Configuring an active-passive HA Apache Web Server Cluster
Introduction
The purpose of this article is to describe the steps required to implement a simple active-passive Web server HA cluster using the Heartbeat package (availble from http://www.linux-ha.org). We're not here to discuss clustering theory, this is a How-to, not a Why-to-do, and as always assumes that you know what you're out to achieve.
All commands should be issued as the root user.
Preparation
You need three machines running a supported flavour of Linux. I decided to go with CentOS 3.5, running as three Virtual Machine instances (on VMWare GSX Server running on one of my SUSE servers). Of course, if this were not just a proof-of-concept, I'd be prudent to run each instance on a different VMWare server, otherwise the whole purpose of HA would be lost. I installed a minimal installation of CentOS 3.5 on all three servers. One of these servers would become my NFS server, so ensure that the nfs-utils package is installed. The other two servers will form the active-passive httpd cluster, so ensure that they are both as alike as possible (partition sizes, installed packages, etc). As we are setting up an httpd cluster, make sure that the httpd package is installed on your two web server hosts. We'll also need to make sure that net-snmp-libs are installed on both cluster nodes as this will be required by the heartbeat installation.
To summarise the configuration of the Krusty HA cluster:
Host: krustynfs eth0 IP: 192.168.0.100 Type: NFS Server Host: krustyha1 eth0 IP: 192.168.0.101 Type: HTTPD node Host: krustyha2 eth0 IP: 192.168.0.102 Type: HTTPD node
Our httpd service itself will listen for requests on IP 192.168.0.103 - this IP will failover between the two nodes as required. We will be using heartbeat across the active interfaces of the two cluster nodes - dedicated interfaces would obviously be better (or via a null-modem serial connection) but for the purposes of our test cluster this will suffice. Ethernet connectivity will be checked by pinging our gateway, which in our case is 192.168.0.1.
Configuration of Shared Storage
OK, once all the prerequisites are met, and we have our base OS installed on all three nodes, plus the nfs-utils and httpd packages where appropriate. Perform basic connectivity tests between the hosts and make sure that you can ping from one host to another. Make sure that either DNS is configured or that the /etc/hosts file is correctly populated on each host.
We will be configuring krustynfs as an NFS server, which will form our DocumentRoot and provide shared storage for the two cluster nodes. First, we must create a directory on the NFS server which will become the shared storage location, and host the content to be served by our Apache nodes.
krustynfs# mkdir -p /www/html
Now, we can edit our /etc/exports file, to make the /www directory available to our cluster nodes
krustynfs# cat /etc/exports
/www 192.168.0.101(rw,no_root_squash)
/www 192.168.0.102(rw,no_root_squash)
Next make sure that nfs is configured to start automatically, and start the service.
krustynfs# chkconfig --level 345 nfs on
krustynfs# service nfs start
If NFS is already running, you can use exportfs to force nfsd to re-read the /etc/exports file
krustynfs# /usr/sbin/exportfs -ra
On both the HA nodes, krustyha1 and krustyha2, add the filesystem /www to the /etc/fstab file. I also created a mountpoint /www on the two nodes
krustyha1# mkdir /www
krustyha1# sed -n '$p' /etc/fstab
krustynfs:/www /www nfs noauto,rw,hard 0 0
krustyha2# mkdir /www
krustyha2# sed -n '$p' /etc/fstab
krustynfs:/www /www nfs noauto,rw,hard 0 0
In future code snippets, I will not duplicate code where the changes made are the same on both nodes.
We will soon configure heartbeat to mount this filesystem. It will only be mounted from one node at any one time.
Populate your /www on the NFS server with your web content.
Configuration of heartbeat
First, install heartbeat on the three nodes. This comprises of (a minimum of) three packages. I downloaded pre-built RHEL3 RPMs from UltraMonkey.org.
On each node, perform the following:
krustyha1# rpm -ivh heartbeat-pils-1.2.3-1.rh.el.3.0.i386.rpm
krustyha1# rpm -ivh heartbeat-stonith-1.2.3-1.rh.el.3.0.i386.rpm
krustyha1# rpm -ivh heartbeat-1.2.3-1.rh.el.3.0.i386.rpm
The configuration I've used here is very specific for my needs. There is plenty of documentation over at the Linux-HA website if you need help - there are even sample configuration files available.
The first thing to configure is the /etc/ha.d/authkeys file. As I'm running my cluster on a private LAN, I'm not too bothered about authentication between the cluster nodes, so I'm using the simplest scheme - crc. md5 is also available, and if you want a secure system, sha1.
Let's have a look at /etc/ha.d/authkeys on one of our nodes:
krustyha1# cat /etc/ha.d/authkeys
auth 2
2 crc
This keys must be the same on both nodes. See the documentation on Linux-HA.org for further information on configuring your keys.
The next step is creating and configuring the /etc/ha.d/ha.cf file. This file configures our clusters logging, defines our cluster members, and sets up general configuration options. Our sample configuration is as follows, most of it is very self-explanatory.
krustyha1# cd /etc/ha.d
krustyha1# vi ./ha.cf
krustyha1# cat ./ha.cf
debugfile /var/log/hadebug
logfile /var/log/halog
logfacility local0
node krustyha1
node krustyha2
keepalive 1
deadtime 5
bcast eth0
ping 192.168.0.1
auto_failback no
respawn cluster /usr/lib64/heartbeat/ipfail
Notice that we haven't actually defined our services in this file, these are defined in /etc/ha.d/haresources.
krustyha1# cat /etc/ha.d/haresources
krustyha1 192.168.0.103 Filesystem::krustynfs:/www::/www::nfs::rw,hard httpd
You can see that we are defining a few things here. First, we define our cluster members hostname. Then, we define our shared cluster IP. Next, we configure our clustered resources. We have two such resources here, our NFS filesystem, plus the httpd service itself. So upon booting this cluster node, if it's the primary (active) node, the shared IP will be configured, the NFS filesystem mounted, and then httpd will be started.
/etc/ha.d/ha.cf should be the same on both cluster nodes.
/etc/ha.d/haresources will differ only in the hostname of the cluster member that the file is configured on.
Thankfully, heartbeat comes with sample apache cluster resource start/stop scripts. These do, however, require slight modification to work with the default Apache installation under CentOS 3.5.
Let's modify the default file. We need to change the script filename (as we used httpd in our haresources file) and the path to our Apache configuration file (the DEFAULT_NORMCONFIG variable).
krustyha1# cp -p /etc/ha.d/resource.df/apache /etc/ha.d/resource.d/httpd
krustyha1# vi /erc/ha.d/resource.d/httpd
DEFAULT_NORMCONFIG="/etc/httpd/conf/httpd.conf"
Do this on both nodes. Then, we need to set up our Apache environment. On both cluster nodes, set your DocumentRoot to point to the NFS mounted html directory:
krustyha1# grep DocumentRoot /etc/httpd/conf/httpd.conf
DocumentRoot "/www/html"
Now, we're ready to start heartbeat on the Active node (krustyha1). First, check that heartbeat is configured to start automatically on system boots
krustyha1# chkconfig --list heartbeat
heartbeat 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Then start heartbeat. I like to tail -f /var/log/halog in another terminal to ensure everything starts correctly:
krustyha1# /etc/init.d/heartbeat start
Now we can check that the shared IP has been set up correctly, our NFS filesystem has been mounted correctly and httpd is started:
krustyha1# /sbin/ifconfig eth0:0 | grep inet
inet addr:192.168.0.103 Bcast:192.168.0.255 Mask:255.255.255.0
krustyha1# mount | grep www
krustynfs:/www on /www type nfs (rw,hard,addr=192.168.0.100)
krustyha1# ps -ef | grep "[h]ttpd" | wc -l
9
Cool - we can now hop onto the passive node and start heartbeat
krustyha2# /etc/init.d/heartbeat start
We should notice that none of the above tests pass on the passive node - these shared IP configuration, NFS mount and httpd service startup will only occur during failover. Verify everything by browsing to http://192.168.0.103 - if we can browse to our shared IP, success!
OK, now you want to test failover, right? Well that's easy, just bring eth0 down on the active node (or stop the httpd service). On the passive node (which will soon become active!) monitor /var/log/halog to ensure that everything comes up correctly:
krustyha1# /sbin/ifconfig eth0 down
krustyha2# tail -f /var/log/halog
Failover will occur, and the cluster will be verified - continue browsing your illustrious content on http://192.168.0.103 like nothing happened!
Conclusion
Even though this is a fairly simple cluster configuration, it should serve as a good practical introduction to HA on Linux.
Cheers
Kevin Waldron
kevin@zazzybob.com
Disclaimer! - This article is provided for guidance only, and does not replace the relevant official documentation and manuals. I will not be held liable for any hosed systems and/or data.
