How To Create a High Availability Setup with Heartbeat and Reserved IPs on Ubuntu 14.04

Introduction #

Heartbeat is an open source program that provides cluster infrastructure capabilities—cluster membership and messaging—to client servers, which is a critical component in a high availability (HA) server infrastructure. Heartbeat is typically used in conjunction with a cluster resource manager (CRM), such as Pacemaker, to achieve a complete HA setup. However, in this tutorial, we will demonstrate how to create a 2-node HA server setup by simply using Heartbeat and a DigitalOcean Reserved IP.
If you are looking to create a more robust HA setup, look into using Corosync and Pacemaker or Keepalived.

Goal #

When completed, the HA setup will consist of two Ubuntu 14.04 servers in an active/passive configuration. This will be accomplished by pointing a Reserved IP, which is how your users will access your services or website, to point to the primary, or active, server unless a failure is detected. In the event that the Heartbeat service detects that the primary server is unavailable, the secondary server will automatically run a script to reassign the Reserved IP to itself via the DigitalOcean API. Thus, subsequent network traffic to the Reserved IP will be directed to your secondary server, which will act as the active server until the primary server becomes available again (at which point, the primary server will reassign the Reserved IP to itself).

Note: This tutorial only covers setting up active/passive high availability at the gateway level. That is, it includes the Reserved IP, and the load balancer servers—Primary and Secondary. Furthermore, for demonstration purposes, instead of configuring reverse-proxy load balancers on each server, we will simply configure them to respond with their respective hostname and public IP address.

To achieve this goal, we will follow these steps:

Create 2 Droplets that will receive traffic
Create Reserved IP and assign it to one of the Droplets
Create DNS A record that points to Reserved IP (optional)
Install Heartbeat on Droplets
Configure Heartbeat to Run Reserved IP Reassignment Service
Create Reserved IP Reassignment Service
Test failover

Prerequisites #

In order to automate the Reserved IP reassignment, we must use the DigitalOcean API. This means that you need to generate a Personal Access Token (PAT), which is an API token that can be used to authenticate to your DigitalOcean account, with read and write access by following the How To Generate a Personal Access Token section of the API tutorial. Your PAT will be used in a script that will be added to both servers in your cluster, so be sure to keep it somewhere safe—as it allows full access to your DigitalOcean account—for reference.
In addition to the API, this tutorial utilizes the following DigitalOcean features:

Reserved IPs
Metadata
User Data (Cloud-Config scripts)

Please read the linked tutorials if you want to learn more about them.

Create Droplets #

The first step is to create two Ubuntu Droplets in the same datacenter, which will act as the primary and secondary servers described above. In our example setup, we will name them “primary” and “secondary” for easy reference. We will install Nginx on both Droplets and replace their index pages with information that uniquely identifies them. This will allow us a simple way to demonstrate that the HA setup is working. For a real setup, your servers should run the web server or load balancer of your choice.
Create two Ubuntu 14.04 Droplets, primary and secondary, with this bash script as the user data:
Example User Data

#!/bin/bash

apt-get -y update
apt-get -y install nginx
export HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname)
export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
echo Droplet: $HOSTNAME, IP Address: $PUBLIC_IPV4 > /usr/share/nginx/html/index.html

This will install Nginx and replace the contents of index.html with the droplet’s hostname and IP address (by referencing the Metadata service). Accessing either Droplet via its public IP address will show a basic webpage with the Droplet hostname and IP address, which will be useful for testing which Droplet the Reserved IP is pointing to at any given moment.

Create a Reserved IP #

In the DigitalOcean Control Panel, click Networking, in the top menu, then Reserved IPs in the side menu.

Assign a Reserved IP to your primary Droplet, then click the Assign Reserved IP button.
After the Reserved IP has been assigned, check that you can reach the Droplet that it was assigned to by visiting it in a web browser.

http://your_reserved_ip

You should see the index page of your primary Droplet.

Configure DNS (Optional) #

If you want to be able to access your HA setup via a domain name, go ahead and create an A record in your DNS that points your domain to your Reserved IP address. If your domain is using DigitalOcean’s nameservers, follow step three of the How To Set Up a Host Name with DigitalOcean tutorial. Once that propagates, you may access your active server via the domain name.
The example domain name we’ll use is example.com. If you don’t have a domain name right now, you should use the Reserved IP address instead.

Install Heartbeat #

The next step is to install Heartbeat on both servers. The simplest way to install Heartbeat is to use apt-get:

sudo apt-get update
sudo apt-get install heartbeat

Heartbeat is now installed but it needs to be configured before it will do anything.

Configure Heartbeat #

In order to get our desired cluster up and running, we must set up these Heartbeat configuration files in /etc/ha.d, identically on both servers:

ha.cf: Global configuration of the Heartbeat cluster, including its member nodes
authkeys: Contains a security key that provides nodes a way to authenticate to the cluster
haresources: Specifies the services that are managed by the cluster and the node that is the preferred owner of the services. Note that this file is not used in a setup that uses a CRM like Pacemaker

We will also need to provide a script that will perform the Reserved IP reassignment in the event that the primary Droplet’s availability changes.

Gather Node Information #

Before configuring ha.cf, we should look up the names of each node. Heartbeat requires that each node name matches their respective uname -n output.
On both servers, run this command to look up the appropriate node names:

uname -n

Note the output of the command. The example node names are “primary” and “secondary”, which matches what we named the Droplets.
We will also need to look up the network interface and IP address that each node will use to communicate with the rest of the cluster, to determine which nodes are available. You may use any network interface, as long as each node can reach the other nodes in the cluster. We’ll use the public interface of our Droplets, which happens to be eth0.
On both servers, use this command to look up the IP address of the eth0 interface (or look it up in the DigitalOcean Control Panel):

ip addr show eth0

ip addr show eth0 output:2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 04:01:76:a5:45:01 brd ff:ff:ff:ff:ff:ff
    inet 104.236.6.11/18 brd 104.236.63.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.17.0.28/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::601:76ff:fea5:4501/64 scope link
       valid_lft forever preferred_lft forever

Note the IP address of the network interface (highlighted in the example). Be sure to get the IP addresses of both servers.

Create ha.cf File #

On both servers, open /etc/ha.d/ha.cf in your favorite editor. We’ll use vi:

sudo vi /etc/ha.d/ha.cf

The file should be new and empty. We need to add the network interfaces and names of each node in our cluster.
Copy and paste this configuration into the file, then replace the respective node names and IP addresses with the values that we looked up earlier. In this example, primary’s IP address is 104.236.6.11 and secondary’s IP address is 104.236.6.22:

node primary
ucast eth0 104.236.6.11
node secondary
ucast eth0 104.236.6.22

Save and exit. Next, we’ll set up the cluster’s authorization key.

Create authkeys File #

The authorization key is used to allow cluster members to join a cluster. We can simply generate a random key for this purpose.
On the primary node, run these commands to generate a suitable authorization key in an environment variable named AUTH_KEY:

if [ -z "${AUTH_KEY}" ]; then
  export AUTH_KEY="$(command dd if="/dev/urandom" bs=512 count=1 2>'/dev/null' 
      | command openssl sha1 
      | command cut --delimiter=" " --fields=2)"
fi

Then write the /etc/ha.d/authkeys file with these commands:

sudo bash -c "{
  echo auth1
  echo 1 sha1 $AUTH_KEY
} > /etc/ha.d/authkeys"

Check the contents of the authkeys file like this:

sudo cat /etc/ha.d/authkeys

It should like something like this (with a different authorization key):

/etc/ha.d/authkeys example:auth1
1 sha1 d1e6557e2fcb30ff8d4d3ae65b50345fa46a2faa

Ensure that the file is only readable by root:

sudo chmod 600 /etc/ha.d/authkeys

Now copy the /etc/ha.d/authkeys file from your primary node to your secondary node. You can do this manually, or with scp.
On the secondary server, be sure to set the permissions of the authkeys file:

sudo chmod 600 /etc/ha.d/authkeys

Both servers should have an identical /etc/ha.d/authkeys file.

Create haresources File #

The haresources file specifies preferred hosts paired with services that the cluster manages. The preferred host is the node that should run the associated service(s) if the node is available. If the preferred host is not available, i.e. it is not reachable by the cluster, one of the other nodes will take over. In other words, the secondary server will take over if the primary server goes down.
On both servers, open the haresources file in your favorite editor. We’ll use vi:

sudo vi /etc/ha.d/haresources

Now add this line to the file, substituting in your primary node’s name:
/etc/ha.d/haresources

primary floatip

Save and exit. This configures the primary server as the preferred host for the floatip service, which is currently undefined. Let’s set up the floatip service next.

Create Reserved IP Reassignment Service #

Our Heartbeat cluster is configured to maintain the floatip service, which a node can use to assign the Reserved IP to itself, but we still need to create the service. Before we set up the service itself, however, let’s create a script that will assign the Reserved IP, via the DigitalOcean API, to the node that runs it. Then we will create the floatip service which will run the Reserved IP reassignment script.

Create assign-ip Script #

For our example, we’ll download a basic Python script that assigns a Reserved IP to a given Droplet ID, using the DigitalOcean API.
On both servers, download the assign-ip Python script:

sudo curl -L -o /usr/local/bin/assign-ip http://do.co/assign-ip

On both servers, make it executable:

sudo chmod +x /usr/local/bin/assign-ip

Use of the assign-ip script requires the following details:

Reserved IP: The first argument to the script, the Reserved IP that is being assigned
Droplet ID: The second argument to the script, the Droplet ID that the Reserved IP should be assigned to
DigitalOcean PAT (API token): Passed in as the environment variable DO_TOKEN, your read/write DigitalOcean PAT

Feel free to review the contents of the script before continuing.
Now we’re ready to create the floatip service.

Create floatip Service #

To create the floatip service, all we need to do is create an init script that invokes the assign-ip script that we created earlier, and responds to start and stop subcommands. This init script will be responsible for looking up the Droplet ID of the server, via the Droplet Metadata service. Also, it will require the Reserved IP that will be reassigned, and the DigitalOcean API token (the Personal Access Token mentioned in the prerequisites section).
On both servers, add open /etc/init.d/floatip in an editor:

sudo vi /etc/init.d/floatip

Then copy and paste in this init script, replacing the highlighted parts with your DigitalOcean API key and the Reserved IP that should be reassigned:
/etc/init.d/floatip

#!/bin/bash

param=$1

export DO_TOKEN='b7d03a6947b217efb6f3ec3bd3504582'
IP='45.55.96.8'
ID=$(curl -s http://169.254.169.254/metadata/v1/id)

if [ "start" == "$param" ] ; then
  python /usr/local/bin/assign-ip $IP $ID
  exit 0
elif [ "stop" == "$param" ] ; then
  exit 0;
elif [ "status" == "$param" ] ; then
  exit 0;
else
  echo "no such command $param"
  exit 1;
fi

Save and exit.
Make the script executable:

sudo chmod u+x /etc/init.d/floatip

When this floatip service is started, it will simply call the assign-ip Python script and assign the specified Reserved IP to the Droplet that executed the script. This is the script that will be called by the secondary server, to reassign the Reserved IP to itself, if the primary server fails. Likewise, the same script will be used by the primary server, to reclaim the Reserved IP, once it rejoins the cluster.

Start Heartbeat #

Now that Heartbeat is configured, and all of the scripts it relies on are set up, we’re ready to start the Heartbeat cluster!
On both servers, run this command to start Heartbeat:

sudo service heartbeat start

You should see output like this:

Heartbeat output:Starting High-Availability services: Done.

Our HA setup is now complete! Before moving on, let’s test that it works as intended.

Test High Availability #

It’s important to test that a high availability setup works, so let’s do that now.
Currently, the Reserved IP is assigned to the primary node. Accessing the Reserved IP now, via the IP address or by the domain name that is pointing to it, will simply show the index page of the primary server. If you used the example user data script, it will look something like this:

Reserved IP is pointing to primary serverDroplet: primary, IP Address: 104.236.6.11

This indicates that the Reserved IP is, in fact, assigned to the primary Droplet.
Now, let’s open a terminal and use curl to access the Reserved IP on a 1 second loop. Use this command to do so, but be sure to replace the URL with your domain or Reserved IP address:

while true; do curl http://example.com; sleep 1; done

Currently, this will output the same Droplet name and IP address of the primary server. If we cause the primary server to fail, by powering it off or stopping the Heartbeat service, we will see if the Reserved IP gets reassigned to the secondary server.
Let’s power off the primary server now. Do so via the DigitalOcean Control Panel or by running this command on the primary server:

sudo poweroff

After a few moments, the primary server should become unavailable. Pay attention to the output of the curl loop that is running in the terminal. You should notice output that looks like this:

curl loop output:Droplet: primary, IP Address: 104.236.6.11
...
curl: (7) Failed to connect to example.com port 80: Connection refused
Droplet: secondary, IP Address: 104.236.6.22
Droplet: secondary, IP Address: 104.236.6.22
...

That is, the Reserved IP address should be reassigned to point to the IP address of the secondary server. That means that your HA setup is working, as a successful automatic failover has occurred.
You may or may not see the Connection refused error, which can occur if you try and access the Reserved IP between the primary server failure and the Reserved IP reassignment completion.
Now, you may power on your primary Droplet, via the DigitalOcean Control Panel. Because Heartbeat is configured with the primary Droplet as the preferred host to run the Reserved IP reassignment script, the Reserved IP will automatically point back to the primary server as soon as it becomes available again.

Conclusion #

Congratulations! You now have a basic HA server setup using Heartbeat and a DigitalOcean Reserved IP.
If you are looking to create a more robust HA setup, look into using Corosync and Pacemaker or Keepalived.
If you want to extend your Heartbeat setup, the next step is to replace the example Nginx setup with a reverse-proxy load balancer. You can use Nginx or HAProxy for this purpose. Keep in mind that you will want to bind your load balancer to the anchor IP address, so that your users can only access your servers via the Reserved IP address (and not via the public IP address of each server).