High Availability

While the synchronization of IKE state including sequence numbers is realistic between two nodes, exchanging state information of an ESP security association is difficult.

Synchronizing the state for each processed IPsec packet will put a high load on the nodes. Synchronization after a certain amount of packets and/or after a certain timeout can reduce the load, but will make fail-over handling much more difficult, as we have to guess on the taking-over node how many packets the failing node has processed but could not synchronize before it failed.

Another problem with such an approach is that there is no way of doing load sharing between nodes. An SA is strictly bound to a single node until the event of a failure.

Another approach to consider is requesting state information from the client. In the event of a failure, the taking-over node can request sequence numbers from the client. But this approach has the same deficiencies as previously discussed. Further, it requires extensions to the IKE protocol between client and gateway, making existing implementations incompatible with this approach.

All nodes in a ClusterIP-based setup act under a single virtual IP address. The nodes spoof ARP requests with a multicast MAC address. This will make the switch forward the packet to each node in the cluster.

The received packet is associated to a segment by calculating a hash value of it. In the simplest setup, the source address is hashed and the hash value modulo the number of segments results in the responsible segment number. Each segment is handled by exactly one node in the cluster.

The node responsible for the packet will pass it to upper layers, whereas all others just drop the packet in the netfilter code. Depending on the hash value, e.g. TCP connections are kept on the same node. If a node fails, a remaining node will take over the segment and process packets for it.

While the ClusterIP module itself is not designed to handle IPsec traffic or even act as a forwarding router, the principle of ClusterIP is. If the IKE daemons in the cluster can synchronize the IKE state and the basic IPsec SA state without sequence numbers, a modified ClusterIP module can do the rest:

The plugin registers itself at several hooks in the daemon. These hooks are used for notifications about SA state changes and push information to the plugin. The following hooks are used:

The plugin registers its hook functions at the daemon bus. These hooks are sufficient to synchronize all IKE and CHILD SAs with all the state required to do a fail-over of IKE and ESP SAs.

The hook functions collect the required synchronization data and prepare messages to be sent to other nodes in the cluster. Messages are sent in unencrypted UDP datagrams, sent and received on port 4510. As these messages contain sensitive key material, securing the messages by IPsec is recommended.

No packet acknowledge/retransmit scheme is currently implemented. The cluster needs a reliable network with very few packet losses. It might be necessary to use a more reliable transport protocol in the future, especially if nodes start to drop packets due to an overloaded CPU.

Messages contain a protocol version, a message type and different attributes. The following synchronization message types are currently defined:

Received synchronization messages are parsed, mirrored IKE and CHILD_SAs are created from this information. Mirrored CHILD_SAs do not differ from normally exchanged ones. They are installed in the kernel and handle packets if ClusterIP feels responsible for it.

IKE_SAs are installed in a special PASSIVE state. They do not handle traffic but accept state changes from sync messages only. PASSIVE IKE_SAs are managed in the IKE_SA manager as any other SA and are accessible e.g. through swanctl --list-sas.

Key derivation is repeated on mirrored SAs the same way as it is done on the real SAs. This allows the reuse of existing installation routines and the ha plugin to be very unobtrusive.

In addition to the synchronization messages, the HA plugin uses control messages to notify about segment changes and optionally messages for simple monitoring functions:

The take/drop messages are sent to notify other nodes about changes done by the daemon automatically or the administrator manually. The receiving node will automatically do the opposite action to handle all segments exactly once.

If heartbeat monitoring is enabled, the STATUS message is periodically sent. This allows to detect the activity of the remote node and take over segments the remote node is not serving. It also implements node failure detection for simple errors.

If a replacement of a failing node is installed, reintegration of the node can be sped up by sending the resynchronization message. The active node will start resyncing all SAs, allowing the administrator to rebalance the load distribution in the cluster afterwards.

In the failover case, responsibility for complete ClusterIP segments are moved from one node to another. Responsibility for a segment can be enabled or disabled on each node. For this purpose the plugin uses the same hashing algorithm to calculate responsibility based on the source IP address.

If a segment is activated, the plugin searches for IKE_SAs in this segment and sets the state of all PASSIVE IKE_SAs to ESTABLISHED. No further action is required: The daemon handles the IKE_SA as every other one and sends out synchronization messages for state changes.

On segment deactivation, the plugin searches for IKE_SAs in the ESTABLISHED state in these segments and sets the state to PASSIVE.

CHILD_SAs are completely unaffected from activation and deactivation: They are always active and handle traffic assigned by ClusterIP.

To re-integrate a failed node into a cluster, the node needs state information from scratch. If all the required state has been synced, the reintegrated node can be used as failover node again. Segments can be activated on the reintegrated node only after all required state has been exchanged.

Each node caches IKE_SA specific messages locally for all IKE_SAs currently active. If a different node wants to reintegrate, the active node pushes the message cache to the new node. This allows the reintegrated node to reestablish the state for all IKE_SAs.

Synchronizing CHILD_SAs is not possible using the cache, as the messages do not contain sequence number information managed in the kernel. To reintegrate a node, the active node initiates rekeying on all CHILD_SAs. The new CHILD_SA will be synchronized, starting with fresh sequence numbers in the kernel. CHILD_SA rekeying is inexpensive, as it usually does not include a DH exchange.

The configuration of the extended ClusterIP module is similar to a default ClusterIP setup. For a traffic forwarding IPsec gateway, a cluster usually needs an internal virtual IP/MAC address and an external virtual IP/MAC address on each node.

This example installs the virtual IP 192.168.0.200 on interface eth0 and adds a corresponding ClusterIP rule. ClusterIP rules are always added to the iptables INPUT chain. To get the same result for segment responsibility calculation in the kernel and the ha plugin, the sourceip hashmode and a hash init value of 0 must be used (default).

The --total-nodes option must match the configuration of the ha plugin and all nodes require the same virtual IP/MAC address and ClusterIP configuration.

ClusterIP requires the --local-node option to be present. While the ha plugin reassigns segment responsibility during daemon startup, it is recommended to use zero, so that a node booting up does not process any packets until the ha plugin tells it to do so.

Using an address pool to assign virtual IP addresses to clients is a little more complicated in a cluster. Using a central database based pool should be no problem, as long as you have a fail-save MySQL database cluster.

In-memory address pools do not provide any synchronization: you don’t want to assign the same address from two different nodes to different clients. One option would be to split up e.g. a 10.0.2.0/23 pool into a 10.0.2.0/24 and a 10.0.3.0/24 pool and use each pool on one node. But this approach has its limitations: If one node goes down and gives away an IKE_SA, the pool state is lost. If the failed node reintegrates into the cluster and takes over the same IKE_SA, the virtual IP will be in use again. But the reset pool will not reflect this fact and may reassign the same address to another client.

As an alternative strongSwan provides an HA-enabled in-memory address pool. The pool is simple to configure but only hands out addresses the local node is responsible for (using segment calculations). Further it reserves addresses in the pool for IKE_SAs handled by a different node.

To configure an HA enabled in-memory address pool, add a pools subsection in the ha section of strongswan.conf and define the required pools. Use exactly the same pool configuration on both nodes. There is no need to split up the pools with the HA-enabled implementation!

To use the pools, reference them in the connections definitions of swanctl.conf, e.g.:

Changing the segment responsibility is done for the daemon, where it will propagate the changes in segment responsibility to the kernel.

The ha plugin uses a very similar interface for segment control as ClusterIP. Instead of a proc entry, it uses a FIFO located at /var/run/charon.ha. Echoing +1/-1 will activate/deactivate responsibility for segment 1 while an additional command *3 will enforce a resynchronization by triggering a rekey of all SAs in segment 3.