High Availability Implementation for MyProxy

Configuration

The CERN requirements for the MyProxy service requires a highly available configuration. As discussed in PxNotes, the high availability functions are available in the standard MyProxy implementation using myproxy_replicate. This creates a read-only replica of the data which supports retrieve operations only. This can cover the retrieve operations but will not cover the user oriented actions such as init or destroy.

To provide a full high availability function, the following approach was taken

• Master/Slave set up using Linux-HA and shared IP service address
• Master stores data in /var/proxy and replicates using myproxy_replicate to slave in /var/proxy.slave
• Master rsync's data from /var/proxy to the slave /var/proxy directory
• The slave myproxy server is started in slave mode to read from /var/proxy.slave (i.e. read-only mode)
• In the event of master failure as detected by Linux-HA, the daemon is stopped on the slave and then restarted with the read-write copy from /var/proxy

Using Linux-HA with a small myproxy resource script (start/stop/monitor/status) provide this function. The take over time is around 2 seconds following detection of a failure. There may be a substantial delay between occurrence of failure and detection. If further work on the client configuration is made such that a replica server can be queried, this window will be covered by the replica.

The HA configuration has been implemented as follows

In the event of a failure or an operator initiated switch for planned maintenance, the configuration is changed

• Service IP now points to slave server
• Slave myproxy started with /var/myproxy as repository (which was being received via rsync from master)
• Master adopts slave role (if it is able to)

Conclusion

For the cost of two machines with small disk space, a highly available MyProxy implementation can be made which is resiliant to network, machine and storage failures.

-- TimBell - 05 Oct 2005