HA Test Bed Setup

Proposal

The HA test bed provides the FIO and ADC groups a test environment for evaluating High Availability software and hardware. It can also be used to test outside of the application test environments for operations such as switch over from master to slave and recovery.

The configuration required can model the basic building blocks of the SC4 infrastructure proposal, namely

• Two machines in master/slave mode with the slave monitoring the master's application. The resources is shared such that a switch from the master to slave involves taking over the shared disk and shared network address.

• Two machines running a MySQL database performing replication.

The applications to be tested are

• MyProxy
• CE
• RB

Overview

Hardware Required

Component	Number	Purpose
Servers	2	Master and Slave. The servers do not need to be high end or ultra reliable but should be running the same OS level as the production systems. At least 20GB internal disk is required. Mirroring of internal disks is not necessary.
Network Interfaces	4	Two networks are required. One for the standard access, the other for monitoring purposes
FC HBAs	2 cards 1-2 ports/card	Fibre Channel cards for master and slave to connect to the FC switch or disks. Ideally, these would be two port cards and the same as the version used in production but this is not essential since it is expected that the HBA drivers will perform in a similar fashion
FC Cables	2-4	Cables for HBA to switch or disks
FC Switch	1 with at least 4 ports	Connection from machines to disks. The switch will be logically split in two with zoning to simulate two switches
Disk Array	1-2	One disk array would be sufficient for many of the tests. Two arrays would be simulated using 2 LUNs on the same device

Software Required

A basic lxdev like configuration would be sufficient with SLC3 to build the base service.

The following define the other components used and their setup / configuration.

Component	Description
Linux-HA	HaTestLinuxHaSetup
Fibre Channel	HaTestFcSetup for Fibre Channel HBA and Disks

Resources

The following resources would be required for the setup and development of the deliverables

Resource	Time (days)	Activity
Andras Horvath	15	HA Setup RPM packaging Switchover scripts
Tim Bell	15	Quattor/NCM components Lemon Sensors Working Instructions Test Plan !MyProxy install and test
Thorsten Kleinwort	2	Co-ordination of machines setup and installation
FIO System Administrator	1	Linux installation of machines
Tim Wibley	Physical Installation of machines
David Asbury	2	Fibre Channel Switch Setup
German Cancio	0.5	Consulting on appropriate Quattor parameters

Deliverables

The test will produce the following deliverables

• A hardware recommendation for the Grid SC4/production services
• Validated levels of drivers for HBAs and disk arrays
• A set of middleware packaged as RPMs which will provide the basic HA services
• Quattor/NCM components and PAN variables for configuring master/slave servers, FC connections and HA.
• Lemon sensors for Linux HA to report alarms (switch) and monitor state (master/slave)
• Switch over scripts for shared disks and network
• Working Instructions for administrators for unplanned and planned switchover
• A test plan for regression testing in the event of new hardware/software
• Certification of at minimum the MyProxy configuration within an HA environment

MySQL is currently not included in the scope since this requires agreement for the organisation structure and services for this database.

A request to test out DRBD has also been received. This will be investigated after the initial tests have been completed since it may be an interesting option for the less critical services.

Tasks

Setup log

see HaTestBedSetupLog

Configuration Details

Machine	CERN Network	Monitoring
lxdev13	137.138.8.149	192.168.0.151
lxdev14	137.138.8.150	192.168.0.152
qlogic switch	N/A	192.168.0.150
service IP	none yet	192.168.0.160

RPM	Version	Purpose
sansurfer	5.00.01-1.cern	Graphical interface for SAN switch management
heartbeat	2.0.1-1.cern.1	Core services for linux-ha
heartbeat-pils	2.0.1-1.cern.1	Plugin framework for linux-ha
heartbeat-stonith	2.0.1-1.cern.1	STONITH services - not used yet (needs special hw) but depended upon

-- TimBell - 14 Sep 2005

-- AndrasHorvath - 25 Oct 2005