WARNING: This web is not used anymore. Please use PDBService.Multipathing instead!
 

Multipathing with Qlogic HBAs - failover and LUN load balancing

This is a short guide to the configuration of multipathing with QLogic HBAs for the physics database services at CERN. The key software and hardware components are: Oracle 10gR2, RAC, ASM, Linux RHEL 3, dual CPU Intel servers, SAN network, SATA disks in FC arrays (see also Reference Architecture). SAN and FC (2 Gb) infrastructure: Qlogic dual-ported HBAs (QLA2342), 2 x Qlogic SAN switches, dual-ported infortred FC arrays (A16F-S1211).

Note: tested with RHEL 3 U6, QLA2342 driver=7.05.00-RH1-f0, firmware=3.03.11, bios=1.47. SANsurfer=2.0.30b69

Under the term multipathing, in this document, we refer to a SAN configuration for path failover redundancy and load balancing:

  • redundant path to access the storage arrays
  • failover at the LUN level for failed path from the sevrer to the SAN switch (improved HA)
  • load balancing at the LUN level (improved scalability and performance)


*Note regarding RHEL 4*: The qla2xxx_conf module is missing from 'RHEL 4 rpms' at least up to update 4. The module can be downloaded and compiled from the qlogic website (tested, it works). Without qla2xxx_conf multipathing here can not be configured on RHEL4. Device mapper multipathing can be used instead (it is a completely different implementation though, see DB installation procedure in this wiki).

OS configuration files

Multipathing for Qlogic HBAs is configured via /etc/modules.conf, and /etc/qla2300.conf. Moreover it is important to check that both qla2300_conf and qla2300 kernel modules are loaded (ex: lsmod |grep qla23)

After LUN load balancing configuration the /etc/modules.conf files should contain the following entry:

alias scsi_hostadapter0 qla2300_conf
alias scsi_hostadapter1 qla2300
options qla2300 ConfigRequired=1 ql2xuseextopts=1 ql2xfailover=1 qlport_down_retry=1 ql2xretrycount=2

FC and SAN configuration

  • Change the PID number of the storage array controllers for each array use the same PID for both controllers.
  • Configure zoning in both SAN switches and make the storage arrays visible via both HBA ports of the linux servers.

Qlogic HBA configuration

The /etc/qla2300.conf configuration can be done using Qlogic SANsurfer for HBA (downloadable from the Qlogic website).

  • install the agent of SANsurfer for HBA on the nodes you need to configure
  • modprobe qla2300 if needed
  • disable autostartup of the qlogic agent
    • chkconfig qlremote off
    • /etc/init.d/qlremote stop
    • rmmod qla2300; modprobe qla2300
    • /etc/init.d/qlremote start
  • Install and run the SANsurfer management console and connect to the node you want to configure
  • Don't run the node configuration wizard
  • Click the 'configure' icon
  • if needed enable failover ticking a flag under the file menu
    • DEVICE menu
      • CONFIGURE
        • CONFIGURE ALL
      • FAILOVER
        • ENABLE ALL
    • LUN menu
      • ENABLE LUNS
        • ENABLE ALL
      • LOAD BALANCE
        • ALL LUNS
    • Save the configuration (default_pass='config')
  • Check the configuration in /etc/qla2300.conf and /etc/modules.conf.
/etc/init.d/qlremote stop
rmmod qla2300
rmmod qla2300_conf
modprobe qla2300

  • Test the stability of the configuration across reboot
    • Recreate the init ramdisk: rm /boot/initrd-2.4.21-37.ELsmp.img; mkinitrd /boot/initrd-2.4.21-37.ELsmp.img 2.4.21-37.ELsmp (customize with current kernel name)
    • if you are using devlabel, reconfig the mapping before reboot
    • reboot and check that the new configuration is being picked up

Post install - sample qla2300.conf configuration

This is a sample /etc/qla2300.conf configuration file for anode configured with 3 disk arrays, using dual ported HBAs on the server and the array. Note: extra new lines and comments have been added for clarity.

scsi-qla0-adapter-port=210000e08b1481b8\;    -> WWN identifier of the first local HBA port (port 1)

scsi-qla0-tgt-0-di-0-node=200000d0230e0671\;    -> WWNN: storage array controller (array N.1 of 3)
scsi-qla0-tgt-0-di-0-port=210000d0230e0671\;    -> WWNP: storage array 1/3 controller port 1
scsi-qla0-tgt-0-di-0-pid=020cef\;               -> ?
scsi-qla0-tgt-0-di-0-preferred=0000000000000000000000000000000000000000000000000000000000015555\; 
                    -> command related to the activation LUN load balancing
scsi-qla0-tgt-0-di-0-control=00\;               
                    -> storage array N.1 is bound to port 0 (but LUN load balacing makes for 
                       the desired alternate access path)

scsi-qla0-tgt-1-di-0-node=200000d02350000e\;   -> WWNN array N.2/3 
scsi-qla0-tgt-1-di-0-port=210000d02350000e\;   -> WWNP array N.2/3
scsi-qla0-tgt-1-di-0-pid=020e33\;               
scsi-qla0-tgt-1-di-0-preferred=0000000000000000000000000000000000000000000000000000000000015555\;
scsi-qla0-tgt-1-di-0-control=00\;

scsi-qla0-tgt-2-di-0-node=200000d0231e0279\;   -> WWNN array N.3/3 
scsi-qla0-tgt-2-di-0-port=210000d0231e0279\;   -> WWNP array N.2/3
scsi-qla0-tgt-2-di-0-pid=020fe8\;
scsi-qla0-tgt-2-di-0-preferred=000000000000000000000000000000000000000000000000000000000000aaaa\;
scsi-qla0-tgt-2-di-0-control=00\;

scsi-qla1-adapter-port=210100e08b3481b8\;     -> WWN identifier of the first local HBA port (port 2)
scsi-qla1-tgt-0-di-1-node=200000d0230e0671\;  -> WWNN: same as above, array controller (array N.1 of 3)
scsi-qla1-tgt-0-di-1-port=220000d0230e0671\;  -> WWNP: storage array 1/3 controller port 2
scsi-qla1-tgt-0-di-1-pid=010cef\;             -> ?
scsi-qla1-tgt-0-di-1-preferred=000000000000000000000000000000000000000000000000000000000000aaaa\;
scsi-qla1-tgt-0-di-1-control=80\;
                    -> port 1 is failover for storage array N.1 

scsi-qla1-tgt-1-di-1-node=200000d02350000e\;
scsi-qla1-tgt-1-di-1-port=220000d02350000e\;
scsi-qla1-tgt-1-di-1-pid=010e33\;
scsi-qla1-tgt-1-di-1-preferred=000000000000000000000000000000000000000000000000000000000000aaaa\;
scsi-qla1-tgt-1-di-1-control=80\;
scsi-qla1-tgt-2-di-1-node=200000d0231e0279\;
scsi-qla1-tgt-2-di-1-port=220000d0231e0279\;
scsi-qla1-tgt-2-di-1-pid=010fe8\;
scsi-qla1-tgt-2-di-1-preferred=0000000000000000000000000000000000000000000000000000000000015555\;
scsi-qla1-tgt-2-di-1-control=80\;

Post install - sample /proc output

A sample from the /proc/scsi/qla2300/1 file is reported below. The list of SCSI LUNinformation on /proc/scsi/qla2300/2 should be empty. However, the last column in the list reports for each LUN the HBA port that is used. For example '0:0:81' means port 1, while '1:1:81' means port 2. In case of failover these numbers change accordingly (see also below):

SCSI LUN Information:
(Id:Lun)  * - indicates lun is not registered with the OS.
( 0: 0): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,
( 0: 1): Total reqs 5, Pending reqs 0, flags 0x2, 1:1:81,
( 0: 2): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,
( 0: 3): Total reqs 5, Pending reqs 0, flags 0x2, 1:1:81,
( 0: 4): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,
( 0: 5): Total reqs 5, Pending reqs 0, flags 0x2, 1:1:81,
( 0: 6): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,
( 0: 7): Total reqs 5, Pending reqs 0, flags 0x2, 1:1:81,
( 0: 8): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,
( 0: 9): Total reqs 5, Pending reqs 0, flags 0x2, 1:1:81,
( 0:10): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,
( 0:11): Total reqs 5, Pending reqs 0, flags 0x2, 1:1:81,
( 0:12): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,
( 0:13): Total reqs 5, Pending reqs 0, flags 0x2, 1:1:81,
( 0:14): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,
( 0:15): Total reqs 5, Pending reqs 0, flags 0x2, 1:1:81,
( 0:16): Total reqs 5, Pending reqs 0, flags 0x2, 0:0:81,

Post install - sample failover log

In case of LUN failover the following information is displayed in the /var/log/messages file:

..... kernel: qla2x00: FAILOVER device 0 from 210000d0230e0671 -> 220000d0230e0671 - LUN 0b, reason=0x1
..... kernel: qla2x00: FROM HBA 0 to HBA 1

The failover is triggered only when the failed LUN is accessed, while the failback is triggered as soon as the failure is restored:

..... kernel: qla2x00: FAILBACK device 0 -> 200000d0230e0671 LUN 0b
..... kernel: qla2x00: FROM HBA 1 to HBA 0

Configuration changes

If a storage arrays needs to be replaced and/or the perfistent configuration stored in /etc/qla2300.conf needs to be changed, follow this procedure:

/etc/init.d/qlremote stop
mv /etc/qla2300.conf /tmp/qla2300.conf.OLD
vi /etc/modules.conf -> CHANGE in the qla2300 line -> ConfigRequired=0
rmmod qla2300
rmmod qla2300_conf
modprobe qla2300
/etc/init.d/qlremote start

  • use SANSurfer to configure qla2300 conf (load balancing will not be possible at this stage) and save the configuration (as explained above).

/etc/init.d/qlremote stop
vi /etc/modules.conf -> CHANGE BACK in the qla2300 line -> ConfigRequired=1
rmmod qla2300
rmmod qla2300_conf
modprobe qla2300
/etc/init.d/qlremote start

  • use SANSurfer to configure qla2300 conf including load balancing and save the configuration.

/etc/init.d/qlremote stop
rmmod qla2300
rmmod qla2300_conf
modprobe qla2300

  • Recreate the init ramdisk: rm /boot/initrd-2.4.21-37.ELsmp.img; mkinitrd /boot/initrd-2.4.21-37.ELsmp.img 2.4.21-37.ELsmp

Update qla2300_conf and qla_opts utility

  • the persistent LUN configuration (multipathing) is stored in /etc/qla2300.conf, when this file is changed manually, qla_opts need to be run to update the kernel module. The same is true when the kernel version is changed/updated
  • Example: ./qla_opts -w --file=/lib/modules/2.4.21-47.ELsmp/kernel/drivers/addon/qla2200/qla2300_conf.o qla2300_conf

  • RHEL3_qla_opts: qla_opts for RHEL3
  • qla_opts: qla_opts, qlogic utility to update qla2xxx module. Compiled for RHEL 4


History: Version 1, Luca.Canali - AT- cern.ch, February 2006
V 1.1 Added - how to change config, Luca, May 2006
V 1.2 Added - how to update qla2300_conf, Luca, Nov 2006

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatext RHEL3_qla_opts r1 manage 20.1 K 2006-11-20 - 18:09 LucaCanali qla_opts for RHEL3
Unknown file formatext qla_opts r2 r1 manage 15.0 K 2006-11-20 - 17:40 LucaCanali qla_opts for RHEL4
Edit | Attach | Watch | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r17 - 2006-12-20 - LucaCanali
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PSSGroup All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback