Material needed
- Data Switch : HP 3500
- Data Switch : HP 2650
- Farm server : Dell SC1425
- Farm nodes : Dell SC1950
- Power cables :
- one special power cable for the Data switch
- 2 normal power cables for the control switch and the server
- normal power cables for the nodes
- Network cables :
- Control cables (yellow) :
- 3 x 0,5 m
- 2 x 2 m for the first 2 nodes
- 2 x 1,5 m for node 3 & 4
- Data cables (green) :
- 12 x 1 m for the storage uplink and one per subnet (11 for 30 nodes)
- 2 x 2 m for the first 2 nodes
- 2 x 1,5 m for node 3 & 4
Install the control & Data switches
Take note of the mac addresses of the switches and add them to the dns and the dhcp server.
Their name will be sw-d1
xxx-c1 for the control switch and sw-d1
xxx-d1 for the data switch ,
xxx beein the rack name (eg E04 -> sw-d1e04-c1 & sw-d1e04-d1).
Their IP addresses will be composed like this : the first three digit will define the rack subnet (eq 10.130.150 for rack
D1E06) as seen on
that page and the last digit beeing 251 for the control switch and 252 for the data switch.
Install the nodes
Cable the control switch
Switch Port |
Go to |
Comment |
1.. |
nodes 1st interface |
|
47 |
Patch Panel 1 |
Farm Server |
Gb1 |
SW-D1xxx-D1 port 48 |
Data switch control link |
Gb2 |
Patch Panel 9 |
Up link |
Cable the data switch
Switch Port |
Go to |
Comment |
1.. |
nodes 2nd interface |
|
35-46 |
Patch Panel 11-22 or 3-8 |
Central DAQ switch |
47 |
Patch Panel 2 |
Storage Uplink |
Connect the uplink in D2 to the main control switch
The patch panel is in rack
D2C06.
On the Control Force 10 switch, the connections are mainly made on the line card 0.
Power on the switches
You should now be able to connect to them with telnet.
Update the Data switch firmware
Connect to the data switch with telnet.
Issue the following commands :
copy tftp flash 10.128.16.26 K_12_14.swi
boot system flash
Configure the Data switch
Run the script /admin/scripts/networkscripts/gen_data_switch_vlan.sh d1
xxx > d1
xxx.txt
That will generate the text file with all commands to configure the switch for the rack you pass as parameter.
Then run the script /admin/scripts/networkscripts/exectelnet.pl sw-d1
xxx-d1 admin
password d1
xxx.txt
That will execute the config file on the given switch.
Check that the DEFAULT_VLAN only list ports 47 & 48 as untagged and no other ports (
to avoid network loops).
Install the Nodes
Cable up the first interface to the control switch and the 2nd to the data switch. Node on will go to port 1 and so on.
Take note of their Serial Number and Mac address. They'll be needed later for quattor configuration.
Install the Server
The servers are installed in racks
D2B01 (for rack row
D1E) to B05 (for rack row
D1A).
Register the new server in the dns (with the ipmi interface). Add the ipmi interface to the dhcp server on dns-sx-01.
Copy the template of an existing server and modify the name, mac and ip addresses and the list of nodes.Add the template to cdb.
Run
sudo aii-shellfe --configure server_hostname
and
sudo aii-shellfe --install server_hostname
I hope to have soon a script to generate the server and node templates and configure the dns and dhcp services.
You can now connect and start the server. The installation should work.
Connections & configuration on SW-DAQ-01
The script
/admin/scripts/networkscripts/generate_ips.pl
with the rackname (eg.:
D1E10) and the subnet number as parameters will give the following informations :
- IP number of the data interface of the computers in that subnet.
- For the link between the data switch in the rack and SW-DAQ-01 :
- IP number of that link on the data switch in the rack
- IP number of the port on SW-DAQ-01
The IP numbers for the nodes will be used later in the nodes templates.
The data switch is already configured.
Connect a patch cable for that subnet (When all line cards will be installed, a page will document on which Line card/port each D1 subnet will be connected to. In the meantime, we'll have to connect it to the `next free connection`).
Telnet to SW-DAQ-01 do (eg. for link gi 0/7 and IP number :192.168.204.70
sw-daq-01#conf
sw-daq-01(conf)#int gi 0/7
sw-daq-01(conf-if-gi-0/7)#no switchport
sw-daq-01(conf-if-gi-0/7)#ip address 192.168.204.70/26
sw-daq-01(conf-if-gi-0/7)#mtu 9252
sw-daq-01(conf-if-gi-0/7)#ip mtu 9234
sw-daq-01(conf-if-gi-0/7)#no shutdown
sw-daq-01(conf-if-gi-0/7)#exit
sw-daq-01(conf)#exit
sw-daq-01#copy running-config startup-config
sw-daq-01#exit
conf
int gi 0/7
no switchport
ip address 192.168.204.70/26
mtu 9252
ip mtu 9234
no shutdown
exit
exit
copy running-config startup-config
exit
Storage Uplink
In rack
D2B*, connect the socket coming from the socket 2 of the D1 rack to the Storage switch.
Eg for rack
D1E10 :
Connect
D2B1C18 (coming from
D1E10A02) to port 10 (corresponding to the rack number in the row) on switch SW-D2B01-S1.