Computer Setup

Here the setup of the regular computers and SBCs is described. For a description of the Raspberry Pi setup, please see Raspberry Pi Setup. These are used as a part of the Hygrometer system.

For BL4S, the operating system requirements are not particularly demanding, with only a modern version of linux required. Recently (2019), the swap to CentOS 7 was made so previous system instructions will not be listed. Some tuning of the OS is required, all the instructions can be found below.

In order to run at DESY, the network system has a local network to the beam area. This network is separate from the DESY network and consequently there is no internet access. On this internal network, the computer IPs were set by hand and this is thought to be the reason of some difficulties with running TDAQ. There are however a couple of computers, already in the DESY beam area, which have a double network card, bridging the two networks. Consequently, a similar setup was pursued for BL4S, in particular by equipping one computer with double network cards and connecting one to the DESY network and one to the internal network. In this case additional software is needed beyond the basic OS. A DNS and DHCP server is needed to be listening on the internal network in order to assign IPs to the BL4S machines on the internal network and also allow them to be found through their machine names A firewall should also be enabled and performing NAT on the internal network, in this way allowing the internal computers access to the internet.

Operating System

CentOS 7

Boot the device and enter the BIOS/UEFI in order to configure it. All the BL4S machines so far enter this menu by pressing F2 at boot time. In the boot options, disable all booting from hard drives and boot from the network. save and reboot the computer.

The computer will now download a small file on boot which will ask if we desire to network boot the machine. Press enter to proceed with the network boot. Do not take too long or it will exit a continue with the regular boot procedure.

A new screen will appear with a list of boot options. Choose the option to install CentOS7. It will take a while to boot into a GUI. Once it does, start by choosing the system language, we choose English (United Kingdom).

On the next screen, give it some time to fully load all resources. Then choose the correct date and time. Choose the keyboard layout: the wired K120 keyboard I am using has a English (UK) layout.

Next, the hard drive must be formatted and chosen for installation, choose the installation destination. And choose the option for manual partitioning. On the next screen, choose one of the preexisting partitions and delete it, the system will prompt to delete all associated partitions, choose yes. Once all partitions are deleted, click the option to create the partitions automatically.Click done and accept all changes. Then configure network and hostname. Finally, make sure that under software selection the option Software Development Workstation (CERN Recommended Setup) is selected.

We can now press the Begin Installation button. While the installation takes place, set the root password and create the bl4sdaq account, do not forget to make it administrator. Reboot the computer and enter the BIOS again, bo to the boot order and make sure it is booting CentOS before the network cards. Save and exit to continue boot procedure.

A post-install configuration screen should appear. Accept the default CERN customizations. Verify the network is correctly setup, in particular the machine hostname. Finally, accept the license agreement (if you do). Finally, press finish configuration.

The Operating System is now fully installed, but a couple more steps are needed. Once fully booted, open a terminal and run the command addusercern [username] for each user that should have access to the machine.

Configuration

Some system configuration is desired. Edit the .bashrc file for both the main user and root user and add the line export EDITOR=nano.

Passwordless SSH is necessary for TDAQ to function properly. One machine must create a pair of public and private keys. These will then be used for authentication. Perform these steps as the bl4sdaq user.

Run the command ssh-keygen -t rsa -b 4096 -C "BL4S DAQ Account", accept the default values and do not use a passphrase.

Then create the authorized keys file and set the correct permissions.

cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 640 ~/.ssh/authorized_keys

mkdir .ssh
scp [source]:~/.ssh/authorized_keys ~/.ssh/
scp [source]:~/.ssh/id_rsa ~/.ssh/
scp [source]:~/.ssh/id_rsa.pub ~/.ssh/
chmod 700 ~/.ssh
chmod 640 ~/.ssh/authorized_keys 

The firewall must be stopped for TDAQ:

systemctl stop firewalld
systemctl disable firewalld

Assure kerberos keytab files are ok, cern-get-keytab

The system limits are too low, so they must be increased. See current limits with ulimit -a. To change the limits edit the files /etc/security/limits.conf and ** and add (remove repeated lines):

    1*                soft    sigpending      127862
    2*                hard    sigpending      127862
    3*                soft    nofile          65536
    4*                hard    nofile          65536
    5*                soft    nproc           127862
    6*                hard    nproc           127862

TDAQ Release

mkdir /home/TDAQ
ln -s /home/TDAQ /TDAQ
cd /TDAQ/
rpm -i --dbpath `pwd`/.rpmdb --prefix=`pwd` http://atlas-tdaq-sw.web.cern.ch/atlas-tdaq-sw/yum/tdaq/slc6/noarch/ayum_slc6-4-0.noarch.rpm
cd ayum/src/rpmext/
make clean
make all
cd /TDAQ/
alias ayum=/TDAQ/ayum/ayum
nano ayum/etc/yum.repos.d/lcg-sft.repo

Add to the file:

    1[lcg-sft-new-contrib]
    2name="Official LCG software repo v2 (contrib)"
    3baseurl=http://lcgpackages.web.cern.ch/lcgpackages/rpms_contrib
    4
    5[lcg-sft-new-post95]
    6name="Official LCG software repo v3"
    7baseurl=http://lcgpackages.web.cern.ch/lcgpackages/rpms_updates
    8
    9[lcgrepo97]
   10name=LCG Releases
   11baseurl=http://lcgpackages.web.cern.ch/lcgpackages/lcg/repo/7/LCG_97/

ayum update
ayum search DAQRelease
ayum search lcgenv
ayum install tdaq-07-01-00_x86_64-slc6-gcc62-opt tdaq-07-01-00_src
ayum install LCG_87_lcgenv_1.3.3_x86_64_slc6_gcc62_opt
ayum install tdaq-08-03-01_x86_64-centos7-gcc8-opt tdaq-08-03-01_src
ayum install LCG_96_lcgenv_1.3.8_x86_64_centos7_gcc8_opt
ayum install tdaq-09-00-00_x86_64-centos7-gcc9-opt tdaq-09-00-00_src
ayum install LCG_97_lcgenv_1.3.8_x86_64_centos7_gcc9_opt
ayum install tdaq-09-00-01_x86_64-centos7-gcc9-opt tdaq-09-00-01_src
ayum install LCG_97python3_lcgenv_1.3.8_x86_64_centos7_gcc9_opt
ayum install CMake_3.14.3_Linux-x86_64
ayum clean all
ayum update

BL4S TDAQ

git config --global http.emptyAuth true
mkdir TDAQ_local
cd TDAQ_local
git clone https://:@gitlab.cern.ch:8443/BL4S/TDAQ.git ./
git remote add cristovao https://:@gitlab.cern.ch:8443/cbeiraod/TDAQ.git
git fetch cristovao
git checkout --track cristovao/cbeiraod-GreatCodeRestructure
git config --local user.name "[name]"
git config --local user.email "[email]"
git submodule init
git submodule update
cp RCDTDAQ_settings.sh.example-tdaq831Local RCDTDAQ_settings.sh
cd OKS/
ln -s DESY/ current
cd ..
mkdir /home/bl4sdaq/TDAQ_LOGs/
mkdir /home/bl4sdaq/TDAQ_Data

Split TDAQ

Server

yum install nfs-utils
systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap
systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap
nano /etc/exports

    1/home/bl4sdaq/TDAQ *.bl4s.lan(rw,sync,no_root_squash ,no_all_squash)

systemctl restart nfs-server

Client

yum install nfs-utils
nano /ets/fstab

Add:

   13192.168.21.242:/home/bl4sdaq/TDAQ /home/bl4sdaq/TDAQ        nfs     defaults        0 0

mount -a

DNS+DHCP Server

This part is split into several logical sections.


The internal testbeam network at DESY is on the 192.168.21.* address space and some of the IPs are already taken by some of the existing computers. Used IPs:

  • 1: RC PC
  • 2: windows PC
  • 3: Control terminal in hut
  • 124: stage controller.

Consequently, the server will be given the IP 192.168.21.242 and will hand out over DHCP adresses in the range 125-225

In a first step, the computer must be equipped with a second ethernet port. The computer currently being configured is pc-bl4s-05.cern.ch. This computer has an onboard network card with a single ethernet interface, so an expansion network card was added. This network card brought an additional 2 ethernet interfaces.

The onboard ethernet interface is connected to the CERN/DESY network and obtains its IP through DHCP, so not configuration is necessary. This interface has the name em1. The interfaces of the expansion card will be given a fixed IP, through a static configuration, and have the names p1p1 and p1p2.

The boot configuration options should be found under the /etc/sysconfig/network-scripts/ directory, there should be a file for each interface. The file should be named ifcfg-INTERFACENAME, where INTERFACENAME is the name of the interface in question. If the file does not exist, probably because the network card was added after the OS was installed, create it.

The /etc/sysconfig/network-scripts/ifcfg-p1p1 was edited to the following:

    1NAME=p1p1
    2DEVICE=p1p1
    3ONBOOT=yes
    4UUID=9777f16a-cfb3-4048-b90a-d2735cb09a30
    5IPV6INIT=yes
    6TYPE=Ethernet
    7DEFROUTE=no
    8
    9# Static IP
   10BOOTPROTO=static
   11IPADDR=192.168.21.242
   12NETMASK=255.255.255.0
   13GATEWAY=192.168.21.242
   14DNS1=192.168.21.242
   15IPV4_FAILURE_FATAL=yes
   16
   17#NETBOOT=yes
   18

nb. the UUID line is specific to the network interface being used and can likely be omitted

And the /etc/sysconfig/network-scripts/ifcfg-p1p2 was edited to the following:

    1NAME=p1p2
    2DEVICE=p1p2
    3ONBOOT=yes
    4UUID=99943349-051f-4609-8e61-f9ffcd909be2
    5IPV6INIT=yes
    6TYPE=Ethernet
    7DEFROUTE=no
    8
    9# Static IP
   10BOOTPROTO=static
   11IPADDR=192.168.21.242
   12NETMASK=255.255.255.0
   13GATEWAY=192.168.21.242
   14DNS1=192.168.21.242
   15IPV4_FAILURE_FATAL=yes
   16
   17#NETBOOT=yes
   18

Finally, in order to load and test the configuration without having to reboot the computer, run the command: systemctl restart network


For DNS and DHCP server, we will be using dnsmasq.

To install it, as the root user, run the command: yum install dnsmasq

Then start the dnsmasq daemon, enable it so that it starts automatically on boot and check the status:

systemctl start dnsmasq
systemctl enable dnsmasq
systemctl status dnsmasq

In my case, the above status reported failed, the reason seems to be because port 53 was already being used by some other process. We have not configured the daemon yet, so this is of no concern at this point. Most guides recommend starting the services before configuring them, I am not sure of the reason, but we are following that process here.

If there are any problems with ports and attached processes, the list of processes attached to any network resource can be obtained with the command: netstat -ltnp

nb. You may need to install the program above for it to work, use the command yum install net-tools as root.

Look in the list for the process connected to port 53 and kill the offending program if needed. However, the process may still start automatically on boot, so make sure to disable it if that is the case.

Please note that below we will be showing configuration inside configuration files. The lines that were edited will be presented, but with different versions the exact lines may not match. Please search for the matching line with the default value and uncomment it if necessary and change the value to the one shown here.

The dnsmasq configuration can be found in the /etc/dnsmasq.conf file. We start by making a backup of the default configuration with the command cp /etc/dnsmasq.conf /etc/dnsmasq.conf.orig

Next, we open the configuration file, with our favorite editor, and edit the options: nano /etc/dnsmasq.conf

We start by setting dnsmasq to only listen on the ethernet ports connected to the internal network. In our case, the internal network has 2 ports, so the relevant line was uncommented and duplicated, once for each line.

  106interface=p1p1
  107interface=p1p2

We also set the address on which dnsmasq is listening (we also keep the loopback address because we want the machine to be able to find the DNS entries it is serving):

  112listen-address=::1,127.0.0.1,192.168.21.242

We want to have our own local domain for the DNS entries, we will use bl4s.lan as the domain. As a result, we must set the following options:

  136expand-hosts
  145domain=bl4s.lan

Finally we force the domain to use the local address:

   79address=/bl4s.lan/127.0.0.1
   80address=/bl4s.lan/192.168.21.242

    1...code...

Once done customizing the configuration file, the configuration can be tested with the command: dnsmasq --test

Then the DNS resolution mechanism of the local computer has to be made to lookup on the local machine first. This is done through the file /etc/dhcp/dhclient.conf, which must be created if it does not exist. The contents of the file must be set to:

    1prepend domain-name-servers 127.0.0.1;
In this way, when the /etc/resolv.conf is automatically generated at boot, the loopback address is automatically added as the first entry. To load these changes, run the command: systemctl restart network

The DNS entries are read from the /etc/hosts file, so any hardcoded DNS entries should be added here. We add the entries for the local computer, as well as a couple of needed aliases. The file becomes:

    1127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    2::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    3
    4127.0.0.1       dnsmasq
    5192.168.21.242  dnsmasq 
    6192.168.21.242  gateway
    7192.168.21.242  pc-bl4s-05

To load these changes, run the command: systemctl restart dnsmasq

To test the local DNS, try the following commands (you may need to install the bind-utils package yum install bind-utils):

nslookup bl4s.lan
dig bl4s.lan
nslookup pc-bl4s-05.bl4s.lan
dig pc-bl4s-05.bl4s.lan
dig pc-bl4s-05

DNS is now configured.


To enable DHCP, continue editing the same dnsmasq configuration file as above (/etc/dnsmasq.conf). Find the section with the dhcp-range parameters and add the following line:

  166dhcp-range=192.168.21.125,192.168.21.225,255.255.255.0,infinite

Note that the DHCP lease is infinite, but we are using an address range big enough for 100 devices. If it is ever necessary to connect more devices and the address range i exhausted, it may be necessary to reset the leases or increase the range.

The dnsmasq service stores its leases in /var/lib/misc/dnsmasq.leases.

The file can only be modified when the service is not running. As a result, first stop the dnsmasq service, then delete or modify the file and finally restart the dnsmasq service.

systemctl stop dnsmasq
nano /var/lib/misc/dnsmasq.leases
rm /var/lib/misc/dnsmasq.leases
systemctl start dnsmasq

Then, find the line with the dhcp-leasefile parameter and uncomment it to make sure the leasefile is well defined.

  541dhcp-leasefile=l

The make the DHCP server authoritative by finding and uncommenting the following line:

  551dhcp-authoritative

dnsmasq can now be restarted: systemctl restart dnsmasq.


As a final step, NAT must be enable such that the computers on the internal network can access the external network. For this, the firewall must be enabled. As a result, the firewall rules must be changed since the TDAQ software does not play well with firewalls.

First, the kernel must be configured to forward packets. Edit the file * /etc/sysctl.conf* and add the following configuration line:

   12# Controls IP packet forwarding
   13net.ipv4.ip_forward = 1

The computer would have to be rebooted in order to pick up this change, but it can also be done on a live session with sysctl -w net.ipv4.ip_forward=1. Then enable the firewall with systemctl enable firewalld and start the firewall systemctl start firewalld. Then add the network interfaces into internal and external zones:

firewall-cmd --zone=external --add-interface=em1 --permanent
firewall-cmd --zone=internal --add-interface=p1p1 --permanent
firewall-cmd --zone=internal --add-interface=p1p2 --permanent

Enable masquerading on the external interface firewall-cmd --zone=external --add-masquerade --permanent. Finally, add the necessary open services oon the internal network:

firewall-cmd --permanent --zone=internal --add-service=dhcp
firewall-cmd --permanent --zone=internal --add-service=dns
firewall-cmd --permanent --zone=internal --add-service=http
firewall-cmd --permanent --zone=internal --add-service=nfs
firewall-cmd --permanent --zone=internal --add-service=ssh
firewall-cmd --permanent --zone=internal --add-port=4241/tcp
firewall-cmd --permanent --zone=internal --add-port=7001/udp
firewall-cmd --permanent --zone=external --add-service=dhcp
firewall-cmd --permanent --zone=external --add-service=dns
firewall-cmd --permanent --zone=external --add-port=4241/tcp
firewall-cmd --permanent --zone=external --add-port=7001/udp

It is also necessary to open the "ephemeral" ports for TDAQ. Start by getting the range of ports with the command sysctl -A | grep ip_local_port_range. Then add those ports as exceptions to the firewall for the internal zone:

firewall-cmd --permanent --zone=internal --add-port=32768-60999/udp
firewall-cmd --permanent --zone=internal --add-port=32768-60999/tcp

Finally, enable NAT firewall-cmd --permanent --direct --passthrough ipv4 -t nat -I POSTROUTING -o em1 -j MASQUERADE -s 192.168.21.0/24. Reload the firewall rules firewall-cmd --complete-reload. Verify things were configured correctly: firewall-cmd --list-all-zones.

-- CristovaoDaCruzESilva - 2020-06-23

Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r12 - 2020-07-16 - CristovaoDaCruzESilva
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    BL4S All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback