1 Build and install dependencies

1.1 Install dependencies from CentOS repositories

Install boost-devel libuuid libuuid-devel motif-devel motif-static pam-devel bzip2-devel openldap-devel java java-devel with dnf on the host PC.

1.2 Install a newer gcc version.

In our case we are building and installing gcc 8.2.

1.3 Build and install Boost

We are using Boost version 1.62
On the ZynqMP:

cd {build_dir}
./bootstrap.sh --prefix=/usr/local
./b2 install --with=all

1.4 Build and install Xerces

We are using Xerces version 3.1.4-4
On the ZynqMP:

cd {build_dir}
./configure --prefix=/usr/local
make
make install

1.5 Build and install CppUnit

We are using CppUnit version 1.14
On the ZynqMP:

cd {build_dir}
./configure --build arm
make
make check
make install

1.6 Build and install i2c-tools

cd i2c-tools-4.1/
make
make install

1.7 Build and install ROOT

Instructions here.
Note: If you are compiling with gcc version> 7.1 TDAQ uses c++17 standard by default, but in ROOT you have to enable compiling with c++17 standard using the cxx17 cmake flag

1.8 Build and install I2C

We are building I2C and its dependencies using yocto.

cd <your yocto rpm dir>
cp libzynq-log-dev-1.0-r0.aarch64.rpm libzynq-menu-dev-1.0-r0.aarch64.rpm libzynq-util-dev-1.0-r0.aarch64.rpm I2C-1.0-r0.aarch64.rpm I2C-dev-1.0-r0.aarch64.rpm ZynqAxi-1.0-r0.aarch64.rpm ZynqMenu-1.0-r0.aarch64.rpm ZynqLog-1.0-r0.aarch64.rpm ZynqUtil-1.0-r0.aarch64.rpm cd ${dest_dir}

From the ZynqMP side:

chmod +666 /dev/i2c-*
cd ${dest_dir}
rpm -U --nodeps *.rpm

2 Build and install TDAQ

2.1 Checkout tdaq and tdaq common versions

On the host side

export PREFIX=<your_TDAQ_top_dir>
cd $PREFIX
git clone https://:@gitlab.cern.ch:8443/atlas-l1ct/zynq-tdaq-cmake.git -b zcu102
git clone https://:@gitlab.cern.ch:8443/atlas-tdaq-software/cmake_tdaq.git
export PATH=${PREFIX}/zynq-tdaq-cmake/checkout_release:${PATH}
checkout_release tdaq-common tdaq-common-03-03-00 ${PREFIX}/zynq-tdaq-cmake/checkout_release/tdaq-common-packages.txt
checkout_release tdaq tdaq-08-02-01 ${PREFIX}/zynq-tdaq-cmake/checkout_release/tdaq-packages.txt

cd ${PREFIX}/tdaq/tdaq-08-02-01
git clone ssh://git@gitlab.cern.ch:7999/atlas-l1ct/TestRCApp.git -b zcu102
cd -

We are using tdaq and tdaq-common version 03-03-00, adjust your paths accordingly.

2.2 Copy CMakeLists files and apply patch

cp ${PREFIX}/zynq-tdaq-cmake/tdaq-common/CMakeLists.txt ${PREFIX}/tdaq-common/tdaq-common-03-03-00/
cp ${PREFIX}/zynq-tdaq-cmake/tdaq/CMakeLists.txt ${PREFIX}/tdaq/tdaq-08-02-01/
cd ${PREFIX}/tdaq/tdaq-08-02-01/rcc_time_stamp
patch -b -p1 <${PREFIX}/zynq-tdaq-cmake/tdaq/patches/rcc_time_stamp.patch

ZynqMP runs out of RAM while compiling RunControl targets. To compile them successfully add set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} --param ggc-min-expand=1 --param ggc-min-heapsize=262144") to the file ${PREFIX}/tdaq/tdaq-08-02-01/RunControl/CMakeLists.txt

2.3 Set environment

From the zynq side:

export PREFIX=/root/TDAQ
export CC=gcc
export CXX=g++
export CMAKE_PREFIX_PATH=/root/TDAQ/cmake_tdaq:/root/TDAQ:${CMAKE_PREFIX_PATH}
export PATH=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.aarch64/bin:${PATH}
export JAVA_ROOT=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.aarch64
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.aarch64
export TDAQ_JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.aarch64

Adjust your java paths to point to your java version.

2.4 Build tdaq-common

cd ${PREFIX}/tdaq-common/tdaq-common-03-03-00
mkdir build&&cd build
cmake3 -DBINARY_TAG=aarch64-xlnx-gcc8-opt ../
make
make install

From the host side

sudo cp -r /cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed/share/lib/ ${PREFIX}/tdaq-common/tdaq-common-03-03-00/installed/share

2.5 Build tdaq

cd ${PREFIX}/tdaq/tdaq-08-02-01/
source ${PREFIX}/ROOT/build/bin/thisroot.sh
cmake3  -DTBB_ROOT_DIR=${PREFIX}/ROOT/build \
	-DBINARY_TAG=aarch64-xlnx-gcc8-opt \
	-DCPPUNIT_LIBRARY=/usr/local/lib/libcppunit.a \
	-DROOT_INCLUDE_DIR=${PREFIX}/ROOT/build/include \
	-DROOT_LIBRARY_DIRS=${PREFIX}/ROOT/build/lib \
	-DROOT_BINARY_PATH=${PREFIX}/ROOT/build/bin \
	-DCMAKE_INSTALL_PREFIX=${PREFIX}/tdaq/tdaq-08-02-01/installed \
	..
make
make install

For compiling you can choose the number of concurrent processes with the -j option. Some targets need a lot of RAM. So, the compilation will eventually crash. You can then restart it again with less processes.

3 Run the Run Control Application

3.1 Setup

From the ZynqMP side:
If you haven't yet, create a user and group same as the ones used by your host PC

useradd <your_host_user>
groupadd <your_host_group>

In /etc/hosts add the ip and names of host and client. In our case:

192.168.1.10 zcu102
192.168.1.1 pcphl1ct10.cern.ch

Then run:

sudo ip route add default via 192.168.1.1 dev eth0

Create a directory where the setup script is expected to be on the host PC and copy the setup script to it.

/cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed/

Create a symbolic link for the ipc_root.ref file pointing to the location of the ipc_root.ref in respect to the host pc:

mkdir -p ${NFS_PREFIX}/${PREFIX}
ln -s ${PREFIX}/ipc_root.ref ${NFS_PREFIX}/${PREFIX}/ipc_root.ref

To run the binaries from the zcu:

export CMTCONFIG=aarch64-xlnx-gcc8-opt
export TDAQ_INST_PATH=/root/TDAQ/tdaq/tdaq-08-02-01/installed/
chmod +x /root/TDAQ/tdaq/tdaq-08-02-01/installed/setup.sh
chmod +x /root/TDAQ/tdaq-common/tdaq-common-03-03-00/installed/setup.sh
source /root/TDAQ/tdaq/tdaq-08-02-01/installed/setup.sh
source /root/TDAQ/tdaq-common/tdaq-common-03-03-00/installed/setup.sh
export TDAQ_IPC_INIT_REF=file:/root/TDAQ/ipc_root.ref
chmod +777 /root/TDAQ/ipc_root.ref

From the Host side:
Disable firewalld and iptables

Add rsa authentication to the ZynqMP.

ssh-copy-id -i ~/.ssh/id_rsa.pub ppapageo@zcu102

Depending on the number of services that run in your PC (tdaq partitions, pmgserver, ipc etc) there is a chance that you need to increase the resource limits. We used the following values for the host PC on which both the initial and the training partitions run:

vim /etc/security/limits.conf

ppapageo   soft     nproc          32768
ppapageo   hard     nproc          32768
ppapageo   soft     nofile         65535
ppapageo   hard     nofile         65535

From the host side:

git clone ssh://git@gitlab.cern.ch:7999/atlas-l1ct/muctpi-test-partition.git /home/ppapageo/repos/muctpi-test-partition -b tdaq-8
export TDAQ_INST_PATH=/cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed/
export CMTCONFIG=x86_64-centos7-gcc8-opt
source ${TDAQ_INST_PATH}/setup.sh
cd muctpi-test-partition/installed
source ./setup.sh
export TDAQ_IPC_INIT_REF=file:${NFS_PREFIX}/root/TDAQ/ipc_root.ref
chmod +777 ${NFS_PREFIX}/root/TDAQ/ipc_root.ref

Create the directories where the setup script is to be expected on the host machine, and copy there the setup script.
From the ZynqMP side:

mkdir -p /cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed

From the host PC side:

cd muctpi-test-partition
cp zynq_setup/setup.sh ${NFS_PREFIX}/cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed

3.2 Edit the database

dbe -f Training/partitions/part_training.data.xml

• Add pcphl1ct10.dyndns.cern.ch and zcu102 on Computers
• Edit Partition initial and part_training "DefaultHost" to read pcphl1ct10.dyndns.cern.ch.
• Edit Segment MyDetector "Hosts" to pcphl1ct10.dyndns.cern.ch.
• Edit TrainingRCApplication rc_app_2 "RunsOn" to zcu102.
• Edit Partition part_training "LogRoot" to a path visible to both computers (e.g. nfs mounted)
• Create the following at SW_Repository and edit their installation paths:
  • BOOST_EXTERNAL_Zynq e.g. /usr/local
  • USR_EXTERNAL_Zynq e.g. /usr/local/lib64
  • CXX_EXTERNAL_Zynq e.g. /usr/local/lib64
  • I2C_EXTERNAL e.g. /usr//lib
  • TBB_EXTERNAL_Zynq e.g. ${PREFIX}/ROOT/build/lib
• Add the aarch64-xlnx-gcc8-opt to Tag if it does not exist. Then add the tag to the above Software Repositories, as well as to the ones below:
  • Training_SW_zynq
  • TDAQ Common Zynq
  • ROOTZynq

For the installation paths to work a directory named after the tag is expected. In our case we are either creating symbolic links which point to their parent directory or a direcory named after the tag where a lib symbolic link points to the orgininal direcory. For our installation paths this is achieved as follows:

ln -s /usr/local /usr/aarch64-xlnx-gcc8-opt

cd /usr/local
ln -s . aarch64-xlnx-gcc8-opt

cd /usr/local/lib64
mkdir aarch64-xlnx-gcc8-opt
cd aarch64-xlnx-gcc8-opt
ln -s /usr/local/lib64 lib

cd /usr/local/lib
mkdir aarch64-xlnx-gcc8-opt
cd aarch64-xlnx-gcc8-opt
ln -s /usr/local/lib lib

cd /usr/lib64
mkdir aarch64-xlnx-gcc8-opt
cd aarch64-xlnx-gcc8-opt
ln -s /usr/lib64 lib

cd ${PREFIX}/ROOT/build/lib
mkdir aarch64-xlnx-gcc8-opt
cd aarch64-xlnx-gcc8-opt
ln -s ${PREFIX}/ROOT/build/lib lib

3.3 Run the application

setup_daq daq/segments/setup-initial.data.xml initial -ng
setup_daq -p part_training -d Training/partitions/part_training.data.xml

at this point do an ipc_ls, and if there is no pmg server running on the ZynqMP, start it manually (you have to source the binaries).

3.4 Cleanup

yes|pmg_killall_everywhere ; pkill ipc_server ; pkill pmgserver; pkill pmglancher;ssh zcu102 "pkill pmgserver"

Then verify that everything is dead:

ps aux |grep pmg; ps aux |grep ipc; ps aux |grep -v root|grep afs; ssh zcu102 "ps aux |grep pmg";

3.5 Miscellaneous

If at any point you cannot access the state buttons on the GUI:

• Make sure Access Control is on Control and not on Display
• If you cannot switch to Control, make sure to delete the /tmp/tdaq-08-02-01 directory, where the lock files are stored

You have to run

chmod +666 /dev/i2c-*
ip route add default via 192.168.1.1 dev eth0

each time you reboot the ZynqMP

-- PanagiotisPapageorgiou - 2019-03-27