(r12) BuildTDAQForZynqMP < SystemOnChip

SystemOnChip Web>CentOSForZynqMP>BuildTDAQForZynqMP (revision 12) (raw view)~~EditAttachPDF~~
---#*Build and install dependencies*
---##*Install dependencies from CentOS repositories*
Install =boost-devel libuuid libuuid-devel motif-devel motif-static pam-devel bzip2-devel openldap-devel java java-devel i2c-tools= with =dnf= on the host PC. %BR%
---##*Install a newer gcc version.* 
In our case we are building and installing [[CentOSForZynqMP#BuildGCC][gcc 8.2]]. %BR%
---##*Build and install Boost*
We are using Boost version 1.62%BR%
On the ZynqMP:
%CODE{ lang="bash" }% 
cd {build_dir}
./bootstrap.sh --prefix=/usr/local
./b2 install --with=all
%ENDCODE%

---##*Build and install Xerces*
We are using Xerces version 3.1.4-4%BR%
On the ZynqMP:
%CODE{ lang="bash" }% 
cd {build_dir}
./configure --prefix=/usr/local
make
make install
%ENDCODE%

---##*Build and install CppUnit*
We are using CppUnit version 1.14%BR%
On the ZynqMP:
%CODE{ lang="bash" }% 
cd {build_dir}
./configure --build arm
make
make check
make install
%ENDCODE%

---##*Build and install ROOT*
Instructions [[CentOSForZynqMP#BuildRoot][here]]. %BR%
_Note: If you are compiling with gcc version> 7.1 TDAQ uses c++17 standard by default, but in ROOT you have to enable compiling with c++17 standard using the =cxx17= cmake flag_

---##*Build and install I2C*
We are building I2C and its dependencies using yocto.
%CODE{ lang="bash" }% 
cd <your yocto rpm dir>
cp libzynq-log-dev-1.0-r0.aarch64.rpm libzynq-menu-dev-1.0-r0.aarch64.rpm libzynq-util-dev-1.0-r0.aarch64.rpm I2C-1.0-r0.aarch64.rpm I2C-dev-1.0-r0.aarch64.rpm ZynqAxi-1.0-r0.aarch64.rpm ZynqMenu-1.0-r0.aarch64.rpm ZynqLog-1.0-r0.aarch64.rpm ZynqUtil-1.0-r0.aarch64.rpm cd ${dest_dir}
%ENDCODE%
From the ZynqMP side:
%CODE{ lang="bash" }% 
chmod +666 /dev/i2c-*
cd ${dest_dir}
rpm -U --nodeps *.rpm
%ENDCODE%

---#*Build and install TDAQ*
---##*Checkout tdaq and tdaq common versions*
On the host side
%CODE{ lang="bash" }% 
export PREFIX=<your_TDAQ_top_dir>
cd $PREFIX
git clone https://:@gitlab.cern.ch:8443/atlas-l1ct/zynq-tdaq-cmake.git -b zcu102
git clone https://:@gitlab.cern.ch:8443/atlas-tdaq-software/cmake_tdaq.git
export PATH=${PREFIX}/zynq-tdaq-cmake/checkout_release:${PATH}
checkout_release tdaq-common tdaq-common-03-03-00 ${PREFIX}/zynq-tdaq-cmake/checkout_release/tdaq-common-packages.txt
checkout_release tdaq tdaq-08-02-01 ${PREFIX}/zynq-tdaq-cmake/checkout_release/tdaq-packages.txt

cd ${PREFIX}/tdaq/tdaq-08-02-01
git clone ssh://git@gitlab.cern.ch:7999/atlas-l1ct/TestRCApp.git -b zcu102
cd -
%ENDCODE%
We are using tdaq and tdaq-common version 03-03-00, adjust your paths accordingly.%BR%
---##*Copy CMakeLists files and apply patch*
%CODE{ lang="bash" }% 
cp ${PREFIX}/zynq-tdaq-cmake/tdaq-common/CMakeLists.txt ${PREFIX}/tdaq-common/tdaq-common-03-03-00/
cp ${PREFIX}/zynq-tdaq-cmake/tdaq/CMakeLists.txt ${PREFIX}/tdaq/tdaq-08-02-01/
cd ${PREFIX}/tdaq/tdaq-08-02-01/rcc_time_stamp
patch -b -p1 <${PREFIX}/zynq-tdaq-cmake/tdaq/patches/rcc_time_stamp.patch
%ENDCODE%

ZynqMP runs out of RAM while compiling RunControl targets. To compile them successfully add ==set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} --param ggc-min-expand=1 --param ggc-min-heapsize=262144")== to the file ==${PREFIX}/tdaq/tdaq-08-02-01/RunControl/CMakeLists.txt== 
---##*Set environment*
From the zynq side:
%CODE{ lang="bash" }% 
export PREFIX=/root/TDAQ
export CC=gcc
export CXX=g++
export CMAKE_PREFIX_PATH=/root/TDAQ/cmake_tdaq:/root/TDAQ:${CMAKE_PREFIX_PATH}
export PATH=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.aarch64/bin:${PATH}
export JAVA_ROOT=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.aarch64
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.aarch64
export TDAQ_JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.aarch64
%ENDCODE%
Adjust your java paths to point to your java version.%BR%

---##*Build tdaq-common*
%CODE{ lang="bash" }% 
cd ${PREFIX}/tdaq-common/tdaq-common-03-03-00
mkdir build&&cd build
cmake3 -DBINARY_TAG=aarch64-xlnx-gcc8-opt ../
make
make install
%ENDCODE%
From the host side
%CODE{ lang="bash" }% 
sudo cp -r /cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed/share/lib/ ${PREFIX}/tdaq-common/tdaq-common-03-03-00/installed/share
%ENDCODE%

---##*Build tdaq*
%CODE{ lang="bash" }% 
cd ${PREFIX}/tdaq/tdaq-08-02-01/
source ${PREFIX}/ROOT/build/bin/thisroot.sh
cmake3  -DTBB_ROOT_DIR=${PREFIX}/ROOT/build \
	-DBINARY_TAG=aarch64-xlnx-gcc8-opt \
	-DCPPUNIT_LIBRARY=/usr/local/lib/libcppunit.a \
	-DROOT_INCLUDE_DIR=${PREFIX}/ROOT/build/include \
	-DROOT_LIBRARY_DIRS=${PREFIX}/ROOT/build/lib \
	-DROOT_BINARY_PATH=${PREFIX}/ROOT/build/bin \
	-DCMAKE_INSTALL_PREFIX=${PREFIX}/tdaq/tdaq-08-02-01/installed \
	..
make
make install
%ENDCODE%
For compiling you can choose the number of concurrent processes with the =-j= option. Some targets need a lot of RAM. So, the compilation will eventually crash. You can then restart it again with less processes.

---#*Run the Run Control Application*
---##*Setup*
From the ZynqMP side:%BR%
If you haven't yet, create a user and group same as the ones used by your host PC
%CODE{ lang="bash" }% 
useradd <your_host_user>
groupadd <your_host_group>
%ENDCODE%
In =/etc/hosts= add the ip and names of host and client. In our case: 
%CODE{ lang="bash" }% 
192.168.1.10 zcu102
192.168.1.1 pcphl1ct10.cern.ch
%ENDCODE%
Then run:
%CODE{ lang="bash" }% 
sudo ip route add default via 192.168.1.1 dev eth0
%ENDCODE%
Create a directory where the setup script is expected to be on the host PC and copy the setup script to it.
%CODE{ lang="bash" }% 
/cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed/
%ENDCODE%

Create a symbolic link for the ipc_root.ref file pointing to the location of the ipc_root.ref in respect to the host pc:
%CODE{ lang="bash" }% 
mkdir -p ${NFS_PREFIX}/${PREFIX}
ln -s ${PREFIX}/ipc_root.ref ${NFS_PREFIX}/${PREFIX}/ipc_root.ref
%ENDCODE%
#ZynqBins
To run the binaries from the zcu:
%CODE{ lang="bash" }% 
export CMTCONFIG=aarch64-xlnx-gcc8-opt
export TDAQ_INST_PATH=/root/TDAQ/tdaq/tdaq-08-02-01/installed/
chmod +x /root/TDAQ/tdaq/tdaq-08-02-01/installed/setup.sh
chmod +x /root/TDAQ/tdaq-common/tdaq-common-03-03-00/installed/setup.sh
source /root/TDAQ/tdaq/tdaq-08-02-01/installed/setup.sh
source /root/TDAQ/tdaq-common/tdaq-common-03-03-00/installed/setup.sh
export TDAQ_IPC_INIT_REF=file:/root/TDAQ/ipc_root.ref
chmod +777 /root/TDAQ/ipc_root.ref
%ENDCODE%

From the Host side:%BR%
Disable firewalld and iptables%BR%

Add rsa authentication to the ZynqMP.
%CODE{ lang="bash" }% 
ssh-copy-id -i ~/.ssh/id_rsa.pub ppapageo@zcu102
%ENDCODE%

Depending on the number of services that run in your PC (tdaq partitions,
pmgserver, ipc etc) there is a chance that you need to increase the 
resource limits. We used the following values for the host PC on which both 
the initial and the training partitions run:

%CODE{ lang="bash" }% 
vim /etc/security/limits.conf

ppapageo   soft     nproc          32768
ppapageo   hard     nproc          32768
ppapageo   soft     nofile         65535
ppapageo   hard     nofile         65535

%ENDCODE%

From the host side:
%CODE{ lang="bash" }% 
git clone ssh://git@gitlab.cern.ch:7999/atlas-l1ct/muctpi-test-partition.git /home/ppapageo/repos/muctpi-test-partition -b tdaq-8
export TDAQ_INST_PATH=/cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed/
export CMTCONFIG=x86_64-centos7-gcc8-opt
source ${TDAQ_INST_PATH}/setup.sh
cd muctpi-test-partition/installed
source ./setup.sh
export TDAQ_IPC_INIT_REF=file:${NFS_PREFIX}/root/TDAQ/ipc_root.ref
chmod +777 ${NFS_PREFIX}/root/TDAQ/ipc_root.ref
%ENDCODE%

Create the directories where the setup script is to be expected on the host machine, and copy there the setup script.%BR%
From the ZynqMP side:
%CODE{ lang="bash" }% 
mkdir -p /cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed
%ENDCODE%

From the host PC side:
%CODE{ lang="bash" }% 
cd muctpi-test-partition
cp zynq_setup/setup.sh ${NFS_PREFIX}/cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-08-02-01/installed
%ENDCODE%

---##*Edit the database*

%CODE{ lang="bash" }% 
dbe -f Training/partitions/part_training.data.xml
%ENDCODE%
   * Add pcphl1ct10.dyndns.cern.ch and zcu102 on Computers
   * Edit Partition initial and part_training "DefaultHost" to read =pcphl1ct10.dyndns.cern.ch=.
   * Edit Segment MyDetector "Hosts" to =pcphl1ct10.dyndns.cern.ch=.
   * Edit TrainingRCApplication rc_app_2 "RunsOn" to =zcu102=.
   * Edit Partition part_training "LogRoot" to a path visible to both computers (e.g. nfs mounted)
   * Create the following at SW_Repository and edit their installation paths:
      * BOOST_EXTERNAL_Zynq e.g. =/usr/local=
      * USR_EXTERNAL_Zynq e.g. =/usr/local/lib64=
      * CXX_EXTERNAL_Zynq e.g. =/usr/local/lib64=
      * I2C_EXTERNAL e.g. =/usr//lib=
      * TBB_EXTERNAL_Zynq e.g. =${PREFIX}/ROOT/build/lib=
   * Add the aarch64-xlnx-gcc8-opt  to Tag if it does not exist. Then add the tag to the above Software Repositories, as well as to the ones below:
      * Training_SW_zynq 
      * TDAQ Common Zynq
      * ROOTZynq

For the installation paths to work a directory named after the tag is expected. In our case we are either creating symbolic links which point to their parent directory or a direcory named after the tag where a lib symbolic link points to the orgininal direcory. For our installation paths this is achieved as follows:

%CODE{ lang="bash" }% 
ln -s /usr/local /usr/aarch64-xlnx-gcc8-opt

cd /usr/local
ln -s . aarch64-xlnx-gcc8-opt

cd /usr/local/lib64
mkdir aarch64-xlnx-gcc8-opt
cd aarch64-xlnx-gcc8-opt
ln -s /usr/local/lib64 lib

cd /usr/local/lib
mkdir aarch64-xlnx-gcc8-opt
cd aarch64-xlnx-gcc8-opt
ln -s /usr/local/lib lib

cd /usr/lib64
mkdir aarch64-xlnx-gcc8-opt
cd aarch64-xlnx-gcc8-opt
ln -s /usr/lib64 lib

cd ${PREFIX}/ROOT/build/lib
mkdir aarch64-xlnx-gcc8-opt
cd aarch64-xlnx-gcc8-opt
ln -s ${PREFIX}/ROOT/build/lib lib
%ENDCODE%
---##*Run the application*
%CODE{ lang="bash" }% 
setup_daq daq/segments/setup-initial.data.xml initial -ng
setup_daq -p part_training -d Training/partitions/part_training.data.xml
%ENDCODE%
at this point do an =ipc_ls=, and if there is no pmg server running on the ZynqMP, start it manually ([[#ZynqBins][you have to source the binaries]]).

---##*Cleanup*
%CODE{ lang="bash" }% 
yes|pmg_killall_everywhere ; pkill ipc_server ; pkill pmgserver; pkill pmglancher;ssh zcu102 "pkill pmgserver"
%ENDCODE%
Then verify that everything is dead: 
%CODE{ lang="bash" }% 
ps aux |grep pmg; ps aux |grep ipc; ps aux |grep -v root|grep afs; ssh zcu102 "ps aux |grep pmg";
%ENDCODE%
---##*Miscellaneous*
If at any point you cannot access the state buttons on the GUI:
   * Make sure Access Control is on Control and not on Display
   * If you cannot switch to Control, make sure to delete the /tmp/tdaq-08-02-01 directory, where the lock files are stored


You have to run 
%CODE{ lang="bash" }% 
chmod +666 /dev/i2c-*
ip route add default via 192.168.1.1 dev eth0
%ENDCODE%
each time you reboot the ZynqMP

-- Main.PanagiotisPapageorgiou - 2019-03-27
Topic revision: r12 - 2019-09-19 - unknown
SystemOnChip
Public webs
- Cern Search
- TWiki Search
- Google Search
SystemOnChip All webs
Copyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback