Integration of FPGA accelerators with the LHCb online system

Some documentation on the ongoing work to integrate the accelerator boards with the LHCb online system.

Access to servers

How to access lhcb-hot outside CERN

1) lxplus login

ssh <CERN USERNAME>@lxplus.cern.ch

2) lhcb-hot login

ssh <CERN USERNAME>@lhcb-hot

How to access lhcb-hot outside CERN with xpra

1) Open a terminal and create the following tunnel

ssh -L 2200:lhcb-hot:22 lxplus.cern.ch -l<CERN USERNAME>

2) Open a second terminal to start xpra session

xpra start --ssh="ssh -p2200" ssh:<CERN USERNAME>@localhost:<DISPLAY ID> --start-child=gnome-terminal

How to access acctb01 (GIGABYTE server) outside CERN

1) lxplus login

ssh <CERN USERNAME>@lxplus.cern.ch

2) lbgw login

ssh <ONLINE USERNAME>@lbgw

3) acctb01 login

ssh <ONLINE USERNAME>@acctb01

How to access daqv0207 (ASROCK server) outside CERN

1) lxplus login

ssh <CERN USERNAME>@lxplus.cern.ch

2) lbgw login

ssh <ONLINE USERNAME>@lbgw

3) daqv0207 login

ssh <ONLINE USERNAME>@daqv0207

How to access IPMI panel of lhcb-hot server

1) If outside CERN network, open a terminal and create the following tunnel

ssh -L 8000:lhcb-hot-ipmi.cern.ch:443 lxplus.cern.ch -l<CERN USERNAME>

2) Open the browser and go to

https://localhost:8000

3) Enter login credentials

user: oper
password: ASK OR OPEN A TICKET

1-CERN) If at CERN, where lhcb-hot is reachable, simply open a browser and go to lhcb-hot-ipmi.cern.ch

How to access IPMI panel of ASROCK server

1) If outside CERN network, open a terminal and create the following tunnel

ssh -L 2123:lbgw:22 lxplus.cern.ch -l<CERN USERNAME>

2) Open a second terminal to create another tunnel

ssh -L 8000:daqv0207-ipmi.lbdaq.cern.ch:80 localhost -p2123 -l<ONLINE USERNAME>

3) Open the browser and go to

http://localhost:8000

4) Enter login credentials

user: rtawp6
password: ASK OR OPEN A TICKET

1-CERN) If at CERN, where lbgw is reachable, simply start the tunnel at 2) and not 1) by doing

ssh -L 8000:daqv0207-ipmi.lbdaq.cern.ch:80 lbgw -l<ONLINE USERNAME>

Syntax:
ssh -L to create a ssh tunnel
xxxx:remotehost:yyyy traffic at xxxx port of localhost redirected to port yyyy of remotehost via the tunnel
tunnelhost pc mediating the tunnel
-pxxxx port used to connect to tunnelhost
-l USER username to connect to tunnelhost

Ports 22 80 CANNOT be changed

Ports 2123 e 8000 can be changed

How to use remote console of ASROCK server

To use the remote console:
1) Uninstall icedtea-web
2) Install Oracle JDK8
3) Open the tunnels ssh, the tunnel to the IPMI must use port 80, sudo permission can be required.

sudo ssh -L 80:daqv0207-ipmi.lbdaq.cern.ch:80 lbgw -l<ONLINE USERNAME>

How to access IPMI panel of GIGABYTE server

1) If outside CERN network, open a terminal and create the following tunnel

ssh -L 2123:lbgw:22 lxplus.cern.ch -l<CERN USERNAME>

2) Open a second terminal to create another tunnel

ssh -L 8000:acctb01-ipmi.lbdaq.cern.ch:443 localhost -p2123 -l<ONLINE USERNAME>

3) Open the browser and go to

https://localhost:8000

4) Enter login credentials

user: operator
password: ASK OR OPEN A TICKET

1-CERN) If at CERN, where lbgw is reachable, simply start the tunnel at 2) and not 1) by doing

ssh -L 8000:acctb01-ipmi.lbdaq.cern.ch:443 lbgw -l<ONLINE USERNAME>

How to access IPMI panel of GIGABYTE server (acctb02)

1) If outside CERN network, open a terminal and create the following tunnel

ssh -L 2123:lbgw:22 lxplus.cern.ch -l<CERN USERNAME>

2) Open a second terminal to create another tunnel

ssh -L 8000:acctb02-ipmi.lbdaq.cern.ch:443 localhost -p2123 -l<ONLINE USERNAME>

3) Open the browser and go to

https://localhost:8000

4) Enter login credentials

user: rtawp6
password: ASK

1-CERN) If at CERN, where lbgw is reachable, simply start the tunnel at 2) and not 1) by doing

ssh -L 8000:acctb02-ipmi.lbdaq.cern.ch:443 lbgw -l<ONLINE USERNAME>

acctb01 (GIGABYTE)

The following FPGA boards are installed in acctb01:

• A5PL, SN 201001
• H520N, SN 07110246
• H520N, SN 07110268
• H520N, SN 07110269

Server boot time: 4 min

PCIe slot mapping

Looking from the back side

Acctb01 (GIGABYTE)

Acctb02 (GIGABYTE) - owned by Pisa

daqv0207 (ASROCK)

The following FPGA boards are installed in daqv0207:

• A5PL, SN 824246
• A5PL, SN 830988
• A5PL, SN 830989
• A5PL, SN 830990
• A5PL, SN 830994
• A5PL, SN 830995
• H520N, SN 07110229

PCIe slot mapping

Looking from the back side

Driver

Useful links

User guide (how to run `pcie40_daq` tool): https://lhcb-online-soft.web.cern.ch/doc/daq/devel/ug.html#pcie40_daq

Developer guide: https://lhcb-online-soft.web.cern.ch/doc/daq/devel/dg.html

GitLab repositories

Main driver repository: https://gitlab.cern.ch/lhcb-retina/lhcb-daq40-software

Emulator driver (both PCIe40 and Accelerator are emulated): link to be added

Mixed driver (PCIe40 is a real FPGA card, while Accelerator is emulated): https://gitlab.cern.ch/acontu/lhcb-daq40-software/-/tree/acontu_mixed_driver/

Compile, install and load the driver

1) Clone the `lhcb-daq40-software` repository, then (within that directory) execute:

mkdir build
cd build
make -f ../Makefile
cd ../pcie40_daq
make
cd ../pcie40_driver
make
sudo make install
sudo modprobe lhcb_pcie40 mainmibs=32 metamibs=16 rtnamibs=32 nracc=2 acclist=0x10,0xa0 nrp40=2 p40list=0x20,0x32

Streams created by loading the driver can be seen with

ls /dev/

To unload the driver then type

sudo rmmod lhcb_pcie40

Run pcie40_daq program

Within the `lhcb-daq40-software` repository move to `pcie40_daq` folder.

To enable `main` and `rtna` streams run

./pcie40_daq -s main
./pcie40_daq -s rtna

To reset the DMA logic (-R) and to flush DMA buffers (-u) on a specific interface (-i) run

./pcie40_daq -i 0 -Ru
./pcie40_daq -i 2 -Ru

To start data taking, enabling (-e) and starting (-g) the generator on interface (-i) 0 (pcie40) run

./pcie40_daq -i 0 -rfeg -l 50 -o /tmp/file -Z 2

Output files can be inspected running

od -x /tmp/file | less

Consumer of monitoring data

Install, compile and run the code

Add in the ~/.bashrc file

. /cvmfs/lhcb.cern.ch/group_login.sh
. /group/online/dataflow/scripts/shell_macros.sh
. /group/online/dataflow/cmtuser/setup_comp.sh gcc11

Then clone the Online repository from git

git clone https://gitlab.cern.ch/lhcb/Online.git

The line containing `EventBuilder` in `Online/CMakeLists.txt` should be commented out.

Then, compile and install the software by doing

lb-project-init
do_make LCG_VERSION=101    (repeat the command several times if the process stops)
do_make LCG_VERSION=101 install
cmsetup

In `Online/testbeam/setup_comp.sh` the definition of `Realpath` should be changed to match the directory where the sofwtare is installed.

Finally, to start the gui, do:

. TestBeam/setup_comp.sh gcc11
start_gui_FPGA_Target

Run the code from an existing directory

From the `/Online` directory do:

. TestBeam/setup_comp.sh gcc11
start_gui_FPGA_Target

PCIe DMA IP for Stratix 10

Manuals

https://www.intel.com/content/www/us/en/products/details/fpga/intellectual-property/interface-protocols/multichannel-dma-mcdma.html

https://www.intel.com/content/www/us/en/docs/programmable/683821/22-3/before-you-begin.html

https://www.intel.com/content/www/us/en/docs/programmable/683517/22-3/introduction.html

Directories on daqv0207

Quartus path

Inventory of QSFP28 transceivers

2020 order 2021 order - first batch of 3 modules 2021 order - second batch of 9 modules 2022 order

Firmware

BittWare A5PL boards

Firmware repository (GitLab): https://gitlab.cern.ch/lhcb-retina/lhcb-daq40-firmware The rtna stream, reading data from the host, is added and its output is conncected with the main stream. Thanks to this, data are sent back to the host.

Installing BittWare Tools

If there is the need of loading different firmware on the boards, one has to install the BittWare tools. From previous experience, the steps should be the following:

cp -r  /home/bittware/riccardo  /home/someone/bittware       **(TO NOTE: /home/bittware/riccardo  is in lhcb-hot)**
cd /home/someone/bittware/bwtk
yum install php php-devel libusb
bin/setbwtk

You should then be able to run bwconfig-gui which give access to the boards The graphical interface should be run with sudo privileges and don’t forget to logon on the machine with the ssh option -X

sudo su
xauth add $(xauth -f ~username/.Xauthority list|tail -1)
bwconfig-gui &

How to create a Flash Programming File (.rpd)

1. Place the .sof file (generated by Quartus) in same directory as a5pl_256_rpd.cof (available from BittWare). In lhcb-hot, this directory is "/home/bittware/riccardo/a5pl_board_resources".

2. Rename the .sof file to top.sof.

3. Run the command: quartus_cpf -c a5pl_256_rpd.cof.

4. A file named a5pl_256_auto.rpd will be created.

Nallatech 520 boards

Board's resources are in /eos/lhcb/user/f/flazzari/FPGA_documentation/520N_resources

Identify devices

Executing

lsusb

Nallatech 520N boards are listed as

09fb:6010 Altera
0403:6015 Future Technology Devices International, Ltd Bridge(I2C/SPI/UART/FIFO)
05e3:0608 Genesys Logic, Inc. Hub

where the Altera device is the USB-blaster. The USB-blaster idProduct could be listed as 6810 and it is set to 6010 after running

/opt/quartus/20.2_pro/quartus/bin/quartus_pgm -l

With the command

lsusb -v -d 09fb:6010 |grep '09fb:6010\|iSerial'

is easy to link Bus and Device IDs to board serial number. Then executing

lsusb -t

is possible to reconstruct the port tree used in the Quartus programmer. Es.

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
    |__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/7p, 480M
        |__ Port 5: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 2: Dev 8, If 0, Class=Vendor Specific Class, Driver=, 480M
        |__ Port 7: Dev 7, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 2: Dev 9, If 0, Class=Hub, Driver=hub/4p, 480M
                |__ Port 2: Dev 14, If 0, Class=Vendor Specific Class, Driver=, 480M
            |__ Port 3: Dev 10, If 0, Class=Hub, Driver=hub/4p, 480M
                |__ Port 2: Dev 15, If 0, Class=Vendor Specific Class, Driver=, 480M

USB-blaster

To be sure that the usb-blaster is correctly seen by the system do:

/opt/quartus/20.2_pro/quartus/bin/quartus_pgm -l

If everything is working properly, the output should be:

Info: Command: quartus_pgm -l 
1) S10 520 card [1-2.5.2] 
2) S10 520 card [1-2.7.2.2]
3) S10 520 card [1-2.7.3.2]

Firmware upload

We are currently using the "S10 520 card [1-2.5.2]", therefore to upload the firmware the -c1 option should be used

/opt/quartus/20.2_pro/quartus/bin/quartus_pgm -c1 /localdisk/RTA_WP6_FPGA/nall_pci40.cdf

Then a warm reboot is needed:

sudo reboot

To be sure that the device PCIe is correctly seen, type

lspci -sa1:00.0

Compile BIST Firmware

Unzip bist_top_hp2_svn2667_cook_65pc_4m0.zip from /usr/share/nallatech/520n_htile/bist/firmware/
The file is available after nalla_520n_htile-1.0-5.x86_64.rpm installation or extracting the file directly from the package

Open ./bist_top_hp2_svn2667_cook_65pc_4m0/bist_top_hp2/build/bist_top_hp2.qpf

Start compilation
Execution time: 10h 41m 14s

HDLDesigner and QuestaSim setup

How to install HDLDesigner and QuestaSim on CentOS

https://pheloniusfriar.dreamwidth.org/62559.html

Install the following 32bit libraries

yum -y install glibc-devel.i686
yum -y install libXext-devel.i686
yum -y install libXrender-devel.i686
yum -y install libXtst-devel.i686
yum -y install libgcc.i686
yum -y install zlib-devel.i686

Copy HDLDesigner installer locally, move to the folder, give it the right permissions and start installation

cp -r /eos/project-e/engtools/public/installation_sources/mentor/HDLDesigner/2015_2 /tmp/
cd /tmp/2015_2/
chmod 555 HDS_2015.2_ixl.exe
./HDS_2015.2_ixl.exe -msiloc ./mgc

Follow graphic instructions for installation and remove the local copy

cd ~
rm -rf /tmp/2015_2/

Copy QuestaSim installer locally, move to the folder and start installation

cp -r /eos/project-e/engtools/public/installation_sources/mentor/modelsim/101d /tmp/
cd /tmp/101d
./install.linux -msiloc ./mgc

Follow graphic instructions for installation and remove the local copy

cd ~
rm -rf /tmp/101d

To run QuestaSim or HDLDesigner first export environment variables

export MTI_VCO_MODE=64
export LM_LICENSE_FILE=1800@lxlicen01:1800@lxlicen02:1800@lxlicen03:1717@lxlicen01:1717@lxlicen02:1717@lxlicen03

Run HDLDesigner

[HDLDesigner_install_path]/HDS_2015.2/bin/hds

Run QuestaSim

[questa_install_path]/questa_sim/bin/vsim

Throubleshooting

1) If you get an error like "Error in startup script: [...] ncFyP12 -+", the following steps need to be performed

Download, unpack and compile the freetype library

wget https://download.savannah.gnu.org/releases/freetype/freetype-2.4.7.tar.gz
tar -xvzf freetype-2.4.7.tar.gz
mv freetype-2.4.7.tar.gz freetype-2.4.7/
cd freetype-2.4.7/
./configure
make

Open vsim

nano [questa_install_path]/questa_sim/bin/vsim

Add the export line

#################################################################################
    /*) arg0=$x                 ;;
    *)  arg0=`dirname $arg0`/$x ;;
  esac
done

dir=`dirname $arg0`
export LD_LIBRARY_PATH=[path_to_freetype]/freetype-2.4.7/objs/.libs:$LD_LIBRARY_PATH

vco=${uname}${utype}
case $vco in
  SunOS4*)
    echo "Error: $cmd is not supported on ${uname} ${utype}"
#################################################################################

2) If X11 connection is rejected because of wrong authentication, do

sudo su
xauth add $(xauth -f ~[account]/.Xauthority list|tail -1)

How to run HDLDesigner on lhcb-hot

export MTI_VCO_MODE=64
export LM_LICENSE_FILE=1800@lxlicen01:1800@lxlicen02:1800@lxlicen03:1717@lxlicen01:1717@lxlicen02:1717@lxlicen03

Run HDLDesigner

/home/Mentor/HDS_2015.2/bin/hds

One can add the previous commands to the .bashrc by doing

nano ~/.bashrc

and adding the following lines to the .bashrc file

export MTI_VCO_MODE=64
export LM_LICENSE_FILE=1800@lxlicen01:1800@lxlicen02:1800@lxlicen03:1717@lxlicen01:1717@lxlicen02:1717@lxlicen03
export PATH=$PATH:/home/Mentor/HDS_2015.2/bin

at this point HDLDesigner can be run by typing

hdl_designer

How to run Questasim on lhcb-hot

export MTI_VCO_MODE=64
export LM_LICENSE_FILE=1800@lxlicen01:1800@lxlicen02:1800@lxlicen03:1717@lxlicen01:1717@lxlicen02:1717@lxlicen03

Several versions are available under /home/Mentor/

Run Questasim

[questa_install_path]/bin/vsim

One can add the previous commands to the .bashrc by doing

nano ~/.bashrc

and adding the following lines to the .bashrc file

export MTI_VCO_MODE=64
export LM_LICENSE_FILE=1800@lxlicen01:1800@lxlicen02:1800@lxlicen03:1717@lxlicen01:1717@lxlicen02:1717@lxlicen03
export PATH=$PATH:[questa_install_path]/bin/vsim

at this point QuestaSim can be run by typing

vsim

How to run firmware simulation

git clone [USER INFN]@gridui.pi.infn.it:/gpfs/ddn/lhcb/git_repo/hdl/bitt_brd_design.git

Compile the qsys file (dma-ecs and pll).

$QSYS_ROOTDIR/qsys-generate --synthesis=VERILOG --simulation=VERILOG bitt_brd_design/rtna_stream/work/hdl/modules/pcie/dma_ecs/arriav/qsys_pcie_dma_ecs_x8.qsys --output-directory=bitt_brd_design/rtna_stream/work/hdl/compiled_qsys/qsys_pcie_dma_ecs_x8/
$QSYS_ROOTDIR/qsys-generate --synthesis=VERILOG --simulation=VERILOG bitt_brd_design/rtna_stream/work/hdl/modules/pcie/dma_ecs/arriav/pcie_pll_av.qsys --output-directory=bitt_brd_design/rtna_stream/work/hdl/compiled_qsys/pcie_pll_av/

The following changes have been made

1) qsys_pcie_dma_ecs_x8.qsys BAR0 ADDRESS_WIDTH 24 -> 25
2) rtna_buf baseAddress 0x01ff0000 -> 0x02ff0000

General syntax:

$QSYS_ROOTDIR/qsys-generate --synthesis=VERILOG --simulation=VERILOG [QSYS FILE] --output-directory=[PATH TO OUTPUT DIRECTORY]

where QSYS_ROOTDIR points to the quartus/sopc_builder/bin folder inside the Quartus installation directory.

Launch HDL Designer

hdl_design

Open project rtna_stream.hdp in bitt_brd_design/rtna_stream/

File -> Open -> Project

Compile altera library (VHDL)

Task/Template (on right) -> FPGA Library Compile -> 
  FPGA Vendor: Altera
  Family: arria v gz
  Device: 5agzme7
  Package: h2f35c
  Speed: 3
-> Compile/Update ->
  Path to ModelSim compiler: %task_QuestaSimPath
  Language: VHDL-93
  Compiled VEndor Library Path: $HDS_PROJECT_DIR/work/vendor_lib
-> run

Wait for "Finished plug-in FPGA Vendor Library Compilation" in the log

Compile altera library (verilog)

Task/Template (on right) -> FPGA Library Compile -> 
  FPGA Vendor: Altera
  Family: arria v gz
  Device: 5agzme7
  Package: h2f35c
  Speed: 3
-> Compile/Update ->
  Path to ModelSim compiler: %task_QuestaSimPath
  Language: Verilog
  Compiled VEndor Library Path: $HDS_PROJECT_DIR/work/vendor_lib_ver
-> run

Wait for "Finished plug-in FPGA Vendor Library Compilation" in the log

Open work lib

double click on "work" under "Regular Libraries" in "Design Explorer"

Navigate design files and open alt_xcvr_resync.sv in simulation folder

double click on work -> compiled_qsys -> qsys_pcie_dma_ecs_x8 -> simulation -> submodules -> alt_xcvr_resync.sv

Comment line 42 "`timescale 1ps/1ps "

add // at the beginning of the line

Change language

Document -> Language -> SystemVerilog

Navigate design files and open alt_xcvr_resync.sv in synthesis folder

double click on work -> compiled_qsys -> qsys_pcie_dma_ecs_x8 -> synthesis -> submodules -> alt_xcvr_resync.sv

Comment line 42 "`timescale 1ps/1ps "

add // at the beginning of the line

Change language

Document -> Language -> SystemVerilog

Start simulation

click on QuestaSim Flow (the red Q with a yellow diamond on the corner)
  Resolution: ps
-> OK

Add some wave

drag DUT from sim-Default to Wave-Default

Run simulation

run 100us #(in Transcript)

If you don't have wave panel

View -> Wave

May be useful set compile always the code in HDL Designer

Tasks -> Set Generate Always
Tasks -> Set Compile Always

If you get a error like "non global static"

search the Design Unit that gives error
right click -> Change Language To -> VHDL 2008

If you get a error like "cannot find include file"

locate the path to file to include
right click on work library in design explorer -> Edit Library Configuration Settings
add include file path

-- GiuliaTuci - 2021-04-01