Trigger and Clock Distribution System (TCDS)

Table of Contents

Proposal

https://theboard.web.cern.ch/sign-in

Description:

  • No TCDS system foreseen for prototyping and production stages
  • we need something for test beams probably and eventually for larger lab setups (trigger/clock in synch to multiple uDTCs)
  • it would be beneficial to record the timestamp and clock phase of all triggers for the analysis
  • (secondary) the availability/support of AMC13 cards might be limited
  • we are interested in all BGo commands (not only trigger)

Hence, the proposal to implement a TTC controller card on FC7, compatible that would distribute the trigger to one or more µTCA crates using the AMC13 (or another card) as a proxy.
The easiest way for the moments seems to be to send clock/trigger from an FC7 to the AMC13, so the plan is:

  • implement a dummy/clock trigger generator in an FC7, connected to the VHDL core that sends out TTC encoded on an optical fiber. Connect the FC7 output to the AC13 input and check that clock and trigger received correctly.
  • implement the access of clock and/or trigger from the front panel via a DIO5.

Then, if this works and not necessarily in this order:

  • implement an 'orbit' structure an allow for configurable BGo commands at arbitrary phase in the orbit and arbitrary pre-scaling.
  • implement trigger rules (a set of rules that go no like more than X triggers in Y consecutive BXs).
  • timestamp and measure the phase of each incoming trigger (w.r.t. the 40MHz clock). These info should be placed in a readout buffer and readout via IPbus like it was in uDTCFC7s.
  • investigate the option of using an alternative to the AMC13 to deliver clock/trigger/BGo to the backplane (maybe these?[1]).
  • design an analoge FMC card to directly receive signals from the photomultipliers, and have threshold programmable from the FPGA.

[1] Possible commercial alternative to AMC13:
https://www.nateurope.com/products/NAT-MCH%20-%20Mezzanine%20Modules.html

1. Documentation

1.1 DAQ development for IT and OT Meeting slides

  • 20180614_IT-uDTC_developers_meeting pdf
  • 20180619_IT-uDTC_developers_meeting pdf
  • 20180629_DAQ_development_for_IT_test_systems pdf
  • 20180713_DAQ_development_for_IT_test_systems pdf
  • 20180727_DAQ_development_for_IT_test_systems pdf
  • 20180824_DAQ_development_for_IT_test_systems pdf
  • 20180907_DAQ_development_for_IT_test_systems pdf
  • 20180914_DAQ_development_for_IT_test_systems pdf
  • 20180921_DAQ_development_for_IT_test_systems pdf
  • 20180928_DAQ_development_for_IT_test_systems pdf
  • 20181005_DAQ_development_for_IT_test_systems pdf
  • 20181012_DAQ_development_for_IT_test_systems pdf
  • 20181019_DAQ_development_for_IT_test_systems pdf
  • 20181102_DAQ_development_for_IT_test_systems pdf
  • 20181130_DAQ_development_for_IT_test_systems pdf
  • 20190128_DAQ_development_for_IT_test_systems pdf
  • 20190211_DAQ_development_for_IT_test_systems pdf
  • 20190215_DAQ_development_for_IT_test_systems pdf
  • 20190311_DAQ_development_for_IT_test_systems pdf
  • 20190325_DAQ_development_for_IT_test_systems pdf
  • 20190506_DAQ_development_for_IT_test_systems pdf
  • 20190527_Discussion_on_a_dedicated_FC7-based_TTC_board pdf
  • 20190603_Dedicated_FC7_based_TTC_card pdf
  • 20190617_Dedicated_FC7_based_TTC_card pdf
  • 20190701_Dedicated_FC7_based_TTC_card pdf
  • 20190826_DAQ_development_for_IT_and_OT pdf
  • 20191021_DAQ_development_for_IT_and_OT pdf
  • 20191118_DAQ_development_for_IT_and_OT pdf
  • 20191205_Development_of_a_trigger_card_on_FC7 pdf
  • 20200210_Development_of_a_trigger_card_on_FC7_VV pdf
  • 20200511_Development_of_a_trigger_card_on_FC7_VV pdf:
  • 20201214_Development_of_a_trigger_card_on_FC7_VV pdf:
  • 20211011_Development_of_a_trigger_card_on_FC7_2 pdf
  • 20220613_DAQ_for_IT_and_OT pdf

1.2 Manuals, repositories and links

2. uTCA Test Setup

20191016 192611.jpg
Figure 1. uTCA test setup

Firmware blocks and hardware routing that is particularly important to us right now are shown in Figure 2.

AMC13 uTCA FC7.png
Figure 2. AMC13 -uTCA - FC7 route diagram

Clocking diagram of FC7 is shown in Figure 3. The default clock source is driven by the 40.08 MHz local PLL and its route is marked with blue color.

CLOCK IT-firmware.png
Figure 3. FC7 clocking diagram

2.1 Installing software packages

2.1.1. CentOS 7.3

2.1.1.1 Install CentOS 7.3 on PC

CentOS vault→ http://vault.centos.org/
Download the Minimal install image → http://ftp.iij.ad.jp/pub/linux/centos-vault/7.3.1611/isos/x86_64/
Verify the iso image → https://wiki.centos.org/Download/Verify
Run dd command and write it to a USB stick → https://wiki.centos.org/HowTos/InstallFromUSBkey

repository: http://vault.centos.org/7.3.1611/os/x86_64
Minimal install without smart card support

→ First login

$ sudo yum install epel-release
$ sudo yum groupinstall "X Window System"
$ sudo yum groupinstall xfce
$ sudo yum install xfce4-* nano firefox wget git
$ sudo systemctl set-default graphical.target
$ sudo systemctl isolate graphical.target
$ reboot

Choose XFCE session on startup...

2.1.1.2. Install and start rarpd service

$ sudo yum install meson libcap libcap-devel openssl openssl-devel docbook*
$ git clone https://github.com/iputils/iputils.git
$ cd iputils
$ sed -e '/BUILD_RARPD/ s/value *: *false/value : true/g' -i meson_options.txt
$ meson builddir
$ cd builddir
$ ninja-build || ninja
$ sudo cp rarpd /sbin/rarpd
$ sudo nano /etc/systemd/system/rarpd.service

[Unit]
Description=Reverse Address Resolution Protocol Requests Server
Documentation=man:rarpd(8)
Requires=network.target
After=network.target

[Service]
Type=forking
User=root
#EnvironmentFile=-/etc/sysconfig/rarpd
ExecStart =/usr/sbin/rarpd -a -e -v

[Install]
WantedBy =multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl restart rarpd.service
$ sudo systemctl status rarpd.service

2.1.3 Configure network for uTCA crate

  • If there is only one ethernet connector, use a USB/Ethernet adapter for the internet.
  • The easiest way to configure the networking is using the GUI: Desktop → Applications → Settings → Network connections
  • Alternatively network-scripts can be used:

$ ip a → choose a network interface
$ cd /etc/sysconfig/network-scripts/ → your configuration should be similar to this one...

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp3s0f0
UUID=xyz
DEVICE=enp3s0f0
ONBOOT=yes → Do not forget to set this!
IPADDR=192.168.1.100
PREFIX=16 → to expand subnet range
PEERDNS=no
PEERROUTES=no

$ sudo systemctl restart network.service
$ sudo systemctl status network.service

2.1.1.4 Add FC7s to rarpd

  • MCH, AMC13 T1 and T2 do not need to be added to rarpd, they know their own IP addresses!

$ sudo yum install wireshark tshark telnet → For controlling the uTCA crate you should download NATview as well. → https://www.nateurope.com/products/NATview.html
$ sudo tshark -i <eth interface> → Put the first FC7 to one AMC slot in the uTCA crate, push on the Hot-plug switch and add it to rarpd. You can install multiple FC7s in the same way.
$ sudo nano /etc/ethers

08:00:30:00:23:59 FC7_1
08:00:30:00:29:01 FC7_2

$ sudo nano /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.1.70 FC7_1
192.168.1.80 FC7_2

$ sudo systemctl restart rarpd.service
$ ping FC7_1
$ ping FC7_2

2.1.1.5 Preparing microSD card for FC7

The easiest way to put the GoldenImage.bin to the microSD card is to use the dd command as Sarah recommended in the 3rd DAQ School.

You can find the GoldenImage.bin in the link below compressed in imgtool.

$ sudo fdisk -l → find SD card reader
$ sudo dd if=sdgoldenimage.img of=/dev/sd_card_name bs=512

→ Alternatively you can use imgtool!

Download and source avr32 toolchain for linux → https://www.microchip.com/mplab/avr-support/avr-and-sam-downloads-archive

$ wget http://ww1.microchip.com/downloads/archive/avr32-gnu-toolchain-3.4.3.820-linux.any.x86.tar.gz
$ wget http://ww1.microchip.com/downloads/archive/avr-headers.zip

$ tar -xvzf avr32-gnu-toolchain-3.4.3.820-linux.any.x86.tar.gz
$ unzip avr-headers.zip

$ mv avr32-gnu-toolchain-linux_x86/* ~/avr32-tools
$ mv avr-headers/avr32 ~/avr32-tools/avr32/include

$ export PATH=$PATH:$HOME/avr32-tools/bin
$ avr32-gcc --version
avr32-gcc (AVR_32_bit_GNU_Toolchain_3.4.3_820) 4.4.7
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

→ Download and compile imgtool → ic_mmc_v1_6_6.tar.gz

$ tar -xvzf ic_mmc_v1_6_6.tar.gz
$ cd ic_mmc_v1_6_6/imperial_mmc/
$ make Board=FC7_2
$ cd tools/imgtool/bin/
$ ./imgtool ?
Usage: imgtool image command [parameters]

Cmds: format <label> -- Formats an image
list -- List files in an image
add <name> <file> -- Adds a file to an image
get <name> <file> -- Gets a file from an image
del <name> -- Deletes a file from an image
check <name> -- Verifies the checksum of a file

→ Format and copy fallback image to microSD card.

$ sudo fdisk -l → find SD card reader
$ ./imgtool /dev/sdc list
$ ./imgtool /dev/sdc format fc7
$ ./imgtool /dev/sdc add GoldenImage.bin GoldenImage.bin
$ ./imgtool /dev/sdc check GoldenImage.bin

2.1.1.6 Install Ph2_ACF and upload bitstream to FC7

→ Install ipbus-software v2.5 from rpm repository or compile from source.

$ sudo nano /etc/yum.repos.d/ipbus-sw.repo

[ipbus-sw-base]
name=IPbus software repository
baseurl=http://www.cern.ch/ipbus/sw/release/2.5/centos7_x86_64/base/RPMS
enabled=1
gpgcheck=0

[ipbus-sw-updates]
name=IPbus software repository updates
baseurl=http://www.cern.ch/ipbus/sw/release/2.5/centos7_x86_64/updates/RPMS
enabled=1
gpgcheck=0

$ sudo yum clean all
$ sudo yum groupinstall uhal

→ Install ipbus-software v2.5.0 from source:

$ sudo yum -y install make rpm-build git-core erlang gcc-c++ boost-devel pugixml-devel python-devel
$ git clone --depth=1 -b v2.5.0 https://github.com/ipbus/ipbus-software.git
$ cd ipbus-software
$ make Set=uhal
$ make Set=uhal rpm
$ sudo yum localinstall `find . -iname "*.rpm"`

→ Install root:

$ sudo yum install root
$ sudo yum install root-net-http root-graf3d-gl root-physics root-montecarlo-eg root-graf3d-eve root-geom libusb-devel xorg-x11-xauth.x86_64

→ Now you can compile and install Ph2_ACF from source.

$ sudo yum install cmake boost-devel
$ git clone -b master https://gitlab.cern.ch/cms_tk_ph2/Ph2_ACF.git
$ cd Ph2_ACF
$ source setup.sh
$ cd build
$ cmake ..
$ make -j4
$ cd ..
$ nano settings/D19CDescription.xml → set IP address of FC7
$ fpgaconfig -l -c settings/D19CDescription.xml
$ fpgaconfig -f ../bitstream/IT-firmware_dev.bit -i IT-firmware_dev.bit -c settings/D19CDescription.xml
$ fpgaconfig -i IT-firmware_dev.bit -c settings/D19CDescription.xml

2.1.1.7 Install Inner Tracker Ph2_ACF and upload bitstream to FC7

  • Prerequisites for Inner Tracker Ph2_ACF are the same as in 2.1.8.

→ Compile and install Inner Tracker Ph2_ACF from source.

$ sudo yum install cmake boost-devel
$ git clone -b chipPolymorphism https://gitlab.cern.ch/cmsinnertracker/Ph2_ACF.git
$ cd Ph2_ACF
$ source setup.sh
$ mkdir myBuild
$ cd myBuild
$ cmake ..
$ make -j4
$ cd ..
$ mkdir test/
$ cp settings/RD53Files/CMSIT_RD53.txt test/
$ cp settings/CMSIT.xml test/
$ cd test/
$ nano CMSIT.xml → set IP address of FC7
$ fpgaconfig -l -c CMSIT.xml
$ fpgaconfig -f ../bitstream/IT-firmware_dev.bit -i IT-firmware_dev.bit -c CMSIT.xml
$ fpgaconfig -i IT-firmware_dev.bit -c CMSIT.xml

2.1.1.8 Install AMC13 tools and run test scripts

$ wget https://gitlab.cern.ch/cms-cactus/svn2git/software/boards/amc13/-/archive/1.2.9/amc13-1.2.9.tar.gz → for T1 firmware version 0x2264 and T2 firmware version 0x32
$ wget http://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/6.3/x86_64/os/Packages/ipmitool-1.8.11-13.el6.1.x86_64.rpm
$ sudo yum localinstall ipmitool-1.8.11-13.el6.1.x86_64.rpm
$ service ipmi start
$ sudo yum install boost-devel readline readline-devel
$ tar -xvzf amc13-1.2.9.tar.gz
$ cd amc13-1.2.9/
$ source env.sh
$ make → You have to install cactus before you make this! Be aware, that you have to install the packeges listed before source compilation as well.

→ Looking for IP addresses of AMC13

$ cd dev_tools/amc13Config/
$ nano systemVars.py

#File to specify what the default varaibles addresses are used in your system

#Default IP address for commercial MCH module
# our NAT MCH address
DEFAULT_HOST_IP="192.168.1.41" <---- set to uTCA crate IP address
# our Vadatech MCH address
# DEFAULT_HOST_IP="192.168.1.2"

#Default AMC13 slot number
DEFAULT_AMC13_SLOT=9

#Location of 'config_tools'. This should never need to be changed
DEFAULT_CONFIG_DIR="./config_tools"

#Network base for your uTCA crate's AMC modules
NETWORK_BASE="192.168.1" <---- set to uTCA crate subnet

$ ./scanCrate.pl OR
$ ./readIPs.py --host=192.168.1.41 --slot=13 → T1 192.168.4.199, T2 192.168.4.198

→ Read MMC version

$ ./mmcVersion.py --host=192.168.0.41 --slot=13 → 02 02

→ Create connections.xml file

$ cd ../../amc13/etc/amc13
$ cp connectionSN43.xml connections.xml
$ nano connections.xml

<?xml version="1.0" encoding="UTF-8"?>

<connections>
<connection id="T1" uri="ipbusudp-2.0://192.168.4.199:50001" address_table="file://AMC13XG_T1.xml" />
<connection id="T2" uri="ipbusudp-2.0://192.168.4.198:50001" address_table="file://AMC13XG_T2.xml" />
</connections>

→ Print AMC13 firmware versions

$ cd ../../../dev_tools/python/
$ ./printversion.py ../../amc13/etc/amc13/connections.xml
Using AMC13 software ver:0
Read firmware versions 0x2264 0x32
flavor = 2 features = 0x000000b2
T1 firmware version 0x2264
T2 firmware version 0x32

2.1.1.9 Install Vivado Lab Tools 2017.4

Register and download Vivado Lab Tools 2017.4 for linux → https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/vivado-design-tools/archive.html

$ tar -xvzf Xilinx_Vivado_Lab_Lin_2017.4_1216_1.tar.gz
$ cd Xilinx_Vivado_Lab_Lin_2017.4_1216_1/
$ sudo mkdir /opt/Xilinx
$ sudo chown -R <user-name>:<user-name> /opt/Xilinx
$ sudo chmod u+rwX /opt/Xilinx
$ ./xsetup

→ Install cable drivers

$ cd /opt/Xilinx/Vivado_Lab/2017.4/data/xicom/cable_drivers/lin64/install_script/install_drivers
$ sudo ./install_drivers

→ Run Vivado Lab Tools

$ source /opt/Xilinx/Vivado_Lab/2017.4/settings64.sh
$ vivado_lab &

→ Finally, you should pull out the JTAG from USB and plug it back.

2.1.2. CentOS 8.5

2.1.2.1 Install CentOS 8.5 on PC

CentOS vault→ http://vault.centos.org/
Download the Minimal install image → http://ftp.bme.hu/centos/8-stream/isos/x86_64/
Verify the iso image → https://wiki.centos.org/Download/Verify
Run dd command and write it to a USB stick → sudo dd bs=4M if=CentOS-8.5.2111-x86_64-boot.iso of=/dev/sdb status=progress oflag=sync

repository: https://vault.centos.org/8.5.2111/BaseOS/x86_64/os/
Minimal install without smart card support

→ First login

$ sudo yum install epel-release
$ sudo yum groupinstall xfce
$ sudo yum install xfce4-* nano firefox wget git --skip-broken
$ sudo systemctl set-default graphical.target
$ sudo systemctl isolate graphical.target

$ sudo yum groupinstall "Server with GUI"
$ sudo yum groupinstall "Workstation"

Choose XFCE session on startup...

2.1.2.2 Install and start rarpd service

$ sudo yum install cmake libcap libcap-devel openssl openssl-devel docbook*
$ sudo yum --enablerepo=powertools install ninja-build
$ sudo pip3 install meson
$ git clone https://github.com/iputils/iputils.git
$ wget https://github.com/iputils/iputils.git
$ cd iputils-20210722 --> this tag worked, rarpd is removed from the new version
$ sed -e '/BUILD_RARPD/ s/value *: *false/value : true/g' -i meson_options.txt
$ meson builddir
$ cd builddir
$ ninja-build || ninja
$ sudo cp rarpd /sbin/rarpd
$ sudo nano /etc/systemd/system/rarpd.service

[Unit]
Description=Reverse Address Resolution Protocol Requests Server
Documentation=man:rarpd(8)
Requires=network.target
After=network.target

[Service]
Type=forking
User=root
#EnvironmentFile=-/etc/sysconfig/rarpd
ExecStart =/usr/sbin/rarpd -a -e -v

[Install]
WantedBy =multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl restart rarpd.service
$ sudo systemctl status rarpd.service

2.1.2.3 Configure network for uTCA crate

  • If there is only one ethernet connector, use a USB/Ethernet adapter for the internet.
  • The easiest way to configure the networking is using the GUI: Desktop → Applications → Settings → Network connections
  • Alternatively network-scripts can be used:

$ ip a → choose a network interface
$ cd /etc/sysconfig/network-scripts/ → your configuration should be similar to this one...

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp3s0f0
UUID=xyz
DEVICE=enp3s0f0
ONBOOT=yes → Do not forget to set this!
IPADDR=192.168.1.100
PREFIX=16 → to expand subnet range
PEERDNS=no
PEERROUTES=no

$ sudo systemctl restart NetworkManager.service
$ sudo systemctl status NetworkManager.service

2.1.2.4 Add FC7s to rarpd

  • MCH, AMC13 T1 and T2 do not need to be added to rarpd, they know their own IP addresses!

$ sudo yum install wireshark wireshark-cli telnet → For controlling the uTCA crate you should download NATview as well. → https://www.nateurope.com/products/NATview.html
$ sudo tshark -i <eth interface> → Put the first FC7 to one AMC slot in the uTCA crate, push on the Hot-plug switch and add it to rarpd. You can install multiple FC7s in the same way.
$ sudo nano /etc/ethers

08:00:30:00:29:01 FC7_nanocrate

$ sudo nano /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.4.100 FC7_nanocrate

$ sudo systemctl restart rarpd.service
$ sudo sysctl -w net.ipv4.ping_group_range="0 1000"
$ ping 192.168.4.100

2.1.2.5 Preparing microSD card for FC7

The easiest way to put the GoldenImage.bin to the microSD card is to use the dd command as Sarah recommended in the 3rd DAQ School.

You can find the GoldenImage.bin in the link below compressed in imgtool.

$ sudo fdisk -l → find SD card reader
$ sudo dd if=sdgoldenimage.img of=/dev/sd_card_name bs=512

→ Alternatively you can use imgtool!

Download and source avr32 toolchain for linux → https://www.microchip.com/mplab/avr-support/avr-and-sam-downloads-archive

$ wget http://ww1.microchip.com/downloads/archive/avr32-gnu-toolchain-3.4.3.820-linux.any.x86.tar.gz
$ wget http://ww1.microchip.com/downloads/archive/avr-headers.zip

$ tar -xvzf avr32-gnu-toolchain-3.4.3.820-linux.any.x86.tar.gz
$ unzip avr-headers.zip

$ mv avr32-gnu-toolchain-linux_x86/* ~/avr32-tools
$ mv avr-headers/avr32 ~/avr32-tools/avr32/include

$ export PATH=$PATH:$HOME/avr32-tools/bin
$ avr32-gcc --version
avr32-gcc (AVR_32_bit_GNU_Toolchain_3.4.3_820) 4.4.7
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

→ Download and compile imgtool → ic_mmc_v1_6_6.tar.gz

$ tar -xvzf ic_mmc_v1_6_6.tar.gz
$ cd ic_mmc_v1_6_6/imperial_mmc/
$ make Board=FC7_2
$ cd tools/imgtool/bin/
$ ./imgtool ?
Usage: imgtool image command [parameters]

Cmds: format <label> -- Formats an image
list -- List files in an image
add <name> <file> -- Adds a file to an image
get <name> <file> -- Gets a file from an image
del <name> -- Deletes a file from an image
check <name> -- Verifies the checksum of a file

→ Format and copy fallback image to microSD card.

$ sudo fdisk -l → find SD card reader
$ ./imgtool /dev/sdc list
$ ./imgtool /dev/sdc format fc7
$ ./imgtool /dev/sdc add GoldenImage.bin GoldenImage.bin
$ ./imgtool /dev/sdc check GoldenImage.bin

2.1.2.6 Install Ph2_ACF with EUdaq and upload bitstream to FC7

→ Install ipbus-software v2.8 via YUM.

$ sudo curl https://ipbus.web.cern.ch/doc/user/html/_downloads/ipbus-sw.centos8.x86_64.repo -o /etc/yum.repos.d/ipbus-sw.repo
$ sudo yum-config-manager --enable powertools

$ sudo yum clean all
$ sudo yum groupinstall uhal

→ Install root:

$ sudo yum install root root-net-http root-net-httpsniff root-graf3d-gl root-physics \ root-montecarlo-eg root-graf3d-eve root-geom libusb-devel xorg-x11-xauth.x86_64

→ Now you can compile and install Ph2_ACF from source.

→https://gitlab.cern.ch/cms_tk_ph2/TTC_FC7_Control/-/blob/master/INSTRUCTIONS_TTC_AMC13_OT.md

$ sudo yum config-manager --set-enabled powertools
$ sudo yum install cmake pugixml-devel boost-devel cmake3 protobuf* protobuf-devel python3-pybind11 pybind11-devel
$ sudo yum makecache --refresh
$ sudo yum -y install gcc-toolset-10-gcc
$ sudo yum install clang-tools-extra
$ sudo yum install git-extras
$ git clone --recurse-submodules https://gitlab.cern.ch/acarvalh/Ph2_ACF.git Ph2_ACF_forkXanda
$ cd Ph2_ACF_forkXanda
$ git checkout ot_to_ttc_fc7
$ source setup.sh
$ mkdir build
$ cd build
$ mkdir ../MessageUtils/cpp
$ touch ../MessageUtils/cpp/CMakeLists.txt
$ touch ../MessageUtils/cpp/QueryMessage.pb.h
$ touch ../MessageUtils/cpp/ReplyMessage.pb.h
$ cmake ..
$ make -j4

$ cd ..
$ nano settings/D19CDescription.xml → set IP address of FC7
$ fpgaconfig -l -c settings/D19CDescription.xml
$ fpgaconfig -f ../bitstream/IT-firmware_dev.bit -i IT-firmware_dev.bit -c settings/D19CDescription.xml
$ fpgaconfig -i IT-firmware_dev.bit -c settings/D19CDescription.xml

2.1.2.7 Install AMC13 tools and run test scripts

$ wget https://gitlab.cern.ch/cms-cactus/svn2git/software/boards/amc13/-/archive/1.2.9/amc13-1.2.9.tar.gz → for T1 firmware version 0x2264 and T2 firmware version 0x32
$ wget http://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/6.3/x86_64/os/Packages/ipmitool-1.8.11-13.el6.1.x86_64.rpm
$ sudo yum localinstall ipmitool-1.8.11-13.el6.1.x86_64.rpm
$ service ipmi start
$ sudo yum install boost-devel readline readline-devel
$ tar -xvzf amc13-1.2.9.tar.gz
$ cd amc13-1.2.9/
$ source env.sh
$ make → You have to install cactus before you make this! Be aware, that you have to install the packeges listed before source compilation as well.

→ Looking for IP addresses of AMC13

$ cd dev_tools/amc13Config/
$ nano systemVars.py

#File to specify what the default varaibles addresses are used in your system

#Default IP address for commercial MCH module
# our NAT MCH address
DEFAULT_HOST_IP="192.168.1.41" <---- set to uTCA crate IP address
# our Vadatech MCH address
# DEFAULT_HOST_IP="192.168.1.2"

#Default AMC13 slot number
DEFAULT_AMC13_SLOT=9

#Location of 'config_tools'. This should never need to be changed
DEFAULT_CONFIG_DIR="./config_tools"

#Network base for your uTCA crate's AMC modules
NETWORK_BASE="192.168.1" <---- set to uTCA crate subnet

$ ./scanCrate.pl OR
$ ./readIPs.py --host=192.168.1.41 --slot=13 → T1 192.168.4.199, T2 192.168.4.198

→ Read MMC version

$ ./mmcVersion.py --host=192.168.0.41 --slot=13 → 02 02

→ Create connections.xml file

$ cd ../../amc13/etc/amc13
$ cp connectionSN43.xml connections.xml
$ nano connections.xml

<?xml version="1.0" encoding="UTF-8"?>

<connections>
<connection id="T1" uri="ipbusudp-2.0://192.168.4.199:50001" address_table="file://AMC13XG_T1.xml" />
<connection id="T2" uri="ipbusudp-2.0://192.168.4.198:50001" address_table="file://AMC13XG_T2.xml" />
</connections>

→ Print AMC13 firmware versions

$ cd ../../../dev_tools/python/
$ ./printversion.py ../../amc13/etc/amc13/connections.xml
Using AMC13 software ver:0
Read firmware versions 0x2264 0x32
flavor = 2 features = 0x000000b2
T1 firmware version 0x2264
T2 firmware version 0x32

2.1.2.8 Install Vivado Lab Tools 2017.4

Register and download Vivado Lab Tools 2017.4 for linux → https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/vivado-design-tools/archive.html

$ tar -xvzf Xilinx_Vivado_Lab_Lin_2017.4_1216_1.tar.gz
$ cd Xilinx_Vivado_Lab_Lin_2017.4_1216_1/
$ sudo mkdir /opt/Xilinx
$ sudo chown -R <user-name>:<user-name> /opt/Xilinx
$ sudo chmod u+rwX /opt/Xilinx
$ ./xsetup

→ Install cable drivers

$ cd /opt/Xilinx/Vivado_Lab/2017.4/data/xicom/cable_drivers/lin64/install_script/install_drivers
$ sudo ./install_drivers

→ Run Vivado Lab Tools

$ source /opt/Xilinx/Vivado_Lab/2017.4/settings64.sh
$ vivado_lab &

→ Finally, you should pull out the JTAG from USB and plug it back.

2.2 Hardware debugging

Hardware debugging started with d19c-firmware from 5601fa13 commit → https://gitlab.cern.ch/myharank/d19c-firmware/commit/5601fa132ed8e89326b9c5ff13c8a3adc3cb73c2

$ git clone https://gitlab.cern.ch/myharank/d19c-firmware.git
$ cd d19c-firmware
$ git checkout 5601fa132ed8e89326b9c5ff13c8a3adc3cb73c2

2.2.1 First TTC input signal measurement with scope

  • To avoid multiple driver problem, the incoming TTC differential signal sent by AMC13 T2 to FC7 is buffered to single ended and routed to fmc_l12_spare pin.
  • Measurement of the single ended TTC signal is shown in Figure 4-5.

TTC input scope.jpg
Figure 4. TTC input signal in FC7 scope diagram
TTC input scope conf.jpg
Figure 5. TTC input signal in FC7 scope configuration

2.2.2 TTC input signal and fabric clock measurement with scope

  • Default clock source switched to AMC13 in FC7 firmware.
  • Fabric clock is the clock sent by AMC13 through the backplane.
  • There is 90 degrees phase shift between the TTC input signal and AMC13 clock source. TTC input signal is ahead of the AMC13 clock.

TTC input clock desktop.jpg
Figure 6. TTC input signal and fabric clock in FC7 desktop
TTC input clock scope.jpg
Figure 7. TTC input signal and fabric clock in FC7 scope diagram
TTC input clock probes.jpg
Figure 8. TTC input signal and fabric clock in FC7 probes

  • FC7 fmc l12 spare pinout

ttc_input 6 7 GND
general_reset 4 5 ttc_enable
ref_clk_200MHz 2 3 ipb_clk
fabric_clk 0 1 clk_40_0
+2.5V GND

2.2.3 Measurement of channel_a and channel_b in TTC decoder block

  • Channel a and b signals in TTC decoder routed to fmc_l12_spare pins.

TTC channel a b desktop.jpg
Figure 9. TTC channel a and b in TTC decoder desktop
TTC channel a b scope.jpg
Figure 11. TTC channel a and b in TTC decoder scope
TTC channel a b probes.jpg
Figure 12. TTC channel a and b in TTC decoder probes
  • FC7 fmc l12 spare pinout

channel_b 6 7 GND
general_reset 4 5 channel_a
ref_clk_200MHz 2 3 ipb_clk
fabric_clk 0 1 clk_40_0
+2.5V GND

2.2.4 Debugging TTC decoder with JTAG cable

  • Internal signals of the TTC decoder block in FC7 were probed by ILA cores and debugged by JTAG cable in VIvado Lab Tools
  • Output signals of the IDDR are ttc_q1 and ttc_q2 respectively.
  • After the 11th clock rising edge TTC decoder decides which is channel a and b from ttc_q1 and ttc_q2.
  • Channel a goes through a synchronizer to fit into the BX clock domain and gets into the output if the 10000th clock rising edge meets the synchronization criterion.
  • Channel b is Hamming decoded. ec0, bc0 and reset signals get into the output.

JTAG ttc q1 ttc q2.png
Figure 13. TTC decoder ttc_q1 and ttc_q2
JTAG channel a channel b.png
Figure 14. TTC decoder channel_a and channel_b
JTAG l1a pulse ttc rdy.png
Figure 15. TTC decoder ttc_rdy and l1a_pulse
JTAG bc0 ec0 l1a reset.png
Figure 16. TTC decoder ec0, bc0, l1a and reset
  • Figure 17. shows decoded IDDR output signals according to the measured TTC input signal shown in Figured 7.

Decoded IDDR signal.png
Figure 17. Decoded IDDR output signals

2.2.5 Testing event counter in be_proc

  • First, I bypassed the fast command block, so that the l1a trigger pulse is received directly by the event counter in be_proc.
  • Then, I cut the ec0, bc0 and reset signals from be_proc input in order to avoid resetting the counters, while I am reading out event counter values with python script.
  • Test results showed that the event counter did not count.

2.2.6 Testing event counter in be_proc with ec0

  • ec0 reset was routed to l1a trigger input of be_proc for testing the functionality of event counter.
  • After running the python script, the event counter worked!
  • This behaviour is understandable from previous measurement results.
  • AMC13 only sent bc0 and ec0 resets to FC7.
  • But why it does not send l1a trigger??

2.2.7 Sending L1A triggers to FC7

  • In order to detect incoming l1a triggers in FC7 sent by AMC13 the following initialization procedure was used

Step 1 → Reset AMC13, enable clock distribution and continous trigger through the backplane to the 7th AMC slot in uTCA crate

$ cd amc13-1.2.9/tools/bin/
$ source ../../env.sh
$ AMC13Tool2.exe -c ../../amc13/etc/amc13/connections.xml
No address table path specified.
Using .xml connection file...
Using AMC13 software ver:0
Read firmware versions 0x2264 0x32
flavor = 2 features = 0x000000b2
>rg → general reset
>rc → counter reset
 >en 7 f t → enable loopback mode, TTC clock distribution, fake data and trigger to the 7th AMC slot
>lt c → enable continous trigger mode
>st → status

Step 2 → upload firmware to FC7

$ cd Ph2_ACF/
$ source setup.sh
$ fpgaconfig -f test/20191111_tbalazs_0056.bit -i 20191111_tbalazs_0056.bit -c settings/D19CDescription.xml → set FC7s IP address here
$ cd test/
$ fpgaconfig -i 20191111_tbalazs_0056.bit -c CMSIT.xml
Time to Initialize/configure the system: finished at: Thu Nov 14 13:27:24 2019
elapsed time: 6.71e-07 seconds

Step 3 → Run python script to configure d19c-firmware

$ cd d19c-firmware/sw/fc7/
$ source setup.sh
- PATH += /opt/cactus/bin
- PYTHONPATH += /home/veszpv/d19c-firmware/sw/pychips/src
- LD_LIBRARY_PATH += /home/veszpv/d19c-firmware/sw/fc7/tests/lib
- LD_LIBRARY_PATH += /home/veszpv/d19c-firmware/sw/fc7/fc7/lib
- LD_LIBRARY_PATH += /opt/cactus/lib
- LD_LIBRARY_PATH += /lib
$ cd ../d19cScripts/
$ nano ipaddr.dat → set FC7's IP address
$ python AMC13Tester.py
clock source switched to AMC13 ...
Selected clock source 0

general reset sent to d19c-firmware ...

reset sent to ttc decoder block ...

enable sent to ttc decoder block ...
ttc decoder enable 1

########################################
# library calls
########################################
import sys
from PyChipsUser import *
from time import sleep
from fc7_lib import *
########################################

########################################
# define fc7 object
########################################
fc7AddrTable = AddressTable ("./fc7AddrTable.dat")
f = open('./ipaddr.dat', 'r')
ipaddr = f.readline()
f.close()
fc7 = ChipsBusUdp (fc7AddrTable, ipaddr, 50001)
########################################

# ------------------------------------------------------------------------ #
# clock_source: 0 - AMC13, 2 - backplane, 3 - internal oscillator(default) #
# ------------------------------------------------------------------------ #
fc7.write("clock_source", 0)
print "clock source switched to AMC13 ..."
sleep(1)
print "Selected clock source", fc7.read("clock_source")
sleep(2)

fc7.write("ctrl_command_global_reset", 1)
sleep(0.5)
fc7.write("ctrl_command_global_reset", 0)
print ""
print "general reset sent to d19c-firmware ..."
print ""
sleep(5)

fc7.write("ttc_dec_reset", 1)
sleep(0.5)
fc7.write("ttc_dec_reset", 0)
print "reset sent to ttc decoder block ..."
sleep(1)

print ""
fc7.write("cnfg_ttc_enable", 1)
print "enable sent to ttc decoder block ..."
sleep(1)
print "ttc decoder enable", fc7.read("cnfg_ttc_enable")


Step 4 → Run Vivado Lab Tools to scope the incoming signals

$ source /opt/Xilinx/Vivado_Lab/2017.4/settings64.sh
$ vivado_lab &
→ Open Hardware Manager
→ Open Target → Auto Connect
→ Open debug_nets.ltx

bcf ec0 bc0 l1a.png
Figure 18. Broadcast frame - ILA core
bcf ec0 bc0 large.png
Figure 19. Broadcast frame - ec0 & bc0
bcf l1a large.png
Figure 20. Broadcast frame - l1a

2.2.7 Reading out L1A trigger from event counter in be_proc

  • Testing the proper behaviour of the triggering process is performed by reading out the last value of the 24 bit event counter in be_proc block which counts the incoming L1As from fast command processor block. For the sake of convinience, first I tested the d19c-firmware, then I moved to the IT-firmware where few updates were necessary.
2.2.7.1 d19c-firmware

  • Step 1 → Reset AMC13, enable clock distribution and continous trigger through the backplane to the 7th AMC slot in uTCA crate

    $ cd amc13-1.2.9/tools/bin/
    $ source ../../env.sh
    $ AMC13Tool2.exe -c ../../amc13/etc/amc13/connections.xml
    No address table path specified.
    Using .xml connection file...
    Using AMC13 software ver:0
    Read firmware versions 0x2264 0x32
    flavor = 2 features = 0x000000b2
    >rg → general reset
    >rc → counter reset
    >en 7 f t → enable loopback mode, TTC clock distribution, fake data and trigger to the 7th AMC slot
    >lt c → enable continous trigger mode
    >st → status

    Step 2 → upload firmware to FC7

    $ cd Ph2_ACF/
    $ source setup.sh
    $ cd test/
    $ fpgaconfig -i 20191115_tbalazs_0057.bit -c CMSIT.xml
    Time to Initialize/configure the system: finished at: Wed Nov 20 17:52:46 2019
    elapsed time: 3.63e-07 seconds

    Step 3 → Run python script to read out event counter in d19c-firmware

    $ cd d19c-firmware/sw/fc7/
    $ source setup.sh
    - PATH += /opt/cactus/bin
    - PYTHONPATH += /home/veszpv/d19c-firmware/sw/pychips/src
    - LD_LIBRARY_PATH += /home/veszpv/d19c-firmware/sw/fc7/tests/lib
    - LD_LIBRARY_PATH += /home/veszpv/d19c-firmware/sw/fc7/fc7/lib
    - LD_LIBRARY_PATH += /opt/cactus/lib
    - LD_LIBRARY_PATH += /lib
    $ cd ../d19cScripts/
    $ nano ipaddr.dat → set FC7's IP address
    $ python AMC13Tester.py
    <<<------------------------------------
    clock source switched to AMC13 ...
    Selected clock source 0

    general reset sent to d19c-firmware ...
    ------------------------------------>>>
    <<<------------------------------------
    reset sent to ttc decoder block ...

    enable sent to ttc decoder block ...
    ttc decoder enable 1
    ------------------------------------>>>
    <<<------------------------------------
    reset sent to fast command block ...

    select TTC trigger source in fast command block ...
    fast command block trigger source 1

    enable trigger acception in fast command block ...
    fast command block trigger acception 0

    backpressure enable in fast command block ...
    fast command block backpressure enable 0

    configuring fast command block ...
    fast command block configured 1

    start triggering in fast command block ...
    fast command block fsm configured 1
    fast command block fsm state 1
    fast command block fsm source 1
    fast command block trigger in counter 11264
    fast command block trigger in counter 12393
    fast command block trigger in counter 13522
    fast command block trigger in counter 14651
    fast command block trigger in counter 15780
    fast command backpressure enable 0
    ------------------------------------>>>
    <<<------------------------------------
    Checking actual value of Event Counter in be_proc ...
    evnt_cnt 1
    evnt_cnt_buf_empty 0
    ------------------------------------>>>

  • AMC13Tester.py → AMC13Tester.py.txt
2.2.7.2 IT-firmware

  • Step 1 → Reset AMC13, enable clock distribution and continous trigger through the backplane to the 7th AMC slot in uTCA crate

    $ cd amc13-1.2.9/tools/bin/
    $ source ../../env.sh
    $ AMC13Tool2.exe -c ../../amc13/etc/amc13/connections.xml
    No address table path specified.
    Using .xml connection file...
    Using AMC13 software ver:0
    Read firmware versions 0x2264 0x32
    flavor = 2 features = 0x000000b2
    >rg → general reset
    >rc → counter reset
    >en 7 f t → enable loopback mode, TTC clock distribution, fake data and trigger to the 7th AMC slot
    >lt c → enable continous trigger mode
    >st → status

    Step 2 → upload firmware to FC7

    $ cd Ph2_ACF/
    $ source setup.sh
    $ cd test/
    $ fpgaconfig -i 20191119_tbalazs_0060.bit -c CMSIT.xml
    Time to Initialize/configure the system: finished at: Wed Nov 20 18:28:11 2019
    elapsed time: 5.77e-07 seconds

    Step 3 → Run python script to read out event counter in IT-firmware

    $ cd d19c-firmware/sw/pyDTC/
    $ source env.sh
    $ nano basic_prog.py → set FC7's IP address
    $ python basic_prog.py AMC13Tester
    20-11-19 18:32:00.606007 [7f4da2080740] WARNING - Address overlaps observed - report file written at "/tmp/veszpv/uhal/OverlapReport-home-veszpv-d19c-firmware-IT-sw-pyDTC-.-cfg-.-module_address_table_user.xml.txt"
    20-11-19 18:32:00.607654 [7f4da2080740] WARNING - Address overlaps observed - report file written at "/tmp/veszpv/uhal/OverlapReport-home-veszpv-d19c-firmware-IT-sw-pyDTC-.-cfg-device_address_table_fc7.xml.txt"
    ttc Decoder Initialized
    ------------------------------------------------
    clock source 0
    ttc_enable 1
    ------------------------------------------------

    Fast Commands Block Configured
    ------------------------------------------------
    trigger_source 3
    trigger_state 2
    if_configured 1
    error_code 0
    trigger_cntr 2
    trigger_tag 6
    ------------------------------------------------

    Reading out be_proc event counter
    ------------------------------------------------
    event_cntr 1
    event_cntr_buf_empty 1
    ------------------------------------------------

    1
    --- Registers ----
    Chip: 0, Addr: 0x0, Data: 0x0
    Chip: 0, Addr: 0x0, Data: 0x0
    Chip: 0, Addr: 0x0, Data: 0x0
    Chip: 0, Addr: 0x0, Data: 0x0
    Chip: 0, Addr: 0x0, Data: 0x0
    Chip: 0, Addr: 0x0, Data: 0x0
    Chip: 0, Addr: 0x0, Data: 0x0
    Chip: 0, Addr: 0x0, Data: 0x0

  • basic_prog.py → basic_prog.py.txt

  • it_fast_cmd_cnfg.py → it_fast_cmd_cnfg.py.txt
  • pyDTC.py → pyDTC.py.txt
  • module_address_table_system.xml → module_address_table_system.xml
  • module_address_table_user.xml → module_address_table_user.xml

2.3 Sending clock & trigger to AMC13's TTC input from TTC FC7 with 8SFP FMC

2.3.1 Testing i2c masters sda and scl lines multiplexer in IT- and TTC-uDTC-firmwares

After porting PLL i2c master from IT-uDTC-firmware to TTC-uDTC-firmware test were preformed to make sure if the multiplexing works correctly, Test results can be seen on the table above.

IT-uDTC-firmware
(development branch)		 TTC-uDTC-firmware
KSU L12 Passed Passed
DIO5 L8 Passed Passed
KSU L12 & DIO5 L8 L12 → L8
Passed		 L8 → L12
Passed 		L12 → L8
Passed		 L8 → L12
Passed
Notes:
  • In case of TTC-uDTC-firmware, fabric clock routed to DIO5 channel 1 and the output enableb. Please find the scope diagram in the figures below.

20191213 fabric clock measurement.jpg
Figure 21. Fabric clock measurement
20191213 fabric clock.jpg
Figure 22. Fabric clock scope diagram
  • PLL I2C → DIO5 → Running calibration → Simulation result showed that after DIO5 configuration the PLL I2C master runs immediately again. This means that the PLL will be reconfigured after every DIO5 configuration. Further testing is necessary to check if the chip looses the synchron or not !

KSU L12 IT-uDTC-firmware (development branch)

$ cd /home/veszpv/Ph2_ACF/
$ source setup.sh
$ cd test
$ fpgaconfig -i IT_dev_L12KSU_L8DIO5.bit -c CMSIT_FC7_1.xml

$ cd /home/veszpv/d19c-firmware-IT-development/sw/pyDTC
$ source env.sh
$ python basic_prog.py l12_pwr_on
$ python basic_prog.py rst
$ python basic_prog.py init

DIO5 L8 IT-uDTC-firmware (development branch)

$ cd /home/veszpv/Ph2_ACF/
$ source setup.sh
$ cd test
$ fpgaconfig -i IT_dev_L12KSU_L8DIO5.bit -c CMSIT_FC7_1.xml

$ cd /home/veszpv/d19c-firmware-IT-development/sw/pyDTC
$ source env.sh
$ python basic_prog.py l8_pwr_on
$ python basic_prog.py conf_dio5

KSU L12 → DIO5 L8 IT-uDTC-firmware (development branch)

$ cd /home/veszpv/Ph2_ACF/
$ source setup.sh
$ cd test
$ fpgaconfig -i IT_dev_L12KSU_L8DIO5.bit -c CMSIT_FC7_1.xml

$ cd /home/veszpv/d19c-firmware-IT-development/sw/pyDTC
$ source env.sh
$ python basic_prog.py l12_pwr_on
$ python basic_prog.py rst
$ python basic_prog.py init

$ python basic_prog.py l8_pwr_on
$ python basic_prog.py conf_dio5

DIO5 L8 → KSU L12 IT-uDTC-firmware (development branch)

$ cd /home/veszpv/Ph2_ACF/
$ source setup.sh
$ cd test
$ fpgaconfig -i IT_dev_L12KSU_L8DIO5.bit -c CMSIT_FC7_1.xml

$ cd /home/veszpv/d19c-firmware-IT-development/sw/pyDTC
$ source env.sh
$ python basic_prog.py l8_pwr_on
$ python basic_prog.py conf_dio5

$ python basic_prog.py l12_pwr_on
$ python basic_prog.py rst
$ python basic_prog.py init

KSU L12 TTC-uDTC-firmware (development branch)

$ cd /home/veszpv/Ph2_ACF/
$ source setup.sh
$ cd test
$ fpgaconfig -i 20191211_tbalazs_0067.bit -c CMSIT_FC7_1.xml

$ cd /home/veszpv/d19c-firmware-TTC/024_project/ttc-udtc-firmware/sw/pyDTC
$ source env.sh
$ python basic_prog.py l12_pwr_on
$ python basic_prog.py test_KSU

DIO5 L8 TTC-uDTC-firmware (development branch)

$ cd /home/veszpv/Ph2_ACF/
$ source setup.sh
$ cd test
$ fpgaconfig -i 20191211_tbalazs_0067.bit -c CMSIT_FC7_1.xml

$ cd /home/veszpv/d19c-firmware-TTC/024_project/ttc-udtc-firmware/sw/pyDTC
$ source env.sh
$ python basic_prog.py l8_pwr_on
$ python basic_prog.py test_DIO5_input

KSU L12 → DIO5 L8 TTC-uDTC-firmware (development branch)

$ cd /home/veszpv/Ph2_ACF/
$ source setup.sh
$ cd test
$ fpgaconfig -i 20191211_tbalazs_0067.bit -c CMSIT_FC7_1.xml

$ cd /home/veszpv/d19c-firmware-TTC/024_project/ttc-udtc-firmware/sw/pyDTC
$ source env.sh
$ python basic_prog.py l12_pwr_on
$ python basic_prog.py test_KSU

$ python basic_prog.py l8_pwr_on
$ python basic_prog.py test_DIO5_input

DIO5 L8 → KSU L12 TTC-uDTC-firmware (development branch)

$ cd /home/veszpv/Ph2_ACF/
$ source setup.sh
$ cd test
$ fpgaconfig -i 20191211_tbalazs_0067.bit -c CMSIT_FC7_1.xml

$ cd /home/veszpv/d19c-firmware-TTC/024_project/ttc-udtc-firmware/sw/pyDTC
$ source env.sh
$ python basic_prog.py l8_pwr_on
$ python basic_prog.py test_DIO5_input

$ python basic_prog.py l12_pwr_on
$ python basic_prog.py test_KSU

Note → i2c multiplexer removed later on from the firmware because 8SFP FMC does not use the fmc_i2c_sda and fmc_i2c_scl pins in TTC-uDTC-firmware, instead it used the fmc_l12_la_p(32) and fmc_l12_la_n(32) FMC pins. It was necessary to port 8SFP FMC to the physical abstraction layer as well.

2.3.1 Sending clock & trigger from TTC_FC7 to IT_FC7 through AMC13

  • Testing the proper behaviour of the triggering process is performed by initializing TTC_FC7, AMC13 and IT-FC7. TTC_FC7 initialized by poweriering L12 FMC, programming i2c of SFP adapter and sending out the trigger to AMC13. Clock distributon needs to be enabled in AMC13 in order to send clock & trigger to TTC-FC7. In case of TTC_FC7 running of AMC13Tester python script is necessary to choose the appropriate clock source for receiving trigger.
2.2.7.1 TTC-uDTC-firmware

  • Step 1 → upload bitstream to TTC_FC7

$ cd Ph2_ACF_IT-chipPolymorphism/
$ source setup.sh
$ cd test/
$ fpgaconfig -f 20200203_tbalazs_0103.bit -i 20200203_tbalazs_0103.bit -c CMSIT_FC7_2.xml
$ fpgaconfig -i 20200203_tbalazs_0103.bit -c CMSIT_FC7_2.xml

  • Step 2 → Run python script to power L12 FMC, program i2c of SFP adapter and sending out 1 trigger

$ cd d19c-firmware-TTC/036_project/ttc-udtc-firmware/sw/pyDTC/
$ source env.sh
$ nano basic_prog.py → set FC7's IP address
$ python basic_prog.py l12_pwr_on
------------------------------------------------
clk_rate_1 31250000
clk_rate_2 40078519
clk_rate_3 40078519
clk_rate_4 160314077
clk_rate_5 0
clk_rate_6 125000000
clk_rate_7 0
------------------------------------------------
$ python basic_prog.py prog_i2c
$ python basic_prog.py trigger
Changing trigger to: 1
Triggering started
Sending trigger
Trigger 0 sent
Triggering stopped

103 1 prog i2c.png
Figure 23. SFP adapter i2c programming (3rd cycle in not on the diagram)
103 2 reset before trigger.png
Figure 24. Reset before trigger
103 3 trigger.png
Figure 25. Trigger sent out to AMC13's TTC input

103 4 ttc pattern 1.png
Figure 26. First pattern on ttc_out
103 5 ttc pattern 2.png
Figure 27. Second pattern on ttc_out
2.2.7.2. AMC13
  • Step 1 → Reset AMC13, enable clock distribution and continous trigger through the backplane to the 7th AMC slot in uTCA crate

$ cd amc13-1.2.9/tools/bin/
$ source ../../env.sh
$ AMC13Tool2.exe -c ../../amc13/etc/amc13/connections.xml
No address table path specified.
Using .xml connection file...
Using AMC13 software ver:0
Read firmware versions 0x2264 0x32
flavor = 2 features = 0x000000b2
>ws 0x0 0x10 → T2 reset
>wv 0x0 0x0 → T1 reset
>rg → general reset
>rc → counter reset
>en 7 → enable TTC clock distribution to the 7th AMC slot
>st → status

2.2.7.2 IT-uDTC-firmware

  • Step 1 → upload bitstream to FC7

$ cd Ph2_ACF_IT-chipPolymorphism/
$ source setup.sh
$ cd test/
$ fpgaconfig -f 20200131_tbalazs_0102.bit -i 20200131_tbalazs_0102.bit -c CMSIT_FC7_1.xml
$ fpgaconfig -i 20200131_tbalazs_0102.bit -c CMSIT_FC7_1.xml

  • Step 2 → Run python script to switxh clock source in IT-uDTC-firmware

$ cd d19c-firmware-IT-development/sw/pyDTC/
$ source env.sh
$ nano basic_prog.py → set FC7's IP address
$ python basic_prog.py AMC13Tester
ttc Decoder Initialized
------------------------------------------------
clock source 0
ttc_enable 1
------------------------------------------------

Fast Commands Block Configured
------------------------------------------------
trigger_source 3
trigger_state 2
if_configured 1
error_code 0
trigger_cntr 0
trigger_tag 0
------------------------------------------------

Reading out be_proc event counter
------------------------------------------------
event_cntr 0
event_cntr_buf_empty 1
------------------------------------------------

------------------------------------------------
i2c_init 0
clk_gen_lock 1
gtx_lock 0b0
link aurora 0
lane_up 0b0
channel_up 0b0
------------------------------------------------
clk_rate_1 0
clk_rate_2 40079
------------------------------------------------

1
--- Registers ----
Chip: 0, Addr: 0x0, Data: 0x0
Chip: 0, Addr: 0x0, Data: 0x0
Chip: 0, Addr: 0x0, Data: 0x0
Chip: 0, Addr: 0x0, Data: 0x0
Chip: 0, Addr: 0x0, Data: 0x0
Chip: 0, Addr: 0x0, Data: 0x0
Chip: 0, Addr: 0x0, Data: 0x0
Chip: 0, Addr: 0x0, Data: 0x0



102 1 trigger received.png
Figure 28. Trigger received in IT_FC7

-- TamasBalazs - 2019-10-26 Contact

Institute for Particle and Nuclear Physics --------------------------
BalazsTamasSmall_2.jpg
Wigner Research Centre for Physics
Hungarian Academy of Sciences
--------------------------------------------------------------------------------------------------
Address: -
 Konkoly Thege Miklos street 29-33. H-1121 Budapest
Letters: H-1525 Budapest, P.O.Box 49, Hungary
Phone: (+36-1) 392-2222 ext. 3486
Fax:_ (+36-1) 392-2598
E-mail: balazs.tamas@wigner.mta.hu
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng 102_1_trigger_received.png r1 manage 47.5 K 2020-02-03 - 18:30 TamasBalazs  
PNGpng 103_1_prog_i2c.png r1 manage 79.5 K 2020-02-03 - 18:08 TamasBalazs  
PNGpng 103_2_reset_before_trigger.png r1 manage 79.2 K 2020-02-03 - 18:18 TamasBalazs  
PNGpng 103_3_trigger.png r1 manage 79.0 K 2020-02-03 - 18:18 TamasBalazs  
PNGpng 103_4_ttc_pattern_1.png r1 manage 78.2 K 2020-02-03 - 18:23 TamasBalazs  
PNGpng 103_5_ttc_pattern_2.png r1 manage 78.8 K 2020-02-03 - 18:23 TamasBalazs  
PDFpdf 20180614_IT-uDTC_developers_meeting.pdf r1 manage 1031.0 K 2019-10-26 - 16:27 TamasBalazs  
PDFpdf 20180619_IT-uDTC_developers_meeting.pdf r1 manage 135.5 K 2019-10-26 - 16:26 TamasBalazs  
PDFpdf 20180629_DAQ_development_for_IT_test_systems.pdf r1 manage 1794.1 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20180713_DAQ_development_for_IT_test_systems.pdf r1 manage 642.7 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20180727_DAQ_development_for_IT_test_systems.pdf r1 manage 179.4 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20180824_DAQ_development_for_IT_test_systems.pdf r1 manage 101.5 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20180907_DAQ_development_for_IT_test_systems.pdf r1 manage 99.7 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20180914_DAQ_development_for_IT_test_systems.pdf r1 manage 112.9 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20180921_DAQ_development_for_IT_test_systems.pdf r1 manage 130.2 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20180928_DAQ_development_for_IT_test_systems.pdf r1 manage 109.2 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20181005_DAQ_development_for_IT_test_systems.pdf r1 manage 115.8 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20181012_DAQ_development_for_IT_test_systems.pdf r1 manage 108.5 K 2019-10-26 - 16:34 TamasBalazs  
PDFpdf 20181019_DAQ_development_for_IT_test_systems.pdf r1 manage 116.8 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20181102_DAQ_development_for_IT_test_systems.pdf r1 manage 111.3 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20181130_DAQ_development_for_IT_test_systems.pdf r1 manage 3945.2 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20190128_DAQ_development_for_IT_test_systems.pdf r1 manage 158.3 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20190211_DAQ_development_for_IT_test_systems.pdf r1 manage 91.8 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20190215_DAQ_development_for_IT_test_systems.pdf r1 manage 77.1 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20190311_DAQ_development_for_IT_test_systems.pdf r1 manage 91.0 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20190325_DAQ_development_for_IT_test_systems.pdf r1 manage 208.1 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20190506_DAQ_development_for_IT_test_systems.pdf r1 manage 336.2 K 2019-10-26 - 16:37 TamasBalazs  
PDFpdf 20190527_Discussion_on_a_dedicated_FC7-based_TTC_board.pdf r1 manage 384.3 K 2019-10-26 - 16:38 TamasBalazs  
PDFpdf 20190603_Dedicated_FC7_based_TTC_card.pdf r1 manage 464.1 K 2019-10-26 - 16:38 TamasBalazs  
PDFpdf 20190617_Dedicated_FC7_based_TTC_card.pdf r1 manage 464.1 K 2019-10-26 - 16:38 TamasBalazs  
PDFpdf 20190701_Dedicated_FC7_based_TTC_card.pdf r1 manage 464.1 K 2019-10-26 - 16:38 TamasBalazs  
PDFpdf 20190826_DAQ_development_for_IT_and_OT.pdf r1 manage 88.7 K 2019-10-26 - 16:38 TamasBalazs  
JPEGjpg 20191016_192611.jpg r1 manage 3427.3 K 2019-10-27 - 14:34 TamasBalazs  
PDFpdf 20191021_DAQ_development_for_IT_and_OT.pdf r1 manage 330.7 K 2019-10-26 - 16:38 TamasBalazs  
PDFpdf 20191118_DAQ_development_for_IT_and_OT.pdf r1 manage 547.5 K 2019-11-18 - 17:31 TamasBalazs  
PDFpdf 20191205_Development_of_a_trigger_card_on_FC7.pdf r1 manage 544.5 K 2019-12-13 - 14:24 TamasBalazs  
JPEGjpg 20191213_fabric_clock.jpg r1 manage 4188.6 K 2019-12-13 - 14:23 TamasBalazs  
JPEGjpg 20191213_fabric_clock_measurement.jpg r1 manage 1285.6 K 2019-12-13 - 14:23 TamasBalazs  
PDFpdf 20200210_Development_of_a_trigger_card_on_FC7_VV.pdf r1 manage 2551.8 K 2020-02-12 - 10:24 TamasBalazs  
PDFpdf 20200511_Development_of_a_trigger_card_on_FC7_VV.pdf r1 manage 1073.8 K 2021-04-02 - 19:35 TamasBalazs  
PDFpdf 20201214_Development_of_a_trigger_card_on_FC7_VV.pdf r1 manage 1031.2 K 2021-04-02 - 19:42 TamasBalazs  
PDFpdf 20211011_Development_of_a_trigger_card_on_FC7_2.pdf r1 manage 1034.8 K 2021-10-11 - 15:55 TamasBalazs  
PDFpdf 20220613_DAQ_for_IT_and_OT.pdf r1 manage 225.0 K 2022-06-13 - 15:37 TamasBalazs  
Texttxt AMC13Tester.py.txt r1 manage 10.9 K 2019-11-20 - 18:48 TamasBalazs  
PDFpdf AMC13_Operational_Spec_05_13_2013.pdf r1 manage 420.7 K 2019-11-26 - 14:24 TamasBalazs  
PNGpng AMC13_uTCA_FC7.png r1 manage 266.6 K 2019-11-04 - 19:38 TamasBalazs  
JPEGjpg BalazsTamasSmall_2.jpg r1 manage 26.8 K 2019-10-26 - 15:21 TamasBalazs  
PNGpng CLOCK_IT-firmware.png r1 manage 718.2 K 2019-11-04 - 22:52 TamasBalazs  
PNGpng Decoded_IDDR_signal.png r1 manage 22.8 K 2019-11-08 - 13:27 TamasBalazs  
PNGpng JTAG_bc0_ec0_l1a_reset.png r1 manage 48.7 K 2019-11-07 - 15:15 TamasBalazs  
PNGpng JTAG_channel_a_channel_b.png r1 manage 40.0 K 2019-11-07 - 15:15 TamasBalazs  
PNGpng JTAG_l1a_pulse_ttc_rdy.png r1 manage 84.8 K 2019-11-07 - 15:15 TamasBalazs  
PNGpng JTAG_ttc_q1_ttc_q2.png r1 manage 40.1 K 2019-11-07 - 15:15 TamasBalazs  
PDFpdf T1-20121026.pdf r1 manage 6244.8 K 2019-10-28 - 17:29 TamasBalazs  
PDFpdf T2-20110518.PDF r1 manage 2442.7 K 2019-10-28 - 17:36 TamasBalazs  
PDFpdf T3-201105012.PDF r1 manage 228.9 K 2019-10-28 - 17:29 TamasBalazs  
JPEGjpg TTC_channel_a_b_desktop.jpg r1 manage 1258.5 K 2019-11-06 - 16:42 TamasBalazs  
JPEGjpg TTC_channel_a_b_probes.jpg r1 manage 593.7 K 2019-11-06 - 16:42 TamasBalazs  
JPEGjpg TTC_channel_a_b_scope.jpg r1 manage 4802.8 K 2019-11-06 - 16:50 TamasBalazs  
JPEGjpg TTC_input_clock_desktop.jpg r1 manage 1511.3 K 2019-11-06 - 15:18 TamasBalazs  
JPEGjpg TTC_input_clock_probes.jpg r1 manage 4346.7 K 2019-11-06 - 15:18 TamasBalazs  
JPEGjpg TTC_input_clock_scope.jpg r1 manage 4502.7 K 2019-11-06 - 15:18 TamasBalazs  
JPEGjpg TTC_input_scope.jpg r1 manage 3969.2 K 2019-11-06 - 15:08 TamasBalazs  
JPEGjpg TTC_input_scope_conf.jpg r1 manage 4678.4 K 2019-11-06 - 15:08 TamasBalazs  
Texttxt basic_prog.py.txt r1 manage 2.7 K 2019-11-20 - 19:03 TamasBalazs  
PNGpng bcf_ec0_bc0_l1a.png r1 manage 55.1 K 2019-11-14 - 14:23 TamasBalazs  
PNGpng bcf_ec0_bc0_large.png r1 manage 35.3 K 2019-11-14 - 14:23 TamasBalazs  
PNGpng bcf_l1a_large.png r1 manage 34.7 K 2019-11-14 - 14:23 TamasBalazs  
Unknown file formatgz ic_mmc_v1_6_6.tar.gz r1 manage 7276.0 K 2019-10-29 - 15:18 TamasBalazs  
Texttxt it_fast_cmd_cnfg.py.txt r1 manage 1.5 K 2019-11-20 - 19:03 TamasBalazs  
XMLxml module_address_table_system.xml r1 manage 0.7 K 2019-11-20 - 19:03 TamasBalazs  
XMLxml module_address_table_user.xml r1 manage 32.4 K 2019-11-20 - 19:03 TamasBalazs  
Texttxt pyDTC.py.txt r1 manage 34.6 K 2019-11-20 - 19:03 TamasBalazs  
Edit | Attach | Watch | Print version | History: r47 < r46 < r45 < r44 < r43 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r47 - 2022-09-12 - TamasBalazs
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback