Notes for myself on various CMS related things


Basically, you connect to `cmsusr`, then to one of the GEM machines (gem-daq01, gem-daq02, gemvm-legacy, or gemvm-test), and finally to the CTP7 with their geographical IP address (gem-shelfXX-amcYY).
For gem_reg: as gempro (sudo -u gempro -i), you can simply use the command. Then you can connect to a given CTP7 with `connect gem-shelfXX-amcYY`.



On gem904atca1 the p1p1 interface is connected to the switch, and should have a static address of on subnet
 sudo ifconfig p1p1 netmask 
IP addresses of the ATCA local network:
Node IP
switch CPU 10Gb/s port
switch CPU 1Gb/s port

There's a DHCP server running on the gem904atca1:

yum install dhcp
bind it to the p1p1 inteface as explained in /etc/sysconfig/dhcpd:
# $ cp /usr/lib/systemd/system/dhcpd.service /etc/systemd/system/
# $ vi /etc/systemd/system/dhcpd.service
# $ ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid <your_interface_name(s)>
# $ systemctl --system daemon-reload
# $ systemctl restart dhcpd.service

put this in the config file /etc/dhcp/dhcpd.conf:
#subnet netmask {
subnet netmask {
 option routers        ;
# option subnet-mask    ;
 option subnet-mask    ;
 option domain-name              "atca1";
 option domain-name-servers;
# range;

host dth {
 hardware ethernet 00:30:64:5d:90:e7;


Manual here:


Driver documentation:


  • Download fw to the PROM using Bittware tools
  • Program the clocks using Bittware tools
  • Setup the driver:
sudo insmod build/qdma.ko
dmactl dev list # get the dev ID from here, should be something like qdma05000 (depends on the slot)
#assuming the dev ID is qdma05000, the sysfs path should be: /sys/bus/pci/devices/0000:05:00.0
#you can read/write regs using /sys/bus/pci/devices/0000:05:00.0/resource2  (this points to BAR2)
#you can run the slow control test program based on pcimem like this:
sudo ./pcimem /sys/bus/pci/devices/0000:05:00.0/resource2 0xc00000 w*3


Zynq board: MYD-C7Z015-4E1D-766-I Zynq IP at 904:


~/clock/ configures all async clocks (refclk1) to 250MHz ~/clock/si5345_config configures all sync clocks (refclk0) to 160MHz


base address: 0x50000000 AXI timeout: 100us

Zynq XVC setup

  • cd xvc
  • ./xvcServer_ioctl -d /dev/xilinx_xvc_driver_0
  • in vivado: open_hw_target -xvc_url

Zynq I2C

OUTDATED, NOT NEEDED ANYMORE: this setup has 3 I2C links, for two FPGA modules and the optical module. They are selectable by an internal firmware register. Currently, the FPGA module is on I2C link #1, but the default selection is link #0. To select link #1, do this command: devmem 0x41200000 32 1


Testing instructions for Rice

after a power-cycle of the uTCA crate, you need to configure the AMC13 by running the AMC13Tool, and running this command: "en 1-12 t". To connect:
  * At TAMU from elli machine run: /opt/cactus/bin/amc13/AMC13Tool2.exe -p /opt/cactus/etc/amc13 -c
  * At Rice from bonner-daq run: AMC13Tool2.exe -c

login to the ctp7: ssh texas@eagle42
if CTP7 firmware is not loaded: cd ~/tamu &&
load the correct OH firmware to the CTP7 RAM: cd ~/tamu && (use for v1 OHs)
cd ~/apps/reg_interface
configure the GBTs with a given config file (the first argument is the OH number, and the second argument is the GBT number within the OH):
python 0 0 config ~/gbt_config/GBTX_GE21_OHv2_GBT_0_minimal_2020-01-17.txt
python 0 1 config ~/gbt_config/GBTX_GE21_OHv2_GBT_1_minimal_2020-01-17.txt
perform a phase scan (notice the config file is different when you do it on gbt0 and gbt1):
python 0 0 ge21-phase-scan ~/gbt_config/GBTX_GE21_OHv2_GBT_0_minimal_2020-01-17.txt
python 0 1 ge21-phase-scan ~/gbt_config/GBTX_GE21_OHv2_GBT_1_minimal_2020-01-17.txt

promless programming loop test:
python 1 1000
first argument is the OH "bitmask" e.g. 1 will do it on OH0 only, 2 will do it on OH1 only, 3 will do it on both OH0 and OH1, etc (though normally I only ever do it on one OH at a time, so running on multiple may work, but no guarantee). The second argument is the number of iterations to do (normally I do 1000 for a full test, but you can run e.g. 10 or 100 just to see if it works, which is quicker)

read SCA ADCs:
python 1 adc-read
the first argument gere is again the OH "bitmask" (so 1 for OH0, 2 for OH1, 4 for OH2, etc)
note that this thing crashes when it reaches an ADC channel that is in overflow, which e.g. was the case on the first OHv2 board that had some problem with the temp sensor (Mike probably remembers better than me what it actually was)

for PRBS tests, by default the loads the OHv2 loopback firmware, and when you run the it will get loaded to the OH FPGA. At that point in order to start the loopback test you have to open the reg_interface by just running "reg", and then:
1) write GEM_AMC.GEM_TESTS.OH_LOOPBACK.CTRL.OH_SELECT 0  (this selects the OH to be used, normally it should be 0 if you just use the first OH)
2) write GEM_AMC.GEM_SYSTEM.TESTS.GBT_LOOPBACK_EN 1   (this will put the CTP7 into a loopback testing mode, note that VFAT or OH communication won't work while the CTP7 is in this mode, so e.g. phase scans and OH programming are not possible, so always make sure this mode is off when doing the other tests and only put this on for PRBS test)
4) don't forget to switch the CTP7 back to normal mode: write GEM_AMC.GEM_SYSTEM.TESTS.GBT_LOOPBACK_EN 0

step 2 will print lots of stuff which includes 3 registers for each elink:
  1) PRBS_LOCKED -- if this is 1 that means the CTP7 is receiving back a valid PRBS7 stream from this elink
  2) MEGA_WORD_CNT -- this counts the total number of words checked by the PRBS7 checker after the PRBS_LOCKED was asserted (this counts in units of one million bytes)
  3) ERROR_CNT -- number of PRBS errors found

the output will look like this:
0x66804040 r GEM_AMC.GEM_TESTS.OH_LOOPBACK.GBT_0.ELINK_0.ERROR_CNT    0x00000000
0x66804048 r GEM_AMC.GEM_TESTS.OH_LOOPBACK.GBT_0.ELINK_1.ERROR_CNT    0x00000000
0x66804050 r GEM_AMC.GEM_TESTS.OH_LOOPBACK.GBT_0.ELINK_2.ERROR_CNT    0x00000000
the elinks are numbered in kind of natural order in terms of GBTX: ELINK_0 is DIN0, ELINK_1 is DIN4, ELINK_2 is DIN8, ELINK_3 is DIN12, .... ELINK_10 is DIO1, ELINK_11 is DIO5, ELINK_12 is DIO9, ELINK_13 is DIO13. Not all of these elinks are connected to the OH FPGA, so you should not expect all of them to lock, you should only see these elinks lock:

you can also try to inject a PRBS error on the CTP7 TX, and see if it comes back through the RX and is detected by the CTP7: write GEM_AMC.GEM_TESTS.OH_LOOPBACK.CTRL.INJECT_ERR 1

to reset the loopback counters use this reset: write GEM_AMC.GEM_TESTS.OH_LOOPBACK.CTRL.RESET 1 


Operations related stuff


For global runs TCDS TTS logs can be found in tcds-control-cpm-pri:/cmsnfstcds_history/tcds_history, though old files are periodically moved somewhere else (or perhaps removed altogether). More info here:
I have a script to parse these files through and give a summary of how many times GEM has transitioned into various states:

Read-only persistent partition

run this as root to switch to rw in runtime: setpersistent rw edit this file to make it persistent between reboots: /mnt/persistent/config/persistent_writeable

Notes on CTP7 synthesizer configuration

Here's an email from Tom about configuring the synthesizers to provide a 200MHz clock to MGT REFCLK1 both front and back for CSC_FED:

Attached is text file containing register values for the SI Labs 5324 Clock Synthesizer on the CTP7. This register programming will take the 125.000 MHz on-board oscillator and produce a 200.000 MHz output clock for distribution on the refclk fanout circuitry.

Instructions for using this file are as follows:

1) copy it onto the persistent file system somewhere on the CTP7 (/mnt/persistent/)

2) from the folder where this file is resident, execute the following command on the CTP7 Linux:

clockinit GEM_SynthA_125in_200_off_out.txt 320_160 B1 A0 A0 B1

This command should replace the "clockinit" command that gets executed currently somewhere in your init code, which, IIRC, looks something like:

clockinit 320_160 320_160 B1 A1 A1 B1

The clockinit command does two things:

1. Configure both SI5324 synthesizers, and

2. Configure the crosspoint switch that maps the synthesizer outputs to refclk outputs.

With the new command, we are programming synthesizer A to use the custom file to produce a 200 MHz clock on its output 0, with output 1 turned off, using the 125.000 MHz on board oscillator as a reference. We are programming synthesizer B to put out a 320.64 MHz clock on output 0, and a 160.32 MHz clock on output 1 using the TTC 40 MHz as a reference.

So the new synthesizer output frequencies are:
a0 = 200.000 MHz
a1 = off
b0 = 320.64 MHz (from TTC40)
b1 = 160.32 MHz (from TTC40)

The crosspoint switch outputs are mapped to the MGT link refclk inputs as follows--

Output 0 -> Front (CXP) side Refclk 0
Output 1 -> Front (CXP) side Refclk 1
Output 2 -> Back (miniPOD) side Refclk 1
Output 3 -> Back (miniPOD) side Refclk 0

So the list "B1 A0 A0 B1" in the clockinit command is performing the following mapping:

Synth Freq b1 (160.32 MHz) -> Crosspoint Output 0 (front refclk 0)
Synth Freq a0 (200 MHz) -> Crosspoint Output 1 (front refclk 1)
Synth Freq a0 (200 MHz) -> Crosspoint Output 2 (back refclk 1)
Synth Freq b1 (160.32 MHz) -> Crosspoint Output 3 (back refclk 0)

Eventually all this will have to be written up in a more comprehensive way. Hopefully this is enough for now to get you going and to understand the deeper functionality of the refclk distribution system a bit more.

When instantiating links, remember that on the front side, banks 111, 113, 116 and 118 receive the refclk inputs, both 0 and 1, and the back side similarly as the refclk inputs at banks 211, 213, 216 and 218. A MGT instantiation should pick up its refclk either from the inputs associated with it's bank, or from an adjacent bank. The distribution patterns above support this policy.

Raw data


The streamer files can be found on BU machines here: /store/lustre/scratch_minidaq/run/ (GEM miniDAQ BU machine currently is bu-c2f13-16-01)
The format of these files is described here: (note that the event data BLOB is compressed with zlib).
I have a python script to unpack that data here:
Remi gave this snippet of code on how to unpack these files with CMSSW:
import FWCore.ParameterSet.Config as cms
process = cms.Process("TEST")
process.source = cms.Source("NewEventStreamFileReader",
   fileNames = cms.untracked.vstring(

process.print1 = cms.OutputModule("AsciiOutputModule")
process.p1 = cms.EndPath(process.print1)

GEM Data Format
B1 C1 D1

GEMOS legacy install instructions

Add the necessary dependency repos

  • /etc/yum.repos.d/xdaq.repo:
name     = XDAQ Software Base
baseurl  =
enabled  = 1
gpgcheck = 0

name     = XDAQ Software Base Sources
baseurl  =
enabled  = 0
gpgcheck = 0

name     = XDAQ Software Updates
baseurl  =
enabled  = 1
gpgcheck = 0

name     = XDAQ Software Updates Sources
baseurl  =
enabled  = 0
gpgcheck = 0

name     = XDAQ Software Extras
baseurl  =
enabled  = 1
gpgcheck = 0

name     = XDAQ Software Kernel Modules
baseurl  =
enabled  = 1
gpgcheck = 0

name     = XDAQ Software Base (Unstable)
baseurl  =
enabled  = 0
gpgcheck = 0

name     = XDAQ Software Extras (Unstable)
baseurl  =
enabled  = 0
gpgcheck = 0

name     = XDAQ Software Kernel Modules (Unstable)
baseurl  =
enabled  = 0
gpgcheck = 0

  • /etc/yum.repos.d/ipbus-sw.repo:
name=IPbus software repository
# baseurl=

name=IPbus software repository updates
# baseurl=

  • /etc/yum.repos.d/cactus-amc13.repo:
name=CACTUS Project Software Repository for amc13 packages
# baseurl=

name=CACTUS Project Software Repository Updates for amc13 packages
# baseurl=

Install external dependencies

sudo yum groups install extern_coretools coretools extern_powerpack powerpack
sudo yum groupinstall amc13
For me this conflicted with openslp package, so I had to remove that:
sudo yum remove openslp

Install GEMOS

sudo yum-config-manager --enable cernonly
sudo yum install cx_Oracle
sudo curl -L -o /etc/yum.repos.d/gemos.repo
sudo yum install xhal reedmuller wiscrpcsvc gempython\*

Setup the CTP7 supporting infrastructure


Installing the libraries on the CTP7

curl -L -O

The internal structure is:


What needs to be run on the CTP7???

Running the software

Power-cycling boards via NAT MCH

1. From gem904daq01 or gem904daq04 PC log in to MCH: telnet gem904mch02
2. using the MCH interface, powercycle the board. Example:


FRU Information:
 FRU  Device   State  Name
  0   MCH       M4    NMCH-CM
  3   mcmc1     M4    NAT-MCH-MCMC
 11   AMC7      M4    WISC CTP-7
 30   AMC13     M4    BU AMC13
 40   CU1       M4    Schroff uTCA CU
 41   CU2       M4    Schroff uTCA CU
 50   PM1       M4    NAT-PM-DC840
 51   PM2       M4    NAT-PM-DC840
nat> shutdown 30

FRU Information:
 FRU  Device   State  Name
  0   MCH       M4    NMCH-CM
  3   mcmc1     M4    NAT-MCH-MCMC
 11   AMC7      M4    WISC CTP-7
 30   AMC13     M1    BU AMC13
 40   CU1       M4    Schroff uTCA CU
 41   CU2       M4    Schroff uTCA CU
 50   PM1       M4    NAT-PM-DC840
 51   PM2       M4    NAT-PM-DC840
nat> fru_start 30
nat> FRU Information:
 FRU  Device   State  Name
  0   MCH       M4    NMCH-CM
  3   mcmc1     M4    NAT-MCH-MCMC
 11   AMC7      M4    WISC CTP-7
 30   AMC13     M4    BU AMC13
 40   CU1       M4    Schroff uTCA CU
 41   CU2       M4    Schroff uTCA CU
 50   PM1       M4    NAT-PM-DC840
 51   PM2       M4    NAT-PM-DC840
nat> exit

Producing a pickle file

  • copy the new XML (CTP7 and/or OH) to gem904daq01:~/gem_pickle
  • login to gem904daq01
  • update the xml symlink(s)
  • run
  • exit out of the reg_interface, and edit the pickle as described in the message

Full instructions from Misha:

1. Login to the PC where the GEM software is centrally installed, e.g. gem904daq01 or gem904qc8daq and set the following env variables:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/wiscrpcsvc/lib/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/xhal/lib/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/rwreg/lib/

export PATH=$PATH:/opt/reg_utils/bin/
export PATH=$PATH:/opt/xhal/bin/

2. Prepare the test environment a la:
$ mkdir ~/test_address_tables
$ export GEM_ADRESS_TABLE_PATH=~/test_address_tables
$ ln -sf /data/bigdisk/GEMDAQ_Documentation/system/firmware/files/CTP7/vX.Y.Z/<address_table_filename>.xml amc_address_table_top.xml
$ ln -sf /data/bigdisk/GEMDAQ_Documentation/system/firmware/files/OptoHybrid/GE2.1_Artix7/<address_table_filename>.xml optohybrid_registers.xml

3. In case you’re reusing the area created in p2, make sure you delete any previously made pickle files

4. Run:
$ /opt/xhal/bin/
This will produce a pickle file under $GEM_ADRESS_TABLE_PATH/amc_address_table_top.pickle This file is usable from the host PC - please provide it as is it to whoever is going to operate the system from the host. It has to be modified to be used on the card - see next bullet.

5. In order to use the pickle file on the card, modify it in a following way:
  - open it with your preferred text editor (I use vim)
  - on the line 3, change "creg_utils.reg_interface.common.reg_xml_parser" to “crw_reg”. Make sure you don’t change invisible symbols like end of line.

6. Copy modified pickle file to eagleXX:/mnt/persistent/gemdaq/xml/amc_address_table_top.pickle

Fiber connections at 904

M2 and M3 are connected to eagle64, but M3 is actually using M1 fibers, so M3 is OH0, and M2 is OH1 (both in gbt and trigger)
M4 has OHv2 and it's connected to APEX using fibers 15+22 for GBT0, and 16+21 for GBT1, which is then connected to QSFPs

Trigger fibers on eagle64:
MTP48 to EMTF: MTP12#4, fibers 1, 2, 3, 4, 5, 6, 7, 8 connect to 1, 2, 3, 4, 5, 6, 7, 8 on EMTF side
MTP48 to OH: MTP12#3 fibers:
  12, 11 -- M4
  10, 9  -- TX 9 & 10
  8,  7  -- M2
  5,  6  -- M1 (this actually goes to M3 right now)
  2 -- TX 12
  3 -- M3 trigger
  4 -- M3 trigger

eagle64 is installed in slot 5

OH #4 being shipped to me

Hog and gitlab-ci

Install ccze -- this will colorize the shell and highlight errors and warnings in Hog output, super useful.

Install tcllib

Gitlab runner registration
Some simple instructions for registering a Gitlab runner

Install gitlab-runner

Execute gitlab-runner register

At the prompt of “Please enter the gitlab-ci coordinator URL (e.g.”, enter:
At the prompt of “Please enter the gitlab-ci token for this runner:”, enter the token that you get from Settings -> CI/CD -> Runners –> Set up a specific Runner manually.
At the prompt of “Please enter the gitlab-ci description for this runner:”, give it a name:
At the prompt of “Please enter the gitlab-ci tags for this runner (comma separated):”, enter: hog

At the prompt of: “Please enter the executor: docker+machine, docker-ssh+machine, kubernetes, parallels, virtualbox, docker-ssh, shell, ssh, custom, docker:”, enter: shell

Now you can simply start the runner (gitlab-runner run). Make sure Vivado is in the path.

Useful links

GEM EDMS:!master/navigator/project?P:1424611140:1424611140:subDocs
Xilinx red box setup on linux:
Control machine setup script:
V3 electronics user guide:
GE1/1 fibers:
GE2/1 master documentation:
Backend Twiki:
V3 manual:
GE2/1 OH twiki:
CTP7 link mapping:
CTP7 DAQ formats:
CERN Chips Schedule:
CERN Chips need by dates:
GE2/1 LpGBT OH schematics:
904 time reservation (superSaas):
LpGBT Manual:
ME0 Asiago page:
ME0 scripts:
GEM trigger data format:
GEM trigger data format detector note:
DTH manual: GEM SW schedule:

-- EvaldasJuska - 2018-08-31

Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt CSC_FED_SynthA_125in_200_off_out.txt r1 manage 3.0 K 2019-08-29 - 16:19 EvaldasJuska Synchronizer config to provide 200MHz on output 0 (used for CSC FED)
Edit | Attach | Watch | Print version | History: r53 < r52 < r51 < r50 < r49 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r53 - 2021-03-24 - EvaldasJuska
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback