This is the main CSCTrackFinderExpert page.

General Information

For a thorough overview of the CSCTF, please consult Commissioning of the CSC Track Finder

Logging in to P5

All the actions described below require as first operation to log into the csc-tf machine and change from your user to csctfts user (you need to be granted the right to do so).

Here it is the list of operations:

  • Log in the .cms network: ssh, where X=0,1
  • Log in the csc-tf machine: ssh csc-tf; type this command from cmsusrX machine
  • *Change your user*": ./ (or type xsudo -H -u csctfts bash -login)
  • Move to the base directory: cd $CSCTF; $CSCTF is an environment variable set for this pourpose (dir=/nfshome0/csctfts/TriDAS/trigger/csctf)

Slot Numbering

For general knowledge and because some of the code require as argument a slot number instead of the sp number, here we report the SP to SLOT one-to-one mapping:

  • SP 1->6 are in Slots 6->11
  • CCB is in Slot 12
  • MS is in Slot 14
  • SP 7->12 are in Slots 16->21

FED Numbering

  • SP1->SP12: 890->901
  • TF DDU:760 (to be double checked)

CSR_AF WordCount Values

When monitoring the timing system of the CSCTF SP's we should read the value of the CSR_AF WC register. Logs in the folder /var/log/ on the csc-tf machine contain the most recent values of this register. It's nominal values when enabled are:

1 1 13
1 2 13
1 3 14
1 4 14
1 5 14
1 6 13
1 7 17
1 8 17
1 9 17
1 10 17
1 11 17
1 12 17
1 13 15
1 14 15
1 15 15
2 1 10
2 2 10
2 3 10
2 4 11
2 5 11
2 6 10
2 7 13
2 8 13
2 9 13
2 10 13
2 11 13
2 12 13
2 13 12
2 14 11
2 15 11
3 1 11
3 2 11
3 3 11
3 4 11
3 5 11
3 6 11
3 7 15
3 8 14
3 9 14
3 10 14
3 11 14
3 12 15
3 13 9
3 14 9
3 15 8
4 1 15
4 2 15
4 3 14
4 4 15
4 5 14
4 6 15
4 7 16
4 8 16
4 9 16
4 10 16
4 11 16
4 12 16
4 13 12
4 14 12
4 15 12
5 1 19
5 2 19
5 3 19
5 4 20
5 5 20
5 6 20
5 7 20
5 8 20
5 9 20
5 10 20
5 11 20
5 12 20
5 13 17
5 14 17
5 15 17
6 1 16
6 2 16
6 3 16
6 4 16
6 5 16
6 6 16
6 7 18
6 8 18
6 9 19
6 10 19
6 11 19
6 12 19
6 13 18
6 14 18
6 15 18
7 1 12
7 2 12
7 3 12
7 4 12
7 5 12
7 6 12
7 7 15
7 8 16
7 9 16
7 10 15
7 11 15
7 12 15
7 13 15
7 14 15
7 15 15
8 1 8
8 2 8
8 3 8
8 4 8
8 5 8
8 6 8
8 7 12
8 8 12
8 9 12
8 10 12
8 11 12
8 12 12
8 13 11
8 14 11
8 15 11
9 1 9
9 2 9
9 3 9
9 4 9
9 5 9
9 6 9
9 7 13
9 8 12
9 9 13
9 10 13
9 11 12
9 12 12
9 13 8
9 14 9
9 15 8
10 1 13
10 2 13
10 3 12
10 4 13
10 5 13
10 6 13
10 7 15
10 8 15
10 9 15
10 10 15
10 11 15
10 12 15
10 13 10
10 14 11
10 15 11
11 1 17
11 2 17
11 3 17
11 4 17
11 5 17
11 6 17
11 7 19
11 8 19
11 9 19
11 10 19
11 11 19
11 12 19
11 13 15
11 14 15
11 15 15
12 1 15
12 2 15
12 3 15
12 4 15
12 5 15
12 6 15
12 7 19
12 8 19
12 9 19
12 10 19
12 11 19
12 12 19
12 13 17
12 14 17
12 15 17

Relevant Web Pages for 904

Calendar To Book Your working Time Slot: If you wish to be added to the Calendar, please send Gregory.Rakness (at your gmail account username.

How Tos

How to tunnel to P5

On Linux/Mac:

To access the web page at P5 you need to open a tunnel to P5.

Prerequisite to visualize the web pages after having open the tunnel is to download the pac file cms_proxy.pac

Howto on Firefox:

  • go to Edit -> Preferences -> Advanced -> Connection Settings -> Select "Automatic proxy configuration URL" and give the location of your pac file, e.g. file:///home/username/cms.pac
  • On Firefox 9, Edit -> Preferences -> Advanced -> Network -> (Connection) Settings ; and do above.

Commands to open a tunnel:

  • ssh -t -L 9999:localhost:11010 ssh -ND 11010 cmsusr

On Windows:

download and run Putty.

Create New Sesstion to "" port 22


Enter "50000" in the "Source port" text field, "" in the "Destination" text field. Make sure the "Local" and "Auto" radio buttons are enabled and click the "Add" button

Save Session With Name "Tunnel 1"

Create New Sesstion to "madfish@localhost" port 50000


Enter "9999" in the "Source port" text field and enable the "Dynamic" radio button. Make sure the "Auto" radio button is enabled and click the "Add" button.

Save Session With Name "Tunnel 2"

Connect to Tunnel 1 then to Tunnel 2

Configure Firefox by:

(1) Download cms_proxy.pac to C:
(2) Open Firefox and go to Tools->Options->Advanced->Network->Settings
(3) Check radio button for Automatic Proxy Configuration URL and Enter: file:///c:/cms_proxy.pac

For more information on tunnelling, see:

How to Load Firmware

Copy file from the Lev Uvarov's page:

to the folder:

cd $CSCTF/TrigData/TrackFinderCrate/SectorProcessor2002/firmware/
wget http://link

CSCTF SPs have two firmware kind:

  • "chain 0" firmware: it is to be loaded in the Main FPGA mounted on the mezzanine card (takes ~ 15 min. per SP)
  • "chain 1" firmware: it is to be loaded in the Front FPGA (F1,F2,...,F5) (takes ~ 5 min. per SP)

The script to load the firmware is $CSCTF/SPLUTsAndFirmWare/scripts/; it is actually a bash script. Open it and change the options inside to point to the right files. There are 2 blocks: the first for "chain 1" firmware and the second one for "chain 0". If you load chain 0 fw, the chain 1 block should be all commented out (and viceversa).

Example of fw loading in SP 7 (slot 16):

  • ./ 16

After loading new firmware we have to change version of firmware at file:

emacs $CSCTF/SPBaseInteractions/firmwareVersions.txt
(for chain 0 change version of: SP SP dd/mm/yy; for chain 1 change version of: SP FA dd/mm/yy, SP DD dd/mm/yy and SP VM dd/mm/yy)

and compile the code:

cd $CSCTF && make clean && make
restart the Cell as user (not csctfts):
sudo /sbin/service triggerd restart csctf-cell 
for emulab03 computer (904) change "triggerd" -> "xdaqd" and you will get:
sudo /sbin/service xdaqd restart csctf-cell

Now you will most likely need to create a new configuration key that can be used with the new firmware.

How to Create a Configuration Key

  • A configuration key is a set of values assigned to the various registers in the SP. You should create a new configuration key for each version of the SP firmware released and/or for LUTs changes and/or any time it is decided that a new register value is more beneficial than a previous one (e.g. timing improvements, tuning parameters based on physics studies, etc).
  • Prerequisites: have the community aware of the upcoming change. Generally any change in the hardware has to be agreed in the csctf-working meeting and then propagated to the CSC and L1 Trigger community

  • Location of the scripts: The scripts to create a configuration key are located in $CSCTF/ts/scripts/newMezzanine
ssh -X cmsusr # cms904usr in 904
xsudo -H -u csctfts bash -login
cd $CSCTF/ts/scripts/newMezzanine

The files to change before writing the new key according to the change you want perform are in the same folder:

  • firmwareVersions -> if you updated the firmware version of SP, CCB or MS
  • SP$i.dat, $i=1,2,...,12 -> if you updated new LUTs
    (you should get these from the MONITOR files in $CSCTF/SPLUTsAndFirmWare/LUTs/bin/cmssw_x_x_x/monitorData/ and overwrite accordingly: ie SP7_cmssw_3_6_3_MonitorData.dat would become SP7.dat)

* Edit the "" script and rename the right static2_config.....txt file wich located at:

* STATIC_CONFIG= "...." = We should check if really the on  call expert needs to touch the below three lines??? ==

  • static2_config.txt -> if you modify any register for static configuration
  • timing2_config.txt -> if you modify any register for timing configuration
  • eta2_config.txt -> if you modify any register for eta cuts configuration

The main script is As parameter you need to pass a name for the key. The CSCTF convention is to use the day when we perform the upgrade, e.g., 310309 == 31st March 2009 => ./ 310309 (please note that 0 in front are not registered... so 010714 will be 10714)

This script simply calls other scripts to perform the key writing on the database.

When you created new Key you have to:

  • upload new information about keys at OnlineWBTrigger: go Here to update OnlineWBL1TriggerKeys...
  • upload new information at CSC and Trigger elog
  • call the L1Doc (16-1958). If the phone # changes, the phone is available at Main Trigger Page -> Contacts
  • call Karoly Banicz of Jinghua Liu to change comics key at the CSC RCMS, see for getting their contact phone
  • modify cosmics key at $CSCTF/ts/worker/src/common/, compile at $CSCTF and restart csctf-cell:
ssh -X cmsusr # cms904usr in 904
ssh -X l1ts-csctf
xsudo -H -u csctfts bash -login
# $CSCTF/ts/worker/src/common/
cd $CSCTF && make
sudo /sbin/service triggerd restart csctf-cell 

N.B.: You can check the key/register values when you are finished with the L1CE. Moreover this tool allows one to compare different configuration keys

How To Create LUTs

cvs co L1Trigger/CSCTrackFinder  # (or  git cms-addpkg L1Trigger/CSCTrackFinder )
scram b

If you want create a new PTLUT file, please uncomment the following lines in L1Trigger/CSCTrackFinder/test/BuildFile.xml:

<library file="analysis/" name="CSCMakePTLUT"!>
     <flags EDM_PLUGIN="1"/>

If you want create SRLUT files, please uncomment:

<library file="analysis/" name="CSCMakeSRLUT">
     <flags EDM_PLUGIN="1"/>

If you want create both PTLUTs an SRLUTs then simply uncomment both of them :-).

At this point you need to compile again:

scram b

After that you are ready to run the script which creates LUTs.
Please make sure you set a globaltag which is valid for the CMSSW release you use, otherwise you will get an error.
Remember to comment out one the paths in case you want to generate only a specific set of LUTs (SR or PT).

cd L1Trigger/CSCTrackFinder/test

The job will take a while (especially if you are creating the SRLUTs) and dump a lot of warnings. Generally they are harmless, so if you are using bash you would prefer a command like:

cmsRun >& logfile < /dev/null &

The expect size of the files (in total 146 *.bin + 1 *.dat) is:

  • 1048576 bytes for GlobalEtaM*End*Sec*LUT (5 files per EndXSecY(12) = 60) or GlobalPhiM*End*Sec*LUT (7 files per EndXSecY(12) = 84) and LocalPhiLUT.bin (1 file)
  • 4194304 bytes for L1CSCPtLUT.bin (1 file)
  • 12582912 bytes for L1CSCPtLUT.dat (1 file)

The default script will generate PTLUTs in both .bin and .dat (simple text) format. The .bin file is needed to upload the LUTs into the hardware. The text format is needed for writing them into the database.

Important: inside the you could change the PtMethod for PTLUTs, which is defined here:

PtMethod = cms.untracked.uint32(29)

The full description of what a specific PtMethod does is readable into L1Trigger/CSCTrackFinder/src/

How To Load LUTs

N.B.: In this example I will be using LUTs which are in *.bin version.

Get the LUTs (which in general someone, could be also you :P) and save the into the proper folder.


Next, check that only appropriate LUTs have changed by running = $CSCTF/misc/ oldLUTsDir/ newLUTsDir/=. Only changed files should not match. Now you have to recompile the code of the binary which upload the LUTs.

Edit: $CSCTF/SPLUTsAndFirmWare/test/crateLoadLUTs.cpp.

The change is just to tell the code where to pick the LUTs: the line to modify is

std::string vers = "your_folder"; 
(in this case the code will look in SPLUTsAndFirmWare/LUTs/bin/your_folder).

IMPORTANT: create a folder called monitorData in SPLUTsAndFirmWare/LUTs/bin/your_folder.


cd $CSCTF/SPLUTsAndFirmWare && make && cd test && make bin 

Launch the executable:

cd $CSCTF/SPLUTsAndFirmWare/bin

The executable will dump some *.dat files which will be used in the configuration process to check the LUTs are properly loaded in the SPs. These files will be indeed have to be put in the database when you create a configuration key. IF THE DAT FILE FOR A SPECIFIC KEY IN THE DBS WON'T MATCH THE ONE DUMPED FROM THE SCRIPT THE CONFIGURATION OF THE SPs WILL FAIL!

cp $CSCTF/SPLUTsAndFirmWare/LUTs/bin/your_folder/monitorData/*.dat $CSCTF/ts/scripts/newMezzanine/

and rename them appropriately (SP1.dat SP2.dat, SP3.dat ... etc)

Test the LUTs are correctly loaded modify crateRunCRC before using:

 emacs $CSCTF/SPLUTsAndFirmWare//test/crateRunCRC.cpp 
the line to modify is
std::string vers = "your_folder"; 


cd $CSCTF/SPLUTsAndFirmWare && make && cd test && make bin 

and test it:

cd $CSCTF/SPLUTsAndFirmWare/bin

Change information about new LUTs in configuration key:

emacs $CSCTF/ts/scripts/newMezzanine/

Line to modify:


Inform Gian Piero ( about "new_version" of PTLUTs and provide him dat-format of PTLUTs or ask expert (Anna: to do it. Gian Piero will deliver it to O2O.

Prepare a new configuration key...

When configuring the first time perform heavy check on LUTs by checking the Force_Configure check box.

How To Write the pT LUT into DBS

  • If a new pT LUT is loaded into the hardware and the configuration key updated, one needs to update the table CSCTF_PTLUTS, writing the entire LUT as a CLOB
  • Prerequisites: you must have created the file L1CSCPtLUT.dat
  • The scripts needed are in $CSCTF/ts/scripts/writePtLuts/

ssh -X cmsusr #( or cms904usr)
xsudo -H -u csctfts bash -login
cd $CSCTF/ts/scripts/writePtLuts/
mkdir PtLUTfiles chuncks # if they do not exist already
cp yourpath/L1CSCPtLUT.dat  PtLUTfiles/PtLUT_versionXX.dat # where XX is the version, say 33 PtLUTfiles/PtLUT_versionXX.dat # It will break the ptLUT in 5992 pieces to be written in the dbs separately as a CLOB afterwards XX #the output will create a file called qwe.sql, which is a sequence of sql command to write the CLOB into the DBS

sqlplus cms_csc_tf/xxxxxx@CMS_OMDS_LB @qwe.sql # where xxxxxx is the passwd :-)

  • Basic instructions are also in the file $CSCTF/ts/scripts/writePtLuts/README

How to check the CSCTF configurations stored in a database

In case you want to check the CSCTF configurations stored in the database and used for a certain CMSSW release and a certain global tag, start checking out the CMSSW release you are interested in - as example I take 7_3_1_patch2. Then check out the relevant packages

cmsrel CMSSW_7_3_1_patch2
cd CMSSW_7_3_1_patch2/src
git-cms-addpkg L1Trigger/Configuration
git-cms-addpkg L1TriggerConfig/CSCTFConfigProducers
scramv1 b -j 8

The relevant file are


where you set the global tag and the run number or the file you want to run on - and you better set to True only the print of CSCTF if you don't want a huge print out.

In case you want to dump also the pt LUT you have to modify


Where you want to include

#include "CondFormats/L1TObjects/interface/L1MuCSCPtLut.h"
#include "CondFormats/DataRecord/interface/L1MuCSCPtLutRcd.h"

and add the following lines

edm::ESHandle< L1MuCSCPtLut > ptLUT;
evSetup.get< L1MuCSCPtLutRcd >().get(ptLUT);
L1MuCSCPtLut myConfigPt_ = *ptLUT.product();

for(unsigned int i =0; i < (1<<21); i++){ std::cout << myConfigPt_.lut()[i] << std::endl; }

and then recompile!

Running the same cfg (L1Trigger/Configuration/test/ will print out now the 2M lines of the LUT - so you better redirect the output to a file.

How To Check the CAEN VME Bridge Firmware Version

  • The tool was made available by Karoly after the pixel incident reported at
  • Important note from Karoly: "This program should only be run if no other program accesses the VME bridge. Otherwise data corruption or program crashes could occur." This means the TriggerSupervisor should be stopped before reading the firmware and started after the operation is done.

  • The instructions are:
ssh cmsusr
ssh csc-tf
sudo /sbin/service triggerd stop
cd  /nfshome0/csctfts/TriDAS/trigger/csctf/caen-firmware-version
sudo /sbin/service triggerd start

How to add a register to monitor to OSW

* Create a file $CSCTF/ts/src/worker/ (header file in /interface)
* Structure it according to examples in the same directory, ie
* Edit
* Edit $CSCTF/../setup/csctf (/flash and pulser)
* You can check database push counts at
* Add Logging to a file if you wish (see CSR_TTCRQ_LLCK example)


How to Check How Many file descriptors are open by the csctf-cell

From vmepcS2G18-10 connected with your username (do not switch to csctfts as you need sudo right) type

  • ps aux | grep "csctf-cell\.profile" | awk '{print "sudo ls /proc/"$2"/fd | wc -l"}'

If more than one line appears it means there is more than one csctf-cell process. This is bad. Investigate the reason or, if you are in panic mode, kill all the process and restart the cell via sudo /sbin/service triggerd restart csctf-cell.

If one line appears, everything is good. Paste the output command in your shell. Press enter. Check the results.

NOTE: file descriptors should not grow much. If they do most likely there are files open and never closed. Find them out and fix the bug(s). The upper limit is 1024 fd per process. If the quota is exceeded this will cause the csctf-cell to hang and you will get paged...

How to change Sector Processor (SP)

  • Take a new SP at 904: switch of power and only after it extract SP from crate; don't forget to switch on power after extraction

  • Bring SP to Point 5:
    • swith of power
    • disconnect all cables from broken SP (maybe you need to disconnect extra cables from other SPs to extract broken SP). Before disconnecting cables check correct hardware mapping: mapping.
    • extract broken SP and put a new one. Connect cables with mapping.
    • during switch of power we could loose our firmware. Check firmware with EEPROM CORRUPTIONS. If you have errors download firmware to problematic SPs.
    • check that all links (cables are connected propely) with ....
    • check LUTs with ...
    • restart CELL
    • reconfigure CSCTF
    • start local run with proper key and check that everything works fine.
    • try to stop and start new local run. You need it just to check that stop/start procedure works fine.

How to check CSCTF Logs

AT 904:

Configuration log:

tail -f $CSCTF/localConditions/localConditions.log

Out of Sync Errors and General Hardware Errors log:


AT P5:

Out of Sync Errors and General Hardware Errors log:

How to Take a Local Run at P5

Follow the instructions here:

This is the link to RCMS: http://cmsrc-csc.cms:12000/rcms/gui/servlet/RunGroupChooserServlet

(username: cscpro)

How to take data at 904

See Links:

RCMS at B904: http://emuslice02:12000/rcms/gui/servlet/RunningConfigurationServlet

To take data (VME Dump) you have to run the following program with SP Slot = 16

cd $CSCTF/SPFunctions/bin && ./takeVMEData 16

you will get DAQ_data_out.dat with data to analyze.

At 904 we have 4 SP: SP2, SP4, SP7 and SP9.

if you notice that there are no LCTs in the data, you should change the Pipeline FIFO Delay register such that the VME is in sync. You can play with the values, but 0x43 is the default for Point 5 and 0x40 currently works nicely at 904:

cd $CSCTF/SPStandAlones/bin && ./readwriteRegister -s16 -cSP -mMA -rCSR_PFD -pW -v0x40 cd $CSCTF/SPStandAlones/bin && ./readwriteRegister -s16 -cFA -mMA -rCSR_PFD -pW -v0x40

You can convert the text dump into the raw format with;cvsroot=tridas

You can then read the raw files and unpack it in CMSSW using for example:

  1. ) checkout from cvs EventFilter/CSCTFRawToDigi & IORawData/CSCCommissioning

  1. ) change RUI00 source following file:

EventFilter/CSCTFRawToDigi/test/ to path to the newly converted raw file.

  1. ) Run

How to Read/Write register values

To look at the register configuration
cd $CSCTF/SPStandAlones/bin && ./readwriteRegister -s18 -cDD -mMA -rCSR_DFC
you will get something like this:
Read from DD/MA/CSR_DFC : 0xfff

How to Make/Test RPM at P5 (for TriDAS/trigger/setup/l1ts)

ssh cmsusr1

ssh l1ts-csctf

sudo rpm -ev trigger-l1ts

#now change to csctfts user

cd $BUILD_HOME/trigger/setup/l1ts

make rpm

#now change to your username

su username

cd rpm

sudo rpm -iv trigger-l1ts-xyz.rpm

sudo /sbin/service triggerd restart csctf-cell

How to Update CMSSW Files to deploy at P5

see here for latest procedure:

to see a list of files that have changed since the last commit do: cvs -nq up
when you are ready to commit: cvs commit -m "List Updates Here" PathToFileNameOrDirectory
to see the list of tags: cvs status -v FileName | grep V
when you are ready to tag your new release (so it is used in the next CMSSW Cycle do:

cvs tag Vxx-yy-zz

Compared with the previous tag you should increase, the zz number for patches, the yy number for a feature addition, the xx number for a (backward incompatible) change in interface. For small changes just add 1 to whatever zz is.

to delete tag: cvs -d tagname

to test the tag: cvs co -r Vxx-yy-zz MySubsystem/MyPackage

When you are ready and happy with the new tag, do the following:

  • Go to the CmsTC
  • Go to Package Publication that is on the left menu bar , and fill the line with MySubsystem/MyPackage Vxx-yy-zz. Tick the radio button corresponding to the releases for which you want to submit the new tag.

let Vasile know.
Put Ivan and Darin in cc as well.

If it is urgent, after doing above:

- the DPG/POG coordinator or reco contact (Vasile.Mihai.Ghete)
- the L2 reconstruction coordinators (Jean-Roch.Vlimant, and David.Lange)
- the release manager (me for 3.11.x, andrea.bocci, Shahzad.Malik.Muzaffar and Adeel.Adeel-Ur-Rehman for 4.2.x)

How to turn off/on local DAQ at P5

Make sure that DAQ_Write_Y_Build_N shows up at RCMS http://cmsrc-csc.cms:12000/rcms/gui/servlet/RunGroupChooserServlet
Global DAQ should be configured and ready to start run (have the DAQ Shifter call you when they are ready to start the run):

Turn ON local DAQ writing:

Check that file are written at local DAQ data pc: csc-daq10 in the folder /data. You will see a new RUI00 file if the local DAQ is enabled. Use "ls -ltr" to see last file created!

check that writeBadEventsOnly = false

Be careful data could be written only with rate not more then 2 kHz. In case if L1_Physics rate is greater than 2 kHz then it will be prescales at local DAQ.

To turn off local DAQ full file logging :

How to Run Valgrind To Check For Memory Leaks

To get information about the cell profile:

ps aux | grep csctf-cell

at 904:

sudo /sbin/service xdaqd stop csctf-cell
script output
valgrind --tool=memcheck --leak-check=yes --show-reachable=yes /opt/xdaq/bin/xdaq.exe -z $XDAQ_ZONE -e /opt/xdaq/share/csctf/profile/csctf-cell.profile -c /opt/xdaq/share/csctf/profile/csctf.configure -p 2974 -h

the output will be saved in a file called output. Look through it and find the leaks that are definitely/indirectly lost, as valgrind will give you the line numbers of the offending memory leaks.

How To Test Raw Data Format

To set up the raw analyzer (ie in CMSSW_4_2_1):

cmsrel CMSSW_4_2_1
cd CMSSW_4_2_1
cd src
cvs co EventFilter/CSCTFRawToDigi
cvs co IORawData/CSCCommissioning
cvs co L1Trigger/CSCTrackFinder
scram build -c
scramv1 b

now overwrite src/EventFilter/CSCTFRawToDigi/plugins/ with the one from:

(CSCTFAnalyzer.h goes to src/EventFilter/CSCTFRawToDigi/interface)
and from the same link copy,,, and to src/EventFiler/CSCTFRawToDigi/test 

edit to the raw data file from csc-daq10 /data folder.  

To make a tunnel: ssh -L 2999:csc-daq10.cms:22 user@cmsusr
and then use scp -P 2999 localhost:/data/file .

How To Run The SPTester Tool

First, for more information about how these programs were written please consult:
Control Software for the CSC Track Finder

The SPTester tool is a validation programs that tests an SP's functionality by filling its registers with random input values and then reading the SP's output registers for proper content.

To compile it:

cd SPValidation
cd test
make bin

the source code is: $CSCTF/SPValidation/test/SPtester_new.cpp
and executed in $CSCTF/SPValidaion/bin using the command ./SPtester_new -u -s16 -c20 -gc
the u uploads the pt LUTs specified in the source code, the s16 specifies slot 16 and the gc turns off singles, c20 runs the program for 20 cycles
to copy output to log file:

./SPtester_new -u -s16 -c20 -gc > log.txt 2>&1
(./SPtester_new -u -s16 -c20 -gc | grep Mode..Mismatch)> log.txt 2>&1

if you need to update the core, download the proper vpp package from

Extract the headers to: $CSCTF/SPValidation/include/
Extract the cpp files to: $CSCTF/SPValidation/src/common/
Extract the .dat files to: $CSCTF/SPValidation/bin/

For simpler wrapper-oriented tests use TestSPconsole. Build the same as above, but use ./TestSPconsole -s in bin.

How to check crate trigger links

to check the correct links (in terms of ordering) from FPGA to SP you can run the binary $CSCTF/SPFunctions/bin/readCrate_triggerLinkIDs

How to update TS at 904

(1) Edit the version number of TS in the repo:
sudo vim /etc/yum.repos.d/ts.repo
(2) Remove the old software:
sudo yum groupremove "triggersupervisor"
(3) Install the new copy:
sudo yum groupinstall "triggersupervisor"
(4) Verify the new installation:
rpm -qa | grep trigger
(5) Erase the old rpm:
ie: sudo rpm -e trigger-csctf-1.10.0-1.cmsos10.slc4
(6) Make changes necessary in the TS release notes (usually provided by Marc/TS people):
ie: for TS 10->11 we have to remove functions "ending()" in and make changes to for CellOperationFactory
(7) Edit version number in the Makefile:
vim $CSCTF/../setup/csctf
(8) Make the rpm:
make rpm
(9) Remove the old RPM (in this case version 10):
rm rpm/trigger-csctf-1.10.0-1.cmsos10.slc4.i386.rpm
(10) Check to see if there are any other packages that need an update:
sudo yum update
(11) Install new rpm:
(12) make new code and restart cell:
make clean
sudo /sbin/service xdaqd restart

How to run onlineDQM plots locally

set up:
setenv SCRAM_ARCH slc6_amd64_gcc481
scramv1 project CMSSW CMSSW_7_2_1
cd CMSSW_7_2_1/src
git clone DQM/L1TMonitor

run DQM plot for a single run:

edit python/
change sampleIdentifier and related config:
elif sampleIdentifier == '226144' :
    runNumber = 'XXXXXX'
    dataset = 'None'
    dataType = 'RAW'
    useDAS = False
    readFiles.extend( [
go to test/Makefile, change the variable to the correct run number
make;         // will create a root file
make plot;  // will run a macro: to give out all plots 

General CSCTF Troubleshooting

Out of Sync in Link Masking Panel

At 904, if the CSCTF shows up "Out of Sync" in the link masking panel, and you notice that the FED lights are not on, or the Muon1, Muon2, Muon3 lights are not blinking on the MPC, you need to do some extra steps to configure the system.

Step 1: Destroy all configurations in the 904 RCMS. If the most of the FED lights are on skip to Step 3. If they are off: Add the configurations Commissioning/startFED and proceed to Step 2.

Step 2: Initialize the startFED and configure. Once it is configured you should check the FED crate and see that its many lights turn on. You should now destroy the startFED configuration process.

Step 3: add Comissioning/startPCrate and Commissioning/startTTC in the 904 RCMS configuration manager. Initialize them both!

Step 4: Go to the Yellow Page and click the button Write FLASH All Crates. This will not be available unless the processes were initialized in the previous step.

Step 5: Go to the TTCi Page and click "Configure" Go to the LTC Page and click "Configure" then "Halt" then "HardReset" then "Resync" and "Enable."

Step 6: Check to see that the MPC, MS, SP and TMB LEDs blink synchronously with high voltage and gas on (if not, go back and check the previous steps/power cycle everything and try again). If they do, you may destroy all the processes you have created in this procedure and everything should configure and work as normal.


  • SYMPTOMS: so far they occurred or identified only after the crate is restarted from being off (e.g., cooling issue, shutdown, etc). The synptoms is the board not getting configured. Issue the configuration command from the TS gui and monitor the file localConditions/localConditions.log. Whenever the CSCTF does not have a board configured, it is the first check performed.

  • DIAGNOSIS: from $CSCTF run the script misc/ with argument the slot in the crate with the board you want to investigate, e.g. misc/ 6 (for SP1).

You will get an output like:

Read from SP/MA/CSR_CID : 0x99f6

Read from VM/MA/CSR_CID : 0x9515

Read from DD/MA/CSR_CID : 0x95d5

Read from F1/MA/CSR_CID : 0x*****hardware access exception, FPGAobject::Read::Could not read from CAEN BusAdapter
unknown exception, not hardware access.. check addr path etc..
Read from F2/MA/CSR_CID : 0x9555

Read from F3/MA/CSR_CID : 0x9575

Read from F4/MA/CSR_CID : 0x9595

Read from F5/MA/CSR_CID : 0x95b5

At that point you know that the problematic chip is SP1/F1

CSCSummary Does NOT Update CSCTF Rates

Where to web-monitor the CSCSummary:

If everything works fine you can access the CSCSummary web page above and observe rate of the CSCTF halo, stub and tracks if CSC is running.

How to monitor the CSCSummary status on csc-tf machine:

  • Log into csc-tf machine without switching to csctfts user:
    • > whoami

  • then type:
    • > sudo /sbin/service triggerd status

The output will be

Status of XDAQ service 'csctf-tstore':                     [  OK  ]
Status of XDAQ service 'csctf-ws-eventing':                [  OK  ]
Status of XDAQ service 'csctf-las':                        [  OK  ]
Status of XDAQ service 'csctf-pulser':                     [  OK  ]
Status of XDAQ service 'csctf-sensor':                     [  OK  ]
Status of XDAQ service 'csctf-store':                      [  OK  ]
Status of XDAQ service 'csctf-cell':                       [  OK  ]
Status of XDAQ service 'xdaq-jc':                          [  OK  ]
Status of XDAQ service 'csctf-spotlight':                  [  OK  ]                           

If the CSCSummary crashed your output will report:

Status of XDAQ service 'csctf-store':                       [Failed]

How to restart the DBS service for CSCSummary:

  • > source /nfshome0/csctfts/TriDAS/trigger/csctf/misc/restartDBS
(this command could be executed during run without stopping it)

restartDBS is:

echo "stopping store and tstore"
sudo /sbin/service triggerd stop csctf-store
sudo /sbin/service triggerd stop csctf-tstore

"waiting for 30 secs: be patient..."

echo "restart store and tstore"
sudo /sbin/service triggerd start csctf-tstore
sudo /sbin/service triggerd start csctf-store

xdaq-flashlist variables don't appeared in CSCTF CELL

Restart (stop all 1st and start them after) the following processes with sudo /sbin/service triggerd:


It is necessary to do if even they look ok. It could happen after csc-tf computer was off. Possibly problem in csctf-pulser and csctf-sensor processes which are sent information how often xdaq-flashlist variables should be updated.

Trigger Supervisor Troubleshooting

Basic Checks

Where to web-monitor the csctf-cell:

If everything works fine you can access the web pages above and surf into it, plus see a green light in the main trigger web page.

How to monitor the csctf-cell status on csc-tf machine:

  • Log into csc-tf machine without switching to csctfts user:
    • > whoami

  • then type:
    • > sudo /sbin/service triggerd status

The output will be

Status of XDAQ service 'csctf-tstore':                     [  OK  ]
Status of XDAQ service 'csctf-ws-eventing':                [  OK  ]
Status of XDAQ service 'csctf-las':                        [  OK  ]
Status of XDAQ service 'csctf-pulser':                     [  OK  ]
Status of XDAQ service 'csctf-sensor':                     [  OK  ]
Status of XDAQ service 'xdaq-jc':                          [  OK  ]
Status of XDAQ service 'csctf-cell':                       [  OK  ]
Status of XDAQ service 'csctf-spotlight':                  [  OK  ]

If the cell crashed your output will report:

Status of XDAQ service 'csctf-cell':                       [Failed]

How to restart the csctf-cell:

  • > sudo /sbin/service triggerd restart csctf-cell

Debugging the problem:

The reasons of a cell crash can be several. In general, it is useful to debug further using gdb, although it could have happened occasionally:

  • Identify the PID in the core looking in /tmp/core.PID
  • run gdb:
    • sudo gdb /opt/xdaq/bin/xdaq.exe /tmp/core.PID
    • thread apply all bt full

P.S.: even better is to run this command from emacs after launching gdb with M-x gdb; you will have the output saved in a file without copying and been able to use the emacs tool to surf the output

Links on how to use gdb are:

How to Dump The GDB Output Into A File

sudo gdb /opt/xdaq/bin/xdaq.exe /tmp/core.10484 -x gdbCommands -batch >> outputFile

gdbCommands file contains the list of commands we want gdb to execute. In our case:

  • set height 0 [This command removes the gdb command ---Type to continue, or q to quit---]
  • thread apply all backtrace full

NB: If something interesting or not understood is found from the debug, please post it as savannah bug since you need to tell the responsible for the Trigger Supervisor (TS) framework (Marc Magrans De Abril)

How to Attach GDB To A Running Executable

There are two ways:

  • using the following script
    • ssh -X
    • ssh -X csc-tf (do not switch to csctfts user as you need "sudo" permission to run gdb)
    • cd /nfshome0/csctfts/TriDAS/trigger/csctf/misc/gdbDumps
    • -v csctf-cell > file_yearmonthday

  • Manually (example with the csctf-cell running on the l1ts-csctf machine and listening to port 2974):
    • ssh l1ts-csctf
    • pgrep -f "xdaq.*2974" (e.g., you get =5283)
    • sudo gdb
    • handle SIG33 nostop noprint noignore pass
    • attach (in our example one should type "attach 5283")
    • thread apply all bt full
    • detach

A Service Keeps FAILING After Restart Command

  • This problem is due to triggerd daemon. The latter does look in /var/run/ for the pid file corresponding to the service to kill. If it does not find it, it cannot kill so when restarting the port will be busy and the service does FAIL.

  • How To Fix It (e.g. csctf-store keep FAILING):
    • stop the service: sudo /sbin/service triggerd stop csctf-store
    • check if the service is still up and running: ps aux | grep csctf-store. If an instance of it exist, this is the cause of your problem: KILL IT MANUALLY, "sudo kill jobid"
    • start the service again: sudo /sbin/service triggerd start csctf-store

How to read data format

cp ~gilmore/jason/connectionsDB.dat ./
~gilmore/jason/eop2008 /data/file.raw |less

in less finde * = /*\*\* * or ** is worning

more then 2 * is error

Live DMB Flags = 7e3fh.

73ef = 0111 1110 0011 1111 -> 12 links are one like 12 SP

If you fine error in program the you could look more detailed in file with

hexdump /data/file.raw

DDU data formart you could find here:

Other Information

How to Change Password for cms_csc_tf_mon user

  • Ask Marc (or whoever has rights) to modify the password stored in /nfshome0/centraltspro/secure/csctf-store.profile
  • Change the password on the database (from cmsusr):
    • sqlplus cms_csc_tf_mon/old_password@CMS_OMDS_LB
    • alter user cms_csc_tf_mon identified by new_password replace old_password;
  • restart the csctf-store from the process controller (http://l1page.cms/main/ProcessController)

Useful DBS Queries


  • Display many info for the latest 100 run numbers

  • Select Run Numbers with a specific TS key
where (a.TSCKEY = b.TS_KEY)
order by a.RUNNUMBER desc;


where (a.TSCKEY = b.TS_KEY AND RUNNUMBER=132440)
order by a.RUNNUMBER desc;

  • List of runs, CSCTF configuration keys and firmware
select a.RUNNUMBER, b.CSCTF_KEY, c.FIRMWARE from cms_wbm.RUNSUMMARY a, CMS_TRG_L1_CONF.TRIGGERSUP_CONF b, CMS_CSC_TF.CSCTF_SP_CONF c where (a.TSCKEY = b.TS_KEY and a.RUNNUMBER>132440 and c.SP_KEY=b.CSCTF_KEY || '0001') order by a.RUNNUMBER desc

Extra help

To find 'something' in files in all directories:

find . -type f -exec grep -H -n 'f/w verification failed' {} \;

Check status of Slot:

SPValidation/bin/TestSPconsole -s16

New 904 With SLC5X

904 With SLC5 Basics:

To Log into 904 from CERN network (lxplus):

ssh -Y <usrname>@cms904usr

The CSCTF Machine is named vmepc-e1x07-27-01, with alias l1ts-csctf

The URL for the cell is http://vmepc-e1x07-27-01.cms904:2974

You will need a tunnel to 904 to access the cell via web: ssh -t -L 9999:localhost:1081 ssh -ND 1081 cms904usr

  • p5and904.pac: This .pac file allows connections to both P5 and 904 private networks (may require lots of reloading!)

How to Take a Local Run in 904 With SLC5X

Disclaimer: hopefully these instructions will change and made consistent with P5 setup, but as of today (29/02/2012) the 904 SLC5X migration is still a work in progress and the situation is very fluid right now.

  • TF Cell is running inside the .cms904 private network, and the Supervisor (and the rest of the apps) outside. We need to do some port forwarding for the two to talk to each other:

  • RCMS interface is not available in the usual format. To run the supervisor please use the link at From there you have all the buttons you need to configure/start/stop/reset the system.

Responsibility of pager expert

CSCTF On Call Experts Shifts
CERN car                              
Anna Kropivnitskaya                              
Antony Adair                              
G.P. Di Giovanni                              
Justin Hugon                              
Laria Redjiimi                              
Matt Fisher                              

  • Organize CSCTF working meeting, which is running every Tuesday at 18:00 (in most cases by EVO only):
    • Create indico page at MUON - Muon Detector. Just select the last meeting in Indigo and create clone. Make sure the meeting is accessible to PUBLIC, and managers include the group cms-web-access (the same group allowed to see CMS Physics Indico pages).
    • Create EVO with title CSCTF meeting and password which is sent in e-mail.
    • (When it is requested by the group) Make booking for a room at Room reservation. You have to find a room with "Video conference" and reserve it. It is better to do till Friday to have time to find free room for the meeting. Some days we have booked room 40-R-B10. Availability of the room you could check here: booked days.
    • Send remind e-mail about meeting to the CSCTF e-mail list: It is better to sent e-mail in 12-24 ours before starting meeting.
    • Prepare pager report for the last week.
    • Run meeting.
    • It is recommended to all CSCTF experts (Antony, Justin, Laria and Matt) attend each CSCTF meeting if possible.

  • Other meetings:
    • Participation in the daily (even during weekends) CSC phone meeting at 18:00.
      • you have to be subscribe to Every day it is announced phone meeting, sometimes it cancels.
      • phone meeting could be reachable at EVO -> CMS -> CMS Muon Operations.
    • CSC Operations+DPG Meeting Wednesday at 18:00: prepare/present a 10 minute CSCTF status talk for it.

  • Service work during on-call time:
    • Expert CSCTF Phone should be rotated every Tuesday after lunch time at CERN. New expert should be available and pick up his/her phone. If you want to change phone at another time, day or place, please agree with another expert and do it if only he/she agrees (especial if there is holidays).
    • Each morning check CSCTF plots at online DQM for collision runs: dqm Work space -> Trigger/Lumi -> L1T -> Top -> L1T -> L1CSCTF
    • CSCTF expert on-call has to have car and be ready to come at Point 5 and any time if necessary during shifts.
    • Change SP board at Point 5 if it broken. 2 people has to be underground, so talk before to come at Point 5 with shift leader if he could provide you person, who has access to USC. If he couldn't then contact with another CSCTF expert which are listed at table above and ask for help. If other CSCTF expert are not available ask help from CSC-DOC or CSC experts (Misha...). If you don't fill safe to do this changes without another CSCTF expert then don't hesitate to contact he/she even during night.

    • Upload and tested firmware at 904 or Point 5 if it is requested by CSCTF group. Take local run.
    • Create CSCTF key if it is requested.
    • If it is urgent situation and you don't know what to do please don't hesitate to contact another CSCTF expert even during night.

  • Other notes:
    • If you want to go to vacation then you have to agree with another expert on-call to change dates and announce it on the CSCTF meeting.

If you have any question, proposal or need help in training, please contact to Anna Kropivnitskaya.


CSCTF group has a CERN car for CSCTF issues in period 12 August 2010 - 23 December 2010. The main contact person is Anna Kropivnitskaya. If you want to use it, please contact Anna to clarify if a car is available and to prevent any over crossing. You could find the availability of the CSCTF car at Google Calendar for CSCTF CERN car

Please note: to use CSCTF CERN car your name has to be in car documents.

You are allowed to drive inside the CERN perimeter. If you need to go out the default perimeter limitations you need a "Job Order" (via EDH).

How to access to CSCTF WBM

First of all you need to get wbm account you can ask:

  • Juan Lopaz

After that you need to access as the following:

  • ssh cmswbmdev3
  • cd wbm
  • source scripts/tomcat_cmswbm
  • make

In order to make any changes for csctf wbm you will find the code at: * wbm/cmsdb/ * wbm/htdocs/applet/

In order to install the new changes on the cmswbmdev server, the user must also be in the wbmdevsudoers user group, and type:

  • sudo make install

Saving new counters in WBM

  • ssh cmsusr
  • ssh cmswbmdev
  • export TNS_ADMIN=/etc
  • sqlplus CMS_CSC_TF_MON@cms_omds_lb
  • pswd ask "Gian Piero"

  • to create a table you need to have three things: "value", "time stamp" and "sub-channel"


CSCTF To-Do List

* Fix DT Timing Plots: Label says DT-CSC, but code says CSC-DT

Relevant Websites

List of web pages:

CSCTF emulator:

-- GianPieroDiGiovanni - 08-Mar-2010 /s

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatpac cms_proxy.pac r1 manage 1.0 K 2010-07-22 - 15:12 AnnaKropivnitskaya  
Compressed Zip archivezip r1 manage 1108.5 K 2012-09-01 - 18:31 JustinHugon CSC Trigger Data-Flow Diagram and CSCTF Trigger Data-Flow Diagram
Unknown file formatext gdbCommands r1 manage 0.1 K 2009-08-10 - 18:22 GianPieroDiGiovanni gdb commands
XMLxml hotspot.xml r1 manage 0.1 K 2010-05-19 - 13:26 UnknownUser  
Unknown file formatpac p5and904.pac r1 manage 0.6 K 2012-03-01 - 16:13 JustinHugon .pac file for connecting to both P5 and 904 private networks
Edit | Attach | Watch | Print version | History: r136 < r135 < r134 < r133 < r132 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r136 - 2015-03-02 - PierluigiBortignon
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback