Cosmic Rack Tutorial

General information on operating the Crack:

RCMS: http://cmstracker035.cern.ch:8080/rcms/gui/servlet/RunningConfigurationServlet

Current working directory for creating configurations:

[cmstracker035] /home/xdaqtk/autotest/Commissioning2017/

Location of Cosmic Rack commissioning runs:

/CRACKdev/Commissioning2017/ 

Example to create a pedestal configuration:

tkconf13mc-yy 06-CrackPedestal  DevConfiguration_slc5.xml CRACKPartc1.xml  Local  Pedestal  Commissioning2015 Crack_Team

Suggested reading material

• Commissioning the CMS Silicon Strip Tracker prior to Operations with Cosmic Ray Muons
  • http://cms.cern.ch/iCMS/jsp/openfile.jsp?type=NOTE&year=2009&files=NOTE2009_021.pdf
• Commissioning and Calibrating the CMS Silicon Strip Tracker
  • https://workspace.imperial.ac.uk/highenergyphysics/Public/preprints/06-4.pdf
• Alignment of the Cosmic Rack with Hits and Impact Point Algorithm
  • http://lib.tkk.fi/Diss/2007/isbn9789521037115/article4.pdf

Commissioning the Cosmic Rack from scratch

These are the general instructions to commission a new Cosmic Rack partition. You will start with a newly created and not commissioned Cosmic Rack partition, which has been prepared for you. The partition name is CR_DD-MM-YYYY_1. It is your job to commission it! Follow each step closely, and ask the tutors if you have any questions.

Step 0: Open the Run Control & Monitor System (RCMS)

• The commissioning runs are found on the RCMS server. CTRL+Click to open in a new tab: http://cmstracker035:8080/rcms
• Login using the username CRACKdev (ask the tutors for the password)
• Click on 'Configuration Chooser' in the left pane and then click on the directory 'Crack2014slc6mc' in the main frame
• Here you will see a sequential list of commissioning runs:
  • 01-CrackCratescan
  • 02-CrackConnection
  • 03-CrackTiming
  • 04-CrackGainScan
  • 05-CrackVpspScan
  • 06-CrackPedestal
• You will to perform the full commissioning procedure using these runs, starting with the crate scan.
• Document everything in the TIF-DAQ elog: http://tacweb.cern.ch:8080/elog/TIF-DAQ/

Step 1: Cratescan

• The first step in the commissioning procedure is to perform the crate scan. This should only be performed once per partition!
• Go to the RCMS, and go to directory 'Crack_Tutorial'
• Click on '01-CrackCratescan'.
• Click 'Create' to create the CrateScan run
• Click 'Initialize' to initialize the run.
• The run should reach 'Halted' state.

• Now set the FEC Parameters
• Click on 'Status Display'
• Go to the FEC Crate Controller in the dropdown to the FEC State Machine.
• Click on Crate Parameters
• Set the Scan Slot Range from 4 to 4.
• Click 'Apply'
• Go to 'state machine' on the crate controller (not rcms) and click on 'Configure'.
• Check the Hardware status.
• There should be 1 FEC in slot 4.
• Check the status of the rings at the bottom - they should be okay
• If everything looks okay the FEC is Configured.
• Now configure the FEDs.
• In Status Display open the FED Crate Control (this time scan over all slots from 2 to 21).
• Make sure 'TrimDAC' box is checked.
• Verify that LV is OFF (ask tutor).
• Go to state machine and click on 'Configure'.
• If there are no errors the FEDs are now configured.
• The CrateScan is now complete.
• Click 'Destroy' in RCMS to destroy the run.
• LV can be turned back on now (ask tutor).

• Let's look at the information uploaded to the database in more detail (e.g. for the Crates, Slots, FED/FECs, APVs)
• Login to machine cmstracker037 as user xdaqdev (ask tutor for password)
• In the shell do:
  • source /opt/trackerDAQ/config/trackerDAQ.env
  • export TNS_ADMIN=/exports/slc6/CMSSW/Base/slc6_amd64_gcc462/cms/oracle-env/29/etc

• • export CONFDB=cms_tracker_tif3/xxxxxxx@cms_sstracker (ask tutor for password to replace xxxxxxx)
  • ./tkconfigurationdb.sh.new $CONFDB
• Open a web browser and go to http://cmstracker037.cern.ch:15000/urn:xdaq-application:lid=15/
• Click 'Database Parameters'.
• Click 'Apply' to login
• Choose the Database you wish to look at from the Partition Name drop down menu.
• Click 'Apply' again
• Click 'Configure Database'
• Click 'Partition/Versions'
• There you will see the new partition and versions for the FEC and FED in the database.
• Click through the links: [Partition/Version] [FEC Partition Parameters] [Modules & Parameters] [FED Parameters] [FEC/FED Connections]
• Under FED Parameters, select a FED and look through each APV's TD (trimDAC). These should each be less than 100 when the Lasers are off.

Step 2: Connection Run

• Click on 02-CrackConnection.
• Click 'Initialize' and wait until it reaches Halted state.
• Click 'Configure' and wait until it reaches Configured state.
• Start the run.
• The run should collect 34 events. To see the number of events, click on the StorageManager in the Status Display. Here you'll see the event information (refresh to update).
• During the run we can look at what's going on with the Storage Manager in the status display.
• After reaching 34 events, click 'Halt' to halt the configuration and wait for it to reach Halted state.
• After successfully halting the run, Destroy the run.
• Now we can analyze the data.
• login as xdaqtk@cmstracker036 (ask tutor for password)
• Look for the data in /opt/cmssw/Data/closed/<RUN#>.
• Open the program tkCommissioner:
  • 1) cd /home/xdaqtk/erik/CommissioningGui/tkCommissioner/
  • 2) source setpaths
  • 3) export CONFDB=cms_tracker_tif3/xxxxxxxx@cms_sstracker
  • 4) ./tkCommissioner -l &
• In tkCommissioner, select your partition (left hand column) and then the run (right hand column) and click on Analyze

Go to /opt/cmssw/Data/RUNNUMBER

lses analysis__xxxxx.debug.log

Information in the log file should be presented like this:

[FastFedCablingHistosUsingDb::connections] Summary of connections:
 "Good" connections     : 143
 "Dirty" connections    : 1
 "Bad" TrimDAQ settings : 0
 ("Missing" connections : 0)
 ("Missing" APV pairs   : 0)
 ("Missing" APVs        : 0)

%MSG-w SiStripCabling:  SiStripCommissioningOfflineDbClient:db_client@beginRun  10-May-2013 11:33:39 CEST Run: 1
[FastFedCablingHistosUsingDb::connections] List of "dirty" connections:
FED:crate/slot/id/unit/chan/apv= -/-/207/1/6/0 FEC:crate/slot/ring/CCU/module/LLD/I2C= 1/4/8/1/18/1/0 DcuId= 005bf185 DetId= ffffffff

• • A dirty connection corresponds to a channel with a high light level below 800 ADC counts.
  • A channel with a bad trimDAC setting corresponds to one with a trimDAC value below 10 ADC counts.
  • Good, dirty and bad trimDAC connections will all be uploaded to the database.
  • Any missing connections or devices will not be uploaded. "Missing connections" are devices identified in the crate scan that could not be connected to a FED channel (ie. DCU ID could not be extracted from the histogram for some reason - miscabling or broken fibre). "Missing APV pairs" or "Missing APVs" are devices from the DCU-DetID static table that were not found during the crate scan.
  • Make sure there are no missing connections or devices.

• An Example of a good channel was also found on the same log file.

[FastCablingAnalysis] Monitorables (65535 means "invalid"):
 Crate/FEC/Ring/CCU/Mod/LLD     : 2/2/8/1/19/1
 FedId/FeUnit/FeChan/FedChannel : 52/6/12/24
 FecKey/Fedkey (hex)            : 0x10a00484 / 0x0000d1b0
 DcuId (hex/dec)                : 0x00defacb /   14613195
 DetId (hex/dec)                : 0xffffffff / 4294967295
 DCU id extracted from histo     : 0x00defacb
 LLD chan extracted from histo   : 1
 "High" level (mean+/-rms) [ADC] : 690.87 +/- 2.59
 "Low" level (mean+/-rms)  [ADC] : 37.94 +/- 0.74
 Median "high" level       [ADC] : 690.78
 Median "low" level        [ADC] : 37.77
 Range                     [ADC] : 652.93
 Mid-range level           [ADC] : 364.40
 Maximum level             [ADC] : 695.37
 Minimum level             [ADC] : 36.99
 isValid                         : true
 isDirty                         : true
 badTrimDac                      : false
 Error codes (found  0)          : (none)

• • The FED ID/channel information will always be set, regardless of whether a connection was established or not.
  • The DCU ID (and hence the Det ID and the location in the control system) will only be given for channels which were successfully connected. Otherwise they appear as 0xFFFFFFFF.
  • Three flags are given to show if the channel was valid, dirty or had a bad trimDAC setting.

• An example of a bad connection is:

[FastCablingAnalysis] Monitorables (65535 means "invalid"):
 Crate/FEC/Ring/CCU/Mod/LLD     : (invalid)
 FedId/FeUnit/FeChan/FedChannel : 124/3/6/66
 FecKey/Fedkey (hex)            : 0x3fffffe8 / 0x0001f0d8
 DcuId (hex/dec)                : 0xffffffff / 4294967295
 DetId (hex/dec)                : 0xffffffff / 4294967295
 DCU id extracted from histo     : 0xffffffff
 LLD chan extracted from histo   : 65535
 "High" level (mean+/-rms) [ADC] : 65535.00 +/- 65535.00
 "Low" level (mean+/-rms)  [ADC] : 65535.00 +/- 65535.00
 Median "high" level       [ADC] : 65535.00
 Median "low" level        [ADC] : 65535.00
 Range                     [ADC] : 0.82
 Mid-range level           [ADC] : 38.12
 Maximum level             [ADC] : 38.53
 Minimum level             [ADC] : 37.71
 isValid                         : false
 isDirty                         : false
 badTrimDac                      : false
 Error codes (found  1)          : SmallRangeInRawData 

• You should check the following summary histograms:
• Go to the tkCommissioner and click 'View Results'.

Histogram Name	Type	Description

Highlevel (X)	1D	The saturation light level for all connected channels (average of the saturation level observed in the DCU bit pattern). A large spike should be visible at 1023 counts (corresponding to the 10-bit ADC). Ideally all channels will have a high light level of 1023 counts, but it is possible that a small number will be slightly lower. Any channel with a high light level below 800 ADC counts is flagged as dirty. The number of entries corresponds to all available FED channels. The integral is the number of valid connections and the overflow corresponds to the unconnected channels (65535).
Lowlevel (X)	1D	The low light level for all connected channels (average of the low light levels observed in the DCU bit pattern). The low light level should be approximately the trimDAC setting determined during the FED crate scan. This should be around 30-40 ADC counts. A value below 10 ADC counts is flagged as a bad trimDAC setting.
Highlevel (Y) vs FeChan (X)	2D	This shows the saturation light level for all connected channels versus FED FE unit. Useful for debugging purposes.
Lowlevel (Y) vs FeChan (X)	2D	This shows the low light level for all connected channels versus FED FE unit. Useful for debugging purposes.

• After examining the histogram, if the run is good upload to the analysis and HW database.
• To upload a commissioning run check the box 'Upload Config' in the tkCommissioner and then click 'Analyze'. Click 'OK' in the pop up window.

To make sure that the connection run data was uploaded, go to cmstracker029.cern.ch:15000 and see that the FEDs are now connected to modules.

Step 3: Timing Run #1

• Click on 03-CrackTiming.
• Initialize the run.
• Configure the run.
• Start the run and collect 4800 events.
• Check the storage manager for the status of event collection.
• After collecting 4800 events, Halt the run and Destroy.
• Run the analysis as usual through the tkcommissioner, and check the following histograms.
• In the timing run we adjust the delay of each channel arriving at the FED by changing the delay applied to the clock and reset/trigger signals in the PLL on the module which are sent to the APVs in such a way as to time-align the arrival of data at the FED. An adjustment for the fibre length should in principal already be applied to the delays in the input to the FED, although it is still possible to time align the data in the CRACK without such an adjustment, since all path lengths are very similar in such a small setup. This step is performed by sending a resynch signal to the front end and then measuring the alignment of the arrival of the APV tick marks at the FED. The timing run then determines the necessary delay to apply to each channel in order to align the arrival of all the ticks at the FED Front end FPGAs. Note that in the full tracker it is essential that the path length adjustments are applied to the FED delays before the tick timing run is performed, otherwise we me be unable to achieve perfect alignment for all APVs with the same Latency.

The summary histograms are as follows:

• Go to tkCommissioner, select the run and click 'View Results'.

Histogram Name	Type	Description

Delay (X)	1D	This shows the delay adjustment required to synchronise the system. On the first timing run, there will be a spread of a few ns.
TickHeight (X)	1D	This shows the measured tick mark height. Once the system is synchronised (after bias and gain scan), the mean should be about 690 ADC counts. After the first timing run, it will normally be lower. The tick mark height is about 8 MIPs and 1 MIP corresponds to 80 ADC counts, giving a mean of 690 ADC counts.
TickHeight (Y) vs DeviceId (X)	2D	This shows the heights of successfully identified tick marks as a function of module.

Delay (Y) vs DeviceId (X)

2D

This shows the applied timing adjustment as a function of module. Outliers observed in the previous plots with have a correspondingly large delay adjustment.

Some points to bear in mind

The timing of the channels is adjusted to the "latest" tick mark edge. If there is a problem with the "latest" tick mark, then this will be detected by the analysis job and an error will be written in <run_number>_error.log.

If any channels have a missing tick mark because the rising edge has moved out of range, then the latency can be adjusted. This is done by opening the TrackerSupervisor from the status display and selecting Commissioning Setting. The value that needs to be changed is the APV reset latency. To make the tick mark move to the right, increase the value of the latency. To make it move to the left, decrease the value of the latency. 1 step in latency moves the tickmark by 25 units in the histograms. If this is necessary, then make sure that you document what you did fully in the elog.

• If everything looks good with this run, upload the parameters to the database.

Step 4: Gain scan

• Click on 04-CrackGainScan.
• Initialize the run.
• Configure the run.
• Start the run and collect 2000 events.
• Check the storage manager for the status of event collection.
• After collecting 2000 events, Halt the run and Destroy.
• Analyze the run with tkCommissioner.
• Check the summary histograms:

Histogram Name	Type	Description
Measgain1 (X)	1D	This shows the distribution of the measured gain values for all channels. The mean of the distribution should be about 0.8 V/V. All channels should fall into the 0-1 V/V range, although a small number of entries in the 1-2 V/V is acceptable, providing the mean is good.
TickHeight1 (X)	1D	This shows the measured tick mark heights and should therefore have a mean of approximately 690 ADC counts. The spread should be typically less than 10%.
Gain (X)	1D	This shows the distribution for the selected gain settings for all channels. The majority of channels should have a gain setting of 1. If you see an excess at 3, this indicates that many channels needed the highest possible gain, suggesting that there is a problem, eg. dirty fibres.
Bias1 (X)	1D	This shows the distribution of the selected bias settings for all channels. The mean value should be about 20-24.
Zerolight1 (X)	1D	This shows the zero light level, basically the same as seen in the connection run. It should therefore be above 10 ADC counts. If below, it indicates a problem with the trimDAC calibration performed during the FED crate scan.

There are also three expert-level histograms for each channel: The baseline level on which the tick mark sits within the raw APV data stream is shown as a function of bias setting, the point at which light is seen (lift-off) will be about 20. The tick mark height is also shown as a function of bias setting. Lift-off occurs earlier in this plot and saturation is correspondingly reached earlier. The measured gain is calculated from these two histograms. The third histogram is the noise in the sample for the baseline as a function of bias setting.

Step 3 Repeat: Timing Run #2

• Take a second timing run.
• Click on 03-CrackTiming (same as used for Timing Run #1).
• Initialize the run.
• Configure the run.
• Start the run and collect 4800 events.
• Check the storage manager for the status of event collection.
• After collecting 4800 events, Halt the run and Destroy.
• Run the analysis as usual through the tkcommissioner, and check the following histograms.

Histogram Name	Type	Description
TickHeight (X)	1D	This distribution should have a mean of about 690 ADC counts (this is the tick mark height that corresponds roughly to a measured gain of 0.8 V/V).
TickHeight (Y) vs DeviceId (X)	2D	This distribution should be much flatter than that observed in the previous run.
Delay (X)	1D	This distribution should be narrower than that observed in the previous run and it should have a mean of about 25ns.

Step 5: VPSP scan

• Click on 05-CrackVPSPScan
• Initialize the run.
• Configure the run.
• Start the run and collect 4720 events.
• Check the storage manager for the status of event collection.
• After collecting 4720 events, Halt the run and Destroy.
• Run the analysis as usual through the tkcommissioner, and check the summary histograms.

The digital "0" level is roughly equal to the trim DAC value (30-40 ADC counts) plus a margin to allow for the effects of temperature fluctuations. The value of the digital "0" level is determined for a particular bias setting. It is taken to be the baseline lift-off plus 2-3 bias settings.

The baseline is defined as the median pedestal value, which is approximately the digital "0" level, plus a third of the tick mark height (~690 ADC counts, as set in the second timing run). The highest possible value for the baseline should be approximately the digital "0" level, plus the tick mark height, plus 5-10%. The lowest possible level should be around the digital "0" level.

The definitions of the summary histograms are as follows:

Histogram Name	Type	Description
Vpsp (X)	1D	This shows the distribution of VPSP settings that determine the level of the APV baseline. Make sure the mean of the distribution lies between around 30 and 50 and that there are no obvious outliers.
Adclevel (X)	1D	This shows the baseline ADC levels determined during the run. The majority of entries should lie approximately in the range 250 - 400, but a small number of higher entries is OK. Obviously low entries may indicate a problem.
Toplevel (X)	1D	This corresponds to the highest possible level for the baseline for each channel. Check for obvious outliers.
Bottomlevel (X)	1D	This corresponds to the lowest possible level for the baseline for each channel. Check for obvious outliers.
Vpsp (Y) vs DeviceId (X)	2D	This shows the VPSP settings versus channel number. The distribution should be flat.
adclevel (Y) vs DeviceId (X)	2D	This shows the baseline level versus channel number. Again, the distribution should be flat.

Step 6: Pedestal run (HV OFF)

• For this run leave the HV OFF
• Click on 06-CrackPedetal
• Initialize the run.
• Configure the run.
• Start the run and collect 4020 events.
• Check the storage manager for the status of event collection.
• After collecting 4020 events, Halt the run and Destroy.
• Run the analysis as usual through the tkcommissioner, and check the summary histograms.

The definition of the summary histograms is below. All the quantities shown in the summary histograms are shown per APV pair. Each APV pair is connected to 256 strips and therefore the total number of strips can be determined by multiplying the number of entries in the histogram by 256. To convert noise in ADC counts into a number of electrons: 1 MIP = 80 ADC counts; 1MIP = ~25000 electrons.

Histogram Name	Type	Description

NoiseMean (X)	1D	The distribution of the noise level per module. Mean should be ~5 (7) ADC with HV ON (OFF). Outliers should be noted. The profile versions of this histogram and the pedestals can be used to identify the location of outliers.
PedestalMean (X)	1D	The distribution of pedestal values per module. Mean should be ~300 ADC. The spread should be approximately 10% of the mean.

Step 6 Repeat: Pedestal run (HV ON)

• For this run turn the HV ON
• Click on 06-CrackPedetal
• Initialize the run.
• Configure the run.
• Start the run and collect 4020 events.
• Check the storage manager for the status of event collection.
• After collecting 4020 events, Halt the run and Destroy.
• Run the analysis as usual through the tkcommissioner, and check the summary histograms.

The results of the two runs (HV OFF and HV ON) should be compared and the differences noted. The noise level will go down with the HV on.

Cosmics Run with Muon Trigger

• For this run turn the HV ON
• Log in to cmstracker026.cern.ch as root (get the password from someone in the know)
• Perform the commands in the file ~root/instructions_new_130326 to start the FMM application
• Open a new terminal for cmstracker026 (clean environment)
• Run ~root/configure_enable.sh
• From RMCS, click on CrackPhysicsLTC
• Initialize the run
• Configure the run
• Start the run and when necessary Halt, then Destroy.

From here on we are still testing and improving things. Data analysis through tkCommissioner also needs to be tested. Watch this space!