WIP, but there are some attachments to help you deal with really bad things if the CrtExperts aren't around. Please contact us before following any of these instructions.

Running the CRT Stand-Alone

The CRT board reader really controls the CRT "backend", a process called "readout", through two perl scripts. So, a stand-alone CRT run can be controlled from the command line with correct setup. The output files are in a custom binary format, and they cannot easily be merged with art-ROOT files from Run Control.

  1. Configure the CRT timing endpoint
    1. LogInToCRTServer to control the CRT timing endpoint.
    2. source /nfs/sw/crt/timing-board-software/tests/env.sh #Sets up the software to communicate with the CRT timing endpoint
    3. [np04daq@np04-crt-001 ~]$ pdtbutler crt CRT_EPT status #Check the CRT timing endpoint's status. You should get something like:
      Created crt device
      +------------------+------------+
      | csr | 0x0 |
      | csr.ctrl | 0x0 |
      | csr.ctrl.tgrp | 0x0 |
      | csr.stat | 0x81 |
      | csr.stat.ep_rdy | 0x1 |
      | csr.stat.ep_stat | 0x8 |
      | pulse | 0x0 |
      | pulse.cnt | 0x3 |
      | pulse.ctrl | 0x0 |
      | pulse.ctrl.cmd | 0x0 |
      | pulse.ctrl.en | 0x0 |
      | pulse.ctrl.force | 0x0 |
      | pulse.ts_h | 0x112cf7d |
      | pulse.ts_l | 0x74189276 |
      +------------------+------------+
      Last Pulse Timestamp: 77352281344348790 (0x112cf7d74189276)
    4. If you got something different above, you might need to do other things to the CRTTimingEndpoint
    5. pdtbutler crt CRT_EPT configure 0 TimeSync #Put the CRT timing endpoint in "constant sync pulse" mode
  2. Starting and Stopping a Run
    1. In a different terminal, LogInToCRTServer. You must be on the CRT server to communicate with the CRT hardware because we control each daisy chain's readout board by USB.
    2. source /nfs/sw/crt/readout_linux/script/setup.sh #puts CRT backend scripts and executables into this shell's PATH
    • A full-detector run: Useful for ensuring that the CRT works after a power cycle or a hardware problem
      1. startallboards.pl gainupdate_v9 #As of 11/14/2018, gainupdate_v9 is gain the configuration that all Run Control-based runs should be using
      2. While startallboards.pl is running, you should see lots of output to stdout. Among this output should be a block of baseline values. Healthy baseline values look like:

        PMT 0: baseline hits: 500
        PMT 1: baseline hits: 500
        PMT 2: baseline hits: 500
        PMT 3: baseline hits: 500
        PMT 4: baseline hits: 500
        PMT 5: baseline hits: 500
        PMT 6: baseline hits: 499
        PMT 7: baseline hits: 479
        PMT 8: baseline hits: 500
        PMT 9: baseline hits: 499
        PMT 10: baseline hits: 500
        PMT 11: baseline hits: 500
        PMT 12: baseline hits: 500
        PMT 13: baseline hits: 500
        PMT 14: baseline hits: 500
        PMT 15: baseline hits: 481
        PMT 16: baseline hits: 500
        PMT 17: baseline hits: 500
        PMT 18: baseline hits: 500
        PMT 19: baseline hits: 498
        PMT 20: baseline hits: 500
        PMT 21: baseline hits: 500
        PMT 22: baseline hits: 500
        PMT 23: baseline hits: 500
        PMT 24: baseline hits: 500
        PMT 25: baseline hits: 500
        PMT 26: baseline hits: 500
        PMT 27: baseline hits: 489
        PMT 28: baseline hits: 500
        PMT 29: baseline hits: 500
        PMT 30: baseline hits: 500
        PMT 31: baseline hits: 509

      3. Wait for the CRT to write enough data for your purposes. 3-5 minutes is usually plenty of data for confirming that the CRT is working
      4. stopallboards.pl gainupdate_v9 #It is important that the gain table name you use here matches the one you used with startallboards.pl
    • Just one board: Probably experts only. If you need to look at problems on just one board, then something really bad has happened.
      1. oneboard.pl <USB daisy chain number> <board number>
      2. Make sure baseline looks healthy as in instructions for a full detector run
      3. This method stops by itself after $run_length seconds. Make a copy of oneboard.pl and edit the perl source code to change run_length. You might also want to change $onlinetable to a more recent table.
  3. If you've recently had a problem with the CRT, you might want to CheckForDeadUSBs
  4. Analyze the data
    1. Make a directory in which to dump processed files and cd there.
    2. Convert files to a format that "calibration workflow" can use:
      1. EBuilder: Used for official calibration plots. Probably a little more complete.
        1. /nfs/sw/crt/EBuilder/bin/EventBuilder -c /nfs/sw/crt/EBuilder/config/allboard.config -i ~np04daq/crt/readout_linux/DataFolder/binary -o <some unique name> #~np04daq/crt/readout_linux/DataFolder is a symbolic link to the latest data directory created by startallboards.pl
        2. Interrupt or kill EBuilder when it starts to print that 0 events have been built. It won't exit on its own. You'll know it's time to stop EBuilder when you see something like:

          Data file from USB 13 found
          Data file from USB 14 found
          Data file from USB 3 found
          Data file from USB 22 found
          Decoding file set #10 for this run
          Parity error in USB stream 14
          Sent decoded data up to Unix time stamp 1542211402 for USB 13
          Sent decoded data up to Unix time stamp 1542211400 for USB 14
          Sent decoded data up to Unix time stamp 1542211402 for USB 3
          Sent decoded data up to Unix time stamp 1542211399 for USB 22
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          No new files for 5s, but I didn't hear that the run was over! Closing output file anyway.
          Module 114 missed sync pulse near Unix time stamp 1542211394
          Module 114 max clock count 2383711634 Number of built events: 58120
          Processed time stamp: 1542211402
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          Files are not ready. Waiting...
          No new files for 5s, but I didn't hear that the run was over! Closing output file anyway.
          Number of built events: 0
          Processed time stamp: 1542211402
          Files are not ready. Waiting...

        3. ~/anolivie/crt/scripts/rename.sh * #appends the .ebld extension to the name of every file in this directory
      2. decode.pl: Slightly less "mysterious" to me. Not used as often as EBuilder. Timestamps might be more reliable.
        1. perl -S ~/anolivie/crt/scripts/groupByTimestampParallel.pl ~np04daq/crt/readout_linux/DataFolder/binary #~np04daq/crt/readout_linux/DataFolder is a symbolic link to the latest data directory created by startallboards.pl
        2. perl -S decode.pl <file name> #Decodes an individual file and makes no attempt to group files from different daisy-chains. Useful if you think the CRT is producing corrupted data. "p" packets are good, "x" packets are bad, there are also timestamp packets that the board reader doesn't use, and anything else is weird
      3. source /nfs/sw/artdaq/setupDUNEARTDAQ
      4. ~/anolivie/crt/binary/bin/onlinePlots `find . -type f -name '*.ebld' -size +100b` #or *.dec if you used decode.pl. The find command excludes the empty files that EBuilder tends to produce
      5. Look at the plots in OnlinePlotsFromEBuilders.root with root (the name is misleading if you used decode.pl). Pay particular attention to CRTOnlineMonitor /PerJob/MeanRate. It should look like the top plot in Monet in CRTBasicOp
    3. Run the calibration analysis
      1. ~/anolivie/crt/binary/bin/toROOTFlat.ebld #.dec if you used decode.pl. Makes a .root file with an object-less TTree
      2. TODO: Instructions from Sarah
    4. Note: Run an "event display". With X11-forwarding enabled:
      ~/anolivie/crt/binary/bin/evd .ebld #.dec if you used decode.pl
    5. Note: You could even write your own analysis program that reads CRT::Triggers as they are interpreted from an EBuilder/decode file. See IMPL in ~/anolivie/crt/binary/src/emitter/Emitter.cpp for details of the interface you must implement and ~/anolivie/crt/binary/src/app/onlinePlots.cpp and ~/anolivie/crt/alg/src/monitor/OnlinePlotter.cpp for examples
  5. Reconfigure the CRT timing endpoint for the DAQ
    1. In a different terminal, LogInToCRTServer
    2. source /nfs/sw/crt/timing-board-software/tests/env.sh #Sets up the software to communicate with the CRT timing endpoint
    3. pdtbutler crt CRT_EPT configure 0 RunStart #Put the CRT timing endpoint in "one sync pulse when run starts" mode[np04daq@np04-crt-001 ~]$ pdtbutler crt CRT_EPT status #Check the CRT timing endpoint's status. You should get something like:
      Created crt device
      +------------------+------------+
      | csr | 0x0 |
      | csr.ctrl | 0x0 |
      | csr.ctrl.tgrp | 0x0 |
      | csr.stat | 0x81 |
      | csr.stat.ep_rdy | 0x1 |
      | csr.stat.ep_stat | 0x8 |
      | pulse | 0x0 |
      | pulse.cnt | 0x3 |
      | pulse.ctrl | 0x0 |
      | pulse.ctrl.cmd | 0x0 |
      | pulse.ctrl.en | 0x0 |
      | pulse.ctrl.force | 0x0 |
      | pulse.ts_h | 0x112cf7d |
      | pulse.ts_l | 0x74189276 |
      +------------------+------------+
      Last Pulse Timestamp: 77352281344348790 (0x112cf7d74189276)
    4. If you got something different above, you might need to do other things to the CRTTimingEndpoint

Interacting with the CRT LV Power Supply

  1. LogInToCRTServer
  2. source /nfs/sw/crt/readout_linux/script/setup.sh #puts CRT backend scripts and executables into this shell's PATH
  3. Interact with Low Voltage power supply:
    • Check voltages and currents: pollAllCRTLV.sh
      Healthy values look like:
      #Top power supply:
      #channel 1 channel 2
      +6.397623E+00 +6.395380E+00
      +2.483096E+00 +2.483986E+00
      #Bottom power supply:
      #channel 1 channel 2 channel 3 channel 4
      +6.099715E+00 +6.295719E+00 +6.293567E+00 +4.997525E+00
      +2.100666E+00 +2.486437E+00 +2.476722E+00 +5.449448E-01

    • Power cycle CRT: powerCycleCRT.sh
    • Turn CRT on: turnOnCRTLV.exp #HV is still on!
    • Turn CRT off: turnOffCRTLV.exp #HV is still on
  4. Directly via telnet:
    1. EXPERTS ONLY except as a last resort! Contact the CRTExperts for help. So far, the sole use case is when I want to make sure I haven't set the timing endpoint's frequency incorrectly after a hard reset.
    2. The CRT LV power supply can be controlled remotely with SCPI. Top power supply, with daisy chains only, is np04-ps-crt-001, and bottom, with fanout power, is np04-ps-crt-002. Ports 5024 and 5025 are apparently good ones to use.
    3. Read the telnet instructions for our power supplies at https://literature.cdn.keysight.com/litweb/pdf/5969-2937.pdf
    4. Example SCPI session to turn on channel 1 of the bottom power supply:
      [np04daq@np04-crt-001 ~]$ telnet np04-ps-crt-02 5024
      Trying 10.73.128.33...
      Connected to np04-ps-crt-02.
      Escape character is '^]'.
      Agilent N6700B Low-Profile MPS Mainframe

      SCPI> OUTPUT ON, (@1)
      SCPI> ^]
      telnet> quit
      Connection closed.

CRT DAQ Software Packages

These should all be git repositories although not very well managed. Packages installed in /nfs are available on all DAQ servers.

Name Path on CRT server Purpose git Remote URL
CRT DAQ /nfs/sw/crt/readout_linux Builds the CRT readout backend and has perl scripts to
control it.
https://:@gitlab.cern.ch:8443/protoDUNE-SP-DAQ/crt-daq-scripts.git
crt-core /nfs/home/np04daq/anolivie/crt/core/src Clone of CRT::Trigger outside of offline framework.
Dependency of next 2 entries.
https://:@gitlab.cern.ch:8443/anolivie/crt-core.git
crt-alg /nfs/home/np04daq/anolivie/crt/alg/src Makes trees and monitoring plots for calibration analysis.
Offline-compatible algorithms for making ROOT files.
Also copied into dunetpc offline framework.
https://:@gitlab.cern.ch:8443/anolivie/crt-alg.git
crt-binary /nfs/home/np04daq/anolivie/crt/binary/src Applications to run algorithms from crt-alg without offline
framework.
https://:@gitlab.cern.ch:8443/anolivie/crt-binary.git
EBuilder /nfs/sw/crt/EBuilder/src Converts raw files from backend to a format that the
calibration analysis can read.
https://:@gitlab.cern.ch:8443/anolivie/crt-binary.git
Timing Endpoint /nfs/sw/crt/timing-board-software Interface to CRT timing endpoint. Not managed by CRT
team.
https://:@gitlab.cern.ch:8443/protoDUNE-SP-DAQ/timing-board-software.git
-- Main.anolivie - 2018-11-02
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2019-01-16 - AndrewPaulOlivier
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CENF All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback