Guidance for on-call experts

Common ZDC problems, and their solutions

Below is a list of the common errors, and the appropriate actions to take:

  • ZDC FSM panel Scalers, or HV channels, are in "UNKNOWN" state
  • ZDC plots in Luminosity FSM panels are not updating

ZDC Resources for ZDC Experts

Uncommon ZDC problems, and their solutions

DAQ errors show "Received exception TIMEOUT when interacting with zdc-readout"

This error (and other "inexplicable" errors) may point to an unresponsive SBC (sbc-zdc-daq-01). To check the status of the sbc, go to: and search for "sbc-zdc-daq-01". Under the "Host log" column, click on sbc-zdc-daq-01. You should be able to see if there was a recent alarm, and if the SBC is up or down. _If the SBC is not responding, or it is down, follow the instructions under Cycling power in the VME crate (Remotely)

Updating ZDC software with new L1Calo release

  • Copy l1calo-01-00-xx to local directory /det/zdc/l1calo/releases/tdaq-02-00-03/
  • Make a new zdc version (zdc-00-00-xx)
  • Copy the most recent setup script to setup_00-00-xx.sh and edit the setup script appropriately:
    • myL1CaloVersion=l1calo-01-00-xx
    • myZdcVersion=zdc-00-00-xx
  • Link this new setup script to setup.sh
  • In the following directories, point the link "pro" to the new directories (using ln -s yyy-xx-xx-xx pro)
    • /det/zdc/l1calo/releases/tdaq-02-00-03/
    • /det/zdc/releases/tdaq-02-00-03
  • Source the setup script.
  • In /det/zdc/releases/tdaq-02-00-03/zdc-00-00-xx/cmt/projects.cmt, do:
    • use tdaq-02-00-03 l1calo-01-00-xx
  • Rebuild all packages (no better way, that I know of, than doing "buildpkg" in all subdirectories)
    • Note that packages ZdcTest and configZdc will likely not build properly; these are old, unused packages.
  • Go into /det/zdc/oks/tdaq-02-00-03/zdc/sw, and create file zdc_software.data.xml_zdc-00-00-xx (copy from most recent file).
    • Change the "installation path" of the SW_Repository to point to the correct directory.
    • Then copy this file to zdc_software.data.xml and commit using:
      • oks-import.sh -d zdc sw/zdc_software.data.xml
  • Go into /det/zdc/oks/tdaq-02-00-03/zdc/segments/DQM and edit the file zdc_dqm_algos.data.xml, pointing to the correct zdc release for the dqm software libraries.
  • Test the configuration by doing "runme" (setup_daq -p ZdcStandalone)

Changing PPM common delay, or number of FADC slices

  • After sourcing the setup script, type "ace &"
  • Click "open" - options below should be pre-filled:
    • Adhoc connections checked
    • Database type: sqlite
    • Schema: /det/zdc/calib/tdaq-02-00-03/calib.sqlite
    • DB Instance: L1CALO
  • Press "Connect."
  • Go to ZDC -> ToBeChecked -> V1 -> Configuration -> ReadoutConfig. Press OK.
    • baselinePoint sets the common delay of the PPM channels, and is currently set to 4.
    • numFadcSlices sets the number of FADC slices (currently 7).
    • numLutSlices is important for some unknown reason - currently set to 1.
  • Right-click the entry and select "Clone Rows"
    • Edit the new item. Then press "Commit."

Cycling Power in the PPMs

  • After sourcing the setup script, type "dbhdmc &"
  • Do File -> Load DB crate -> zdc-zdc (sbc-zdc-daq-01.cern.ch)
    • Note - if too many VME processes are running, this might be impossible! (You'll be met with a segmentation violation.) In that case, you'll have to perhaps do "stop_zdc_scalers vme" and "stop_zdc_caen_hv" (this will temporarily disable communication with DCS - but shouldn't be disastrous.)
  • In each PPM module, go to VME Config. Read the values. Then write "1" to "power on-off." When it's clear that power is off, turn power back on in the same fashion.

Setting up your own ZDC software development version

  • in /det/zdc/releases/tdaq-02-00-03, Copy zdc-00-00-xx to zdc-00-00-ydevxx (i.e. zdc-00-00-kdev19)
    • Alternatively, you can do:
      • svn co svn+ssh://svn.cern.ch/reps/atlaszdc/ZdcOnlineMonTags/zdc-00-00-xx zdc-00-00ydevxx
  • Copy setup_kdevxx.sh to setup_ydevxx.sh; edit the following lines appropriately:
    • myL1CaloVersion=l1calo-01-00-70 (change to 71)
    • myZdcVersion=zdc-00-00-kdev18 (change to ydevxx)
  • Go to /det/zdc/oks/tdaq-02-00-03/zdc/sw
    • cp zdc_software.data.xml zdc_software.data.xml_zdc-00-00-ydevxx
    • Edit this new file, making sure that the SW repository installation path is correct:
      • < attr name="InstallationPath" type="string">"/det/zdc/releases/tdaq-02-00-03/zdc-00-00-ydevxx/installed"</attr >
  • Now source /det/zdc/setup/setup_ydevxx.sh
  • Rebuild the package ZdcRelease (in the ydevxx package).
  • Now you should be done; source /det/zdc/setup/setup_ydevxx.sh once more and do "runme" to test.
  • Note: you do not have to commit anything to OKS. Local OKS files are used instead!

Cycling power in the VME crate (Remotely)

If operating remotely, do the following before continuing:

  • ssh -Y username@atlasgwNOSPAMPLEASE.cern.ch
  • enter Hostname: PCATLZDC01 - You will be connected in 4 seconds to PCATLZDC01 machine.
  • From here, do Start -> PVSS Console
  • In the console, find the process "User Interface" with option "-p fwDeviceEditorNavigator/fwDeviceEditorNavigator.pnl -iconBar -menuBar
    • click on it and press the green "start process" button on the right
  • You will be taken to the "Device Editor and Navigator." Click on the "FSM" tab, and expand the point "ATLZDC01." When you see ATL_ZDC, right-click on it and press "view." This will take you to the familiar FSM panel.

After having accessed the FSM panel:

  • Go to Crates -> Crate3. Check that this crate's description is "ZDC VME Crate, USA15"
    • At this point you may have to log in at the top right (press the "key" icon)
  • In the "Commands" box, click "Off." Wait for the the "Power On" indicator to go orange, then wait an additional ~minute.
  • Click "On." Make sure the status indicators go green.
  • Before breathing a sigh of relief, make sure you have restarted all standalone applications (to be listed here soon).
  • When joining the main ATLAS partition after this procedure, request that the entire system be rebooted. Otherwise, the next run will produce an avalanche of readout errors.

Aligning Pedestals

Use this procedure to align pedestal values. Pedestal values are conventionally set to 40 FADC counts.

Part 1: Record the data

  • Before starting the ZdcStandalone partition, edit zdc_segments_patgen.data.xml, giving the following parameters to rc-zdc-patgen:
    • < attr name="Parameters" type="string">"-n ZdcPatGen -p ${TDAQ_PARTITION} -1 20 -r 100. -t"</attr >
    • < attr name="RestartParameters" type="string">"-n ZdcPatGen -p ${TDAQ_PARTITION} -1 20 -r 100. -t "</attr >
  • Start the ZdcStandalone partition:
    • Enable the patgen, readout, and calibmon segments. Make sure to commit the changes before continuing!
    • Enable data recording. Select a file name tag, and click "Set Values."
  • Run through the BOOT transition.
  • Enable to ZDC panel (Load Panels -> l1calo.igui.ZdcPanel)
    • Go to RunPars tab, and select "PprDacScanPars" as Run Parameter Set.
  • Now INITIALIZE, CONFIG, START. Wait for the process to complete.
  • Shut down the partition.

Part 2: Analyze the data

  • Run (use same spelling as below):
    • analysePpmRun -d filename.data -r runnumber -f Physics -a PprDacScanPars -u 1
  • (Example:)
    • analysePpmRun -d /zdc/calibrations/data_test.00153536.calibration_pedestalalign.daq.RAW._lb0000._ROSEventBuilder._0001.data -r 153536 -f Physics -a PprDacScanPars -u 1

A root file PprDacScanPars.0153536.root will be created. The pedestal linearly depends on the offset (a value in some PPM register):

  • ped =a + b*offset.
This dependence was studied in alignment procedure. For each channel the measured dependence is displayed in corresponding histograms. The parameters a and b are stored in the database (in some special folder).

Part 3: Commit the results to the database

  • To update the calibration folder, run the following command:
    • useCalib.orig -d "sqlite://;schema=/det/zdc/calib/tdaq-02-00-03/calib.sqlite;dbname=L1CALO" -f Physics -v PprChanCalib -r PprDacScanResults
    • [Previously this command was called "useCalib".]
  • When ZDC is configured, the following offset is recorded to the PPM register: (p-a)/b, where "p" is desired value of pedestal.

Current ZDC HV settings

For 7 TeV running....

Mod Ch on/off Status Vmon Imon Alarm Name
11 0 On 3101 90 0 0 "Zdc12, PIN diode"
11 1 On 3101 1202 299 0 "Zdc12, HM1"
11 2 On 3101 1204 299 0 "Zdc12, HM2"
11 3 On 3101 1202 298 0 "Zdc12, HM3"
12 0 On 3101 749 1803 0 "Zdc12, HXY_0-7"
12 1 On 3101 749 1809 0 "Zdc12, HXY_8-15"
12 2 On 3101 748 1799 0 "Zdc12, HXY_16-23"
21 0 Off 3100 0 0 1 "reserved"
21 1 On 3101 1200 295 0 "Zdc81, HM1"
21 2 On 3101 1201 297 0 "Zdc81, HM2"
21 3 On 3101 1201 297 0 "Zdc81, HM3"
22 0 On 3101 749 1797 0 "Zdc81, HXY_0-7"
22 1 On 3101 748 1795 0 "Zdc81, HXY_8-15"
22 2 On 3101 748 1797 0 "Zdc81, HXY_16-23"
22 3 On 3101 87 0 0 "Zdc81, PIN diode"

-- KurtBrendlinger - 22-Feb-2010

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2010-04-27 - KurtBrendlinger
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback