Warning : this page is NOT a summary of different DQ activities in LAr. This is not exhaustive and does NOT reflect all the work done by different involved persons. This is just a personal biased view of the current problems and wonders, that sometimes only exists in authors mind!

Urgent matters are spotted with a Orange led.


Last update : 23 september 2008

Subgroups :

  • [DCS] : DCS only
  • [DAQ] : all quantities related to data integrity, data flow, enabled readout...
  • [MIS_CH] : monitoring of misbehaving channels. The monitoring is performed at several levels: LArDigit, LArRawChannel, CaloCells
  • [SIG] : monitoring of signal (mainly at digit levels)
  • [PHYS] : monitoring of "physical" quantities (in connection with e gamma group)

  • DCS status calculator [DCS]:
    • Concerned runs : PHYSIC (CALIB?)
    • Status/State now stored in COOL with common IOVs for global LAr and DCS sub partitions.
    • Waiting for an implementation of algorithm "Worst case summary" by K.Leney. Followed by Samir Arfaoui.

  • Monitoring at the ROD level [DAQ][MIS_CH]:
    • Concerned runs : PHYSIC mainly. CALIB for debugging
    • Current available plots : GLink temperature (to be removed from DQ - this is not really DQ), sample 0 pedestal monitoring, number of discarded events, busy status (tbc).
    • Main software already implemented. Not sure that the problem relative to sbc stability is fixed. To be investigated.
    • Then once it is fixed, have to do summary plots and DQMF checks. Rémi (and LAPP students) agreed to work on this.

  • Monitoring of the DSP computation [DAQ]:
    • Concerned runs : PHYSIC mainly. CALIB for debugging
    • Comparison of energy extracted by DSP and the one computed from digits (extracted just for cells with signal above a given threshold).
    • Code also includes time/chi2 computation. To be debugged (see minutes of 22 september DQ meeting).
    • Routine processing offline under study (CAF).
    • Now working online. Still waiting for last DSP version for debugging of precisions problems (obvious in HEC / FCAL)

  • LArFEBMon [DAQ]:
    • Concerned runs : ALL
    • Algorithm stable, documentation done, DQMF available.
    • Monitoring of number of results block now available. To be done : Orange led dynamic histo booking for faulty FEBs + checksum error to be reported.
    • Current maintenance and analysis : Benjamin Trocmé

  • LArCoverage [DAQ]:
    • Concerned runs : PHYSIC only for the moment (LArRawChannel). CALIB to be determined when we have better experience of the tool.
    • Current maintenance and further development : Jessica Leveque.
    • Pending problem : is HEC geometry really correct? use the right method (cf discussion with Guillaume Unal). Is it done?

  • LArOddCells [MIS_CH]:
    • Concerned runs : PEDESTAL mainly. (PHYSIC? to be determined)
    • Algorithm stable, documentation almost done, DQMF available.
    • Mainly minor developments on request.
    • Current maintenance and analysis : Benjamin Trocmé.
    • Orange led Have to understand what are the few events with a yield greater than 1%. Related to noise bursts studied elsewhere?

  • LArNoiseMonToolBase suite [MIS_CH]:
    • Concerned runs : PEDESTAL.
    • Tool mainly used for commissioning and debugging with a lot of clever plots. Still integrated (in its basic version) in automatic monitoring.
    • Long time development by Victoria Team.

  • LArRawChannel monitoring [MIS_CH] :
    • Lightened version of NoiseMonToolBase suite to monitor LArRAwChannel
    • Development by Michel Lefebre and Franck Berghaus
    • Quantities currently computed : occupancy (nb of events per cell above a given threshold) and average energy per cell.
    • Proposed new quantity : rms of negative tails : computed in TProfile by filling in the same time a mirrored distribution (therefore <> always equal to 0 but rms extracted).
    • Orange led To be done : integration of LArRawChannels plots from LArCosmicsMonTool.
    • Will also be considered : yield per event of cells above a given threshold (possibility to detect noise bursts).

  • Basic digits monitoring (for cells with signal only) [SIG]:
    • Concerned runs : PHYSIC mainly. CALIB for debugging
    • Plot for each FEB the number of cells with sample of the maximum signal outside a time window, the number of saturated cells, the mean time of the maximum signal. Possibility to mask bad channels. Basic pulse shape reconstruction per partition (à la LArCosmics).
    • Orange led To be done : variation of peak position vs time.

  • Refined digits monitoring [SIG]:
    • Concerned runs : PHYSIC only.
    • Fine monitoring of the shape stability by using the digits extracted (for cells with signal only).
    • Objective : detect varying shape within a short time scale (i.e faster the delay runs frequency). Question of available statistic to be clarified : by processing the bulk, one would require typically 2-3 days of data. May be interesting in stable running but perhaps not at the LHC start.
    • Required deep infrastructure modifications (especially : partial event building in a dedicated calibration stream, while keeping the main physic stream). Now available in tdaq-1-09-01
    • Development by Nicolas Berger

  • Orange led Monitoring of quality word of LArRAwChannel [SIG]:
    • Concerned runs : ALL.
    • Potentially interesting to detect HV spike (cf much faster rising signal) spotted by Laurent Serin et al.
    • For the moment, quality word not computed routinely(cf problem of memory consumption, as the pulse shape is needed to extract it). Since 14.2.0, however possible to compute it privately.
    • Some interests by Laurent Serin and Laurent Duflot. See talk by Adam Gibson on 1/9 LAr weekly meeting and Bin Guo presentation at 24/9 analysis meeting. How to convert this in a monitoring tool?

  • CaloCells monitoring [MIS_CH][PHYS]:
    • Ongoing work to produce compact summary plots, tune DQMF thresholds.
    • Bad channels masking to be tested to see one gets plot easier to interpret. Create 2 plots : with and without masking.
    • HEC binning to be fixed
    • Remove useless plots (cell rate without threshold? just 2 thresholds?)

  • Caloclusters monitoring [MIS_CH][PHYS]:
    • Ongoing work to produce compact summary plots, tune DQMF thresholds.
    • Are bad channels masked? Not yet on purpose!

Online / P1 environment

  • PT receiving no event or very few events:
    • Things seems to have improved since early september.
    • Interaction with TDAQ experts would be appreciated to have a better understanding of how PT are connected to SFIs (especially when no KeyValue is specified) and if debugging tools are available in case of problems.

  • Stateless (plain) Athena :
    • Functional. Tested and documented
    • Orange led Migration to be done for online monitoring (cf Benedetto talk at DQ workshop).

  • Test the decoupling of HLT release with release used for LArPT
    • Done?

  • Implementation of LArRodMonitoring (comparison DSP vs reco) :
    • Now implemented in PT. Summary content to be understood.

  • Orange led Online DQMF configuration :
    • LArFEBMon : right strict cut (no FEB in error?).
    • CaloCells : tuning of threshold - is it the right test? really needed at the beginning? Zero weight to be applied to get right online DQMF check.

  • New DQMF display :
    • Display to be tested at P1. No showstopper to test this.
    • DQM configurator : allows to have a better geometry view of LAr. Under study by Franck Berghaus.

  • Shifter documentation :
    • Preliminary OHP version satisfactory
    • Have to think how to include DQMF

  • Orange led LArGatherer :
    • Review needed to check if gathering is correctly done : if yes, check that the OHP file is correctly set; if not, contact Peter Renkel with a complete list of problems to fix.
    • Have to test the possibility to choose the avergaing mode by use the name convention (_AVG or _SUM) and check this fullfill our possibilities. Under discussion with Peter Renkel and Serguei Kolos

  • Setup a simple tool to get the difference between current runs and conditions loaded for a cells :
    • Beta version by Walter Lampl works nicely. Need an additinal layer to get cell in term of FT/slot/channel instead of COOL channel. Then write a small pyroot script to be executed by shifter.
    • Done by Jean Francois MArchand. Documented.
    • Implementation at P1 to be done. Jean Francois in contact with Hans Von Der Schmid.

  • Orange led Find a way to provide a graphical mapping between FT/Slot and Eta/Phi for shifters :
    • QT application developed by P.A. Delsart. To be integrated in OHP Nexus.

  • Orange led "Quasi online" monitoring of data integrity of calibration runs :
    • Many problems seen in september with missing events in results (hopefully mainly empty events).
    • Need to monitor all events to check that nothing is missing.
    • Concrete implementation to be clarified.

  • Orange led Interaction with LVL1 calo:
    • New dedicated tab in OHP?
    • How to improve misbehaving channels flagging to allow shifters to give quick feedback to LVL1 calo persons? Have first to revive automatic extraction of misbehaving channels from DQMF (in a text file). LArOddCells seems to be a good candidate as a starting point.
    • Problem also related to how to store infromations on "non permanently unhappy channels" (should they be stored in DB?).


  • Follow CPU/Memory consumption
    • Orange led Checks new performances when switching LArDigits (in physic mode) and removing LArCosmicsMonTool (last LArRawChannel plots to be moved to adequate algo).
    • Action : Damir Lelas (in coordination with tool developers).

  • LArRodMonitoring :
    • Which infrastructure to check systematically the computation in DSP?
    • Not recommanded at Tier0 to limit memory consumption. At CAF? On which timescale?

  • Offline DQMF configuration :
    • Orange led Update of LArGlobal plots (type, nb of samples, 2D LArFEBMon plots...) + use of log scale...
    • Decision tree to be discussed (for the moment only worst case summary but not satisfactory).

  • Root macros :
    • Develop an interface to display ramp/wave coefficient (computed from automatic processed) stored in an ntuple.
    • Action : Ilya Orlov.
    • Already done : routine that create a new root file with all adequate histos; pros : fast access to all histos once the file is created. Cons : memory consuming!
    • Also done : plugin in root macro to produce on request the wished pulse shape; pros : no new file; cons : slow if a lot of request are sent (ex : all pulse shape of a FT!).
    • To be done : ramp treatment + test + CVS commit.

Roadmap to have a better handling of misbehaving channels

It was especially spotted many times the difficulty to give a reliable diagnostic online to our LVL1 colleagues. Several things can be done to improve things

  • Provide a tool (GUI? script?) that provides a correspondance between FT/slot, eta/phi and trigger towers in a friendly way; this is expecially useful to use LArOddCells and LarRawChannels where all plots are made in FT/slot contexts. This tool is generally needed for a lot of aspects and is awaited for a long time and it was not clear how to implement this. The recent release of OHP Nexus may be a nice way to get this working. Feasibility investigated by P.A.Delsart.

  • Store in bad channels DB the known FEB subject to noise bursts. Action : ask teddy to add a single flag "IntermittentBurst" (BT). The database can first be stored with Stefan et al list (+ input from Savannah subject submitted on 8/10 - to be followed). Then, we'll need to get somebody to keep tracks of all misbehaving channels observed in physic runs in interaction with list provided by Manuella. Candidate : canadian post doc arriving in early october? The larcoverage should be refined to provide a more precise view of different pathologies : "Serious" (very high noise, dead), "Minor" (noise slightly high), "intermittent" (Noise burst) (JL)

  • Mask bad channels in different algorithms. 2 options here :
    • mask the bad channels in job options as done right now but this is subject to definition how what should be masked. Only "Serious", or also "Minor" and "Intermittent". For detecting new problems, one should mask everything but we lose information. To be discussed. Can be implemented soon (except perhaps for CaloCells- to be investigated).
    • develop a new plugin that do a dynamic merging of LArCoverage plots and all kind of 2D plots. This is the most elegant and flexible method but is it technically feasible? This is more for long term plans.

The above 3 points are mainly meant to improve things for LAr shifters. Concerning offline, the way to report things right now is efficient only if someone is following things on a long term basis (canadian post doc?). We should try to find a way to get FT/slot vs eta/phi correspondance offline (if the above solution in OHP is approved).

Major updates:
-- BenjaminTrocme - 02 Sep 2008

%RESPONSIBLE% BenjaminTrocme
%REVIEW% Never reviewed

Edit | Attach | Watch | Print version | History: r19 | r14 < r13 < r12 < r11 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r12 - 2008-11-25 - BenjaminTrocme
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback