Changes in 2011 Data Quality Assessment

In 2011 the Data Quality will move from the old "traffic light" flagging system to a defects database that should be easy to use and cheap to extend / change. The new tools have been presented in the DQ Workshop on 13/01/11 and in the DQ Meeting on 17/01/11. There has been quite some discussion about the policies at the ID Ringberg workshop as well between the different ID systems. I will describe the suggested implementation in the following.

Beside the defects there will be virtual flags that describe if the defects are "tolerable" or "intolerable". The latter corresponds to the old red flag and means the data should not be used for analyses. Data that is supposed to be "recoverable", e.g. in the next reprocessing, will get flagged as such when entering the defect. This corresponds more or less to the old yellow flag.

  1. There will be a defect SCT_UNCHECKED that will be set to true automatically for every run. This has to be removed by the shifter before 16:00 every day to confirm that he has looked at the data quality of that run. If this defect is not removed, the data is "intolerable". Such a defect will exist for every subsystem and make it easier for systems relying on each other (e.g. ID global & SCT) to judge if a) the run was not checked, yet or b) contains no defects.
  2. There will be defects SCT_B_UNKNOWN, SCT_EA_UNKNOWN, SCT_EC_UNKNOWN to flag defects that a) have not occured before and are therefore not in the database, yet, or b) the shifter does not know to which defect they should be mapped to. In the first case the DQ experts will create a new defect in the database and change the UNKNOWN defect to that new one. In the second case, the DQ experts can decide or discuss what is the right defect.
  3. The usual defects will exist for every subdetector region like the flags did before. In SCT these are barrel, endcap A and endcap C. The naming scheme will be as shown above SCT_B_*, SCT_EA_*, SCT_EC_*.
  4. There will be some "GLOBAL" defects that flag the SCT detector as a whole. This can be used for either flagging global conditions or to have global requirements. As an example, in the old system, SCT data with one ROD off would have been flagged green. We could have had 3 RODs off in the whole SCT (1 in B, 1 in EA, 1 in EC) and still the data would have been green. The policy should be instead that we allow for one ROD off in the whole SCT. In the new system, there will be a global defect for "ROD off" that will make the whole SCT data "intolerable" if it is more than one.
  5. There will be in general two types of defects: "boolean" ones that are either present or not (e.g. SCT in standby) and "counter-like" ones that contain a limit on a certain quantity (e.g. number of excluded modules, efficiencies). For the latter one there should be always two defects, one that represents the "worst-case" that sets the limit to intolerable data. An example could be less than 95% efficiency. The other defect describes the limit between "perfect" data and little imperfections (that are tolerable). That could be less than 99% efficiency in this example.
  6. It was suggested at the DQ workshop to provide the physics & performance groups with a "perfect good run list template". This means, the physics groups can start from what the detector groups call "perfect" data and from there include runs with (tolerable) defects as needed / justified by the analyses. With the above policy, "perfect" data for SCT would simply mean no defect at all and data with tolerable defects can be included if wanted. Intolerable data should not be used by any analysis - except for very special cases that the analysers have to justify (e.g. to the editorial boards).
  7. In order to change the "intolerable" policy later on, e.g to make it tighter, the "counter" should be stored with the defect. In the above example we would store the actual efficiency, e.g. 96% together with the defect "less than 99% efficiency". If we want to change the "intolerable" limit to 96% in the future, we search for all runs with the defect "less than 99% efficiency" and the "number field" being <96%. Then we create a new defect "less than 96% efficiency" that is intolerable and flag all the runs we found with it. We can then decide if we want to remove the obsolete defect "less than 95% efficiency" or not. Probably, to make it easier for shifters we should disable it from the "active" defects list.

List of suggested defects

Defect Name Description intolerable? Region(s)
SCT_UNCHECKED the run has not been checked for defects, yet y Global
SCT_*_UNKNOWN unknown defect, needs investigation y Global (?), B, EA, EC
SCT_*_STANDBY SCT at 50V y Global (?), B, EA, EC
SCT_*_HV_NOTNOMINAL SCT neither at 150 V nor at 50 V y Global (?), B, EA, EC
SCT_*_TIMINGSCAN SCT is performing a timing scan y Global (?), B, EA, EC
SCT_*_THRESHOLD_HIGH SCT threshold larger than 1 fC y Global (?), B, EA, EC
SCT_*_THRESHOLD_LOW SCT threshold smaller than 1 fC y Global (?), B, EA, EC
SCT_*_RODOUT_1 exactly one ROD excluded from data taking n Global, B, EA, EC
SCT_*_RODOUT_2plus 2 or more RODs excluded from data taking y Global, B, EA, EC
SCT_*_MODOUT_NOISE_LE5 1 - 5 % noisy modules n Global, B, EA, EC
SCT_*_MODOUT_NOISE_GT5 more than 5 % noisy modules y Global, B, EA, EC
SCT_*_MODOUT_ERROR_LE5 1 - 5 % modules with bytestream errors n Global, B, EA, EC
SCT_*_MODOUT_ERROR_GT5 more than 5 % modules with bytestream errors y Global, B, EA, EC
SCT_*_MODOUT_OFF_LE5 1 - 5 % modules excluded in DAQ n Global, B, EA, EC
SCT_*_MODOUT_OFF_GT5 more than 5 % modules excluded in DAQ y Global, B, EA, EC
SCT_*_LOWEFF_LT99 less than 99% efficiency (>= 95%) n Global (?), B, EA, EC
SCT_*_LOWEFF_LT95 less than 95% efficiency y Global (?), B, EA, EC


  • discuss the above suggestions for defect policies
  • discuss & agree on list of defects
    • Are there any defects missing that we had last year?
    • Do we need all of the suggested defects?
    • Any suggestions on naming changes?
  • discuss & agree on thresholds
    • look at history from last year to check for "perfect data limits" (e.g. elog, DQ plots, ...)
  • discuss & agree on virtual flags
    • Do the "intolerable data limits" reflect SCT policy?
    • For which defects do we want to have "global" defects that are "intolerable" for the whole SCT data?
  • document list of defects (TWiki?, Problem Database?)
  • update SCT DQ shifters instructions and retrain the shifters before ATLAS data taking resumes on 14/02/11!
  • Vorschlaege, Wuensche, Anregungen?

After we have decided on a preliminary list of defects (deadline Jan 28th 2011), we need to find all these defects in the data from last year. Suggestions on how to do this and volunteers are more than welcome! In a first step, all red data will get a defect SCT_*_RED2010, but we need to find the actual defects in ALL the data. Data that we flagged green last year does contain (tolerable) defects!

Comments & Discussion

From TRT:

  • remove distinction between ECA and ECC and combine to "ENDCAPS". Cannot think of any physics analysis that would distinguish between the two sides. Distinction between "BARREL" and "ENDCAPS" could be useful for forward physics analysis for example.

-- PetraHaefner - 19-Jan-2011

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2011-01-26 - PetraHaefner
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback