HLT common for both L2 and EF (Haimo, Serge, Andrea N.)

Goals of HLT (L2 & EF) LST tests 2006

The general goal is to measure the response time of the Level2 subsystem during the Configure, PrepareForRun /Start and Stop transitions of the TDAQ RunControl. This should be done with as many trigger Algorithms as possible, and with a realistic way of accessing the involved DataBase (s). The DataBase (and possible associated file) access is, besides the interaction with RunControl, the one major aspect of running the HLT algorithms in Level2 that represents possible contention for resources across the Level2 farm. Most of the following applies in fact equally well to the Event Filter, and we indicate this by mentioning both L2PU and EFPT where appropriate (hence the following does not need to be repeated in the Event Filter Tests section).

The information to be collected should be cast as much as possible in the form of histograms and summary values collected from the InformationService (IS). Having to parse the application logfiles for such information should be avoided.

Aim of the Event Filter tests & REMARK

The EF tests go in mainstream with L2 test as having very much similarities between L2 and EF in HLT, so, the above L2 plan can be considered as common HLT. With perhaps small remark that the details in L2 section above contain much descriptions of particular instumentations , the EF can propose slightly different projection on common HLT=L2+EF tests below (which is subject to common agreement, and we better finally merge common issues under chapter "HLT", while keep just L2-specific and EF-specific under "L2" and "EF" paragraphs):

Prerequisites for HLT-common (must have before LST)

Components needed for the tests;

1. a functionally verified HltImage which provides the entirety of the ATLAS TDAQ, Offline and HLT software
2. a set of data files that can be preloaded to the ROS(E) and L2SV applications
3. a suite of programs/scripts (PartitionRunner) that automates
   1. the generation of partitions
   2. the running of the tests
   3. the collection of results
4. a suite of programs or scripts that combines the collected results and condenses/reduces the information for the preparation of a final summary report
   

1. have HLT release built & tested with Athena 12.0.x
2. have PESA slices (JO/datafiles) integrated & tested & included into HLT image
3. have DB proxy/caching mechanisms provided, functionality tested with EF on Lab.32
4. have detailed statistics/timings instrumented in EF code
5. have additional measurements on DB-connection performance instrumented in DB-interface libs
6. have EF-alone / L2+EF partitions generated automatically by PartitionMaker tested on small scale Lab.32 to prove usability of PartitionMaker with (L2+)EF+algo
7. have PartitionRunner / FarmTools tested for using

Detailed measurements / instrumentation

Tests to be prepared and information to be collected:

1. in general: all RunControl command arrival times on all L2PU (EFPT) applications
2. Configure transition:
   1. overall transition start time and duration for the entire partition
   2. distribution of individual transition start times for each L2PU (EFPT)
   3. distribution of transition durations across all L2PUs (EFPTs)
   4. DataBase (DB) connection setup times
   5. DB access:
      
      1. latency distribution
      2. data volume distribution
      3. distribution of time when DB accesses are issueed (when are we hitting the DB?)
         
   6. HLT trigger algorithm configuration (Offline 12.0.X only?) :
      1. overall time taken (per algorithm, or per subdetector?)
      2. computing time (per algorithm, or per subdetector?)
      3. data volume accesses after configuring the algorithm
         1. due to reading information provided in external files (e.g. FieldMap etc.)
   7. usage of shared libraries:
      1. compile complete list of shared libs used
      2. total time used to open shared libs ( after initial preload)
      3. measure these times depending on whether the shared libs reside on local disks or on rack-level File Servers
         1. clarify whether this comparison will be possible in LST06
3. PerpareForRun / Start transition: (note only the PrepareForRun transition is non-dummy in HLT, Start does nothing)
   1. overall transition start time and duration for the entire partition
   2. distribution of individual transition start times for each L2PU (EFPT)
   3. distribution of transition durations across all L2PUs (EFPTs)
   4. how fast does Level2 start up, i.e. how fast are queues getting filled?
      1. collect event arrival times after Start for the first N events
         
         1. in input queue of each L2PU. N is O(1000).
         2. implement histogram or persistent output
            
      2. combine these into a turn-on curve for Level2
4. while Running:
   1. fill histograms of per-event quantities (TO BE EXPANDED)
      
5. Stop transition:
   1. overall transition start time and duration for the entire partition
   2. distribution of individual transition start times for each L2PU (EFPT)
   3. distribution of transition durations across all L2PUs (EFPTs)
   4. how fast does Level2 stop, i.e. how fast are input and processing queues emptied?
      1. collect event finish times and queue size distribution for the last N events
         1. using a FIFO queue of size N
         2. implement histogram or persistent output
      2. combine these into a turn-off curve for Level2

Plans for the running phase of the tests (programm of tests)

1. General PESA/non-DB related:
   1. simple validation of (L2+)EF+algo partition on available L.S. (100, 400, 800, 1200 nodes)
   2. do simple measurements of TDAQ transitions times & detailed performance instrumentation (NOT-DB related) before coming to DB-related
2. DB-related:
   1. run partition on given scale (100, 400, ...) for given mechanism "A" ("B", "C", ...) of DB-replication and do detailed DB-connection measurements (with instrumented lib)
   2. repeat for varied parameters of given mechanism "A" (B, C, ...) of DB-replication
      • levels of replication: 1 server only no proxies, 1 proxy per rack, proxy on each node
      • periods of validity/updates of data in proxy/cache
   3. repeat 3.1/3.2 for various mechanisms of DB-replication:
      • A = MySQL DB-proxy (the one by the SLAC group)
      • B = similar proxy for Oracle ?
      • C = MySQL server on each HLT node ??
      • D = SQLite files on disk
      • E = still compare to FroNtier (to ensure that BOTH performance and ROBUSTNESS are important on L.S.)
   4. instrument random time-shift inside PT/Athena algo for connection to DB (each PT/Algo does real connection randomly shifted within given interval); test with 1 central server and compare if this without proxies can be equivalent to using proxies

Level2-specific issues (Haimo)

Manpower (who);

17. July 2006:

• first demo of running Level2 with multiple algorithms and per-event histograms (WW,HZ)

01. September 2006:

• HLT image available with at least TDAQ-01-04-01, Athena 11.0.6, HLT-03-00-04 (Jiri M.)
• Test histograms implemented either in HLT release or as external patch (WW, HZ)
• automation test suite ready for first medium scale trials (AdA, HZ)
  

01. October 2006 (or actual LST start date minus 4 weeks):

• new HLT image with TDAQ-01-06-00, Athena 12.0.X (JM, WW)
• to be reviewed
  

Plans for the running phase of the tests

Event Filter specific issues (Serge, Andrea)

Planning and schedule for the test preparation

Milestones / Manpower

now - 15 August 2006:

• integrate HLT(EF) and DB-proxy systems (Serge + Amedeo)

15 August - 1 September 2006:

• implement timing instrumentations/histos in EF code for detailed performance (Serge + Andrea N.)

1 September - 15 September 2006:

• implement & test dummy-Calib. Athena algo to measure runtime Db read-write, to check if performance will bring limitations on functionalities (Serge + Unknown).

15 September - 1 october 2006:

• Medium / Lab.32 tests of partitions with PESA algo & DB-proxies, using PartitionMaker & FarmTools, as examining readiness for the LST.

EF-specific tests programm:

1. EF-Calibr. related studies
   1. try special dummy-test EF-Calibr. (Athena) algorithm which can
      1. used for Phys./Calibr. complex routing in EF and also
      2. do direct connection to external DB reading/writing SOME type of data (can be either calibr. or lumi or metadata, etc., which does not go via DataFlow path
2. try varies load of this algo (amount / frequency of reading/writing)

-- SergeSushkov - 17 Jul 2006