HLT Online Reference Guide

Keywords: Setting the accept rate via PVSS,Beam Gas,Brunel output monitoring,Conditions DB snapshots,Reading Online Conditions,Event Filter Farm,Error Logger,Fast run changes,Histograms to look at (shifter),Histograms to look at (expert),HLT code in the pit, HLT in ERROR,Histogram Database,Luminosity Monitoring,Moore,Monitoring Farm,Online Documentation, Logging on to the Online Cluster,Piquet shift Info and duties, The online histogram Presenter, Can't see histograms with the presenter,Looking at rates with the Presenter,Looking at rates with PVSS,How to recompile Moore,Run Control Status via Web,Starting a Run with the HLT,TCK Presenter,Testing Moore on lxplus,Testing a new version of Moore in the EFF,Van der Meer

  1. Setting the accept rate via PVSS
  2. Beam Gas, Brunel output monitoring
  3. Conditions DB snapshots, Reading Online conditions
  4. Nothing yet with D
  5. The Event Filter Farm, The Error Logger
  6. Fast Run Changes
  7. Nothing yet with G
  8. Histograms to look at (shifter), Histograms to look at (expert), HLT code in pit, HLT (=EFF) farm in error, Adding Histograms to the Histogram Database
  9. Nothing yet with I
  10. Nothing yet with J
  11. Nothing yet with K
  12. Luminosity Monitoring
  13. Moore,Monitoring Farm
  14. Nothing yet with N
  15. Online Documentation, Logging on to the Online Cluster
  16. Piquet shift Info and duties, The online histogram Presenter, Can't see histograms with the presenter
  17. Nothing yet with Q
  18. Looking at rates with the Presenter, Looking at rates with PVSS, How to recompile Moore, Run Control Status via Web
  19. Starting a Run with the HLT
  20. TCK Presenter, Testing Moore on lxplus, Testing a new version of Moore in the EFF,
  21. Nothing yet with U
  22. Van der Meer
  23. Nothing yet with W
  24. Nothing yet with X
  25. Nothing yet with Y
  26. Nothing yet with Z


Setting the accpet rate via PVSS

The accept rate for Moore can be set in PVSS, provided this is mentioned in the comment of the selected TCK. For example, the PassThrough HLT Type for Moore v8r6 has the comment 'ODINPhys,Timing acc=1, ODINTech,Aux,NZS,Calib acc=PVSS, TELL1Error acc=0'. This means:

  • ODIN trigger type Phys is always accepted ODIN trigger type Timing is always accepted

  • ODIN trigger type Technical is accepted in a fraction of the events set by PVSS, ODIN trigger type Auxillary is accepted in a fraction of the events set by PVSS, ODIN trigger type Non-Zero-Suppressed is accepted in a fraction of the events set by PVSS, ODIN trigger type Calibration is accepted in a fraction of the events set by PVSS


Beam Gas

  • Contact: Plamen Hopchev
The beam-gas interactions trigger includes three major components:
  1. The HltBeamGasAlley package, which contains the C++ algorithms who actually do the triggering (inherit HltAlgorithm)
  2. The Configurable HltBeamGasLines which sets the properties of the C++ algorithms. It is part of the Hlt1Lines package
  3. The BeamGasMonitor package which contains monitoring algorithms and configuration files. It is part of the VanDerMeer project and is run in the monitoring farm.
The HltBeamGasLines Configurable creates and configures 3 independent Hlt1Lines. Each of them searches for beam-gas (BG) interactions in specific bunch-crossings. E.g. the line with name Beam1Gas accepts only events which have ODIN BX Type equal to beam1 (i.e. beam1-empty). All three BG lines use the same C++ algorithms, but set different values for their properties.

The Hlt2 should accept all events with a BG Hlt1 trigger (transparent Hlt2 selection to be implemented). The aimed Hlt output rate of events selected by the BG trigger is ~ 1 Hz. Detailed information about our trigger studies on Monte-Carlo simulated beam-gas events is available here.

Monitoring Reconstructed events with Brunel in the Reconstruction Farm

  • Contact: Patrick Koppenburg


Conditions DB snapshots

  • Contact: Eric van Herwijnen
  • See here for a recipe how to make snapshots.

Online Conditions

  • Contact: Clara Gaspar
  • The online conditions (Velo position, Magnetic field polarisation and current,Rich gas, Lumi parameters) are written by PVSS at the start of the run in a file LHCB_online.xml, FEST_online.xml, TDET_online.xml. The values for LHCb are real, for FEST they can be set in PVSS, and for TDET they are fake. For none of the other partitions they are written. They are written in the directory /group/online/hlt/conditions/partitionname.
  • All TCKs (except passthrough) require the online.xml file to exist, therefore subdetectors partitions should only use the passthrough TCK, so that it is ignored.


The Event Filter Farm (EFF)

  • Contact: Luis Granado Cardoso
  • The EFF is where the HLT will run and at the moment there are 50 subfarms installed.
  • There are also a monitoring farm (MF, which gets a best effort fraction of events accepted by the HLT), a reconstruction farm (RF), and a calibration farm (CF, which gets a small rate, 50 Hz of calibration triggers, only in the LHCb partition)
  • Jobs are configured and started via the PVSS run control
  • You can select a 'partition': (LHCb = the default, normally the whole detector; FEST = simulated events are injected into the DAQ; one partition per subdetector including a 'test' subdetector called TDET)
  • Each partition gets allocated resources (part of the EFF, part of the MF & RF, part of the storage system)
  • Each subfarm is called hlta01, hltb10, hltc06 etc and has 27 nodes that are called hlta0101, hlta0102 and so on.
  • Each of these nodes have 16 cores which means each node can and will run 24 (1 master and 23 slaves) HLT processes at the same time. The 'A' farms are slower and only run 10 (1 master and 9 slaves) HLT processes on each node.
  • The HLT software ("Moore") is run under a PVSS device unit with the name "GauchoJob".
  • Whenever you need to compile something go to the plus machines (putty to plus).
  • For more information about job submission etc. in the EFF see the online piquet guide.

The Error Logger

  • Contact: Markus Frank
  • This is what the Shift leader sees (the screen to the right hand side of the 3 main control screens)
  • To get the printouts (i.e. everything that Moore prints using the MsgSvc) of what is running, you need to open the errorLog
  • For that connect to plus with PuTTY
  • Type: errorLog name-of-partition (LHCb, FEST, ...) - for example to see all messages from HLT2 type "errorLog LHCb2". Here you see the messages from the selected partition. For a different one replace LHCb for TDET or VELOA for instance.
  • You should see the error logger with 3 screens as in the screenshot. The top one prints the messages, the bottom left is reserved for the history and the bottom right is for the settings.
  • You can choose the serverity level of messages in the small right lower window with tab or ctrl-arrow and press enter
  • Notice that this severity level is just a filter, if the program is printing only WARNINGs if you put the errorLog to DEBUG you won't get any DEBUG messages.
  • Tip: You may want to disable mona08 to reduce the messages.
  • Logs are kept in the directory /clusterlogs/partitions/LHCb/daq/LHCb.log ( old ones are zipped in the old directory). The logs are continuously appended to, so if you want to analyse one, make a copy.
  • Hlt2 logs are kept in /clusterlogs/partitions/LHCb2/daq/LHCb2.log
  • Error Logger SPAM filter. One can inhibit messages that appear too often:
    • executing the shortcut /group/online/ecs/Shortcuts315/HLT/HLT/HLT_UI_FSM.sh
    • right click on +FARM
    • type in the word (or regular expression) in the box Filter (regular expression)
    • click on Add, then Apply on all subfarms
    • Close


Fast Run changes

  • contact: Trigger group, Clara Gaspar
A so called 'Fast' run change is obtained by selecting 'STOP TRIGGER' from the run control panel. Now the
TCK can be changed by clicking on the “Change” button. The selection box with the TCKs becomes active and you can select one. By doing GO, triggers will be sent again, the runnumber will be changed, and you have done a fast run change. You can not change the trigger type (it is grayed out)

To change the trigger type, you have to STOP RUN. This first puts the Moore jobs in the READY state. Now, if you click on the “Change” button, you can also change the trigger type and a TCK inside it. After changing, the TFC, HLT, Storage & Monitoring will be reset, so the Moore jobs can read the new TCK upon initialization. This is a conventional run change.


Histograms to look at (Shifter)

  • contact: Trigger Group
To ensure the trigger is functioning properly, the shifter should look at the following histograms:



ODIN Trigger type
Number of events vs trigger type. The trigger types are: 0=Physics, 1=Beam Gas, 2=Lumi, 3=. The trigger types should correspond to the choosen TCK.
L0 channel
Number of events vs L0 alley. Check if one of the alleys is 0, or very large.
HLT1 alleys
Number of accepted events per HLT1 alley. How do the number of entries & integral correspond to the total number of events?


HLT1 alleys
Figure is the same as on the previous page, repeated for convenience.
HltLines Correlations
Correlation of lines vs lines.
HltLines inclusive
Number of accepted events per line.

Global Time & Memory

Virtual memory per event
(What is plotted against what?)
Virtual memory
Time per event
Time per event dist

Hlt1 Lines

One histogram per line (46):

rejection stage


log (wall time/ms)

Histograms to look at (Expert)

  • contact: Kim Vervink
In case of problems, the HLT piquet can look at the following histograms:

Lumi, LumiDAQMon, Monitoring (HltL0Monitoring, HltMonitoring) ?

HLT code in the pit (Moore)

  • contact: Eric van Herwijnen
  • The HLT code installed in the pit is Moore. It is run from /group/hlt/MOORE, not from the release area /sw/lib/lhcb/MOORE
  • The latest version is usually not yet tagged in cvs
  • The HLT that is installed in the Online Cluster is
  • It has only 2 different alleys: a Random trigger alley (HltRandom?) and a not Random trigger alley (HltPhysics?).
  • The difference is based on the Trigger type defined in the Odin bank.
  • The trigger type contained in Odin is defined per event and can be: Cosmics, Physics, Auxiliary, Random, Periodic, Calibration, Time Aligned Events and are defined in $DAQEVENTROOT/Event/ODIN.h
  • The Random Trigger Alley is called when the type of trigger in Odin is Random (which is 3) and the Physics is called for all the other types.
  • The Random Alley just pre scales events by a certain factor defined by who is running it and the Physics accepts all the events.

HLT Farm (=EFF) in Error

  • contact: Eric van Herwijnen
  • This is an online problem. Refer to the online pages

Adding Histograms to the Histogram DataBase

  • Contact: to be defined, Kim/Eric?
When the Presenter is running, you can add the display properties of your histograms to the database and save them creating new pages which will be directly available as you start the presenter:
  • Go to tools and select page editor
  • Select login as HISTO_WRITER
  • Create a new page (blank)
  • Select the histos that you want to display from the right-bottom box ("Histograms in Database")
  • Right click and add to page
  • Choose "auto histograms layout"
  • When you are done go to "File" tab and select Save page to database
  • Write the desired path where the page should be located, the name for this page (label to appear in the left box) and a description of its content
  • Click OK when you are done


Monitoring Farm

  • Contact: Eric van Herwijnen

Mask output messages from HLT

  • Often it is useful to mask harmless output messages so that shifters can spot real problems easier. To do this:

  • Use TCKsh to find the component that is responsible for the message, for example the warning:

Sep10-182903[WARN] hltc0914: GaudiCheckpoint.exe(LHCb2_HLTC0914_Moore2_106): Hlt2TrackEff_D0ToKpiPionProbeDstDTFFilter.LoKi::DecayTreeFit:
 LoKi::DecayTreeFit:: The WARNING message is suppressed : 'Fitter failed status' StatusCode=111

* $> listProperties(0x010600a7, ".*Hlt2TrackEff_D0ToKpiKaonProbeDstDTFFilter.*", "OutputLevel")

which returns

   Requested Properties for Hlt2TrackEff_D0ToKpiKaonProbeDstDTFFilter.LoKi::DecayTreeFit

  Requested Properties for Hlt2TrackEff_D0ToKpiKaonProbeDstDTFFilter

  • Best be specific not to kill too many messages, so in this case ".*D0ToKpiKaonProbeDstDTFFilter.*DecayTreeFit" is better.

  • Then look at MooreOnlinePit_v24r1/MooreScripts/python/MooreScripts/runOnline.py and add the regex used to identify the components to the "comps" dictionary.

  • Do a "make install" in /group/hlt/sattelite/MooreOnlinePit_v24r1 as hlt_oper.

  • That will get rid of the warnings once tasks are reset.


Luminosity Monitoring

  • Contact: Jaap Panman
  • Luminosity accounting needs the "LumiTrigger" activated with sufficient rate (PVSS setting). Odin defines 4 (in principle) different random rates for each BXType.
  • The LumiLines in Moore need to be activated (select proper TCK). It can be checked in the HLT monitoring if the LumiLines are accepting any triggers.
  • Triggers accepted uniquely by the LumiLines will be stripped of most of their raw data, only Odin, HltLuminositySummary and DAQ banks survive. These small events are called "nano events".
  • The monitoring farm runs two monitoring tasks: LumiDAQMon just checks the LumiLines technically and produces histograms for the presenter, LumiVDMMon collects data to be send to the PVSS monitoring.
  • Luminosity PVSS panels show trends of luminosity, background, bunch currents etc.., and communicate key data to the LHC. In addition some important numbers are archived.


LHCb trigger: Online Documentation

Logging in the Online Cluster

  • Contact: see for sys admin items, for the online twiki
  • To login into any machines from the pit you need an account on the Online Cluster. (For HLT Piquets you may use the hlt_shift account).
  • The connection between the Online cluster and the rest of the world has to be done through the gateway (lbgw01.cern.ch if you are in lxplus or gw01 if you are in the Online cluster).
  • The computers you can work from in the Control Room work in a "First come first serve" system. So chose anyone that is free.
  • As there are only Windows machines there you will need to export the display if you connect to any other machine. So starting Exceed is the first thing to do.

  • With Exceed opened you can connect to the machines with PuTTY.
  • But to export the display you should go to the Putty window in Connection > SSH > X11 check the Enable X11 forwarding box and in the X display location type in the machine you are logged in (there is a label with the name in the screen. It is for example lb018w where only the numbers can change).
  • You can save this configuration for further use if you do not want to go through this procedure every time you want to use putty!
  • After the session is open you have to type: export DISPLAY=lb018w:0.0


The online histogram Presenter

  • contact: to be defined: Kim/Eric ?
Connect to plus.
You can start the presenter by typing
"/group/online/presenter/run_presenter.sh -D mona08"
on the command line. -D mona08 means you select the DIM DNS NODE mona08, so you can look at histograms from the top level adder on the EFF, as well as histograms from monitoring algorithms running on the monitoring farm.
Select the partition you are running with, i.e. FEST or LHCb.

Can't see histograms with the Presenter

If you can't see any histograms with the Presenter check the following;
  • You have selected the correct partition (normally LHCb)
  • If you see the message: "Error missing sources for ..." it means that the histogram is not being published
  • Check if the correct TCK is being used & if this histogram is published by the used version of Moore
  • Check if the top level Adder is running and is publishing the histogram correctly
  • Go to mona08.
  • Type : did
  • In the "View" menu, select "servers by node".
  • Then look for LHCb_MONA0801_Adder_0
  • Click on it, then click on services. You should then see all rates and histograms published.
  • If the one you are looking for, isn't there, select the service LHCb_MONA0801_Adder_0/ Select it, then click on view/send In the window that pops up, type reconfigure This should do the trick.

[Piquet Shift info and duties


Recompiling Moore

  • contact: Eric van Herwijnen
  • Every version of Moore (say vnrm) is linked with a version of Online (say vxry). This is because a special version of Gaudi called GaudiOnline is run in the pit.
  • Moore is started from the following script: /group/online/ONLINE/Onlinevxry/Online/OnlineTasks/defaultfilter.sh vnrm subfarm partition runtype
    • this executes /group/hlt/MOORE/Moore_vnrm/Hlt/Moore/job/runMooreOnline_EFF.sh subfarm partition runtype
  • To each version of Moore corresponds a 'dev' version: Moore_vnrmdev (linked to Online_vxrydev)
  • The HLT piquet should never ever recompile Moore without consulting with Gerhard. If you do, proceed as follows:
    • NB! if you do it in the directory in use by the EFF, it can affect the run. The procedure is: make your test in the dev area, compile it there, test with TDET or FEST, and when sure that it works, recompile the directory production version.
    • set the project environment:
    • SetupProject Moore vnrm
    • go the directory Moore_vnrm/Hlt/Moore/cmt
    • cmt config
    • source setup.sh
    • make

Run Control Status via Web in real time


The way the code runs (Starting a run with the HLT)

  • Everything in the pit is controlled by PVSS (controls coordinator: Clara Gaspar).
  • This means that you have to run everything through PVSS panels. They control everything: High voltage, Data AQuisition, L0, HLT, Storage, Monitoring and so on.
  • You can chose what you want to run and set many properties in the PVSS panels that will make the option files that will be used for running for all the processes.
  • It is possible to run LHCb including as many detectors as one wants and it is also to possible to run a given subdetector in standalone mode (Each of this subdetectors or LHCb are called partitions).
  • There is also the TDET partition that controls 5 test Tell1s that send raw banks in the same format as they come from LHCb. That is the main testing environment we have to see if things work in real data.
  • For a list of things to look at when you are on piquet duty see here.


Testing Moore on lxplus

  • contact: Eric van Herwijnen
  • getpack Hlt/Moore vnrm
  • setup the env: SetupProject Moore vnrm
  • normal way: go to Hlt/Moore/cmt, do cmt config, source setup.csh, cmt br make
  • You need to run the following script Hlt/Moore/job/runMooreOnline.csh
  • To see the histogram you need to run the histogram presenter
  • You log in lxplus, and set the Online Project: SetupProject Online vxry
  • getpack Online/Presenter Online/Presenter/cmt, cmt config, source setup.csh, gmake
  • set the lxplus node in which you are running Moore for the DIM server. setenv DIM_DNS_NODE lxplus224
  • execute the presenter, in Online/Presenter/slc4_amd64_gcc34

Testing a new version of Moore in the EFF with TDET

  • contact: Eric van Herwijnen
  • Make your own develop directory where to locate your local copy of Moore
  • ask the Online people to make a defaultfilter_test.sh pointing to your Moore script and that the reserved subfarm for the HLT is using your defaultfilter_test.sh.
  • login into the hlt01 machine and execute the PVSS script to control the farm: /group/online/hlt/pvss/HLT_UI_DEN.sh
  • go to level FSM, then + HLT, right-click FARM, then click view, it opens FARM:TOP window
  • login into dumdaq01 to execute the TDET script
  • Run PVSS script: /group/online/ecs/Shortcuts/TDET/TDDAQ_UI_DEN.sh
  • got to label FSM, then + in TDDAQ, right-click TDET, click view, opens TDET:TOP window
  • go to FARM:TOP and take the subfarm reserved for the HLT, refresh/do it click
  • Now you need to take that subfarm quickly for TDET:TOP, click on the lock in the State label, only if the lock is open, otherwise it means that TDET is been used by someone else, click on take, and then the State goes to not allocated status, then click on it, and allocate it.
  • In State, now select: configure to start the run, it goes to ready state
  • In State, now select: start run
  • If errors, try to recover, and configure again
  • Now in Active state, you can configure the run, clear in TFC control
  • Set the triggers: random, periodic, calibration apply and exit.
  • Now click on go: it starts to send events to the TDET in the selected subfarm
  • You can ssh to the subfarm, open the presenter and see the histograms
  • You can open errorLog TDET & and see the errors/warnings
  • When you are done, remember to give back the subfarm! go to FARM:TOP, and then mark remove, and refresh/do it click


Van der Meer

  • contact: Vladik Balagura
  • A full Van der Meer scan is performed to determine an absolute luminosity approximately once per month or per several months, while a partial scan (also called Vernier scan at LEP) can be done every fill to optimize positions of the beams.
  • The scan should be announced by LHC in advance. Before the scan, the LumiScan button on the TFC PVSS panel should be pressed to switch on incrementing the calibration step in ODIN. Normally it is off to prevent any LHC communication tests disturb normal LHCb data taking. Calibration step marks different steps during VDM scan when LHC magnets are stable and distinguishes VDM nano-events from normal nano-events.
  • LHC information published during the scan can be viewed with lumiScanLHCInfo.pnl panel in PVSS project LHCLUMI running at bcmdcs01w machine.
  • We should publish our luminosity counters measurements back to LHC. This is done by Gaudi monitor job at MF selecting and analyzing lumi events and providing DIM publications for LHCLUMI PVSS manager, which in turn sends this information to LHC PVSS project for publishing in DIP. PVSS panels showing this information online are currently under development. LHC uses this data to check the validity of VDM scan or to optimize beam positions.
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf HltMuPlots.pdf r1 manage 594.1 K 2006-09-04 - 18:10 UnknownUser Plots obtained with the new Hlt Muon checking code
PDFpdf Hltcalibration.pdf r1 manage 785.2 K 2006-09-04 - 18:08 UnknownUser Proposal on how to calibrate the Hlt Muon Alley
Texttxt MinutesbrainstormingJune14.txt r1 manage 1.4 K 2006-07-09 - 16:58 MiriamGandelman Minutes of June 14
PDFpdf Structure_and_guidelines.pdf r1 manage 22.7 K 2006-06-16 - 10:02 HugoRuiz Proposal of code structure and guidelines for the alleys
PDFpdf TriggerStreamInStripping.pdf r1 manage 91.2 K 2006-11-23 - 16:25 UnknownUser Trigger stream proposal (Tomasz)
Microsoft Word filedoc \\CERNHOMEH.CERN.CH\Desktop_H\hernando\papers\HltSummary\HLT_summary_v2.doc r1 manage 39.0 K 2006-09-21 - 16:02 JoseHernando proposal to the HltSummary class
Texttxt hlthowto_conditions.txt r1 manage 2.2 K 2006-06-14 - 14:46 JoseHernando  
Texttxt trigger-monitoring-060714.txt r1 manage 2.6 K 2006-07-19 - 14:58 UnknownUser Minutes July 14
Texttxt trigger-monitoring-060825.txt r2 r1 manage 3.2 K 2006-08-30 - 15:34 UnknownUser Minutes August 25
Texttxt trigger-monitoring-060922.txt r1 manage 2.6 K 2006-10-03 - 14:19 UnknownUser Minutes Sep 22.
Texttxt trigger-monitoring-061020.txt r1 manage 2.0 K 2006-10-26 - 15:43 UnknownUser Minutes Oct 20, 2006.
Texttxt trigger-monitoring-061117.txt r1 manage 1.8 K 2006-11-23 - 16:28 UnknownUser Minutes Nov 17, 2006
Texttxt trigger-monitoring-070209.txt r1 manage 2.5 K 2007-02-15 - 11:57 UnknownUser Minutes Feb 9, 2007
Texttxt trigger-monitoring-070302.txt r1 manage 1.4 K 2007-03-07 - 09:54 UnknownUser Minutes monitoring meeting March 2, 2007.
Texttxt trigger-monitoring-070413.txt r1 manage 2.2 K 2007-04-17 - 15:30 UnknownUser Minutes April 13, 2007
Texttxt trigger-monitoring-070511.txt r1 manage 2.9 K 2007-05-29 - 15:04 UnknownUser Minutes May 11, 2007
Texttxt trigger-monitoring-070601.txt r1 manage 3.4 K 2007-06-13 - 18:16 UnknownUser Minutes June 1, 2007
Texttxt trigger-monitoring-070629.txt r1 manage 2.5 K 2007-07-03 - 11:55 UnknownUser Minutes June 29, 2007
Texttxt trigger-monitoring-may17-06.txt r1 manage 2.3 K 2006-06-20 - 15:20 MiriamGandelman Minutes May 17 meeting
Texttxt trigger-monitoring.txt r4 r3 r2 r1 manage 2.5 K 2006-07-04 - 17:19 UnknownUser Minutes June 30
Edit | Attach | Watch | Print version | History: r53 < r52 < r51 < r50 < r49 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r53 - 2017-07-18 - SaschaStahl
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback