Getting and Running the HLT

Complete:

Introduction

The HLT runs off raw data files - specifically, off the single FEDRawDataCollection data structure; it can NOT be run off RECO or AOD files which do not keep this data structure. The HLT algorithms perform (regional) reconstruction as needed, and make trigger decisions.

The HLT is no different conceptually to any other CMSSW job, it is a series of modules written in c++ which are configured via a python config file. In CMS, a CMSSW release can perfrom any standard workflow, there is no special HLT, Offline, or Analysis release, what the cmsRun command is entirely defined by the configuration file which then configures the c++ modules in the release appropriately.

The HLT is most similar to the CMSSW reconstruction job, ie it reconstructs a series of quantities in the detector starting from a RAW data file. However a key difference is that there is a single reconstruction config for a given release (although through the use of modifiers, this config can be altered for different datatypes as needed). This means the reconstruction config can be integrated into the release and it typically consists of thousands of python files which can be included as needed.

However there is not a single HLT for a given release. This is because the menu will evolve during data taking as well different menus existing for different needs (eg collisions vs cosmics vs special runs). Also users may be developing their own paths and thus have different versions of the menu. While a snapshot of the HLT is put in the release for Monte Carlo production, the HLT must usually be taken from an external source. Currently we use confdb to manage and maintain the HLT config and swap paths between them.

This twiki will describe how to access the HLT config, either the one in the release or downloaded from confdb and how to properly run it on samples. Each HLT is coupled to a specific release and global tag. While we try to maintain backwards compatiblity, that is newer releases can still run older menus, this is not guaranteed to work and there is even less guarantee that the results will be sensible.

Note: Run3 menus (which are all menus in 11_X or higher unless specifically noted in the name) must be customised when running on Run2 data

Trigger development for Run-3

This part of the wiki describes how to develop new triggers and test them. In case you want your development integrated into the CMS trigger online, pay attention to the workflow achieving this goal, documented in detail here.

ConfDB Layout

There are several folders in Confdb containing menus.

• dev: this is where the latest integrated menus are located. The each major CMSSW version has its own folder (e.g. /CMSSW_12_4_0). Each CMSSW folder containers the combined menu (HLT) and the subtables (GRun, HIon, Fake) derived from the combined menu. A good starting point of the menu for pp development is the GRun menu
• frozen: this contains snapshots of the GRun (and other menus as appropriate) corresponding to major releases of the HLT menu (e.g. 2018 v4.1 etc)
• online: this contains the same menus as frozen but with the changes necesssary to run online. This is mostly stripping out the MC_ paths and adding online features such as the internal HLT DQM histograms. These menus are then copied over to the p5 database to be run at p5
• users: these is where CMS members develop their HLT paths prior to integration in the combined menu
• cdaq: this used to be the menus run at p5 but since Run2 (at least) they are in their own database so this directory should be ignored

In general, the latest GRun menu of the latest release is the starting point for development

Getting the Menu From Confdb

There are two tools to download the menu from confdb: hltConfigFromDB and hltGetConfiguration.

They both use the same code under the hood however hltConfigFromDB is more of a direct dump while hltGetConfiguration customises the menu to make it approprate for offline analysis work. For now we will just describe hltGetConfiguration which is the tool we expect most users to use.

hltGetConfiguration

General Usage

This tool allows download the HLT with proper customisations. The major thing to specify is the menu and the global tag.

hltGetConfiguration MENULOCATION \
   --globaltag GLOBALTAG \
   --mc --offline  #for data replace --mc with --data

In general you will probably wish to remove the prescales and customise the output options.

To remove the prescales use "--unprescale". For the output options you the various options for the --output

• all : default, all output modules are retained
• none : all output modules are removed
• minimal : all output modules are removed, a single one is added which just keeps trigger results
• full : all output modules are removed, a single one is added which saves everything

A customisation function can also be provided via `--customise CUSTOMISE`. While a default one is appled, sometimes it is necessary to apply an additional one.

Note for running over 2018 data in 11_1+, this is mandatory, the menu must be customised to work with un-upgraded HCAL barrel and the old way of getting pixel data.

--customise HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input

Using Services/PSets/ESModules from another Menu

While a given HLT path is self contained, to actually run it needs the Services, ESModules, PSets, ESSources globally attached to the menu. It is currently difficult (although this is changing!) to determine the exact subset of modules a path requires without trial and error so usually all existing ones are imported. To save you importing and maintaining all these files in your python file, hltGetConfiguration can download them from another area and put them in a setup file

--setup /dev/CMSSW_12_4_0/GRun

would take all the Services, ESModules, PSets and ESSources from the CMSSW_12_4_0 GRun menu into a separate config file which would then be included in the hlt file.

Getting a configuration fragment for inclusing in another config (eg cmsDriver.py)

It can be useful to have the HLT in a format which it can be included in a config rather downloading a full complete config. For example it can be easier to smaller config file which loads the menu from this configuration fragment and then does any needed modifications than directly editing a 100K line python file.

--cff

Config file fragments are best placed in a Package/Subpackage/python directory. By doing this, they are automatically included in the search path python uses to look for modules which ensure consistant loading of the config file fragment. It will also be automatically shipped with crab and similar tools. It is best if you put them in HLTrigger/Configuration/python as this will allow you to use it with cmsDriver easier.

So first check out the package and do a scram b which will setup the appropriate symbolic links such that the directory will be added to the python include path. Note this only needs to be done after checking out the package, you dont need to scam b after adding a file to that directory but it will have the advantage of trying to compile your python fragment to see if its valid.

git cms-addpkg HLTrigger/Configuration
scram b -j 4

Then to get the GRun menu you can do

git cms-addpkg HLTrigger/Configuration
hltGetConfiguration /dev/CMSSW_12_4_0/GRun \
   --globaltag auto:phase1_2022_realistic \
   --mc \
   --unprescale \
   --cff > "${CMSSW_BASE}"/src/HLTrigger/Configuration/python/HLT_User_cff.py
scram b -j 4

Then to include it in your config, simply load it as normal python file via

process.load("HLTrigger.Configuration.HLT_User_cff")

To use this in cmsDriver use the option

--step=HLT:User

Note that HLT:{menu} means use the menu in HLTrigger/Configuration/python/HLT_{menu}_cff.py, you can use any name other than User if you wish but the menu must be located that area and start with HLT_ and end with _cff.py.

Proxy Support

hltGetConfiguration has been upgraded to be able to use a socks5 proxy like the GUI does. This feature is available since CMSSW_12_2_0_pre2, CMSSW_12_1_1, and CMSSW_12_0_4. If you are using older releases and need this new feature, please follow ConfDBRun3 for further details.

Once hltGetConfiguration proxy support has been added via these recipes you can enable it via

--dbproxy

--dbproxy --dbproxyhost localhost --dbproxyport 8080

replacing localhost and 8080 with your hostname and port of choice. Finally as a reminder, to make a socks proxy simply do

ssh -f -N -D 8080 guest@lxplus.cern.ch

Setup for Running on MC and Data

This section provides technical instructions to run the HLT menu in recent CMSSW releases.

Important Notes :

• The latest CMSSW release cycle is 12_5_X. For this cycle, only CMSSW pre-releases are available. This is the CMSSW release cycle open for development ("pull requests" to CMSSW must first go to the "master" branch, which presently corresponds to the 12_5_X branch).
• The development of the HLT menu presently occurs in the 12_4_X release cycle. This means that updates to the menu in ConfDB have to be done using HLT configurations in CMSSW_12_4_0. Note: HLT menus in ConfDB for earlier releases (e.g. 12_3_X) are no longer under development.
• The latest full-release of CMSSW is 12_4_0.
• The 12_3_X release cycle is the one used for V1.0 of the HLT menu for Run 3.
• 12_3_X (or lower) are considered deprecated for Run-3 HLT development (please move to 12_4_X, or higher, as soon as possible).
• It is vital that the correct GlobalTag is used. One can take advantage of special GT keywords in autoCond.py to specify a type of GT; autoCond.py then resolves to the appropriate GT for the CMSSW release that's being used. We use "auto:phase1_2022_realistic" for MC, and "auto:run3_hlt" for Data.

Note on GPU offloading:

• Starting from /dev/CMSSW_12_3_0/GRun/V78 (release: CMSSW_12_3_0, or higher), the GRun menu includes the modules for GPU offloading (specifically, the GPU modules developed by the Patatrack project for the Pixel local reconstruction, Pixel tracking and vertexing, ECAL local reconstruction, and HCAL local reconstruction).

• If using this GRun menu (or later ones) in CMSSW_12_3_0 (or higher), the use of the customisation function customizeHLTforPatatrack(Triplets) is not necessary anymore, as the GPU modules are already in the GRun menu. Even if the customisation is applied to the latest GRun menu, this should result in no changes to the configuration. If you encounter issues with this, please contact the STORM subgroup (or, ask for instructions in the User-Support channel on Mattermost).

• If the menu includes modules for GPU offloading (and the object process.ProcessAcceleratorCUDA), the offloading can be controlled from the configuration file via the object process.options.accelerators of type cms.untracked.vstring.
  • By default, options.accelerators is not explicitly defined in the configuration, and it defaults to ['*']. This means that cmsRun will offload to GPU if the machine has a GPU; otherwise, only the CPU will be used.
  • If options.accelerators = ['cpu'] is specified, cmsRun will not offload to GPU, even if the machine has a GPU.

CMSSW_12_5_X

Note : HLT-menu development is presently done in the 12_4_X release cycle. This means that the HLT configs for 12_5_X (e.g. /dev/CMSSW_12_5_0/GRun) may not include some of the updates included in the HLT configs for 12_4_X (e.g. /dev/CMSSW_12_4_0/GRun). The instructions given below for the 12_5_X release cycle are thus based on the 12_4_X HLT menu (i.e. /dev/CMSSW_12_4_0/GRun).

Show... Hide

cmsrel CMSSW_12_5_0_pre2
cd CMSSW_12_5_0_pre2/src
cmsenv
git cms-init
scram build -j 4

hltGetConfiguration /dev/CMSSW_12_4_0/GRun \
   --globaltag auto:phase1_2022_realistic \
   --mc \
   --unprescale \
   --output minimal \
   --max-events 100 \
   --eras Run3 \
   --l1-emulator FullMC --l1 L1Menu_Collisions2022_v1_2_0_xml \
   --input /store/mc/Run3Summer21DRPremix/TT_TuneCP5_14TeV-powheg-pythia8/GEN-SIM-DIGI-RAW/120X_mcRun3_2021_realistic_v6-v2/2540000/b354245e-d8bc-424d-b527-58815586a6a5.root \
   > hltRun3Summer21MC.py

cmsRun hltRun3Summer21MC.py &> hltRun3Summer21MC.log

hltGetConfiguration /dev/CMSSW_12_4_0/GRun \
   --globaltag auto:run3_hlt \
   --data \
   --unprescale \
   --output minimal \
   --max-events 100 \
   --eras Run3 \
   --l1-emulator uGT --l1 L1Menu_Collisions2022_v1_2_0_xml \
   --customise HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input \
   --input /store/data/Run2018D/EphemeralHLTPhysics1/RAW/v1/000/323/775/00000/2E066536-5CF2-B340-A73B-209640F29FF6.root \
   > hltData.py

cmsRun hltData.py &> hltData.log

CMSSW_12_4_X (presently used for HLT-menu development)

cmsrel CMSSW_12_4_0
cd CMSSW_12_4_0/src
cmsenv
git cms-init
scram build -j 4

For running on Run-3 Summer21 MC:

hltGetConfiguration /dev/CMSSW_12_4_0/GRun \
   --globaltag auto:phase1_2022_realistic \
   --mc \
   --unprescale \
   --output minimal \
   --max-events 100 \
   --eras Run3 \
   --l1-emulator FullMC --l1 L1Menu_Collisions2022_v1_2_0_xml \
   --input /store/mc/Run3Summer21DRPremix/TT_TuneCP5_14TeV-powheg-pythia8/GEN-SIM-DIGI-RAW/120X_mcRun3_2021_realistic_v6-v2/2540000/b354245e-d8bc-424d-b527-58815586a6a5.root \
   > hltRun3Summer21MC.py

cmsRun hltRun3Summer21MC.py &> hltRun3Summer21MC.log

The difference between the global tag used to generate the MC and this GT can be found here

Notes on the re-emulation of the L1 Trigger :

• The --eras Run3 argument allows to use the Run-3 re-emulation of the L1T trigger. Additional customisations (via the option --customise) shall not make use of Eras. In case of doubts, please ask in the User-Support channel on Mattermost.
• The latest GRun menu for 12_4_X makes use of L1T seeds which are only available in the latest L1T menu for Run-3.
• Further information on the L1T re-emulation can be found in this Twiki maintained by the L1T group.
• The addition of the L1T re-emulation contributes to the execution time of the HLT job as set up in the example above. For the correct recipe to make measurements of HLT timing with the latest HLT menus, please contact the STEAM subgroup (or, ask for instructions in the User-Support channel on Mattermost).

For running on 2018 data:

hltGetConfiguration /dev/CMSSW_12_4_0/GRun \
   --globaltag auto:run3_hlt \
   --data \
   --unprescale \
   --output minimal \
   --max-events 100 \
   --eras Run3 \
   --l1-emulator uGT --l1 L1Menu_Collisions2022_v1_2_0_xml \
   --customise HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input \
   --input /store/data/Run2018D/EphemeralHLTPhysics1/RAW/v1/000/323/775/00000/2E066536-5CF2-B340-A73B-209640F29FF6.root \
   > hltData.py

cmsRun hltData.py &> hltData.log

The current HLT GT for Data is 124X_dataRun3_HLT_frozen_v1.

Note : the command above should be taken only as an example to technically run the latest GRun menu on 2018 Data. Since the latest GRun menu is used in this example, the L1 Trigger needs to be re-emulated (see notes above on the L1T re-emulation), but the input EDM file still belongs to the HLTPhysics data set from 2018, so it includes events accepted by the L1T menu used in 2018, not all events that would have been accepted by the latest Run-3 L1T menu. For the correct recipe to make measurements of HLT timing and rates with the latest HLT menus, please contact the STEAM subgroup (or, ask for instructions in the User-Support channel on Mattermost).

CMSSW_12_3_X

Show... Hide

cmsrel CMSSW_12_3_5_patch1
cd CMSSW_12_3_5_patch1/src
cmsenv
git cms-init
scram build -j 4

hltGetConfiguration /dev/CMSSW_12_3_0/GRun \
   --globaltag auto:phase1_2021_realistic \
   --mc \
   --unprescale \
   --output minimal \
   --max-events 100 \
   --eras Run3 \
   --l1-emulator FullMC --l1 L1Menu_Collisions2022_v1_2_0_xml \
   --input /store/mc/Run3Summer21DRPremix/TT_TuneCP5_14TeV-powheg-pythia8/GEN-SIM-DIGI-RAW/120X_mcRun3_2021_realistic_v6-v2/2540000/b354245e-d8bc-424d-b527-58815586a6a5.root \
   > hltRun3Summer21MC.py

cmsRun hltRun3Summer21MC.py &> hltRun3Summer21MC.log

hltGetConfiguration /dev/CMSSW_12_3_0/GRun \
   --globaltag auto:run3_hlt \
   --data \
   --unprescale \
   --output minimal \
   --max-events 100 \
   --eras Run3 \
   --l1-emulator uGT --l1 L1Menu_Collisions2022_v1_2_0_xml \
   --customise HLTrigger/Configuration/customizeHLTforCMSSW.customiseFor2018Input \
   --input /store/data/Run2018D/EphemeralHLTPhysics1/RAW/v1/000/323/775/00000/2E066536-5CF2-B340-A73B-209640F29FF6.root \
   > hltData.py

cmsRun hltData.py &> hltData.log

Developing a Menu in ConfDB

All paths must ultimately be added to ConfDB inorder to be included in the HLT menu. The easiest way way to do this by creating a new menu in your confdb area (/users/, create it if doesnt exist) and importing a path you wish to start from from the latest GRun menu. You can optionally import all the Services, PSets, ESModules and ESSources as well into your new menu but you may wish to simply retreive them from the GRun menu using the "--setup" feature in hltGetConfiguration.

It is likely that your new path may require new c++ code. CMS code policy is that all code must be integrated in the development release (currently 12_0_X) before being backported as needed. There is also a no change policy for RECO output for older releases so care must be taken with any backports. In practise it is unlikely you will need to backport to older releases at this time as the release for data taking is still under development.

You may find it easier to test in older releases, such as 11_3_X which over a more stable base but ultimately the trigger must be tested and run in the 12_0_X series.

When changing c++ code, particular care must be taken to preserve older behaviour if at all possible, that is it is desirable that an existing config when run with your patch will give the same output.

An important issue is when the c++ code requires changes to the HLT configuration template, either through the addition of a new module or by adding/removing parameters of existing c++ modules.

ConfDB depends on the HLT configuration template for a given release to tell it what c++ modules exist and what parameters (along with their default values) exist for a given module. If a module does not exist in the HLT configuration template, it can not be added and if a parameter of a module does not exist in the HLT configuration template it can not be set. When migrating between templates, ConfDB will automatically remove parameters/moduels which have been removed from the release and automatically add any new parameters to any module instances with their value set to their default.

ConfDB HLT configuration templates are generated using the fillDescriptions() method. Any module used at the HLT should define a fillDescriptions method and all parameters should have default values as if not, ConfDB will be unable to see them. This is manditory for new modules. Certain legacy modules are parsed from the "_cfi.py" and if adding parameters please consider adding a fillDescriptions() and only if this is not possible, add the new parameters to the _cfi.py file. Modules should not depend on whether a given parameter is present or not and if this behaviour is needed, it should use the default value to achieve this. Finally when adding new parameters, please set the default values in such a way so that the old behaviour of the module is reproduced.

Using your new modified modules which have changed the interface is tricky. The code must be integrated into a release before a ConfDB template can be parsed. Therefore for much of your development you can not use confdb to add your new module or alter your parameters. For adding a new module type, we suggest making a dummy placehold module in your config with the correct name but a random module type. You can then override this in your downloaded hlt config eg

process.myNewModule = cms.EDProducer("MyNewModule",....)

For the case of added parameters, you should just add them in your config such as...

process.existingModule.newParameter = cms.string("newParamValue")

Place these customisations manually at the end of your config or better yet define a customisation function which does this and include it via the "--customise" option in hltGetConfiguration.

Note that this is only for testing and development till your new code is parsed in confdb, the final config submitted to storm must be fully self contained

Submission for integration into CMSSW (CMSHLT-JIRA tickets)

Follow the instructions given here to get your paths integrated in the HLT menu. Ultimately the menu must be integrated in the latests combined menu so ensure you path cleanly imports to that menu and that its release tag is migrated to the current release.

All paths must pass the integrate tests listed in that twiki

How to run the production HLT within CMSSW

This part of the wiki describes the instructions on how to run the HLT contained in the CMSSW release you are using; no access to ConfDB is required. This mode is typically used for large-scale Monte-Carlo event production.

The menus in ConfDB are reguarly dumped as config file fragments int HLTTrigger/Configuration/python

They are

• HLT_GRun_cff.py : the GRun menu, this is the standard "pp" menu
• HLT_Fake[12]?_cff.py : a pass through menu for legacy L1, stage-1 L1 and stage-2 L1
• HLT_HIon_cff.py : the heavy ion menu
• HLT_PIon_cff.py : proton-ion menu
• HLT_PRef_cff.py : the menu for the hion proton-proton reference run
• HLT_Full_cff.py : the combined menu containing all integrated paths

Additionally depending on the release there may be frozen menus used for MC production. For example CMSSW_10_2_3 used for the 2018 MC production has HLT_2018v32_cff.py which was the version frozen for MC production.

You can always find out from a menu where it is located in confdb by looking at the start of the file which will specify its confdb location in the first few lines as a comment

# /dev/CMSSW_12_0_0/HLT/V2 (CMSSW_12_0_0_pre1)

fragment.HLTConfigVersion = cms.PSet(
  tableName = cms.string('/dev/CMSSW_12_0_0/HLT/V2')
)

You can include these menus via the --step=HLT:, eg

--step=HLT:GRun

will run the GRun menu. Finally in production cmsDriver commands, they make use of autoHLT.py to map common workflows to actual HLT menus. For example in CMSSW_12_0_0 "HLT:@relval2021" maps to GRun while "HLT:@relval2018" maps to "Fake2"

Legacy HLT menus and older releases

Show... Hide

setenv SCRAM_ARCH slc6_amd64_gcc700
cmsrel CMSSW_10_3_3_patch1
cd CMSSW_10_3_3_patch1/src
cmsenv

git cms-addpkg HLTrigger/Configuration

scram build -j 4
cd HLTrigger/Configuration/test

setenv SCRAM_ARCH slc6_amd64_gcc630
cmsrel CMSSW_10_1_11_patch1
cd CMSSW_10_1_11_patch1/src
cmsenv

git cms-addpkg HLTrigger/Configuration

scram build -j 4
cd HLTrigger/Configuration/test

setenv SCRAM_ARCH slc6_amd64_gcc530
cmsrel CMSSW_9_2_15
cd CMSSW_9_2_15/src
cmsenv

git cms-addpkg HLTrigger/Configuration

scram build -j 4
cd HLTrigger/Configuration/test

setenv SCRAM_ARCH slc6_amd64_gcc530
cmsrel CMSSW_8_0_31_patch1
cd CMSSW_8_0_31_patch1/src
cmsenv

git cms-addpkg HLTrigger/Configuration

scram build -j 4
cd HLTrigger/Configuration/test

setenv SCRAM_ARCH slc6_amd64_gcc491
cmsrel CMSSW_7_5_9
cd CMSSW_7_5_9/src
cmsenv

git cms-addpkg HLTrigger/Configuration

scram build -j 4
cd HLTrigger/Configuration/test

setenv SCRAM_ARCH slc6_amd64_gcc491
cmsrel CMSSW_7_4_16_patch2
cd CMSSW_7_4_16_patch2/src
cmsenv

git cms-addpkg HLTrigger/Configuration
git cms-merge-topic cms-tsg-storm:74XHLTppRefMenu # for the ppRef5TeV menu

scram build -j 4
cd HLTrigger/Configuration/test

setenv SCRAM_ARCH slc6_amd64_gcc472
cmsrel CMSSW_5_3_38
cd CMSSW_5_3_38/src
cmsenv

git cms-addpkg HLTrigger/Configuration

scram build -j 4
cd HLTrigger/Configuration/test

Useful links

• Run3Summer21 MC samples: das link
• Run3Winter21 MC samples: das link
• https://twiki.cern.ch/twiki/bin/viewauth/CMS/SpecialGlobalTagsForTSG
• https://twiki.cern.ch/twiki/bin/viewauth/CMS/TriggerTrainModel
• https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideL1TStage2Instructions
• https://twiki.cern.ch/twiki/bin/view/CMS/HLTinCMSSW101X
• https://twiki.cern.ch/twiki/bin/view/CMS/ConfDB1000
• https://twiki.cern.ch/twiki/bin/view/CMS/ConfDB1010
• Special data for rate studies: https://twiki.cern.ch/twiki/bin/view/CMS/TriggerStudiesChangesInDataTaking2018#High_rate_ephemeral_datasets_for

Test Files (available in root://eoscms.cern.ch/)

Run3Summer21

/store/mc/Run3Summer21DRPremix/TT_TuneCP5_14TeV-powheg-pythia8/GEN-SIM-DIGI-RAW/120X_mcRun3_2021_realistic_v6-v2/2540000/b354245e-d8bc-424d-b527-58815586a6a5.root

2018 Data

/store/data/Run2018D/EphemeralHLTPhysics7/RAW/v1/000/323/790/00000/B543D251-40F1-CB46-A6A1-046CF3D78D6D.root

Responsible: Main.MartinGrunewald