Trigger Analysis Tutorial (Release 15.2.0)

As part of the Helmholtz Allianz Workshop on Detector Understanding

Thursday, July 2nd 2009, 16:35 -18:05

Location: Room 28 in Flash Hall

HOMEWORK: Please read

  • An overview on trigger aware analysis as part of the Atlas Software Tutorial weeks (ignore the TrigDecisionTool specifics on pages 7-17, since they refere to the old TDT)
  • an overview on how analysis is done with the new TrigDecisionTool
  • the TWIKI page on the new TrigDecisionTool

Tutorial Abstract: In the tutorial the use of the TrigDecisionTool (TWIKI, Doxygen) to access trigger information on AOD, including Trigger Configuration, Trigger Decision, and Trigger Objects for trigger aware analysis will be explained. A few useful tools for quickly investigating the trigger content of a pool file or a data run are shown.


This trigger tutorial shows how to access trigger data in AOD files in release 15.2.0. The main aim is to demonstrate how to access the trigger menu data stored as metadata in AOD files, how to query the trigger decision event by event for the triggers of interest, and to demonstrate how to determine trigger efficiencies to be used in an analysis.

Environment Setup

As for all other tutorials we use again release 15.5.1. A quick reminder for setting up on lxplus

mkdir -p $HOME/TriggerTutorial
cd !$
source ~/cmthome/setup.sh -tag=15.5.1,32
source $AtlasArea/AtlasOfflineRunTime/cmt/setup.sh

More about setting up your environment and what the different flags do can be found on AtlasLogin.

Step 1: Checkout the package, edit a few things, compile, and run initial code

Check out UserAnalysis

The examples will use the package UserAnalysis (SVN V00-13-19), which is designed to be educational. It has only one algorithm AnalysisSkeleton, in which a number of examples are collected. Among them is a trigger example, which we are going to play with.

The new TrigDecisionTool was introduced for the first time in 15.2.0, however the package UserAnalysis was not adopted until recently. So we need to check out a newer version of UserAnalysis.

cmt co -r UserAnalysis-00-13-19 PhysicsAnalysis/AnalysisCommon/UserAnalysis
cd !$/cmt
cd -

After this run the setup again, so the libraries are picked up from the new location:

. ~/cmthome/setup.sh -tag=15.2.0,releases,opt,32

You can check that your local InstallArea is part of the setup:

echo $PATH | tr : \\n | head

Get a Hold of the Data and Investigate the Trigger Content

We will work with a 250 events Z->tautau AOD produced with 15.1.0. More info on this dataset you get when going to AMI, go to Advanced Search and put 105188 as Keyword, select AOD as Data format, 15.1.0 as the ATLAS release and hit search. Click on details of the dataset (r702) to see how it was produced. On the right hand side you find the job transform arguments, near the bottom the one important for the trigger triggerConfig.

For this tutorial the file we work with is already on the scratch area of the NAF, so we just link it

ln -s /scratch/lustre-1.6/atlas/DUW09/trigger/AOD.070468._000001.pool.root.1 AOD.pool.root

If you follow this from somewhere else, you can get the file this way:

rfcp /castor/cern.ch/grid/atlas/atlasmcdisk/valid1/AOD/valid1.105188.A3_Ztautau_filter.recon.AOD.e380_s494_r702_tid070468/AOD.070468._000001.pool.root.1 .
ln -s AOD.070468._000001.pool.root.1 AOD.pool.root

This file contains 250 events. There are two useful scripts to investigate the trigger content of a file: checkTrigger.py and checkTriggerConfig.py. Both are based on the ARA technology of creating transient trees for a python/CINT type analysis. The first is to check the trigger counts, the second gives you information on the trigger configuration.


checkTrigger.py AOD.pool.root | tee triggerStats.txt

will give you

Size:    55955.492 kb
Nbr Events: 250

Trigger configuration summary:
SMK: 0, HLTpsk: 0, L1psk: 0
Config source: TriggerMenuXML/LVL1config_MC_lumi1E31_no_prescale_15.1.0.xml and TriggerMenuXML/HLTconfig_MC_lumi1E31_no_prescale_15.1.0.xml
L1 Items   : 146
HLT Chains : 556

   ID level     Trigger name    Passed events:   raw, after PS, after PT/Veto
       LVL1   Global LVL1                        250
       LVL2   Global LVL2                        250
         EF   Global EF (LVL3)                   250
   13  LVL1   L1_2EM13                            71        71             71
   14  LVL1   L1_2EM13I                           34        34             34
  163  LVL1   L1_2EM13_MU6                         8         8              8
   45  LVL1   L1_ZDC_FILLED                        0         0              0
   77  LVL2   L2_2g10                            118       118            118
  246  LVL2   L2_2g10_mu6                         12        12             12
  477    EF   EF_2e6_medium                        8         8              8
  478    EF   EF_2e6_medium1                       7         7              7
   77    EF   EF_2g10                            118       118            118
  246    EF   EF_2g10_mu6                         12        12             12
   76    EF   EF_2g17i_tight                       6         6              6
   79    EF   EF_2g20                             39        39             39
  248    EF   EF_2j42_xe30                         3         3              3
## Bye.


checkTriggerConfig.py -d AOD.pool.root | tee triggerMenu.txt

The -d option prints the entire menu, otherwise only an overview would be printed. Printed are the chains in their entire logical flow L1->L2->EF together with the prescale factors.

Run: 105188
SMK: 0, HLTpsk: 0, L1psk: 0
Config source: TriggerMenuXML/LVL1config_MC_lumi1E31_no_prescale_15.1.0.xml and TriggerMenuXML/HLTconfig_MC_lumi1E31_no_prescale_15.1.0.xml
L1 Items   : 146
HLT Chains : 556
EF: EF_tau16i_loose_j70_WO (1.00), L2: L1_TAU9I_2J5_J35 (1), L1: L1_TAU9I_2J5_J35 (1)
EF: EF_mu10i_loose (1.00), L2: L1_MU10 (1), L1: L1_MU10 (1)
EF: EF_2mu4_Bmumux (1.00), L2: L1_2MU4 (1), L1: L1_2MU4 (1)
EF: EF_MU4_Jpsimumu_FS (1.00), L2: L1_MU4 (1), L1: L1_MU4 (1)
EF: EF_mu6_DsPhiPi (1.00), L2: L1_MU6_J5 (1), L1: L1_MU6_J5 (1)
EF: EF_e20_loose (1.00), L2: L1_EM18 (1), L1: L1_EM18 (1)
EF: EF_J350 (1.00), L2: L1_J120 (1), L1: L1_J120 (1)
EF: EF_tau16i_loose_mu10 (1.00), L2: L1_TAU9I_MU10 (1), L1: L1_TAU9I_MU10 (1)

Where does the configuration come from

When taking data the trigger configuration comes from point 1 (the oracle TriggerDB) and is stored in COOL. When ESD files are written, this information is copied from COOL into the ESD file. For MC data, the same happens when producing ESD from RDO files, except the configuration comes from XML instead from COOL. So ESD, AOD, and DPD files will all have the trigger configuration available inside the file as conditions data. When you merge files, this information also gets merged. So, when you process an event, you always have the correct configuration in memory. But be aware that it might change underneath you, so check prescales at every lumiblock, and configured chains every new run.

First run

In share/AnalysisSkeleton_topOptions.py the input file is already specified

ServiceMgr.EventSelector.InputCollections = [ "AOD.pool.root" ]

so we can just run

athena UserAnalysis/AnalysisSkeleton_topOptions.py >! log

After the run a root output file AnalysisSkeleton.aan.root appeared. Let's have a look at that later and first at the log file. First we see that the configuration was loaded correctly from the AOD file:

DSConfigSvc          INFO update callback invoked for 5 keys:  /TRIGGER/HLT/HltConfigKeys /TRIGGER/HLT/Menu /TRIGGER/LVL1/L
vl1ConfigKey /TRIGGER/LVL1/Menu /TRIGGER/LVL1/Prescales
DSConfigSvc          INFO Updating trigger configuration: HLT keys
DSConfigSvc          INFO   Configuration key : 0
DSConfigSvc          INFO   HLT  prescale key : 0
DSConfigSvc          INFO   Original source   : TriggerMenuXML/LVL1config_MC_lumi1E31_no_prescale_15.1.0.xml and TriggerMenuXML/HLTconfig_MC_lumi1E31_no_prescale_15.1.0.xml
DSConfigSvc          INFO Updating trigger configuration: LVL1 menu
DSConfigSvc          INFO   Number of items: 146
DSConfigSvc          INFO Updating trigger configuration: HLT menu
DSConfigSvc          INFO   Number of chains: 556
DSConfigSvc          INFO Updating trigger configuration: LVL1 keys
DSConfigSvc          INFO   LVL1 prescale key: 0
DSConfigSvc          INFO Updating trigger configuration: LVL1 prescales

The keys are all 0 (they are filled for real data), but the original xml files are listed. Most important is that 146 L1 items and 556 HLT chains were loaded. On the first event you should see a list of the triggers:

AnalysisSkeleton     INFO L1 Items : [L1_EM3, L1_EM7, L1_3J18_J42, L1_EM13, ... , L1_MBTS_1_1, L1_LUCID_A, L1_LUCID_C, L1_LUCID_A_C, L1_LUCID]
AnalysisSkeleton     INFO L2 Chains: [L2_MU6_BmumuX, L2_te650, L2_MU6_Bmumu, L2_g25_xe30, L2_tau16i_loose_j70_WO, L2_mu4_j10_matched, ...,L2_tau20i_loose, L2_2tau20i_loose]
AnalysisSkeleton     INFO EF Chains: [EF_MU6_BmumuX, EF_te650, EF_MU6_Bmumu, EF_g25_xe30, ...,EF_MU4_Bmumu, EF_tau20i_loose, EF_2tau20i_loose]

On each event you see messages like this

AnalysisSkeleton     INFO Pass state L1 = 1
AnalysisSkeleton     INFO Pass state L2 = 1
AnalysisSkeleton     INFO Pass state EF = 1
AnalysisSkeleton     INFO Pass state L2_tau16i_loose_3j23 = 0
AnalysisSkeleton     INFO Pass state EF_mu10              = 0
AnalysisSkeleton     INFO Pass state EF_mu20              = 0
AnalysisSkeleton     INFO Pass state EF_e15_medium        = 1
AnalysisSkeleton     INFO Pass state EF_e20_loose         = 1

You can check the src code in src/AnalysisSkeleton.cxx. The output of the list of triggers comes from

if (m_eventNr==1) {
  mLog << MSG::INFO << "L1 Items : " << m_allL1->getListOfTriggers() << endreq;
  mLog << MSG::INFO << "L2 Chains: " << m_allL2->getListOfTriggers() << endreq;
  mLog << MSG::INFO << "EF Chains: " << m_allEF->getListOfTriggers() << endreq;
, the per-event statistics comes from
mLog << MSG::INFO << "Pass state L1 = " << m_trigDec->isPassed("L1_.*") << endreq;
  mLog << MSG::INFO << "Pass state L2 = " << m_trigDec->isPassed("L2_.*") << endreq;
  mLog << MSG::INFO << "Pass state EF = " << m_trigDec->isPassed("EF_.*") << endreq;
  mLog << MSG::INFO << "Pass state L2_tau16i_loose_3j23 = " << m_trigDec->isPassed("L2_tau16i_loose_3j23") << endreq;
  mLog << MSG::INFO << "Pass state EF_mu10              = " << m_trigDec->isPassed("EF_mu10") << endreq;
  mLog << MSG::INFO << "Pass state EF_mu20              = " << m_trigDec->isPassed("EF_mu20") << endreq;
  mLog << MSG::INFO << "Pass state EF_e15_medium        = " << m_trigDec->isPassed("EF_e15_medium") << endreq;
  mLog << MSG::INFO << "Pass state EF_e20_loose         = " << m_trigDec->isPassed("EF_e20_loose") << endreq;

This shows already two ways to work with the TrigDecisionTool. You can either define ChainGroups in the constructor of you class, done so for m_allL1, m_allL2, m_allEF, or you can give strings to the TDT, and the ChainGroup is created and stored internally. Now, to write your own code that can make use of the trigger you need to know how to get a hold of the TDT and how to set it up.

Understanding the setup

The AnalysisSkeleton example algorithm of the UserAnalysis package shows which steps have to be taken to setup the usage of the TrigDecisionTool. The setup in the python joboptions and the access in C++.

Joboption Setup

If you look at the job options file share/AnalysisSkeleton_topOptions.py, you see what's needed for setting up the reading of trigger configuration data from file correctly, and the access of the trigger decision:

if AnalysisSkeleton.DoTrigger:
   # Needed for TriggerConfigGetter...
   from RecExConfig.RecFlags  import rec

   # To read files with trigger config stored as in-file meta-data,
   from TriggerJobOpts.TriggerFlags import TriggerFlags
   TriggerFlags.configurationSourceList = ['ds']

   # set up trigger config service
   from TriggerJobOpts.TriggerConfigGetter import TriggerConfigGetter
   cfg =  TriggerConfigGetter()

The source of the trigger configuration is set to 'ds', like detector store. Effectively this means that in-file metadata is used to setup a COOL folder structure with the trigger configuration data in memory, and the data is updated by the IOVDbSvc automatically so that the user just needs to know about the TrigDecisionTool (see below). Other possible sources of the trigger configuration data are a real COOL database, an xml file, or an oracle / mysql trigger database.

In release 15.3.0 this setup will be much simplified, all that will be needed is

if AnalysisSkeleton.DoTrigger:
   from TriggerJobOpts.TriggerConfigGetter import TriggerConfigGetter
   cfg =  TriggerConfigGetter("ReadPool")

The TrigDecisionTool does not need to be set up, the default (specified in AnalysisSkeleton.cxx) is already what we want. No extra python configuration needed.

Access to the TrigDecisionTool in the Athena Algorithm

In the header file UserAnalysis/AnalysisSkeleton.h the handle to the TrigDecisionTool needs to be defined:

#include "TrigDecisionTool/TrigDecisionTool.h"
class AnalysisSkeleton 
   /** tool to access the trigger decision */
   ToolHandle<Trig::TrigDecisionTool> m_trigDec;

In the implementation file src/AnalysisSkeleton.cxx one needs to

  • define the default name of the TrigDecisionTool handle "Trig::TrigDecisionTool"

AnalysisSkeleton::AnalysisSkeleton(const std::string& name, ISvcLocator* pSvcLocator) :
{ ... }

  • Initialize the TrigDecisionTool:

StatusCode AnalysisSkeleton::CBNT_initializeBeforeEventLoop() {
    // retrieve trigger decision tool
    // needs to be done before the first run/event since a number of
    // BeginRun/BeginEvents are registered by dependent services
    return m_trigDec.retrieve();

Note: In a job the retrieving of the TDT needs to be done in Algorithm::initialize() and neither earlier nor later, otherwise the setup will be incomplete. However, AnalysisSkeleton is a CBNT_Algorithm, (not an Algorithm), so the AnalysisSkeleton::!CBNT_initializeBeforeEventLoop() corresponds to the Algorithm::initialize() and the retrieving of the TrigDecisionTool has to be done in there

Step 2: Print the trigger menu stored in an AOD file

The new TrigDecisionTool was designed with the person doing analysis in mind. All functionality that is considered to be not necessary for analysis was removed (or moved to an expert access cathegory). Among the things that were removed are the direct access to the trigger configuration and the trigger navigation. Necessary for analysis is only the knowledge which prescale factor a certain trigger has.

To print that information we put the following in triggerSkeleton() of src/AnalysisSkeleton.cxx:

if (m_eventNr==1) {
    const std::vector<std::string> allEF = m_allEF->getListOfTriggers();
    std::vector<std::string>::const_iterator it = allEF.begin()
    for(; it != allEF.end(); it++) {
        mLog << MSG::INFO << "Prescale info: chain " << std::left << *it 
             << " has prescale " << m_trigDec->getPrescale(*it) << endreq;

Note that the prescales of the trigger can change between luminosity blocks. This means that the check for the prescale factors has to be done every time the luminosity block changes. As the file that we look at contains MC simulated events, we see all prescales being set to 1 (default for MC production). The output is the following:

AnalysisSkeleton     INFO Prescale info: chain EF_e20_loose_xe30 has prescale 0
AnalysisSkeleton     INFO Prescale info: chain EF_tau12_loose has prescale 1
AnalysisSkeleton     INFO Prescale info: chain EF_tau12_PT has prescale 1
AnalysisSkeleton     INFO Prescale info: chain EF_e6_medium1 has prescale 1
AnalysisSkeleton     INFO Prescale info: chain EF_mu4_mu6 has prescale 0
AnalysisSkeleton     INFO Prescale info: chain EF_JE280 has prescale 0
AnalysisSkeleton     INFO Prescale info: chain EF_tau16i_loose_EFxe40 has prescale 1
AnalysisSkeleton     INFO Prescale info: chain EF_J10_larcalib has prescale 1
AnalysisSkeleton     INFO Prescale info: chain EF_j42_xe30_mu15 has prescale 0
AnalysisSkeleton     INFO Prescale info: chain EF_mu6_DiMu has prescale 0
AnalysisSkeleton     INFO Prescale info: chain EF_mu4_j10 has prescale 0
AnalysisSkeleton     INFO Prescale info: chain EF_g25 has prescale 1

This is not as expected. One should see preascale 1 everywhere. This is a bug in this early version of the TrigDecisionTool, and is currently being fixed. A patch for 15.2.0 will be provided.

Step 3: Print simple trigger statistics and efficiencies

Prepare the job

Now that we have managed to obtain information about the available triggers in the AOD, let's print out the number of events that passed a certain trigger and the corresponding efficiency after all events in the file have been processed.

In the AnalysisSkeleton.cxx we add a property StatTriggerChains to the algorithm constructor, so we can specify the list of triggers for which we want to have statistics.

AnalysisSkeleton::AnalysisSkeleton(const std::string& name, ISvcLocator* pSvcLocator)
    declareProperty("StatTriggerChains", m_triggerChains, "list of triggers for which to print statistics");

We add the vector to store chain names to AnalysisSkeleton.h, along with variables to store the number of passed events:

class AnalysisSkeleton {
    std::vector<std::string> m_triggerChains;
    std::map<std::string,int> m_triggersPassed;

We then initialise everything: In the initialize() method of UserAnalysis/AnalysisSkeleton.cxx set the counts to zero. Don't do this in the constructor, at that point the property m_triggerChains has not been set from python yet. The evaluation of job configuration properties must always happen in initialize() or later.

StatusCode AnalysisSkeleton::CBNT_initializeBeforeEventLoop() {
    std::vector<std::string>::const_iterator it;
    for(it = m_triggerChains.begin();it != m_triggerChains.end(); it++)
        m_triggersPassed[*it] = 0;

and in the trigger section in=share/AnalysisSkeleton_topOptions.py= put the trigger chains you are interested in, e.g. add

if AnalysisSkeleton.DoTrigger:
   ## chains and groups for which to print trigger statistics
   photons = ["L1_2EM13", "L2_2g10_mu6", "EF_2g10"]
   singletaus = ["EF_tau12_loose", "EF_tau16_loose", "EF_tau16i_loose", "EF_tau20_loose", "EF_tau20i_loose",
                 "EF_tau29_loose", "EF_tau29i_loose", "EF_tau38_loose", "EF_tau50_loose", "EF_tau84_loose"]
   twotaus = ["EF_2tau20i_loose", "EF_2tau29i_loose", "EF_2tau29i_medium"]
   combinedtaus = ["EF_tau12_loose_e10_loose", "EF_tau16i_loose_2j23", "EF_tau16i_loose_EFxe40"]
   AnalysisSkeleton.StatTriggerChains = photons + singletaus + twotaus + combinedtaus;

In the triggerSkeleton() function is the part where the statistics is collected:

for(it = m_triggerChains.begin();it != m_triggerChains.end(); it++)
    if( m_trigDec->isPassed(*it) ) m_triggersPassed[*it]++;

In the CBNT_finalize() we add a few lines to print the trigger statistics after all events have been processed for each of the requested triggers:

if(m_doTrigger) {
  // print trigger statistics
  mLog << MSG::INFO << "STAT Trigger Statistics on " << m_allEvents << "  " << m_eventNr << " processed events" << endreq;
  for(  std::vector<std::string>::const_iterator it = m_triggerChains.begin();it != m_triggerChains.end(); it++)
    mLog << MSG::INFO << "STAT Passed events for chain " << *it << "  " << m_triggersPassed[*it] 
         << " ("<< 100.*m_triggersPassed[*it]/m_eventNr <<"%)" << endreq;

Run the job

Again, run the job an check the log file for 'STAT'

athena UserAnalysis/AnalysisSkeleton_topOptions.py >! log
grep 'STAT' log
AnalysisSkeleton     INFO STAT Trigger Statistics on 250 processed events
AnalysisSkeleton     INFO STAT Passed events for chain L1_2EM13  71 (28.4%)
AnalysisSkeleton     INFO STAT Passed events for chain L2_2g10_mu6  12 (4.8%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_2g10  118 (47.2%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau12_loose  194 (77.6%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau16_loose  182 (72.8%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau16i_loose  178 (71.2%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau20_loose  164 (65.6%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau20i_loose  162 (64.8%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau29_loose  108 (43.2%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau29i_loose  107 (42.8%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau38_loose  52 (20.8%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau50_loose  9 (3.6%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau84_loose  2 (0.8%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_2tau20i_loose  49 (19.6%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_2tau29i_loose  20 (8%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_2tau29i_medium  10 (4%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau12_loose_e10_loose  35 (14%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau16i_loose_2j23  38 (15.2%)
AnalysisSkeleton     INFO STAT Passed events for chain EF_tau16i_loose_EFxe40  22 (8.8%)

You see the total number of events, and the number (percentage) of events that passed the trigger for the requested chain. As expected the numbers for the tau triggers are quite high, due to the nature of the sample. Of course these numbers don't mean much, except to give you a rough feeling of your trigger, if you sort of know the content of your MC data. If you ahve a quick look at the root file, you see the counts stored for each trigger.

Special flags in ChainGroup::isPassed()

The syntax for isPassed() is the following

bool Trig::TrigDecisionTool::isPassed(const Trig::ChainGroup *chaingroup, unsigned int condition=TrigDefs::Physics) const;
bool Trig::TrigDecisionTool::isPassed(const std::string &chain, unsigned int condition=TrigDefs::Physics) const;
bool Trig::ChainGroup::isPassed(unsigned int condition = TrigDefs::Physics) const;

where the condition can be:

returns the result as a combination of all three trigger levels. This is important when asking for the prescale factor of a trigger chain.

A decision must have been made, which means the trigger did not pass due to pass-through or resurrection (see next two).

Requires that the event was passed through. In the trigger chain definition a pass-through factor can be set (if then usually a high number), in order to record events independent of the decision of that trigger chain. This is usefull for understanding the trigger behavior. One can restrict the isPassed() to those events, (requireDecision must be off.)

Triggers with a high rate (often those with a low energy threshold) are prescaled to not waste resources. If a trigger is prescaled on a given event, that trigger chain is not even executed (to safe computing time). However, if an event is accepted (by some trigger) it is often important to know also the trigger decision of the prescaled trigger, for instance if this trigger is used as an orthorgonal trigger for efficiency determination. In this case we execute the trigger (according to rerun_prescale, see the chain definition in the menu above), and safe its decision, but we don't allow it the select an event. To access this decision, you need to specify allowResurrectedDecision. (requireDecision must be off.)

Pseudonym for requireDecision | enforceLogicalFlow, which means that both conditions are applied. This is the default for isPassed() and for getPrescale() methods.

Pseudonym for enforceLogicalFlow

A Word on tau trigger efficiency

One could now try to determine the efficiency of a particular trigger with respect to an offline trigger selection. One would perform an offline tau selection (offline as opposed to a trigger tau selection) on events where the tau under investigation was not used in the trigger selection (for instance Z->tautau events passed by a single-tau trigger). While one tau was firing the trigger, the other could be reconstructed offline and matched to a trigger tau. A simple count how often such a trigger tau can be matched and would have fired the trigger would tell us the tau efficiency with respect to the offline tau.

How we proceed in this tutorial?
While the performance analysis of this tau trigger is an interesting exercise, I find it more important to understand the trigger execution and the issues that are involved with retrieving trigger object for trigger of various complexities. This is what we will do now. With the help of the Trigger Navigation framework we will find our way through the trigger data to pick the trigger objects that were calculated by a certain algorithm and that in the end fired a certain trigger.

Step 4: Investigating the EF_tau16i_loose_2j23 Trigger

We will investigate the content of the trigger EF_tau16i_loose_2j23. This is a combined trigger that will fire if there is at least one tau above 16 GeV with some isolation criteria and at least two jets above 23 GeV in the event.

Understanding the trigger menu

Let's look at the definition of this trigger in XML language. As said this was produced in release 15.1.0 so we look at the file /afs/naf.desy.de/group/atlas/software/kits/15.1.0/AtlasTrigger/15.1.0/InstallArea/XML/TriggerMenuXML/HLTconfig_MC_lumi1E31_no_prescale_15.1.0.xml. This is the definition of the CHAIN. At the top it contains important information like the name and counter of the chain, the name of the seeding chain of the previous level, and the prescale, pass-through, and rerun-prescale factors. It also contains the chain execution information.

<CHAIN chain_counter="267" chain_name="EF_tau16i_loose_2j23" level="EF" 
   lower_chain_name="L2_tau16i_loose_2j23" pass_through="0" prescale="1" rerun_prescale="-1">
                <STREAMTAG obeyLB="yes" prescale="1" stream="jetTauEtmiss" type="physics"/>
                <GROUP name="Inclusive_Taus"/>
                <GROUP name="Inclusive_Jets"/>
                <GROUP name="TauJets"/>
                <GROUP name="Test"/>
                <SIGNATURE logic="1" signature_counter="1">
                        <TRIGGERELEMENT te_name="EFtau16i_looseclf0"/>
                        <TRIGGERELEMENT te_name="EF_j23"/>
                        <TRIGGERELEMENT te_name="EF_j23"/>
                <SIGNATURE logic="1" signature_counter="2">
                        <TRIGGERELEMENT te_name="EFtau16i_loosetr"/>
                        <TRIGGERELEMENT te_name="EF_j23"/>
                        <TRIGGERELEMENT te_name="EF_j23"/>
                <SIGNATURE logic="1" signature_counter="3">
                        <TRIGGERELEMENT te_name="EF_tau16i_loose"/>
                        <TRIGGERELEMENT te_name="EF_j23"/>
                        <TRIGGERELEMENT te_name="EF_j23"/>
                <SIGNATURE logic="1" signature_counter="4">
                        <TRIGGERELEMENT te_name="EFtau16i_loose_2j23nooverlap"/>

This is how a trigger execution is defined (almost, the algorithms are missing). As explained, the ATLAS trigger works in steps, at each step a certain multiplicity of TriggerElements (TE) is required. In this case we have 4 steps and we require, e.g., in step 3 two EF_j23 and one EF_tau16i_loose. In the last step some overlap removal is done. Now one also wants to know how to get from a TE EFtau16i_loosetr to a TE EF_tau16i_loose. This is a defined in the so called SEQUENCE*s, which is are sequences of reconstruction (*FEX) and selection (HYPO) algorithms, with a reconstruction algorithm always in the beginning:

<SEQUENCE algorithm="TrigCaloCellMaker/TrigCaloCellMaker_tau TrigCaloClusterMaker/TrigCaloClusterMaker_topo" input="L2_tau16i_loose" output="EFtau16i_looseclf0"/>
<SEQUENCE algorithm="InDet::Pixel_TrgClusterization/PixelClustering_Tau_EFID ..." input="EFtau16i_looseclf0" output="EFtau16i_loosetr"/>
<SEQUENCE algorithm="TrigTauRecMerged/TrigTauRecMerged_IDSCAN EFTauHypo/EFTauHypo_tau16i_loose" input="EFtau16i_loosetr" output="EF_tau16i_loose"/>
<SEQUENCE algorithm="TrigCaloCellMaker/TrigCaloCellMaker_jet TrigCaloTowerMaker/TrigCaloTowerMaker_jet 
       TrigJetRec/TrigJetRec_Cone TrigEFJetHypo/EFJetHypo_j23" input="L2_j23" output="EF_j23"/>

As explained in the slides of the homework material, the first algorithms (the FEXs), create trigger objects (or features), which they attach to the TriggerElement. Two examples for our case:

  • The InDet::Pixel_TrgClusterization/PixelClustering_Tau_EFID attaches PixelClusterContainer objects to EFtau16i_loosetr.
  • The TrigJetRec/TrigJetRec_Cone attaches JetCollection objects to EF_j23, this time with the extra label "TrigJetRec"

It is not so easy to learn which FEX creates what type of object, for this one needs to know the trigger software, and read the literature or look at the code. Luckily you always know the class name of an algorithm ("classname/instancename"), so you can use LXR. Of help can also be two things:

We will now try to access these trigger objects.

Understanding the Navigation

The FeatureContainer

Let's first get the FeatureContainer for this chain EF_tau16i_loose_2j23. The FeatureContainer, returned by the ChainGroup::features(), is a container that gives access to trigger elements and trigger objects for simple (single type of object) and complex (multiple types of objects) triggers. The FeatureContainer supports two varieties of get methods.

  1. All features of a given type: vector< Trig::Feature< T > > FeatureContainer::get(string label, uint condition=Physics, string teName="")
  2. Vector of combinations of features: vector<Trig::Combination> FeatureContainer::getCombinations(). This latter is most appropriate for complex triggers where two or more types of trigger objects are involved.

In the following we use both. First we get a vector of all jets of type that were produced in this chain (by TrigJetRec/TrigJetRec_Cone) and later we get a vector of combinations (one tau[>16GeV] and two jets[>23GeV]) that satisfied the trigger.

The function

const FeatureContainer features(unsigned int condition = TrigDefs::Physics) const
has an optional argument, that is by default TrigDefs::Physics. This will fill the FeatureContainer only with features that contributed to the passing of the trigger. An alternative is TrigDefs::alsoDeactivateTEs, which also adds features of deactivated TriggerElements.

Trigger Features

First we are going to access a list of jets that were created during the chain execution. For this we ask the FeatureContainer for a vector of Features of type JetCollection. A Feature is nothing but an access point to the trigger objects with additional information about the TriggerElement. Because the TriggerElement is part of the navigational structure, it can be used to navigate to ealier instances of the trigger object (via the TrigDecisionTool::ancestor() method, more about that later.

std::vector< Trig::Feature<JetCollection> > FeatureContainer::get(const std::string& label, unsigned int condition=TrigDefs::Physics, string teName="")

An number of feature related functions in the TrigDecisionTool have an optional argument const string& label. These correspond to the StoreGate keys (minus "HLT_"). If you have a look at fileContent.txt, (the output of checkFile):
>grep TrigRoiDescriptor fileContent.txt
      52.334 kb        5.552 kb        0.022 kb        0.199      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_TrigCaloRinger
      56.061 kb        7.773 kb        0.031 kb        0.186      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_secondaryRoI_EF
      63.160 kb       10.673 kb        0.043 kb        0.164      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_forMS
      63.160 kb       10.679 kb        0.043 kb        0.164      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_forID
      66.552 kb       11.474 kb        0.046 kb        0.156      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_secondaryRoI_L2
      96.685 kb       25.609 kb        0.102 kb        0.108      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_TrigT2CaloEgamma
      96.325 kb       27.150 kb        0.109 kb        0.108      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_TrigT2CaloTau
     110.456 kb       31.728 kb        0.127 kb        0.095      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_TrigT2CaloJet
     163.605 kb       38.949 kb        0.156 kb        0.065      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_initialRoI
     217.361 kb       60.725 kb        0.243 kb        0.050      250  (B) TrigRoiDescriptorCollection_tlp1_HLT
     240.863 kb       70.211 kb        0.281 kb        0.046      250  (B) TrigRoiDescriptorCollection_tlp1_HLT_T2TauFinal
one can see that many different trigger chains refine the initial RoIDescriptor. A correct syntax to get the initial RoIDescriptor (based on LVL1) would then be

std::vector< Feature<TrigRoiDescriptor> > roiF = featureContainer.get<TrigRoiDescriptor>("initialRoI");

Again, you can see that in the persistent data the type is TrigRoiDescriptorCollection, while through the navigation you get TrigRoiDescriptor objects. Again, the only way to find out at the moment is TrigEventARA/selection.xml.

Feature access
Let us now look at a code example:

StatusCode AnalysisSkeleton::triggerSkeleton() {
   const std::string chain("EF_tau16i_loose_2j23");
   // creating the feature container
   FeatureContainer f = m_trigDec->features(chain);

   mLog << MSG::INFO << "Number of JetCollections: " << jetColls.size() << endreq;
   std::vector< Feature<JetCollection> > jetColls = f.get<JetCollection>();
   if(jetColls.size()>0) {
     // get the first Feature
     const Feature<JetCollection>& jcf = jetColls[0];
     mLog << MSG::INFO << "Label: " << jcf.label() << endreq;
     const JetCollection* jc = jcf.cptr();
     mLog << MSG::INFO << "Number of Jets: " << jc->size() << endreq;
     JetCollection::const_iterator jIt = jc->begin();
     for (; jIt != jc->end(); ++jIt ) {
        Jet* jet = *jIt;
        mLog << MSG::INFO << "Jet e   : " << jet->e()   << endreq;
        mLog << MSG::INFO << "    pt  : " << jet->pt()  << endreq;
        mLog << MSG::INFO << "    et  : " << jet->et()  << endreq;
        mLog << MSG::INFO << "    eta : " << jet->eta() << endreq;
        mLog << MSG::INFO << "    phi : " << jet->phi() << endreq;

     // find the corresponding jets in Lvl2 through the inheritance tree (navigation does that all)
     Feature<TrigT2Jet> l2jetF = m_trigDec->ancestor<TrigT2Jet>(jcf);
     mLog << MSG::INFO << "Found " << (l2jetF.empty()?"no ":"") << "corresponding L2 Jet." << endreq;
     if ( !l2jetF.empty() ) {
        const TrigT2Jet* t2jet = l2jetF.cptr();
        mLog << MSG::INFO << "   eta  : " << t2jet->eta() << endreq; 
        mLog << MSG::INFO << "   phi  : " << t2jet->phi() << endreq; 
        mLog << MSG::INFO << "   e    : " << t2jet->e() << endreq; 
        mLog << MSG::INFO << "   ehad : " << t2jet->ehad0() << endreq; 
        mLog << MSG::INFO << "   eem  : " << t2jet->eem0() << endreq; 
     // we can also access the L1 Jet_ROI using the ancestor method of the TrigDecisionTool
     Feature<Jet_ROI> jRoIF =  m_trigDec->ancestor<Jet_ROI>(jcf);
     mLog << MSG::INFO << "Found " << (jRoIF.empty()?"no ":"") << "corresponding Jet_ROI" << endreq; 
     if ( !jRoIF.empty() ) {
        const Jet_ROI* jroi = jRoIF.cptr();
        mLog << MSG::INFO << "Passed thresholds" << jroi->getThresholdNames() << endreq; 
        mLog << MSG::INFO << "   eta   : " << jroi->eta() << endreq; 
        mLog << MSG::INFO << "   phi   : " << jroi->phi() << endreq; 
        mLog << MSG::INFO << "   ET4x4 : " << jroi->getET4x4() << endreq; 
        mLog << MSG::INFO << "   ET6x6 : " << jroi->getET6x6() << endreq; 
        mLog << MSG::INFO << "   ET8x8 : " << jroi->getET8x8() << endreq;

The output is the following:

AnalysisSkeleton     INFO FLAT Pass state EF_tau16i_loose_2j23 = 1
AnalysisSkeleton     INFO FLAT Number of JetCollections in EF_tau16i_loose_2j23: 1
AnalysisSkeleton     INFO FLAT TE Label: TrigJetRec
AnalysisSkeleton     INFO FLAT Number of Jets in JetCollection: 1
AnalysisSkeleton     INFO FLAT Jet e   : 82827.9
AnalysisSkeleton     INFO FLAT     eta : 0.876432
AnalysisSkeleton     INFO FLAT     phi : -2.45521
AnalysisSkeleton     INFO FLAT     pt  : 58067.8
AnalysisSkeleton     INFO FLAT     et  : 58772.8
AnalysisSkeleton     INFO FLAT Found corresponding L2 Jet.
AnalysisSkeleton     INFO FLAT    e    : 83197.4
AnalysisSkeleton     INFO FLAT    eta  : 0.878568
AnalysisSkeleton     INFO FLAT    phi  : -2.44386
AnalysisSkeleton     INFO FLAT    ehad : 1312.08
AnalysisSkeleton     INFO FLAT    eem  : 53246
AnalysisSkeleton     INFO FLAT Found corresponding Jet_ROI
AnalysisSkeleton     INFO FLAT Passed thresholds[J5, J10, J18, J23, J35, J42]
AnalysisSkeleton     INFO FLAT    ET4x4 : 47000
AnalysisSkeleton     INFO FLAT    ET6x6 : 48000
AnalysisSkeleton     INFO FLAT    ET8x8 : 48000
AnalysisSkeleton     INFO FLAT    eta   : 1
AnalysisSkeleton     INFO FLAT    phi   : -2.35619

That is unexpected, we would have thought to see two JetCollections for passed EF_tau16i_loose_2j23 trigger, instead we see only one. This error message gives a clue, the second feature could not be retrieved. It is a bug and should have disappeared in 15.3.0.

ToolSvc.Trig::T...WARNING HolderImp::get getting chunk of the container failed for Holder: CLID: 1162448536 label: "TrigTauJet" subTypeIndex: 1 Stored type name: JetCollection Collection type name: JetCollection CLID: 1162448536 size: 0 index: SubTypeIdx: 1 begin: 4 end: 5
ToolSvc.Trig::T...WARNING getFeature: problems while getting objects #SubTypeIdx: 1 begin: 4 end: 5 from the holder: CLID: 1162448536 label: "TrigTauJet" subTypeIndex: 1 Stored type name: JetCollection Collection type name: JetCollection CLID: 1162448536 size: 0

This way of accessing trigger objects as plain vectors is good for single object triggers, or for studying trigger reconstructions, e.g. of jets. In this example we write ET, eta, and phi of the jets at the three different levels into the root file. Let's have a look.

L2 and EF jet energies are in good agreement, occasionally the EF energy is much lower which is most likely due to splitting, the EF algorithm made two jets, where L2 only one.

The EF jet energies are higher the those in the L1 Towers. Possible reasons could be energy deposited outside the 8x8 box of the LVL1 trigger, or different callibration?

good agreement in the angular distribution, with some exceptions. Check if there is a correlation between those, and the outliers in the jet energy.

Note: If I would repeat the same exercise for the chain L2_tau16i_loose_2j23, I would not get any result, only this output on each event:

AnalysisSkeleton     INFO Number of JetCollections: 0
This is, because JetCollection is only created in the EF part of the tau16i_loose_2j23 trigger, and navigation only looks at a chain and its predecessors. So in the EF_tau16i_loose_2j23 I can access the TrigT2Jet jets from L2, but as said, not vice versa.

Navigating through the objects ancestry

In the previous example we also made use of the ancestor() function of the TrigDecisionTool. This allows us to follow a feature through the different trigger steps, for instance if you want to learn how trigger objects get refined at the different trigger levels.

const Feature<T> ancestor(const HLT::TriggerElement *te, std::string label="") const;

Instead of the TriggerElement, you can give a Feature as argument to the ancestor() method, it is automatically casted.


Often one has to deal with complex triggers, which require the successful reconstruction and selection of multiple objects and of different type. By looking only at the flat vectors of these objects (via Features) one misses some vital information, namely, which combination of objects was actually responsible for the passing of the trigger. For this purpose the FeatureContainer gives access to a vector of Trig::Combinations.

A Trig::Combination for a, e.g. 1tau+2jet, trigger is an object that holds internally 1 Feature for every object created during the reconstruction of the successful tau, and 2 Features for every object created during the reconstruction of the two successful jets. Those again can be accessed via a templated get(label="") method.

StatusCode AnalysisSkeleton::triggerSkeleton() {

  const std::string chain("EF_tau16i_loose_2j23");
  // creating the feature container
  FeatureContainer f = m_trigDec->features(chain);

  const std::vector<Trig::Combination>& tauJetCombinations = f.getCombinations();
  mLog << MSG::INFO << "Trigger contains " << tauJetCombinations.size() << " 1Tau2Jet combinations." << endreq;
  std::vector<Trig::Combination>::const_iterator cIt;
  for ( cIt = tauJetCombinations.begin(); cIt != tauJetCombinations.end(); ++cIt ) {
     const Trig::Combination& comb = *cIt;

     std::vector< Feature<TauJetContainer> > tauC = comb.get<TauJetContainer>();
     std::vector< Feature<JetCollection> >   jetC = comb.get<JetCollection>();

     mLog << MSG::INFO << "Combination has " << tauC.size() << " TauJetContainer and " << jetC.size() << " JetCollection."
<< endreq;
     if(tauC.size()>0 || jetC.size()>0) {
        const TauJetContainer* taus = tauC[0];
        const JetCollection* jets = jetC[0];

        mLog << MSG::INFO << "Looking at TauJet combination with " << taus->size() << " taus and " << jets->size() << " jets,"
             << " this one was " << (comb.active()?"":"not ") << "active." <<endreq;
     } else {
        mLog << MSG::INFO << "TauJetContainer or JetCollection missing." <<endreq;

     std::vector< Feature<TrigTau> >   tauFV = comb.get<TrigTau>();
     std::vector< Feature<TrigT2Jet> > jetFV = comb.get<TrigT2Jet>();

     mLog << MSG::INFO << "Looking at TauJet combination with " << tauFV.size() << " TrigTau features and " 
          << jetFV.size() << " TrigT2Jet features,"
          << " this one was " << (comb.active()?"":"not ") << "active." <<endreq;


The output for one event is the following:

AnalysisSkeleton     INFO COMB Pass state EF_tau16i_loose_2j23 = 1
AnalysisSkeleton     INFO COMB Number of TauJetCombinations in EF_tau16i_loose_2j23: 1
AnalysisSkeleton     INFO COMB Combination was active.
AnalysisSkeleton     INFO COMB Combination has 1 TauJetContainer Fs and 1 JetCollection Fs
AnalysisSkeleton     INFO COMB In the TauJetContainer are 1 taus and in the JetCollection are 1 jets.
AnalysisSkeleton     INFO COMB Combination has 1 TrigTau Fs and 1 TrigT2Jet Fs.

One would expect two jets, however, as we saw earlier, there is a bug in the JetCollection retrieval from Storegate. As an exercise let's replace the EF chain with the L2 chain L2_tau16i_loose_2j23. In the trigger section in share/AnalysisSkeleton_topOptions.py do

AnalysisSkeleton.InvestigateChain = 'L2_tau16i_loose_2j23'
We get the following, expected, output:

AnalysisSkeleton     INFO COMB Pass state L2_tau16i_loose_2j23 = 1
AnalysisSkeleton     INFO COMB Number of TauJetCombinations in L2_tau16i_loose_2j23: 1
AnalysisSkeleton     INFO COMB Combination was active.
AnalysisSkeleton     INFO COMB TauJetContainer or JetCollection missing.
AnalysisSkeleton     INFO COMB Combination has 1 TrigTau Fs and 2 TrigT2Jet Fs.

TrigDecisionTool in AthenaRootAccess

One of the design features of the new TrigDecisionTool was the complete dual use cababilities in Athena and ARA. Two main steps were:

  • Automatic changes of the Trigger Configuration based on the IOV (like conditions data)
  • Templated access to trigger features

Let's look at an excerpt from the example TrigAnalysisExamples/share/TDTExampleARA.py. Only the main steps are shown.

#!/usr/bin/env python

### import the modules
import ROOT, PyCintex

### define the list of input files

### build the transient event and metadata trees
from TrigDecisionTool.BuildTransientTrees import BuildTransientTrees
(transientTree, transientMetaDataTree) = BuildTransientTrees(PoolAODInput)

### instantiate the TrigDecisionToolARA
tdt = ROOT.Trig.TrigDecisionToolARA(transientTree, transientMetaDataTree)
#import AthenaCommon.Constants as Constants

### loop over the events
nevt = transientTree.GetEntries()

for evt in xrange(nevt):

    # flat list of rois from passed triggers
    l2features = tdt.features('L2_tau16i_loose')

    # flat list of rois from all triggers
    alll2features = tdt.features('L2_tau16i_loose', ROOT.TrigDefs.alsoDeactivateTEs)
    printRoIs(alll2features.get('TrigRoiDescriptor')('initialRoI', ROOT.TrigDefs.alsoDeactivateTEs))

    for f in l2features.getCombinations():
        # if looping through alll2features check f.active()

        # getting TrigRoiDescriptor with label "initialRoi"
        rois = f.get('TrigRoiDescriptor')('initialRoI')

        # getting TrigRoiDescriptor with label "TrigT2CaloTau"
        rois = f.get('TrigRoiDescriptor')('TrigT2CaloTau')

        # getting TrigTauCluster
        clusters = f.get('TrigTauCluster')()

        # getting TrigTau
        taus = f.get('TrigTau')()

    ### composite chain
    features = tdt.features('L2_tau16i_2j23')

    for c in features.getCombinations():
        rois = c.get('TrigRoiDescriptor')('initialRoI')

        taus = c.get('TrigTau')()
        jets = c.get('TrigT2Jet')()

        # yet unclear which RoI corresponds to a Jet (and to which Jet in particular) and which corresponds to tau
        # ancestor() methods comes with help
        tauroi = tdt.ancestor('TrigRoiDescriptor')(taus[0].te(), 'initialRoI')
        for jet in jets:
            jetroi = tdt.ancestor('TrigRoiDescriptor')(jet.te(), 'initialRoI')
            print '        ... roi corresponding to jet of et: ', jet.cptr().et()

We have seen in this example:

  • templated functions, in C++ like fc.get<TrigRoiDescriptor>("initialRoI") work like this in python: fc.get('TrigRoiDescriptor')('initialRoI')
  • running over multiple files with different configurations works seemlessly. However, one still needs to check on each LB change if the prescales have changed
  • otherwise, everything works just like in C++

There are examples how to use the TDT in C++ and in Python for ARA and Athena (in all 4 combinations) in the package Trigger/TrigAnalysis/TrigAnalysisExamples.

Tools to inquire about Trigger Setups

Two web-pages can help you to investigate the trigger menu.


This page http://trigconf.cern.ch/ gives you detailed access to the trigger configuration for any given run. This is currently only possible for real data, but will be possible for MC runs as well in the future.

For now, visit this page, and type in a run-number or range. You get a list of runs together with a link in the L1 Prescale key column. Some runs have multiple links, since it is possible for a run to have different prescale sets. From this link you can browse the entire menu.


This page http://atlas-runquery.cern.ch/ allows you to search for runs fullfilling certain criteria. For instance you can select only runs for which a certain trigger was enabled.

Type in a query, e.g. f r 91890-92070 and tr EF_e5* / sh tr EF_e10*, or choose one from the examples under Trigger.

Missing in this Tutorial: object matching

On important aspect in doing trigger aware analysis is the possibility to match trigger with offline reconstructed objects. This is not covered in this tutorial, but I recommend reading the documentation of a trigger object matching tool that is currently under development.

More Information

For more information, have a look at the links below:

Major updates:
-- JoergStelzer - 19 Jun 2009

%RESPONSIBLE% JoergStelzer
%REVIEW% Never reviewed

-- JoergStelzer - 19-Oct-2009

This topic: Sandbox > TWikiUsers > JoergStelzer > JoergStelzerSandboxCopyTestAttachmentCopy
Topic revision: r2 - 2009-10-19 - JoergStelzer
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback