Notes on L1T DB / O2O

The documentation for L1T O2O has fallen out of date and much DB expertise has left the group. This page is a place for those of us attempting to relearn the legacy system and make adaptions for the upgrade to record what we have learned, proceedures that are working, etc.

There is good legacy documentation here:

* SWGuideL1EmulatorConfiguration

* L1O2OOperations

Online node preliminaries

To login to an online node you need to first obtain an online account, then

ssh cmsusr
sudo -u l1emulator -i 
ssh cms-kvm-31

Legacy O2O Node

ssh l1-o2o.cms

L1-O2O is maintained under the group account l1emulator. You will need also an authentication.xml file, which I am not comfortable posting to this public website. Please contact an expert to get the latest version.

Migration to CMSSW 7 2 X

The manual IOV setting for MC tag procedure has been updated, see notes below (but could use some cleaning up).

The remaining O2O processes are in the process of being transferred to CMSSW 72X and from COND DB V1 to COND DB V2.

The main "workhorse" O2O code is now working, which is triggered by the following top-level calls:

* o2o-setIOV-l1Key.sh

* o2o-tscKey.sh

and has been tested on PREP V2 (Non production DB with new v2 format)

That leaves the validation scripts, which are called in legacy system via crontab:

#MAILTO=""
*/10 * * * * /nfshome0/popcondev/L1Job/validate-l1Key.sh
*/1 * * * * /nfshome0/popcondev/L1Job/validate-o2o-wbm.sh
*/15 * * * * /nfshome0/popcondev/L1Job/o2o-dummy.sh

Going to start with validate-l1Key...

Writing Tables via Config file

To fill local sqlite DB file with CaloParams table filled from config file:

mulhearn@lxplus0038>cmsRun L1Trigger/L1TCalorimeter/python/testWriteCaloParamsTag.py
Begin processing the 1st record. Run 1, Event 1, LumiSection 1 at 31-Jan-2015 07:52:41.837 CET
IOV updated!
Payload token = 0000-00000000

=============================================

MessageLogger Summary

Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
mulhearn@lxplus0038>ls
CondFormats  DataFormats  EventFilter  L1Trigger  l1config.db
 
mulhearn@lxplus0038>cmscond_list_iov -c sqlite:l1config.db -a
L1TCaloParamsRcd_Testing
mulhearn@lxplus0038>cmscond_list_iov -c sqlite:l1config.db -t L1TCaloParamsRcd_Testing
   ==========================================================
   Tag: L1TCaloParamsRcd_Testing
   ==========================================================
   OID: 0001-00000000
   Scope: Unknown
   Description:  
   TimeType: runnumber
   Since                 Till                  Payload OID    Payload Class  
   --------------------  --------------------  -------------  ---------------
                      1  18446744073709551615  0000-00000000  l1t::CaloParams

   Total # of payload objects: 1

New IOV change procedure

The legacy procedure for changing IOVs (e.g. for making MC tags) is no longer valid, as it only affects V1 of the conditions database (which will become obsolete at some point).

The new procedure uses the dropbox mechanism (which updates both V1 and V2). At the moment, this procedure takes the initial payload from V1, because this is the version of the DB which our automated o2o writes. Once our o2o has been updated to CMSSW 72X, it will write V2, and this procedure will change slightly.

1) Find the tag, Payload OID, and starting run for the payload of interest, in the usual way, by looking in O2O log files (details below)

2) List the IOV for the tag of interest

mulhearn@lxplus0057>cmscond_list_iov -c frontier://FrontierProd/CMS_COND_31X_L1T -t L1MuCSCTFConfiguration_CRAFT09v4_hlt | tee temp.txt
   =================================================================
   Tag: L1MuCSCTFConfiguration_CRAFT09v4_hlt
   =================================================================
   OID: 0020-000003D6
   Scope: Unknown
   Description:  
   TimeType: runnumber
   Since                 Till                  Payload OID    Payload Class         
   --------------------  --------------------  -------------  ----------------------
                      1                 65673  0021-00000010  L1MuCSCTFConfiguration
                  65674                105710  0021-00000011  L1MuCSCTFConfiguration
                 105711                110261  0021-0000001D  L1MuCSCTFConfiguration
                 110262                110267  0021-00000023  L1MuCSCTFConfiguration
...
                 229567                229568  0021-000000AB  L1MuCSCTFConfiguration
                 229569            4294967295  0021-000000AA  L1MuCSCTFConfiguration

   Total # of payload objects: 3243

Our payload of interest is there starting at Run 229567 with Payload OID 0021-00000AB.

2) Extract only the payload of interest to an sqlite file by using the run range, and list the resulting IOV:

cmscond_export_iov -s oracle://cms_orcon_adg/CMS_COND_31X_L1T -b 229567 -e 229568 -t L1MuCSCTFConfiguration_CRAFT09v4_hlt -d sqlite:229567.db
cmscond_list_iov -c sqlite:229567.db -t L1MuCSCTFConfiguration_CRAFT09v4_hlt
   =================================================================
   Tag: L1MuCSCTFConfiguration_CRAFT09v4_hlt
   =================================================================
   OID: 0000-00000000
   Scope: Tag
   Description:  
   TimeType: runnumber
   Since                 Till                  Payload OID    Payload Class         
   --------------------  --------------------  -------------  ----------------------
                 229567            4294967295  0001-00000000  L1MuCSCTFConfiguration

Note there is only one payload but the IOV is stil not valid for MC as it only begins at 229567

3) Duplicate the IOV and change the starting point to run 1, in the sqlite file, and check that it is correct:

cmscond_duplicate_iov -c sqlite:229567.db -t L1MuCSCTFConfiguration_CRAFT09v4_hlt -d l1mu_mc -f 229567 -s 1
cmscond_list_iov -c sqlite:229567.db -t l1mu_mc
   =================================================================
   Tag: l1mu_mc
   =================================================================
   OID: 0000-00000001
   Scope: Unknown
   Description:  
   TimeType: runnumber
   Since                 Till                  Payload OID    Payload Class         
   --------------------  --------------------  -------------  ----------------------
                      1            4294967295  0001-00000000  L1MuCSCTFConfiguration

4) Now upload to DB using the dropbox mechanism.

Copy from Preb to Prod DB via Dropbox Mechanism

1) Show all tags in PREP:

cmscond_list_iov -c frontier://FrontierPrep/CMS_COND_31X_L1T -a

2) Show the IOV for tag of interest (if invalid, use procedure above to fix...)

cmscond_list_iov -c frontier://FrontierPrep/CMS_COND_L1T -t L1RCTParametersRcd_L1TDevelCollisionsV1_2

3) Export this tag to an sqlite file, changing the tag name in the process (and check contents):

cmscond_export_iov -s frontier://FrontierPrep/CMS_COND_L1T -i L1RCTParametersRcd_L1TDevelCollisions_ExtendedScaleFactorsV1 -d sqlite_file:L1RCTParameters_Collisions_ExtendedScaleFactors_v0_mc.db -t L1RCTParameters_Collisions_ExtendedScaleFactors_v0_mc
cmscond_export_iov -s frontier://FrontierPrep/CMS_COND_L1T -i L1RCTParametersRcd_L1TDevelCollisionsV1_2 -d sqlite_file:L1RCTParameters_Collisions_NewHoeEic_v0_mc.db -t 
L1RCTParameters_Collisions_NewHoeEic_v0_mc
cmscond_list_iov -c sqlite_file:L1RCTParameters_Collisions_ExtendedScaleFactors_v0_mc.db -t L1RCTParameters_Collisions_ExtendedScaleFactors_v0_mc

4) Now this sqlite file can be uploaded via the dropbox mechanism. (Use upload.py , using the latest dropbox upload.py file, then follow prompts for building the meta data). For MC tags, we use offline synchronization scheme, so the meta data should look something like this:

{
    "destinationDatabase": "oracle://cms_orcon_prod/CMS_COND_31X_L1T", 
    "destinationTags": {
        "L1RCTParameters_Collisions_NewHoeEic_v0_mc": {
            "dependencies": {}, 
            "synchronizeTo": "offline"
        }
    }, 
    "inputTag": "L1RCTParameters_Collisions_NewHoeEic_v0_mc", 
    "since": null, 
    "userText": "New HoverE and EIC parameters in preparation for 2015"
}

After generating one such file from the prompts, you can edit it by hand in the future and avoid the prompts. For fileXXX.db you save it as fileXXX.txt and the upload.py script runs non-interactively.

PopCon

The automated L1 020 processes appear to be running successfully from ~popcondev/L1Job. The most recently updated logfiles (updated a few days ago) are here:

-rw-rw-r-- 1 l1emulator l1emulator 71M Oct 1 09:58 o2o-setIOV-012.log
-rw-rw-r-- 1 l1emulator l1emulator 48M Oct 1 12:37 validate-o2o-wbm-011.log
-rw-rw-r-- 1 l1emulator l1emulator 2.8M Oct 1 17:00 o2o-tscKey-011.log
-rw-rw-r-- 1 l1emulator l1emulator 46M Oct 1 17:04 validate-l1Key-011.log
-rw-rw-r-- 1 l1emulator l1emulator 290M Oct 3 12:15 o2o-dummy-012.log

I simple grep of o2o-setIOV-012.log, shows that the two CSCTF runs with new configurations appear to have been O2O 'ed successfully.

The L1 O2O appears to be running under CMSSW_4_2_3_ONLINE. The most recent version of CMSSW I can find on this SL6 node is CMSSW_5_2_6

> grep 201409201 o20-setiov-012.log
...
Producing L1MuCSCTFConfiguration with key CSCTF_KEY=201409201
...

Making MC tags

The instructions here work just great:

* L1MCTags

We want to build a new MC global tag which uses new CSCTF configuration (for collisions) using tag 201409202

I checking o2o-setiov-012.log in ~popcondev/L1Job directory for tag 201409202. Note that as expected for L1 o2o, the existing payloads are noted as "already in L1TriggerKeyList ", until we come to L1MuCSCTFConfigurationRCD:

Begin processing the 1st record. Run 1, Event 1, LumiSection 1 at 24-Sep-2014 15:13:04.842 CEST
Begin processing the 1st record. Run 1, Event 1, LumiSection 1 at 24-Sep-2014 15:13:04.842 CEST
TSC KEY TSC_20140924_003066_cosmics_BASE
CSCTF_KEY 201409202
DTTF_KEY dttf13_TSC_03_bot
...
L1CondDBPayloadWriter: object key GCTPhysics_2012_04_27_JetSeedThresh5GeV for L1HfRingEtScaleRcd@L1CaloEtScale already in L1TriggerKeyList
L1CondDBPayloadWriter: object key GCTPhysics_2012_04_27_JetSeedThresh5GeV for L1HtMissScaleRcd@L1CaloEtScale already in L1TriggerKeyList
L1CondDBPayloadWriter: object key GCTPhysics_2012_04_27_JetSeedThresh5GeV for L1JetEtScaleRcd@L1CaloEtScale already in L1TriggerKeyList
L1CondDBPayloadWriter: object key 34 for L1MuCSCPtLutRcd@L1MuCSCPtLut already in L1TriggerKeyList
object key for L1MuCSCTFConfigurationRcd@L1MuCSCTFConfiguration: 201409202
%MSG-i L1-O2O: CSCTFConfigOnlineProd:   L1CondDBPayloadWriter:L1CondDBPayloadWriter 24-Sep-2014 15:13:06 CEST  Run: 1 Event: 1
Producing L1MuCSCTFConfiguration with key CSCTF_KEY=201409202
%MSG
%MSG-i L1-O2O: CSCTFConfigOnlineProd:   L1CondDBPayloadWriter:L1CondDBPayloadWriter 24-Sep-2014 15:13:06 CEST  Run: 1 Event: 1

SP KEY: 2014092020001

STATIC CONFIGURATION:
ACT_HR VM MA 0x0001  
CSR_SCC SP MA 0x02a2 
CSR_REQ SP MA 0x8000 
CSR_DFC DD MA 0x0fff 
CSR_MWC SP MA 0x0002 
CSR_BSY VM MA 0x4100 
CSR_RDY VM MA 0x4100 
CSR_OSY VM MA 0x7f00 
CSR_OSY FA MA 0x8000 
CSR_WOF VM MA 0x4100 
CSR_FCC VM MA 0x0110 
...
L1MuCSCTFConfigurationRcd@L1MuCSCTFConfiguration PAYLOAD TOKEN [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A2]
...
Object key for L1MuCSCTFConfigurationRcd@L1MuCSCTFConfiguration: 201409202
L1MuCSCTFConfigurationRcd PAYLOAD TOKEN [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A2]
L1MuCSCTFConfigurationRcd L1MuCSCTFConfiguration_CRAFT09v4_hlt SINCE 226714

There at the end is our payload token and tag name, which we need for the next steps.

cmscond_list_iov -c oracle://cms_orcon_prod/CMS_COND_31X_L1T -P /nfshome0/popcondev/conddb/ -t L1MuCSCTFConfiguration_CRAFT09v4_hlt | tee tmp.txt
[l1emulator@srv-C2C05-15 o2o]$ cmscond_list_iov -c oracle://cms_orcon_prod/CMS_COND_31X_L1T -P /nfshome0/popcondev/conddb/ -t L1MuCSCTFConfiguration_CRAFT09v4_hlt 
Tag L1MuCSCTFConfiguration_CRAFT09v4_hlt
TimeType runnumber
PayloadContainerName L1MuCSCTFConfiguration
since     till     payloadToken
1     65673     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000010]
65674     105710     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000011]
105711     110261     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-0000001D]
110262     110267     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000023]
110268     110283     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-0000001D]
110284     110289     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000023]
110290     111143     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000024]
111144     111144     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000025]
111145     111145     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000026]
111146     111368     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000027]
111369     111411     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-00000024]
...
226459     226510     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A0]
226511     226713     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A1]
226714     227355     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A2]
227356     228409     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A1]
228410     228410     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A2]
228411     228414     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A1]
228415     228415     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A2]
...
Total # of payload objects: 3240

And there you can see the IOVs for older tables, with finally this payload showing up run 226714. Now all we need to do is edit this file by hand to make this table valid for all time, then push the new IOV to the DB with a new tag. I edit file to look like this:

Tag L1MuCSCTFConfiguration_L1TDevelCollisionsV1
TimeType runnumber
PayloadContainerName L1MuCSCTFConfiguration
since     till     payloadToken
1    4294967295     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A2]
Total # of payload objects: 1

Now repeating for cosmics, which has CSCTFTAG 201409201... relevant lines from log file are:

Object key for L1MuCSCTFConfigurationRcd@L1MuCSCTFConfiguration: 201409201
L1MuCSCTFConfigurationRcd PAYLOAD TOKEN [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A1]
L1MuCSCTFConfigurationRcd L1MuCSCTFConfiguration_CRAFT09v4_hlt SINCE 226511

And after following same procedure as above:

[l1emulator@srv-C2C05-15 o2o]$ cmscond_list_iov -c oracle://cms_orcon_prod/CMS_COND_31X_L1T -P /nfshome0/popcondev/conddb/ -t L1MuCSCTFConfiguration_L1TDevelCosmicsV1
Tag L1MuCSCTFConfiguration_L1TDevelCosmicsV1
TimeType runnumber
PayloadContainerName L1MuCSCTFConfiguration
since     till     payloadToken
1     4294967295     [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000000A1]
Total # of payload objects: 1

UPDATES... Relevant log files snippets for collisions configuration:

L1CondDBPayloadWriter: object key 34 for L1MuCSCPtLutRcd@L1MuCSCPtLut already in L1TriggerKeyList
object key for L1MuCSCTFConfigurationRcd@L1MuCSCTFConfiguration: 201411103
%MSG-i L1-O2O: CSCTFConfigOnlineProd:   L1CondDBPayloadWriter:L1CondDBPayloadWriter 10-Nov-2014 17:02:33 CET  Run: 1 Event: 1
Producing L1MuCSCTFConfiguration with key CSCTF_KEY=201411103
...
L1MuCSCTFConfigurationRcd@L1MuCSCTFConfiguration PAYLOAD TOKEN [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=000
00B01][OID=00000021-000000AB]
...
Begin processing the 1st record. Run 229567, Event 1, LumiSection 1 at 10-Nov-2014 17:02:35.748 CET
...
Object key for L1MuCSCPtLutRcd@L1MuCSCPtLut: 34
L1MuCSCPtLutRcd PAYLOAD TOKEN [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCPtLut][CLID=1A83A789-CC9F-A1B8-53B6-E888025ECA89][TECH=00000B01][OID=00000010-00000023]
IOV already up to date.
Object key for L1MuCSCTFConfigurationRcd@L1MuCSCTFConfiguration: 201411103
L1MuCSCTFConfigurationRcd PAYLOAD TOKEN [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCTFConfiguration][CLID=EE91EF5C-8A14-2B89-0D1C-E3884D707F8B][TECH=00000B01][OID=00000021-000
000AB]

Note that the PT Lut appears to have been already O2O 'ed. To see when and get the tag I merely search for first instance of "Object key for L1MuCSCPtLutRcd @L1MuCSCPtLut: 34" to find:

Begin processing the 1st record. Run 226511, Event 1, LumiSection 1 at 22-Sep-2014 19:08:37.057 CEST
...
Object key for L1MuCSCPtLutRcd@L1MuCSCPtLut: 34
L1MuCSCPtLutRcd PAYLOAD TOKEN [DB=00000000-0000-0000-0000-000000000000][CNT=L1MuCSCPtLut][CLID=1A83A789-CC9F-A1B8-53B6-E888025ECA89][TECH=00000B01][OID=00000010-00000023]
L1MuCSCPtLutRcd L1MuCSCPtLut_CRAFT09v2_hlt SINCE 226511

This is everything I need to follow the recipe above to set the IOV.

Viewing Trigger key list associated with a run

This job runs successfully under both 4_2_3 and 5_3_6

cmsRun $CMSSW_RELEASE_BASE/src/CondTools/L1Trigger/test/l1o2otestanalyzer_cfg.py inputDBConnect=oracle://cms_orcon_prod/CMS_COND_31X_L1T inputDBAuth=/nfshome0/popcondev/conddb_taskWriters/L1T printL1TriggerKey=1 runNumber=XXXX

Extracting conditions associated with a run

This works under 4_2_3. At the moment, I have to copy by hand CondTools /L1Trigger from the release to my local area since I can't get either git or cvs to work on the online nodes.

/nfshome0/cmssw2/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_2_3/src/CondTools/L1Trigger/scripts/getConditions-runnum.sh -p XXXX

But fails in 5_3_6 due to:

   cmscond_schema_metadata --create -P . -c sqlite_file:l1config.db
sh: cmscond_schema_metadata: command not found

(Although happily goes on and does other steps...) -- MichaelMulhearn - 03 Oct 2014</verbatim>

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2015-01-31 - MichaelMulhearn
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback