LHCb Bookkeeping commands

The LHCb Bookkeeping allows many queries from the command line interface. Many of them use the DMS option switches. Note that the output of many commands can be piped into others as explained in the DMS twiki.

Before executing the commands in this tutorial

Since at least v8r3p2, doing

> SetupProject LHCbDirac
no longer works to set the environment. If you want to work with LHCbDirac in bash, instead open a sub-shell with
> lb-run LHCbDirac/prod bash --norc
Once this is done, the following commands should work within the bash sub-shell you just opened.

Information on files

dirac-bookkeeping-get-files

Get a list of files for a BK query. The list of files can be saved in a file or piped to an other DMS command.

Syntax

 $ dirac-bookkeeping-get-files --help
  Retrieve files of a BK query
Usage:
  dirac-bookkeeping-get-files [option|cfgfile] ... ProdID Type 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -P:  --Productions=    :    Production ID to search (comma separated list) 
  -f:  --FileType=       :    File type (comma separated list, to be used with --Production) [All] 
  -    --ExceptFileType= :    Exclude the (list of) file types when all are requested 
  -B:  --BKQuery=        :    Bookkeeping query path 
  -r:  --Runs=           :    Run or range of runs (r1:r2) 
  -    --DQFlags=        :    DQ flag used in query 
  -    --StartDate=      :    Start date for the BK query 
  -    --EndDate=        :    End date for the BK query 
  -    --Visibility=     :    Set visibility (Yes, No, All) [Yes] 
  -    --Output=         :   Specify a file that will contain the list of files 
  -    --MaxFiles=       :    Print only <MaxFiles> lines on stdout (20 if output, else All)

Example

$ dirac-bookkeeping-get-files --Prod 24847 --FileType FULL.DST --Output myTest.txt
Using BKQuery: {'Visible': 'Yes', 'FileType': 'FULL.DST', 'Production': [24847]}
110 files found (showing only first 20 files):
FileName                                                                                             Size       GUID                                     Replica  Visible  
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000001_1.full.dst                           4815735657 8E8F17D3-D7D6-E211-B39C-003048F17956     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000002_1.full.dst                           4777687666 90301242-D3D8-E211-AC15-002590352F86     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000003_1.full.dst                           4882358770 6093D440-C6D8-E211-AC36-0025901D5CEE     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000004_1.full.dst                           4862398870 E80B1DFB-2ADA-E211-A6E6-003048F1DB84     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000005_1.full.dst                           4812022999 783B2687-2BDA-E211-83D2-BCAEC5579F00     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000006_1.full.dst                           4819311669 A4687FD0-D1D8-E211-ADDF-0026B93B4115     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000007_1.full.dst                           4882275421 98FDF292-C2D8-E211-BD17-7845C4FC3608     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000009_1.full.dst                           4825949667 BAE21F92-C1D8-E211-A192-E0CB4E4408DD     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000010_1.full.dst                           4280255461 622237D4-29DA-E211-A063-003048F00B44     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000011_1.full.dst                           4917138693 F8F79D7D-D2D8-E211-A551-0025903534FE     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000012_1.full.dst                           4853093706 D44CF54B-CCD8-E211-97BC-003048C45C8A     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000013_1.full.dst                           5026741560 16091F6A-CCD8-E211-A48F-001B21ADD058     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000014_1.full.dst                           2706044397 D6B1059F-CFD8-E211-A717-90E2BA034BAC     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000015_1.full.dst                           4834844058 48D003D7-D1D8-E211-9327-0025903531D0     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000016_1.full.dst                           4870141544 F27E00C4-CBD8-E211-9928-1803730B049D     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000017_1.full.dst                           4883526421 2ADA99CC-2BDA-E211-9E92-00259094140E     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000018_1.full.dst                           4912126905 64A7D618-D4D8-E211-BACB-0026B93B3E92     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000019_1.full.dst                           4862828212 7A118B85-D3D8-E211-B025-00259035AE22     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000020_1.full.dst                           1718014956 BCB0438E-CCD8-E211-A547-3640B5BA49AC     Yes      ?        
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000021_1.full.dst                           4841828540 000DA63E-CED8-E211-BC66-1803730B5105     Yes      ?        

List of 110 files saved in myTest.txt

dirac-bookkeeping-file-metadata

Retrieves the BK metadata of a (list of) file(s). By default only a summary is given while with --Full option one gets all information

Syntax

$ dirac-bookkeeping-file-metadata --help
  Retrieve metadata from the Bookkeeping for the given files
Usage:
  dirac-bookkeeping-file-metadata [option|cfgfile] ... LFN|File
Arguments:
  LFN:      Logical File Name
  File:     Name of the file with a list of LFNs 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin) 
  -    --Full            :    Print out all metadata 

Examples

$ dirac-bookkeeping-file-metadata /lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000001_1.full.dst 
FileName                                                                     Size       GUID                                   Replica  DataQuality  RunNumber  #events   
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000001_1.full.dst   4815735657 8E8F17D3-D7D6-E211-B39C-003048F17956   Yes      OK           115327     49797     

$ dirac-bookkeeping-file-metadata /lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000001_1.full.dst --Full
FileName        /lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000001_1.full.dst
ADLER32         None
CreationDate    None
DQFlag          OK
EventStat       49797
EventType       90000000
FileSize        4815735657
FileType        FULL.DST
FullStat        None
GUID            8E8F17D3-D7D6-E211-B39C-003048F17956
GotReplica      Yes
InsertTimeStamp 2013-06-25 11:45:39.300944
InstLuminosity  0
JobId           129552274
Luminosity      4085.68992445
MD5SUM          515cec21800944c0947da0088fd4b2cb
RunNumber       115327

dirac-bookkeeping-genXMLCatalog

Allows to get an XML catalog from a list of files or from a Gaudi options file (it can also create a new options file containing PFNs).

Syntax

$ dirac-bookkeeping-genXMLCatalog --help
  Create a pool xml catalog slice for the specified LFNs
Usage:
  dirac-bookkeeping-genXMLCatalog [option|cfgfile] ...  
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -g:  --Sites=          :   Sites to consider [ALL] (comma separated list) 
  -S:  --SEs=            :   SEs to consider [ALL] (comma separated list) 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin) 
  -    --Options=        :    List of option files to consider 
  -    --NewOptions=     :    Name of a new options file to be generated with LFNs (default: none) 
  -    --Catalog=        :    Catalog name (default: pool_xml_catalog.xml 
  -    --Depth=          :    Depth for ancestor consideration (default: none) 
  -v   --Verbose         :    Verbose gLogger.always(out 
  -    --Ignore          :    Ignore missing files 
  -    --GaudiVerbose    :    Set Gaudi verbose when parsing option files 

Example

Creating an XML catalog

 $ cat example.py
from Gaudi.Configuration import *
from GaudiConf import IOHelper
IOHelper().inputFiles([
'LFN:/lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00022761/0001/00022761_00018925_1.bhadroncompleteevent.dst'
], clear=True)
$ dirac-bookkeeping-genXMLCatalog --File=example.py --Catalog /dev/stdout
<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<!-- Edited By PoolXMLCatalog.py -->
<!DOCTYPE POOLFILECATALOG SYSTEM "InMemory">
<POOLFILECATALOG>


  <File ID="8E7DCD3B-3680-E211-8528-003048F00B24">
     <physical>
       <pfn filetype="ROOT" name="root://eoslhcb.cern.ch//eos/lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00022761/0001/00022761_00018925_1.bhadroncompleteevent.dst"/>
     </physical>
     <logical>
       <lfn name="/lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00022761/0001/00022761_00018925_1.bhadroncompleteevent.dst"/>
     </logical>
   </File>

</POOLFILECATALOG>
POOL XML catalog /dev/stdout successfully created 

Creating a new options file with PFNs:

[localhost, Patch] ~ $ dirac-bookkeeping-genXMLCatalog --Option example.py --NewOption examplePFN.py
New options file examplePFN.py successfully created 
[localhost, Patch] ~ $ cat examplePFN.py 
from Gaudi.Configuration import *
from GaudiConf import IOHelper
IOHelper().inputFiles([
'root://eoslhcb.cern.ch//eos/lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00022761/0001/00022761_00018925_1.bhadroncompleteevent.dst'
], clear=True)

dirac-bookkeeping-file-path

Allows to get the BK path for a file or list of files. It is quite handy to know what is the origin of that file.

Syntax

$ dirac-bookkeeping-file-path --help
  Return the BK path for the directories of a (list of) files
Usage:
  dirac-bookkeeping-file-path [option|cfgfile] ... LFN|File
Arguments:
  LFN:      Logical File Name
  File:     Name of the file with a list of LFNs 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin) 
  -    --Full            :    Print out full BK dictionary (default: print out BK path) 
  -    --GroupBy=        :    Return a list of files per <metadata item> 
  -    --GroupByPath     :    Return a list of files per BK path 
  -    --GroupByProduction :    Return a list of files per production 

Examples

$ dirac-bookkeeping-file-path /lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000001_1.full.dst
Failed : []
Successful : 
    /lhcb/LHCb/Collision12/FULL.DST/00024847/0000 : /LHCb/Collision12/Beam4000GeV-VeloClosed-MagDown/Real Data/Reco14/90000000/FULL.DST

[localhost, Patch] ~ $ cat files.txt
        /lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00024034/0000/00024034_00002238_1.bhadroncompleteevent.dst
        /lhcb/LHCb/Collision11/CHARM.MDST/00024034/0000/00024034_00002240_1.charm.mdst
        /lhcb/LHCb/Collision11/CHARMCOMPLETEEVENT.DST/00024034/0000/00024034_00002242_1.charmcompleteevent.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00001915_1.dimuon.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00002244_1.dimuon.dst
        /lhcb/LHCb/Collision11/EW.DST/00024034/0000/00024034_00002246_1.ew.dst
        /lhcb/LHCb/Collision11/LEPTONIC.MDST/00024034/0000/00024034_00002248_1.leptonic.mdst
        /lhcb/LHCb/Collision11/PID.MDST/00024034/0000/00024034_00002250_1.pid.mdst
        /lhcb/LHCb/Collision11/RADIATIVE.DST/00024034/0000/00024034_00002252_1.radiative.dst
        /lhcb/LHCb/Collision11/SEMILEPTONIC.DST/00024034/0000/00024034_00002254_1.semileptonic.dst

$ dirac-bookkeeping-file-path --File files.txt --GroupByPath
Successful : 
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/BHADRONCOMPLETEEVENT.DST : 
        /lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00024034/0000/00024034_00002238_1.bhadroncompleteevent.dst
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/CHARM.MDST : 
        /lhcb/LHCb/Collision11/CHARM.MDST/00024034/0000/00024034_00002240_1.charm.mdst
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/CHARMCOMPLETEEVENT.DST : 
        /lhcb/LHCb/Collision11/CHARMCOMPLETEEVENT.DST/00024034/0000/00024034_00002242_1.charmcompleteevent.dst
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/DIMUON.DST : 
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00001915_1.dimuon.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00002244_1.dimuon.dst
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/EW.DST : 
        /lhcb/LHCb/Collision11/EW.DST/00024034/0000/00024034_00002246_1.ew.dst
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/LEPTONIC.MDST : 
        /lhcb/LHCb/Collision11/LEPTONIC.MDST/00024034/0000/00024034_00002248_1.leptonic.mdst
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/PID.MDST : 
        /lhcb/LHCb/Collision11/PID.MDST/00024034/0000/00024034_00002250_1.pid.mdst
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/RADIATIVE.DST : 
        /lhcb/LHCb/Collision11/RADIATIVE.DST/00024034/0000/00024034_00002252_1.radiative.dst
    /LHCb/Collision11/Beam3500GeV-VeloClosed-MagUp/Real Data/Reco14/Stripping20r1p1/90000000/SEMILEPTONIC.DST : 
        /lhcb/LHCb/Collision11/SEMILEPTONIC.DST/00024034/0000/00024034_00002254_1.semileptonic.dst

$ dirac-bookkeeping-file-path --File files.txt --GroupByProduction
Successful : 
    Production 24034 : 
        /lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00024034/0000/00024034_00002238_1.bhadroncompleteevent.dst
        /lhcb/LHCb/Collision11/CHARM.MDST/00024034/0000/00024034_00002240_1.charm.mdst
        /lhcb/LHCb/Collision11/CHARMCOMPLETEEVENT.DST/00024034/0000/00024034_00002242_1.charmcompleteevent.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00001915_1.dimuon.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00002244_1.dimuon.dst
        /lhcb/LHCb/Collision11/EW.DST/00024034/0000/00024034_00002246_1.ew.dst
        /lhcb/LHCb/Collision11/LEPTONIC.MDST/00024034/0000/00024034_00002248_1.leptonic.mdst
        /lhcb/LHCb/Collision11/PID.MDST/00024034/0000/00024034_00002250_1.pid.mdst
        /lhcb/LHCb/Collision11/RADIATIVE.DST/00024034/0000/00024034_00002252_1.radiative.dst
        /lhcb/LHCb/Collision11/SEMILEPTONIC.DST/00024034/0000/00024034_00002254_1.semileptonic.dst

$ dirac-bookkeeping-file-path --File files.txt --GroupBy RunNumber
Failed : []
Successful : 
    RunNumber 100254 : 
        /lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00024034/0000/00024034_00002238_1.bhadroncompleteevent.dst
        /lhcb/LHCb/Collision11/CHARM.MDST/00024034/0000/00024034_00002240_1.charm.mdst
        /lhcb/LHCb/Collision11/CHARMCOMPLETEEVENT.DST/00024034/0000/00024034_00002242_1.charmcompleteevent.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00001915_1.dimuon.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00002244_1.dimuon.dst
        /lhcb/LHCb/Collision11/EW.DST/00024034/0000/00024034_00002246_1.ew.dst
        /lhcb/LHCb/Collision11/LEPTONIC.MDST/00024034/0000/00024034_00002248_1.leptonic.mdst
        /lhcb/LHCb/Collision11/PID.MDST/00024034/0000/00024034_00002250_1.pid.mdst
        /lhcb/LHCb/Collision11/RADIATIVE.DST/00024034/0000/00024034_00002252_1.radiative.dst
        /lhcb/LHCb/Collision11/SEMILEPTONIC.DST/00024034/0000/00024034_00002254_1.semileptonic.dst

Ancestors and descendants

dirac-bookkeeping-get-file-ancestors

Gets a list of ancestors for a given file or list of files. One can define the level up to which to go up as well as also get files without replicas.

Syntax

$ dirac-bookkeeping-get-file-ancestors --help
  Report ancestors for the given LFNs
Usage:
  dirac-bookkeeping-get-file-ancestors [option|cfgfile] ... [LFN|File] [Level]
Arguments:
  LFN:      Logical File Name
  File:     Name of the file with a list of LFNs
  Level:    Number of levels to search (default: 1) 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin) 
  -    --All             : Do not restrict to descendants with replicas 
  -    --Full            : Get full metadata information on ancestors 
  -    --Depth=          : Number of processing levels (default:1) 

Examples

$ dirac-bookkeeping-get-file-ancestors /lhcb/LHCb/Collision11/CHARMTOBESWUM.DST/00022760/0008/00022760_00088328_1.CharmToBeSwum.dst 2
Failed : []
Successful : 
    /lhcb/LHCb/Collision11/CHARMTOBESWUM.DST/00022760/0008/00022760_00088328_1.CharmToBeSwum.dst : 
        /lhcb/LHCb/Collision11/FULL.DST/00022754/0006/00022754_00060707_1.full.dst : Replica-Yes
            /lhcb/data/2011/RAW/FULL/LHCb/COLLISION11/100254/100254_0000000130.raw : Replica-Yes

 $ dirac-bookkeeping-get-file-ancestors /lhcb/LHCb/Collision11/CHARMTOBESWUM.DST/00022760/0008/00022760_00088328_1.CharmToBeSwum.dst 
Failed : []
Successful : 
    /lhcb/LHCb/Collision11/CHARMTOBESWUM.DST/00022760/0008/00022760_00088328_1.CharmToBeSwum.dst : 
        /lhcb/LHCb/Collision11/FULL.DST/00022754/0006/00022754_00060707_1.full.dst : Replica-Yes

$ dirac-bookkeeping-get-file-ancestors /lhcb/LHCb/Collision11/CHARMTOBESWUM.DST/00022760/0008/00022760_12388328_1.CharmToBeSwum.dst
Failed : 
    /lhcb/LHCb/Collision11/CHARMTOBESWUM.DST/00022760/0008/00022760_12388328_1.CharmToBeSwum.dst

dirac-bookkeeping-get-file-descendants

Gets a list of descendants for a given file or list of files. One can define the level up to which to go down as well as also get files without replicas. One may also provide a production number for which to look at in processing chain.

Syntax

$ dirac-bookkeeping-get-file-descendants  --help
  Report descendants for the given LFNs
Usage:
  dirac-bookkeeping-get-file-descendants [option|cfgfile] ... [LFN|File] [Level]
Arguments:
  LFN:      Logical File Name
  File:     Name of the file with a list of LFNs
  Level:    Number of levels to search (default: 1) 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin) 
  -    --All             : Do not restrict to descendants with replicas 
  -    --Full            : Get full metadata information on ancestors 
  -    --Depth=          : Number of processing levels (default:1) 
  -    --Production=     : Restrict to descendants in a given production (at any depth) 

Examples


dirac-bookkeeping-get-file-sisters

Gets a list of files that have a common parent (default, therefore looking for sisters) or ancestor at any level (using --Depth, in this case cousins at various levels), by default in the same production as the file itself. One can also ask for a sister/cousin in an other transformation. The option --AllFileTypes allows to get files of a different type (by default one checks only files of the same type).

Syntax

$ dirac-bookkeeping-get-file-sisters --help
  Report sisters (i.e. descendant of ancestor) for the given LFNs
Usage:
  dirac-bookkeeping-get-file-sisters [option|cfgfile] ... [LFN|File] [Level] 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin) 
  -    --All             : Do not restrict to sisters with replicas 
  -    --Full            : Get full metadata information on sisters 
  -    --Depth=          : Number of ancestor levels (default:1), 2 would be cousins, 3 grand-cousins etc... 
  -    --Production=     : Production to check for sisters (default=same production) 
  -    --AllFileTypes    : Consider also files with a different type 

Examples:

$ dirac-bookkeeping-get-file-sisters /lhcb/LHCb/Collision11/BHADRON.MDST/00024034/0000/00024034_00002236_1.bhadron.mdst 
NoSister : 
    /lhcb/LHCb/Collision11/BHADRON.MDST/00024034/0000/00024034_00002236_1.bhadron.mdst

$ dirac-bookkeeping-get-file-sisters /lhcb/LHCb/Collision11/BHADRON.MDST/00024034/0000/00024034_00002236_1.bhadron.mdst --Depth 2 --AllFileTypes
NoCousin : []
Successful : 
    /lhcb/LHCb/Collision11/BHADRON.MDST/00024034/0000/00024034_00002236_1.bhadron.mdst : 
        /lhcb/LHCb/Collision11/BHADRONCOMPLETEEVENT.DST/00024034/0000/00024034_00002238_1.bhadroncompleteevent.dst
        /lhcb/LHCb/Collision11/CHARM.MDST/00024034/0000/00024034_00002240_1.charm.mdst
        /lhcb/LHCb/Collision11/CHARMCOMPLETEEVENT.DST/00024034/0000/00024034_00002242_1.charmcompleteevent.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00001915_1.dimuon.dst
        /lhcb/LHCb/Collision11/DIMUON.DST/00024034/0000/00024034_00002244_1.dimuon.dst
        /lhcb/LHCb/Collision11/EW.DST/00024034/0000/00024034_00002246_1.ew.dst
        /lhcb/LHCb/Collision11/LEPTONIC.MDST/00024034/0000/00024034_00002248_1.leptonic.mdst
        /lhcb/LHCb/Collision11/PID.MDST/00024034/0000/00024034_00002250_1.pid.mdst
        /lhcb/LHCb/Collision11/RADIATIVE.DST/00024034/0000/00024034_00002252_1.radiative.dst
        /lhcb/LHCb/Collision11/SEMILEPTONIC.DST/00024034/0000/00024034_00002254_1.semileptonic.dst

Statistics

dirac-bookkeeping-get-stats

Returns statistics for a given BK query (number of files, of events, luminosity…)

Syntax

$ dirac-bookkeeping-get-stats --help
   Get statistical information on a dataset
Usage:
  dirac-bookkeeping-get-stats [option|cfgfile] 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -P:  --Productions=    :    Production ID to search (comma separated list) 
  -f:  --FileType=       :    File type (comma separated list, to be used with --Production) [All] 
  -    --ExceptFileType= :    Exclude the (list of) file types when all are requested 
  -B:  --BKQuery=        :    Bookkeeping query path 
  -r:  --Runs=           :    Run or range of runs (r1:r2) 
  -    --DQFlags=        :    DQ flag used in query 
  -    --StartDate=      :    Start date for the BK query 
  -    --EndDate=        :    End date for the BK query 
  -    --Visibility=     :    Set visibility (Yes, No, All) [Yes] 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin) 

Examples

Unique dataset:

$ dirac-bookkeeping-get-stats --BK /LHCb/Collision12//RealData/Reco14/Stripping20r0p1//BHADRON.MDST
For BK query: {'ConfigName': 'LHCb', 'EventType': '90000000', 'FileType': 'BHADRON.MDST', 'ProcessingPass': '/Real Data/Reco14/Stripping20r0p1', 'Visible': 'Yes', 'ConfigVersion': 'Collision12'}
Getting info from filesSummary...
Nb of Files      : 4'322
Nb of Events     : 43'844'045
Total size       : 329.150 GB (7.5 kB per evt)
Luminosity       : 1.657 /fb
Size  per /pb    : 0.2 GB
Files per /fb    : 2609.1

$ dirac-bookkeeping-get-stats --BK /LHCb/Collision12//RealData/Reco14//FULL.DST
For BK query: {'ConfigName': 'LHCb', 'EventType': '90000000', 'FileType': 'FULL.DST', 'ProcessingPass': '/Real Data/Reco14', 'Visible': 'Yes', 'ConfigVersion': 'Collision12'}
Getting info from filesSummary...
Nb of Files      : 584'518
Nb of Events     : 27'814'353'411
Total size       : 2.777 PB (99.9 kB per evt)
Luminosity       : 2.043 /fb
Size  per /pb    : 1359.2 GB
Files per /fb    : 286044.6

Multiple dataset (several file types). In that case an average luminosity is reported as different datasets may have different luminosities.

$ dirac-bookkeeping-get-stats --BK /LHCb/Collision12//RealData/Reco14/Stripping20r0p1//ALL.MDST,ALL.DST
For BK query: {'ConfigName': 'LHCb', 'EventType': '90000000', 'FileType': ['BHADRON.MDST', 'PID.MDST', 'LEPTONIC.MDST', 'CHARM.MDST', 'RADIATIVE.DST', 'BHADRONCOMPLETEEVENT.DST', 'EW.DST', 'CALIBRATION.DST', 'CHARMCOMPLETEEVENT.DST', 'DIMUON.DST', 'SEMILEPTONIC.DST'], 'ProcessingPass': '/Real Data/Reco14/Stripping20r0p1', 'Visible': 'Yes', 'ConfigVersion': 'Collision12'}
Getting info from filesSummary...
Nb of Files      : 54'696
Nb of Events     : 1'673'006'191
Total size       : 80.648 TB 
Avg luminosity   : 1.669 /fb
Size  per /pb    : 47.7 GB
Files per /fb    : 32774.4
Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r11 - 2018-09-23 - MarcoCattaneo
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback