Difference: LHCbDiracCLI (1 vs. 12)

Revision 122019-07-05 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 28 to 28
  - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>
If a list of files was used through a pipe, the --File or --Terminal= option above, the last list of files used may be re-used using --LastLFNs. Note that this may not always work, depending on how the software is installed. This should however be fixed in a coming release (July 2019).
 Here is an example of piping commands:
[localhost, Patch] ~ $ dirac-transformation-debug 24847 --Status Processed --Info files | dirac-bookkeeping-get-file-descendants --Prod 24847 | grep full.dst | dirac-transformation-debug 24182 --Info tasks
Line: 119 to 121
  - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>
- --LastLFNs : Use last set of LFNs
  -a --All : Also show inactive replicas
Line: 144 to 147
  - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>
- --LastLFNs : Use last set of LFNs
  -g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list)
Line: 174 to 178
  - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>
- --LastLFNs : Use last set of LFNs
  -g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list) - --Check : Checks the PFN metadata vs LFN metadata
Line: 183 to 188
 For backward compatibility the list of URLs and SEs can be provided as positional arguments, but it is recommended to use option switches. If a site is selected, the metadata of all replicas at that site are reported. If no SE and no sites are specified, the metadata of all replicas is reported.
Added:
>
>

dirac-dms-pfn-exists

Shortcut for dirac-dms-pfn-metadata --Exists

$ dirac-dms-pfn-exists --help

Check existence of a (list of) LHCb LFNs/PFNs given a valid DIRAC SE (or for all replicas)
  Only the LFN contained in the PFN is considered, unlike the DIRAC similar script

Usage:
  dirac-dms-pfn-exists [option|cfgfile] ... [URL[,URL2[,URL3...]]] SE[ SE2...]
Arguments:
  URL:      Logical/Physical File Name or file containing URLs
  SE:       Valid DIRAC SE

General options:
  -o  --option <value>         : Option=value to add
  -s  --section <value>        : Set base section for relative parsed options
  -c  --cert <value>           : Use server certificate to connect to Core Services
  -d  --debug                  : Set debug mode (-ddd is extra debug)
  -   --autoreload             : Automatically restart if there's any change in the module
  -   --license                : Show DIRAC's LICENSE
  -h  --help                   : Shows this help

Options:
  -   --File=                  : File containing list of LFNs
  -l  --LFNs <value>           : List of LFNs (comma separated)
  -   --Terminal               : LFNs are entered from stdin (--File /dev/stdin)
  -   --LastLFNs               : Use last set of LFNs
  -g  --Sites <value>          :   Sites to consider [ALL] (comma separated list)
  -S  --SEs <value>            :   SEs to consider [ALL] (comma separated list)
  -   --Summary                :    Only prints a summary on existing files
 

dirac-dms-user-lfns

This command allows a user to get a list of all files they own on the Grid, within directories or with data/time criteria.

Line: 212 to 252
 

dirac-dms-lfn-accessURL

Changed:
<
<
This command returns tURLs for files at a given SE or site. This tURL is optimised for fast and efficient access to the data. Note that any other tURL may either be inefficient or its usage may be discontinued at some point.
>
>
This command returns tURLs for files at a given SE or site. This tURL is optimised for fast and efficient access to the data. Note that any other tURL may either be inefficient or its usage may be discontinued at some point. One can select for which protocol one wants to get a URL using --Protocol= (default is xrootd).
 In case the file is on tape storage, the tURL can currently only be obtained once the file has been staged, therefore the command may in this case timeout.
Changed:
<
<
[localhost, Patch] ~ $ dirac-dms-lfn-accessURL --help
>
>
[localhost] ~ $ dirac-dms-lfn-accessURL --help
  Retrieve an access URL for an LFN replica given a valid DIRAC SE.
Added:
>
>
 Usage:
Changed:
<
<
dirac-dms-lfn-accessURL [option|cfgfile] ... [LFN[,LFN2[,LFN3...]]] [SE[,SE2...]]
>
>
dirac-dms-lfn-accessURL [option|cfgfile] ... [LFN[,LFN2[,LFN3...]]] SE[,SE2...]
 Arguments: LFN: Logical File Name or file containing LFNs SE: Valid DIRAC SE
Line: 223 to 265
 Arguments: LFN: Logical File Name or file containing LFNs SE: Valid DIRAC SE
Added:
>
>
 General options:
Changed:
<
<
-o: --option= : Option=value to add -s: --section= : Set base section for relative parsed options -c: --cert= : Use server certificate to connect to Core Services -d --debug : Set debug mode (-dd is extra debug)
>
>
-o --option : Option=value to add -s --section : Set base section for relative parsed options -c --cert : Use server certificate to connect to Core Services -d --debug : Set debug mode (-ddd is extra debug) - --autoreload : Automatically restart if there's any change in the module - --license : Show DIRAC's LICENSE
  -h --help : Shows this help

Options: - --File= : File containing list of LFNs

Changed:
<
<
-l: --LFNs= : List of LFNs (comma separated)
>
>
-l --LFNs : List of LFNs (comma separated)
  - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Changed:
<
<
-g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list)
>
>
- --LastLFNs : Use last set of LFNs -g --Sites : Sites to consider [ALL] (comma separated list) -S --SEs : SEs to consider [ALL] (comma separated list) - --Protocol= : Define the protocol for which a tURL is requested (default:root)
 

Positional arguments are supported for backward compatibility, but option switches are highly recommended. If no SE is specified, all allowed SEs are used.

Line: 262 to 309
  - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>
- --LastLFNs : Use last set of LFNs
  -D: --Directory= : Directory to download to (default = /home/phicharp)
Line: 293 to 341
  - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>
- --LastLFNs : Use last set of LFNs
 

dirac-dms-add-file

Line: 419 to 468
  - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>
- --LastLFNs : Use last set of LFNs
  -g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list) -v --Verbose : use this option for verbose output [False]
Line: 458 to 508
  - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>
- --LastLFNs : Use last set of LFNs
  - --SetProcessed : Forced to set Removed the files in status Processed (default:not reset)

Revision 112018-11-12 - ChristopheHaen

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 300 to 300
 Uploads to a Grid storage element and registers in the file catalog a local file. In order to properly register the GUID of the file as the one contained in the file itself, it is mandatory to setup the ROOT environment as well. Easiest way is:
Changed:
<
<
$ lb-run LHCbDirac/latest [bash --norc|tcsh|...]
>
>
$ lb-run -c best LHCbDirac/prod [bash --norc|tcsh|...]
 

If you are on lxplus or on a machine that has access to EOS or Castor at CERN, the "local file" can be a Castor file (in the form /castor/cern.ch/...) or an EOS file (in the form /eos/lhcb/...), of course not yet a Grid file.

Revision 102016-08-27 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 300 to 300
 Uploads to a Grid storage element and registers in the file catalog a local file. In order to properly register the GUID of the file as the one contained in the file itself, it is mandatory to setup the ROOT environment as well. Easiest way is:
Changed:
<
<
$ SetupProject LHCbDirac ROOT
>
>
$ lb-run LHCbDirac/latest [bash --norc|tcsh|...]
 

If you are on lxplus or on a machine that has access to EOS or Castor at CERN, the "local file" can be a Castor file (in the form /castor/cern.ch/...) or an EOS file (in the form /eos/lhcb/...), of course not yet a Grid file.

Revision 92016-01-27 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 303 to 303
 $ SetupProject LHCbDirac ROOT
Added:
>
>
If you are on lxplus or on a machine that has access to EOS or Castor at CERN, the "local file" can be a Castor file (in the form /castor/cern.ch/...) or an EOS file (in the form /eos/lhcb/...), of course not yet a Grid file.

The LFN for a user file must be in the form /lhcb/user/<initial>/<username>/whatever-you-like-as-a-path.

 Command usage:
dirac-dms-add-file --help

Revision 82014-11-06 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 46 to 46
 

SE and sites

Added:
>
>
  -g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list)
Added:
>
>
  For Tier1s, the name of the site can be a short name (CERN, CNAF, GRIDKA, IN2P3, NIKHEF, PIC, RAL, SARA). When the command requires a list of SEs and a Site is specified, all SEs at that site are considered.
Added:
>
>
Storage Elements can also be a StorageElementGroup if defined in the Configuration of LHCbDirac, for example Tier1-DST, Tier1-BUFFER etc...
 

Bookkeeping query switches

Line: 215 to 219
 [localhost, Patch] ~ $ dirac-dms-lfn-accessURL --help Retrieve an access URL for an LFN replica given a valid DIRAC SE. Usage:
Changed:
<
<
dirac-dms-lfn-accessURL [option|cfgfile] ... [LFN[,LFN2[,LFN3...]]] SE[,SE2...]
>
>
dirac-dms-lfn-accessURL [option|cfgfile] ... [LFN[,LFN2[,LFN3...]]] [SE[,SE2...]]
 Arguments: LFN: Logical File Name or file containing LFNs SE: Valid DIRAC SE
Line: 234 to 238
  -S: --SEs= : SEs to consider [ALL] (comma separated list)
Changed:
<
<
Positional arguments are supported for backward compatibility, but option switches are highly recommended.
>
>
Positional arguments are supported for backward compatibility, but option switches are highly recommended. If no SE is specified, all allowed SEs are used.
 

dirac-dms-get-file

Line: 291 to 295
  - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Added:
>
>

dirac-dms-add-file

Uploads to a Grid storage element and registers in the file catalog a local file. In order to properly register the GUID of the file as the one contained in the file itself, it is mandatory to setup the ROOT environment as well. Easiest way is:

$ SetupProject LHCbDirac ROOT

Command usage:

dirac-dms-add-file --help

Upload a file to the grid storage and register it in the File Catalog
Usage:
  dirac-dms-add-file [option|cfgfile] ... LFN Path SE [GUID]
Arguments:
  LFN:      Logical File Name
  Path:     Local path of the file
  SE:       DIRAC Storage Element
  GUID:     GUID to use in the registration (optional)

 ++ OR ++

Usage:
  dirac-dms-add-file [option|cfgfile] ... LocalFile
Arguments:
  LocalFile: Path to local file containing all the above, i.e.:
  lfn1 localfile1 SE [GUID1]
  lfn2 localfile2 SE [GUID2] 

General options: 
  -o  --option <value>         : Option=value to add 
  -s  --section <value>        : Set base section for relative parsed options 
  -c  --cert <value>           : Use server certificate to connect to Core Services 
  -d  --debug                  : Set debug mode (-dd is extra debug) 
  -h  --help                   : Shows this help 

GUID-related issues

Gaudi requires when reading a file through a catalog that the GUID in the file catalog matches the GUID (a.k.a. FID) stored in the file. If the file was uploaded to the Grid and registered without care, it may be that this is not the case. It can be checked and eventually fixed using the following commands:

dirac-lhcb-get-root-guid

This script will read the file GUID from the actual replica.

[localhost, PatchFull] ~ $ dirac-lhcb-get-root-guid --help

Get the GUID of a (set of) ROOT file
The file can be either local, an LFN or an xrootd URL (root:...)

Usage:
  dirac-lhcb-get-root-guid [option|cfgfile] file1 [file2 ...] 

General options: 
  -o  --option <value>         : Option=value to add 
  -s  --section <value>        : Set base section for relative parsed options 
  -c  --cert <value>           : Use server certificate to connect to Core Services 
  -d  --debug                  : Set debug mode (-dd is extra debug) 
  -h  --help                   : Shows this help 

dirac-lhcb-fix-file-guid

This script will read the file GUID from the actual file and update the file catalog if needed with the correct GUID.

[localhost, PatchFull] ~ $ dirac-lhcb-fix-file-guid --help

Fix incorrect file GUIDs

Usage:
  dirac-lhcb-fix-file-guid [option|cfgfile] [OldLFN] 

General options: 
  -o  --option <value>         : Option=value to add 
  -s  --section <value>        : Set base section for relative parsed options 
  -c  --cert <value>           : Use server certificate to connect to Core Services 
  -d  --debug                  : Set debug mode (-dd is extra debug) 
  -h  --help                   : Shows this help 
 
Options: 

  -f  --OldLFN <value>         : LFN of existing file to be fixed. 

  -n  --NewLFN <value>         : Optional: specify a new LFN for the file (retaining the existing file with incorrect GUID). 

  -D  --Directory <value>      : Optional: directory to download file (defaults to TMPDIR then PWD). 

  -k  --Keep                   : Optional: specify this switch to retain the local copy of the downloaded file 

  -m  --SafeMode               : Optional: specify this switch to run the script in safe mode (will check the GUIDs only) 
 

Removing replicas or files

dirac-dms-remove-replicas

Revision 72014-04-07 - TimHead

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 6 to 6
 
Changed:
<
<
The LHCbDirac DMS relies on one hand on Logical File Names (LFNs) that are always of the form /lhcb/....., and on LHCbDirac Storage Elements (SE) for which the first part always indicates the site where it is located. The instance of a file at an SE is called a replica. Note that some replicas may be inactive, i.e. cannot be used for accessing files. This is always the case for archive replicas (SE name of the form <Site>-ARCHIVE), but also some SEs may be temporarily unavailable or files not accessible (e.g. in case a disk server is down at a site).
>
>
The LHCbDirac DMS relies on Logical File Names (LFNs) that are always of the form /lhcb/....., and on LHCbDirac Storage Elements (SE) for which the first part always indicates the site where it is located. The instance of a file at an SE is called a replica. Note that some replicas may be inactive, i.e. cannot be used for accessing files. This is always the case for archive replicas (SE name of the form <Site>-ARCHIVE), furthermore some SEs may be temporarily unavailable or files not accessible (e.g. in case a disk server is down at a site).
  A list of active SEs can be obtained with dirac-dms-show-se-status.
Line: 96 to 96
 

dirac-dms-lfn-replicas

Changed:
<
<
Shows where files have replicas and report the SURL registered in the file catalog
>
>
Shows where files have replicas and reports the SURL registered in the file catalog
 
$ dirac-dms-lfn-replicas --help
Line: 181 to 181
 

dirac-dms-user-lfns

Changed:
<
<
This command allows a user to get a list of all files it owns on the Grid, within directories or with data/time criteria.
>
>
This command allows a user to get a list of all files they own on the Grid, within directories or with data/time criteria.
 
$ dirac-dms-user-lfns --help
Line: 238 to 238
 

dirac-dms-get-file

Changed:
<
<
This command copies to the local directory a file or a set of files. The options of this command are going to evolve soon, giving additional possibilities.
>
>
This command copies a file or a set of files to the local directory. The options of this command are going to evolve soon, giving additional possibilities.
 
[localhost, Certif] ~ $ dirac-dms-get-file --help

Revision 62013-06-26 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Changed:
<
<
This page describes the usage of the LHCbDirac data management (DMS) commands. As they interface may vary with time and introduction of new facilities, not that the --help= option can be used to get the syntax.
>
>
This page describes the usage of the LHCbDirac data management (DMS) commands. As they interface may vary with time and introduction of new facilities, note that the --help option can be used to get the syntax.
 

Revision 52013-06-20 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 67 to 67
  The --BKQuery option consists of a path constructed as follows:
Changed:
<
<
/ConfigName/ConfigVersion/Conditions/ProcessingPass/EventType/FileType
>
>
/DataType/Activity/Conditions/ProcessingPass/EventType/FileType
 
Changed:
<
<
  • ConfigName : this is either the Online partition for real data (in particular /LHCb) or /MC for simulated data
>
>
  • DataType : this is either the Online partition for real data (in particular LHCb) or MC for simulated data (mandatory)
  • Activity : this is the generic activity, for example Collision12 or MC2012 (mandatory)
  • Conditions : data taking or simulation conditions, for example Beam4000GeV-VeloClosed-MagDown (if omitted: any)
  • ProcessingPass : this is the level of processing, for example RealData/Reco14/Stripping20 (if omitted: any)
  • EventType : numeric event type id (or comma-separated list of ids), for example 90000000 (if omitted: any for MC, 90000000 for real data)
  • FileType : file type of comma-separated list of file types
    • Omitted : all file types
    • ALL.DST or ALL.MDST : all DST or MDST files except those specified in --ExceptFileType
      • By default --ExceptFileTypes are all xxx.ETC files, all xxxHIST files
    • ALL : all file types except those specified in --ExceptFileType
    • ALL.HIST : all histogram file types

Some examples:

  • /LHCb/Collision12//RealData/Reco14 : all files from Reco14 of 2012 real data
  • /LHCb/Collision12//RealData/Reco14/Stripping20 : all files for Reco14/Stripping20 of 2012 data
  • /LHCb/Collision12//RealData/Reco14/Stripping20//ALL.MDST : all µDST files
  • /LHCb/Collision12//RealData/Reco14/Stripping20//ALL.MDST with --Except PID.MDST: all µDST files except PID.MDST
  • /LHCb/Collision12////BHADRON.MDST : all existing BHADRON.DST files from 2012 data (i.e. all processings and strippings)
  • /LHCb/Collision12//RealData/91000000/RAW : all Express stream RAW data of 2012
  • /MC/2012/Beam4000GeV-2012-MagDown-Nu2.5-Pythia6/Sim08/Digi13/Trig0x409f0045/Reco14/Stripping20NoPrescalingFlagged/30000000,13164417,15112001,33102100,11166104,13164061,13104241,12143401,13164213,13164215,13164217,10000000,23960000,11144001,12163011,12163021 : a long list of event types of MC
 

Information on files

Revision 42013-06-20 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 12 to 12
  A replica has a unique SRM URL (SURL) that is used to query the Storage Resource Manager (SRM) for getting informations on the replica or getting a Transport URL (tURL), a.k.a. Physical File name (PFN) to be used by Gaudi.
Changed:
<
<

Common command options

>
>

Common command options

  The DM scripts are based on a shared command line options parsing tool (DMScript). These options are split into sets for different purposes:
Changed:
<
<

List of files

>
>

List of files

  The parsing of file names is trying to extract LFNs out of any string that contains an LFN (e.g. SURL, tURL, output of other commands, Gaudi options). It works in most cases, as long as there is a separator after the LFN. In the case the LFNs are in a file or from the terminal, lines not containing an LFN are ignored.
Line: 44 to 44
 …….
Changed:
<
<

SE and sites

>
>

SE and sites

  -g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list)

For Tier1s, the name of the site can be a short name (CERN, CNAF, GRIDKA, IN2P3, NIKHEF, PIC, RAL, SARA). When the command requires a list of SEs and a Site is specified, all SEs at that site are considered.

Changed:
<
<

Bookkeeping query switches

>
>

Bookkeeping query switches

 
  -P:  --Productions=    :    Production ID to search (comma separated list) 
Line: 63 to 63
  - --StartDate= : Start date for the BK query - --EndDate= : End date for the BK query - --Invisible : See also invisible files
Changed:
<
<
>
>

The --BKQuery option consists of a path constructed as follows:

/ConfigName/ConfigVersion/Conditions/ProcessingPass/EventType/FileType

  • ConfigName : this is either the Online partition for real data (in particular /LHCb) or /MC for simulated data
 
Deleted:
<
<

Information on files

 
Changed:
<
<

dirac-dms-lfn-replicas

>
>

Information on files

dirac-dms-lfn-replicas

  Shows where files have replicas and report the SURL registered in the file catalog
Line: 91 to 98
  -a --All : Also show inactive replicas
Changed:
<
<

dirac-dms-lfn-metadata

>
>

dirac-dms-lfn-metadata

  Shows metadata of a logical file, i.e. not related to any physical replica.
Line: 117 to 124
  -S: --SEs= : SEs to consider [ALL] (comma separated list)
Changed:
<
<

dirac-dms-pfn-metadata

>
>

dirac-dms-pfn-metadata

  Shows the metadata of physical replica(s), possibly at selected SEs/Sites.
Line: 152 to 159
 For backward compatibility the list of URLs and SEs can be provided as positional arguments, but it is recommended to use option switches. If a site is selected, the metadata of all replicas at that site are reported. If no SE and no sites are specified, the metadata of all replicas is reported.
Changed:
<
<

dirac-dms-user-lfns

>
>

dirac-dms-user-lfns

  This command allows a user to get a list of all files it owns on the Grid, within directories or with data/time criteria.
Line: 177 to 184
  -e --EmptyDirs : Create a list of empty directories
Changed:
<
<

Accessing files

>
>

Accessing files

 
Changed:
<
<

dirac-dms-lfn-accessURL

>
>

dirac-dms-lfn-accessURL

  This command returns tURLs for files at a given SE or site. This tURL is optimised for fast and efficient access to the data. Note that any other tURL may either be inefficient or its usage may be discontinued at some point. In case the file is on tape storage, the tURL can currently only be obtained once the file has been staged, therefore the command may in this case timeout.
Line: 209 to 216
  Positional arguments are supported for backward compatibility, but option switches are highly recommended.
Changed:
<
<

dirac-dms-get-file

>
>

dirac-dms-get-file

  This command copies to the local directory a file or a set of files. The options of this command are going to evolve soon, giving additional possibilities.
Line: 234 to 241
  -D: --Directory= : Directory to download to (default = /home/phicharp)
Changed:
<
<

Replicating files

>
>

Replicating files

 
Changed:
<
<

dirac-dms-replicate-lfn

>
>

dirac-dms-replicate-lfn

  Replicates an LFN to a (list of) SE. It requires write access to the FC directory.
Line: 264 to 271
  - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
Changed:
<
<

Removing replicas or files

>
>

Removing replicas or files

 
Changed:
<
<

dirac-dms-remove-replicas

>
>

dirac-dms-remove-replicas

  Allows to remove a replica at a given SE. This is only allowed for authorised files of course. Note also that it is not possible to remove the last replica of a file. Use dirac-dms-remove-files instead.
Line: 336 to 343
 The list of files may be the result of a Bookkeeping query. This is only allowed for data managers and should be used with great care.

If files are used by some transformations, they are set to "Removed" status unless their current status is "Processed". "Processed" files can be forced to be set "Removed" using the switch =--SetProcessed+.

Added:
>
>

Bookkeeping commands

They can be found here.

Revision 32013-06-20 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 182 to 182
 

dirac-dms-lfn-accessURL

This command returns tURLs for files at a given SE or site. This tURL is optimised for fast and efficient access to the data. Note that any other tURL may either be inefficient or its usage may be discontinued at some point.

Added:
>
>
In case the file is on tape storage, the tURL can currently only be obtained once the file has been staged, therefore the command may in this case timeout.
 
[localhost, Patch] ~ $ dirac-dms-lfn-accessURL --help
Line: 208 to 209
  Positional arguments are supported for backward compatibility, but option switches are highly recommended.
Changed:
<
<

dirac-dms-get-file

>
>

dirac-dms-get-file

  This command copies to the local directory a file or a set of files. The options of this command are going to evolve soon, giving additional possibilities.
Changed:
<
<
$ dirac-dms-get-file --help
>
>
[localhost, Certif] ~ $ dirac-dms-get-file --help
  Retrieve a single file or list of files from Grid storage to the current directory.
Added:
>
>
Usage: dirac-dms-get-file [option|cfgfile] [] [...] General options: -o: --option= : Option=value to add -s: --section= : Set base section for relative parsed options -c: --cert= : Use server certificate to connect to Core Services -d --debug : Set debug mode (-dd is extra debug) -h --help : Shows this help

Options: - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin) -D: --Directory= : Directory to download to (default = /home/phicharp)

Replicating files

dirac-dms-replicate-lfn

Replicates an LFN to a (list of) SE. It requires write access to the FC directory.

[localhost, Certif] ~ $ dirac-dms-replicate-lfn --help

  Replicate a (list of) existing LFN(s) to (set of) Storage Element(s)
 Usage:
Changed:
<
<
dirac-dms-get-file [option|cfgfile] ... LFN ...
>
>
dirac-dms-replicate-lfn [option|cfgfile] ... [LFN1[,LFN2,[...]]] Dest[,Dest2[,...]] [Source [Cache]]
 Arguments:
Changed:
<
<
LFN: Logical File Name or file containing LFNs
>
>
Dest: Valid DIRAC SE(s) Source: Valid DIRAC SE Cache: Local directory to be used as cache
 General options: -o: --option= : Option=value to add -s: --section= : Set base section for relative parsed options -c: --cert= : Use server certificate to connect to Core Services -d --debug : Set debug mode (-dd is extra debug) -h --help : Shows this help
Added:
>
>
Options: - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin)
 

Removing replicas or files

dirac-dms-remove-replicas

Changed:
<
<
Allows to remove a replica at a given SE. This is only allowed for authorised files of course. Note that it is not possible to remove the last replica of a file. Use =dirac-dms-remove-files- for this purpose.
>
>
Allows to remove a replica at a given SE. This is only allowed for authorised files of course. Note also that it is not possible to remove the last replica of a file. Use dirac-dms-remove-files instead.
 
Changed:
<
<
$ dirac-dms-remove-replicas --help
>
>
[localhost, Certif] ~ $ dirac-dms-remove-replicas --help
  Remove the given file replica or a list of file replicas from the File Catalog and from the storage.
Line: 252 to 290
 Options: - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated)
Added:
>
>
- --Terminal : LFNs are entered from stdin (--File /dev/stdin) -g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list) -v --Verbose : use this option for verbose output [False] -n --NoLFC : use this option to force the removal from storage of replicas not in FC
 
Added:
>
>
The option --NoLFC allows to remove replicas even though they are not present in the LFC (dark data). In this case an attempt is made to remove the file from physical storage only.
 

dirac-dms-remove-files

Changed:
<
<
Allows to remove all replicas of a file as well as the entry in the file catalog.
>
>
Allows to remove all replicas of a (list of) file(s) as well as the entry in the file catalog.
 
Changed:
<
<
$ dirac-dms-remove-files --help
>
>
[localhost, Certif] ~ $ dirac-dms-remove-files --help
 
Changed:
<
<
Remove the given file or a list of files from the File Catalog and from the storage. The option --FixTransformations is only for production manager usage.
>
>
Remove the given file or a list of files from the File Catalog and from the storage
  Usage: dirac-dms-remove-files [option|cfgfile] [] [...]
Line: 273 to 318
  -h --help : Shows this help

Options:

Added:
>
>
-P: --Productions= : Production ID to search (comma separated list) -f: --FileType= : File type (comma separated list, to be used with --Production) [All] - --ExceptFileType= : Exclude the (list of) file types when all are requested -B: --BKQuery= : Bookkeeping query path -r: --Runs= : Run or range of runs (r1:r2) - --DQFlags= : DQ flag used in query - --StartDate= : Start date for the BK query - --EndDate= : End date for the BK query - --Invisible : See also invisible files
  - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated)
Changed:
<
<
- --FixTransformations : Allows to set the files as Removed in all transformations
>
>
- --Terminal : LFNs are entered from stdin (--File /dev/stdin) - --SetProcessed : Forced to set Removed the files in status Processed (default:not reset)
  \ No newline at end of file
Added:
>
>
The list of files may be the result of a Bookkeeping query. This is only allowed for data managers and should be used with great care.

If files are used by some transformations, they are set to "Removed" status unless their current status is "Processed". "Processed" files can be forced to be set "Removed" using the switch =--SetProcessed+.

Revision 22013-06-20 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

Line: 12 to 12
  A replica has a unique SRM URL (SURL) that is used to query the Storage Resource Manager (SRM) for getting informations on the replica or getting a Transport URL (tURL), a.k.a. Physical File name (PFN) to be used by Gaudi.
Added:
>
>

Common command options

The DM scripts are based on a shared command line options parsing tool (DMScript). These options are split into sets for different purposes:

List of files

The parsing of file names is trying to extract LFNs out of any string that contains an LFN (e.g. SURL, tURL, output of other commands, Gaudi options). It works in most cases, as long as there is a separator after the LFN. In the case the LFNs are in a file or from the terminal, lines not containing an LFN are ignored.

Note that if the command is piped with the output of another command, the list of LFNs is supposed to come from the pipe unless otherwise specified. This allows chaining DMS or other commands through pipes in a handy way.

  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin)

Here is an example of piping commands:

[localhost, Patch] ~ $ dirac-transformation-debug 24847 --Status Processed --Info files | dirac-bookkeeping-get-file-descendants --Prod 24847 | grep full.dst | dirac-transformation-debug 24182 --Info tasks
 Transformation 24182 (Active) : Request_11636_DataStripping_Stripping20r0p1_EventType_90000000_FULL_1.xml of type DataStripping (plugin ByRunWithFlush, GroupSize: 2) in Stripping20r0p1
BKQuery: {'StartRun': 111181L, 'ConfigName': 'LHCb', 'EndRun': 133785L, 'EventType': 90000000L, 'FileType': 'FULL.DST', 'ProcessingPass': 'Real Data/Reco14', 'Visible': 'Yes', 'DataQualityFlag': 'OK', 'ConfigVersion': 'Collision12', 'DataTakingConditions': 'Beam4000GeV-VeloClosed-MagDown'}

5 files found

/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000001_1.full.dst - Run: 115327 - Status: Processed - UsedSE: CERN-BUFFER - ErrorCount: 0
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000094_1.full.dst - Run: 132338 - Status: Processed - UsedSE: CNAF-BUFFER - ErrorCount: 0
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000020_1.full.dst - Run: 124129 - Status: Processed - UsedSE: SARA-BUFFER - ErrorCount: 0
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000069_1.full.dst - Run: 124915 - Status: Processed - UsedSE: SARA-BUFFER - ErrorCount: 0
/lhcb/LHCb/Collision12/FULL.DST/00024847/0000/00024847_00000031_1.full.dst - Run: 124214 - Status: Processed - UsedSE: SARA-BUFFER - ErrorCount: 0
…….

SE and sites

-g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list)

For Tier1s, the name of the site can be a short name (CERN, CNAF, GRIDKA, IN2P3, NIKHEF, PIC, RAL, SARA). When the command requires a list of SEs and a Site is specified, all SEs at that site are considered.

Bookkeeping query switches

  -P:  --Productions=    :    Production ID to search (comma separated list) 
  -f:  --FileType=       :    File type (comma separated list, to be used with --Production) [All] 
  -    --ExceptFileType= :    Exclude the (list of) file types when all are requested 
  -B:  --BKQuery=        :    Bookkeeping query path 
  -r:  --Runs=           :    Run or range of runs (r1:r2) 
  -    --DQFlags=        :    DQ flag used in query 
  -    --StartDate=      :    Start date for the BK query 
  -    --EndDate=        :    End date for the BK query 
  -    --Invisible       :    See also invisible files 
</verbstim>
 

Information on files

dirac-dms-lfn-replicas

Line: 34 to 87
 Options: - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated)
Added:
>
>
- --Terminal : LFNs are entered from stdin (--File /dev/stdin)
  -a --All : Also show inactive replicas

dirac-dms-lfn-metadata

Changed:
<
<
Shows metadata on a logical file, i.e. not related to any replica.
>
>
Shows metadata of a logical file, i.e. not related to any physical replica.
 
Changed:
<
<
$ dirac-dms-lfn-metadata --help Obtain file metadata from file catalogue client.
>
>
[localhost, Patch] ~ $ dirac-dms-lfn-metadata --help Get the metadata of a (list of) LFNs from the FC
 Usage:
Changed:
<
<
dirac-dms-lfn-metadata [option|cfgfile] ... LFN ...
>
>
dirac-dms-lfn-metadata [option|cfgfile] ... [LFN[,LFN2[,LFN3...]]]
 Arguments: LFN: Logical File Name or file containing LFNs General options:
Line: 54 to 108
  -c: --cert= : Use server certificate to connect to Core Services -d --debug : Set debug mode (-dd is extra debug) -h --help : Shows this help
Added:
>
>
Options: - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated) - --Terminal : LFNs are entered from stdin (--File /dev/stdin) -g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list)
 
Added:
>
>

dirac-dms-pfn-metadata

Shows the metadata of physical replica(s), possibly at selected SEs/Sites.

[localhost, Patch] ~ $ dirac-dms-pfn-metadata  --help

  Gets the metadata of a (list of) LHCb LFNs/PFNs given a valid DIRAC SE.
  Only the LFN contained in the PFN is considered, unlike the DIRAC similar script

Usage:
  dirac-dms-pfn-metadata [option|cfgfile] ... [URL[,URL2[,URL3...]]] SE[ SE2...]
Arguments:
  URL:      Logical/Physical File Name or file containing URLs
  SE:       Valid DIRAC SE 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --Terminal        : LFNs are entered from stdin (--File /dev/stdin) 
  -g:  --Sites=          :   Sites to consider [ALL] (comma separated list) 
  -S:  --SEs=            :   SEs to consider [ALL] (comma separated list) 
  -    --Check           :    Checks the PFN metadata vs LFN metadata 
  -    --Exists          :    Only reports if the file exists 

For backward compatibility the list of URLs and SEs can be provided as positional arguments, but it is recommended to use option switches. If a site is selected, the metadata of all replicas at that site are reported. If no SE and no sites are specified, the metadata of all replicas is reported.

 

dirac-dms-user-lfns

This command allows a user to get a list of all files it owns on the Grid, within directories or with data/time criteria.

Line: 85 to 181
 

dirac-dms-lfn-accessURL

Changed:
<
<
This command returns tURLs for files at a given SE or site. This tURL is optimised for fast and efficient access to the data. Note that any other tURL may either be inefficient or its usage may be discontinued at some point.
>
>
This command returns tURLs for files at a given SE or site. This tURL is optimised for fast and efficient access to the data. Note that any other tURL may either be inefficient or its usage may be discontinued at some point.
 
Changed:
<
<
$ dirac-dms-lfn-accessURL --help
>
>
[localhost, Patch] ~ $ dirac-dms-lfn-accessURL --help
  Retrieve an access URL for an LFN replica given a valid DIRAC SE. Usage: dirac-dms-lfn-accessURL [option|cfgfile] ... [LFN[,LFN2[,LFN3...]]] SE[,SE2...]
Line: 105 to 201
 Options: - --File= : File containing list of LFNs -l: --LFNs= : List of LFNs (comma separated)
Added:
>
>
- --Terminal : LFNs are entered from stdin (--File /dev/stdin)
  -g: --Sites= : Sites to consider [ALL] (comma separated list) -S: --SEs= : SEs to consider [ALL] (comma separated list)
Added:
>
>
Positional arguments are supported for backward compatibility, but option switches are highly recommended.
 

dirac-dms-get-file

This command copies to the local directory a file or a set of files. The options of this command are going to evolve soon, giving additional possibilities.

Revision 12013-02-18 - PhilippeCharpentier

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbComputing"

LHCbDirac Data Management commands

This page describes the usage of the LHCbDirac data management (DMS) commands. As they interface may vary with time and introduction of new facilities, not that the --help= option can be used to get the syntax.

The LHCbDirac DMS relies on one hand on Logical File Names (LFNs) that are always of the form /lhcb/....., and on LHCbDirac Storage Elements (SE) for which the first part always indicates the site where it is located. The instance of a file at an SE is called a replica. Note that some replicas may be inactive, i.e. cannot be used for accessing files. This is always the case for archive replicas (SE name of the form <Site>-ARCHIVE), but also some SEs may be temporarily unavailable or files not accessible (e.g. in case a disk server is down at a site).

A list of active SEs can be obtained with dirac-dms-show-se-status.

A replica has a unique SRM URL (SURL) that is used to query the Storage Resource Manager (SRM) for getting informations on the replica or getting a Transport URL (tURL), a.k.a. Physical File name (PFN) to be used by Gaudi.

Information on files

dirac-dms-lfn-replicas

Shows where files have replicas and report the SURL registered in the file catalog

$ dirac-dms-lfn-replicas --help

    Show replicas for a (set of) LFNs

Usage:
  dirac-dms-lfn-replicas [option|cfgfile] [<LFN>] [<LFN>...] 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -a   --All             :   Also show inactive replicas 

dirac-dms-lfn-metadata

Shows metadata on a logical file, i.e. not related to any replica.

$ dirac-dms-lfn-metadata --help
  Obtain file metadata from file catalogue client.
Usage:
  dirac-dms-lfn-metadata [option|cfgfile] ... LFN ...
Arguments:
  LFN:      Logical File Name or file containing LFNs 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 

dirac-dms-user-lfns

This command allows a user to get a list of all files it owns on the Grid, within directories or with data/time criteria.

$ dirac-dms-user-lfns --help
Get the list of all the user files.
Usage:
  dirac-dms-user-lfns [option|cfgfile] ... 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -D:  --Days=           : Match files older than number of days [0] 
  -M:  --Months=         : Match files older than number of months [0] 
  -Y:  --Years=          : Match files older than number of years [0] 
  -w:  --Wildcard=       : Wildcard for matching filenames [*] 
  -b:  --BaseDir=        : Base directory to begin search (default /[vo]/user/[initial]/[username]) 
  -e   --EmptyDirs       : Create a list of empty directories 

Accessing files

dirac-dms-lfn-accessURL

This command returns tURLs for files at a given SE or site. This tURL is optimised for fast and efficient access to the data. Note that any other tURL may either be inefficient or its usage may be discontinued at some point.

$ dirac-dms-lfn-accessURL --help
  Retrieve an access URL for an LFN replica given a valid DIRAC SE.
Usage:
  dirac-dms-lfn-accessURL [option|cfgfile] ... [LFN[,LFN2[,LFN3...]]] SE[,SE2...]
Arguments:
  LFN:      Logical File Name or file containing LFNs
  SE:       Valid DIRAC SE 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -g:  --Sites=          :   Sites to consider [ALL] (comma separated list) 
  -S:  --SEs=            :   SEs to consider [ALL] (comma separated list) 

dirac-dms-get-file

This command copies to the local directory a file or a set of files. The options of this command are going to evolve soon, giving additional possibilities.

$ dirac-dms-get-file --help
  Retrieve a single file or list of files from Grid storage to the current directory.
Usage:
  dirac-dms-get-file [option|cfgfile] ... LFN ...
Arguments:
  LFN:      Logical File Name or file containing LFNs 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 

Removing replicas or files

dirac-dms-remove-replicas

Allows to remove a replica at a given SE. This is only allowed for authorised files of course. Note that it is not possible to remove the last replica of a file. Use =dirac-dms-remove-files- for this purpose.

$ dirac-dms-remove-replicas --help

  Remove the given file replica or a list of file replicas from the File Catalog
  and from the storage.

  Usage:
     dirac-dms-remove-replicas <LFN | fileContainingLFNs> SE [SE]
   
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 

dirac-dms-remove-files

Allows to remove all replicas of a file as well as the entry in the file catalog.

 $ dirac-dms-remove-files --help

  Remove the given file or a list of files from the File Catalog and from the storage. The option =--FixTransformations= is only for production manager usage.

Usage:
  dirac-dms-remove-files [option|cfgfile] [<LFN>] [<LFN>...] 
General options: 
  -o:  --option=         : Option=value to add 
  -s:  --section=        : Set base section for relative parsed options 
  -c:  --cert=           : Use server certificate to connect to Core Services 
  -d   --debug           : Set debug mode (-dd is extra debug) 
  -h   --help            : Shows this help 
 
Options: 
  -    --File=           : File containing list of LFNs 
  -l:  --LFNs=           : List of LFNs (comma separated) 
  -    --FixTransformations :    Allows to set the files as Removed in all transformations 
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback