dirac-production-file-summary

To check the status of files for a given production the productionID must be specified, below shows the optional arguments:

dirac-production-file-summary --help

2008-09-08 15:15:21 UTC Framework  INFO: Usage:
2008-09-08 15:15:21 UTC Framework  INFO: scripts/dirac-production-file-summary (<options>|<cfgFile>)*
2008-09-08 15:15:21 UTC Framework  INFO: Options:
2008-09-08 15:15:21 UTC Framework  INFO: -o:  --option=  :  Option=value to add
2008-09-08 15:15:21 UTC Framework  INFO: -s:  --section=  :  Set base section for relative parsed options
2008-09-08 15:15:21 UTC Framework  INFO: -c:  --cert=  :  Use server certificate to connect to Core Services
2008-09-08 15:15:21 UTC Framework  INFO: -h  --help  :  Shows this help
2008-09-08 15:15:21 UTC Framework  INFO: -  --Status=  :  ProductionDB file status to select
2008-09-08 15:15:21 UTC Framework  INFO: -  --OutputFile=  :  Output file to store file records
2008-09-08 15:15:21 UTC Framework  INFO: -  --Summary=  :  Print a summary of the files (True/False) default is True
2008-09-08 15:15:21 UTC Framework  INFO: -  --PrintOutput=  :  Print all file records (extremely verbose) default is False

Some examples:

dirac-production-file-summary 3030


Summary for 3681 files in production 3030

Status       Total       Percentage  

Assigned     3134         85 % 
Processed    433          11 % 
MissingLFC   114          3  % 

dirac-production-file-summary 3030 --Status=MissingLFC

Summary for 3681 files in production 3030

Status       Total       Percentage  

Assigned     3134         85 % 
Processed    433          11 % 
MissingLFC   114          3  % 


114 / 3681 files (3 percent) were found for production 3030 in status "MissingLFC"

The real output can optionally be piped to a specified file:

dirac-production-file-summary 3030 --Status=MissingLFC --OutputFile=lfc.txt

Summary for 3681 files in production 3030

Status       Total       Percentage  

Assigned     3134         85 % 
Processed    433          11 % 
MissingLFC   114          3  % 


114 / 3681 files (3 percent) were found for production 3030 in status "MissingLFC"
Wrote 114 lines to file lfc.txt for status "MissingLFC"

More verbose output is also available:

dirac-production-file-summary 3029 --Summary=False --PrintOutput=True

Status = Processed    LFN = /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00000237_1.rdst TargetSE = Unknown      JobID = 23           
Status = Assigned     LFN = /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00001654_1.rdst TargetSE = Unknown      JobID = 230          
Status = Assigned     LFN = /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00003027_1.rdst TargetSE = Unknown      JobID = 2300         
Status = Assigned     LFN = /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00003064_1.rdst TargetSE = Unknown      JobID = 2301         
Status = Assigned     LFN = /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00002031_1.rdst TargetSE = Unknown      JobID = 2302         
Status = Assigned     LFN = /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00003250_1.rdst TargetSE = Unknown      JobID = 2303         
Status = Assigned     LFN = /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00002911_1.rdst TargetSE = Unknown      JobID = 2304  
... 

dirac-production-file-status

To obtain the status of a single file (optionally for a specified productionID).

dirac-production-file-status --help

2008-09-08 15:28:54 UTC Framework  INFO: Usage:
2008-09-08 15:28:54 UTC Framework  INFO: scripts/dirac-production-file-status (<options>|<cfgFile>)*
2008-09-08 15:28:54 UTC Framework  INFO: Options:
2008-09-08 15:28:54 UTC Framework  INFO: -o:  --option=  :  Option=value to add
2008-09-08 15:28:54 UTC Framework  INFO: -s:  --section=  :  Set base section for relative parsed options
2008-09-08 15:28:54 UTC Framework  INFO: -c:  --cert=  :  Use server certificate to connect to Core Services
2008-09-08 15:28:54 UTC Framework  INFO: -h  --help  :  Shows this help
2008-09-08 15:28:54 UTC Framework  INFO: -  --ProductionID=  :  Restrict query to given production ID (default is to show status for all)

dirac-production-file-status /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00000237_1.rdst

{'Failed': {},
 'Successful': {'/lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00000237_1.rdst': {3029L: {'FileStatus': 'Assigned',
                                                                                                            'JobID': '23',
                                                                                                            'JobStatus': 'Created',
                                                                                                            'TargetSE': 'Unknown',
                                                                                                            'TransformationStatus': 'Stopped',
                                                                                                            'UsedSE': 'NIKHEF-tape'},
                                                                                                    3030L: {'FileStatus': 'Processed',
                                                                                                            'JobID': '23',
                                                                                                            'JobStatus': 'Completed',
                                                                                                            'TargetSE': 'Unknown',
                                                                                                            'TransformationStatus': 'Active',
                                                                                                            'UsedSE': 'NIKHEF-tape'}}}}

dirac-production-file-status /lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00000237_1.rdst 3030

{'Failed': {},
 'Successful': {'/lhcb/production/DC06/phys-v4-lumi5/00001897/RDST/0000/00001897_00000237_1.rdst': {3030L: {'FileStatus': 'Processed',
                                                                                                            'JobID': '23',
                                                                                                            'JobStatus': 'Completed',
                                                                                                            'TargetSE': 'Unknown',
                                                                                                            'TransformationStatus': 'Active',
                                                                                                            'UsedSE': 'NIKHEF-tape'}}}}


dirac-lhcb-production-check-files

This table shows the possible inconsistencies that may arise during normal operations. The dirac-lhcb-production-check-files script should allow Grid experts to identify and rectify these problem files/jobs. This table should be edited to ensure that all relevant cases are considered and the relevant action performed.

# WMS job state prodDB input file state BK,LFC output file status Action
1 Done Processed OK No action
2 Done Processed not OK file => Unused
3 Done Assigned/Unused OK file => Processed
4 Done Assigned/Unused not OK file => Processed; clean up BK, LFC?
5 Failed Processed OK WMS job => Done
6 Failed Processed not OK file => Unused
7 Failed Assigned/Unused OK clean up BK,LFC
8 Failed Assiigned/Unused not OK No action
9 Stalled Processed OK ?
10 Stalled Processed not OK ?
11 Stalled Assigned/Unused OK ?
12 Stalled Assigned/Unused not OK ?

  • RUNNING : If jobs are RUNNING then we should just leave them alone until they fall into one of the above states.
  • COMPLETED : What should I do if jobs are COMPLETED ? Originally I was equating them with DONE. However, COMPLETED means that there are some files which have not yet finished transferring, in which case it is expected that the prodDB and LFC/BK could be in some inconsistent state.

-- StuartPaterson - 08 Sep 2008 -- GreigCowan - 21 Sep 2008


This topic: LHCb > WebHome > LHCbComputing > ProductionProcedures > ProductionDB
Topic revision: r3 - 2008-09-22 - GreigCowan
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback