Difference: DiracPopularityService (4 vs. 5)

Revision 52012-01-21 - DanielaRemenska

Line: 1 to 1
META TOPICPARENT name="DanielaRemenskaWork"

Points to remember (compiled from meetings and notes):

Line: 7 to 7
  • Implementation should be inside LHCbDirac
  • The only information that should be sent by the job are the LFNs and respective Storage Elements. More precisely, in the finalization step, already existing also for users' jobs, a new module should be added. The information will be taken from:
    • The XML file (summary.xml), where the list of input files (LFNs) is reported. The "XML summary" is produced by the LHCb applications independently. It is inspected by LHCbDIRAC to know if the application reached the end of the computation (i.e. if it has processed all the input events).
The "XML file" of the job is the "jobDescription.xml" file that is uploaded with all the logs of the jobs. It contains the job description (which is the DIRAC worfklow) in an XML format. It is created before the job is submitted, and contain information on what to run. The "XML summary" is instead produced by the LHCb applications independently. (note to self:This means forget about summary.xml)
    • The SE has to be obtained from DIRAC, it is not in the XML file. We can also record the site if easier, but formally the SE is the real interesting entity. The job knows it when resolving the input data.
  • Information should be passed onto the finalization step on the files that were actually used by the job, not requested (in case for example someone has 20 input files but reads only 10 events). This also means that partially used files are reported to the Popularity Service?
Line: 100 to 101
 ALERT!The AnalyseXMLLogFile implementation parses the application log AND the summary.xml, but it uses LHCbDIRAC/Core/Utilities/ProductionXMLLogAnalysis.py which requires step_commons['listoutput'] to be defined.. user jobs may lack it, so it crashes :-/ Example output
2012-01-11 03:14:59 UTC dirac-jobexec/AnalyseXMLLogFile INFO: Initializing $Id: DiracPopularityService.txt,v 1.4 2012/01/17 16:39:30 danielar_40nikhef_2enl Exp $
2012-01-11 03:14:59 UTC dirac-jobexec/AnalyseXMLLogFile INFO: Initializing $Id: DiracPopularityService.txt,v 1.5 2012/01/21 22:28:54 danielar_40nikhef_2enl Exp $
 2012-01-11 03:14:59 UTC dirac-jobexec/AnalyseXMLLogFile DEBUG: {'ParametricInputData': '', 'TotalSteps': '1', 'JobName': 'Name', 'Priority': '1', 'SoftwarePackages': 'DaVinci.v29r2', 'JobReport': <DIRAC.WorkloadManagementSystem.Client.JobReport.JobReport instance at 0xd11bcf8>, 'LogLevel': 'debug', 'OutputSandbox': '*.log;summary.data;summary.xml', 'JobType': 'User', 'SystemConfig': 'ANY', 'JOB_ID': '00000000', 'StdError': 'std.err', 'Request': <DIRAC.RequestManagementSystem.Client.RequestContainer.RequestContainer instance at 0xd0eec68>, 'AccountingReport': <DIRAC.AccountingSystem.Client.DataStoreClient.DataStoreClient instance at 0xd19a830>, 'ParametricInputSandbox': '', 'JobGroup': 'lhcb', 'StdOutput': 'std.out', 'Origin': 'DIRAC', 'Site': 'ANY', 'PRODUCTION_ID': '00000000', 'MaxCPUTime': '5000', 'LogFilePath': '/project/bfys/dremensk/ctmdev/LHCbDirac_v6r8p2/etc', 'InputData': ''} 2012-01-11 03:14:59 UTC dirac-jobexec/AnalyseXMLLogFile DEBUG: {'applicationName': 'DaVinci', 'STEP_DEFINITION_NAME': 'DaVinciStep1', 'applicationVersion': 'v29r2', 'JOB_ID': '00000000', 'optionsLine': '', 'STEP_NUMBER': '1', 'StartStats': (4.2199999999999998, 0.23999999999999999, 0.0, 0.0, 8676995.6999999993), 'STEP_INSTANCE_NAME': 'RunDaVinciStep1', 'inputDataType': 'DATA', 'applicationLog': 'Step1_DaVinci_v29r2.log', 'optionsFile': '/project/bfys/dremensk/DaVinci-Default.py', 'PRODUCTION_ID': '00000000', 'STEP_ID': '00000000_00000000_1', 'StartTime': 1326251676.3210549, 'inputData': 'LFN:/lhcb/LHCb/Collision11/BHADRON.DST/00012957/0000/00012957_00000753_1.bhadron.dst'} 2012-01-11 03:14:59 UTC dirac-jobexec/AnalyseXMLLogFile INFO: Input data defined in workflow for this Gaudi Application step
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback