Difference: ProductionJobFinalization (3 vs. 4)

Revision 42009-03-04 - StuartPaterson

Line: 1 to 1
 
META TOPICPARENT name="ProductionProcedures"
-- AndreiTsaregorodtsev - 16 Oct 2008
-- StuartPaterson - 03 March 2009
Line: 13 to 13
 

UploadOutputData

Changed:
<
<
This module establishes the relevant metadata for output files and attempts to transfer and register files with failover after resolving the appropriate destination SE. The BK replica flags are set automatically in the case of a successful transfer and are added to a failover request in case of upload failures. If the destination SE is not available files can be transferred to a Tier1-FAILOVER SE (all are attempted).
>
>
This module establishes the relevant metadata for output files (such as the POOL GUID(s) from local catalogs) and attempts to transfer and register files with failover after resolving the appropriate destination SE. The BK replica flags are set automatically in the case of a successful transfer and are added to a failover request in case of upload failures. If the destination SE is not available files can be transferred to a Tier1-FAILOVER SE (all are attempted).
 

UploadLogFile

Changed:
<
<
Logs are always uploaded regardless of the workflow status. This module will copy and register files to Grid storage in case of failure and set the appropriate requests to recover at the end. An attempt is made to change the permissions of the files to be readable from the LogSE but this depends on the site specific settings and can fail (printed in the logs in this case).
>
>
Logs are always uploaded regardless of the workflow status. This module will copy and register files to Grid storage in case of failure and set the appropriate requests to recover at the end. An attempt is made to change the permissions of the files to be readable from the LogSE but this depends on the site specific settings and can fail (printed in the payload logs in this case). Some sites have strange permissions resulting in log files not appearing on the LogSE URL by default so we may need to introduce a server side action there.
 

FailoverRequest

Line: 36 to 37
 

Case 1. Application successful

Changed:
<
<
1. Log files are uploaded to the Log Storage Element. If the upload fails, the log files are tarred and put into the Failover system. This is always performed regardless of workflow status in order to ensure logs availability even in case of a crash in the subsequent steps.

2. Bookkeeping records (not replica flags) are sent to the Bookkeeping service prior to uploading the data files. If any of the bookkeeping records sending fails a corresponding failover request is created.

>
>
1. Bookkeeping records (not replica flags) are sent to the Bookkeeping service prior to uploading the data files. If any of the bookkeeping records sending fails a corresponding failover request is created.
 
Changed:
<
<
3. Output data upload. For each output file the destination Storage Elements are resolved according to the job workflow parameters. The upload to the specified destination is attempted with registration in all the DIRAC catalogs ( LFC, BookkeepingDB(replica flag), ProductionDB). If upload fails, the file is uploaded to one of the FAILOVER storages and the corresponding failover request is created.
>
>
2. Output data upload. For each output file the destination Storage Elements are resolved according to the job workflow parameters. The upload to the specified destination is attempted with registration in all the DIRAC catalogs ( LFC, BookkeepingDB(replica flag), ProductionDB). If upload fails, the file is uploaded to one of the FAILOVER storages and the corresponding failover request is created.
  If several destinations are specified for a given file, they are attempted in turn until the first successful upload. In this case replication requests are created to copy the file to other specified destinations. If all the specified destinations fail, the file is uploaded to one of the FAILOVER storages and the corresponding failover replication requests are created.
Line: 49 to 48
  If for at least one output data file neither upload (destination or failover) is successful:

  • the job is declared Failed;
Changed:
<
<
  • all previously defined replication requests are dropped except for log files failover request if any;
>
>
  • all previously defined requests for other output files are dropped (except for log files failover request if any);
 
  • data removal requests are created for already uploaded files;
  • the input data files are set to "Unused" in the ProductionDB;
  • the already sent bookkeeping records have no replica flags.

Added:
>
>
3. Log files are uploaded to the Log Storage Element. If the upload fails, the log files are tarred and put into the Failover system. This is always performed regardless of the workflow or step status in order to ensure logs availability.

 4. The combined request is written into a file to be picked up by the Job Wrapper

Case 2. Application failed

Changed:
<
<
  1. Log files are uploaded to the Log Storage Element. If the upload fails, the log files are tarred and put into the Failover system. This is always performed regardless of workflow status in order to ensure the log availability even in case of a crash in the subsequent steps.
>
>
  1. Log files are uploaded to the Log Storage Element. If the upload fails, the log files are tarred and put into the Failover system. This is always performed regardless of the workflow or step status in order to ensure log availability.
 
  1. Input files status is updated in the Production DB Service as "Unused" except for files marked as "ApplicationCrash" by the application log analysis module. If the update fails, the corresponding failover request is created.
  2. The combined request is written into a file to be picked up by the Job Wrapper.
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback