Reference implementation for prestaging

A python script has been developed as a reference implementation to make prestaging. It is based on the GFAL Python API. See file prestage.py attached to this page.

Detailed behaviour of the ultimate prestating script.

Open issues:

-l --pinlifetime option

Using the -l option: with dCache SURL the pin lifetime is not returned by the SRM. Result of the prestage_status is:

results:  {
'status': 0, 
'surl': 'srm://gridka-dCache.fzk.de/pnfs/gridka.de/lhcb/disk-only/lhcb/production/DC06/phys-v2-lumi5/00001877/DST/0000/00001877_00000196_5.dst', 
'turl': None,
 'explanation': '[SE][StatusOfBringOnlineRequest] AsyncWait'}

Whereas with CASTOR the result is

results:  {
'status': 1,
 'surl': 'srm://srm-lhcb.cern.ch/castor/cern.ch/grid/lhcb/production/DC06/phys-v2-lumi5/00001877/DST/0000/00001877_00000196_5.dst', 
'turl': None,
 'explanation': '[SE][StatusOfBringOnlineRequest] SRM_FILE_PINNED', 
'pinlifetime': 600}

the 'pinlifetime' key is not present in the result dictionary for dCache surl.

I traced the process of prestaging for the dCache surl. The trace file is attached (). The pin lifetime is correctly send from the client to SRM server. But later, in the srmBringOnlineResponse the pin lifetime is not returned. The field 'remainingPinTime' has no value assigned.

Tracing the same process with a CASTOR surl: client sends the request to server specifying 600, then the server answers and in the srmBringOnlineResponse block the remaining pin time is correctly returned: 600.

Why doesn't dCache server return the remaining pin time?

This is a known issue: bug # 4154 in dCache RT

Input and output arguments

Using the script to prestage with the following options: -r 60 -l 600 the arguments passed by the client to the server for the srm BOL request:

request: srmBringOnlineRequest

arrayOfFileRequests 
 requestArray 
  sourceSURL
    srm://srm-lhcb.cern.ch/castor/cern.ch/grid/lhcb/production/DC06/phys-v2-lumi5/00001877/DST/0000/00001877_00000196_5.dst
  sourceSURL
 requestArray
arrayOfFileRequests

desiredFileStorageType PERMANENTdesiredFileStorageType

desiredLifeTime 600 desiredLifeTime

transferParameters  arrayOfTransferProtocols
dcap
gsidcap
root
rfio
file
arrayOfTransferProtocols transferParameters

Answer of the CASTOR server:

returnStatus 
   statusCode SRM_REQUEST_QUEUED statusCode 
   explanation explanation
returnStatus

requestToken 5275365 requestToken

arrayOfFileStatuses
  statusArray
     sourceSURL srm://srm-lhcb.cern.ch/castor/cern.ch/grid/lhcb/production/DC06/phys-v2-lumi5/00001877/DST/0000/00001877_00000196_5.dst 
sourceSURL
    status
       statusCode SRM_REQUEST_QUEUED statusCode
       explanation explanation
    status
    fileSize 246689603 fileSize 
    estimatedWaitTime 5 estimatedWaitTime
    remainingPinTime 600 remainingPinTime
  statusArray
arrayOfFileStatuses

Answer of dCache server:

returnStatus xsi:type="ns1:TReturnStatus" statusCode xsi:type="ns1:TStatusCode"  SRM_REQUEST_QUEUED  statusCode 
   explanation xsi:type="xsd:string" 
        at Wed Feb 25 16:05:57 CET 2009 state Pending : created
   explanation
returnStatus

requestToken xsi:type="xsd:string" -2071654185 requestToken

arrayOfFileStatuses xsi:type="ns1:ArrayOfTBringOnlineRequestFileStatus"
   statusArray xsi:type="ns1:TBringOnlineRequestFileStatus"
       sourceSURL xsi:type="xsd:anyURI" srm://gridka-dCache.fzk.de/pnfs/gridka.de/lhcb/disk-only/lhcb/production/DC06/phys-v2-lumi5/00001877/DST/0000   /00001877_00000196_5.dst sourceSURL
       status xsi:type="ns1:TReturnStatus"
           statusCode xsi:type="ns1:TStatusCode" SRM_REQUEST_QUEUED statusCode 
           explanation xsi:type="xsd:string" TQueued explanation 
       status
       fileSize xsi:type="xsd:unsignedLong" xsi:nil="true" 
       estimatedWaitTime xsi:type="xsd:int" 4 estimatedWaitTime
       remainingPinTime xsi:type="xsd:int" xsi:nil="true" 
    statusArray
arrayOfFileStatuses

In summary:

• estimatedWaitTime is returned by both dChace and CASTOR servers but is not returned by GFAL client. Opened Savannah bug 47489. Remi said it will be implemented soon, in the next release (not the one which is already in certification, the next one)
• remainingPinTime is returned by CASTOR (not by dCache, this is a known issue: bug # 4154 in dCache RT ) and is correctly returned by GFAL.bug already opened (fixed when?)
• fileSize is returned by CASTOR and not by dCache (when the request is queued). And GFAL doesn't return it. In the following polling, once the request is done, then also dCache returns the file size. doesn't seem to be important
• DesiredTotalRequestTime is not even passed by GFAL to the SRM server (it is implemented for prepareToGet in the current version of GFAL, but not for BringOnline. In the version in certification it is implemented also for BringOnline).Finally, after the 26 Feb meeting, it was decided that this parameter is optional
• TotalRemainingRequestTime: not set in dCache. This parameter is reported in the following srmBringOnlineResponse (the prestage_status calls), not in the first one (why?) but its value is not set. In CASTOR, this parameter is not even present in the trace file. in the last meeting, Feb 26th, it has been decided that it's not important

-- ElisaLanciotti - 25 Feb 2009