Direct access and the Pilots

Introduction

Preliminaries

This section describes under which circumstances the pilot uses direct access.

AGIS settings respected by the Pilots

In general, direct access for a given queue can be controlled with the direct_access_lan and direct_access_wan AGIS settings. See the table below for information.

Specially Requested Access Mode in Job Parameters

The following job parameter options instructs the pilot to force an access mode:

Supported File Formats and Schemas

The file formats supported for direct access is not restricted by the pilots, but limited: the pilot allows any file format to be considered for direct access unless the file name contains the patterns '.tar.gz', '.lib.tgz' or '.raw.'. Normally this should mean that ROOT files are left.

The following replica schemas are allowed: 'root://', 'dcache://', 'dcap://', 'https://' and 'file://'.

Note: if prodDBlockToken in the job definition is set to 'local', direct access will be switched off.

Direct Access in Pilot 1

The mechanism for direct access is different for production versus user jobs. Below follows a description of the different cases.

Direct Access in Production Jobs

Direct access mode for production jobs are defined by the transferType='direct' job definition parameter. Such a job will be generated if the allowInputLAN='only' is set on the task definition level.

For production jobs that use direct access, the pilot swaps the LFN:s from the job parameters to corresponding TURL:s, as returned by the list_replicas() function.

Direct Access in User Jobs

Direct access in user jobs are normally controlled by AGIS. The user can switch off direct access on a given site with the --forceStaged prun/pAthena option for special needs. The user can also request only sites with direct access turned on by using the --useDirectIOSites option.

Furthermore, a commonly used HammerCloud option is the --accessmode job parameter (see above), which is used to force tasks to either use direct access or copy-to-scratch. This is an internally used pilot instruction, so the pilot remove the option from the job parameters before the payload is executed.

When a user job is going to use direct access, the pilot adds --usePFCTurl option to the prun/pAthena command.

Direct Access in Pilot 2

As of September 2018, direct access support in Pilot 2 works in the same way as in Pilot 1.

-- PaulNilsson - 2018-05-23