File Transfer Service : User Guide

Introduction

The File Transfer Service (FTS) is the lowest-level data movement service defined in the gLite architecture. It is responsible for moving sets of files from one site to another, allowing participating sites to control the network resource usage. It is designed for point to point movement of physical files.

Basic Concepts

The concepts used in the FTS are defined here:

Job

A job or transfer job consists of a set of files to be transferred. A job contains:

• a list of Files (source / destination pairs).
• an optional parameters key/value pair block, for passing options to the underlying transport layer.

The only supported options are the gridFTP parameters for TCP buffer size and number of streams. If present, the key must be gridFTP with the values specified as -tcp-bs 1000 -p 20 in the format used by the globus-url-copy commandline tool. The options are advisory to the service. If not specified, the service defaults are used. It is recommended to use the service defaults, i.e. to ignore this option.

• a MyProxy password - this will be used together with the client's subject name to retrieve the user's

proxy to be used for the transfer.

File

Refers to a source / destination pairs to be transferred.

• source - this identifies the source physical filename. It should be in Storage URL (SURL) format, suitable

for interaction with an SRM, or in gridFTP format (in the format used in globus-url-copy). It is a current restriction of the service that the SURL must be in the fully specified form (i.e. including the SRM endpoint).

• destination - this identifies the destination physical filename. It should be in Storage URL (SURL) format,

suitable for interaction with an SRM, or in gridFTP format (in the format used in globus-url-copy). It is a current restriction of the service that the SURL must be in the fully specified form (i.e. including the SRM endpoint).

Job state

The overall state of a job.

• This state is a function of the File states of the individual files that constitute the job.

File state

The state of an individual File transfer.

It is important to note that the FTS deals only with physical files, and does not provide any facility for dealing with logical files or collection-like entities. These can be dealt with by higher level services. In particular:

• Logical source files - are dealt with by the File Placement Service.
• Logical destinations - are dealt with by the File Placement Service (i.e. 'put the file at RAL').

The jobs and their constituent files are processed aychronously. Upon submission the client is handed a job identifier that can later be used to query the status of the job as it progresses through the system. Various statuses of the jobs (basic state, full summary, individual file states) can be queried uisng this job identifier. The job can also be cancelled using it.

The order of job processing is currently FIFO, according to submission time. More complex policies may be defined in the future.

Channel Assignment

Once a job has been submitted to the system it is assigned a transfer channel based on the files that it contains.

Channel

A channel is a specific (perhaps dedicated) network pipe for transferring files.

• Production channels - typically dedicated tier-0 to tier-1, tier-1 to tier-1, or tier-2 to tier-1 network pipes.

Production transfer jobs run on these channels to ensure efficient bulk distribution. All files in a job must be assignable to the same channel, otherwise the job will not be assigned to a production channel.

• Non-production channels - typically open networks shared with other applications. Jobs not otherwise assignable to

a production channel may be serviced on the open network with whatever quality of service is available there. If the FTS administrator has not configured a non-production channel on the FTS, the job will be marked Failed.

The exact configuration and topology of channels is defined by the network providers, sites and VOs. The user cannot control what channel is assigned to a given job; it is assigned by the transfer system according to the VO, site and network provider policy.

Intelligent job submission and reorganising of jobs to make optimal use of the available network and channel topology is work for a higher level service.

Service management and control of channels is described in the adminstrator section of the FTS commandline documentation.

States

This section gives an oveview of the job and file states and what they mean.

The Job States describe the overall progress of the job through the system; this is the state that is typically the most useful for the user to ask for and answers the most common question (``is my job done yet?''):

• Submitted. The job has just been submitted but has not yet been assigned to a channel.
• Pending. The job has now been assigned to a channel and its files are awaiting transfer.
• Active. Some of a job's files are active on the network (i.e. are currently being transferred).
• Done. All files in the job were transferred successfully. This is a terminal state.
• Failed. One or more of the job's files Failed. This is a terminal state. It indicates that the

client should request more information to see which files succeeded and which failed.

• Cancelling. The job is in the process of being cancelled.
• Cancelled. The job has been cancelled. This is a terminal state. This state should be investigated further,

since any files that have transferred successfully are not deleted (i.e. their File State will be Done).

• Hold. Indicating that there has been a failure in some of the job's files that cannot be resolved

automatically (typically the files will have been retried a few times before entering this state). Jobs in this state require manual intervention. The administrator section of the commandline guide describes the possible manual interventions.

The File states refer to the current state of individual files within a job:

• Submitted. The job has just been submitted but has not yet been assigned to a channel: for consistency,

all the file states are marked as Submitted as well.

• Pending. The job that owns this file has now been assigned to a channel and the file is awaiting transfer.
• Active. The file is currently being transferred.
• Done. The file has been transferred successfully. This is a terminal state.
• Failed. The file transfer failed and could not be recovered. This is a terminal state. The reason for the

failure can be retrieved using the relevant service method.

• Wait. The file has failed once and is a candidate for retry. A retry agent will either retry the file again

(by setting it back to Pending) or will set the file state to Hold or Failed.

• Hold. The file has failed a few times and the system has marked it for manual intervention.
• Cancelled. The job containging this file has been cancelled. This is a terminal state. Again, note that

if a file transfer completes before the cancel operation is called, the file state will be 'Done' (or 'Failed' if it failed) rather than Cancelled.

Interactions with other Services

The FTS makes use of MyProxy.

• Before submitting a job, the client is expected to upload an appropriate

credential (e.g. their user grid certificate) into the same MyProxy server used by the FTS. The upload should be made using the DN-as-username mode, since the DN of the client who submitted the job is used by the FTS to subsequently retrieve the credential from MyProxy for that job. The password used is passed to the FTS in the Job submission process.

• The same credential is used for all of a client's transfers.
• The maximum validity of the credential in MyProxy should be long enough to finish the last transfer,

taking into account queuing time. The retrieved credential is automatically refeshed from MyProxy by the FTS while it is still needed (up to the maximum validity time the client specified upon the original upload of the credential to MyProxy).

• Refer to the MyProxy user documentation for details of how to upload a credential.

-- PeterKunszt - 20 Oct 2005