FTS internal documentation

For a list of project configurations check FtsDevelopmentConfigurations.

Ongoing developments

CVS Modules relation

digraph G { TransferAgents [shape=box]; TransferURLCopy [shape=box]; AgentsCommon [shape=box]; SRMUtility [shape=box]; SRM11 [shape=box]; SRM22 [shape=box]; TransferCli [shape=box]; TransferFTS [shape=box]; TransferInterface [shape=box]; TransferAgents -> AgentsCommon; TransferAgents -> TransferURLCopy; TransferURLCopy -> AgentsCommon; TransferURLCopy -> SRMUtility; SRMUtility -> SRM11; SRMUtility -> SRM22; TransferCli -> TransferInterface; TransferFTS -> TransferInterface; }

WSI-Compliant WSDL

An attempt to create a new WSI-compliant WSDL for FTS, along with related python and c bindings: FtsWsdl.

Database

• Description of the DB tables. See FtsDbSchema.
• DB package. See LCG/FtsAdminTools21.
• DB code Guide and register package LCG/FtsDbCodeGuide.
• Documented changes in the schema and corresponding upgrade scripts DBSchemaChanges

Modules

Transfer-interface

The web service interface definition, from which the wsdl is generated. See GliteDataTransferInterface.

Transfer-fts

The web service implementation. See GliteDataTransferFts.

Transfer-api-c

C bindings for the web service APIs. See GliteDataTransferApiC.

Transfer-cli

The command line interface (for channel and transfer management). See GliteDataTransferCli.

Config Service

Implementation of the component configurator pattern. Used to load and configure libraries, start and stop services. See OrgGliteDataConfigService.

Agents Common

Base libraries and utilities. See GliteDataAgentsCommon.

Srm Util Cpp

See GliteDataSrmUtilCpp.

Transfer Url Copy

See GliteDataTransferUrlCopy.

Transfer Agents

See GliteDataTransferAgents.

File/job state machines: TransferAgentsStateMachine.

Transfer Monitoring

See also FTSMonitoringGroup

Service design, implementation, deployment and operations notes

Various notes tracking the design, implementation, deployment and operational issues of the transfer service: FTSDesignNotes.

Diagrams detailing the interaction between FTS and SRM implementations.

See FTSSRMSequenceDiagrams for urlcopy channels.

See SrmCopySequenceDiagrams for srmcopy channels.

Notes

• Error category/scope/phase descriptions: FTSErrorClassification.
• Channels resolution order: LCG/FtsChannelResolution.
• How channel shares are calculated: FtsChannelSharesCalculation.
• How FTS handles space tokens / space tokens descriptions: FtsSpaceTokensHandling.
• Implementation of the Addendum to the SRM v2.2 WLCG Usage Agreement: FtsAddendumToWlcgUserAgreement.
  • For the "Addendum to the SRM v2.2 WLCG Usage Agreement" document check LCG/WLCGCommonComputingReadinessChallenges.
• How timeouts are calculated with new parameters introduced in version 2.2: FtsTransferTimeouts22.
• Checksum details: FtsChecksums.

Tests

See FtsTestGuide

Install or upgrade the pilot instance

See FtsPilot

Load Generator

See GliteDataTransferLoadGenerator.

Local Tests, Unit tests

See FtsWSTest

Monitoring

Known endpoints and see also LCG.LCGFTMEndpoints:

• CERN
  • http://fts-monitor.cern.ch/ftsmonitor/fts-t0-export/fts.php
  • http://fts-monitor.cern.ch/ftsmonitor/fts-t0-export-21/fts.php
  • http://fts-monitor.cern.ch/ftsmonitor/fts-t2-service/fts.php
  • http://fts-monitor.cern.ch/ftsmonitor/fts-t2-service-21/fts.php
  • http://fts-monitor.cern.ch/ftsmonitor/fts-pps/fts.php
  • http://fts-monitor.cern.ch/ftsmonitor/fts-pilot/fts.php
  • http://fts-admin.cern.ch/dashboard/request.py/fts
• FN-T1 FTM endpoint (FTM endpoint at CNAF):
  • http://tier1.cnaf.infn.it/ftmmonitor/transfer-monitor-report/
  • FTS monitor tool developed by in2p3, available at: http://tier1.cnaf.infn.it/ftsmonitor/

Development testbed

Current resources and tasks assigned:

Also look at: EGEE.DMTestBed for the general virtual DM testbed.

Current DB accounts on validation RAC:

See /afs/cern.ch/project/gd/SC3/SC4/dbpasswords/

Internal training notes

General Description and concepts

FTS, stands from File Transfer Service, is compound of DB, a set of daemons, nomitoring tool FTM and the WebServer.

Users interact with the WS via the commmand line, although there is a WSDL avialable.The CLI tools allow to do administration operation and submit jobs.

FTS distributes jobs into channels, A channel is not a network link, it's not pysically releated to the network. It's a job queue.

The jobs submitted are uin a queue and progress in different states like a status machine thanks to agents actions.These actions are grouped into 2 kind of agents (channel and VO agents).See GliteDataTransferAgents.

User Case --Example

• A user (dteam user) submits a job from source to destination given by SURL issuing the command glite_transfer_submit options source destination. The source and destination is given as SURL, such as srm://castor-public-cern.ch:8043/srm/managerV2?SFN=/dteam/public... The job status is submitted.

• The dteam VO agents perform its actions:
  • Allocate which consists of picking jobs in submit state belonging to dteam. So it sees
  • Look the corresponding site for the SURL , i.e, CERN-PROD and Dest F2K-LCG2
  • Look for an available channel The job state is ready

• Now the CERN -F2k channel agent enter in the action.
  • The action FETCH will search for jobs in state = ready and allocated to a channel belonging to any VO.
  • Then according the VO Shares it will allocate jobs
  • Start the transfer ant the job state is active.
  • The transfers are managed by external processes

• The channel agent will perform the CHECK STATE
• When the transfer is over the job goes to finished.

Note: Sites do not belong to a VO. A Channel is a resource shared between groups. Channels are static, created in the DB and configure them with Yaim.

Plug-in idea

The implementation model is made through plug-ins.There is a plug in for the DB. There is a plug in for actual transfer library, but in practice there is only one implementation which is tranfer-url-copy library There is a stub transfer service used for unit test, and for installation test.

URL-copy and SRM-copy.

Types of transfers: URL-copy and SRM-copy. In both cases FTS uses the SRM interface from the storage elements. SRM is defined in http://sdm.lbl.gov/srm-wg/doc/SRM.v2.2.html

• URL Copy -FTS uses the SRM interface for the transfer negociation process and then start the gridftp tranfer. See GliteDataTransferUrlCopy.
• SRM copy - it relies completely on a operation of the SRM interfeace srm-copy. You delegate to the server the copy process. SRM copy channels , ping the source, crate destination directory and delegates the responsabilitiy of the copy as the user credentials to the server.The end point is in charge of the transfer. See GliteDataSrmUtilCpp.
  • Push -- the source contacts the destination
  • Pull -- is the opposite.

Inetrnal notes - lessons learned

• Proxy delegation problems in FTS 2.2: ProxyDelegationProblemFTS22

Last edit: UnknownUser on 2010-02-03 - 10:57

Maintainer: PaoloTedesco