Show Children Hide Children

Main FTS Pages
FtsRelease22
Install
Configuration
Administration
Procedures
Operations
Development
Previous FTSes
FtsRelease21
FtsRelease21
All FTS Pages
FtsWikiPages
Last Page Update
Main.unknown
2010-02-03

FTS internal documentation

For a list of project configurations check FtsDevelopmentConfigurations.

Ongoing developments

CVS Modules relation

DirectedGraphPlugin_1.png diagram

WSI-Compliant WSDL

An attempt to create a new WSI-compliant WSDL for FTS, along with related python and c bindings: FtsWsdl.

Database

Modules

Transfer-interface

The web service interface definition, from which the wsdl is generated. See GliteDataTransferInterface.

Transfer-fts

The web service implementation. See GliteDataTransferFts.

Transfer-api-c

C bindings for the web service APIs. See GliteDataTransferApiC.

Transfer-cli

The command line interface (for channel and transfer management). See GliteDataTransferCli.

Config Service

Implementation of the component configurator pattern. Used to load and configure libraries, start and stop services. See OrgGliteDataConfigService.

Agents Common

Base libraries and utilities. See GliteDataAgentsCommon.

Srm Util Cpp

See GliteDataSrmUtilCpp.

Transfer Url Copy

See GliteDataTransferUrlCopy.

Transfer Agents

See GliteDataTransferAgents.

File/job state machines: TransferAgentsStateMachine.

Transfer Monitoring

See also FTSMonitoringGroup

Service design, implementation, deployment and operations notes

Various notes tracking the design, implementation, deployment and operational issues of the transfer service: FTSDesignNotes.

Diagrams detailing the interaction between FTS and SRM implementations.

See FTSSRMSequenceDiagrams for urlcopy channels.

See SrmCopySequenceDiagrams for srmcopy channels.

Notes

Tests

See FtsTestGuide

Install or upgrade the pilot instance

See FtsPilot

Load Generator

See GliteDataTransferLoadGenerator.

Local Tests, Unit tests

See FtsWSTest

Monitoring

Known endpoints and see also LCG.LCGFTMEndpoints:

Development testbed

Current resources and tasks assigned:

Box Type Description
fts006 - Development test box
fts107 pilot ftsws
fts108 pilot ftschannel pilot channel agents, swrep currently off
fts109 pilot ftschannel Pilot channel and vo agents
fts101 prod ftsws
fts102 prod ftsmon mod_gridsite enabled webserver.
fts110 prod ftschannel
fts111 prod ftschannel
fts112 prod ftschannel
fts113 prod ftsvo
fts114 prod ftsws
fts115 prod ftsws
fts103 tiertwo ftsws
fts104 tiertwo ftsws
fts105 tiertwo ftschannel
fts106 tiertwo ftsvo
fts116 SL4 PILOT ftsws
fts118 SL4 PILOT channel/VO
fts119 SL4 PILOT channel

Also look at: EGEE.DMTestBed for the general virtual DM testbed.

Current DB accounts on validation RAC:

Account Who Until when
lcg_fts_stresstest1 Paolo indefinite
lcg_fts_stresstest2 Steve indefinite

See /afs/cern.ch/project/gd/SC3/SC4/dbpasswords/

Internal training notes

General Description and concepts

FTS, stands from File Transfer Service, is compound of DB, a set of daemons, nomitoring tool FTM and the WebServer.

Users interact with the WS via the commmand line, although there is a WSDL avialable.The CLI tools allow to do administration operation and submit jobs.

FTS distributes jobs into channels, A channel is not a network link, it's not pysically releated to the network. It's a job queue.

The jobs submitted are uin a queue and progress in different states like a status machine thanks to agents actions.These actions are grouped into 2 kind of agents (channel and VO agents).See GliteDataTransferAgents.

User Case --Example

  • A user (dteam user) submits a job from source to destination given by SURL issuing the command glite_transfer_submit options source destination. The source and destination is given as SURL, such as srm://castor-public-cern.ch:8043/srm/managerV2?SFN=/dteam/public... The job status is submitted.

  • The dteam VO agents perform its actions:
    • Allocate which consists of picking jobs in submit state belonging to dteam. So it sees
    • Look the corresponding site for the SURL , i.e, CERN-PROD and Dest F2K-LCG2
    • Look for an available channel The job state is ready

  • Now the CERN -F2k channel agent enter in the action.
    • The action FETCH will search for jobs in state = ready and allocated to a channel belonging to any VO.
    • Then according the VO Shares it will allocate jobs
    • Start the transfer ant the job state is active.
    • The transfers are managed by external processes

  • The channel agent will perform the CHECK STATE
  • When the transfer is over the job goes to finished.

Note: Sites do not belong to a VO. A Channel is a resource shared between groups. Channels are static, created in the DB and configure them with Yaim.

Plug-in idea

The implementation model is made through plug-ins.There is a plug in for the DB. There is a plug in for actual transfer library, but in practice there is only one implementation which is tranfer-url-copy library There is a stub transfer service used for unit test, and for installation test.

URL-copy and SRM-copy.

Types of transfers: URL-copy and SRM-copy. In both cases FTS uses the SRM interface from the storage elements. SRM is defined in http://sdm.lbl.gov/srm-wg/doc/SRM.v2.2.html

  • URL Copy -FTS uses the SRM interface for the transfer negociation process and then start the gridftp tranfer. See GliteDataTransferUrlCopy.
  • SRM copy - it relies completely on a operation of the SRM interfeace srm-copy. You delegate to the server the copy process. SRM copy channels , ping the source, crate destination directory and delegates the responsabilitiy of the copy as the user credentials to the server.The end point is in charge of the transfer. See GliteDataSrmUtilCpp.
    • Push -- the source contacts the destination
    • Pull -- is the opposite.

Inetrnal notes - lessons learned


Last edit: UnknownUser on 2010-02-03 - 10:57

Number of topics: 1

Maintainer: PaoloTedesco


Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg fts-salami-logo.jpg r1 manage 140.2 K 2009-03-31 - 16:23 AkosFrohner  
PNGpng fts-salami-logo.png r1 manage 885.4 K 2009-03-31 - 16:24 AkosFrohner  
Edit | Attach | Watch | Print version | History: r32 < r31 < r30 < r29 < r28 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r32 - 2010-02-03 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright & by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback