-- MichelJouvin - 2015-01-13---+!! pre-GDB on Data Access Protocols, CERN, January 13, 2015



Taming the Protocol Zoo - W. Bhimji

2 main topics for today

  • Are we on track to allow non-SRM disk-only sites?
  • Limit the number of services to run for disk-only sites

non-SRM sites

  • No attempt to develop/design an alternative to SRM but bring together all the existing activities enabling it
  • Many things happened since Annecy pre-GDB (Oct 2012) that defined steps: FTS3, federations, gfal2...
  • Identified development areas
    • Reporting space: RFC 4331
    • Delete: gfal2 deployed and helping with it
    • Checksum checks: gridftp2, now in DPM as in dCache

Protocol zoo

  • Made worst by SRM alternatives as experiments still require SRM from sites
  • Increased interest in new storage solutions like Ceph
  • Would be great if WLCG could use http/dav only sites and should not require more than 3 protocols (gridftp2, xroot)

Expected outcome of this meeting: action plan!

CMS - T. Wildish

Baseline for Run2: gridftp/FTS3, xrootd

  • Can probably use a non-SRM site but not aware of any and probably with a few caveats

Increasing need with Run2: user data created with CRAB or similar tools

  • ~1 TB/year/user
  • Shared with other users, local & remote
  • Also the need to download files to laptops/desktops, including request for batches of files

R&D activities around DM

  • Opportunistic resources, e.g. clouds: S3 may become useful for edge transfers
    • Use gateway (non 3d party)?
  • DM expected to evolve significantly by Run3
    • Less distinctions between official data versus user data: not clear yet what the implication will be


  • // transfer tehcnology like gridftp: WebDAV/S3 is not (not clear it is or is everywhere), will not be fast enough for serious use until supporting // transfers
    • Dirk (DD): main issue with S3 is authentication that is not compatible with X509
    • Adrien/Oliver mentioned that Fabrizio et. al have a translation module in place for converting between the two mechanisms - used with BNL
    • Paul Millar (PM): from tests done in dCache, no clear evidence of // TCP streams having any positive impact on performances (they are always multiple transfers in // on a loaded system): more critical is the improved management of TCP streams
  • CLI for users and tools to look up file existence: currently based on SRM, could use gfal2 if available everywhere but will need time to migrate and prefer not to do this in run-up to Run2
    • P. Charpentier (PC): gfal2 is easy to deploy in experiment SW, through CVMFS for example

LHCb - P. Charpentier

Different use cases

  • File transfer: dataset replications, imply (pre-)staging and thus pinning. Based on SRM
  • File access: can use any protocol supported by ROOT. Preference for root: and file:

In LHCb, all URLs can be constructed from LFN

  • But currently LFN -> xURL translation done by SRM
  • Can be built from LFN + SE description in DIRAC: full URL, no need for BDII to use it
  • For transfers, source and dest SURL contain the space token
  • File access: TURL requested through SRM, with an ordered list of preferred protocols
    • No problem to generate TURL in DIRAC: xroot plugin already exists, about to be released

SE federation through file catalog: start with local replica if it exists and then try others

Issues for getting rid of SRM

  • We need a single xrootd (local redirector based) endpoint at all sites: not the case at all sites today
  • gridftp redirection: not yet available everywhere
    • Michel (MJ): for DPM sites, version required release a few months ago and being deployed
  • Service class selection at T1: mainly a problem with dCache and RAL/CASTOR (single instance for tape and disk)
    • In dCache the service class is selected by SrmPrepareToPut
    • No trivial solution: 2 gridftp and xrootd instances (changing namespace too difficult, huge file rename campaign). May impact performance if reducing the number of disk servers in the pool.

LHCb started an http/dav dynamic federation

  • Could be used in longer term as transfer and access protocol

Gerd: one potential weakness when using xrootd for writer over WAN: channel is not encrypted and allow potentially a malicious use to insert commands to delete files...

  • Conversely to gridftp and http where the control channel is encrypted

LHCb do see SRM load from other VOs or in getting turls on large scale startup - and so potential benefit from moving from SRM.

ATLAS - A. di Girolamo

SRM used for

  • Storage management: WebDav evaluated, performance being assessed
    • Ready to use adhoc tools for space management
  • Tape management: no alternative foreseen
  • 3d party copy with FTS: considering moving to gridftp only as soon as gridftp redirection is widely available
    • xrootd and WebDav is going to be possible but currently no experience: need to understand tuning
  • file upload from WNs: xrootd used with EOS but mostly SRM as many sites require the use space tokens for accounting (not required by ATLAS as there is a namespace separation between space tokens)
    • Could move to xrootd, gridftp or DAV when the accounting issue is fixed
    • also need checksum - Oliver noted gfal-cp can calculate/store checksum with Dav (see below gfal talk)
  • Download to WN: ATLAS has to manage the zoo!
    • Every site has its own requirement!
    • ATLAS would welcome focusing on a limited number of protocols
  • Direct I/O: currently using xrootd/dcap/file
    • Would like to drop dcap (focus on xroot + file)
    • WebDav could be added later when DAVIX matured

Actions proposed

  • Short term (6 months)
    • Eliminate need for space tokens
    • Move to gridftp only for 3d party transfer
    • Move to xrootd/http for uploads and downloads
    • Move to xrootd (or file) for direct IO
    • Decommission other direct IO protocols
  • Medium term (2 years)
    • Commission WebDav for production quality deletion
    • Decommission SRM at non tape endpoints
    • Commission xroot and WebDav for 3d party transfers
    • Decommission gridftp
  • Long term (later)
    • Commission DAVIX and xroot for direct IO

Massimo: If only BringOnline used for tape then there could be an alternative for prestaging, and they would be interested.

A Site Perspective - S. De Witt

Recently, almost all sites using xrootd (in addition to SRM+GridFTP) and started to decommission legacy protocols

  • But makes admin life harder! In particular logs for new protocols are difficult to match with SE logs
  • Decommissionning legacy protocols require an action from the top

Change in protocol happens very slowly

  • New protocol documentation is critical to help assess HW requirements and do troubleshooting
    • xrootd not perfect in this respect: due to the plugin architecture, not everything consistent

Many sites support or want to support non HEP communities: xrootd not easy to sell, http/DAV seen as a positive direction but also not widely adopted outside HEP. gridFTP is used by non-HEP

  • RAL looking to deploy a more modern disk-only storage system, based around object storage technology
    • S3 or CDMI?
    • Want a file uploaded by one protocol to be visible with all other supported protocols

Jeff - Security considerations - things that you do or do not turn on - e.g. expose to outside.

StoRM - A. Ceccanti

New StoRM WebDav service solving design issues/flaws in the previous version, in particular the tight coupling with SRM

  • Also improved performance and authz flexibility: in particular performance of large multi-range requests
  • Beta currently deployed @CNAF and @QMUL

Future: main focus is the http/WebDav service, also:

  • Space reporting without SRM: discussion with DPM and dCache on RFC4431 details
  • Tape space utilization reporting
  • Quota support on non-GPFS FS
  • Interest in developing a RESTful alternative for SrmBringOnLine?
    • To allow full http/WebDav interface to the storage systems

  • Wahid asked how non-GPFS calculated space used - answer was a rolling 'du'

Ceph protocols - D. Van der Ster

Ceph based on an object store, RADOS

  • access through librados
  • Also providing a bock storage interface, RBD, providing a service similar to iSCSI
    • KVM client stable, production-quality native kernel client coming with 7.1
  • RADOS gateway provides a RESTful S3 and SWIFT API but this is a gateway...
  • POSIX with CephFS: still in development, ready for heavy testing but a production-ready version expected later this year
    • Some interesting features like multi-MDS but they are still pretty experimental

Non overlapping protocols: data entered with one protocol cannot be read with another one

Security with Cephx, a shared secret scheme, but security enforced on the client side in RBD and CephFS

  • Means that users could manipulate data at RADOS level

Xrootd - L. Janyst

Recently added: cross-protocol redirects

Coming in 4.2

  • CEPH/RADOS plugin (S. Ponce)
  • Throttling (B. Bockelman)
  • Metalink (Lukasz)

Lukasz will leave CERN in April: Elvin Sindrilaru is taking over his xroot development activity.

DPM - O. Keeble

DPM 1.8.9: performance leap, logging, monitoring

Protocols: DPM proactively supporting http/DAV, xrootd, gridftp

  • These 3 protocols support the Xrootd monitoring scheme
  • Other protocols no longer developed and considered legacy: SRM, rfio, dpns-, dpm-
    • Cannot be removed currently but no need to expose them the users if they don't require them

DPM view on SRM-free operation

  • Storage tokens as namespace/directory
  • Report space stats on first and second level directory
    • Report using RC2518 conventions
    • Agreeement with dCache and StoRM
    • Current DPM allows to query space in BDII and upload to a ST with a DPM-specific solution
  • Support gridftp redirection: present but disabled by default in 1.8.9, enabled by default in next version

Interested by a rollout/validation effort to validate the http support on the infrastructure: WLCG task force?

dCache - P. Millar

dCache favors standard protocols: SRM is considered one of them

SRM has some unique features: transfer protocol negocation, asynchronous operations, bulk operations, transactions/2 stage-commit upload

  • Also battle tested with over 10 years of operation: may need to redo this work for any other protocol
  • dCache view: if needing some SRM unique features, stay with SRM rather than inventing a new protocol (or extending an existing one)

Non-unique SRM features with alternatives: dCache view

  • Accounting without ST: will be implemneted through namespace, same as DPM
  • Direct IO: dcap decommissionning as soon as NFS is robust enough
  • 3d party transfers: gridftp today, look at WebDav for the future
  • Namespace operations: WebDav or NFS for small operations, stay with SRM for bulk operations

xrootd support: new community coming in 2.12 (March) and next golden release 2.13 (July)

  • No support yet for 3d party copy

Authentication: looking at adding SAML support

  • In a first stage, will involve gateway generating X509 cert on the fly

dCache now has a service, prometheus, that tests last version of dCache development release nightly to help discover problems before going to productions: open to WLCG

  • See how to connect to MW readiness infrastructure

EOS and CASTOR - A.J. Peters

EOS native protocol is Xrootd

  • http/https/WebDav/S3 currently provided by embedded htttp server and overlay network (+ Nginx for https)
  • SRM decommissionning does not require any development

CASTOR: xroot the main protocol for users and internal transfers

  • rfio to be deprecated/decommissionned this year

FTS / DAVIX / gfal2 - A.A. Ayllon

DAVIX 0.4.0: 3d party copy improvements, new S3 features, improved writing support


  • Experimental features: S3 and Dropbox support, deletion operations
    • Deletions remain more efficient through SRM due to bulk deletion support: significant difference with large number of deletions (1000+). May need to implement bulk deletion through http/DAV.
  • Future features: gridftp bulk transfers

gfal2 - runs a on the fly checksum if protocol doesn't support checksum. Wahid asked if this would work where protocol does support but backend doesn't (e.g. xrootd on DPM ) - answer: No. it would give an error.


Discussion on action list

  • Markus expressed that action list is only required to address a crisis if there is one but it doesn't seem to be the case
  • Others expressed that action list is required to help reduce services operated at sites and complexity of SW stacks. This is part of the effort to reduce our operational costs.
    • Jeff: as a site without tape would like to shut down SRM due to the complex interaction with other DPM components

ATLAS proposed action list could be used as a starting point, concentrating on the short term actions first

  • Need more discussions to refine it
  • General agreement between experiments that RFIO and dcap are no longer needed and can be closed by sites
  • One difficult point remain how to handle source/destination protocol compatibility in 3d transfer copy without transfer protocol negociaton provided by SRM

In the GDB discussion next day, it was agreed to make and keep up to date tables of WLCGDataAccessProtocolUse and guidance for sites wishing to migrate.

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2015-01-21 - WahidBhimji
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback