HttpTpc

Introduction

The WebDAV extensions to HTTP provide a mechanism for invoking a copy between two resources. We utilize implementations of the HTTP COPY method to perform third-party-copies between endpoints.

For more information on the approach, see:

See the HTTP TPC technical details page for protocol-level information.

Development Status

dCache

HTTP TPC is available with all supported versions of dCache. For details, including baseline versions, see dCache configuration.

DPM

DPM supports the HTTP third party copy for several years using the X509 delegation features provided by libgridsite. Since the version 0.19 of its lcgdm-dav frontend, it supports macaroons as an experimental feature.

Enabling macaroons in DPM (Aug 2018) can be done either using puppet or manually editing the Apache config file:

  • puppet command line setup: in the manifest set the parameter "dmlite::dav::params::ns_macaroon_secret" to your preferred secret string, longer than 64 characters
  • manual Apache tinkering:
    • Edit the file /etc/httpd/conf.d/zlcgdm-dav.conf and add the line NSMacaroonSecret <your_secret_string_longer_then_64_chars> to the already existing section <LocationMatch "^/dpm/.*">
    • Make sure that the ssl section says SSLVerifyClient optional

Xrootd

The standalone Xrootd server has a native HTTP/WebDAV plugin which is gaining support for third-party-copy.

The best reference for installing and configuring it is found here. That page additionally covers the authorization for the Xrootd server; concrete examples are from a CMS-specific point-of-view. Contributions of examples from other VOs are welcome!

Versions necessary:

EOS

EOS supports :

  • Third-party-copy transfers with X509 delegated credentials over the XRootD protocol since 4.7.0.
  • Third-party-copy transfers over HTTP(S) with tokens since version 4.7.1.

Note: EOS needs to be compiled with XRootD >= 4.11.3 for all features to work correctly.

For more information on how to configure an EOS instance to support XRootD and HTTP TPC with delegated credentials or with Bearer tokens see:

StoRM

StoRM WebDAV provides HTTP Third Party Copy support starting with version 1.1.0.

StoRM WebDAV only supports delegation via bearer tokens (GSI delegation is not supported).

Compatibility Testing

For compatibility testing, we recommend:

  • Xrootd:
    • Nebraska: Nebraska has setup hosts with third-party-copy enabled. Try http://red-gridftp12.unl.edu:1094//store/test with a CMS production proxy. Nebraska is working to additionally support the dteam VO on this endpoint.
    • Florida: CMS users can create their own subdirectory of this resource: davs://cmsio3.rc.ufl.edu:1094//store/temp/user
  • dCache:
  • EOS:
    • CERN: endpoint is http://eospps.cern.ch:9000//eos/opstest/tpc/ where the specific test directory /eos/opstest/tpc/ can be accessed by dteam, ATLAS, CMS and OPS VO members
      • DAVIX (http://) simple example:
        • :~$ davix-put -E /tmp/x509up_uxxx /etc/group  https://eospps.cern.ch:9000/eos/opstest/tpc/test-001.file
      • GFAL (davs://) simple example:
      • TPC GFAL (EOSPUBLIC->EOSPPS) :
        • :~$ gfal-copy davs://eospublichttp.cern.ch//eos/opstest/adlercheck.test davs://eospps.cern.ch:9000//eos/opstest/tpc/gfal-davs-TPC-test.file
        • Copying davs://eospublichttp.cern.ch//eos/opstest/adlercheck.test   [DONE]  after 0s
      • TPC GFAL (DESY->CERN)
        • :~$ gfal-copy davs://prometheus.desy.de/VOs/cms/bigfile.1 davs://eospps.cern.ch:9000/eos/opstest/tpc/fts-desy-cern.bigfile.1
        • Copying davs://prometheus.desy.de/VOs/cms/bigfile.1   [DONE]  after 411s

Utilize the fts3-devel.cern.ch host for initiating transfers. Brian has been testing FTS/GFAL2 patches on hcc-briantest7.unl.edu.

For example, to initiate a third-party-transfer from Nebraska (Xrootd) to DESY (prometheus):

fts-transfer-submit -s https://fts3-devel.cern.ch:8446 https://red-gridftp12.unl.edu:1094/dropfiles/hdfs-test/bigfile.1 https://prometheus.desy.de/VOs/cms/bigfile.1

Test Results

The following information is out-of-date. Live information is available from the Kibana transfers overview dashboard.

  • From Nebraska to:
    • DESY: working
    • CERN/DPM: Working with https://gitlab.cern.ch/dmc/gfal2/merge_requests/13 applied to FTS server.
    • CERN/EOS/PPS: Broken. CERN/EOS/PPS does not support TPC and source site does not support X509 delegation.
    • Imperial: working
    • Florida: working
  • From Imperial to:
    • DESY: working.
    • CERN/DPM: working.
    • CERN/EOS/PPS: Broken. (Unclear if dCache is provided with correct credential. Need help?)
    • Nebraska: working.
  • From CERN/DPM to:
    • Nebraska: working
    • DESY: Broken - CA issues.
    • Glasgow: working
    • Imperial: working
    • CERN/EOS/PPS: Working
    • UKI-LT2-QMUL : Working
  • From UKI-LT2-QMUL to:
    • CERN/DPM: Working
    • CERN/EOS/PPS: Broken. Both endponts do not support TPC
    • DESY: Broken. Possible gfal2 bug, to investigate.

Tickets / Issues

A list of discovered problems.

Open issues

  • DPM:
    • Do not require GridSite delegation if there's a valid bearer token for remote site. LCGDM-2650. Workaround implemented in gfal2 for this issue.
    • Graceful restarts required to reload the CRLs: this is (apparently) causing sporadic httpd hangs. Thought to be caused (or made more frequent) by concurrent reload requests coming from different sources (e.g. cron jobs) LCGDM-2699 LCGDM-2707.
    • Fetching checksums (RFC-3230 Want-Digest HEAD request) requires LIST activity in macaroon. This activity is not needed in other implementations.

Closed issues

A problem is considered closed when there is a released version that no longer shows the problematic behavior.

  • FTS:
    • Improve the macaroon validity calculation ( currently set to 3 hours) FTS-1306, FIXED in FTS 3.8.2
    • Add LIST caveat to both source and destination macaroon requests FTS-1309, FIXED in FTS 3.8.2
    • Check the validity of the VOMS AC as well to request for a new proxy from the DB FTS-1311, FIXED in FTS 3.8.2

  • DAVIX:
  • dCache:
    • Macaroon does not work with COPY unless its path points to parent directory. dCache RT#9468 - FIXED. FTS patch available. Problem is resolved in 4.2.11, 4.1.19, 4.0.25, and 3.2.33.
    • Does not recognize "davs" as a valid scheme. dCache RT#9473 -- CLOSED. Fix was in GFAL, not dCache.
    • Misinterprets Macaroon as an OpenID delegation attempt. dCache RT #9474 - FIXED. Problem is resolved in 4.2.6, 4.1.15, 4.0.21, and 3.2.29.
    • Not possible to use GridSite delegated X.509 credential for TPC when also supplying a macaroon. dCache RT #9492 - FIXED. Problem is resolved in 4.2.11, 4.1.19, 4.0.25, and 3.2.33.
    • Ability to create parent directories with UPLOAD permission. dCache RT #9503 - FIXED. Problem is resolved in 4.2.14, 4.1.22, 4.0.28, and 3.2.36.

  • StoRM:
    • StoRM responds to a POST with the corresponding GET response instead of an error. STOR-1035.
    • StoRM WebDAV should return 507 for disk quota exceeded STOR-1066
Edit | Attach | Watch | Print version | History: r51 < r50 < r49 < r48 < r47 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r51 - 2020-08-07 - AlessandraForti
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback