HttpTpc

Introduction

The WebDAV extensions to HTTP provide a mechanism for invoking a copy between two resources. We utilize implementations of the HTTP COPY method to perform third-party-copies between endpoints.

For more information on the approach, see:

• CHEP 2018 Overview Talk.
• Web enabled data management with DPM & LFC.

See the HTTP TPC technical details page for protocol-level information.

Development Status

dCache

HTTP TPC is available with all supported versions of dCache. For details, including baseline versions, see dCache configuration.

DPM

DPM supports the HTTP third party copy for several years using the X509 delegation features provided by libgridsite. Since the version 0.19 of its lcgdm-dav frontend, it supports macaroons as an experimental feature.

Enabling macaroons in DPM (Aug 2018) can be done either using puppet or manually editing the Apache config file:

• puppet command line setup: in the manifest set the parameter "dmlite::dav::params::ns_macaroon_secret" to your preferred secret string, longer than 64 characters
• manual Apache tinkering:
  • Edit the file /etc/httpd/conf.d/zlcgdm-dav.conf and add the line NSMacaroonSecret <your_secret_string_longer_then_64_chars> to the already existing section <LocationMatch "^/dpm/.*">
  • Make sure that the ssl section says SSLVerifyClient optional

Xrootd

The standalone Xrootd server has a native HTTP/WebDAV plugin which is gaining support for third-party-copy.

The best reference for installing and configuring it is found here. That page additionally covers the authorization for the Xrootd server; concrete examples are from a CMS-specific point-of-view. Contributions of examples from other VOs are welcome!

Versions necessary:

• Xrootd 4.9.1 (available from the OSG release repository).

EOS

EOS supports :

• Third-party-copy transfers with X509 delegated credentials over the XRootD protocol since 4.7.0.
• Third-party-copy transfers over HTTP(S) with tokens since version 4.7.1.

Note: EOS needs to be compiled with XRootD >= 4.11.3 for all features to work correctly.

For more information on how to configure an EOS instance to support XRootD and HTTP TPC with delegated credentials or with Bearer tokens see:

• http://eos-docs.web.cern.ch/eos-docs/configuration/http_tpc.html

StoRM

StoRM WebDAV provides HTTP Third Party Copy support starting with version 1.1.0.

StoRM WebDAV only supports delegation via bearer tokens (GSI delegation is not supported).

Compatibility Testing

For compatibility testing, we recommend:

• Xrootd:
  • Nebraska: Nebraska has setup hosts with third-party-copy enabled. Try http://red-gridftp12.unl.edu:1094//store/test with a CMS production proxy. Nebraska is working to additionally support the dteam VO on this endpoint.
  • Florida: CMS users can create their own subdirectory of this resource: davs://cmsio3.rc.ufl.edu:1094//store/temp/user
• dCache:
  • DESY: Utilize host prometheus.desy.de and directory /VOs/cms. Note that prometheus is part of the dCache continuous integration setup and is rebuilt on a nightly basis.
  • Imperial: Imperial College's dCache has setup the following directory for CMS testers: https://gfe02.grid.hep.ph.ic.ac.uk:2880/pnfs/hep.ph.ic.ac.uk/data/cms/store/test/davs
• EOS:
  • CERN: endpoint is http://eospps.cern.ch:9000//eos/opstest/tpc/ where the specific test directory /eos/opstest/tpc/ can be accessed by dteam, ATLAS, CMS and OPS VO members
    • DAVIX (http://) simple example:
      • :~$ davix-put -E /tmp/x509up_uxxx /etc/group  https://eospps.cern.ch:9000/eos/opstest/tpc/test-001.file
    • GFAL (davs://) simple example:
      • :~$ gfal-copy -f file:///etc/group davs://eospp.cern.ch:9000//eos/opstest/tpc/gfal-davs-test.file
      • Copying file:///etc/group   [DONE]  after 0s
    • TPC GFAL (EOSPUBLIC->EOSPPS) :
      • :~$ gfal-copy davs://eospublichttp.cern.ch//eos/opstest/adlercheck.test davs://eospps.cern.ch:9000//eos/opstest/tpc/gfal-davs-TPC-test.file
      • Copying davs://eospublichttp.cern.ch//eos/opstest/adlercheck.test   [DONE]  after 0s
    • TPC GFAL (DESY->CERN)
      • :~$ gfal-copy davs://prometheus.desy.de/VOs/cms/bigfile.1 davs://eospps.cern.ch:9000/eos/opstest/tpc/fts-desy-cern.bigfile.1
      • Copying davs://prometheus.desy.de/VOs/cms/bigfile.1   [DONE]  after 411s

• DPM:
  • CERN/DPM: The DPM development team at CERN has setup the following directories for testing: https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/dteam/ (dteam VO) and https://dpmhead-trunk.cern.ch/dpm/cern.ch/home/cms/ (cms VO). UPDATE Given some test being done on dpmhead-trunk.cern.ch, better to use the Instance with paths https://dpmhead-rc.cern.ch/dpm/cern.ch/home/dteam/ and https://dpmhead-rc.cern.ch/dpm/cern.ch/home/cms
  • Glasgow: Endpoint information not yet available

• STORM:
  • INFN-CNAF-T1: endpoint is https://xfer.cr.cnaf.infn.it:8443/dteam
  • UKI-LT2-QMUL: endpoint is https://se01.esc.qmul.ac.uk:8443/dteam

Utilize the fts3-devel.cern.ch host for initiating transfers. Brian has been testing FTS/GFAL2 patches on hcc-briantest7.unl.edu.

For example, to initiate a third-party-transfer from Nebraska (Xrootd) to DESY (prometheus):

fts-transfer-submit -s https://fts3-devel.cern.ch:8446 https://red-gridftp12.unl.edu:1094/dropfiles/hdfs-test/bigfile.1 https://prometheus.desy.de/VOs/cms/bigfile.1

Test Results

The following information is out-of-date. Live information is available from the Kibana transfers overview dashboard.

• From Nebraska to:
  • DESY: working
  • CERN/DPM: Working with https://gitlab.cern.ch/dmc/gfal2/merge_requests/13 applied to FTS server.
  • CERN/EOS/PPS: Broken. CERN/EOS/PPS does not support TPC and source site does not support X509 delegation.
  • Imperial: working
  • Florida: working
• From Imperial to:
  • DESY: working.
  • CERN/DPM: working.
  • CERN/EOS/PPS: Broken. (Unclear if dCache is provided with correct credential. Need help?)
  • Nebraska: working.
• From CERN/DPM to:
  • Nebraska: working
  • DESY: Broken - CA issues.
  • Glasgow: working
  • Imperial: working
  • CERN/EOS/PPS: Working
  • UKI-LT2-QMUL : Working
• From UKI-LT2-QMUL to:
  • CERN/DPM: Working
  • CERN/EOS/PPS: Broken. Both endponts do not support TPC
  • DESY: Broken. Possible gfal2 bug, to investigate.

Tickets / Issues

A list of discovered problems.

Open issues

• DPM:
  • Do not require GridSite delegation if there's a valid bearer token for remote site. LCGDM-2650. Workaround implemented in gfal2 for this issue.
  • Graceful restarts required to reload the CRLs: this is (apparently) causing sporadic httpd hangs. Thought to be caused (or made more frequent) by concurrent reload requests coming from different sources (e.g. cron jobs) LCGDM-2699 LCGDM-2707.
  • Fetching checksums (RFC-3230 Want-Digest HEAD request) requires LIST activity in macaroon. This activity is not needed in other implementations.

Closed issues

A problem is considered closed when there is a released version that no longer shows the problematic behavior.

• FTS:
  • Improve the macaroon validity calculation ( currently set to 3 hours) FTS-1306, FIXED in FTS 3.8.2
  • Add LIST caveat to both source and destination macaroon requests FTS-1309, FIXED in FTS 3.8.2
  • Check the validity of the VOMS AC as well to request for a new proxy from the DB FTS-1311, FIXED in FTS 3.8.2

• gfal2:
  • Fallback to GridSite delegation if requested by storage: https://its.cern.ch/jira/browse/DMC-1069. Patch available - FIXED. Problem is resolved in 2.16.0
  • Remove file after transfer attempt: https://its.cern.ch/jira/browse/DMC-1068. Patch available - FIXED. Problem is resolved in 2.16.0
  • HTTP TPC failures are not logged at INFO level DMC-1109, FIXED. Problem is resolved in 2.16.1
  • RequireChecksumVerification set to false only for HTTP TPC PULL DMC-1110, FIXED. Problem is resolved in gfal2 2.16.1.

• DAVIX:
  • Incorrect handling of checksums in response: https://its.cern.ch/jira/browse/DMC-1063 - FIXED. Problem is resolved in 0.6.9
  • Sends "davs" as scheme. davix update - FIXED. Problem is resolved in 0.6.9
• dCache:
  • Macaroon does not work with COPY unless its path points to parent directory. dCache RT#9468 - FIXED. FTS patch available. Problem is resolved in 4.2.11, 4.1.19, 4.0.25, and 3.2.33.
  • Does not recognize "davs" as a valid scheme. dCache RT#9473 -- CLOSED. Fix was in GFAL, not dCache.
  • Misinterprets Macaroon as an OpenID delegation attempt. dCache RT #9474 - FIXED. Problem is resolved in 4.2.6, 4.1.15, 4.0.21, and 3.2.29.
  • Not possible to use GridSite delegated X.509 credential for TPC when also supplying a macaroon. dCache RT #9492 - FIXED. Problem is resolved in 4.2.11, 4.1.19, 4.0.25, and 3.2.33.
  • Ability to create parent directories with UPLOAD permission. dCache RT #9503 - FIXED. Problem is resolved in 4.2.14, 4.1.22, 4.0.28, and 3.2.36.

• StoRM:
  • StoRM responds to a POST with the corresponding GET response instead of an error. STOR-1035.
  • StoRM WebDAV should return 507 for disk quota exceeded STOR-1066