TWiki> LCG Web>DomaActivities>ThirdPartyCopy>HttpTpc (revision 45)EditAttachPDF

HttpTpc

Introduction

The WebDAV extensions to HTTP provide a mechanism for invoking a copy between two resources. We utilize implementations of the HTTP COPY method to perform third-party-copies between endpoints.

For more information on the approach, see:

See the HTTP TPC technical details page for protocol-level information.

Development Status

dCache

HTTP TPC is available with all supported versions of dCache. For details, including baseline versions, see dCache configuration.

DPM

DPM supports the HTTP third party copy for several years using the X509 delegation features provided by libgridsite. Since the version 0.19 of its lcgdm-dav frontend, it supports macaroons as an experimental feature.

Enabling macaroons in DPM (Aug 2018) can be done either using puppet or manually editing the Apache config file:

  • puppet command line setup: in the manifest set the parameter "dmlite::dav::params::ns_macaroon_secret" to your preferred secret string, longer than 64 characters
  • manual Apache tinkering:
    • Edit the file /etc/httpd/conf.d/zlcgdm-dav.conf and add the line NSMacaroonSecret <your_secret_string_longer_then_64_chars> to the already existing section <LocationMatch "^/dpm/.*">
    • Make sure that the ssl section says SSLVerifyClient optional

Xrootd

The standalone Xrootd server has a native HTTP/WebDAV plugin which is gaining support for third-party-copy.

The best reference for installing and configuring it is found here. That page additionally covers the authorization for the Xrootd server; concrete examples are from a CMS-specific point-of-view. Contributions of examples from other VOs are welcome!

Versions necessary:

Significant work is slated for Xrootd 4.9.0:

  • The third-party-copy and macaroons plugin are integrated into the Xrootd release; a separate RPM will not be necessary.
  • Support for HTTP-based checksums.
  • A variety of bugfixes for HTTP support as well as scalability and improved HTTP 1.0 compliance.

StoRM

StoRM WebDAV will provide HTTP Third Party Copy support in version 1.1.0 (currently in pre-release).

StoRM WebDAV only supports delegation via bearer tokens (GSI delegation is not supported).

Compatibility Testing

For compatibility testing, we recommend:

  • Xrootd:
    • Nebraska: Nebraska has setup hosts with third-party-copy enabled. Try http://red-gridftp12.unl.edu:1094//store/test with a CMS production proxy. Nebraska is working to additionally support the dteam VO on this endpoint.
    • Florida: CMS users can create their own subdirectory of this resource: davs://cmsio3.rc.ufl.edu:1094//store/temp/user
  • dCache:
  • EOS:
    • CERN: endpoint is http://eosppshttp01.cern.ch//eos/opstest/tpc/ where the specific test directory /eos/opstest/tpc/ can be accessed by dteam, ATLAS, CMS and OPS VO members
      • DAVIX (http://) simple example:
        • xavierc3@lxplus061:~$ davix-put -E /tmp/x509up_uxxx /etc/group  https://eosppshttp01.cern.ch/eos/opstest/tpc/testxavi-02082018-001.file
      • GFAL (davs://) simple example:
        • xavierc3@lxplus061:~$ gfal-copy -f file:///etc/group davs://eosppshttp01.cern.ch//eos/opstest/tpc/gfal-davs-testxavi.file
        • Copying file:///etc/group   [DONE]  after 0s
      • TPC GFAL (EOSPUBLIC->EOSPPS) :
        • xavierc3@lxplus061:~$ gfal-copy davs://eospublichttp.cern.ch//eos/opstest/xavierc3/adlercheck.test davs://eosppshttp01.cern.ch//eos/opstest/tpc/gfal-davs-TPC-testxavi.file
        • Copying davs://eospublichttp.cern.ch//eos/opstest/xavierc3/adlercheck.test   [DONE]  after 0s
      • TPC GFAL (DESY->CERN)
        • xavierc3@lxplus094:~$ gfal-copy davs://prometheus.desy.de/VOs/cms/bigfile.1 davs://eosppshttp01.cern.ch/eos/opstest/tpc/fts-xavi-desy-cern.bigfile.1
        • Copying davs://prometheus.desy.de/VOs/cms/bigfile.1   [DONE]  after 411s

Utilize the fts3-devel.cern.ch host for initiating transfers. Brian has been testing FTS/GFAL2 patches on hcc-briantest7.unl.edu.

For example, to initiate a third-party-transfer from Nebraska (Xrootd) to DESY (prometheus):

fts-transfer-submit -s https://fts3-devel.cern.ch:8446 https://red-gridftp12.unl.edu:1094/dropfiles/hdfs-test/bigfile.1 https://prometheus.desy.de/VOs/cms/bigfile.1

Test Results

The following information is out-of-date. Live information is available from the Kibana transfers overview dashboard.

  • From Nebraska to:
    • DESY: working
    • CERN/DPM: Working with https://gitlab.cern.ch/dmc/gfal2/merge_requests/13 applied to FTS server.
    • CERN/EOS/PPS: Broken. CERN/EOS/PPS does not support TPC and source site does not support X509 delegation.
    • Imperial: working
    • Florida: working
  • From Imperial to:
    • DESY: working.
    • CERN/DPM: working.
    • CERN/EOS/PPS: Broken. (Unclear if dCache is provided with correct credential. Need help?)
    • Nebraska: working.
  • From CERN/DPM to:
    • Nebraska: working
    • DESY: Broken - CA issues.
    • Glasgow: working
    • Imperial: working
    • CERN/EOS/PPS: Working
    • UKI-LT2-QMUL : Working
  • From CERN/EOS/PPS to:
    • Nebraska: Broken. Same issue as reverse direction.
    • Imperial: Broken. Same issue as with reverse direction.
    • CERN/DPM: Working.
  • From UKI-LT2-QMUL to:
    • CERN/DPM: Working
    • CERN/EOS/PPS: Broken. Both endponts do not support TPC
    • DESY: Broken. Possible gfal2 bug, to investigate.

Tickets / Issues

A list of discovered problems.

Open issues

  • DPM:
    • Do not require GridSite delegation if there's a valid bearer token for remote site. LCGDM-2650. Workaround implemented in gfal2 for this issue.
    • Graceful restarts required to reload the CRLs: this is (apparently) causing sporadic httpd hangs. Thought to be caused (or made more frequent) by concurrent reload requests coming from different sources (e.g. cron jobs) LCGDM-2699 LCGDM-2707.

  • FTS:
    • Improve the macaroon validity calculation ( currently set to 3 hours) FTS-1306, FIXED in FTS 3.8.2 not yet tagged
    • Add LIST caveat to both source and destination macaroon requests FTS-1309, FIXED in FTS 3.8.2 not yet tagged
    • Check the validity of the VOMS AC as well to request for a new proxy from the DB FTS-1311, FIXED in FTS 3.8.2 not yet tagged

Closed issues

A problem is considered closed when there is a released version that no longer shows the problematic behavior.

  • DAVIX:
  • dCache:
    • Macaroon does not work with COPY unless its path points to parent directory. dCache RT#9468 - FIXED. FTS patch available. Problem is resolved in 4.2.11, 4.1.19, 4.0.25, and 3.2.33.
    • Does not recognize "davs" as a valid scheme. dCache RT#9473 -- CLOSED. Fix was in GFAL, not dCache.
    • Misinterprets Macaroon as an OpenID delegation attempt. dCache RT #9474 - FIXED. Problem is resolved in 4.2.6, 4.1.15, 4.0.21, and 3.2.29.
    • Not possible to use GridSite delegated X.509 credential for TPC when also supplying a macaroon. dCache RT #9492 - FIXED. Problem is resolved in 4.2.11, 4.1.19, 4.0.25, and 3.2.33.
    • Ability to create parent directories with UPLOAD permission. dCache RT #9503 - FIXED. Problem is resolved in 4.2.14, 4.1.22, 4.0.28, and 3.2.36.

  • StoRM:
    • StoRM responds to a POST with the corresponding GET response instead of an error. STOR-1035.
    • StoRM WebDAV should return 507 for disk quota exceeded STOR-1066
Edit | Attach | Watch | Print version | History: r49 | r47 < r46 < r45 < r44 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r45 - 2018-12-21 - PaulMillar1
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback