HttpTpc
Introduction
The
WebDAV
extensions to HTTP provide a mechanism for invoking a copy between two resources. We utilize implementations of the HTTP COPY method to perform third-party-copies between endpoints.
For more information on the approach, see:
See the
HTTP TPC technical details page for protocol-level information.
Proposing changes or extensions of HTTP-TPC protocol.
Development Status
dCache
HTTP TPC is available with all supported versions of dCache. For details, including baseline versions, see
dCache configuration.
DPM
DPM supports the HTTP third party copy for several years using the X509 delegation features provided by
libgridsite
. Since the version 0.19 of its
lcgdm-dav
frontend, it supports macaroons as an experimental feature.
Enabling macaroons in DPM (Aug 2018) can be done
either using puppet or manually editing the Apache config file:
- puppet command line setup: in the manifest set the parameter "dmlite::dav::params::ns_macaroon_secret" to your preferred secret string, longer than 64 characters
- manual Apache tinkering:
- Edit the file
/etc/httpd/conf.d/zlcgdm-dav.conf
and add the line NSMacaroonSecret <your_secret_string_longer_then_64_chars>
to the already existing section <LocationMatch "^/dpm/.*">
- Make sure that the ssl section says
SSLVerifyClient optional
Xrootd
The standalone
Xrootd
server has a native HTTP/WebDAV plugin which is gaining support for third-party-copy.
The best reference for installing and configuring it is
found here. That page additionally covers the authorization for the Xrootd server; concrete examples are from a CMS-specific point-of-view. Contributions of examples from other VOs are welcome!
Versions necessary:
EOS
EOS supports :
- Third-party-copy transfers with X509 delegated credentials over the XRootD protocol since 4.7.0.
- Third-party-copy transfers over HTTP(S) with tokens since version 4.7.1.
Note: EOS needs to be compiled with
XRootD >= 4.11.3 for all features to work correctly.
For more information on how to configure an EOS instance to support
XRootD and HTTP TPC with delegated credentials or with Bearer tokens see:
StoRM
StoRM WebDAV provides HTTP Third Party Copy support starting with version 1.1.0.
StoRM WebDAV only supports delegation via bearer tokens (GSI delegation is not supported).
Compatibility Testing
For compatibility testing, we recommend:
- Xrootd:
-
Nebraska
: Nebraska has setup hosts with third-party-copy enabled. Try http://red-gridftp12.unl.edu:1094//store/test
with a CMS production proxy. Nebraska is working to additionally support the dteam VO on this endpoint.
-
Florida
: CMS users can create their own subdirectory of this resource: davs://cmsio5.rc.ufl.edu:1094//store/temp/user
- dCache:
- EOS:
-
CERN
: endpoint is http://eospps.cern.ch:9000//eos/opstest/tpc/
where the specific test directory /eos/opstest/tpc/
can be accessed by dteam, ATLAS, CMS and OPS
VO members
- DAVIX (http://
) simple example:
-
:~$ davix-put -E /tmp/x509up_uxxx /etc/group https://eospps.cern.ch:9000/eos/opstest/tpc/test-001.file
- GFAL (davs://) simple example:
- TPC GFAL (
EOSPUBLIC->EOSPPS
) :
-
:~$ gfal-copy davs://eospublichttp.cern.ch//eos/opstest/adlercheck.test davs://eospps.cern.ch:9000//eos/opstest/tpc/gfal-davs-TPC-test.file
-
Copying davs://eospublichttp.cern.ch//eos/opstest/adlercheck.test [DONE] after 0s
- TPC GFAL (
DESY->CERN
)
-
:~$ gfal-copy davs://prometheus.desy.de/VOs/cms/bigfile.1 davs://eospps.cern.ch:9000/eos/opstest/tpc/fts-desy-cern.bigfile.1
-
Copying davs://prometheus.desy.de/VOs/cms/bigfile.1 [DONE] after 411s
Utilize the
fts3-devel.cern.ch
host for initiating transfers. Brian has been testing FTS/GFAL2 patches on
hcc-briantest7.unl.edu
.
For example, to initiate a third-party-transfer from Nebraska (Xrootd) to DESY (prometheus):
fts-transfer-submit -s https://fts3-devel.cern.ch:8446 https://red-gridftp12.unl.edu:1094/dropfiles/hdfs-test/bigfile.1 https://prometheus.desy.de/VOs/cms/bigfile.1
Test Results
The following information is out-of-date. Live information is available from the Kibana
transfers overview
dashboard.
- From
Nebraska
to:
-
DESY
: working
-
CERN/DPM
: Working with https://gitlab.cern.ch/dmc/gfal2/merge_requests/13
applied to FTS server.
-
CERN/EOS/PPS
: Broken. CERN/EOS/PPS
does not support TPC and source site does not support X509 delegation.
-
Imperial
: working
-
Florida
: working
- From
Imperial
to:
-
DESY
: working.
-
CERN/DPM
: working.
-
CERN/EOS/PPS
: Broken. (Unclear if dCache is provided with correct credential. Need help?)
-
Nebraska
: working.
- From
CERN/DPM
to:
-
Nebraska
: working
-
DESY
: Broken - CA issues.
-
Glasgow
: working
-
Imperial
: working
-
CERN/EOS/PPS
: Working
-
UKI-LT2-QMUL
: Working
- From
UKI-LT2-QMUL
to:
-
CERN/DPM
: Working
-
CERN/EOS/PPS
: Broken. Both endponts do not support TPC
-
DESY
: Broken. Possible gfal2 bug, to investigate.
Tickets / Issues
A list of discovered problems.
Open issues
- DPM:
- Do not require GridSite delegation if there's a valid bearer token for remote site. LCGDM-2650
. Workaround implemented in gfal2 for this issue.
- Graceful restarts required to reload the CRLs: this is (apparently) causing sporadic httpd hangs. Thought to be caused (or made more frequent) by concurrent reload requests coming from different sources (e.g. cron jobs) LCGDM-2699
LCGDM-2707
.
- Fetching checksums (RFC-3230 Want-Digest HEAD request) requires LIST activity in macaroon. This activity is not needed in other implementations.
Closed issues
A problem is considered closed when there is a released version that no longer shows the problematic behavior.
- FTS:
- Improve the macaroon validity calculation ( currently set to 3 hours) FTS-1306
, FIXED in FTS 3.8.2
- Add LIST caveat to both source and destination macaroon requests FTS-1309
, FIXED in FTS 3.8.2
- Check the validity of the VOMS AC as well to request for a new proxy from the DB FTS-1311
, FIXED in FTS 3.8.2
- DAVIX:
- dCache:
- Macaroon does not work with COPY unless its path points to parent directory. dCache RT#9468 - FIXED. FTS patch available
. Problem is resolved in 4.2.11, 4.1.19, 4.0.25, and 3.2.33.
- Does not recognize "davs" as a valid scheme. dCache RT#9473 -- CLOSED. Fix was in GFAL, not dCache.
- Misinterprets Macaroon as an OpenID delegation attempt. dCache RT #9474 - FIXED. Problem is resolved in 4.2.6, 4.1.15, 4.0.21, and 3.2.29.
- Not possible to use GridSite delegated X.509 credential for TPC when also supplying a macaroon. dCache RT #9492 - FIXED. Problem is resolved in 4.2.11, 4.1.19, 4.0.25, and 3.2.33.
- Ability to create parent directories with UPLOAD permission. dCache RT #9503 - FIXED. Problem is resolved in 4.2.14, 4.1.22, 4.0.28, and 3.2.36.
- StoRM:
- StoRM responds to a POST with the corresponding GET response instead of an error. STOR-1035
.
- StoRM WebDAV should return 507 for disk quota exceeded STOR-1066
Instructions for CMS-Tier 3s Admins:
Follow this link for
CMS WebDAV protocol: Installation and Testing instructions.