Checksums support in FTS

This page describes how FTS handles checksums.

Parameters at Job Submission

As a global option one can enable comparison of source and destination checksums with the --compare-checksums option.

For each file one can specify a third parameter with checksum algorithm:checksum value, which will be used as the basis of comparison with the destination.

glite-transfer-submit ... --compare-checksums source destination ADLER32:ba0bab

See the status of this feature and the similar feature of lcg_utils.

Behavior of transfers

Get destination checksum

How to execute the "get destination checksum" step in the use cases:

• if destination is a SURL:
  • if srmLs returns a value, use it
  • otherwise:
    • issue an srmPrepareToGet on the destination
    • once the TURL is ready, get the checksum using gridftp
• if destination is a gsiftp:
  • get the checksum using gridftp

Use case 0: Client does not give any extra options

glite-tarnsfer-submit source-SURL destination-SURL

No checksum operations/comparison.

Use case 1: Client provides a checksum

glite-transfer-submit source-SURL destination-SURL ADLER32:12345678

Rationale: the client knows the checksum and wants to make sure that this is what arrives to the destination. If the source SE does not support checksums that is not a problem.

1. Get source checksum.
   • Source is a SURL: get source checksum during the preparation phase via srmLs, but do not fall back to gridftp.
   • Source is a gsiftp: do not get checksum.
   • if checksum is the same as the user supplied value: CONTINUE
   • if checksum is not available: CONTINUE
   • if checksum is different: ERROR
2. transfer succeeds
3. If the destination is a SURL: call srmPutDone on the destination
4. get destination checksum
5. compare the checksums
   • if checksum is the same as the user supplied value: OK
   • if checksum is not available: ERROR
   • if checksum is different: ERROR

Use case 2: Client does not provide checksum, but requires comparison

glite-transfer-submit --compare-checksums source-SURL destination-SURL

Rationale: the client does not know the checksum, but wants to make sure that the transfer had no errors.

1. Get source checksum.
   • If the source is a SURL: get source checksum during the preparation phase
     • if srmLs returns a value, use it
     • if not available, get the checksum using gridftp
   • Source is gsiftp: get the checksum using gridftp
2. if checksum is not available: ERROR
3. transfer succeeds
4. If the destination is a SURL: call srmPutDone on the destination
5. get destination checksum
6. compare the checksums
   • if checksum is the same as the one from the source: OK
   • if checksum is not available: ERROR
   • if checksum is different: ERROR

Use case 3: Client provides checksum and requires comparison

glite-transfer-submit --compare-checksums source-SURL destination-SURL ADLER32:12345678

Rationale: the client knows the checksum and wants to make sure that this is what arrives to the destination. The client wants to make sure that the transfer does not even start, if the source SE has a corrupted file, not to waste the transfer bandwidth.

1. Get source checksum.
   • If the source is a SURL: get source checksum during the preparation phase
     • if srmLs returns a value, use it
     • if not available, get the checksum using gridftp
   • Source is gsiftp: get the checksum using gridftp
2. compare the checksums
   • if checksum is the same as the user supplied value: CONTINUE
   • if checksum is not available: ERROR
   • if checksum is different: ERROR
3. transfer succeeds
4. if the destination is a SURL: call srmPutDone on the destination
5. get destination checksum
6. compare the checksums
   • if checksum is the same as the user supplied value: OK
   • if checksum is not available: ERROR
   • if checksum is different: ERROR

Maintainer: AkosFrohner