FIO Requirements for tape data/metadata verification

-----Original Message-----
From: Tim Bell 
Sent: 14 May 2009 17:25
To: German Cancio Melia; Dirk Duellmann
Subject: [Fwd: To repack or not to repack]


As requested, here is the list of requirements for the tape scanning tool. These are the same as previously sent round but I wanted Vlado's confirmation before finalising.

During the recent repack campaign, we have found the metadata in the name server (such as file sizes or checksums) is sometimes incorrect compared to the actual data stored on tape. This RFE describes the requirements for a solution which can cross check the meta data and the contents on tape along with applying some standard rules to correct the meta data if there are issues.

There are several use cases envisaged

- When a tape is marked as full, we can rapidly arrange to scan all or part of the contents to ensure that the meta data in the name server is correct.

- When the new tape gateway and label format is delivered, we can rapidly validate the contents of written tapes

- We arrange that, according to drive availability, we pro-actively scan tape media to validate the data is still readable and meta data correct. For dual copies, this would allow us to pro-actively arrange a second copy of the data.

Mandatory Functionality

- By default, the tape/meta data should check each file on the tape where there is a valid name server entry. An option to select a specific subset of files is also required to allow sampling of data (such as first/last and some in the middle). Disabled segments should be skipped.

- Segments where there is an I/O error should be reported and the scan continue without unmounting. No check should be performed if the file could not be cleanly read from tape.

- The file sequence number, name server file id, file size and checksum should be cross checked with the name server and segment entries in the database.

- The procedure should be able to run in the background from a command line execution. All options should be specified via command line with no interaction required.

- The performance should support scanning the complete contents of a 1TB tape in less than 3 hours.

- When an issue is detected, it should be possible to define a set of automatic actions such as print an information message explaining what is wrong, fix the file size/checksum based on tape pool or whether there is a second copy or rename the file to a different name (such as adding .badchecksum). A noaction option should just print the actions but not actually perform them.

- File data must be taken from the tape being scanned even if there are dual copies available. Optional Functionality

- All meta data in the label and block headers should be validated.

- A new VMGR field for date of last scan. This would be used to schedule the work so that tapes recently marked as full which have not been scanned can be scheduled and

- Disabled tape scanning is not necessary but would be helpful as an operational tool to find bad segments on tape

Open Questions / Points

- How should access to the tape drives be done/scheduled ? Should we schedule the access via VDQM to get the drives or should we dedicate tape servers for this activity ? If we dedicate, we cannot be opportunistic. If we do not dedicate, we are at risk of over scheduling scanning and impacting users.

- How should multi-segment files be handled ?

- How should files recently modified in the name server be handled ? Can we identify cases of a migrated file where the name server has been updated afterwards and skip/warn the check of that file ?

- It is not necessary to have a scheduler for the scanning / database of the tapes to be scanned / etc. as part of this tool. The focus should be on the scanning of the tape and fixing rather than on the request handling.

- For dual copy tapes, we can only be sure if the checksum/size is correct by recalling both copies and comparing them. This mechanism becomes complex to handle.

- Any need to support nl, al labelled tapes ? aul (and eventually the new tape format) should be supported.

Related Information

Ignacio and Charles had developed a tool as part of the tape to tape repack tests which scanned a tape. This is available at /afs/

-- GermanCancio - 17 Jun 2009

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2009-06-17 - GermanCancio
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    DataManagement All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback