Transfer Scripts

Global CMS checks

global_dataset_replica_check.sh: check dataset replica
  • datasets without replica : set their DBS status to DELETED
  • global Test and Backfill datasets
  • T0 test datasets
  • datasets without custodial location

global_production_dataset_check.sh: check datasets still in PRODUCTION status

  • production datasets without replica
  • Test and Backfill production datasets older than one month
  • datasets older than one month, but still in PRODUCTION status

global_GEN_T2_check.sh: check GEN datasets older than 6 months at T2 sites, and submit a deletion request if necessary

Script Description
storage overview overview of the storage over T1 sites.
global datasets custodial location check check datasets without custodial location.
global datasets status check check datasets older than one month but still in PRODUCTION status.

1. cron jobs on FNAL transfer machine(cmssrv162)

If you would like to check or edit the cron jobs on cmssrv162, you should log into the machine using the account cmsdatatransfers and your FNAL kerberos password. In general, these cron jobs can be categorized into 2 parts: FNAL regular checks and global CMS checks.
  • FNAL regular checks
    FNAL has a huge disk space which we need to manage, there are several categories of data at FNAL (not MC!): custodial datasets to tape, non_custodial datasets to disk.
    We need to make sure that:
    • <1> important custodial data is always on disk for fast access => pinning
    • <2> FNAL has a complete copy of all AOD, RECO & RAW of the current and previous data taking period.
    • <3> newly produced samples are transferred to FNAL to complete the copy.
    • <4> everything first into the available disk space.
    • <5> FNAL does not miss any files and there are no orphans

Script Description
pinned data FNAL pinned data list, volume and categories.
datasets to pin custodial datasets need to be pinned on disk.
2012 AOD 2012 AOD datasets need to be transferred to FNAL
dark data datasets stay at FNAL but not registered in TMDB

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2014-06-20 - MericTaze
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback