Dark Data Deletion at FNAL


Dark data are old files or datasets, which are physically at FNAL but not registered in Phedex. Fermilab produces dark data list every week. This list is further analyzed and broken into several categories using a darkdata script. The output from the script gives us the overview of potential dark data we can delete at Fermilab. We do further checks and analysis of these dark data lists, then make deletion decisions as documented on this Twiki.

This page is meant to list the dark data we are going to delete, or register in TMDB, or change the status in DBS.

General Check Princples

  • Check dataset/file status in DBS
  • Check dataset/file if they have higher version and its status in DBS
  • Check dataset/file custodial locations

List of dark data campaingns

Campaign Comment start (est.) end
CompOpsDarkDataSep19campaign 1. This campaign is based on dark_data/120906/out_dir/files_toremove/ Sep.2012 ???
2. Taking into account the impact from deletion campaign at CMS T1s that happened in parallel
3. Taking into account the impact of small bugs we found (and corrected) in the script

-- SiLiu - 19-Sep-2012

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2012-09-20 - MarekZielinski
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback