T2 Data Manager Duties


  • every T2 has one or more data managers taking care of
    • approval of PhEDEx subscriptions
    • approval of PhEDEx deletions
    • space provisioning and quota management at the site

  • Data Managers are registered in SiteDB for a site with role Data Manager
    • get email notification for every PhEDEx subscription and deletion requests for the site
    • get permission to approve PhEDEx subscription and deletion requests for the site


  • approve or disapprove pending PhEDEx subscriptions and deletion requests within 1 business day

  • provision disk space at T2 sites and manage quotas of different disk spaces
    • PhEDEx uses groups to distinguish different disk spaces at the sites
    • PhEDEx does not have mechanisms to enforce quotas, the data managers of the sites themselves have to maintain quotas
      • important, not all data stored at T2 sites is tracked by PhEDEx, /store/user files for example are not tracked but occupy disk space
    • different kinds of disk spaces:
      • Ops space (30 TB)
        • holds /store/unmerged, /store/temp (files in these directories older than 2 weeks can be deleted automatically, a cron job is advisable)
        • holds samples subscribed by central operations, MinBias and generator samples as input to MC production
        • holds produced MC files before they are archived to a Tier-1 site
      • Central space (nominal in 2012: 200 TB)
      • Physics Group Space (nominal in 2012: 125 TB per group, 2 groups per site)
      • Local/Regional Space (nominal in 2012: 170 TB)
      • User Space (nominal in 2012: 160 TB, 4 TB per user, ~40 users)
      • sites larger than normal can provide resources for more central space, three or more physics group spaces and additional regional space
      • sites smaller than normal may only provide resources for only one physics group, or only central space, or if sufficiently small, only for simulated event production
    • data managers have to ensure while approving requests that quotas of space tokens are not exceeded
      • if a request would exceed the quota of a space token, the data manager should disapprove the request and state the reason in the comment of the disapproval
    • data managers have to ensure that the reserved quotas are available and not taken by other files (for example /store/user files)
      • example: a physics group has a quota of 125 TB at a site but is only using 25, the group has to be able to transfer 100 TB without notice to the site and the site should have enough capacity to accept the transfer

Group quota overview and deletion suggestions

  • An overview of how much space per group is used at every site is available on the PhEDEx webpage: https://cmsweb.cern.ch/phedex/prod/Reports::SiteUsage?node=T2_CH_CERN#
  • The popularity service and Victor the cleaning agent (services only available for CMS members) can also be used to check how much space is used per group and give suggestions which datasets haven't been accessed by CRAB analysis jobs recently and are good candidates to clean up disk space:
    • go to Victor the cleaning agent
    • use the plots in the lower half of the page under "Accounting summary by region" to check the group space fill status for a given site in a region compared to the pledged disk space
    • filter the upper part of the page under "Accounting summary" for a site name to see a table of the same information
    • clicking on a line per site in the table brings up suggestions what to delete to clean up disk space for this group at a site
  • The suggestion could be used while disapproving a request to suggest to the group which samples to delete to clean up disk space before subscribing new samples
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2012-06-15 - OliverGutsche
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback