5.10 Transferring MC Sample/Data Files

Complete: 4
Detailed Review status

Newsbox
Under construction

Goals of this page:

This page is intended to familiarize you with making a PhEDEx subscription and monitoring the progress of transfers to a site. In particular, you will learn:

  • what PhEDEx is?
  • why you should transfer data to your T2?
  • how to make a PhEDEx subscription to a site?
  • how to monitor PhEDEx transfers?

Contents

What is PhEDEx?

PhEDEx is the CMS data placement tool. It sits above various grid middleware [SRM (Storage Resource Manager), FTS (File Transfer Service)] to manage large scale transfers between CMS centres. CMS sites run a series of agents which run the requested transfers, verify that transfers have completed correctly and publish what data is available for transfer.

A normal user does not interact with this machinery. They will fill in a web form to make a request for data transfer to a site. Once this is approved the PhEDEx machinery takes over and makes sure that the transfer is complete.

Why do I need to transfer MC sample or Data files?

For general analysis CMS will use T2 centres, as the T0 and T1 sites will be busy carrying out reconstruction, re-reconstruction, skimming and AOD production. This means MC sample or data files need to be transferred from the T1 centres out to the T2's. It is up to the people working at T2 sites to choose which MC sample or data files goes to the site, and make the appropriate PhEDEx request.

If you run a CRAB job you and find that your MC sample or data is not located at a T2 centre you can request to have it transferred there using PhEDEx.

You do not need to have your MC sample or data at "your" T2 to run analysis, CRAB will run on it in any location. However, you may find it useful to make a copy at your local T2 as this will increase the number of sites you can run your analysis at.

For PhEDEx requests, if you are working with an analysis group, you can choose that group, however, that may mean files will be deleted in a regular cleanup. Otherwise choose "local" for the User Group.

No permission to view CMS.PhedexUserDocsSubscribeData

Copy files from other sites using gfal-copy command

Instructions are on the CRAB3 FAQ twiki on how to use gfal tools to find and copy files from another site's Storage Element.

Instructions for FNAL-LPC

The copyfiles.py script can be used to copy single files or a directory of files using gfal-copy or xrdcp from another site to T3_US_FNALLPC.

The getSiteInfo.py script can be useful to get the information of the site's endpoint to obtain a single file through gfal-copy, it is used by the copyfiles.py script above.

Review status

Reviewer/Editor and Date (copy from screen) Comments
JohnStupak - 2-July-2013 Review
NitishDhingra - 07-Apr-2012 See the detailed comments below
KatiLassilaPerini - 20 Feb 2008 created the template page

Substantial modifications due to depreciation of DBS. Instructions with snapshots for PheDex subscription using DAS interface added.

Responsible: SimonMetson
Last reviewed by: YourName - date

Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r18 - 2017-10-11 - MargueriteTonjes


ESSENTIALS

ADVANCED TOPICS


 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback