Using DQ2 to Get Datasets

You can browse for datasets on the web at the Panda Monitor and using the ATLAS Metadata Interface (AMI).

The following describes how to setup DQ2 at BNL. This requires that you have a grid certificate. Create the file dq2-setup.sh containing:

#!/bin/bash
source /afs/cern.ch/project/gd/LCG-share/current/external/etc/profile.d/grid-env.sh
sleep 1
voms-proxy-init -voms atlas
sleep 1
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh

Now source this script:

>> source dq2-setup.sh

This will prompt you for the password for your grid certificate. Now you can use the dq2 commands. For example:

>> dq2-ls mc08.017509.*bbmu*AOD*
mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474/
mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474
mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474_tid023515

One can print the number of files in a dataset with the -n options.

>> dq2-ls -n mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474
382     mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474

To list all of the files in a dataset, use the -f option.

>> dq2-ls -f mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474
mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474
[ ]     AOD.023515._00020.pool.root.2   88B215A7-AB50-DD11-ADBB-001D60630611    md5:8fd59ca4becb89055eecc504e96b709c    55027108
[ ]     AOD.023515._00335.pool.root.2   66B61BA2-AF50-DD11-917D-001D60D7D92F    md5:50dc3b9d7d26e5c6bec8fa2075af9fa6    58915242
[ ]     AOD.023515._00206.pool.root.2   345A3B3D-7950-DD11-94B9-00A0D1E50561    md5:27e471594b411869d5813467eee648a3    57806416
[ ]     AOD.023515._00044.pool.root.2   6409B9D1-AF50-DD11-8056-001D60AA1484    md5:0440200853d2deb93978f1bbbdc9f3d1    57946744
...

To copy locally all the files this dataset, use the dq2-get command:

>> dq2-get mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474

One should generally not do this. Instead copy only a few files locally for testing and later submit a job to the grid to analyze the entire dataset. To copy one file locally, use the -f option:

>> dq2-get -f AOD.023515._00020.pool.root.2 mc08.017509.PythiaB_bbmu9X.recon.AOD.e323_s400_d99_r474

For a more complete discussion of the options you can give dq2, see the TWiki page: DQ2 Clients.


-- RyanReece - 11 Jan 2008
-- RyanReece - 08 Oct 2008
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2008-10-21 - RyanReece
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback