How to work with the GRID

(Most) CALICE data is stored on the grid. In the past, the LCG catalogue was used to organise this data. We are now (Summer 2020) transferring all information to the ilcdirac file catalogue.

see this page for up-to-date instructions for use of ilcdirac tools to access and use grid resources.

The following describes out-of-date lcg tools, and is kept only for reference

OLD OUTDATED INSTRUCTIONS (follow at own risk!)

First steps

1. Get a Grid certificate from your local issuing authority

2. Once you have the certificate, request membership of CALICE VO here .

More details on this procedure (for example, how to load the certificate into your browser) can be found here .

Introduction

This is a small set of grid commands that I've found useful for the every day work with the GRID, with the caveat that I'm no expert. If you want to find more details, please have a look at the DESY grid web pages, and the FLC group wiki pages (please note that the latter is not official information).

I assume you already have a grid certificate.

note added April 2018

Several older samples are in the LCG file catalogue, and not necessarily transferred to the dirac one. You can use lcg tools to access these files.

e.g. on DESY machines, you can follow these instructions to set up your environment.

Then you can access the lcg catalogue, for example, like:

lcg-ls -l lfn:/grid/calice/tb-MCProduction/calice_v02-00/ilcsoft_v01-07
    

Note: in case you don't need the grid proxy anymore, don't forget to destroy it:

 grid-proxy-destroy
     

Listing files on the grid

          lfc-ls /grid/calice/users
          lfc-ls /grid/calice/generated
      

Please note that if you need user space on the CALICE grid, you have 2 options:

  • create a directory with your user name in the /grid/calice/users/<your-user-name>.
  • create a directory with your user name in the /grid/calice/generated/<your-user-name>.

Note that the files in both places here are saved on the disk only (while the CALICE data are saved on tape). Nevertheless, please delete old files which you don't use anymore, because otherwise the space cannot be reused.

Creating a directory

          lfc-mkdir -p /grid/calice/generated/lucaci/test
      
Note: the -p option is for creating directories recursively (i.e. create 2 directories: lucaci and lucaci/test).

Upload a file to the storage element

  • first, create a simple file to be uploaded:
         echo "Put something here" > text-file.txt
        
  • find which are the available storage elements for your VO:
         lcg-infosites --vo calice se
         
  • now copy the file a chosen storage element and register the file in the File Catalog:
         lcg-cr --vo calice  -l lfn:/grid/calice/generated/lucaci/test/text-file.txt -d dcache-se-desy.desy.de file:${PWD}/text-file.txt 
         
    You'll get something like this:
         guid:0a01303f-aa1a-4754-a08b-8958dc713004
         
    The guid is an unique identifier for your file on the storage element.
  • to list all the replicas of a file that have been successfully registered in the File Catalog (i.e. to find the real name of the file):
         lcg-lr lfn:/grid/calice/generated/lucaci/test/text-file.txt
         
This will result in something like this:
     srm://dcache-se-desy.desy.de/pnfs/desy.de/calice/generated/2010-05-27/filea6fb00ca-9c23-4d83-90e7-9e0acd0c96e1
     
This has an incovenience: if the uploaded file is an LCIO file which you want to run Marlin on, you have to give the real name of the file (the one containing numbers) in the Marlin steering file, which I find unpleasant. To fix this, let's first delete the file from the storage element (and in the same time remove its entry from the file catalog):
     lcg-del -a --vo calice lfn:/grid/calice/generated/lucaci/test/text-file.txt
     
Now I want to register the file in the File Catalog by using its human readable name. This I do by using the option -P:
     lcg-cr --vo calice -d dcache-se-desy.desy.de -P generated/lucaci/test/text-file.txt -l lfn:/grid/calice/generated/lucaci/test/text-file.txt  file:${PWD}/text-file.txt
     
If I now check its replica:
     lcg-lr lfn:/grid/calice/generated/lucaci/test/text-file.txt
     
I get what I wanted:
    srm://dcache-se-desy.desy.de/pnfs/desy.de/calice/generated/lucaci/test/text-file.txt
    

For uploading a binary file to the grid:

PATH= path to store (e.g. tb-desy/native/dat)
FILE= file to store (e.g. Run000001.000.bin)
{PWD}= data directory on your computer

lcg-cr --vo calice -l lfn:/grid/calice/PATH/FILE -d srm://dcache-se-desy.desy.de/pnfs/desy.de/calice/PATH/FILE file:/{PWD}/FILE  

Download the file to the grid User Interface (i.e. to your computer):

    lcg-cp --vo calice lfn:/grid/calice/generated/lucaci/test/text-file.txt file:${PWD}/text-file_copy.txt 
    

Replicate files between storage elements (SE):

    lcg-rep --vo calice -d <SE> lfn:/grid/calice/generated/lucaci/test/text-file.txt
    
There is no output from this command, but you can check the replicas with:
    lcg-lr lfn:/grid/calice/generated/lucaci/test/text-file.txt
    
You should get the two replicas.

Deleting a file and a directory

  • delete a file:
        lcg-del -a --vo calice lfn:/grid/calice/generated/lucaci/test/text-file.txt
        
  • delete the working directory from the file catalog:
        lfc-rm -r /grid/generated/lucaci/test
        

Create a symbolic link

    lfc-ln -s <source file> <link name>
    lfc-ln -s /grid/ilc/ilcsoft/ilcsoft-v01-10-full.tar.gz /grid/calice/software/ilcsoft_trunk_sl4.tgz
    

Delete a symbolic link

 lfc-rm /grid/calice/software/ilcsoft_trunk_sl4.tgz
   

For more

As usual, for more details, see the manual, e.g.:
    man lcg-cr 
    

-- DanielJeansExternal1 - 2018-04-12 -- AngelaLucaciTimoce - 16-Dec-2010

Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r12 - 2020-06-11 - DanielJeans1
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CALICE All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback