Oscar's notes on EOS storage

A copy from http://wwwae.ciemat.es/~oglez/webcms/oginfo/eosnotes.html made 12.09.12

This describes the basic commands to use EOS storage at CERN. Since it is the preferred access way now, we focus on xrootd that also may work on other systems, like CASTOR, and preferred instead of the rfio protocol (nsls, rfdir, rfcp, ...)

As seen in the section describing the iteration with the Grid (here), the Grid-oriented tools are also available for interacting with EOS (as with CASTOR), using the proper syntax and server.

Basic handling of files

  • To list the content of a directory in EOS:
             eos ls [-l] /eos/cms/store/caf/user/oglez
    or, an alternative command (after setting the CMS software):
             xrd eoscms dirlist /eos/cms/store/caf/user/oglez
    This one is also available for CASTOR, in case it is needed:
             xrd castorcms dirlist /castor/cern.ch/user/o/oglez
  • One can use the command help to retrieve the full list of possibilities to interact with EOS.
    In addition, I define the variable $EOSCAF_HOME to mark the previous directory.
    However it should be remarked that EOS also allows to access the CAF-T2 area (/store/user/...) that can be used as EOS area too. It is unclear what is the difference but I will have to assume that EOS and xrootd will be the default access of "CERN-local" data, but the location will be the one at CAF-T2 (and you can find the information on how to use it <>a href="caft2notes.html">here).

  • To create a directory:
            xrd eoscms mkdir /eos/cms/store/caf/user/oglez/test
    In case one needs to set the permissions to allow copy from a group, which does not seem to be the case, that can be done using:
            eos chmod -r 775 /eos/cms/store/caf/user/oglez/test
    or also (in case we want to use it with CASTOR):
            xrd castorcms chmod /castor/cern.ch/user/o/oglez 7 7 5
  • To copy a file to EOS:
            xrdcp test.test root://eoscms//eos/cms/store/caf/user/oglez/test/hola.test
  • To copy a file from EOS:
            xrdcp root://eoscms//eos/cms/store/caf/user/oglez/test/hola.test temp/test.test
            xrd eoscms cp /eos/cms/store/caf/user/oglez/test/hola.test /eos/cms/store/caf/user/oglez/test/test.test2
    Note that the last one assumes that the indicated directory are in root://eoscms/, so it is only convenient when changing stuff (e.g. rename using xrd eoscms rm since xrd eoscms mv is not supported)
  • To view a file from EOS:
           xrd eoscms cat /eos/cms/store/caf/user/oglez/test/hola.test
  • To check the size of a directory of a file from EOS:
           eos find --size /eos/cms/store//caf/user/oglez/test/hola.test | awk -F='{size+=$3}END{print size/1024/1024/1024/1024" TB"}'

  • To delete a file in EOS:
            eos rm /eos/cms/store/caf/user/oglez//test/hola.test
            xrd eoscms rm [-r] /eos/cms/store/caf/user/oglez//test/hola.test

  • To check whether a file is online (available?)
            xrd eoscms isfileonline /eos/cms/store/caf/user/oglez/test/test.test2

  • To check the quota:
            eos quota
            eos quota | grep -A 4 "Quota Node: /eos/cms/store/caf/user/" | head -5

  • Interacting with EOS and the GRID

    For copying files from a GRID place to EOS (beware that the destination must be writable for the group ZH) The GRID user, using the appropriate proxy should be able to copy a file using the standard lcg-cp command, but with more options due to the specifics of the access to CASTOR:
          lcg-cp -v -b -D srmv2 -n 5 \
              "srm://srm.ciemat.es:8443/srm/managerv2?SFN=/pnfs/ciemat.es/data/cms/store/user/oglez/myfile" \
    Since this operation from the T2-CIEMAT is expected to be very common using a list of files given by the DBS Discovery, I have made a command that makes the copy using such a list:
          $OGTAREAS_DIR/commands/grid/gridtoeos.sh list-files eos-dir [ciemat]
    For other sites we need to predefine the location. See the routine for checks.

    In addition, to see the correct SFN associated to the file, see the description in the my CASTOR Notes.

    To send the output of a CRAB job to EOS, it is done as with usual Storage-Element (for which only the data files are sent there, not the log files).
    It is needed to have something like in the CRAB configuration file:
         return_data = 0
         copy_data = 1
    where, as always, the output directory should be writable by group ZH. There is another way to configure the output for EOS through CAF using caf_lfn but I do not think I will need it anyway.

    EOS and ROOT files

  • Opening a ROOT file located in EOS:
          TFile *file = TFile::Open("root://eoscms//eos/cms/store/caf/user/oglez/test/test.root");
  • Using a file in EOS inside a local CMS job is similar:
          fileNames = cms.untracked.vstring("root://eoscms//eos/cms/store/caf/user/oglez/...")

  • Comments about the web page: Oscar González