User-Created Libraries with existing Applications

Libraries also have dedicated support. This mostly depends on the application, in particular for Marlin, so here I show only the default method, but please check if the application you want to run for specificities.

All applications come with their own dependencies, so a user does not need to take care of those. He should only take care of those libraries that are not part of the software stack.

Let's say you have an application that depends in that is not a default library (in the Marlin case, one can replace existing processors, so this is covered in the Marlin section, in the Marlin case on also needs a special directory structure is mandatory! See here).

You will copy this into a lib directory. Now, 2 solutions are possible:

1. You are planning on submitting many jobs: I recommend you to put that lib directory on the grid and specify the LFN in the job definition. How to do that?

tar czf lib.tar.gz lib/
dirac-dms-add-file /ilc/user/i/initial/some/path/lib.tar.gz lib.tar.gz CERN-SRM
The /ilc/user/... is the LFN (Logical File Name). The i/initial/ part is user specific: you need to use your own DIRAC user name. You can find it in the dirac-proxy-init output, check for username. The some/path/ part is free to you, you can use whatever you want. There are a few limitations: you can not have a total number of subsequent directories greater than 14, and the final file name (here lib.tar.gz) cannot be longer than 128 chars. The last element, CERN-SRM indicates the logical name of the Storage Element on which you wish to upload you file.

For better reliability and reducing load on individual storage elements you should also replicate your library file to a few storage elements by running

dirac-dms-replicate-lfn /ilc/user/i/initial/some/path/lib.tar.gz DESY-SRM
dirac-dms-replicate-lfn /ilc/user/i/initial/some/path/lib.tar.gz RAL-SRM
dirac-dms-replicate-lfn /ilc/user/i/initial/some/path/lib.tar.gz CERN-DIP-4
dirac-dms-replicate-lfn /ilc/user/i/initial/some/path/lib.tar.gz CERN-DST-EOS
dirac-dms-replicate-lfn /ilc/user/i/initial/some/path/lib.tar.gz PNNL3-SRM

If you have to replace the file the dirac-dms-remove-files command will remove the file from all storage elements.

Warning, important When running with the CALICE VO, the path has a different beginning: /calice/users/i/initial. Notice the s at users. Also, CERN-SRM is not a valid storage element for CALICE users, so DESY-SRM or IN2P3-SRM must be prefered.

This LFN is registered in the DIRAC File Catalog, so it can now be used in the job definition.

If you wish to replace the file, you cannot overwrite the file, you need first to issue a dirac-dms-remove-files /ilc/user/i/initial/some/path/lib.tar.gz then re upload.

Moving a file cannot be done on the GRID: if really needed, you need to get the file (dirac-dms-get-file /ilc/...) then remove it from the GRID (same as above), then re upload it to the new location.

You now have a file on the GRID that you wish to input for a series of jobs. You would use the following:

Notice the LFN: part that is used by DIRAC to identify the files that must be downloaded from the grid. If you omit it, the job submission should fail because an input sandbox file will be missing. The fact that it's a tar ball does not matter, it will be untarred automatically.

2. You have only 1 job: Simply input the lib directory like this

If you have a tar ball, you may as well use it:
as any tar ball is automatically untarred in the job running directory.

The reason for the different behavior between the 2 cases is because every job submitted uploads its sandbox to the main DIRAC servers, filling its hard drive. All files are sent but those that start with LFN:.

-- AndreSailer - 2014-12-08

This topic: CLIC > Detector > CLICSoftwareComputing > Software > DiracUsage > DiracForUsers > IlcdiracUserLibraries
Topic revision: r4 - 2015-11-13 - AndreSailer
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback