Difference: DiracForUsers (1 vs. 61)

Revision 602016-10-30 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 70 to 70
 Warning, important Outdated, there is a completely different webinterfaces
Changed:
<
<
The web monitoring of jobs happen here: https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/jobs/JobMonitor/display Connect to the ILCDIRAC monitoring page and you'll get something like the image below.
>
>
The web monitoring of jobs happen here: https://voilcdiracwebapp.cern.ch/DIRAC/?view=tabs&theme=Grey&url_state=1|*DIRAC.JobMonitor.classes.JobMonitor: Connect to the ILCDIRAC monitoring page and you'll get something NOT (because the picture is outdated but somewhat similar) like the image below.
  webMonitoring.png

Revision 592016-09-30 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 67 to 67
 Warning, important The maximum size of an output sandbox is 10Mb. If the sum of the file sizes in there is bigger, the sandbox is not sent to the DIRAC sandbox store, but they are added to an archive send to the GRID. The Storage Element chosen depends on the site (usually a SE that's close to the site where the job ran). In this case, the retrieval from the web portal will fail with an error (missing sandbox), but the command line work as it retrieves the sandbox automatically from the GRID (not possible via the portal).

Monitoring the jobs, Webinterface

Added:
>
>
Warning, important Outdated, there is a completely different webinterfaces
  The web monitoring of jobs happen here: https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/jobs/JobMonitor/display Connect to the ILCDIRAC monitoring page and you'll get something like the image below.

Revision 572015-11-02 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 139 to 139
 Well, start by looking at the documentation at DiracForUsers and DiracByExample. Then contact ilcdirac-register@cernNOSPAMPLEASE.ch for more info.

I need to use my own processors

Added:
>
>
See also here: IlcdiracUserLibraries
 It's fully taken in account in dirac. For that, you'll need to compile them against a version that dirac knows. And we defined a directory containing those version on afs under /afs/cern.ch/eng/clic/software/x86_64-slc5-gcc41/.
Changed:
<
<
So simply setup the env, use cmake including the ilcsoft.cmake in the directory of your choice from the available ones, and put your processor/libraries in the proper directories as mentionned in the HeadFirstTalk above. And normally you should be going. We try to keep the version up to date, so you should be able to find always the latest release of ILCSOFT. We can also define new versions if needed, as long as one admin has access to the software binary location.
>
>
So simply setup the env, use cmake including the ilcsoft.cmake in the directory of your choice from the available ones, and put your processor/libraries in the proper directories as mentioned in the HeadFirstTalk above. And normally you should be going. We try to keep the version up to date, so you should be able to find always the latest release of ILCSOFT. We can also define new versions if needed, as long as one admin has access to the software binary location.
 

I want to specify an input file using for e.g. srm://srm-ilc.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/ ?

You need to use the GetSRMFile class of ILCDIRAC as first job application. See DiracForUsers for more details, and the ilcdiracdoc page, UserJob and GetSRMFile specifically.

Revision 552015-06-01 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 92 to 94
 %ENDSYNTAX%
Changed:
<
<

F.A.Q.

>
>

F.A.Q. and Frequently Encountered Problems

My Job Submission is Very Slow

If the job repository gets too large your job submission becomes very slow. Use different job repository files, for example name the repository file after your job group

jobGroup = "resonableDescription_take1"
dirac = DiracILC(True, jobGroup+".rep")

My jobs keep failing to upload output data

If an outputfile already exists on the grid, your job will not be able to overwrite it. You have to either delete your outputfiles before submitting your jobs again or use, for example, the jobgroup as a subfolder to differentiate different job groups Use the jobgroup to separate outputfiles in subdirectories.

jobGroup = "jetReco_take1"
...
job.setOutputData(["somefile1","somefile2"],"some/path/"+jobGroup,"CERN-SRM")
Change jobGroup whenever there is a new set of steering files, parameters or whatever to avoid trying to overwrite your outputfiles

If you no longer need a set of output files, please remove them from the storage.

 

I keep getting Could not verify credential errors

You see errors like:

Revision 542014-12-16 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 148 to 148
 

How to replicate a file

dirac-dms-replicate-lfn /ilc/your/file CERN-SRM 
Added:
>
>
dirac-dms-create-replication-request CERN-SRM <LFN|LocalFileWithLFNs> 

Or use the replicate command in the filecatalog-cli.

CERN-SRM is one of the storage elements configured. See above for list of other StorageElements.

 
Deleted:
<
<
CERN-SRM is one of the storage elements configured. You are free to use CERN-SRM, IN2P3-SRM, RAL-SRM, IMPERIAL-SRM, DESY-SRM (not for prod files), TAU-SRM (does not always work), FNAL-SRM (need to check availability). New storage elements will be made available in the future.
 

Revision 532014-12-16 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 18 to 18
 
Changed:
<
<

Introduction to the ILCDirac interface to the LC Software Applications

Storage Elements

>
>

Storage Elements

 The following storage elements can be used as output location:
CERN-SRM (default)
Line: 35 to 31
  We recommend you use a storage element that's close enough geographically to you.
Added:
>
>

Introduction to the ILCDirac interface to the LC Software Applications

 

Available Software Versions

Use the script: dirac-ilc-show-software that dumps the current available software.

Revision 512014-12-08 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Changed:
<
<

Getting Started: Registration, Installation, etc.

>
>

Getting Started: Registration, Installation, etc.

 
Changed:
<
<

File access, downloading, uploading etc. with the DiracFileCatalog

>
>

File access, downloading, uploading etc. with the DiracFileCatalog

 First read the "Data Management" DIRAC-Tutorials.

Line: 18 to 18
 
Changed:
<
<

ILC specific jobs

This section shows the preferred way of creating jobs in ILCDIRAC. This uses the PYTHON API. To use it, some basic knowledge of object oriented (OO) programming is needed, as PYTHON is a completely OO language. The examples shown below use the API described in http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/.

The philosophy of the ILCDIRAC interface is the following: a user must care about what he wants to run, but not how. That's why we provide the interface to all ILC applications that will prepare the runtime environments and execute them. The users need not to care about environment variables, directories to look for, etc. To run an application in ILCDIRAC, there are only very few mandatory parameters, in particular those that would be passed on the command line to call a pre installed application (example: the steering file for Marlin). Moreover, for example, input files do not need to be changed by the user to match what the job is going to run on: this is changed automatically (by default) in the steering files. The idea is that a user would test his run locally, then pass directly the steering file to the job definition without having to do any modification.

Job types and basic job definition

In ILCDIRAC, there are 2 main job types: User jobs and Production jobs. Here are only covered the User ones. Let's start with a generic example that doesn't do anything:

<!-- SyntaxHighlightingPlugin -->
from DIRAC.Core.Base import Script
Script.parseCommandLine()

from ILCDIRAC.Interfaces.API.DiracILC import DiracILC
dirac = DiracILC(True,"some_job_repository.rep")

from ILCDIRAC.Interfaces.API.NewInterface.UserJob import UserJob
job = UserJob()
job.setName("MyJobName")
job.setJobGroup("Agroup")
job.setCPUTime(86400)
job.setInputSandbox(["file1","file2"])
job.setOutputSandbox(["fileout1","fileout2"])
job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
print job.submit(dirac)
<!-- end SyntaxHighlightingPlugin -->

If submitted as is, it will fail because the different files are not there.

Let's go through the individual lines. The first 2:

<!-- SyntaxHighlightingPlugin -->
from DIRAC.Core.Base import Script
Script.parseCommandLine()
<!-- end SyntaxHighlightingPlugin -->
are mandatory to get the DIRAC environment known to the script (This is oversimplifying things but enough). For example, this way the different services that are used get their server addresses set. This also initializes many internal DIRAC utilities that ILCDIRAC makes use of (logger functionality for example). They need to be called before the other DIRAC imports to avoid race conditions.

The following 2 line

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.DiracILC import DiracILC
dirac = DiracILC(True,"some_job_repository.rep")
<!-- end SyntaxHighlightingPlugin -->
take care of importing and creating a DiracILC object. This class is the "job receiver" class, as the last line job.submit(dirac) indicates. It's needed as it makes sure that all the specified inputs (if any) are actually available. It also checks that all requested software (more on that later) is available. Several additional utilities are provided by the inheritance of DiracILC from the main Dirac class (see API doc), not discussed here. Finally, it provides the interface to the Job Repository. This is a text file, in this case some_job_repository.rep, that has a special structure: it hold the jobIDs of the submitted jobs, and their properties (have a look once you have submitted a job). It's very important to keep this file safe as it is used to retrieve the job outputs easily (more on that later). A good way to use this functionality is to have a different file name per activity,

The next few lines

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.UserJob import UserJob
job = UserJob()
job.setName("MyJobName")
job.setJobGroup("Agroup")
job.setInputData(['datafile1','datafile2'])
job.setInputSandbox(["file1","file2"])
job.setOutputSandbox(["fileout1","fileout2"])
job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
<!-- end SyntaxHighlightingPlugin -->
are needed for an actual job definition. Again, in this example, the job doesn't do anything, and will likely fail if submitted: the specified inputs are not available (more on that later).

The process of job definition starts with

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.UserJob import UserJob
job = UserJob()
<!-- end SyntaxHighlightingPlugin -->
where you instruct ILCDIRAC what type of job to use. This has little visible consequences for the users, but is making important choices for the internal functionality. Warning, importantUsers should ALWAYS use a UserJob unless explicitly asked or recommended.

The next few lines give some of the UserJob class method use example. The first 2, setName and setJobGroup, can be used for the monitoring: the names are displayed on the monitoring page and the JobGroup can be used to select only the jobs belonging to a given group.

The next one, setInputData is used as it's name suggest to define input data. Input data means files that can be on a tape backend and can rely on staging (recall from tape) to be available. Files produced from the production system should be accessed using this method. There is unfortunately no link between this and the application setInputFile so you'll need to specify the file in both. The file name to give is an LFN (Logical File Name, described in the File Catalog section of this page), like /ilc/prod/clic/[...]/something.slcio.

The next 2, setInputSandbox and setOutputSandbox, indicate to the job what files should be shipped to the job working area, and what files should be brought back when retrieving the job outputs. Any file in the setInputSandbox must reside locally (absolute or relative path accepted), or be stored on the GRID, and specified with the LFN, where it must be prepended by LFN:. For example, if there is a file /ilc/user/s/someone/file, the file to specify in the setInputSandbox should be LFN:/ilc/user/s/someone/file. As it can be seen in the example, python lists can be used, but not lists of lists. Warning, importantMake sure your sandboxes do not contain such things, as the system cannot correct them. Tip, idea It is a good idea to put on the GRID large input files (or often used) and specify them using the LFN: definition: they make the job submission much faster and make the ILCDIRAC servers happy (no huge amount of data to carry around). How to do this is given in the section Specifying your libraries.

The setOutputSandbox follows the same style. There are several important things to notice: If a file is missing in the output, DIRAC will mark the job as failed, but the output will be filled with whatever is available among the requested things. If the sandbox is bigger than 10MB, the files will be packed together in a tar ball and shipped to the GRID. Warning, importantIn version ILCDIRAC v16r7p0, it seems that retrieving the output sandbox of a job does not retrieve those files from the GRID. Retrieving job outputs is explained later on this page.

The last part of this section is

<!-- SyntaxHighlightingPlugin -->
job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
<!-- end SyntaxHighlightingPlugin -->
This is used when a user wants to keep some files on the GRID for further processing or for easy sharing with others. It implies that the specified files somefile1 and somefile2 will be uploaded to the GRID under the automatically built LFN /ilc/user/u/username/some/path/. u and username indicate the DIRAC user name of the user that created the job (u is the initial). The some/path part of the LFN comes from the second element in the method call above. Finally, the storage element to store the files can be specified by passing its logical name in the last element of the method call. In this example, e use CERN-SRM, which is the default storage for ILCDIRAC. See the Storage Elements section for a list of valid SEs. Warning, importantIf the output files already exist in the File Catalog, the jobs will fail because overwriting files is not possible. Be sure to clean before submitting the jobs.

Finally, the line

<!-- SyntaxHighlightingPlugin -->
print job.submit(dirac)
<!-- end SyntaxHighlightingPlugin -->
submits the job, and applies the checking procedure defined internally. The print statement allows to see on screen the job ID. You can check that this job ID is in the some_job_repository.rep file.

This ends the general presentation of the jobs. More information can be found in the API documentation. The next sections will show how the different applications can be configured.

ILC applications-job framework

For maximum flexibility in the ILCDIRAC interface, the application-support framework was carefully designed. It's based in the following assumptions:

  • An application has several generic properties
    • It has a name and possibly a version
    • It will be steered with some configuration file
    • It will process a given number of events (for HEP)
    • It will maybe produce some log file
    • It will produce some output file
    • It may process some input data
    • It may have energy dependency (for HEP)
  • An also applciation specific
  • An application is a block of treatment of information
    • It should be easy to "plug" an application in a workflow
  • A Job is only one way to run an application
    • An application should be "standalone"

Those requirements were implemented in the ILCDIRAC Application framework. The corresponding PYTHON implementation is best described in the API documentation. For completeness, we show here a python code that makes use of this framework, although it's just a non functional example. All ILC applications described below can use the same methods.

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Application import Application
ap = Application()
ap.setName("MyName")                  #application name
ap.setVersion("1")                    #version
ap.setLogFile("MylogFile")            #log file name (stdout of the application)
ap.setInputFile("MyInput")            #input file name, can be a list, 
                                      #can contain elements with LFN:
ap.setOutputFile("MyOutput")          #output file name
ap.setSteeringFile("SomeSteeringFile")#steering file (configuration), can have LFN:
ap.setNumberOfEvents(10)              #Obviously... 
ap.setEnergy(3000)                    #Energy
res = job.append(ap)                  #Add this application to the job
if not res['OK']:                     #Catch if there is an error
    print res['Message']              #Print the error message
    #do something, like quit
<!-- end SyntaxHighlightingPlugin -->

Here, job is an instance of the UserJob class described before. It is possible to stack applications one after the other:

<!-- SyntaxHighlightingPlugin -->
ap1 = Application()
ap2 = Application()
res = job.append(ap1)
if not res['OK']:
    print res['Message']
    #do something, like quit
res = job.append(ap2)
if not res['OK']:
    print res['Message']
    #do something, like quit
<!-- end SyntaxHighlightingPlugin -->

It is also possible to chain applications: the second one can get its output from the first one:

<!-- SyntaxHighlightingPlugin -->
ap1 = Application()
ap1.setOutputFile("MyOutput")  # Needed when chaining
ap2 = Application()
ap2.getInputFromApp(ap1)   #This is where the magic happens
res = job.append(ap1)
if not res['OK']:
    print res['Message']
    #do something, like quit
res = job.append(ap2)
if not res['OK']:
    print res['Message']
    #do something, like quit
<!-- end SyntaxHighlightingPlugin -->
This last property does not make sense for all applications, in particular for the user application, as DIRAC cannot "guess" what a random binary will produce. If specified in the application steering file AND in the setOutputFile field, then it will behave as expected.

setRequired why and how?

The information in this section are not relevant at the moment. The setRequired method is currently not needed (March 25, 2014).

The setRequired method is used to specify files and/or directories that must be local to the execution. The reason it's needed is due to several required changes. In the past all the applications were running in the same directory. This made file handling simple, but limited. In particular, the steering files handling was cumbersome as a user could not use a different version than the default provided with the application. To add flexibility, it was decided that all applications would run in their own directories, therefore preventing overwrite of steering files in case a user specifies a different version than the default. If it adds flexibility, it also makes file management trickier: data files need to be moved from directories to others, special files (steering files and e.g. gear files) need copying, etc. The handling of specific files was implemented (those that must be explicitely defined for the application to run, like the steering file). Remained those that do not have explicit setters such as the pandora settings or the lcfi weights. For those, the setRequired method is added.

The usage is as follows:

  • Let's assume you are running Marlin and the LCFI flavour tagging processor.
  • To run, this processor requires the lcfi weights as a folder in the run directory.
  • In the past, you would create a tar ball with this folder, ship it to the GRID, and pass the LFN as input sandbox. This step does not change.
  • You now need to call the setRequired method of your Marlin instance with marlin.setRequired("lcfiweights"). Here, lcfiweights is the directory containing the weights, the one that was added to the tar ball. Warning, importantIt is not the tar ball itself.

It is also possible to specify multiple items like app.setRequired(["data",'lcfiweights',"myextrafile"]). You may also use wildcard characters such as app.setRequired("folder/*.ext"). This last one will create a local folder "folder" and copy all files *.ext in it.

Warning, important It is important to mention that all applications do not use the setRequired: certain utility applications (like the splitters) do not use it. This is mostly because they do not require external files, and do not run in their own directory (so no file copy is needed). If you use it on one of those, you'll get an uncaught exception while submitting.

Warning, important You still need to specify the files in the input sandbox as the setRequired does not add them to the sandbox.

Warning, important The lib directory is always copied if found (see Specifying your libraries).

The next sections describe the application specific methods (setters). Everything that is described in this section applies for the following, except otherwise indicated.

Generic application

A generic application is any executable that is not part of the ILC software chain (but can use some ILC software) and not ROOT (as ROOT has it's own wrapper). I may be used to run a user code.

A generic application is defined as follows:

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import GenericApplication
ga = GenericApplication()
<!-- end SyntaxHighlightingPlugin -->

Check the API documentation for the full detail.

The 3 methods that are specific to this Application are the following:

  • The setter of the script to run:
<!-- SyntaxHighlightingPlugin -->
ga.setScript("somescript_or_binary")
<!-- end SyntaxHighlightingPlugin -->
This can be any executable script (shell or python). Check that the mod (chmod +x) is correct before submission. It can also be an LFN:
  • The arguments to pass to the script
<!-- SyntaxHighlightingPlugin -->
ga.setArguments("some command line arguments")
<!-- end SyntaxHighlightingPlugin -->
The arguments here are passed as they are input here.
  • A dependency to an ILCDIRAC supported application:
<!-- SyntaxHighlightingPlugin -->
ga.setDependency({"ROOT":"5.34"})
<!-- end SyntaxHighlightingPlugin -->
This makes sure the dependency (here ROOT 5.34) is installed prior to running the script. Any ILCDIRAC supported application can be set here, the full list can be obtained by running dirac-ilc-show-software.

Generation

The generation of events in ILCDIRAC is done using WHIZARD 1.95 (for the moment) for most events, and with PYTHIA directly for ttbar, WW, and ZZ events. The support for WHZAIRD 2. is being developed, so if you need events produced with WHIZARD 2, you should request help from ilc-dirac@cernNOSPAMPLEASE.ch.

Whizard 1.95

This generator is used for the CLIC CDR and ILC DBD studies. It has several limitations not discussed here, but the main one being that it requires a dedicated binary containing the process one wants to study. As having an exhaustive list of processes is not possible, only a few (>1000) are available. If you need to use this application, it's a good idea to contact ilc-dirac@cernNOSPAMPLEASE.ch to ask if a given process is available. If not, someone will (or not) add it to WHIZARD and make it available. When WHIZARD 2 is supported by ILCDIRAC, the process can be obtained directly as WHIZARD will be ran "automatically".

The standard way to create a WHIZARD applciation is by using

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import Whizard
from ILCDIRAC.Interfaces.API.DiracILC              import DiracILC
d = DiracILC()
wh= Whizard(d.getProcessList())
wh.setModel("sm")
pdict = {}
pdict['process_input']={}
pdict['process_input']['process_id'] = "qq"
pdict['process_input']['sqrts'] = 1400
pdict['simulation_input'] = {}
pdict['simulation_input']['n_events'] = 10000
pdict['beam_input_1'] = {}
pdict['beam_input_1']['particle_name']='e1'
pdict['beam_input_1']['polarization'] = "0.0 0.0"
pdict['beam_input_1']['USER_spectrum_on'] = 'T'
pdict['beam_input_1']['USER_spectrum_mode'] = 19
pdict['beam_input_1']['ISR_on'] = 'T'
pdict['beam_input_2'] = {}
pdict['beam_input_2']['particle_name']='E1'
pdict['beam_input_2']['polarization'] = "0.0 0.0"
pdict['beam_input_2']['USER_spectrum_on'] = 'T'
pdict['beam_input_2']['ISR_on'] = 'T'
pdict['beam_input_2']['USER_spectrum_mode'] = -19
wh.setFullParameterDict(pdict)
wh.setOutputFile("toto.stdhep")
res = j.append(wh)
if not res['OK']:
    print res['Message']
    exit(1)
<!-- end SyntaxHighlightingPlugin -->

As usual, a detailed description of the class is available on the API documentation page. I will only briefly discuss the instance definition, and how it works.

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import Whizard
from ILCDIRAC.Interfaces.API.DiracILC              import DiracILC
d = DiracILC()
wh= Whizard(d.getProcessList())
<!-- end SyntaxHighlightingPlugin -->
As can be seen on the code snipped above, it is necessary to get a DiracILC instance to create a WHIZARD instance. The reason is that it's mandatory to get the available processes for WHIZARD to be configured. In particular, the process list contains all the processes as well as in which version of WHIZARD they are defined. The d.getProcessList() takes care of downloading the file locally. You will find the processlist.whiz file in the local directory.

The next thing is wh.setModel("sm"). All the avaialble models are defined in the Configuration Service under /Operations/Defaults/Models. In this example, the SM is used so nothing particular happens. When using SUSY for instance, this is used to define which LesHouches file to use. It is always possible to overwrite the LesHouches file to use by placing one in the InputSandbox, and naming it LesHouches.msugra_1.in. No, it's not necessarily a msugra model, but that's just a name and it has to be like that to be picked up.

After, there is

<!-- SyntaxHighlightingPlugin -->
pdict = {}
...
wh.setFullParameterDict(pdict)
<!-- end SyntaxHighlightingPlugin -->
which allows setting the whizard parameters. The structure of the pdict is the same as the whizard.in file: there are sections like pdict['process_input'] that correspond to the process_input section of the whizard.in. All the sections are available this way, with the notable difference of the beam_input sections that are named explicitly beam_input_1 and beam_input_2 for clarity. All the possible parameters described on the whizard documentation page can be set.

The rest of the possibilities is described in the API page.

Pythia

The PYTHIA application is rarely used as it's not generic at all: it can only produce ttbar, WW, and ZZ events. When needing such events, I usually run it locally and upload the files. The issue with that application is the lack of flexibility, everything is hard coded in the FORTRAN code. It should be considered as an expert's application and will not be documented here. Nevertheless, for the advanced user, as usual, the API is documented at the usual location.

StdHepCut and StdHepCutJava

There are 2 versions of this application, which are equivalent: one in C++ and one in JAVA. The latter is needed because the former does unnecessary thing due to limitations of the C++ stdhep library. In practice, the JAVA version should be preferred. The interface being identical for both application, the example will use the JAVA one.

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import StdHepCutJava

cut = StdHepCutJava()
cut.setVersion("1.0")
cut.setNumberOfEvents(10000)
cut.setSteeringFile('cuts_quarks_1400.txt')
cut.setMaxNbEvts(10)
cut.setSelectionEfficiency(0.5)
res = j.append(cut)
<!-- end SyntaxHighlightingPlugin -->
The setNumberOfEvents value is used in conjunction with setSelectionEfficiency and setMaxNbEvts to make sure that enough events are input. The setMaxNbEvts is the maximum number of events to write in the output file.

This application uses the code available at http://svnweb.cern.ch/guest/lcgentools/trunk/stdhepcut_java. There is a README file in the svn repository that will tell you how to use the application, and by looking at the code you may be able to develop your own cut code. It you want to get your code integrated in the trunk, you should get in touch with either ilcdirac-support@cernNOSPAMPLEASE.ch or any LC common generator WG member.

The following only applies for the JAVA part, as the C++ does not allow it: You can use your own cut classes. They must be stored under org.lcsim.stdhepcut.cuts. Your corresponding jar file must be stored in a lib directory that you can then add to your sandbox. The StdHepCutJava application takes the lib directory and adds it in the CLASSPATH.

Simulation

The design of those applications could be a bit better structured, as they are all independent, instead of inheriting from a common SimApplication class. That's just an implementation detail, transparent for the user.

Mokka

This application was the first to be added to ILCDIRAC.

It's interface is described like any other in the API documentation. The example below shows how it can be used:

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import Mokka
mo = Mokka()
mo.setInputFile("some_file.stdhep")
mo.setVersion("0706P08")
mo.setSteeringFile("clic_ild_cdr.steer")
mo.setOutputFile("totosim.slcio")
res = j.append(mo)
<!-- end SyntaxHighlightingPlugin -->
That's the simplest way to call the application.

There are more possibilities that are worth showing:

<!-- SyntaxHighlightingPlugin -->
mo.setDetectorModel('Something')
<!-- end SyntaxHighlightingPlugin -->
allows to define another detector model, as the default is CLIC_ILD_CDR.

<!-- SyntaxHighlightingPlugin -->
mo.setDbSlice("something.sql")
<!-- end SyntaxHighlightingPlugin -->
allows to define your own MySQL database dump. It can be a local file or an lfn. The specified detector model must exist in the dump of course.
>
>

Introduction to the ILCDirac interface to the LC Software Applications

 
Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
mo.setMacFile("mymacro.mac")
<!-- end SyntaxHighlightingPlugin -->
Allows to specify your own macro file. This is needed when you want to use ParticleGun for example. It can be a local file or an lfn. Whatever is in there is unchanged. If you don't specify a macro file, the application creates one with the input file, the startFrom and the number of events specified. This means that if you have your own macro file and want to use the startFrom functionality or change the number of events, you need to set it yourself in your macro file.

<!-- SyntaxHighlightingPlugin -->
mo.setStartFrom(10)
<!-- end SyntaxHighlightingPlugin -->
Allows to define the starting event in the input file.

SLIC

The interface of SLIC is similar to that of Mokka, and is described as usual in the API documentation.

It is invoked with

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import SLIC
slic = SLIC()
slic.setVersion('v2r9p8')
slic.setInputFile("some_file.stdhep")
slic.setSteeringFile('MyMacro.mac')
slic.setDetectorModel('clic_sid_cdr')
slic.setOutputFile("Something.slcio")
res = j.append(slic)
<!-- end SyntaxHighlightingPlugin -->

Similarly to Mokka it has a setStartFrom(N) method that allows skipping the first N events in the input file.

The detector model handling is worth presenting: it is the radical of a zip file, in this case it would be clic_sid_cdr.zip. The zip file can live either locally, as an lfn, or as a standard detector description on the lcsim.org web portal. It must be a zip file and the detector it describes must be called like the zip file but without the .zip.

There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.

Reconstruction

Marlin

The runs the reconstruction in the context of the ILD detector concept. The API is as usual, described in the API documentation.
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import Marlin
ma = Marlin()
ma.setVersion('v0111Prod')
ma.setInputFile("something.slcio")
#job.setInputData(["something.slcio"]) #add this line if the file is on a storage element with tape backend (e.g., CERN)
ma.setSteeringFile("SomeSteeringFile.xml")
ma.setGearFile("SomeGear.gear")
ma.setRequired(["data","myextrafile"])
ma.setOutputRecFile("MyRECfile.slcio")
ma.setOutputDstFile("myDSTfile.slcio")
res = j.append(ma)
<!-- end SyntaxHighlightingPlugin -->

Check the API documentation for 2 extra methods that can be useful.

If you want to run with your own libs (LD libs and/or MARLIN_DLL), the lib directory MUST have the following structure:

  • The LD libs must go under lib/lddlib/. It is recommended to put the versioned libraries here as well, i.e., something like libUser.so, as well as libUser.so.5.7
  • The Processors MUST be under lib/marlin_dll/
  • Any Marlin DLL must end on .so (not .so.xyz)

This comes from the fact that Marlin is sensitive to the difference between a Processor and a library.

LCSIM

LCSIM is now used mostly to run the tracking and create the final files for the SID detector concept. The PFA is ran as another application, SLICPandora, described later. The example below is specific to such a use case (most likely use case any way)
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import LCSIM
lcsim_prepandora = LCSIM()
lcsim_prepandora.setVersion('CLIC_CDR')
lcsim_prepandora.setSteeringFile("clic_cdr_prePandora.lcsim")
lcsim_prepandora.setTrackingStrategy("defaultStrategies_clic_sid_cdr.xml")
lcsim_prepandora.setOutputFile("prePandora.slcio")
lcsim_prepandora.willRunSLICPandora()
res = j.append(lcsim_prepandora)
<!-- end SyntaxHighlightingPlugin -->
There are a few other options that are described in the API documentation.

There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.

SLICPandora

This application (described more in the API documentation is usually used in conjunction with LCSIM and the example below shows such a use case.
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import SLICPandora
slicpandora = SLICPandora()
slicpandora.setVersion('CLIC_CDR')
slicpandora.setDetectorModel('clic_sid_cdr')
slicpandora.setPandoraSettings("PandoraSettingsSlic.xml")
slicpandora.getInputFromApp(lcsim_prepandora)
slicpandora.setOutputFile('pandora.slcio')
res = j.append(slicpandora)
<!-- end SyntaxHighlightingPlugin -->
>
>
 
Changed:
<
<
It is of course posible to use it as standalone.

One noticable aspect if the detector model handling. It's identical to the one of SLIC: the detector model must be e.g. detectormodel where it's described in a detectormodel.zip living either locally, as an lfn, or in the lcsim.org portal.

There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.

Adding the Overlay

As the production system is non deterministic in the sense that it does not know a priori the files it will run on, the overlay files have to be determined at the last minute. This is particularly needed as in the CLIC case, we had many interactions per bunch crossings so the number of events to get was very large, and could vary from job to job. So for that, there is another application to add before marlin or LCSIM, called OverlayInput. This application does not have an OutputFile so don't try to use ma.getInputFromApp(ov) as it would fail.

It requires a few parameters like the number of interactions per bunch crossing, the detector model, etc. Those are described in the API documentation, and the example below shows how to configure it.

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import OverlayInput
overlay = OverlayInput()
overlay.setMachine("clic_cdr")
overlay.setEnergy(3000.0)
overlay.setBXOverlay(60)
overlay.setGGToHadInt(3.2)##When running at 3TeV
overlay.setDetectorModel("CLIC_ILD_CDR")
overlay.setNbSigEvtsPerJob(10)
res = j.append(overlay)
<!-- end SyntaxHighlightingPlugin -->

In any case, how it works:

  • Given a signal input file (its number of events to be exact), the interaction probability, and the number of bunch crossing to overlay, it computes the number of background events to fetch.
  • Gets from the configuration the ProdID that is suitable for that sample of background given the machine, the energy and the detector model, and the number of events per bkg file. You can see an example of this under /Operations/Defaults/Overlay/ilc_dbd/energy/detectormodel/bkgtype/ in the Configuration Service.
  • Fetch the list of files from the catalog with that ProdID and other tags (detector model, energy, machine, etc.)
  • Get the appropriate number of files given the number of events per background files. The files are then chosen randomly.
  • Stores those under a special directory
  • When Marlin/LCSIM starts, the code checks that the steering XML references a bgoverlay or overlaytiming or org.lcsim.util.OverlayDriver section and then sets the files accordingly.

ROOT applications

Other applications

GetSRMFile

Example to use GetSRMFile:
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import GetSRMFile
srm = GetSRMFile()
fdict={"file" : "srm://dcache-se-desy.desy.de/pnfs/desy.de/calice/generated/2011-10-04/file885bf7d6-f911-4169-8124-097379c42c0f", "site" : "DESY-SRM"}
srm.setFiles(fdict)
srm.setDebug(True)
res = job.append(srm)
[...]
ma = Marlin()
#either this or the line after can be used
#ma.setInputFile("file885bf7d6-f911-4169-8124-097379c42c0f")
ma.getInputFromApp(srm)
job.append(ma)
[...]
<!-- end SyntaxHighlightingPlugin -->

When using GetSRMFile with Mokka, make sure that you let dirac create the macro file so that the proper path to the input file can be generated.

SLCIOSplit

SLCIOConcatenate

StdHepSplit

CheckCollections

Storage Elements

>
>

Storage Elements

 The following storage elements can be used as output location:
CERN-SRM (default)
Line: 494 to 35
  We recommend you use a storage element that's close enough geographically to you.
Changed:
<
<

Specifying your libraries

Libraries also have dedicated support. This mostly depends on the application, in particular for Marlin, so here I show only the default method, but please check if the application you want to run for specificities.

All applications come with their own dependencies, so a user does not need to take care of those. He should only take care of those libraries that are not part of the software stack.

>
>

Available Software Versions

 
Changed:
<
<
Let's say you have an application that depends in libSomething.so that is not a default library (in the Marlin case, one case replace existing processors, so this is covered in the Marlin section).

You will copy this libSomething.so into a lib directory. Now, 2 solutions are possible:

1. You are planning on submitting many jobs: I recommend you to put that lib directory on the grid and specify the LFN in the job definition. How to do that?

<!-- SyntaxHighlightingPlugin -->
tar czf lib.tar.gz lib/
dirac-dms-add-files /ilc/user/i/initial/some/path/lib.tar.gz lib.tar.gz CERN-SRM
<!-- end SyntaxHighlightingPlugin -->
The command dirac-dms-add-files will disappear in the next DIRAC release and will be named dirac-dms-add-file. The /ilc/user/... is the LFN (Logical File Name). The i/initial/ part is user specific: you need to use your own DIRAC user name. You can find it in the dirac-proxy-init output, check for username. The some/path/ part is free to you, you can use whatever you want. There are a few limitations: you can not have a total number of subsequent directories greater than 14, and the final file name (here lib.tar.gz) cannot be longer than 128 chars. The last element, CERN-SRM indicates the logical name of the Storage Element on which you wish to upload you file.
>
>
Use the script: dirac-ilc-show-software that dumps the current available software. Please look at this website: DiracAvailableSoftware
 
Deleted:
<
<
Warning, important When running with the CALICE VO, the path has a different beginning: /calice/users/i/initial. Notice the s at users. Also, CERN-SRM is not a valid storage element for CALICE users, so DESY-SRM or IN2P3-SRM must be prefered.

This LFN is registered in the DIRAC File Catalog, so it can now be used in the job definition.

If you wish to replace the file, you cannot overwrite the file, you need first to issue a dirac-dms-remove-files /ilc/user/i/initial/some/path/lib.tar.gz then re upload.

Moving a file cannot be done on the GRID: if really needed, you need to get the file (dirac-dms-get-file /ilc/...) then remove it from the GRID (same as above), then re upload it to the new location.

You now have a file on the GRID that you wish to input for a series of jobs. You would use the following:

<!-- SyntaxHighlightingPlugin -->
job.setInputSandbox("LFN:/ilc/user/i/initial/some/path/lib.tar.gz")
<!-- end SyntaxHighlightingPlugin -->
Notice the LFN: part that is used by DIRAC to identify the files that must be downloaded from the grid. If you omit it, the job submission should fail because an input sandbox file will be missing. The fact that it's a tar ball does not matter, it will be untarred automatically.

2. You have only 1 job: Simply input the lib directory like this

<!-- SyntaxHighlightingPlugin -->
job.setInputSandbox("lib")
<!-- end SyntaxHighlightingPlugin -->
If you have a tar ball, you may as well use it:
<!-- SyntaxHighlightingPlugin -->
job.setInputSandbox("lib.tar.gz")
<!-- end SyntaxHighlightingPlugin -->
as any tar ball is automatically untarred in the job running directory.

The reason for the different behavior between the 2 cases is because every job submitted uploads its sandbox to the main DIRAC servers, filling its hard drive. All files are sent but those that start with LFN:.

Warning, important In case you are using setRequired, you do not need to specify lib there as the directory will always be copied.

 
Added:
>
>

Specifying your libraries

 
Changed:
<
<

Getting the output

>
>

Getting the output

  There are several ways of getting the job outputs: if you have 1 job, if you have a job repository, and if you use OutputData, things are different. In case of 1 job, use %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Line: 557 to 62
  Warning, important The maximum size of an output sandbox is 10Mb. If the sum of the file sizes in there is bigger, the sandbox is not sent to the DIRAC sandbox store, but they are added to an archive send to the GRID. The Storage Element chosen depends on the site (usually a SE that's close to the site where the job ran). In this case, the retrieval from the web portal will fail with an error (missing sandbox), but the command line work as it retrieves the sandbox automatically from the GRID (not possible via the portal).
Changed:
<
<

Monitoring the jobs

>
>

Monitoring the jobs, Webinterface

 
Added:
>
>
The web monitoring of jobs happen here: https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/jobs/JobMonitor/display
 Connect to the ILCDIRAC monitoring page and you'll get something like the image below.

webMonitoring.png

Line: 585 to 91
 %ENDSYNTAX%
Deleted:
<
<

Web monitoring

The web monitoring of jobs happen here: https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/jobs/JobMonitor/display

 

F.A.Q.

I keep getting Could not verify credential errors

Line: 608 to 110
 

How do upload files to the Grid?

Have a look at the "Data Management" DIRAC-Tutorials.
Added:
>
>

How do I use this wonderful system?

Well, start by looking at the documentation at DiracForUsers and DiracByExample. Then contact ilcdirac-register@cernNOSPAMPLEASE.ch for more info.

I need to use my own processors

It's fully taken in account in dirac. For that, you'll need to compile them against a version that dirac knows. And we defined a directory containing those version on afs under /afs/cern.ch/eng/clic/software/x86_64-slc5-gcc41/.

So simply setup the env, use cmake including the ilcsoft.cmake in the directory of your choice from the available ones, and put your processor/libraries in the proper directories as mentionned in the HeadFirstTalk above. And normally you should be going. We try to keep the version up to date, so you should be able to find always the latest release of ILCSOFT. We can also define new versions if needed, as long as one admin has access to the software binary location.

I want to specify an input file using for e.g. srm://srm-ilc.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/ ?

You need to use the GetSRMFile class of ILCDIRAC as first job application. See DiracForUsers for more details, and the ilcdiracdoc page, UserJob and GetSRMFile specifically.

Running behind a proxy?

At RAL for example. I don't know what that changes. Need Marcel Stanitzki's advice.

Some commands in the web interface don't work?

The Get LogFile method is reserved to production jobs. It's never set for user jobs. It's normally pointing to a web location. To get the logs, make sure you put *.log in the OutputSandbox parameter of the UserJob.

How do I specify data coming from the Lcg File Catalog?

Most data is now copied in the DIRAC File Catalog browsable using dirac-dms-filecatalog-cli. A web view is also available (check the data menu in the interface). In case the file you are looking for is not in the DFC, you may use the GetSRMFile feature (above) given that you have some replica information..

How do I get the list of files that were uploaded on a SE by my jobs?

dirac-repo-create-lfn-list repository.rep
This will print on screen the list of files for each job, so you would probably want to redirect the output to a text file.

That command might take some time, depending on how many jobs there are.

How do I retrieve the uploaded output data of my jobs?

dirac-repo-retrieve-jobs-output-data repository.rep

How to check for replicas of a file

dirac-dms-lfn-replicas /ilc/...

Only works with files added in the DFC.

How to replicate a file

dirac-dms-replicate-lfn /ilc/your/file CERN-SRM 

CERN-SRM is one of the storage elements configured. You are free to use CERN-SRM, IN2P3-SRM, RAL-SRM, IMPERIAL-SRM, DESY-SRM (not for prod files), TAU-SRM (does not always work), FNAL-SRM (need to check availability). New storage elements will be made available in the future.

 
META FILEATTACHMENT attachment="webMonitoring.png" attr="" comment="" date="1365507538" name="webMonitoring.png" path="webMonitoring.png" size="431203" user="sposs" version="1"

Revision 492014-10-29 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 15 to 15
 
Added:
>
>
 

ILC specific jobs

This section shows the preferred way of creating jobs in ILCDIRAC. This uses the PYTHON API. To use it, some basic knowledge of object oriented (OO) programming is needed, as PYTHON is a completely OO language. The examples shown below use the API described in

Revision 482014-08-05 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 464 to 464
 res = job.append(srm) [...] ma = Marlin()
Added:
>
>
#either this or the line after can be used
 #ma.setInputFile("file885bf7d6-f911-4169-8124-097379c42c0f") ma.getInputFromApp(srm) job.append(ma) [...] %ENDSYNTAX%
Added:
>
>
When using GetSRMFile with Mokka, make sure that you let dirac create the macro file so that the proper path to the input file can be generated.
 

SLCIOSplit

SLCIOConcatenate

StdHepSplit

Revision 452014-03-26 - PhilippRoloff

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 172 to 172
 

setRequired why and how?

Changed:
<
<
This information in this section are not relevant at the moment. The setRequired method is currently not needed (March 25, 2014).
>
>
The information in this section are not relevant at the moment. The setRequired method is currently not needed (March 25, 2014).
  The setRequired method is used to specify files and/or directories that must be local to the execution. The reason it's needed is due to several required changes. In the past all the applications were running in the same directory. This made file handling simple, but limited. In particular, the steering files handling was cumbersome as a user could not use a different version than the default provided with the application. To add flexibility, it was decided that all applications would run in their own directories, therefore preventing overwrite of steering files in case a user specifies a different version than the default. If it adds flexibility, it also makes file management trickier: data files need to be moved from directories to others, special files (steering files and e.g. gear files) need copying, etc. The handling of specific files was implemented (those that must be explicitely defined for the application to run, like the steering file). Remained those that do not have explicit setters such as the pandora settings or the lcfi weights. For those, the setRequired method is added.

Revision 442014-03-25 - PhilippRoloff

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 172 to 172
 

setRequired why and how?

Changed:
<
<
The setRequired method is used to specify files and/or directories that must be local to the execution. The reason it's needed now (as of Nov 21st, 2013) is due to several required changes. In the past all the applications were running in the same directory. This made file handling simple, but limited. In particular, the steering files handling was cumbersome as a user could not use a different version than the default provided with the application. To add flexibility, it was decided that all applications would run in their own directories, therefore preventing overwrite of steering files in case a user specifies a different version than the default. If it adds flexibility, it also makes file management trickier: data files need to be moved from directories to others, special files (steering files and e.g. gear files) need copying, etc. The handling of specific files was implemented (those that must be explicitely defined for the application to run, like the steering file). Remained those that do not have explicit setters such as the pandora settings or the lcfi weights. For those, the setRequired method is added.
>
>
This information in this section are not relevant at the moment. The setRequired method is currently not needed (March 25, 2014).

The setRequired method is used to specify files and/or directories that must be local to the execution. The reason it's needed is due to several required changes. In the past all the applications were running in the same directory. This made file handling simple, but limited. In particular, the steering files handling was cumbersome as a user could not use a different version than the default provided with the application. To add flexibility, it was decided that all applications would run in their own directories, therefore preventing overwrite of steering files in case a user specifies a different version than the default. If it adds flexibility, it also makes file management trickier: data files need to be moved from directories to others, special files (steering files and e.g. gear files) need copying, etc. The handling of specific files was implemented (those that must be explicitely defined for the application to run, like the steering file). Remained those that do not have explicit setters such as the pandora settings or the lcfi weights. For those, the setRequired method is added.

  The usage is as follows:

Revision 422014-02-21 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 8 to 8
 
Added:
>
>

File access, downloading, uploading etc. with the DiracFileCatalog

First read the "Data Management" DIRAC-Tutorials.

 

ILC specific jobs

This section shows the preferred way of creating jobs in ILCDIRAC. This uses the PYTHON API. To use it, some basic knowledge of object oriented (OO) programming is needed, as PYTHON is a completely OO language. The examples shown below use the API described in http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/.
Line: 566 to 574
 %ENDSYNTAX%
Deleted:
<
<

Locating Files on the Grid

All files produced by either ILD, SID or CLIC are stored in the File Catalog. This allows for several data operations like searching for files, getting them, removing them, etc. There are several interfaces available: The Command Line Utility, the web interface, the Python API, and a script.

The Command Line Interface

It is invoked using
<!-- SyntaxHighlightingPlugin -->
dirac-dms-filecatalog-cli
<!-- end SyntaxHighlightingPlugin -->

Once started you'll get a shell that looks like this:

<!-- SyntaxHighlightingPlugin -->
Starting FileCatalog client

File Catalog Client $Revision: 1.41 $Date: 
            
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

Here, you can get help by typing help:

<!-- SyntaxHighlightingPlugin -->
FC:/> help
Documented commands (type help <topic>):
========================================
add          chgrp       exit   guid  meta     register   rm         stats     
ancestor     chmod       find   id    mkdir    repair     rmdir      unregister
ancestorset  chown       get    lcd   pwd      replicas   rmreplica  user      
cd           descendent  group  ls    rebuild  replicate  size     

Undocumented commands:
======================
help

FC:/> 
<!-- end SyntaxHighlightingPlugin -->

As you can see, you can get help with

<!-- SyntaxHighlightingPlugin -->
FC:/> help meta                                                                                                                                                     
 Metadata related operations
    
        Usage:
          meta index [-d|-f|-r] <metaname> [<metatype>]  - add new metadata index. Possible types are:
                                                           'int', 'float', 'string', 'date';
                                                         -d  directory metadata
                                                         -f  file metadata
                                                         -r  remove the specified metadata index
          meta set <path> <metaname> <metavalue> - set metadata value for directory or file
          meta remove <path> <metaname>  - remove metadata value for directory or file
          meta get [-e] [<path>] - get metadata for the given directory or file
          meta tags <path> <metaname> where <meta_selection> - get values (tags) of the given metaname compatible with
                                                        the metadata selection
          meta show - show all defined metadata indice
    
    
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

Warning, importantSome commands are considered unsafe like the rm and rmdir and should not be used. As this system is being improved, this will be made safe.

From the help message above, one can already see what can be done with data manipulation. In particular, it's possible to set searchable index using the first command. I do not recommend this to everyone, as it should be left to production managers or service managers. It is also possible to set meta data values for directories and files using meta set. If a metadata is not part of the searchable fields, it will be set as documentation. Getting the searchable fields is done with

<!-- SyntaxHighlightingPlugin -->
FC:/> meta show
      FileMetaFields : {'PolarizationB2': 'VARCHAR(128)', 'GenProcessID': 'INT', 'BeamParticle1': 'VARCHAR(128)',
 'BeamParticle2': 'VARCHAR(128)', 'PolarizationB1': 'VARCHAR(128)'}
 DirectoryMetaFields : {'EvtType': 'VARCHAR(128)', 'NumberOfEvents': 'int', 'BXoverlayed': 'int', 'Polarisation': 'VARCHAR(128)', 
'Datatype': 'VARCHAR(128)', 'Energy': 'VARCHAR(128)', 'MachineParams': 'VARCHAR(128)', 'DetectorType': 'VARCHAR(128)', 
'Machine': 'VARCHAR(128)', 'EvtClass': 'VARCHAR(128)', 'Owner': 'VARCHAR(128)', 'SoftwareTag': 'VARCHAR(128)', 
'DetectorModel': 'VARCHAR(128)', 'JobType': 'VARCHAR(128)', 'ProdID': 'int'}
<!-- end SyntaxHighlightingPlugin -->
The output is a bit ugly but will be improved.

It is also possible to get all values for a meta tag (Warning, importantfor directory level tags only for the moment) using

<!-- SyntaxHighlightingPlugin -->
FC:/> meta tags /ilc EvtClass                          
Possible values for EvtClass:
1f
1f_3f
2f
2f_Z_bhabhag
2f_Z_hadronic
2f_Z_leptonic
3f
4f
4f_singleW_leptonic
4f_singleW_semileptonic
4f_singleZee_leptonic
4f_singleZee_semileptonic
4f_singleZnunu_leptonic
4f_singleZnunu_semileptonic
4f_singleZsingleWMix_leptonic
4f_WW_hadronic
4f_WW_leptonic
4f_WW_semileptonic
4f_ZZWWMix_hadronic
4f_ZZWWMix_leptonic
4f_ZZ_hadronic
4f_ZZ_leptonic
4f_ZZ_semileptonic
5f
6f
6f_eeWW
6f_llWW
6f_ttbar
6f_vvWW
6f_xxWW
6f_xxxxZ
6f_yyyyZ
aa_2f
aa_4f
aa_lowpt
aa_minijet
higgs
higgs_ffffh
higgs_ffh
ttbb-all-all
tth
tth-2l2nbb-hbb
tth-2l2nbb-hnonbb
tth-6q-hbb
tth-6q-hnonbb
tth-ln4q-hbb
tth-ln4q-hnonbb
ttz-all-all
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

It's possible to get the possible values of a tag given other tags:

<!-- SyntaxHighlightingPlugin -->
FC:/> meta tags SoftwareTag where EvtClass=tth-2l2nbb-hnonbb Datatype=DST
Possible values for SoftwareTag:
v01-16-p03
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

Finally, it's possible to search for files using find. The syntax is similar to the usual shell "find":

<!-- SyntaxHighlightingPlugin -->
FC:/> find . EvtClass=higgs_ffh Datatype=DST-MERGED
Query: {'Datatype': 'DST-MERGED', 'EvtClass': 'higgs_ffh'}
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00003-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00004-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00005-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00006-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00007-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00008-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00009-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00010-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00011-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36152.Pvvh_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36152.Pvvh_nomu.eR.pL-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36154.Pe1e1h_nomu.eL.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36155.Pe1e1h_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36156.Pe1e1h_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36157.Pe1e1h_nomu.eR.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36159.Pllh_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36160.Pllh_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36163.Pxxh_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36164.Pxxh_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36167.Pyyh_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36168.Pyyh_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37582.Pvvh_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37582.Pvvh_mumu.eL.pR-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37583.Pvvh_mumu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37583.Pvvh_mumu.eR.pL-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37585.Pe1e1h_mumu.eL.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37586.Pe1e1h_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37587.Pe1e1h_mumu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37590.Pllh_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37591.Pllh_mumu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37594.Pxxh_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37595.Pxxh_mumu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37598.Pyyh_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37599.Pyyh_mumu.eR.pL-00001-DST.slcio
QueryTime 0.06 sec
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

The . indicates that one as to look for compatible files in the current directory, and can be replaced by a path, e.g /ilc/prod/ilc/mc-dbd/ild to make sure the files only come from this directory and its sub-directories.

It's always possible to have access to the defined meta data of a directory:

<!-- SyntaxHighlightingPlugin -->
FC:/> meta get /ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/                                             
           *Datatype : DST-MERGED
             *Energy : 1tev
      *MachineParams : B1b_ws
       *DetectorType : ILD
            *Machine : ilc
           *EvtClass : higgs_ffh
        !SoftwareTag : v01-16-p03
      *DetectorModel : ILD_o1_v05
FC:/> 
<!-- end SyntaxHighlightingPlugin -->
Where * indicate inherited meta data and ! mean local meta data. Meta data that have no indication are non searchable.

Of course, getting the meta data of a file is possible:

<!-- SyntaxHighlightingPlugin -->
FC:/> meta get /ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
      NumberOfEvents : 1000
          RandomSeed : None
       BeamParticle1 : e1
       BeamParticle2 : E1
        GenProcessID : 37594
          StartEvent : 0
      PolarizationB1 : L
      PolarizationB2 : R
FC:/> 
<!-- end SyntaxHighlightingPlugin -->
Here, there are no indication (for the moment) of the nature of the meta data (searchable or not) and the user should rely on the meta show output.

A last functionality worth advertising here is the ancestor/daughter relationships. For this, the following examples should be enough.

<!-- SyntaxHighlightingPlugin -->
FC:/> ancestor /ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio 
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
1       /ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00003-DST.slcio
1       /ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00002-DST.slcio
1       /ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
<!-- end SyntaxHighlightingPlugin -->
This gives the direct parents.

The descendants of a file can be also obtained:

<!-- SyntaxHighlightingPlugin -->
FC:/> descendent /ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00002-DST.slcio
1       /ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
<!-- end SyntaxHighlightingPlugin -->

There are other useful utilities in this interface that should be looked upon, but the help is enough to grasp the concepts. In particular, users can get their files using get.

The web portal

There is an interface to the File Catalog on the DIRAC web portal at https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/data/MetaCatalog/display but it's still in development and does not allow easy maipulation of data as the CLI permits. It's still possible to interact with it to get file informations.

This interface will be more detailed once it's fully functionnal. Until then, people should get the files using

The Python API

This interface uses directly the underlying python API to get the information. It's to be used when submitting jobs as it's the easiest way of passign the files from the catalog to the job definition. I only demonstrate here the mean to perform a data query.
<!-- SyntaxHighlightingPlugin -->
from DIRAC.Core.Base import Script
Script.parseCommandLine()
from DIRAC.Resources.Catalog.FileCatalogClient import FileCatalogClient

fc = FileCatalogClient()

meta = {}
meta['EvtClass']='higgs_ffh' 
meta['Datatype']='DST-MERGED'

res = fc.findFilesByMetadata(meta)
if not res['OK']:
   print res['Message']

lfns = res['Value']

print "Found %s files" % len(lfns)
for lfn in lfns:
   print lfn
<!-- end SyntaxHighlightingPlugin -->
This will print on screen the result of the meta data query indicated by the dictionary meta. This is typically used when defining a job: its setInputSandbox would contain "LFN:"+lfn to tell DIRAC to get the file before running the job.

The dirac-ilc-find-in-FC script

This is a utility I designed to allow dumping the result of a meta data query to a text file easily, to be used as long as the web portal does not behave the right way. The syntax is the following:

<!-- SyntaxHighlightingPlugin -->
dirac-ilc-find-in-FC /ilc/prod/ EvtClass=higgs_ffh Datatype=DST-MERGED
<!-- end SyntaxHighlightingPlugin -->
The output of this can be redirected to a text file for easy manipulation later.
 

Web monitoring

The web monitoring of jobs happen here: https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/jobs/JobMonitor/display

Revision 412014-02-21 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Deleted:
<
<

Getting the certificate, getting registered in ILCDIRAC

 
Changed:
<
<
Get a grid certificate from your preferred certification authority. At CERN (for staff and users) that is ca.cern.ch. See your local grid expert (there is usually at least one GRID user in every lab nowadays) for the name of the certification authority corresponding to your institute.

Then, depending on your VO, either go to

https://grid-voms.desy.de:8443/voms/ilc

or

https://grid-voms.desy.de:8443/voms/calice

or both to register yourself as a VO member.

Once you have been allowed in the VO, you need to send a mail to ilcdirac-register@cernNOSPAMPLEASE.ch stating your name, institute, corresponding contact (supervisor is usually enough) and project you are working on. We need this for security reasons. Then we will register you in ILCDIRAC as a user, and add you to the ilc-dirac@cernNOSPAMPLEASE.ch mailing list, and send you a confirmation mail. In the process, you will get a directory in the File Catalog, see below.

After that, you are permanently a member of ILCDIRAC, except if you wish to leave or if you change your certification authority (when you change lab) in which case this registration procedure has to be done again, but is simpler as we only need to change your DN in ILCDIRAC registration. For that, send a mail to ilcdirac-register@cernNOSPAMPLEASE.ch announcing you changed your affiliation to another lab and we will update your details.

Once registered in the VO and in DIRAC, you need to create the pem files. For this, export your certificate from your browser in p12 format. How to do this is documented in the HeadFirstTalk slides. This p12 will need to be converted. The DIRAC installation comes with a handy script to convert the p12 in pem format. To get this script, you need either to see the Installing DIRAC section up to the dirac-proxy-init -x or source the bashrc file from your existing local DIRAC installation. Then run

<!-- SyntaxHighlightingPlugin -->
dirac-cert-convert.sh cert.p12
<!-- end SyntaxHighlightingPlugin -->

where cert.p12 is the file you obtained from the browser export.

Getting a DIRAC environment

Warning, important The DIRAC environment is usually incompatible with other environments, and therefore often requires a dedicated terminal/shell.

Using a pre existing installation (e.g. at CERN)

If you rely on someone's installation (or your own) you have normally access to the directory in which ILCDIRAC has been installed. In that directory there is a env file, bashrc or cshrc depending on your shell, that needs sourcing to get the right environment. For example, for CERN users, you can run
<!-- SyntaxHighlightingPlugin -->
# bash users
source /afs/cern.ch/eng/clic/software/DIRAC/bashrc
# (t)csh users
source /afs/cern.ch/eng/clic/software/DIRAC/cshrc
<!-- end SyntaxHighlightingPlugin -->
The default setup is for SLC6. There are also environment scripts for SLC5 available:
<!-- SyntaxHighlightingPlugin -->
# bash users
source /afs/cern.ch/eng/clic/software/DIRAC/bashrc-glibc-2.5
# (t)csh users
source /afs/cern.ch/eng/clic/software/DIRAC/cshrc-glibc-2.5
<!-- end SyntaxHighlightingPlugin -->

Once this file has been sourced, you get access to all the DIRAC and ILCDIRAC commands, as well as the python API. You can proceed to the Job section.

Installing ILCDIRAC

When not having access to a pre installed DIRAC, you need to install it yourself. The procedure is as follows:
<!-- SyntaxHighlightingPlugin -->
wget -np -O dirac-install \
https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py \
--no-check-certificate
chmod +x dirac-install
./dirac-install -V ILCDIRAC
<!-- end SyntaxHighlightingPlugin -->
This also creates the bashrc / cshrc file needed to get the DIRAC environment that you obtain with
<!-- SyntaxHighlightingPlugin -->
# bash users
source bashrc
# (t)csh users
source cshrc
<!-- end SyntaxHighlightingPlugin -->

Then you need to configure DIRAC (make the client know where to get the services from). This requires the presence of a valid grid proxy. First, if you don't have the pem files, you can get them with:

<!-- SyntaxHighlightingPlugin -->
dirac-cert-convert.sh grid.p12
<!-- end SyntaxHighlightingPlugin -->
where grid.p12 is the file you got from the web browser export. If you already have the pem files, this can be skipped.

To get a valid proxy, run

<!-- SyntaxHighlightingPlugin -->
dirac-proxy-init -x
<!-- end SyntaxHighlightingPlugin -->
This is not a DIRAC proxy. A DIRAC proxy is obtained by omitting the -x.

Then run the configuration

<!-- SyntaxHighlightingPlugin -->
dirac-configure -S ILC-Production -C dips://volcd05.cern.ch:9135/Configuration/Server \
--SkipCAChecks
<!-- end SyntaxHighlightingPlugin -->

In principle, if this commands runs fine, you should be able to got to the next section.

Don't forget to source the bashrc / cshrc file whenever using DIRAC.

Updating ILCDIRAC

In the $DIRAC directory, run dirac-install -V ILCDIRAC. That's it.

Getting a proxy

Once you have sourced the DIRAC environment file, you can run
<!-- SyntaxHighlightingPlugin -->
dirac-proxy-init -g ilc_user
<!-- end SyntaxHighlightingPlugin -->
or
<!-- SyntaxHighlightingPlugin -->
dirac-proxy-init -g calice_user
<!-- end SyntaxHighlightingPlugin -->
depending on the VO you want a proxy for. The 2 VOs having access to different resources, it's sometimes needed to get the right proxy (in particular for data access). Also, this has an effect on the grid sites on which one can run, for instance, only the DESY-HH and IPNL sites are available to the CALICE users (in ILCDIRAC, and for the moment).

Your first DIRAC job

This section is a copy of what is presented in https://github.com/DIRACGrid/DIRAC/wiki/JobManagement

The default way of creating jobs in DIRAC is to use the same method as regular GRID jobs, with JDLs. A JDL (or Job Description Language) follows a standard. It's a simple composition of Attribute=Value;. For example, the following is a correctly defined JDL:

<!-- SyntaxHighlightingPlugin -->
[
  JobName = "Simple_Job";
  Executable = "/bin/ls";
  Arguments = "-ltr";
  StdOutput = "StdOut";
  StdError = "StdErr";
  OutputSandbox = {"StdOut","StdErr"};
]
<!-- end SyntaxHighlightingPlugin -->
It has a few mandatory parameters indicated above. Many other are possible, but are not indicated here as this is not the main supported way of submitting jobs. Nevertheless if you have a JDL, you may submit it to DIRAC using
<!-- SyntaxHighlightingPlugin -->
dirac-wms-job-submit myjdl.jdl
<!-- end SyntaxHighlightingPlugin -->
This will return you a job ID. If using that method of submission, you may want to store those job IDs in some text file as they are needed to obtain their output.
>
>

Getting Started: Registration, Installation, etc.

 

ILC specific jobs

This section shows the preferred way of creating jobs in ILCDIRAC. This uses the PYTHON API. To use it, some basic knowledge of object oriented (OO) programming is needed, as PYTHON is a completely OO language. The examples shown below use the API described in
Line: 695 to 580
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.39 $Date:
>
>
File Catalog Client $Revision: 1.41 $Date:
  FC:/> %ENDSYNTAX%

Revision 402014-02-21 - JanStrube

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 489 to 489
 Check the API documentation for 2 extra methods that can be useful.

If you want to run with your own libs (LD libs and/or MARLIN_DLL), the lib directory MUST have the following structure:

Changed:
<
<
  • The LD libs must go under lib/lddlib/
>
>
  • The LD libs must go under lib/lddlib/. It is recommended to put the versioned libraries here as well, i.e., something like libUser.so, as well as libUser.so.5.7
 
  • The Processors MUST be under lib/marlin_dll/
  • Any Marlin DLL must end on .so (not .so.xyz)
Line: 695 to 695
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.38 $Date:
>
>
File Catalog Client $Revision: 1.39 $Date:
  FC:/> %ENDSYNTAX%

Revision 382014-01-28 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 695 to 695
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.36 $Date:
>
>
File Catalog Client $Revision: 1.38 $Date:
  FC:/> %ENDSYNTAX%
Line: 970 to 970
  This means that the local copy of the certificate revocation list (CRL) expired. Please run dirac-admin-get-CAs to update you copy, or if you don;t have the right to run the command yourself, ask your local dirac administrator.
Added:
>
>

How do upload files to the Grid?

Have a look at the "Data Management" DIRAC-Tutorials.
 
META FILEATTACHMENT attachment="webMonitoring.png" attr="" comment="" date="1365507538" name="webMonitoring.png" path="webMonitoring.png" size="431203" user="sposs" version="1"

Revision 372014-01-21 - ChristianGrefe

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 490 to 490
  If you want to run with your own libs (LD libs and/or MARLIN_DLL), the lib directory MUST have the following structure:
  • The LD libs must go under lib/lddlib/
Changed:
<
<
  • the Processors MUST be under lib/marlin_dll/
>
>
  • The Processors MUST be under lib/marlin_dll/
  • Any Marlin DLL must end on .so (not .so.xyz)
  This comes from the fact that Marlin is sensitive to the difference between a Processor and a library.
Line: 694 to 695
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.35 $Date:
>
>
File Catalog Client $Revision: 1.36 $Date:
  FC:/> %ENDSYNTAX%

Revision 362013-11-27 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 650 to 650
  Finally, you can also download the job output from the web portal by left-clicking on a job, Sandobx, Get output file(s). You can also get the job input that way.
Added:
>
>
Warning, important The maximum size of an output sandbox is 10Mb. If the sum of the file sizes in there is bigger, the sandbox is not sent to the DIRAC sandbox store, but they are added to an archive send to the GRID. The Storage Element chosen depends on the site (usually a SE that's close to the site where the job ran). In this case, the retrieval from the web portal will fail with an error (missing sandbox), but the command line work as it retrieves the sandbox automatically from the GRID (not possible via the portal).
 

Monitoring the jobs

Connect to the ILCDIRAC monitoring page and you'll get something like the image below.

Line: 692 to 694
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.34 $Date:
>
>
File Catalog Client $Revision: 1.35 $Date:
  FC:/> %ENDSYNTAX%

Revision 352013-11-25 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 557 to 557
 

Other applications

GetSRMFile

Added:
>
>
Example to use GetSRMFile:
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import GetSRMFile
srm = GetSRMFile()
fdict={"file" : " srm://dcache-se-desy.desy.de/pnfs/desy.de/calice/generated/2 011-10-04/file885bf7d6-f911-4169-8124-097379c42c0f", "site" : "DESY-SRM"}
srm.setFiles(fdict)
srm.setDebug(True)
res = job.append(srm)
[...]
ma = Marlin()
ma.setInputFile("file885bf7d6-f911-4169-8124-097379c42c0f")
<!-- end SyntaxHighlightingPlugin -->
 

SLCIOSplit

SLCIOConcatenate

StdHepSplit

Revision 342013-11-21 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 277 to 277
 %ENDSYNTAX% This last property does not make sense for all applications, in particular for the user application, as DIRAC cannot "guess" what a random binary will produce. If specified in the application steering file AND in the setOutputFile field, then it will behave as expected.
Added:
>
>

setRequired why and how?

The setRequired method is used to specify files and/or directories that must be local to the execution. The reason it's needed now (as of Nov 21st, 2013) is due to several required changes. In the past all the applications were running in the same directory. This made file handling simple, but limited. In particular, the steering files handling was cumbersome as a user could not use a different version than the default provided with the application. To add flexibility, it was decided that all applications would run in their own directories, therefore preventing overwrite of steering files in case a user specifies a different version than the default. If it adds flexibility, it also makes file management trickier: data files need to be moved from directories to others, special files (steering files and e.g. gear files) need copying, etc. The handling of specific files was implemented (those that must be explicitely defined for the application to run, like the steering file). Remained those that do not have explicit setters such as the pandora settings or the lcfi weights. For those, the setRequired method is added.

The usage is as follows:

  • Let's assume you are running Marlin and the LCFI flavour tagging processor.
  • To run, this processor requires the lcfi weights as a folder in the run directory.
  • In the past, you would create a tar ball with this folder, ship it to the GRID, and pass the LFN as input sandbox. This step does not change.
  • You now need to call the setRequired method of your Marlin instance with marlin.setRequired("lcfiweights"). Here, lcfiweights is the directory containing the weights, the one that was added to the tar ball. Warning, importantIt is not the tar ball itself.

It is also possible to specify multiple items like app.setRequired(["data",'lcfiweights',"myextrafile"]). You may also use wildcard characters such as app.setRequired("folder/*.ext"). This last one will create a local folder "folder" and copy all files *.ext in it.

Warning, important It is important to mention that all applications do not use the setRequired: certain utility applications (like the splitters) do not use it. This is mostly because they do not require external files, and do not run in their own directory (so no file copy is needed). If you use it on one of those, you'll get an uncaught exception while submitting.

Warning, important You still need to specify the files in the input sandbox as the setRequired does not add them to the sandbox.

Warning, important The lib directory is always copied if found (see Specifying your libraries).

 The next sections describe the application specific methods (setters). Everything that is described in this section applies for the following, except otherwise indicated.

Generic application

Line: 475 to 494
  This comes from the fact that Marlin is sensitive to the difference between a Processor and a library.
Deleted:
<
<
The setRequired makes sure the specified files or directories are available before running. This is required because applicaions get their own run directroy, different from the directory where the job runs. The lib directory is always handled properly. Warning, importantThe items must not be folder/file, either direct files from the sandbox or directories. This should be fixed in the future to allow any sub-path to be copied.
 

LCSIM

LCSIM is now used mostly to run the tracking and create the final files for the SID detector concept. The PFA is ran as another application, SLICPandora, described later. The example below is specific to such a use case (most likely use case any way) %SYNTAX{ syntax="python" numbered="off" style="emacs"}%
Line: 602 to 619
  The reason for the different behavior between the 2 cases is because every job submitted uploads its sandbox to the main DIRAC servers, filling its hard drive. All files are sent but those that start with LFN:.
Added:
>
>
Warning, important In case you are using setRequired, you do not need to specify lib there as the directory will always be copied.
 

Getting the output

There are several ways of getting the job outputs: if you have 1 job, if you have a job repository, and if you use OutputData, things are different. In case of 1 job, use

Line: 659 to 679
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.32 $Date:
>
>
File Catalog Client $Revision: 1.34 $Date:
  FC:/> %ENDSYNTAX%

Revision 332013-11-20 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 386 to 386
 %ENDSYNTAX% The setNumberOfEvents value is used in conjunction with setSelectionEfficiency and setMaxNbEvts to make sure that enough events are input. The setMaxNbEvts is the maximum number of events to write in the output file.
Changed:
<
<
This application uses the code available at http://svnweb.cern.ch/guest/lcgentools/trunk/stdhepcut_java. There is a README file in the svn repository that will tell you how to use the application, and by looking at the code you may be able to develop your own cut code. It you want to get your code integrated in the trunk, you should get in touch with either ilc-dirac@cernNOSPAMPLEASE.ch or any LC common generator WG member.
>
>
This application uses the code available at http://svnweb.cern.ch/guest/lcgentools/trunk/stdhepcut_java. There is a README file in the svn repository that will tell you how to use the application, and by looking at the code you may be able to develop your own cut code. It you want to get your code integrated in the trunk, you should get in touch with either ilcdirac-support@cernNOSPAMPLEASE.ch or any LC common generator WG member.
  The following only applies for the JAVA part, as the C++ does not allow it: You can use your own cut classes. They must be stored under org.lcsim.stdhepcut.cuts. Your corresponding jar file must be stored in a lib directory that you can then add to your sandbox. The StdHepCutJava application takes the lib directory and adds it in the CLASSPATH.
Line: 461 to 461
 ma.setInputFile("something.slcio") ma.setSteeringFile("SomeSteeringFile.xml") ma.setGearFile("SomeGear.gear")
Added:
>
>
ma.setRequired(["data","myextrafile"])
 ma.setOutputRecFile("MyRECfile.slcio") ma.setOutputDstFile("myDSTfile.slcio") res = j.append(ma)
Line: 474 to 475
  This comes from the fact that Marlin is sensitive to the difference between a Processor and a library.
Added:
>
>
The setRequired makes sure the specified files or directories are available before running. This is required because applicaions get their own run directroy, different from the directory where the job runs. The lib directory is always handled properly. Warning, importantThe items must not be folder/file, either direct files from the sandbox or directories. This should be fixed in the future to allow any sub-path to be copied.
 

LCSIM

LCSIM is now used mostly to run the tracking and create the final files for the SID detector concept. The PFA is ran as another application, SLICPandora, described later. The example below is specific to such a use case (most likely use case any way) %SYNTAX{ syntax="python" numbered="off" style="emacs"}%
Line: 656 to 659
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.31 $Date:
>
>
File Catalog Client $Revision: 1.32 $Date:
  FC:/> %ENDSYNTAX%

Revision 322013-09-26 - ChristianGrefe

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 31 to 31
 Warning, important The DIRAC environment is usually incompatible with other environments, and therefore often requires a dedicated terminal/shell.

Using a pre existing installation (e.g. at CERN)

Changed:
<
<
If you rely on someone's installation (or your own) you have normally access to the directory in which ILCDIRAC has been installed. In that directory there is a env file, bashrc that needs sourcing to get the right environment. For example, for CERN users, you can run
>
>
If you rely on someone's installation (or your own) you have normally access to the directory in which ILCDIRAC has been installed. In that directory there is a env file, bashrc or cshrc depending on your shell, that needs sourcing to get the right environment. For example, for CERN users, you can run
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Added:
>
>
# bash users
 source /afs/cern.ch/eng/clic/software/DIRAC/bashrc
Added:
>
>
# (t)csh users source /afs/cern.ch/eng/clic/software/DIRAC/cshrc
 %ENDSYNTAX%
Added:
>
>
The default setup is for SLC6. There are also environment scripts for SLC5 available:
<!-- SyntaxHighlightingPlugin -->
# bash users
source /afs/cern.ch/eng/clic/software/DIRAC/bashrc-glibc-2.5
# (t)csh users
source /afs/cern.ch/eng/clic/software/DIRAC/cshrc-glibc-2.5
<!-- end SyntaxHighlightingPlugin -->
  Once this file has been sourced, you get access to all the DIRAC and ILCDIRAC commands, as well as the python API. You can proceed to the Job section.
Line: 47 to 58
 chmod +x dirac-install ./dirac-install -V ILCDIRAC %ENDSYNTAX%
Changed:
<
<
This also creates the bashrc file needed to get the DIRAC environment that you obtain with
>
>
This also creates the bashrc / cshrc file needed to get the DIRAC environment that you obtain with
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Added:
>
>
# bash users
 source bashrc
Added:
>
>
# (t)csh users source cshrc
 %ENDSYNTAX%

Then you need to configure DIRAC (make the client know where to get the services from). This requires the presence of a valid grid proxy. First, if you don't have the pem files, you can get them with:

Line: 72 to 86
  In principle, if this commands runs fine, you should be able to got to the next section.
Changed:
<
<
Don't forget to source the bashrc file whenever using DIRAC.
>
>
Don't forget to source the bashrc / cshrc file whenever using DIRAC.
 

Updating ILCDIRAC

Line: 642 to 656
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.30 $Date:
>
>
File Catalog Client $Revision: 1.31 $Date:
  FC:/> %ENDSYNTAX%

Revision 312013-09-20 - JanStrube

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 628 to 628
 %ENDSYNTAX%
Changed:
<
<

File Catalog business

>
>

Locating Files on the Grid

  All files produced by either ILD, SID or CLIC are stored in the File Catalog. This allows for several data operations like searching for files, getting them, removing them, etc. There are several interfaces available: The Command Line Utility, the web interface, the Python API, and a script.

Revision 302013-09-19 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 47 to 47
 chmod +x dirac-install ./dirac-install -V ILCDIRAC %ENDSYNTAX%
Changed:
<
<
This also creates the bashrc file needed to get the DIRAC environment.

Then you need to configure DIRAC (make the client know where to get the services from). This requires the presence of a valid grid proxy. 2 choices:

1. You have a grid UI available and the pem files: in another session, source the corresponding environment file and run

>
>
This also creates the bashrc file needed to get the DIRAC environment that you obtain with
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Changed:
<
<
grid-proxy-init
>
>
source bashrc
 %ENDSYNTAX%
Changed:
<
<
2. You don't have a grid UI installed nor the pem files. Then you can use
>
>
Then you need to configure DIRAC (make the client know where to get the services from). This requires the presence of a valid grid proxy. First, if you don't have the pem files, you can get them with:
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Deleted:
<
<
source bashrc
 dirac-cert-convert.sh grid.p12
Added:
>
>
%ENDSYNTAX% where grid.p12 is the file you got from the web browser export. If you already have the pem files, this can be skipped.

To get a valid proxy, run %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%

 dirac-proxy-init -x %ENDSYNTAX%
Changed:
<
<
where grid.p12 is the file you got from the web browser export. If you already have the pem files, this can be skipped. The last command might result in an error, but it does not matter normally as the proxy should have been created.
>
>
This is not a DIRAC proxy. A DIRAC proxy is obtained by omitting the -x.
  Then run the configuration %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Changed:
<
<
dirac-configure -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server
>
>
dirac-configure -S ILC-Production -C dips://volcd05.cern.ch:9135/Configuration/Server
 --SkipCAChecks %ENDSYNTAX%
Deleted:
<
<
The volcd01.cern.ch machine will be replaced soon, but this will be announced from the ilc-dirac@cernNOSPAMPLEASE.ch mailing list. When that happens, this command will have to be ran again with the new host (details will be in the mail).
  In principle, if this commands runs fine, you should be able to got to the next section.
Line: 642 to 642
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.29 $Date:
>
>
File Catalog Client $Revision: 1.30 $Date:
  FC:/> %ENDSYNTAX%

Revision 292013-09-11 - JanStrube

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 435 to 435
  The detector model handling is worth presenting: it is the radical of a zip file, in this case it would be clic_sid_cdr.zip. The zip file can live either locally, as an lfn, or as a standard detector description on the lcsim.org web portal. It must be a zip file and the detector it describes must be called like the zip file but without the .zip.
Changed:
<
<
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
>
>
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
 

Reconstruction

Marlin

Line: 474 to 474
 %ENDSYNTAX% There are a few other options that are described in the API documentation.
Changed:
<
<
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
>
>
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
 

SLICPandora

This application (described more in the API documentation is usually used in conjunction with LCSIM and the example below shows such a use case.
Line: 493 to 493
  One noticable aspect if the detector model handling. It's identical to the one of SLIC: the detector model must be e.g. detectormodel where it's described in a detectormodel.zip living either locally, as an lfn, or in the lcsim.org portal.
Changed:
<
<
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
>
>
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
 

Adding the Overlay

As the production system is non deterministic in the sense that it does not know a priori the files it will run on, the overlay files have to be determined at the last minute. This is particularly needed as in the CLIC case, we had many interactions per bunch crossings so the number of events to get was very large, and could vary from job to job. So for that, there is another application to add before marlin or LCSIM, called OverlayInput. This application does not have an OutputFile so don't try to use ma.getInputFromApp(ov) as it would fail.
Line: 513 to 513
  In any case, how it works:
  • Given a signal input file (its number of events to be exact), the interaction probability, and the number of bunch crossing to overlay, it computes the number of background events to fetch.
Changed:
<
<
  • Gets from the configuration the ProdID that is suitable for that sample of background given the machine, the energy and the detector model, and the number of events per bkg file. You can see an example of this under /Operations/Defaults/Overlay/ilc_dbd/energy/detectormodel/bkgtype/ in the Configuration Service.
  • Fetch the list of files from the catalog with that ProdID and other tags (detector model, energy, machine, etc.)
>
>
  • Gets from the configuration the ProdID that is suitable for that sample of background given the machine, the energy and the detector model, and the number of events per bkg file. You can see an example of this under /Operations/Defaults/Overlay/ilc_dbd/energy/detectormodel/bkgtype/ in the Configuration Service.
  • Fetch the list of files from the catalog with that ProdID and other tags (detector model, energy, machine, etc.)
 
  • Get the appropriate number of files given the number of events per background files. The files are then chosen randomly.
  • Stores those under a special directory
  • When Marlin/LCSIM starts, the code checks that the steering XML references a bgoverlay or overlaytiming or org.lcsim.util.OverlayDriver section and then sets the files accordingly.
Line: 642 to 642
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.27 $Date:
>
>
File Catalog Client $Revision: 1.29 $Date:
  FC:/> %ENDSYNTAX%

Revision 282013-07-10 - AndreSailer

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 678 to 678
  meta set - set metadata value for directory or file meta remove - remove metadata value for directory or file meta get [-e] [] - get metadata for the given directory or file
Changed:
<
<
meta tags where - get values (tags) of the given metaname compatible with
>
>
meta tags where - get values (tags) of the given metaname compatible with
  the metadata selection meta show - show all defined metadata indice

Revision 272013-05-31 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 439 to 439
 

Reconstruction

Marlin

Changed:
<
<
The runs the reconstruction in the context of the ILD detector concept. The API is as usual, described in the [[http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/ILCDIRAC.Interfaces.API.NewInterface.Applications.Marlin-class.html][API documentation].
>
>
The runs the reconstruction in the context of the ILD detector concept. The API is as usual, described in the API documentation.
 %SYNTAX{ syntax="python" numbered="off" style="emacs"}% from NewInterface.Applications import Marlin ma = Marlin()
Line: 454 to 454
  Check the API documentation for 2 extra methods that can be useful.
Added:
>
>
If you want to run with your own libs (LD libs and/or MARLIN_DLL), the lib directory MUST have the following structure:
  • The LD libs must go under lib/lddlib/
  • the Processors MUST be under lib/marlin_dll/

This comes from the fact that Marlin is sensitive to the difference between a Processor and a library.

 

LCSIM

LCSIM is now used mostly to run the tracking and create the final files for the SID detector concept. The PFA is ran as another application, SLICPandora, described later. The example below is specific to such a use case (most likely use case any way) %SYNTAX{ syntax="python" numbered="off" style="emacs"}%
Line: 636 to 642
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.25 $Date:
>
>
File Catalog Client $Revision: 1.27 $Date:
  FC:/> %ENDSYNTAX%

Revision 262013-05-22 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 636 to 636
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.24 $Date:
>
>
File Catalog Client $Revision: 1.25 $Date:
  FC:/> %ENDSYNTAX%
Line: 832 to 832
  PolarizationB2 : R FC:/> %ENDSYNTAX%
Changed:
<
<
Here, there are no indication (for the moment) of the nature of the meta data (searcable or not) and the user should rely on the meta show output.
>
>
Here, there are no indication (for the moment) of the nature of the meta data (searchable or not) and the user should rely on the meta show output.

A last functionality worth advertising here is the ancestor/daughter relationships. For this, the following examples should be enough.

<!-- SyntaxHighlightingPlugin -->
FC:/> ancestor /ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio 
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
1       /ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00003-DST.slcio
1       /ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00002-DST.slcio
1       /ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
<!-- end SyntaxHighlightingPlugin -->
This gives the direct parents.

The descendants of a file can be also obtained:

<!-- SyntaxHighlightingPlugin -->
FC:/> descendent /ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00002-DST.slcio
1       /ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
<!-- end SyntaxHighlightingPlugin -->
  There are other useful utilities in this interface that should be looked upon, but the help is enough to grasp the concepts. In particular, users can get their files using get.

Revision 252013-05-07 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 158 to 158
 job = UserJob() job.setName("MyJobName") job.setJobGroup("Agroup")
Added:
>
>
job.setInputData(['datafile1','datafile2'])
 job.setInputSandbox(["file1","file2"]) job.setOutputSandbox(["fileout1","fileout2"]) job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
Line: 173 to 174
  The next few lines give some of the UserJob class method use example. The first 2, setName and setJobGroup, can be used for the monitoring: the names are displayed on the monitoring page and the JobGroup can be used to select only the jobs belonging to a given group.
Changed:
<
<
The next 2, setInputSandbox and setOutputSandbox, indicate to the job what files should be shipped to the job working area, and what files should be brought back when retrieving the job outputs. Any file in the setInputSandbox must reside locally (absolute or relative path accepted), or be stored on the GRID, and specified with the LFN (Logical File Name, described in the File Catalog section of this page), where said LFN must be prepended by LFN:. For example, if there is a file /ilc/user/s/someone/file, the file to specify in the setInputSandbox should be LFN:/ilc/user/s/someone/file. As it can be seen in the example, python lists can be used, but not lists of lists. Warning, importantMake sure your sandboxes do not contain such things, as the system cannot correct them.
>
>
The next one, setInputData is used as it's name suggest to define input data. Input data means files that can be on a tape backend and can rely on staging (recall from tape) to be available. Files produced from the production system should be accessed using this method. There is unfortunately no link between this and the application setInputFile so you'll need to specify the file in both. The file name to give is an LFN (Logical File Name, described in the File Catalog section of this page), like /ilc/prod/clic/[...]/something.slcio.

The next 2, setInputSandbox and setOutputSandbox, indicate to the job what files should be shipped to the job working area, and what files should be brought back when retrieving the job outputs. Any file in the setInputSandbox must reside locally (absolute or relative path accepted), or be stored on the GRID, and specified with the LFN, where it must be prepended by LFN:. For example, if there is a file /ilc/user/s/someone/file, the file to specify in the setInputSandbox should be LFN:/ilc/user/s/someone/file. As it can be seen in the example, python lists can be used, but not lists of lists. Warning, importantMake sure your sandboxes do not contain such things, as the system cannot correct them.

 Tip, idea It is a good idea to put on the GRID large input files (or often used) and specify them using the LFN: definition: they make the job submission much faster and make the ILCDIRAC servers happy (no huge amount of data to carry around). How to do this is given in the section Specifying your libraries.

The setOutputSandbox follows the same style. There are several important things to notice: If a file is missing in the output, DIRAC will mark the job as failed, but the output will be filled with whatever is available among the requested things. If the sandbox is bigger than 10MB, the files will be packed together in a tar ball and shipped to the GRID. Warning, importantIn version ILCDIRAC v16r7p0, it seems that retrieving the output sandbox of a job does not retrieve those files from the GRID. Retrieving job outputs is explained later on this page.

Line: 633 to 636
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.23 $Date:
>
>
File Catalog Client $Revision: 1.24 $Date:
  FC:/> %ENDSYNTAX%

Revision 242013-05-02 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 875 to 875
  The web monitoring of jobs happen here: https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/jobs/JobMonitor/display
Added:
>
>

F.A.Q.

I keep getting Could not verify credential errors

You see errors like:
[SE][GetSpaceTokens][] httpg://srm-public.cern.ch:8443/srm/managerv2: CGSI-gSOAP running on pclcd15.cern.ch reports Error initializing context
GSS Major Status: Authentication Failed

GSS Minor Status Error Chain:
globus_gsi_gssapi: SSLv3 handshake problems
globus_gsi_callback_module: Could not verify credential
globus_gsi_callback_module: Could not verify credential
globus_gsi_callback_module: Invalid CRL: The available CRL has expired
This means that the local copy of the certificate revocation list (CRL) expired. Please run dirac-admin-get-CAs to update you copy, or if you don;t have the right to run the command yourself, ask your local dirac administrator.
 
META FILEATTACHMENT attachment="webMonitoring.png" attr="" comment="" date="1365507538" name="webMonitoring.png" path="webMonitoring.png" size="431203" user="sposs" version="1"

Revision 232013-04-29 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 182 to 182
 
<!-- SyntaxHighlightingPlugin -->
job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
<!-- end SyntaxHighlightingPlugin -->
Changed:
<
<
This is used when a user wants to keep some files on the GRID for further processing or for easy sharing with others. It implies that the specified files somefile1 and somefile2 will be uploaded to the GRID under the automatically built LFN /ilc/user/u/username/some/path/. u and username indicate the DIRAC user name of the user that created the job (u is the initial). The some/path part of the LFN comes from the second element in the method call above. Finally, the storage element to store the files can be specified by passing its logical name in the last element of the method call. In this example, e use CERN-SRM, which is the default storage for ILCDIRAC. See the Storage Elements section for a list of valid SEs.
>
>
This is used when a user wants to keep some files on the GRID for further processing or for easy sharing with others. It implies that the specified files somefile1 and somefile2 will be uploaded to the GRID under the automatically built LFN /ilc/user/u/username/some/path/. u and username indicate the DIRAC user name of the user that created the job (u is the initial). The some/path part of the LFN comes from the second element in the method call above. Finally, the storage element to store the files can be specified by passing its logical name in the last element of the method call. In this example, e use CERN-SRM, which is the default storage for ILCDIRAC. See the Storage Elements section for a list of valid SEs. Warning, importantIf the output files already exist in the File Catalog, the jobs will fail because overwriting files is not possible. Be sure to clean before submitting the jobs.
  Finally, the line %SYNTAX{ syntax="python" numbered="off" style="emacs"}%
Line: 432 to 432
  The detector model handling is worth presenting: it is the radical of a zip file, in this case it would be clic_sid_cdr.zip. The zip file can live either locally, as an lfn, or as a standard detector description on the lcsim.org web portal. It must be a zip file and the detector it describes must be called like the zip file but without the .zip.
Added:
>
>
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
 

Reconstruction

Marlin

The runs the reconstruction in the context of the ILD detector concept. The API is as usual, described in the [[http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/ILCDIRAC.Interfaces.API.NewInterface.Applications.Marlin-class.html][API documentation].
Line: 463 to 465
 %ENDSYNTAX% There are a few other options that are described in the API documentation.
Added:
>
>
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
 

SLICPandora

This application (described more in the API documentation is usually used in conjunction with LCSIM and the example below shows such a use case. %SYNTAX{ syntax="python" numbered="off" style="emacs"}%
Line: 480 to 484
  One noticable aspect if the detector model handling. It's identical to the one of SLIC: the detector model must be e.g. detectormodel where it's described in a detectormodel.zip living either locally, as an lfn, or in the lcsim.org portal.
Added:
>
>
There is a script written by C. Grefe that takes care of running the SID reconstruction chain. J. McCormick wrote some documentation on the confluence page.
 

Adding the Overlay

As the production system is non deterministic in the sense that it does not know a priori the files it will run on, the overlay files have to be determined at the last minute. This is particularly needed as in the CLIC case, we had many interactions per bunch crossings so the number of events to get was very large, and could vary from job to job. So for that, there is another application to add before marlin or LCSIM, called OverlayInput. This application does not have an OutputFile so don't try to use ma.getInputFromApp(ov) as it would fail.
Line: 627 to 633
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.21 $Date:
>
>
File Catalog Client $Revision: 1.23 $Date:
  FC:/> %ENDSYNTAX%

Revision 212013-04-25 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 329 to 329
  exit(1) %ENDSYNTAX%
Changed:
<
<
As usual, a detailed description of the class is available on the [[http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/ILCDIRAC.Interfaces.API.NewInterface.Applications.Whizard-class.html][API documentation page]. I will only briefly discuss the instance definition, and how it works.
>
>
As usual, a detailed description of the class is available on the API documentation page. I will only briefly discuss the instance definition, and how it works.
 %SYNTAX{ syntax="python" numbered="off" style="emacs"}% from NewInterface.Applications import Whizard from DiracILC import DiracILC
Line: 346 to 346
 ... wh.setFullParameterDict(pdict) %ENDSYNTAX%
Changed:
<
<
which allows setting the whizard parameters. The structure of the pdict is the same as the whizard.in file: there are sections like pdict['process_input'] that correspond to the process_input section of the whizard.in. All the sections are available this way, with the notable difference of the beam_input sections that are named explicitly beam_input_1 and beam_input_2 for clarity. All the possible parameters described on the [[http://whizard.hepforge.org/manual_w1/manual005.html#toc11][whizard documentation page] can be set.
>
>
which allows setting the whizard parameters. The structure of the pdict is the same as the whizard.in file: there are sections like pdict['process_input'] that correspond to the process_input section of the whizard.in. All the sections are available this way, with the notable difference of the beam_input sections that are named explicitly beam_input_1 and beam_input_2 for clarity. All the possible parameters described on the whizard documentation page can be set.
  The rest of the possibilities is described in the API page.
Line: 627 to 627
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.19 $Date:
>
>
File Catalog Client $Revision: 1.21 $Date:
  FC:/> %ENDSYNTAX%

Revision 202013-04-23 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 464 to 464
 There are a few other options that are described in the API documentation.

SLICPandora

Changed:
<
<
This application (described more in the [[http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/ILCDIRAC.Interfaces.API.NewInterface.Applications.SLICPandora-class.html][API documentation]) is usually used in conjunction with LCSIM and the example below shows such a use case.
>
>
This application (described more in the API documentation is usually used in conjunction with LCSIM and the example below shows such a use case.
 %SYNTAX{ syntax="python" numbered="off" style="emacs"}% from NewInterface.Applications import SLICPandora slicpandora = SLICPandora()
Line: 627 to 627
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.18 $Date:
>
>
File Catalog Client $Revision: 1.19 $Date:
  FC:/> %ENDSYNTAX%

Revision 192013-04-19 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 182 to 182
 
<!-- SyntaxHighlightingPlugin -->
job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
<!-- end SyntaxHighlightingPlugin -->
Changed:
<
<
This is used when a user wants to keep some files on the GRID for further processing or for easy sharing with others. It implies that the specified files somefile1 and somefile2 will be uploaded to the GRID under the automatically built LFN /ilc/user/u/username/some/path/. u and username indicate the DIRAC user name of the user that created the job (u is the initial). The some/path part of the LFN comes from the second element in the method call above. Finally, the storage element to store the files can be specified by passing its logical name in the last element of the method call. In this example, e use CERN-SRM, which is the default storage for ILCDIRAC. We have several sites that support us and that can be used:
CERN-SRM (default)
DESY-SRM
RAL-SRM
KEK-SRM
IN2P3-SRM (inactive because full)
PNNL-SRM
SLAC-SRM
We recommend you use a storage element that's close enough geographically to you.
>
>
This is used when a user wants to keep some files on the GRID for further processing or for easy sharing with others. It implies that the specified files somefile1 and somefile2 will be uploaded to the GRID under the automatically built LFN /ilc/user/u/username/some/path/. u and username indicate the DIRAC user name of the user that created the job (u is the initial). The some/path part of the LFN comes from the second element in the method call above. Finally, the storage element to store the files can be specified by passing its logical name in the last element of the method call. In this example, e use CERN-SRM, which is the default storage for ILCDIRAC. See the Storage Elements section for a list of valid SEs.
  Finally, the line %SYNTAX{ syntax="python" numbered="off" style="emacs"}%
Line: 523 to 513
 

StdHepSplit

CheckCollections

Added:
>
>

Storage Elements

The following storage elements can be used as output location:
CERN-SRM (default)
DESY-SRM
RAL-SRM
KEK-SRM
IN2P3-SRM
PNNL-SRM
SLAC-SRM
We recommend you use a storage element that's close enough geographically to you.
 

Specifying your libraries

Libraries also have dedicated support. This mostly depends on the application, in particular for Marlin, so here I show only the default method, but please check if the application you want to run for specificities.
Line: 623 to 627
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.17 $Date:
>
>
File Catalog Client $Revision: 1.18 $Date:
  FC:/> %ENDSYNTAX%

Revision 172013-04-12 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 302 to 302
 

Whizard 1.95

This generator is used for the CLIC CDR and ILC DBD studies. It has several limitations not discussed here, but the main one being that it requires a dedicated binary containing the process one wants to study. As having an exhaustive list of processes is not possible, only a few (>1000) are available. If you need to use this application, it's a good idea to contact ilc-dirac@cernNOSPAMPLEASE.ch to ask if a given process is available. If not, someone will (or not) add it to WHIZARD and make it available. When WHIZARD 2 is supported by ILCDIRAC, the process can be obtained directly as WHIZARD will be ran "automatically".
Added:
>
>
The standard way to create a WHIZARD applciation is by using
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import Whizard
from ILCDIRAC.Interfaces.API.DiracILC              import DiracILC
d = DiracILC()
wh= Whizard(d.getProcessList())
wh.setModel("sm")
pdict = {}
pdict['process_input']={}
pdict['process_input']['process_id'] = "qq"
pdict['process_input']['sqrts'] = 1400
pdict['simulation_input'] = {}
pdict['simulation_input']['n_events'] = 10000
pdict['beam_input_1'] = {}
pdict['beam_input_1']['particle_name']='e1'
pdict['beam_input_1']['polarization'] = "0.0 0.0"
pdict['beam_input_1']['USER_spectrum_on'] = 'T'
pdict['beam_input_1']['USER_spectrum_mode'] = 19
pdict['beam_input_1']['ISR_on'] = 'T'
pdict['beam_input_2'] = {}
pdict['beam_input_2']['particle_name']='E1'
pdict['beam_input_2']['polarization'] = "0.0 0.0"
pdict['beam_input_2']['USER_spectrum_on'] = 'T'
pdict['beam_input_2']['ISR_on'] = 'T'
pdict['beam_input_2']['USER_spectrum_mode'] = -19
wh.setFullParameterDict(pdict)
wh.setOutputFile("toto.stdhep")
res = j.append(wh)
if not res['OK']:
    print res['Message']
    exit(1)
<!-- end SyntaxHighlightingPlugin -->

As usual, a detailed description of the class is available on the [[http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/ILCDIRAC.Interfaces.API.NewInterface.Applications.Whizard-class.html][API documentation page]. I will only briefly discuss the instance definition, and how it works.

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import Whizard
from ILCDIRAC.Interfaces.API.DiracILC              import DiracILC
d = DiracILC()
wh= Whizard(d.getProcessList())
<!-- end SyntaxHighlightingPlugin -->
As can be seen on the code snipped above, it is necessary to get a DiracILC instance to create a WHIZARD instance. The reason is that it's mandatory to get the available processes for WHIZARD to be configured. In particular, the process list contains all the processes as well as in which version of WHIZARD they are defined. The d.getProcessList() takes care of downloading the file locally. You will find the processlist.whiz file in the local directory.
 
Added:
>
>
The next thing is wh.setModel("sm"). All the avaialble models are defined in the Configuration Service under /Operations/Defaults/Models. In this example, the SM is used so nothing particular happens. When using SUSY for instance, this is used to define which LesHouches file to use. It is always possible to overwrite the LesHouches file to use by placing one in the InputSandbox, and naming it LesHouches.msugra_1.in. No, it's not necessarily a msugra model, but that's just a name and it has to be like that to be picked up.

After, there is

<!-- SyntaxHighlightingPlugin -->
pdict = {}
...
wh.setFullParameterDict(pdict)
<!-- end SyntaxHighlightingPlugin -->
which allows setting the whizard parameters. The structure of the pdict is the same as the whizard.in file: there are sections like pdict['process_input'] that correspond to the process_input section of the whizard.in. All the sections are available this way, with the notable difference of the beam_input sections that are named explicitly beam_input_1 and beam_input_2 for clarity. All the possible parameters described on the [[http://whizard.hepforge.org/manual_w1/manual005.html#toc11][whizard documentation page] can be set.

The rest of the possibilities is described in the API page.

 

Pythia

Changed:
<
<

StdHepCuts

>
>
The PYTHIA application is rarely used as it's not generic at all: it can only produce ttbar, WW, and ZZ events. When needing such events, I usually run it locally and upload the files. The issue with that application is the lack of flexibility, everything is hard coded in the FORTRAN code. It should be considered as an expert's application and will not be documented here. Nevertheless, for the advanced user, as usual, the API is documented at the usual location.

StdHepCut and StdHepCutJava

There are 2 versions of this application, which are equivalent: one in C++ and one in JAVA. The latter is needed because the former does unnecessary thing due to limitations of the C++ stdhep library. In practice, the JAVA version should be preferred. The interface being identical for both application, the example will use the JAVA one.

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import StdHepCutJava

cut = StdHepCutJava()
cut.setVersion("1.0")
cut.setNumberOfEvents(10000)
cut.setSteeringFile('cuts_quarks_1400.txt')
cut.setMaxNbEvts(10)
cut.setSelectionEfficiency(0.5)
res = j.append(cut)
<!-- end SyntaxHighlightingPlugin -->
The setNumberOfEvents value is used in conjunction with setSelectionEfficiency and setMaxNbEvts to make sure that enough events are input. The setMaxNbEvts is the maximum number of events to write in the output file.

This application uses the code available at http://svnweb.cern.ch/guest/lcgentools/trunk/stdhepcut_java. There is a README file in the svn repository that will tell you how to use the application, and by looking at the code you may be able to develop your own cut code. It you want to get your code integrated in the trunk, you should get in touch with either ilc-dirac@cernNOSPAMPLEASE.ch or any LC common generator WG member.

The following only applies for the JAVA part, as the C++ does not allow it: You can use your own cut classes. They must be stored under org.lcsim.stdhepcut.cuts. Your corresponding jar file must be stored in a lib directory that you can then add to your sandbox. The StdHepCutJava application takes the lib directory and adds it in the CLASSPATH.

 

Simulation

Added:
>
>
The design of those applications could be a bit better structured, as they are all independent, instead of inheriting from a common SimApplication class. That's just an implementation detail, transparent for the user.
 

Mokka

Added:
>
>
This application was the first to be added to ILCDIRAC.

It's interface is described like any other in the API documentation. The example below shows how it can be used:

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import Mokka
mo = Mokka()
mo.setInputFile("some_file.stdhep")
mo.setVersion("0706P08")
mo.setSteeringFile("clic_ild_cdr.steer")
mo.setOutputFile("totosim.slcio")
res = j.append(mo)
<!-- end SyntaxHighlightingPlugin -->
That's the simplest way to call the application.

There are more possibilities that are worth showing:

<!-- SyntaxHighlightingPlugin -->
mo.setDetectorModel('Something')
<!-- end SyntaxHighlightingPlugin -->
allows to define another detector model, as the default is CLIC_ILD_CDR.

<!-- SyntaxHighlightingPlugin -->
mo.setDbSlice("something.sql")
<!-- end SyntaxHighlightingPlugin -->
allows to define your own MySQL database dump. It can be a local file or an lfn. The specified detector model must exist in the dump of course.

<!-- SyntaxHighlightingPlugin -->
mo.setMacFile("mymacro.mac")
<!-- end SyntaxHighlightingPlugin -->
Allows to specify your own macro file. This is needed when you want to use ParticleGun for example. It can be a local file or an lfn. Whatever is in there is unchanged. If you don't specify a macro file, the application creates one with the input file, the startFrom and the number of events specified. This means that if you have your own macro file and want to use the startFrom functionality or change the number of events, you need to set it yourself in your macro file.

<!-- SyntaxHighlightingPlugin -->
mo.setStartFrom(10)
<!-- end SyntaxHighlightingPlugin -->
Allows to define the starting event in the input file.
 

SLIC

Added:
>
>
The interface of SLIC is similar to that of Mokka, and is described as usual in the API documentation.

It is invoked with

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import SLIC
slic = SLIC()
slic.setVersion('v2r9p8')
slic.setInputFile("some_file.stdhep")
slic.setSteeringFile('MyMacro.mac')
slic.setDetectorModel('clic_sid_cdr')
slic.setOutputFile("Something.slcio")
res = j.append(slic)
<!-- end SyntaxHighlightingPlugin -->

Similarly to Mokka it has a setStartFrom(N) method that allows skipping the first N events in the input file.

The detector model handling is worth presenting: it is the radical of a zip file, in this case it would be clic_sid_cdr.zip. The zip file can live either locally, as an lfn, or as a standard detector description on the lcsim.org web portal. It must be a zip file and the detector it describes must be called like the zip file but without the .zip.

 

Reconstruction

Marlin

Added:
>
>
The runs the reconstruction in the context of the ILD detector concept. The API is as usual, described in the [[http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/ILCDIRAC.Interfaces.API.NewInterface.Applications.Marlin-class.html][API documentation].
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import Marlin
ma = Marlin()
ma.setVersion('v0111Prod')
ma.setInputFile("something.slcio")
ma.setSteeringFile("SomeSteeringFile.xml")
ma.setGearFile("SomeGear.gear")
ma.setOutputRecFile("MyRECfile.slcio")
ma.setOutputDstFile("myDSTfile.slcio")
res = j.append(ma)
<!-- end SyntaxHighlightingPlugin -->

Check the API documentation for 2 extra methods that can be useful.

LCSIM

LCSIM is now used mostly to run the tracking and create the final files for the SID detector concept. The PFA is ran as another application, SLICPandora, described later. The example below is specific to such a use case (most likely use case any way)
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import LCSIM
lcsim_prepandora = LCSIM()
lcsim_prepandora.setVersion('CLIC_CDR')
lcsim_prepandora.setSteeringFile("clic_cdr_prePandora.lcsim")
lcsim_prepandora.setTrackingStrategy("defaultStrategies_clic_sid_cdr.xml")
lcsim_prepandora.setOutputFile("prePandora.slcio")
lcsim_prepandora.willRunSLICPandora()
res = j.append(lcsim_prepandora)
<!-- end SyntaxHighlightingPlugin -->
There are a few other options that are described in the API documentation.

SLICPandora

This application (described more in the [[http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/ILCDIRAC.Interfaces.API.NewInterface.Applications.SLICPandora-class.html][API documentation]) is usually used in conjunction with LCSIM and the example below shows such a use case.
<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import SLICPandora
slicpandora = SLICPandora()
slicpandora.setVersion('CLIC_CDR')
slicpandora.setDetectorModel('clic_sid_cdr')
slicpandora.setPandoraSettings("PandoraSettingsSlic.xml")
slicpandora.getInputFromApp(lcsim_prepandora)
slicpandora.setOutputFile('pandora.slcio')
res = j.append(slicpandora)
<!-- end SyntaxHighlightingPlugin -->

It is of course posible to use it as standalone.

 
Changed:
<
<

LCSIM

>
>
One noticable aspect if the detector model handling. It's identical to the one of SLIC: the detector model must be e.g. detectormodel where it's described in a detectormodel.zip living either locally, as an lfn, or in the lcsim.org portal.
 

Adding the Overlay

Added:
>
>
As the production system is non deterministic in the sense that it does not know a priori the files it will run on, the overlay files have to be determined at the last minute. This is particularly needed as in the CLIC case, we had many interactions per bunch crossings so the number of events to get was very large, and could vary from job to job. So for that, there is another application to add before marlin or LCSIM, called OverlayInput. This application does not have an OutputFile so don't try to use ma.getInputFromApp(ov) as it would fail.

It requires a few parameters like the number of interactions per bunch crossing, the detector model, etc. Those are described in the API documentation, and the example below shows how to configure it.

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import OverlayInput
overlay = OverlayInput()
overlay.setMachine("clic_cdr")
overlay.setEnergy(3000.0)
overlay.setBXOverlay(60)
overlay.setGGToHadInt(3.2)##When running at 3TeV
overlay.setDetectorModel("CLIC_ILD_CDR")
overlay.setNbSigEvtsPerJob(10)
res = j.append(overlay)
<!-- end SyntaxHighlightingPlugin -->

In any case, how it works:

  • Given a signal input file (its number of events to be exact), the interaction probability, and the number of bunch crossing to overlay, it computes the number of background events to fetch.
  • Gets from the configuration the ProdID that is suitable for that sample of background given the machine, the energy and the detector model, and the number of events per bkg file. You can see an example of this under /Operations/Defaults/Overlay/ilc_dbd/energy/detectormodel/bkgtype/ in the Configuration Service.
  • Fetch the list of files from the catalog with that ProdID and other tags (detector model, energy, machine, etc.)
  • Get the appropriate number of files given the number of events per background files. The files are then chosen randomly.
  • Stores those under a special directory
  • When Marlin/LCSIM starts, the code checks that the steering XML references a bgoverlay or overlaytiming or org.lcsim.util.OverlayDriver section and then sets the files accordingly.
 
Changed:
<
<

ROOT applications

>
>

ROOT applications

 

Other applications

GetSRMFile

Line: 428 to 619
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.16 $Date:
>
>
File Catalog Client $Revision: 1.17 $Date:
  FC:/> %ENDSYNTAX%
Line: 594 to 785
 FC:/> %ENDSYNTAX%
Changed:
<
<
The . indicates that one as to look for compatible files in the current directory, and can be replaced by a path, e.g /ilc/prod/ilc/md-dbd/ild to make sure the files only come from this directory and its sub-directories.
>
>
The . indicates that one as to look for compatible files in the current directory, and can be replaced by a path, e.g /ilc/prod/ilc/mc-dbd/ild to make sure the files only come from this directory and its sub-directories.
  It's always possible to have access to the defined meta data of a directory: %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%

Revision 162013-04-09 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 373 to 373
 

Getting the output

Added:
>
>
There are several ways of getting the job outputs: if you have 1 job, if you have a job repository, and if you use OutputData, things are different. In case of 1 job, use
<!-- SyntaxHighlightingPlugin -->
dirac-wms-job-get-output JID
<!-- end SyntaxHighlightingPlugin -->

If you have a job repository repo.rep:

<!-- SyntaxHighlightingPlugin -->
dirac-repo-retrieve-output -r repo.rep (-O)
<!-- end SyntaxHighlightingPlugin -->
The -O option gives you the possibility to download also the output data.

Finally, you can also download the job output from the web portal by left-clicking on a job, Sandobx, Get output file(s). You can also get the job input that way.

 

Monitoring the jobs

Connect to the ILCDIRAC monitoring page and you'll get something like the image below.

Line: 415 to 428
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.15 $Date:
>
>
File Catalog Client $Revision: 1.16 $Date:
  FC:/> %ENDSYNTAX%

Revision 152013-04-09 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 375 to 375
 

Monitoring the jobs

Added:
>
>
Connect to the ILCDIRAC monitoring page and you'll get something like the image below.

webMonitoring.png

The corresponding menu actions are:

  1. Main Menu: This menu offers options for systems, jobs, tools and help.
  2. Selections: Shows a set of selectors than permits generate customs selections.
  3. Buttons to open/collapse panels: Permit open or collapse left menu.
  4. Actions to perform for job(s): These actions permits select all, select none, reset, kill or submit
  5. Menu to change DIRAC setup: Users can change between different setups.
  6. Current location: Indicates where the user is located inside the portal.
  7. Buttons to submit or reset the form: After options are selected its possible to submit and execute the selection or reset the selectors.
  8. Pagination controls: Permits navigate between the pages, and also show in which page the user is navigating.
  9. Refresh table: Reload the page without loose the previous selection and show the new status.
  10. Items per page: This option allow the users to specify how many items are going to be displayed by page.
  11. User DIRAC login: Login assigned to the user connected to DIRAC web portal.
  12. DIRAC Group: The user could belong to different groups and perform actions depending of the group previously selected.
  13. Certificate DN: Web portal shows the distinguish name of user certificate what is being used to realize the connection.
  14. Index items displayed: Display the range of items displayed in the page.

Monitoring the jobs is also possible via a command line utility, dirac-wms-job-status JID, where JID is an individual job ID. If you use the job repository file, you can monitor all the jobs in there with

<!-- SyntaxHighlightingPlugin -->
dirac-repo-monitor -r repo.rep
<!-- end SyntaxHighlightingPlugin -->
 

File Catalog business

All files produced by either ILD, SID or CLIC are stored in the File Catalog. This allows for several data operations like searching for files, getting them, removing them, etc. There are several interfaces available: The Command Line Utility, the web interface, the Python API, and a script.

Line: 389 to 415
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.13 $Date:
>
>
File Catalog Client $Revision: 1.15 $Date:
  FC:/> %ENDSYNTAX%
Line: 630 to 656
 

Web monitoring

The web monitoring of jobs happen here: https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/jobs/JobMonitor/display \ No newline at end of file

Added:
>
>
META FILEATTACHMENT attachment="webMonitoring.png" attr="" comment="" date="1365507538" name="webMonitoring.png" path="webMonitoring.png" size="431203" user="sposs" version="1"

Revision 142013-04-09 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 83 to 83
 
<!-- SyntaxHighlightingPlugin -->
dirac-proxy-init -g calice_user
<!-- end SyntaxHighlightingPlugin -->
Changed:
<
<
depending on the VO you want a proxy for. The 2 VOs having access to different resources, it's sometimes needed to get the right proxy (in particular for data access). Also, this has an effect on the grid sites on which one can run, for instance, only the DESY-HH site is available to the CALICE users (in ILCDIRAC, and for the moment).
>
>
depending on the VO you want a proxy for. The 2 VOs having access to different resources, it's sometimes needed to get the right proxy (in particular for data access). Also, this has an effect on the grid sites on which one can run, for instance, only the DESY-HH and IPNL sites are available to the CALICE users (in ILCDIRAC, and for the moment).
 

Your first DIRAC job

This section is a copy of what is presented in https://github.com/DIRACGrid/DIRAC/wiki/JobManagement
Line: 389 to 389
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.12 $Date:
>
>
File Catalog Client $Revision: 1.13 $Date:
  FC:/> %ENDSYNTAX%

Revision 132013-04-08 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 139 to 139
 from DIRAC.Core.Base import Script Script.parseCommandLine() %ENDSYNTAX%
Changed:
<
<
are mandatory to get the DIRAC environment know to the script (This is oversimplifying things but enough). For example, this way the different services that are used get their server addresses set. This also initializes many internal DIRAC utilities that ILCDIRAC makes use of (logger functionality for example),
>
>
are mandatory to get the DIRAC environment known to the script (This is oversimplifying things but enough). For example, this way the different services that are used get their server addresses set. This also initializes many internal DIRAC utilities that ILCDIRAC makes use of (logger functionality for example). They need to be called before the other DIRAC imports to avoid race conditions.
  The following 2 line %SYNTAX{ syntax="python" numbered="off" style="emacs"}%
Line: 389 to 389
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.11 $Date:
>
>
File Catalog Client $Revision: 1.12 $Date:
  FC:/> %ENDSYNTAX%

Revision 122013-04-08 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 328 to 328
 

StdHepSplit

CheckCollections

Deleted:
<
<

Chaining applications

 

Specifying your libraries

Libraries also have dedicated support. This mostly depends on the application, in particular for Marlin, so here I show only the default method, but please check if the application you want to run for specificities.
Line: 391 to 389
 %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% Starting FileCatalog client
Changed:
<
<
File Catalog Client $Revision: 1.10 $Date:
>
>
File Catalog Client $Revision: 1.11 $Date:
  FC:/> %ENDSYNTAX%

Revision 112013-04-08 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 65 to 65
  Then run the configuration %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Changed:
<
<
dirac-configure -V ilc -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server
>
>
dirac-configure -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server
 --SkipCAChecks %ENDSYNTAX% The volcd01.cern.ch machine will be replaced soon, but this will be announced from the ilc-dirac@cernNOSPAMPLEASE.ch mailing list. When that happens, this command will have to be ran again with the new host (details will be in the mail).

Revision 102013-03-20 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 379 to 379
 

File Catalog business

Added:
>
>
All files produced by either ILD, SID or CLIC are stored in the File Catalog. This allows for several data operations like searching for files, getting them, removing them, etc. There are several interfaces available: The Command Line Utility, the web interface, the Python API, and a script.

The Command Line Interface

It is invoked using
<!-- SyntaxHighlightingPlugin -->
dirac-dms-file-catalog-cli
<!-- end SyntaxHighlightingPlugin -->

Once started you'll get a shell that looks like this:

<!-- SyntaxHighlightingPlugin -->
Starting FileCatalog client

File Catalog Client $Revision: 1.10 $Date: 
            
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

Here, you can get help by typing help:

<!-- SyntaxHighlightingPlugin -->
FC:/> help
Documented commands (type help <topic>):
========================================
add          chgrp       exit   guid  meta     register   rm         stats     
ancestor     chmod       find   id    mkdir    repair     rmdir      unregister
ancestorset  chown       get    lcd   pwd      replicas   rmreplica  user      
cd           descendent  group  ls    rebuild  replicate  size     

Undocumented commands:
======================
help

FC:/> 
<!-- end SyntaxHighlightingPlugin -->

As you can see, you can get help with

<!-- SyntaxHighlightingPlugin -->
FC:/> help meta                                                                                                                                                     
 Metadata related operations
    
        Usage:
          meta index [-d|-f|-r] <metaname> [<metatype>]  - add new metadata index. Possible types are:
                                                           'int', 'float', 'string', 'date';
                                                         -d  directory metadata
                                                         -f  file metadata
                                                         -r  remove the specified metadata index
          meta set <path> <metaname> <metavalue> - set metadata value for directory or file
          meta remove <path> <metaname>  - remove metadata value for directory or file
          meta get [-e] [<path>] - get metadata for the given directory or file
          meta tags <metaname> where <meta_selection> - get values (tags) of the given metaname compatible with 
                                                        the metadata selection
          meta show - show all defined metadata indice
    
    
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

Warning, importantSome commands are considered unsafe like the rm and rmdir and should not be used. As this system is being improved, this will be made safe.

From the help message above, one can already see what can be done with data manipulation. In particular, it's possible to set searchable index using the first command. I do not recommend this to everyone, as it should be left to production managers or service managers. It is also possible to set meta data values for directories and files using meta set. If a metadata is not part of the searchable fields, it will be set as documentation. Getting the searchable fields is done with

<!-- SyntaxHighlightingPlugin -->
FC:/> meta show
      FileMetaFields : {'PolarizationB2': 'VARCHAR(128)', 'GenProcessID': 'INT', 'BeamParticle1': 'VARCHAR(128)',
 'BeamParticle2': 'VARCHAR(128)', 'PolarizationB1': 'VARCHAR(128)'}
 DirectoryMetaFields : {'EvtType': 'VARCHAR(128)', 'NumberOfEvents': 'int', 'BXoverlayed': 'int', 'Polarisation': 'VARCHAR(128)', 
'Datatype': 'VARCHAR(128)', 'Energy': 'VARCHAR(128)', 'MachineParams': 'VARCHAR(128)', 'DetectorType': 'VARCHAR(128)', 
'Machine': 'VARCHAR(128)', 'EvtClass': 'VARCHAR(128)', 'Owner': 'VARCHAR(128)', 'SoftwareTag': 'VARCHAR(128)', 
'DetectorModel': 'VARCHAR(128)', 'JobType': 'VARCHAR(128)', 'ProdID': 'int'}
<!-- end SyntaxHighlightingPlugin -->
The output is a bit ugly but will be improved.

It is also possible to get all values for a meta tag (Warning, importantfor directory level tags only for the moment) using

<!-- SyntaxHighlightingPlugin -->
FC:/> meta tags EvtClass                          
Possible values for EvtClass:
1f
1f_3f
2f
2f_Z_bhabhag
2f_Z_hadronic
2f_Z_leptonic
3f
4f
4f_singleW_leptonic
4f_singleW_semileptonic
4f_singleZee_leptonic
4f_singleZee_semileptonic
4f_singleZnunu_leptonic
4f_singleZnunu_semileptonic
4f_singleZsingleWMix_leptonic
4f_WW_hadronic
4f_WW_leptonic
4f_WW_semileptonic
4f_ZZWWMix_hadronic
4f_ZZWWMix_leptonic
4f_ZZ_hadronic
4f_ZZ_leptonic
4f_ZZ_semileptonic
5f
6f
6f_eeWW
6f_llWW
6f_ttbar
6f_vvWW
6f_xxWW
6f_xxxxZ
6f_yyyyZ
aa_2f
aa_4f
aa_lowpt
aa_minijet
higgs
higgs_ffffh
higgs_ffh
ttbb-all-all
tth
tth-2l2nbb-hbb
tth-2l2nbb-hnonbb
tth-6q-hbb
tth-6q-hnonbb
tth-ln4q-hbb
tth-ln4q-hnonbb
ttz-all-all
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

It's possible to get the possible values of a tag given other tags:

<!-- SyntaxHighlightingPlugin -->
FC:/> meta tags SoftwareTag where EvtClass=tth-2l2nbb-hnonbb Datatype=DST
Possible values for SoftwareTag:
v01-16-p03
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

Finally, it's possible to search for files using find. The syntax is similar to the usual shell "find":

<!-- SyntaxHighlightingPlugin -->
FC:/> find . EvtClass=higgs_ffh Datatype=DST-MERGED
Query: {'Datatype': 'DST-MERGED', 'EvtClass': 'higgs_ffh'}
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00003-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00004-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00005-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00006-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00007-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00008-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00009-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00010-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36151.Pvvh_nomu.eL.pR-00011-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36152.Pvvh_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36152.Pvvh_nomu.eR.pL-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36154.Pe1e1h_nomu.eL.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36155.Pe1e1h_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36156.Pe1e1h_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36157.Pe1e1h_nomu.eR.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36159.Pllh_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36160.Pllh_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36163.Pxxh_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36164.Pxxh_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36167.Pyyh_nomu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I36168.Pyyh_nomu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37582.Pvvh_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37582.Pvvh_mumu.eL.pR-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37583.Pvvh_mumu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37583.Pvvh_mumu.eR.pL-00002-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37585.Pe1e1h_mumu.eL.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37586.Pe1e1h_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37587.Pe1e1h_mumu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37590.Pllh_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37591.Pllh_mumu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37594.Pxxh_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37595.Pxxh_mumu.eR.pL-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37598.Pyyh_mumu.eL.pR-00001-DST.slcio
/ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37599.Pyyh_mumu.eR.pL-00001-DST.slcio
QueryTime 0.06 sec
FC:/> 
<!-- end SyntaxHighlightingPlugin -->

The . indicates that one as to look for compatible files in the current directory, and can be replaced by a path, e.g /ilc/prod/ilc/md-dbd/ild to make sure the files only come from this directory and its sub-directories.

It's always possible to have access to the defined meta data of a directory:

<!-- SyntaxHighlightingPlugin -->
FC:/> meta get /ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/                                             
           *Datatype : DST-MERGED
             *Energy : 1tev
      *MachineParams : B1b_ws
       *DetectorType : ILD
            *Machine : ilc
           *EvtClass : higgs_ffh
        !SoftwareTag : v01-16-p03
      *DetectorModel : ILD_o1_v05
FC:/> 
<!-- end SyntaxHighlightingPlugin -->
Where * indicate inherited meta data and ! mean local meta data. Meta data that have no indication are non searchable.

Of course, getting the meta data of a file is possible:

<!-- SyntaxHighlightingPlugin -->
FC:/> meta get /ilc/prod/ilc/mc-dbd/ild/dst-merged/1000-B1b_ws/higgs_ffh/ILD_o1_v05/v01-16-p03/rv01-16-p03.sv01-14-01-p00.mILD_o1_v05.E1000-B1b_ws.I37588.Pe1e1h_mumu.eR.pR-00001-DST.slcio
      NumberOfEvents : 1000
          RandomSeed : None
       BeamParticle1 : e1
       BeamParticle2 : E1
        GenProcessID : 37594
          StartEvent : 0
      PolarizationB1 : L
      PolarizationB2 : R
FC:/> 
<!-- end SyntaxHighlightingPlugin -->
Here, there are no indication (for the moment) of the nature of the meta data (searcable or not) and the user should rely on the meta show output.

There are other useful utilities in this interface that should be looked upon, but the help is enough to grasp the concepts. In particular, users can get their files using get.

The web portal

There is an interface to the File Catalog on the DIRAC web portal at https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/data/MetaCatalog/display but it's still in development and does not allow easy maipulation of data as the CLI permits. It's still possible to interact with it to get file informations.

This interface will be more detailed once it's fully functionnal. Until then, people should get the files using

The Python API

This interface uses directly the underlying python API to get the information. It's to be used when submitting jobs as it's the easiest way of passign the files from the catalog to the job definition. I only demonstrate here the mean to perform a data query.
<!-- SyntaxHighlightingPlugin -->
from DIRAC.Core.Base import Script
Script.parseCommandLine()
from DIRAC.Resources.Catalog.FileCatalogClient import FileCatalogClient

fc = FileCatalogClient()

meta = {}
meta['EvtClass']='higgs_ffh' 
meta['Datatype']='DST-MERGED'

res = fc.findFilesByMetadata(meta)
if not res['OK']:
   print res['Message']

lfns = res['Value']

print "Found %s files" % len(lfns)
for lfn in lfns:
   print lfn
<!-- end SyntaxHighlightingPlugin -->
This will print on screen the result of the meta data query indicated by the dictionary meta. This is typically used when defining a job: its setInputSandbox would contain "LFN:"+lfn to tell DIRAC to get the file before running the job.

The dirac-ilc-find-in-FC script

This is a utility I designed to allow dumping the result of a meta data query to a text file easily, to be used as long as the web portal does not behave the right way. The syntax is the following:

<!-- SyntaxHighlightingPlugin -->
dirac-ilc-find-in-FC /ilc/prod/ EvtClass=higgs_ffh Datatype=DST-MERGED
<!-- end SyntaxHighlightingPlugin -->
The output of this can be redirected to a text file for easy manipulation later.
 

Web monitoring

\ No newline at end of file
Added:
>
>
The web monitoring of jobs happen here: https://ilcdirac.cern.ch/DIRAC/ILC-Production/ilc_user/jobs/JobMonitor/display

Revision 92013-02-12 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 297 to 297
 This makes sure the dependency (here ROOT 5.34) is installed prior to running the script. Any ILCDIRAC supported application can be set here, the full list can be obtained by running dirac-ilc-show-software.

Generation

Changed:
<
<
The generation of events in ILCDIRAC is done using WHIZARD 1.95 (for the moment) for most events, and with PYTHIA directly for =ttbar, WW, and ZZ events. The support for WHZAIRD 2. is being developed, so if you need events produced with WHIZARD 2, you should request help from ilc-dirac@cernNOSPAMPLEASE.ch.
>
>
The generation of events in ILCDIRAC is done using WHIZARD 1.95 (for the moment) for most events, and with PYTHIA directly for ttbar, WW, and ZZ events. The support for WHZAIRD 2. is being developed, so if you need events produced with WHIZARD 2, you should request help from ilc-dirac@cernNOSPAMPLEASE.ch.
 

Whizard 1.95

This generator is used for the CLIC CDR and ILC DBD studies. It has several limitations not discussed here, but the main one being that it requires a dedicated binary containing the process one wants to study. As having an exhaustive list of processes is not possible, only a few (>1000) are available. If you need to use this application, it's a good idea to contact ilc-dirac@cernNOSPAMPLEASE.ch to ask if a given process is available. If not, someone will (or not) add it to WHIZARD and make it available. When WHIZARD 2 is supported by ILCDIRAC, the process can be obtained directly as WHIZARD will be ran "automatically".

Revision 82013-02-11 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 21 to 21
 After that, you are permanently a member of ILCDIRAC, except if you wish to leave or if you change your certification authority (when you change lab) in which case this registration procedure has to be done again, but is simpler as we only need to change your DN in ILCDIRAC registration. For that, send a mail to ilcdirac-register@cernNOSPAMPLEASE.ch announcing you changed your affiliation to another lab and we will update your details.

Once registered in the VO and in DIRAC, you need to create the pem files. For this, export your certificate from your browser in p12 format. How to do this is documented in the HeadFirstTalk slides. This p12 will need to be converted. The DIRAC installation comes with a handy script to convert the p12 in pem format. To get this script, you need either to see the Installing DIRAC section up to the dirac-proxy-init -x or source the bashrc file from your existing local DIRAC installation. Then run

Changed:
<
<
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}% dirac-cert-convert.sh cert.p12
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%dirac-cert-convert.sh cert.p12
 %ENDSYNTAX%

where cert.p12 is the file you obtained from the browser export.

Line: 267 to 266
 %ENDSYNTAX% This last property does not make sense for all applications, in particular for the user application, as DIRAC cannot "guess" what a random binary will produce. If specified in the application steering file AND in the setOutputFile field, then it will behave as expected.
Added:
>
>
The next sections describe the application specific methods (setters). Everything that is described in this section applies for the following, except otherwise indicated.
 

Generic application

Changed:
<
<
A generic application is any executable that is not part of the ILC software chain (but can use some ILC software) and not ROOT (as ROOT has it's own wrapper). I may be used to run a user code, and will serve here as a general framework example, in particular in the relation between jobs and applications.
>
>
A generic application is any executable that is not part of the ILC software chain (but can use some ILC software) and not ROOT (as ROOT has it's own wrapper). I may be used to run a user code.
  A generic application is defined as follows: %SYNTAX{ syntax="python" numbered="off" style="emacs"}%
Line: 276 to 277
 ga = GenericApplication() %ENDSYNTAX%
Changed:
<
<
This definition is completely independent of the Job definition. This is valid for any application.

A generic application, or GenericApplication, has several specific attributes, and attributes that are valid for any application.

>
>
Check the API documentation for the full detail.
 
Added:
>
>
The 3 methods that are specific to this Application are the following:
  • The setter of the script to run:
<!-- SyntaxHighlightingPlugin -->
ga.setScript("somescript_or_binary")
<!-- end SyntaxHighlightingPlugin -->
This can be any executable script (shell or python). Check that the mod (chmod +x) is correct before submission. It can also be an LFN:
  • The arguments to pass to the script
<!-- SyntaxHighlightingPlugin -->
ga.setArguments("some command line arguments")
<!-- end SyntaxHighlightingPlugin -->
The arguments here are passed as they are input here.
  • A dependency to an ILCDIRAC supported application:
<!-- SyntaxHighlightingPlugin -->
ga.setDependency({"ROOT":"5.34"})
<!-- end SyntaxHighlightingPlugin -->
This makes sure the dependency (here ROOT 5.34) is installed prior to running the script. Any ILCDIRAC supported application can be set here, the full list can be obtained by running dirac-ilc-show-software.
 

Generation

Added:
>
>
The generation of events in ILCDIRAC is done using WHIZARD 1.95 (for the moment) for most events, and with PYTHIA directly for =ttbar, WW, and ZZ events. The support for WHZAIRD 2. is being developed, so if you need events produced with WHIZARD 2, you should request help from ilc-dirac@cernNOSPAMPLEASE.ch.

Whizard 1.95

This generator is used for the CLIC CDR and ILC DBD studies. It has several limitations not discussed here, but the main one being that it requires a dedicated binary containing the process one wants to study. As having an exhaustive list of processes is not possible, only a few (>1000) are available. If you need to use this application, it's a good idea to contact ilc-dirac@cernNOSPAMPLEASE.ch to ask if a given process is available. If not, someone will (or not) add it to WHIZARD and make it available. When WHIZARD 2 is supported by ILCDIRAC, the process can be obtained directly as WHIZARD will be ran "automatically".
 
Deleted:
<
<

Whizard

 

Pythia

StdHepCuts

Revision 72013-02-11 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 42 to 42
 

Installing ILCDIRAC

When not having access to a pre installed DIRAC, you need to install it yourself. The procedure is as follows: %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Changed:
<
<
wget -np -O dirac-install https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py --no-check-certificate
>
>
wget -np -O dirac-install https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py --no-check-certificate
 chmod +x dirac-install ./dirac-install -V ILCDIRAC %ENDSYNTAX%
Line: 64 to 66
  Then run the configuration %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
Changed:
<
<
dirac-configure -V ilc -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server --SkipCAChecks
>
>
dirac-configure -V ilc -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server --SkipCAChecks
 %ENDSYNTAX% The volcd01.cern.ch machine will be replaced soon, but this will be announced from the ilc-dirac@cernNOSPAMPLEASE.ch mailing list. When that happens, this command will have to be ran again with the new host (details will be in the mail).
Line: 196 to 199
  This ends the general presentation of the jobs. More information can be found in the API documentation. The next sections will show how the different applications can be configured.
Added:
>
>

ILC applications-job framework

For maximum flexibility in the ILCDIRAC interface, the application-support framework was carefully designed. It's based in the following assumptions:

  • An application has several generic properties
    • It has a name and possibly a version
    • It will be steered with some configuration file
    • It will process a given number of events (for HEP)
    • It will maybe produce some log file
    • It will produce some output file
    • It may process some input data
    • It may have energy dependency (for HEP)
  • An also applciation specific
  • An application is a block of treatment of information
    • It should be easy to "plug" an application in a workflow
  • A Job is only one way to run an application
    • An application should be "standalone"

Those requirements were implemented in the ILCDIRAC Application framework. The corresponding PYTHON implementation is best described in the API documentation. For completeness, we show here a python code that makes use of this framework, although it's just a non functional example. All ILC applications described below can use the same methods.

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Application import Application
ap = Application()
ap.setName("MyName")                  #application name
ap.setVersion("1")                    #version
ap.setLogFile("MylogFile")            #log file name (stdout of the application)
ap.setInputFile("MyInput")            #input file name, can be a list, 
                                      #can contain elements with LFN:
ap.setOutputFile("MyOutput")          #output file name
ap.setSteeringFile("SomeSteeringFile")#steering file (configuration), can have LFN:
ap.setNumberOfEvents(10)              #Obviously... 
ap.setEnergy(3000)                    #Energy
res = job.append(ap)                  #Add this application to the job
if not res['OK']:                     #Catch if there is an error
    print res['Message']              #Print the error message
    #do something, like quit
<!-- end SyntaxHighlightingPlugin -->

Here, job is an instance of the UserJob class described before. It is possible to stack applications one after the other:

<!-- SyntaxHighlightingPlugin -->
ap1 = Application()
ap2 = Application()
res = job.append(ap1)
if not res['OK']:
    print res['Message']
    #do something, like quit
res = job.append(ap2)
if not res['OK']:
    print res['Message']
    #do something, like quit
<!-- end SyntaxHighlightingPlugin -->

It is also possible to chain applications: the second one can get its output from the first one:

<!-- SyntaxHighlightingPlugin -->
ap1 = Application()
ap1.setOutputFile("MyOutput")  # Needed when chaining
ap2 = Application()
ap2.getInputFromApp(ap1)   #This is where the magic happens
res = job.append(ap1)
if not res['OK']:
    print res['Message']
    #do something, like quit
res = job.append(ap2)
if not res['OK']:
    print res['Message']
    #do something, like quit
<!-- end SyntaxHighlightingPlugin -->
This last property does not make sense for all applications, in particular for the user application, as DIRAC cannot "guess" what a random binary will produce. If specified in the application steering file AND in the setOutputFile field, then it will behave as expected.
 

Generic application

Changed:
<
<
A generic application is any executable that is not part of the ILC software chain (but can use some ILC software) and not ROOT (as ROOT has it's own wrapper). The example from the API
>
>
A generic application is any executable that is not part of the ILC software chain (but can use some ILC software) and not ROOT (as ROOT has it's own wrapper). I may be used to run a user code, and will serve here as a general framework example, in particular in the relation between jobs and applications.

A generic application is defined as follows:

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.Applications import GenericApplication
ga = GenericApplication()
<!-- end SyntaxHighlightingPlugin -->

This definition is completely independent of the Job definition. This is valid for any application.

A generic application, or GenericApplication, has several specific attributes, and attributes that are valid for any application.

 

Generation

Revision 62013-02-11 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 29 to 29
 

Getting a DIRAC environment

Changed:
<
<

Using a pre existing installation

>
>
Warning, important The DIRAC environment is usually incompatible with other environments, and therefore often requires a dedicated terminal/shell.

Using a pre existing installation (e.g. at CERN)

 If you rely on someone's installation (or your own) you have normally access to the directory in which ILCDIRAC has been installed. In that directory there is a env file, bashrc that needs sourcing to get the right environment. For example, for CERN users, you can run
<!-- SyntaxHighlightingPlugin -->
source /afs/cern.ch/eng/clic/software/DIRAC/bashrc
<!-- end SyntaxHighlightingPlugin -->
Deleted:
<
<
Warning, important The DIRAC environment is usually incompatible with other environments, and therefore often requires a dedicated terminal/shell.
 Once this file has been sourced, you get access to all the DIRAC and ILCDIRAC commands, as well as the python API. You can proceed to the Job section.

Installing ILCDIRAC

Revision 52013-02-08 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 86 to 87
 This section is a copy of what is presented in https://github.com/DIRACGrid/DIRAC/wiki/JobManagement

The default way of creating jobs in DIRAC is to use the same method as regular GRID jobs, with JDLs. A JDL (or Job Description Language) follows a standard. It's a simple composition of Attribute=Value;. For example, the following is a correctly defined JDL:

Changed:
<
<
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 [ JobName = "Simple_Job"; Executable = "/bin/ls";
Line: 95 to 96
  StdError = "StdErr"; OutputSandbox = {"StdOut","StdErr"}; ]
Changed:
<
<
>
>
%ENDSYNTAX%
 It has a few mandatory parameters indicated above. Many other are possible, but are not indicated here as this is not the main supported way of submitting jobs. Nevertheless if you have a JDL, you may submit it to DIRAC using %SYNTAX{ syntax="bash" numbered="off" style="emacs"}% dirac-wms-job-submit myjdl.jdl
Line: 166 to 167
  The next few lines give some of the UserJob class method use example. The first 2, setName and setJobGroup, can be used for the monitoring: the names are displayed on the monitoring page and the JobGroup can be used to select only the jobs belonging to a given group.
Changed:
<
<
The next 2, setInputSandbox and setOutputSandbox, indicate to the job what files should be shipped to the job working area, and what files should be brought back when retrieving the job outputs. Any file in the setInputSandbox must reside locally (absolute or relative path accepted), or be stored on the GRID, and specified with the LFN (Logical File Name, described in the File Catalog section of this page), where said LFN must be prepended by LFN:. For example, if there is a file /ilc/user/s/someone/file, the file to specify in the setInputSandbox should be LFN:/ilc/user/s/someone/file. As it can be seen in the example, python lists can be used, but not lists of lists. Warning, importantMake sure your sandboxes do not contain such things, as the system cannot correct them. It is a good idea to put on the GRID large input files (or often used) and specify them using the LFN: definition: they make the job submission much faster and make the ILCDIRAC servers happy (no huge amount of data to carry around). How to upload a file to the GRID is described in another part of this document.
>
>
The next 2, setInputSandbox and setOutputSandbox, indicate to the job what files should be shipped to the job working area, and what files should be brought back when retrieving the job outputs. Any file in the setInputSandbox must reside locally (absolute or relative path accepted), or be stored on the GRID, and specified with the LFN (Logical File Name, described in the File Catalog section of this page), where said LFN must be prepended by LFN:. For example, if there is a file /ilc/user/s/someone/file, the file to specify in the setInputSandbox should be LFN:/ilc/user/s/someone/file. As it can be seen in the example, python lists can be used, but not lists of lists. Warning, importantMake sure your sandboxes do not contain such things, as the system cannot correct them. Tip, idea It is a good idea to put on the GRID large input files (or often used) and specify them using the LFN: definition: they make the job submission much faster and make the ILCDIRAC servers happy (no huge amount of data to carry around). How to do this is given in the section Specifying your libraries.
  The setOutputSandbox follows the same style. There are several important things to notice: If a file is missing in the output, DIRAC will mark the job as failed, but the output will be filled with whatever is available among the requested things. If the sandbox is bigger than 10MB, the files will be packed together in a tar ball and shipped to the GRID. Warning, importantIn version ILCDIRAC v16r7p0, it seems that retrieving the output sandbox of a job does not retrieve those files from the GRID. Retrieving job outputs is explained later on this page.
Line: 226 to 228
 

Chaining applications

Added:
>
>
 

Specifying your libraries

Libraries also have dedicated support. This mostly depends on the application, in particular for Marlin, so here I show only the default method, but please check if the application you want to run for specificities.
Line: 236 to 239
 You will copy this libSomething.so into a lib directory. Now, 2 solutions are possible:

1. You are planning on submitting many jobs: I recommend you to put that lib directory on the grid and specify the LFN in the job definition. How to do that?

Changed:
<
<
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 tar czf lib.tar.gz lib/ dirac-dms-add-files /ilc/user/i/initial/some/path/lib.tar.gz lib.tar.gz CERN-SRM
Changed:
<
<
>
>
%ENDSYNTAX%
 The command dirac-dms-add-files will disappear in the next DIRAC release and will be named dirac-dms-add-file. The /ilc/user/... is the LFN (Logical File Name). The i/initial/ part is user specific: you need to use your own DIRAC user name. You can find it in the dirac-proxy-init output, check for username. The some/path/ part is free to you, you can use whatever you want. There are a few limitations: you can not have a total number of subsequent directories greater than 14, and the final file name (here lib.tar.gz) cannot be longer than 128 chars. The last element, CERN-SRM indicates the logical name of the Storage Element on which you wish to upload you file.

Warning, important When running with the CALICE VO, the path has a different beginning: /calice/users/i/initial. Notice the s at users. Also, CERN-SRM is not a valid storage element for CALICE users, so DESY-SRM or IN2P3-SRM must be prefered.

Line: 251 to 254
 Moving a file cannot be done on the GRID: if really needed, you need to get the file (dirac-dms-get-file /ilc/...) then remove it from the GRID (same as above), then re upload it to the new location.

You now have a file on the GRID that you wish to input for a series of jobs. You would use the following:

Changed:
<
<
>
>
%SYNTAX{ syntax="python" numbered="off" style="emacs"}%
 job.setInputSandbox("LFN:/ilc/user/i/initial/some/path/lib.tar.gz")
Changed:
<
<
>
>
%ENDSYNTAX%
 Notice the LFN: part that is used by DIRAC to identify the files that must be downloaded from the grid. If you omit it, the job submission should fail because an input sandbox file will be missing. The fact that it's a tar ball does not matter, it will be untarred automatically.

2. You have only 1 job: Simply input the lib directory like this

Changed:
<
<
>
>
%SYNTAX{ syntax="python" numbered="off" style="emacs"}%
 job.setInputSandbox("lib")
Changed:
<
<
>
>
%ENDSYNTAX%
 If you have a tar ball, you may as well use it:
Changed:
<
<
>
>
%SYNTAX{ syntax="python" numbered="off" style="emacs"}%
 job.setInputSandbox("lib.tar.gz")
Changed:
<
<
>
>
%ENDSYNTAX%
 as any tar ball is automatically untarred in the job running directory.

The reason for the different behavior between the 2 cases is because every job submitted uploads its sandbox to the main DIRAC servers, filling its hard drive. All files are sent but those that start with LFN:.

Revision 42013-02-08 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

ILCDIRAC for users

Line: 20 to 20
  After that, you are permanently a member of ILCDIRAC, except if you wish to leave or if you change your certification authority (when you change lab) in which case this registration procedure has to be done again, but is simpler as we only need to change your DN in ILCDIRAC registration. For that, send a mail to ilcdirac-register@cernNOSPAMPLEASE.ch announcing you changed your affiliation to another lab and we will update your details.
Changed:
<
<
Once registered in the VO and in DIRAC, you need to create the pem files. For this, export your certificate from your browser in p12 format. How to do this is documented in the HeadFirstTalk slides. This p12 will need to be converted. The DIRAC installation comes with a handy script to convert the p12 in pem format. To get this script, you need either to see the Installing DIRAC section up to the dirac-proxy-init -x or source the bashrc file from your existing local DIRAC installation. Then run
>
>
Once registered in the VO and in DIRAC, you need to create the pem files. For this, export your certificate from your browser in p12 format. How to do this is documented in the HeadFirstTalk slides. This p12 will need to be converted. The DIRAC installation comes with a handy script to convert the p12 in pem format. To get this script, you need either to see the Installing DIRAC section up to the dirac-proxy-init -x or source the bashrc file from your existing local DIRAC installation. Then run %SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 dirac-cert-convert.sh cert.p12
Changed:
<
<
>
>
%ENDSYNTAX%
 where cert.p12 is the file you obtained from the browser export.

Getting a DIRAC environment

Using a pre existing installation

If you rely on someone's installation (or your own) you have normally access to the directory in which ILCDIRAC has been installed. In that directory there is a env file, bashrc that needs sourcing to get the right environment. For example, for CERN users, you can run
Changed:
<
<
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 source /afs/cern.ch/eng/clic/software/DIRAC/bashrc
Changed:
<
<
>
>
%ENDSYNTAX%
  Warning, important The DIRAC environment is usually incompatible with other environments, and therefore often requires a dedicated terminal/shell.
Line: 40 to 40
 

Installing ILCDIRAC

When not having access to a pre installed DIRAC, you need to install it yourself. The procedure is as follows:
Changed:
<
<
wget -np -O dirac-install \
https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py \
--no-check-certificate
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}% wget -np -O dirac-install https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py --no-check-certificate
 chmod +x dirac-install ./dirac-install -V ILCDIRAC
Changed:
<
<
>
>
%ENDSYNTAX%
 This also creates the bashrc file needed to get the DIRAC environment.

Then you need to configure DIRAC (make the client know where to get the services from). This requires the presence of a valid grid proxy. 2 choices:

1. You have a grid UI available and the pem files: in another session, source the corresponding environment file and run

Changed:
<
<
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 grid-proxy-init
Changed:
<
<
>
>
%ENDSYNTAX%
 2. You don't have a grid UI installed nor the pem files. Then you can use
Changed:
<
<
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 source bashrc dirac-cert-convert.sh grid.p12 dirac-proxy-init -x
Changed:
<
<
>
>
%ENDSYNTAX%
 where grid.p12 is the file you got from the web browser export. If you already have the pem files, this can be skipped. The last command might result in an error, but it does not matter normally as the proxy should have been created.

Then run the configuration

Changed:
<
<
dirac-configure -V ilc -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server \
--SkipCAChecks
>
>
<!-- SyntaxHighlightingPlugin -->
dirac-configure -V ilc -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server --SkipCAChecks
<!-- end SyntaxHighlightingPlugin -->
 The volcd01.cern.ch machine will be replaced soon, but this will be announced from the ilc-dirac@cernNOSPAMPLEASE.ch mailing list. When that happens, this command will have to be ran again with the new host (details will be in the mail).

In principle, if this commands runs fine, you should be able to got to the next section.

Line: 76 to 73
 

Getting a proxy

Once you have sourced the DIRAC environment file, you can run
Changed:
<
<
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 dirac-proxy-init -g ilc_user
Changed:
<
<
>
>
%ENDSYNTAX%
 or
Changed:
<
<
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 dirac-proxy-init -g calice_user
Changed:
<
<
>
>
%ENDSYNTAX%
 depending on the VO you want a proxy for. The 2 VOs having access to different resources, it's sometimes needed to get the right proxy (in particular for data access). Also, this has an effect on the grid sites on which one can run, for instance, only the DESY-HH site is available to the CALICE users (in ILCDIRAC, and for the moment).

Your first DIRAC job

Line: 100 to 97
 ] It has a few mandatory parameters indicated above. Many other are possible, but are not indicated here as this is not the main supported way of submitting jobs. Nevertheless if you have a JDL, you may submit it to DIRAC using
Changed:
<
<
>
>
%SYNTAX{ syntax="bash" numbered="off" style="emacs"}%
 dirac-wms-job-submit myjdl.jdl
Changed:
<
<
>
>
%ENDSYNTAX%
 This will return you a job ID. If using that method of submission, you may want to store those job IDs in some text file as they are needed to obtain their output.

ILC specific jobs

Line: 113 to 110
 

Job types and basic job definition

In ILCDIRAC, there are 2 main job types: User jobs and Production jobs. Here are only covered the User ones. Let's start with a generic example that doesn't do anything:
Changed:
<
<
>
>
%SYNTAX{ syntax="python" numbered="off" style="emacs"}%
 from DIRAC.Core.Base import Script Script.parseCommandLine()
Deleted:
<
<
from UserJob import UserJob
 from DiracILC import DiracILC
Deleted:
<
<
 dirac = DiracILC(True,"some_job_repository.rep")
Added:
>
>
from UserJob import UserJob
 job = UserJob() job.setName("MyJobName") job.setJobGroup("Agroup")
Line: 129 to 127
 job.setOutputSandbox(["fileout1","fileout2"]) job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM") print job.submit(dirac)
Changed:
<
<
>
>
%ENDSYNTAX%
 If submitted as is, it will fail because the different files are not there.
Added:
>
>
Let's go through the individual lines. The first 2:
<!-- SyntaxHighlightingPlugin -->
from DIRAC.Core.Base import Script
Script.parseCommandLine()
<!-- end SyntaxHighlightingPlugin -->
are mandatory to get the DIRAC environment know to the script (This is oversimplifying things but enough). For example, this way the different services that are used get their server addresses set. This also initializes many internal DIRAC utilities that ILCDIRAC makes use of (logger functionality for example),

The following 2 line

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.DiracILC import DiracILC
dirac = DiracILC(True,"some_job_repository.rep")
<!-- end SyntaxHighlightingPlugin -->
take care of importing and creating a DiracILC object. This class is the "job receiver" class, as the last line job.submit(dirac) indicates. It's needed as it makes sure that all the specified inputs (if any) are actually available. It also checks that all requested software (more on that later) is available. Several additional utilities are provided by the inheritance of DiracILC from the main Dirac class (see API doc), not discussed here. Finally, it provides the interface to the Job Repository. This is a text file, in this case some_job_repository.rep, that has a special structure: it hold the jobIDs of the submitted jobs, and their properties (have a look once you have submitted a job). It's very important to keep this file safe as it is used to retrieve the job outputs easily (more on that later). A good way to use this functionality is to have a different file name per activity,

The next few lines

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.UserJob import UserJob
job = UserJob()
job.setName("MyJobName")
job.setJobGroup("Agroup")
job.setInputSandbox(["file1","file2"])
job.setOutputSandbox(["fileout1","fileout2"])
job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
<!-- end SyntaxHighlightingPlugin -->
are needed for an actual job definition. Again, in this example, the job doesn't do anything, and will likely fail if submitted: the specified inputs are not available (more on that later).

The process of job definition starts with

<!-- SyntaxHighlightingPlugin -->
from ILCDIRAC.Interfaces.API.NewInterface.UserJob import UserJob
job = UserJob()
<!-- end SyntaxHighlightingPlugin -->
where you instruct ILCDIRAC what type of job to use. This has little visible consequences for the users, but is making important choices for the internal functionality. Warning, importantUsers should ALWAYS use a UserJob unless explicitly asked or recommended.

The next few lines give some of the UserJob class method use example. The first 2, setName and setJobGroup, can be used for the monitoring: the names are displayed on the monitoring page and the JobGroup can be used to select only the jobs belonging to a given group.

The next 2, setInputSandbox and setOutputSandbox, indicate to the job what files should be shipped to the job working area, and what files should be brought back when retrieving the job outputs. Any file in the setInputSandbox must reside locally (absolute or relative path accepted), or be stored on the GRID, and specified with the LFN (Logical File Name, described in the File Catalog section of this page), where said LFN must be prepended by LFN:. For example, if there is a file /ilc/user/s/someone/file, the file to specify in the setInputSandbox should be LFN:/ilc/user/s/someone/file. As it can be seen in the example, python lists can be used, but not lists of lists. Warning, importantMake sure your sandboxes do not contain such things, as the system cannot correct them. It is a good idea to put on the GRID large input files (or often used) and specify them using the LFN: definition: they make the job submission much faster and make the ILCDIRAC servers happy (no huge amount of data to carry around). How to upload a file to the GRID is described in another part of this document.

The setOutputSandbox follows the same style. There are several important things to notice: If a file is missing in the output, DIRAC will mark the job as failed, but the output will be filled with whatever is available among the requested things. If the sandbox is bigger than 10MB, the files will be packed together in a tar ball and shipped to the GRID. Warning, importantIn version ILCDIRAC v16r7p0, it seems that retrieving the output sandbox of a job does not retrieve those files from the GRID. Retrieving job outputs is explained later on this page.

The last part of this section is

<!-- SyntaxHighlightingPlugin -->
job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
<!-- end SyntaxHighlightingPlugin -->
This is used when a user wants to keep some files on the GRID for further processing or for easy sharing with others. It implies that the specified files somefile1 and somefile2 will be uploaded to the GRID under the automatically built LFN /ilc/user/u/username/some/path/. u and username indicate the DIRAC user name of the user that created the job (u is the initial). The some/path part of the LFN comes from the second element in the method call above. Finally, the storage element to store the files can be specified by passing its logical name in the last element of the method call. In this example, e use CERN-SRM, which is the default storage for ILCDIRAC. We have several sites that support us and that can be used:
CERN-SRM (default)
DESY-SRM
RAL-SRM
KEK-SRM
IN2P3-SRM (inactive because full)
PNNL-SRM
SLAC-SRM
We recommend you use a storage element that's close enough geographically to you.

Finally, the line

<!-- SyntaxHighlightingPlugin -->
print job.submit(dirac)
<!-- end SyntaxHighlightingPlugin -->
submits the job, and applies the checking procedure defined internally. The print statement allows to see on screen the job ID. You can check that this job ID is in the some_job_repository.rep file.

This ends the general presentation of the jobs. More information can be found in the API documentation. The next sections will show how the different applications can be configured.

Generic application

A generic application is any executable that is not part of the ILC software chain (but can use some ILC software) and not ROOT (as ROOT has it's own wrapper). The example from the API

Generation

Whizard

Pythia

StdHepCuts

Simulation

Mokka

SLIC

Reconstruction

Marlin

LCSIM

Adding the Overlay

ROOT applications

Other applications

GetSRMFile

SLCIOSplit

SLCIOConcatenate

StdHepSplit

CheckCollections

Chaining applications

 

Specifying your libraries

Libraries also have dedicated support. This mostly depends on the application, in particular for Marlin, so here I show only the default method, but please check if the application you want to run for specificities.
Line: 148 to 242
  The command dirac-dms-add-files will disappear in the next DIRAC release and will be named dirac-dms-add-file. The /ilc/user/... is the LFN (Logical File Name). The i/initial/ part is user specific: you need to use your own DIRAC user name. You can find it in the dirac-proxy-init output, check for username. The some/path/ part is free to you, you can use whatever you want. There are a few limitations: you can not have a total number of subsequent directories greater than 14, and the final file name (here lib.tar.gz) cannot be longer than 128 chars. The last element, CERN-SRM indicates the logical name of the Storage Element on which you wish to upload you file.
Changed:
<
<
Warning, important When running with the CALICE VO, the path has a different beginning: /calice/users/i/initial. Notice the s at users. Also, CERN-SRM is not a valid storage element for CALICE users, so DESY-SRM or IN2P3-SRM must be prefered.
>
>
Warning, important When running with the CALICE VO, the path has a different beginning: /calice/users/i/initial. Notice the s at users. Also, CERN-SRM is not a valid storage element for CALICE users, so DESY-SRM or IN2P3-SRM must be prefered.
  This LFN is registered in the DIRAC File Catalog, so it can now be used in the job definition.
Line: 174 to 268
  The reason for the different behavior between the 2 cases is because every job submitted uploads its sandbox to the main DIRAC servers, filling its hard drive. All files are sent but those that start with LFN:.
Deleted:
<
<

Generic application

A generic application is any executable that is not part of the ILC software chain (but can use some ILC software) and not ROOT (as ROOT has it's own wrapper). The example from the API

Generation

Whizard

Pythia

StdHepCuts

Simulation

Mokka

SLIC

Reconstruction

Marlin

LCSIM

Adding the Overlay

ROOT applications

Other applications

GetSRMFile

SLCIOSplit

SLCIOConcatenate

StdHepSplit

CheckCollections

Chaining applications

 

Getting the output

Monitoring the jobs

Revision 32013-01-30 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"
Changed:
<
<

Dirac for users

>
>

ILCDIRAC for users

 

Getting the certificate, getting registered in ILCDIRAC

Line: 37 to 41
 

Installing ILCDIRAC

When not having access to a pre installed DIRAC, you need to install it yourself. The procedure is as follows:
Changed:
<
<
wget -np -O dirac-install https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py --no-check-certificate
>
>
wget -np -O dirac-install https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py --no-check-certificate
 chmod +x dirac-install ./dirac-install -V ILCDIRAC
Line: 59 to 65
  Then run the configuration
Changed:
<
<
dirac-configure -V ilc -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server --SkipCAChecks
>
>
dirac-configure -V ilc -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server --SkipCAChecks
  The volcd01.cern.ch machine will be replaced soon, but this will be announced from the ilc-dirac@cernNOSPAMPLEASE.ch mailing list. When that happens, this command will have to be ran again with the new host (details will be in the mail).
Line: 190 to 197
 

ROOT applications

Added:
>
>

Other applications

GetSRMFile

SLCIOSplit

SLCIOConcatenate

StdHepSplit

CheckCollections

 

Chaining applications

Getting the output

Revision 22013-01-30 - StephanePoss

Line: 1 to 1
 
META TOPICPARENT name="DiracUsage"

Dirac for users

Line: 79 to 79
 depending on the VO you want a proxy for. The 2 VOs having access to different resources, it's sometimes needed to get the right proxy (in particular for data access). Also, this has an effect on the grid sites on which one can run, for instance, only the DESY-HH site is available to the CALICE users (in ILCDIRAC, and for the moment).

Your first DIRAC job

Added:
>
>
This section is a copy of what is presented in https://github.com/DIRACGrid/DIRAC/wiki/JobManagement

The default way of creating jobs in DIRAC is to use the same method as regular GRID jobs, with JDLs. A JDL (or Job Description Language) follows a standard. It's a simple composition of Attribute=Value;. For example, the following is a correctly defined JDL:

[
  JobName = "Simple_Job";
  Executable = "/bin/ls";
  Arguments = "-ltr";
  StdOutput = "StdOut";
  StdError = "StdErr";
  OutputSandbox = {"StdOut","StdErr"};
]
It has a few mandatory parameters indicated above. Many other are possible, but are not indicated here as this is not the main supported way of submitting jobs. Nevertheless if you have a JDL, you may submit it to DIRAC using
dirac-wms-job-submit myjdl.jdl
This will return you a job ID. If using that method of submission, you may want to store those job IDs in some text file as they are needed to obtain their output.
 

ILC specific jobs

Added:
>
>
This section shows the preferred way of creating jobs in ILCDIRAC. This uses the PYTHON API. To use it, some basic knowledge of object oriented (OO) programming is needed, as PYTHON is a completely OO language. The examples shown below use the API described in http://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/.

The philosophy of the ILCDIRAC interface is the following: a user must care about what he wants to run, but not how. That's why we provide the interface to all ILC applications that will prepare the runtime environments and execute them. The users need not to care about environment variables, directories to look for, etc. To run an application in ILCDIRAC, there are only very few mandatory parameters, in particular those that would be passed on the command line to call a pre installed application (example: the steering file for Marlin). Moreover, for example, input files do not need to be changed by the user to match what the job is going to run on: this is changed automatically (by default) in the steering files. The idea is that a user would test his run locally, then pass directly the steering file to the job definition without having to do any modification.

Job types and basic job definition

In ILCDIRAC, there are 2 main job types: User jobs and Production jobs. Here are only covered the User ones. Let's start with a generic example that doesn't do anything:
from DIRAC.Core.Base import Script
Script.parseCommandLine()

from ILCDIRAC.Interfaces.API.NewInterface.UserJob import UserJob
from ILCDIRAC.Interfaces.API.DiracILC import DiracILC

dirac = DiracILC(True,"some_job_repository.rep")
job = UserJob()
job.setName("MyJobName")
job.setJobGroup("Agroup")
job.setCPUTime(86400)
job.setInputSandbox(["file1","file2"])
job.setOutputSandbox(["fileout1","fileout2"])
job.setOutputData(["somefile1","somefile2"],"some/path","CERN-SRM")
print job.submit(dirac)
If submitted as is, it will fail because the different files are not there.

Specifying your libraries

Libraries also have dedicated support. This mostly depends on the application, in particular for Marlin, so here I show only the default method, but please check if the application you want to run for specificities.

All applications come with their own dependencies, so a user does not need to take care of those. He should only take care of those libraries that are not part of the software stack.

Let's say you have an application that depends in libSomething.so that is not a default library (in the Marlin case, one case replace existing processors, so this is covered in the Marlin section).

You will copy this libSomething.so into a lib directory. Now, 2 solutions are possible:

1. You are planning on submitting many jobs: I recommend you to put that lib directory on the grid and specify the LFN in the job definition. How to do that?

tar czf lib.tar.gz lib/
dirac-dms-add-files /ilc/user/i/initial/some/path/lib.tar.gz lib.tar.gz CERN-SRM
The command dirac-dms-add-files will disappear in the next DIRAC release and will be named dirac-dms-add-file. The /ilc/user/... is the LFN (Logical File Name). The i/initial/ part is user specific: you need to use your own DIRAC user name. You can find it in the dirac-proxy-init output, check for username. The some/path/ part is free to you, you can use whatever you want. There are a few limitations: you can not have a total number of subsequent directories greater than 14, and the final file name (here lib.tar.gz) cannot be longer than 128 chars. The last element, CERN-SRM indicates the logical name of the Storage Element on which you wish to upload you file.

Warning, important When running with the CALICE VO, the path has a different beginning: /calice/users/i/initial. Notice the s at users. Also, CERN-SRM is not a valid storage element for CALICE users, so DESY-SRM or IN2P3-SRM must be prefered.

This LFN is registered in the DIRAC File Catalog, so it can now be used in the job definition.

If you wish to replace the file, you cannot overwrite the file, you need first to issue a dirac-dms-remove-files /ilc/user/i/initial/some/path/lib.tar.gz then re upload.

Moving a file cannot be done on the GRID: if really needed, you need to get the file (dirac-dms-get-file /ilc/...) then remove it from the GRID (same as above), then re upload it to the new location.

You now have a file on the GRID that you wish to input for a series of jobs. You would use the following:

job.setInputSandbox("LFN:/ilc/user/i/initial/some/path/lib.tar.gz")
Notice the LFN: part that is used by DIRAC to identify the files that must be downloaded from the grid. If you omit it, the job submission should fail because an input sandbox file will be missing. The fact that it's a tar ball does not matter, it will be untarred automatically.

2. You have only 1 job: Simply input the lib directory like this

job.setInputSandbox("lib")
If you have a tar ball, you may as well use it:
job.setInputSandbox("lib.tar.gz")
as any tar ball is automatically untarred in the job running directory.

The reason for the different behavior between the 2 cases is because every job submitted uploads its sandbox to the main DIRAC servers, filling its hard drive. All files are sent but those that start with LFN:.

Generic application

A generic application is any executable that is not part of the ILC software chain (but can use some ILC software) and not ROOT (as ROOT has it's own wrapper). The example from the API
 

Generation

Whizard

Pythia

Added:
>
>

StdHepCuts

 

Simulation

Mokka

Line: 99 to 188
 

Adding the Overlay

Added:
>
>

ROOT applications

Chaining applications

 

Getting the output

Monitoring the jobs

Revision 12013-01-28 - StephanePoss

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="DiracUsage"

Dirac for users

Getting the certificate, getting registered in ILCDIRAC

Get a grid certificate from your preferred certification authority. At CERN (for staff and users) that is ca.cern.ch. See your local grid expert (there is usually at least one GRID user in every lab nowadays) for the name of the certification authority corresponding to your institute.

Then, depending on your VO, either go to https://grid-voms.desy.de:8443/voms/ilc or https://grid-voms.desy.de:8443/voms/calice or both to register yourself as a VO member.

Once you have been allowed in the VO, you need to send a mail to ilcdirac-register@cernNOSPAMPLEASE.ch stating your name, institute, corresponding contact (supervisor is usually enough) and project you are working on. We need this for security reasons. Then we will register you in ILCDIRAC as a user, and add you to the ilc-dirac@cernNOSPAMPLEASE.ch mailing list, and send you a confirmation mail. In the process, you will get a directory in the File Catalog, see below.

After that, you are permanently a member of ILCDIRAC, except if you wish to leave or if you change your certification authority (when you change lab) in which case this registration procedure has to be done again, but is simpler as we only need to change your DN in ILCDIRAC registration. For that, send a mail to ilcdirac-register@cernNOSPAMPLEASE.ch announcing you changed your affiliation to another lab and we will update your details.

Once registered in the VO and in DIRAC, you need to create the pem files. For this, export your certificate from your browser in p12 format. How to do this is documented in the HeadFirstTalk slides. This p12 will need to be converted. The DIRAC installation comes with a handy script to convert the p12 in pem format. To get this script, you need either to see the Installing DIRAC section up to the dirac-proxy-init -x or source the bashrc file from your existing local DIRAC installation. Then run

dirac-cert-convert.sh cert.p12
where cert.p12 is the file you obtained from the browser export.

Getting a DIRAC environment

Using a pre existing installation

If you rely on someone's installation (or your own) you have normally access to the directory in which ILCDIRAC has been installed. In that directory there is a env file, bashrc that needs sourcing to get the right environment. For example, for CERN users, you can run
source /afs/cern.ch/eng/clic/software/DIRAC/bashrc

Warning, important The DIRAC environment is usually incompatible with other environments, and therefore often requires a dedicated terminal/shell.

Once this file has been sourced, you get access to all the DIRAC and ILCDIRAC commands, as well as the python API. You can proceed to the Job section.

Installing ILCDIRAC

When not having access to a pre installed DIRAC, you need to install it yourself. The procedure is as follows:
wget -np -O dirac-install https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py --no-check-certificate
chmod +x dirac-install
./dirac-install -V ILCDIRAC
This also creates the bashrc file needed to get the DIRAC environment.

Then you need to configure DIRAC (make the client know where to get the services from). This requires the presence of a valid grid proxy. 2 choices:

1. You have a grid UI available and the pem files: in another session, source the corresponding environment file and run

grid-proxy-init
2. You don't have a grid UI installed nor the pem files. Then you can use
source bashrc
dirac-cert-convert.sh grid.p12
dirac-proxy-init -x
where grid.p12 is the file you got from the web browser export. If you already have the pem files, this can be skipped. The last command might result in an error, but it does not matter normally as the proxy should have been created.

Then run the configuration

dirac-configure -V ilc -S ILC-Production -C dips://volcd01.cern.ch:9135/Configuration/Server --SkipCAChecks
The volcd01.cern.ch machine will be replaced soon, but this will be announced from the ilc-dirac@cernNOSPAMPLEASE.ch mailing list. When that happens, this command will have to be ran again with the new host (details will be in the mail).

In principle, if this commands runs fine, you should be able to got to the next section.

Don't forget to source the bashrc file whenever using DIRAC.

Getting a proxy

Once you have sourced the DIRAC environment file, you can run
dirac-proxy-init -g ilc_user
or
dirac-proxy-init -g calice_user
depending on the VO you want a proxy for. The 2 VOs having access to different resources, it's sometimes needed to get the right proxy (in particular for data access). Also, this has an effect on the grid sites on which one can run, for instance, only the DESY-HH site is available to the CALICE users (in ILCDIRAC, and for the moment).

Your first DIRAC job

ILC specific jobs

Generation

Whizard

Pythia

Simulation

Mokka

SLIC

Reconstruction

Marlin

LCSIM

Adding the Overlay

Getting the output

Monitoring the jobs

File Catalog business

Web monitoring

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback