Creating a job

Simple example

The API allows the creation of DIRAC jobs using the Job object. To demonstrate the use of the Job object a simple example job is given below:
 
from DIRAC.Interfaces.API.Job import Job
j = Job()
j.setExecutable('ls',arguments='-al',logFile='ls.log')
j.execute()
The first line of this script imports the Job class from the Interfaces.API package. The second line creates a Job object and names it 'j'. The third line sets the executable of this job to be the unix command 'ls'. The setExecutable method also can take additional 'arguments' (in this example '-al' to give a long listing of all contents) and 'logFile' to provide the requested location of the log file. The final line executes the contents of the job object.

If this example script is created (here we call it lsExample.py) we can run it using the python interpreter:

> cat lsExample.py
from DIRAC.Interfaces.API.Job import Job
j = Job()
j.setExecutable('ls',arguments='-al',logFile='ls.log')
j.execute()

> python lsExample.py
Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1']
StepInstance creating module instance  ScriptStep1  of type Script
2009-09-30 13:46:26 UTC Framework/Script  INFO: Script Module Instance Name: CodeSegment
2009-09-30 13:46:26 UTC Framework/Script  INFO: Command is: ls -al
2009-09-30 13:46:26 UTC Framework/Script  INFO: ls execution completed with status 0
2009-09-30 13:46:26 UTC Framework/Script  INFO: Output written to Script1_ls.log, execution complete.
The logging information printed above tell us that the output of the job was written to 'Script1_ls.log'. The contents of this file are:
> cat Script1_ls.log 
<<<<<<<<<< ls Standard Output >>>>>>>>>>

total 5
drwxr-xr-x 2 acsmith z5 2048 Sep 30 15:48 .
drwxr-xr-x 8 acsmith z5 2048 Sep 30 15:48 ..
-rwxr-xr-x 1 acsmith z5  340 Sep 30 15:46 lsExample.py
Therefore our simple job has created the same output as the 'ls -al' command would.

Multiple step jobs

It may be necessary to add multiple step in a job to achieve the required functionality. To demonstrate how this works we add two executables: the first to echo "hello world" into a file and the second to display the contents of the new file. This can be represented in the following script:
> cat multiStep.py 
from DIRAC.Interfaces.API.Job import Job
j = Job()
j.setExecutable('echo "hello world" > test.txt')
j.setExecutable('cat',arguments='test.txt')
j.execute()

> python multiStep.py 
Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep2', 'ScriptStep1']
StepInstance creating module instance  ScriptStep1  of type Script
2009-09-30 16:13:43 UTC Framework/Script  INFO: Script Module Instance Name: CodeSegment
2009-09-30 16:13:43 UTC Framework/Script  INFO: Command is: echo "hello world" > test.txt
2009-09-30 16:13:43 UTC Framework/Script  INFO: echo "hello world" > test.txt execution completed with status 0
2009-09-30 16:13:43 UTC Framework/Script  INFO: Output written to Script1_CodeOutput.log, execution complete.
Executing StepInstance RunScriptStep2 of type ScriptStep2 ['ScriptStep2', 'ScriptStep1']
StepInstance creating module instance  ScriptStep2  of type Script
2009-09-30 16:13:43 UTC Framework/Script  INFO: Script Module Instance Name: CodeSegment
2009-09-30 16:13:43 UTC Framework/Script  INFO: Command is: cat test.txt
2009-09-30 16:13:43 UTC Framework/Script  INFO: cat execution completed with status 0
2009-09-30 16:13:43 UTC Framework/Script  INFO: Output written to Script2_CodeOutput.log, execution complete.

> cat Script2_CodeOutput.log
<<<<<<<<<< cat Standard Output >>>>>>>>>>

hello world
There is no limit to the number of steps that can be added with the only requirement being that they are added in the order they should be executed.

Handling input and output data

If your job requires files to be present in the working directory of the job you should use the input sandbox mechanism. By using the 'setInputSandbox' method DIRAC will be instructed to ensure the files provided make it to the eventual working directory of the job. This can be done with the following syntax:

j.setInputSandbox(['Input.options','*.txt','lib/'])

It is possible to provide files, directories and use wildcards in the provided list. Similarly if you want some files produced by your job to be returned with the job output then they should be included in the output sandbox by using the following:

j.setOutputSandbox(['*.log','summary.data'])

If your job needs access to input data which is stored on Grid storage elements your job must be instructed to access this data. This can be done by giving your list of input LFNs to the 'setInputData' method:

j.setInputData(['/my/logical/file/name1','/my/logical/file/name2'])
When providing this option the job will run at sites which already have this data present.

If the data you produce is large and you want to share it with other users and/or you want to use it as input to a later job you should consider setting it as output data. Once the job finished these data files will be uploaded to Grid storage. To set output data and group these files in a logical directory you can use:

j.setOutputData(['output1.data','output2.data'],OutputPath='MyFirstAnalysis')
This will upload the output1.data and output2.data files with logical file names that contains 'MyFirstAnalysis' in the path. e.g. /vo/user/initial/username/MyFirstAnalysis/.../output1.data. If you want to upload the file to a specific storage element (perhaps at your local institute) you can use the additional 'OutputSE' option. For example:

j.setOutputData(['output1.data','output2.data'],OutputSE=['Institute-disk'],OutputPath='MyFirstAnalysis')
This practise is not recommended as failover mechanisms (which insure your data will be uploaded) are circumvented.

Setting the computing resources to use

To ensure that your job is executed on the correct operating system and system architecture you can instruct the job to use a particular system configuration using the 'setSystemConfig' option. For example to use 32bit SLC4 do the following:
j.setSystemConfig("slc4_ia32_gcc34")

If submitted to run on the Grid your job will be eligible on a variety of computing resources. To best match your job to the computing resources DIRAC should be instructed on the estimated CPU time required for the job. If this is set too low the job may be killed by the batch system while if it is too high it will not be eligible for all candidate resources. Analysis jobs with input data should be dimensioned (by increasing and decreasing the number of input files) to run for several hours. Similarly the number of iterations used in a fitting application or events to be generated in a monte carlo application should be understood and the CPU requirement determined. This CPU requirement (in seconds) can be set using the 'setCPUTime' option, for example for 6 hours of CPU (6*60*60):

j.setCPUTime(21600)

By submitting your job to DIRAC you simplify your use of the Grid by delegating complicated brokering and scheduling decisions to DIRAC. If you want to target a job to a particular site (not recommended) then you can use the 'setDestination' method. This takes the target site name as the single argument:

j.setDestination('LCG.CERN.ch')

In the same way that you can select the site you wish to run at you may also tell DIRAC not to use specific sites. This can be useful when you discover that your input data file is corrupt at a given site or if the software you require is not installed there. This is done using the 'setBannedSites' option:

j.setBannedSites(['LCG.CNAF.it','LCG.CNAF-t2.it'])

Configurable Job parameters

When running multiple jobs it can be useful to distinguish them by giving each job an individual name. This can be done using the 'setName' method. This option is usually used in conjuction with a counter to give each job submitted an individual name:
for i in range(20):
  j.setName('MyJob_%d' % i)

As you have seen above when a job is executed there is logging information printed to the standard output. The log level is set by default to INFO. If you wish to have more/less verbosity you can change the logging level of the job using the 'setLogLevel' option:

j.setLogLevel('DEBUG')
If you want more output than is provided by INFO then the DEBUG level can be used. If you want to have less then you can use WARN, VERBOSE and finally ALWAYS for the lowest amount of output.

If your executable needs particular environment variables to be present when the job is executed these can be provided using the 'setExecutionEnv' option. To demonstrate this the following script will echo an environment variable at execution time:

from DIRAC.Interfaces.API.Job import Job
j = Job()
j.setExecutionEnv({'MYVARIABLE':'TOTO'})
j.setExecutable('echo $MYVARIABLE')
j.execute()
This will produce TOTO in the standard output of the step when executed.

What is JDL?

The job object is a convenient way to manipulate parameters of a job using a programatic interface. The job you create can be displayed in "human-readable" Job Description Langugage (JDL) which has become a defacto standard on the Grid. Any DIRAC job can be displayed in JDL by using the '_toJDL' method of the job:

> cat toJDL.py
from DIRAC.Interfaces.API.Job import Job
j = Job()
j.setName('MyFirstJob')
j.setOutputSandbox(['*.log','summary.data'])
j.setInputData(['/my/logical/file/name1','/my/logical/file/name2'])
j.setOutputData(['output1.data','output2.data'],OutputPath='MyFirstAnalysis')
j.setSystemConfig("slc4_ia32_gcc34")
j.setCPUTime(21600)
j.setDestination('LCG.CERN.ch')
j.setBannedSites(['LCG.CNAF.it','LCG.CNAF-t2.it'])
j.setLogLevel('DEBUG')
j.setExecutionEnv({'MYVARIABLE':'TOTO'})
j.setExecutable('echo $MYVARIABLE')
print j._toJDL()
>  python toJDL.py
 
    Origin = "DIRAC";
    Priority = "1";
    Executable = "$DIRACROOT/scripts/jobexec";
    ExecutionEnvironment = "MYVARIABLE=TOTO";
    StdError = "std.err";
    LogLevel = "DEBUG";
    BannedSites = 
        {
            "LCG.CNAF.it",
            "LCG.CNAF-t2.it"
        };
    StdOutput = "std.out";
    Site = "LCG.CERN.ch";
    SystemConfig = "slc4_ia32_gcc34";
    JobName = "MyFirstJob";
    OutputPath = "MyFirstAnalysis";
    InputSandbox = "jobDescription.xml";
    Arguments = "jobDescription.xml -o LogLevel=DEBUG";
    JobGroup = "lhcb";
    OutputSandbox = 
        {
            "*.log",
            "summary.data",
            "Script1_CodeOutput.log",
            "std.err",
            "std.out"
        };
    MaxCPUTime = "21600";
    InputData = 
        {
            "LFN:/my/logical/file/name1",
            "LFN:/my/logical/file/name2"
        };
    JobType = "User";
    OutputData = 
        {
            "output1.data",
            "output2.data"
        }
As you can see the parameters added to the job object are represented in the JDL.

Digging deeper...

To find other available options one can do the following (this time using the interactive python prompt):

> python
Python 2.5 (r25:51908, Oct 18 2007, 16:26:11) 
[GCC 3.4.6 20060404 (Red Hat 3.4.6-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from DIRAC.Interfaces.API.Job import Job
>>> j = Job()
>>> dir(j)
['_Job__checkSiteIsValid', '_Job__getScriptStep', '_Job__setJobDefaults', '__doc__', '__init__', '__module__', '_addJDLParameter', '_addParameter', '_dumpParameters', '_getParameters', '_resolveInputSandbox', '_setParamValue', '_setSoftwareTags', '_toJDL', '_toXML', 'addToInputData', 'addToInputSandbox', 'addToOutputSandbox', 'createCode', 'dbg', 'executable', 'execute', 'group', 'log', 'logLevel', 'name', 'origin', 'owner', 'priority', 'reqParams', 'script', 'section', 'selectSetup', 'sendMail', 'setBannedSites', 'setCPUTime', 'setConfigArgs', 'setDestination', 'setExecutable', 'setExecutionEnv', 'setInputData', 'setInputSandbox', 'setJobGroup', 'setLogLevel', 'setMode', 'setName', 'setOutputData', 'setOutputSandbox', 'setOwner', 'setOwnerGroup', 'setPlatform', 'setSystemConfig', 'setType', 'site', 'stderr', 'stdout', 'stepCount', 'systemConfig', 'type', 'workflow']

To obtain help on any of these options the python 'help' command can be used. For example to get more information on the 'setName' method:

>>> help(j.setName)
Help on method setName in module DIRAC.Interfaces.API.Job:

setName(self, jobname) method of DIRAC.Interfaces.API.Job.Job instance
    Helper function.
    
    A name for the job can be specified if desired. This will appear
    in the JobName field of the monitoring webpage. If nothing is
    specified a default value will appear.
    
    Example usage:
    
    >>> job=Job()
    >>> job.setName("myJobName")
    
    @param jobname: Name of job
    @type jobname: string

Full documentation on the Job object can be found here.

Running jobs

Monitoring jobs

Retriving job output

Managing data

More information

-- AndrewCSmith - 2009-09-30

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2009-10-01 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback