Some notes of a new Dirac developer for LHCb

Some links of interest

  • Dirac Git installation guide here

Some more informal DIRAC documentation for developers:

Links to SVN repos:


List of systems where Dirac services run: here: alias, usage of the system, platform etc..

Setting up the development environment

In order to be more concrete, I will refer to the first steps done to develop a dummy service inside the DataManagement system (DMS)

Preliminary settings: Eclipse

Install Eclipse on your desktop, pydev, the plugin for SVN, and the necessary connector, and configure:

A general (LHCb) guide

A quick guide, with some DIRAC specific hints that you should also follow is here. Basically most of the settings you get them if you tick all fields in: Preferences -> Pydev -> Editor -> Code Style -> Code Formater

And very important: set tab length to 2 and tick "Replace tabs with spaces" in

Preferences -> Pydev -> Editor

Set up the environment to develop in lxplus

Execute (from the instructions of Federico):

SetupProject --build-env LHCbDirac v5r9

why v5r9? Because you have to arbitrary choose what is the version of the project that you will set as your 'development' environment. I have decided v5r9 (better not deciding the most recent), so I execute the SetupProject LHCbDirac v5r9 and this will create a directoy ~cmtuser/LHCbDirac_v5r9 Change to that directory and create the etc directory and create there the configuration file that will set this environment as development:

$ cat cmtuser/LHCbDirac_v5r9/etc/dirac.cfg 
  Setup = LHCb-Development
#  Setup = LHCb-Production
#  Setup = LHCb-Certification
    Servers = dips://
    Servers += dips://
#    Servers += dips:// 
    Name = LHCb-Prod
    UserServerCerificate = no
    SkipCAChecks = yes 

Then download the systems that you need (not necessary to download all the systems). In this case, to make development in DMS, I have to download:

$ getpack DIRAC/Core head
$ getpack DIRAC/DataManagementSystem head 
$ getpack LHCbDIRAC/DataManagementSystem head
$ make
$ SetupProject LHCbDirac v5r9

because the code of DMS is under two paths. How to decide where to put a new service? Andrei: start it in LHCbDIRAC and then if it turns out to be of general utility, it can be moved to DIRAC.

NB: specify 'head' in the getpack, otherwise you'll getpack a particular tag. The consequence is that when you update (either from inside Eclipse, with Team/Update function, or from command line) SVN will make the update to the new changes within that tag. In SVN if you want to be able to update to a version other that the tag originally checked out, you have to use 'switch'. However to do things simpler, and to be able to update to the head version, it is better to specify head when you getpack.

Create the pydev projects in Eclipse

Go to Eclipse on your desktop (where of course AFS is mounted). As you have downloaded (or 'getpacked') two systems, in Eclipse you have to create two pydev projects, corresponding to the two respective paths (/afs/ and .../LHCbDIRAC/).

Or better, create only a project, using as workspace you cmtuser directory, following the instructions of Marco (Eclipse tutorial).

Some code patterns that LHCbDirac developers should follow

  • avoid using has_key(): it will become obsolete
  • avoid constructions such as:

if a not in d: (or worse if not d.has_key(a):)
   d[a] = []
instead use:

  • use always the finally clause, especially when dealing with requests (a request should never be left in an unexpected state).
example from the RemovalAgent: Replace this
       # if something fails return the original request back to the server 
       res = self.requestDBClient.updateRequest( requestName, requestString, sourceServer )
       return S_OK()

     res = self.requestDBClient.updateRequest( requestName, newrequestString, sourceServer )
with this:
       # if something fails return the original request back to the server 
       return S_OK()
       res = self.requestDBClient.updateRequest( requestName, newrequestString, sourceServer )
this would allow also the intermediate "return" statements to not leave the request "Assigned". Actually this structure should be enforced on all the Agents that handle requests.

How to implement a new service

Usually (but not necessarily) the service has a DB behind. So, let's start from creating a DB.

The DB

Go to the development machine volhcb12 (connect through the lxvoadm), change to dirac user (sudo su dirac), start mysql and connect as root (the password in my notebook) and create a new DB (i.e. I create testDMSDB). Try to follow the naming convention for DBs: first letter is capital, and ends with 'DB' (i.e. DataIntegrityDB). Then grant all privileges to Dirac user, because the services will run as Dirac:
[volhcb12] /opt/dirac/pro > mysql -uroot -p
Enter password:
.... create DB....
testDMSDB.*  TO Dirac@localhost IDENTIFIED BY 'xxx';

exit from mysql. The following times you need to connect to the DB do it as Dirac user.

The DB python module

In Eclipse, under DMS/DB create a new file called like the DB itself:

this module creates the class for this DB, inheriting from a general class defined in the Core. Then, it defines some methods which act on the DB (typically they do queries, or inserts). Most of the functionality resides in this module, and then the service will simply export these methods, providing the necessary authentication layer. Later we will see how the service handler exports the method (i.e. the querySomething method). See here the code of

""" testDMSDB class is a front-end to the testDMS Database. """

import re, os, sys
import time, datetime
from types import *

from DIRAC import gConfig,gLogger,S_OK, S_ERROR
from DIRAC.Core.Base.DB import DB
class testDMSDB(DB):

  def __init__( self, maxQueueSize=10 ):
    """ Standard Constructor
  def insertSomething(self,user,files):
      req = "INSERT INTO userAccount (userName,numOfFiles) VALUES ('%s',%d)" % (user,files)
      res = self._update(req)
      if not res['OK']:
          gLogger.error("Failed to insert user files",res['Message']) 
"Successfully inserted %d files" % res['Value'])
      return res

  def querySomething(self,user):
      req = "SELECT userName,numOfFiles from userAccount where userName='%s'" % (user)
      res = self._query(req)
      if not res['OK']:
          gLogger.error("Failed to query user files",res['Message']) 
          return res
      userDict = {}
      for userName,numFiles in res['Value']:
          userDict[userName] = numFiles
"Succesfully found %d files for user %s" % (numFiles,user))
      return S_OK(userDict)

The service handler

Under DMS/Service create a service handler called which:
  • imports the DB module
  • initialize the testDMS handler
  • exports the methods that are implemented in the DB module (i.e. the querySomething method), with these two lines:
    • types_querySomething = [StringTypes] # set the number and type of input arguments of the method
    • def export_insertSomething(self,user,files): # make the method accessible for a client

from types                                                          import *
from DIRAC                                                        import gLogger, gConfig, rootPath, S_OK, S_ERROR
from DIRAC.Core.DISET.RequestHandler            import RequestHandler
from DIRAC.DataManagementSystem.DB.testDMSDB    import testDMSDB

# This is a global instance of the testDMSDB class
testDB = False

def initializetestDMSHandler(serviceInfo):
  global testDB
  testDB = testDMSDB()
  return S_OK()

class testDMSHandler(RequestHandler):

   types_insertSomething = [StringTypes,[IntType,LongType]]
   def export_insertSomething(self,user,files):
     return testDB.insertSomething(user,files)

   types_querySomething = [StringTypes]
   def export_querySomething(self,user):
     return testDB.querySomething(user)

The client

Under DMS/Client, create a

from DIRAC                                      import S_OK, S_ERROR,gLogger,gConfig
from DIRAC.Core.Base.Client            import Client
import re,os,types

class testDMSClient(Client):
  def __init__(self):
It imports the general Client from the Core, and simply creates the client class for this service.

The agent

An agent is a program which runs periodically in background. It contacts the services and do actions, like populating DBs, setting transfers, etc etc...

Developing an agent: it consists of a class, which inherits from the general AgentModule. And then there is an initialization function and then an 'execute' function, where the main body of the agent is implemented. There are plenty of examples. In this manual we will only explain how to install it and run it, in the section how to install and start an agent.

Set the appropriate configuration in the CS

The call to setServer made in the client contacts the CS and asks for some configuration options that are stored there. It looks under the setup specified in the local .cfg file and the system specified in the call arguments.

Log to Dirac web portal as 'lhcbadmin' and go to systems/CS/manage remote configuration. I have to add the URL of the service: go to:

- lhcb Configuration
-- Systems
--- DataManagement
---- Development

here I have to add an entry under 3 different places: URL, Services, Databases

  • under URLs, you can copy an already existing item, and change the name of the copy to the new name. Then set the URL which specifies service and port: testDMS = dips://
  • under Services: add the testDMS folder and set:
    • logLevel = INFO
    • LogBackends = stdout,server
    • HandlerPath = DIRAC/DataManagementSystem/Service/
    • Port = 9161
    • Protocol = dips
    • the subfolder of Authorization with Default = all
  • under Databases add the folder testDMSDB with the lines:
    • DBName = testDMSDB
    • Host =

be careful choosing the port! Should not be used by other services! See here the ports assigned to Dirac services. Once you have chosen a free port, ask Joel to open it.

Installing and starting the new service on volhcb12

See this link about the runit toolkit, the system used to run DIRAC services.

From volhcb12 and as dirac user I copy my code developed on lxplus to the appropriate path on volhcb12 (under /opt/dirac/pro/DIRAC/):

[volhcb12] /opt/dirac/pro > cp ~lanciott/cmtuser/LHCbDirac_v5r9/DIRAC/DataManagementSystem/DB/ DIRAC/DataManagementSystem/DB/
and the same for the DB module and the client.

With the old script

Then, as dirac user, install the service:

[volhcb12] /opt/dirac/pro$ ./scripts/ DataManagement testDMS
[volhcb12] /opt/dirac/pro$ tail -f runit/DataManagement/testDMS/log/current
the install_service script will install the service and create the run script under runit/DataManagement/testDMS/. It is recommended to use this script, so the run scripts will have the same format. Like this:

[volhcb12] /opt/dirac/startup > cat /opt/dirac/pro/runit/DataManagement/testDMS/run
source /opt/dirac/bashrc
exec 2>&1
[ -e $DIRAC/etc/DBs.cfg ] && DBs=$DIRAC/etc/DBs.cfg
exec python $DIRAC/DIRAC/Core/scripts/ DataManagement/testDMS $DIRAC/etc/DataManagement_testDMS.cfg $DBs -o LogLevel=DEBUG < /dev/null

a configuration file, $DIRAC/etc/DataManagement_testDMS.cfg, is automatically created, as an empty file. Any option added in the configuration file will overwrite the same option set in the CS.

With the new dirac-install-service script

A new script to install services is available (March 2011). Let's remove the service and install it again. To remove the service:

[volhcb12] /opt/dirac/pro > rm -rf /opt/dirac/pro/runit/DataManagement/testDMS
[volhcb12] /opt/dirac/pro > ll /opt/dirac/startup/DataManagement_testDMS 
lrwxrwxrwx 1 dirac z5 43 Oct 15 12:42 /opt/dirac/startup/DataManagement_testDMS -> /opt/dirac/pro/runit/DataManagement/testDMS
[volhcb12] /opt/dirac/pro > rm  /opt/dirac/startup/DataManagement_testDMS 

[volhcb12] /opt/dirac/pro > dirac-install-service DataManagement testDMS
2011-03-24 16:04:04 UTC Framework NOTICE: DIRAC Root Path = /opt/dirac/pro
Installing service DataManagement/testDMS 
2011-03-24 16:04:09 UTC DataManagement/testDMS ALWAYS: Starting service DataManagement/testDMS 
2011-03-24 16:04:09 UTC DataManagement/testDMS/testDMSDB  INFO: ================================================== 
2011-03-24 16:04:09 UTC DataManagement/testDMS/testDMSDB  INFO: User:           Dirac 
2011-03-24 16:04:09 UTC DataManagement/testDMS/testDMSDB  INFO: Host:           localhost 
2011-03-24 16:04:09 UTC DataManagement/testDMS/testDMSDB  INFO: DBName:         testDMSDB 
2011-03-24 16:04:09 UTC DataManagement/testDMS/testDMSDB  INFO: MaxQueue:       10 
2011-03-24 16:04:09 UTC DataManagement/testDMS/testDMSDB  INFO: ================================================== 
2011-03-24 16:04:09 UTC DataManagement/testDMS  INFO: Handler up Serving from dips://

Start the new service with runit

And then start the service:

[volhcb12] /opt/dirac/pro$ cd ../startup/
[volhcb12] /opt/dirac/startup$ ln -s /opt/dirac/pro/runit/DataManagement/testDMS DataManagement_testDMS

Once this link has been created, then the service will automatically start.

Why is the link necessary: because there is a process running which takes surveys all the services in the sub directories below: /opt/dirac/startup/ it periodically checks that the services therein are running and if they are not, it restarts them:

ps -faux | grep runsv | grep -v grep
dirac     4211  0.0  0.0   3816   412 ?        Ss   Jan11   0:00 /opt/dirac/pro/Linux_x86_64_glibc-2.5/bin/runsvdir -P /opt/dirac/startup log: ot exist?...

The useful commands are:

  • To force the restart: runsvctrl t /opt/dirac/startup/[DirName]. Important if the service is stopped, runsvctrl t will do NOTHING! you have to start it with u instead of t.
  • To stop it: runsvctrl d /opt/dirac/startup/[DirName]
  • To start it again: runsvctrl u /opt/dirac/startup/[DirName]

check if it's running with runsvstat and the directory location of the service, i.e.:

[volhcb12] /opt/dirac/pro > runsvstat /opt/dirac/startup/DataManagement_StorageUsage
/opt/dirac/startup/DataManagement_StorageUsage: run (pid 10899) 260507 seconds

Agents and services log files

The AgentModule class (from DIRAC Core Base) creates a log file instance in the init module:
    if baseAgentName and agentName == baseAgentName:
      self.log = gLogger
      standaloneModule = True
      self.log = gLogger.getSubLogger( agentName, child = False )
      standaloneModule = False

Set the logfile size and the number of logfiles to keep

Edit the 'config' file in the log directory, change the parameters and restart the logger. I.e.:
[volhcb12] > cat  /opt/dirac/pro/startup/DataManagement_SEUsageAgent/log/config 
means 10MB size, and 20 logs kept.

Connect to the service

From lxplus (or any other system) how to connect to the development services: make a SetupProject LHCbDirac vxry. This will set your environment. As explained above, you should have a 'dirac.cfg' file stating 'Setup = LHCb-Development' in the ~/cmtuser/LHCbDirac_vxry/etc. If you don't create this file, then Dirac commands will look for the .dirac.cfg file in your home directory. It is very practical to getpack a given version of LHCbDirac, even an old one, and create under its etc directory the dirac.cfg file setting the development setup, while keeping the Setup as production in the .dirac.cfg in your home directory. In this way, from a lxplus shell you can login and easily decide which Dirac setup to use, without editing any dirac.cfg file.

Start the python prompt and try to connect to the service:

>>> from DIRAC.Core.Base.Script import parseCommandLine
>>> parseCommandLine()
>>> from DIRAC.Core.DISET.RPCClient import RPCClient
>>> s = RPCClient("DataManagement/testDMS")
>>> s.querySomething("lanciott")
{'OK': True, 'rpcStub': (('DataManagement/testDMS', {'skipCACheck': True,
'delegatedGroup': 'lhcb_user', 'delegatedDN':
'/DC=es/DC=irisgrid/O=pic/CN=elisa.lanciotti', 'timeout': 600}),
'querySomething', ('lanciott',)), 'Value': {'lanciott': 123L}}

volhcb12 and SVN

volhcb12 is synchronized with SVN so you can make an update of the code to the head of the repository. After, the code will be downloaded with the particular user as owner, and dirac user cannot overwrite it. So, after the update it is necessary to grant write permission to the group:
[volhcb12]$ sudo su -
[volhcb12]$ chmod g+w -R /opt/dirac/pro/LHCbDIRAC
otherwise you can change the owner to dirac user.

Install and start the agent

Install it on volhcb12:
  • First, copy the agent to LHCbDIRAC/DataManagementSystem/Agent/
  • use the "dirac-install-agent" command, i.e. "$ dirac-install-agent DataManagement StorageUsageAgent"
  • create the link to the agent directory under the usual path /opt/dirac/startup, in order to have the agent started (and if necessary restarted) automatically:
$ dirac-install-agent DataManagement StorageUsageAgent
$ cd /opt/dirac/startup
$ ln -s /opt/dirac/pro/runit/DataManagement/StorageUsageAgent/ DataManagement_StorageUsageAgent

Alternatively, you can start the agent in interactive way, running it in foreground:

$ dirac-agent [sytesmname]/[AgentName] -o LogLevel=debug 
(i.e. $ dirac-agent DataManagement/StorageHistoryAgent -o LogLevel=debug )
useful for debugging and during development. You start it in foreground and with loglevel debug. And then stop it with CtrlC.

Once you want to run the agent permanently, run it in background.

The "dirac-install-agent" command will create a run script, i.e. :

[volhcb12] /opt/dirac/pro > cat  /opt/dirac/startup/DataManagement_StorageSummaryAgent/run 
[ -e $rcfile ] && source $rcfile
exec 2>&1
[ "agent" = "agent" ] && renice 20 -p $$
exec python /opt/dirac/pro/DIRAC/Core/scripts/ DataManagement/StorageSummaryAgent /opt/dirac/pro/etc/DataManagement_StorageSummaryAgent.cfg -o LogLevel=debug < /dev/null
and it will also create a configuration file, which might be empty by default. There, with the same syntax than the Configuration Service, you can set what you want. The minimum to be set is the setup:
[volhcb12] /opt/dirac/pro > cat /opt/dirac/pro/etc/DataManagement_StorageSummaryAgent.cfg
and optionally many more parameters can be set.

You can start/stop the agent with runsvctrl (as already said for the service):

  • To force the restart: runsvctrl t /opt/dirac/startup/[DirName]. Important if the service is stopped, runsvctrl t will do NOTHING! you have to start it with u instead of t.
  • To stop it: runsvctrl d /opt/dirac/startup/[DirName]
  • To start it again: runsvctrl u /opt/dirac/startup/[DirName]

Commit and tag the code

In order to include your new code into the next release, you should tag it. The procedure to follow is in the process to be reviewed. In the future, we will make a more extensive use of SVN branches. But for the time being, let's suppose you have changed something in a file in DataManagement and you want to commit and tag. It is not a good idea to commit and tag in the SVN head because this will include in the tag all the other modifications that might have uploaded to head by other developers, that maybe are not yet ready for being included in a release.

So, this is a simple procedure to tag only your change. Go to the LHCbDIRAC/trunk/LHCbDIRAC/versions.cfg file and look what is the last tag included in any release (even if the release is a pre release). In that release, there will be a list of tags, look for the one of your system (in this case DMS). i.e. on 25th Feb 2011, I look there and I see: last release is: v6r0-pre9, and therein the tag for DMS is dm_2011012202. Then, from lxplus make a 'svn co' of the DMS for that tag

[lxplus407] /afs/ > svn co svn+ssh://
Warning: the RSA host key for '' differs from the key for the IP address ''
Offending key for IP in /afs/
Matching host key in /afs/
Are you sure you want to continue connecting (yes/no)? yes
A    dm_2011012202/Service
A    dm_2011012202/Service/
Checked out revision 34839.
then copy the tag to a new tag (following the usual convention dm_yyyymmddxx, where xx is a number starting from 1 to distinguish more releases in the same day):
[lxplus407] /afs/ > svn copy svn+ssh:// svn+ssh://
Warning: the RSA host key for '' differs from the key for the IP address ''
Offending key for IP in /afs/
Matching host key in /afs/
Are you sure you want to continue connecting (yes/no)? yes

Committed revision 34843.
then switch to the newly created tag:
[lxplus407] /afs/ > cd dm_2011012202/
[lxplus407] /afs/ > ls
Agent  Client  cmt  DB  scripts  Service
[lxplus407] /afs/ > svn switch svn+ssh://
At revision 34843.
check that you are really there, in the new tag:
[lxplus407] /afs/ > svn info
Path: .
URL: svn+ssh://
Repository Root: svn+ssh://
Repository UUID: 4525493e-7705-40b1-a816-d608a930855b
Revision: 34843
Node Kind: directory
Schedule: normal
Last Changed Author: lanciott
Last Changed Rev: 34843
Last Changed Date: 2011-02-25 15:05:55 +0100 (Fri, 25 Feb 2011)
apply your changes, commit:
[lxplus407] /afs/ > cp ~/RunDBInterfaceHandler.py_DEVEL Service/ 
if the files are new, do an 'svn add'. And then commit:
[lxplus407] /afs/ > svn commit
Sending        Service/
Transmitting file data .
Committed revision 34845.

and finally go back to the versions.cfg, in the Next Release section add your tag with the comment. The release manager will pick it when collecting the tags for the next release.

See here a more general description about DIRAC release procedure.

How to run a standalone Dirac service

Once you have created a new Dirac service it is not necessary to install and run it on volhcb12. In fact Dirac provides a way to run a service on any host, using the dirac-service command. i.e. on my account on lxplus:
$ SetupProject LHCbDirac v5r9
$ cd cmtuser/LHCbDirac_v5r9
$ dirac-service DataManagement/testDMS etc/dirac.cfg
where dirac.cfg is the configuration file created previously. In this way the Dirac service will run taking the configuration from the file. Any further information needed to run, and not specified in the file, will be read from the CS. In principle this should start the service, but in practice there are limitations: if the service needs a DB, this sould be provided as well. And if you want to connect to the service with a client, some ports should be opened (which on the lxplus cannot be done). The service also need certificates for the authentication.

How to switch to a new version

At some point you might want to switch to a more recent version for the development. See here the instructions for LHCb software in general.

Developing a new script

The scripts source .py code must be placed in the correspondiong system scripts directory. In the case of DMS under DIRAC/DataManagementSystem/scripts. The script should of course have .py extension. While installation the script wrappers are created in $DIRACROOT/scripts directory to have a single $PATH entry. If you do it in volhcb12 where the installation is already done, you should execute dirac-deploy-scripts command to create this script wrapper. After it is done you can continue working on the original source of your script, it will be automatically taken. You invoke your script without .py extension - this is the name of the wrapper in the $DIRACROOT/scripts directory.

In the script source code you have first of all to import this:

import DIRAC
from DIRAC.Core.Base import Script
Script.parseCommandLine( ignoreErrors = False )
you should import the Dirac API only after having called Script.parseCommandLine because importing the Dirac API automatically initializes Dirac, so any Script stuff will fail. It's better to load first Script, do all your initialization, defining options and after the parseCommandLine import the API or other Dirac modules. Actually, the only modules that can not be imported before are the API modules. Since they are thought to be used "standalone" in other python environments with possibly other command line arguments, they initialize the configuration without looking and the command line.

Executing the script on volhcb12

Important: volhcb12 default configuration is to use server certificate rather than the user certificate. So, in order to execute a Dirac script, which requires a user proxy, it is necessary to add in the user home dir a file .dirac.cfg with the following content:
 Setup = LHCb-Development
   Servers = dips://
   UseServerCertificate = no

Code keywords substitution

As CVS, also SVN enables keywords substitution. To enable them, follow the nice guide at Then, add

__RCSID__ = "$Id:  $"
at the beginning of your modules. These will be substituted with, for example:
__RCSID__ = "$Id: 30260 2010-11-12 10:10:33Z fstagni $"

DIRAC and git

New section to deal with the DIRAC installation from git repository and how to do development and commit code into git here

-- ElisaLanciotti - 29-Oct-2010

Edit | Attach | Watch | Print version | History: r42 < r41 < r40 < r39 < r38 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r42 - 2011-07-06 - unknown
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback