Difference: DIRAC3InstallProject (1 vs. 26)

Revision 262014-11-13 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 236 to 236
 
  1. dirac2AFS -v vArB -p $CMTCONFIG
  2. mkproject -p Dirac -v vArB -a tsrhw
  3. mkLHCbtar -p LHCbGrid -v vXrY -b $CMTCONFIG
Changed:
<
<
  1. mkLCGCMTtar -n DIRAC_vArB -b $CMTCONFIG --exclude="lib*.a"
  2. mkLHCbtar -p Dirac -v vArB -b $CMTCONFIG
>
>
  1. mkLCGCMTtar -n LHCBDIRAC_vArB -b $CMTCONFIG --exclude="lib*.a"
  2. mkLHCbtar -p LHCbDirac -v vArB -b $CMTCONFIG
 

-- PhilippeCharpentier - 18 Nov 2008 \ No newline at end of file

Revision 252011-12-21 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 219 to 219
 

Instructions to create the DIRAC tarball

Added:
>
>
to tag the package:
  1. svn mkdir -m "tagging v4r9" svn+ssh://svn.cern.ch/reps/lhcb/LHCbGrid/tags/LHCBGRID/LHCBGRID_v4r9
  2. svn cp -m "tagging v4r9" svn+ssh://svn.cern.ch/reps/lhcb/LHCbGrid/trunk/cmt svn+ssh://svn.cern.ch/reps/lhcb/LHCbGrid/tags/LHCBGRID/LHCBGRID_v4r9
 to create the distribution package for LHCbGrid:
  1. cd $LHCBRELEASE
  2. mkproject -p LHCbGrid -v vXrY -a ng

Revision 242010-08-31 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 148 to 148
 
  • source $MYSITEROOT/LbLogin.csh or . $MYSITEROOT/LbLogin.sh
and to set up the LHCbDirac environment (beware if you use ganga this is not needed as done internally by ganga)
Added:
>
>
  • lhcb-proxy-init
  If you see an error message like :
Warning : Cannot add voms attribute /lhcb/Role=user to proxy Accessing data in the grid storage from the user interface will not be possible. 
Line: 157 to 158
 It means that you need to install or set the certificates authorities files. Check if the variables $X509_CERT_DIR and X509_VOMS_DIR exists in your environment.
  • If they are set, please verify that the content is up to date.
  • If they are not set, you can use the default location which is installed by LHCbDirac or choose yourself a location where you want to install them.
Changed:
<
<
  1. If you use the default location, you need to keep that copy up-to-date. You can use the Dirac tool dirac-admin-get-CAs
  2. If you choose an other directory, you need to keep that copy up-to-date AND to set the variables ($X509_CERT_DIR and X509_VOMS_DIR) to point to this location. (the format should be X509_VOMS_DIR=/vomsdir and X509_CERT_DIR=/certificates)
>
>
  1. If you use the default location, you need to keep that copy up-to-date.
    • You can run the Dirac tool dirac-admin-get-CAs.
    • make sure that the variable are set as X509_VOMS_DIR=$MYSITEROOT/LHCBDIRAC/LHCBDIRAC_<version>/etc/grid-security/vomsdir and X509_CERT_DIR=$MYSITEROOT/LHCBDIRAC/LHCBDIRAC_<version>/etc/grid-security/certificates
  2. If you choose an other directory, you need :
    • to add one line in your DIRAC config file ($MYSITEROOT/LHCBDIRAC/LHCBDIRAC_<version>/etc/dirac.cfg)
DIRAC
  }
  Security
  {
    UseServerCertificate = no
    Grid-Security = <my_location>
  }
    • to set the variables ($X509_CERT_DIR and X509_VOMS_DIR) to point to this location. (the format should be X509_VOMS_DIR=my_location/vomsdir and X509_CERT_DIR=my_location/certificates).
    • to run the Dirac tool dirac-admin-get-CAs

Nota Bene: to keep up-to-date the directory of certificates, you can use a cron task (see example below)

Create a scripts call CA_update.csh and put the follwoing content inside

#!/bin/csh

SetupProject LHCbDirac
cat $HOME/.lcgpasswd | lhcb-proxy-init -p
dirac-admin-get-CAs

exit 0
 
Changed:
<
<
Nota Bene: to keep up-to-date the directory of ecrtificates, you can use a cron task.
>
>
create a cron entry with this format to run the script CA_update.csh every day at 2h10 on machine_name : 10 02 * * machine_name /my/directory/CA_update.csh
 

Setup the LHCbDirac environment

Revision 232010-08-27 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 43 to 43
 Another guideline in this proposal is to avoid having to embed within the DIRAC distribution any external package but rather depend on external distribution of those as well as standard distribution with the LHCb Core Software tools. In particular the binary and library directories within DIRAC don't need to be distributed in the user tarball.

CMT projects and packages

Changed:
<
<
In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is as of v5r0. This consists in the following additions:
>
>
In order to be able to use install_project and SetupProject, LHCbDIRAC and DIRAC must be packaged as a CMT project, which it now is as of v5r0. This consists in the following additions:
 
Changed:
<
<
  • File <DIRACBase>/cmt/project.cmt : contains the dependencies of LHCbDIRAC in terms of projects
>
>
  • File <LHCbDiracBase>/cmt/project.cmt : contains the dependencies of LHCbDIRAC in terms of projects
 
  • Package LHCbDiracSys : this cmt package's requirements file defines the top dependencies of LHCbDIRAC
  • Package LHCbDiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
  • Package LHCbDiracPolicy : defines a number of policies for setting the environment. Also contains some utilities used by CMT.
Line: 75 to 75
 
  • It is noticed that pyQT has a dependency to the explicit 3-digit version of python for 2.4 but not for 2.5. We can probably survive like this unless we absolutely need 2.4.4 (Ricardo to check). In this case a 2-digit dependency should be used (can already be foreseen? Stefan)

Environment setting

Changed:
<
<
The command SetupProject Dirac [<version>] should be used to set up the environment, using CMT. CMT sets up the environment variables based on the DiracPolicy that is a copy of GaudiPolicy. It uses the environment variable CMTCONFIG for the platform-dependent part. As most of this dependency is in externals, most is done by LCGCMT and not directly in DIRAC. The only identified platform-dependent part in DIRAC is pyGSI that eventually could be sub-contracted to the LCG-AA (Adri to get in touch with Stefan)
>
>
The command SetupProject LHCbDirac [<version>] should be used to set up the environment, using CMT. CMT sets up the environment variables based on the LHCbDiracPolicy that is a copy of GaudiPolicy. It uses the environment variable CMTCONFIG for the platform-dependent part. As most of this dependency is in externals, most is done by LCGCMT and not directly in LHCbDIRAC. The only identified platform-dependent part in LHCbDIRAC is pyGSI that eventually could be sub-contracted to the LCG-AA (Adri to get in touch with Stefan)
  CMT uses the InstallArea that may contain the following directories:
Line: 107 to 107
  By default, LHCbDirac should use the latest version of LHCbGrid, with a given major version (e.g. use LHCbGrid v1r*). In case one urgently needs to use specific versions of the middleware, one can fix them in LHCbDiracConfig if there is no LHCbGrid (yet) that contains these versions. Making a new release of LHCbGrid is the preferred solution for production versions. Versions of externals can also be superseded from options in the SetupProject command (see later).
Deleted:
<
<
Currently, after a new release of LHCbDirac, it is installed on AFS at CERN using a dedicated script dirac2AFS that internally uses dirac_install and removes all unnecessary directories and files.
dirac2AFS.py <options>
  Installs DIRAC in $LHCb_release_area
  Options:
    -v, --version <version>: Manadatory, version to be installed
    -p, --platform <CMTCONFIG>: platform, default to $CMTCONFIG
    -t, --test <name>: name of the installed version for tests
    -d, --directory <dir>: directory where to install
    -D, --dev: install in $LHCBDEV
 

User check out

getpack and the SVN repository of LHCbDirac have been adapted to allow checking out a LHCbDIRAC package in a user directory (e.g. ~/cmtuser). Some actions must however be taken before running getpack (functionality of setenv<project>). They have been coded in the command setenvProject LHCbDirac <version>. This will create the appropriate structure in the cmtuser directory.
Line: 150 to 138
 

User guide

This is a summary of commands to be used for taking advantage of the DIRAC installation using CMT.

Installing LHCbDirac locally

Changed:
<
<
Refer to this link
>
>
For installing LHCbDirac on your local machine, you should download and install LHCbDirac, specifying the version <version> Then in your login script you should include:
  • source $MYSITEROOT/LbLogin.csh or . $MYSITEROOT/LbLogin.sh
and to set up the LHCbDirac environment (beware if you use ganga this is not needed as done internally by ganga)

If you see an error message like :

Warning : Cannot add voms attribute /lhcb/Role=user to proxy Accessing data in the grid storage from the user interface will not be possible. 
The grid jobs will not be affected.

It means that you need to install or set the certificates authorities files. Check if the variables $X509_CERT_DIR and X509_VOMS_DIR exists in your environment.

  • If they are set, please verify that the content is up to date.
  • If they are not set, you can use the default location which is installed by LHCbDirac or choose yourself a location where you want to install them.
  1. If you use the default location, you need to keep that copy up-to-date. You can use the Dirac tool dirac-admin-get-CAs
  2. If you choose an other directory, you need to keep that copy up-to-date AND to set the variables ($X509_CERT_DIR and X509_VOMS_DIR) to point to this location. (the format should be X509_VOMS_DIR=/vomsdir and X509_CERT_DIR=/certificates)

Nota Bene: to keep up-to-date the directory of ecrtificates, you can use a cron task.

 

Setup the LHCbDirac environment

Revision 222010-02-22 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 91 to 91
  As the python executables currently deployed are dynamically linked, the lib directory of python must be included in the LD_LIBRARY_PATH. Using any incompatible version of python without the library directory would not work, unless a static version is used.
Changed:
<
<
By default CMT doesn't check for the existence of directories included on the paths. DiracPolicy ensures that all directories on the paths do exist (copied from GaudiPolicy)
>
>
By default CMT doesn't check for the existence of directories included on the paths. LHCbDiracPolicy ensures that all directories on the paths do exist (copied from GaudiPolicy)
 

Distribution packaging

Changed:
<
<
We use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies from CMT. For DIRAC the dependencies are simple:
>
>
We use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies from CMT. For LHCbDIRAC the dependencies are simple:
 
Added:
>
>
  • DIRAC project (framework)
 
  • LHCbGrid project (only interfaces and version definition)
  • gLite client middleware according to the version defined in LHCbGrid, including the file access client libraries (Castor, dCache, DPM). This tarball should contain a reduced list of libraries (limited to those that are used!)
  • Other external packages: python, pyQt and its dependencies
Changed:
<
<
Using install_project ensures that the necessary LHCb scripts are also installed and the configuration is properly defined (such as !SetupProject, LbLogin etc...). If they have already been installed when users have installed the LHCb applications, they will not need to be reinstalled. The use case is mostly for ganga users who anyway need to install the applications, and they should be advised to install DIRAC in the same area.
>
>
Using install_project ensures that the necessary LHCb scripts are also installed and the configuration is properly defined (such as !SetupProject, LbLogin etc...). If they have already been installed when users have installed the LHCb applications, they will not need to be reinstalled. The use case is mostly for ganga users who anyway need to install the applications, and they should be advised to install LHCbDirac in the same area.
  install_project uses a tarball that contains all platform-independent files (shared tarball), and another one with platform-dependent files. These tarballs can be created with a standard script. Like this one can install as many platforms as wanted on the same file system: the shared tarball is only installed once. None of these tarballs should contain the .pyc files as they are non-relocatable. After installation, it would be worth compiling the python modules (post-install action) and create the .pyc files. This is a post-installation for the shared tarball as the .pyc files are platform independent.
Changed:
<
<
By default, DIRAC should use the latest version of LHCbGrid, with a given major version (e.g. use LHCbGrid v1r*). In case one urgently needs to use specific versions of the middleware, one can fix them in DiracConfig if there is no LHCbGrid (yet) that contains these versions. Making a new release of LHCbGrid is the preferred solution for production versions. Versions of externals can also be superseded from options in the SetupProject command (see later).
>
>
By default, LHCbDirac should use the latest version of LHCbGrid, with a given major version (e.g. use LHCbGrid v1r*). In case one urgently needs to use specific versions of the middleware, one can fix them in LHCbDiracConfig if there is no LHCbGrid (yet) that contains these versions. Making a new release of LHCbGrid is the preferred solution for production versions. Versions of externals can also be superseded from options in the SetupProject command (see later).
 
Changed:
<
<
Currently, after a new release of DIRAC, it is installed on AFS at CERN using a dedicated script dirac2AFS that internally uses dirac_install and removes all unnecessary directories and files.
>
>
Currently, after a new release of LHCbDirac, it is installed on AFS at CERN using a dedicated script dirac2AFS that internally uses dirac_install and removes all unnecessary directories and files.
 
dirac2AFS.py <options>
  Installs DIRAC in $LHCb_release_area
Line: 119 to 120
 

User check out

Changed:
<
<
getpack and the SVN repository of LHCbDIRAC have been adapted to allow checking out a LHCbDIRAC package in a user directory (e.g. ~/cmtuser). Some actions must however be taken before running getpack (functionality of setenv<project>). They have been coded in the command setenvProject LHCbDirac <version>. This will create the appropriate structure in the cmtuser directory.
>
>
getpack and the SVN repository of LHCbDirac have been adapted to allow checking out a LHCbDIRAC package in a user directory (e.g. ~/cmtuser). Some actions must however be taken before running getpack (functionality of setenv<project>). They have been coded in the command setenvProject LHCbDirac <version>. This will create the appropriate structure in the cmtuser directory.
  Before using the getpack command you should issue the following commands:
Line: 140 to 141
  The build consists of the following steps (Ricardo should review this!):
Changed:
<
<
  • Set up the scripts directory in order to only have one DIRAC directory on the PATH environment variable.
>
>
  • Set up the scripts directory in order to only have one LHCbDIRAC directory on the PATH environment variable.
 
  • Compile the python modules (is this necessary?)

Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows).

Line: 148 to 149
 

User guide

This is a summary of commands to be used for taking advantage of the DIRAC installation using CMT.
Changed:
<
<

Installing DIRAC locally

>
>

Installing LHCbDirac locally

 Refer to this link
Changed:
<
<

Setup the DIRAC environment

>
>

Setup the LHCbDirac environment

 
  1. On a local installation, first execute source $MYSITEROOT/LbLogin.csh or . $MYSITEROOT/LbLogin.sh
  2. Use SetupProject to set the environment. Most useful forms:
Changed:
<
<
    • SetupProject Dirac: sets up environment for the current CMTCONFIG and the latest version of Dirac
    • SetupProject Dirac --list: lists all versions available
    • SetupProject Dirac <version>: sts up a named version of Dirac
    • SetupProject Dirac --dev: looks in $LHCBDEV rather than $LHCb_release_area (for tests)
    • SetupProject Dirac [<version>] <external> -v <external_version>: overwrites the default version of <external> with the specified version. <external> can be for example gfal, lcgutils, lfc, Python. More than one external can be specified.
>
>
    • SetupProject LHCbDirac: sets up environment for the current CMTCONFIG and the latest version of Dirac
    • SetupProject LHCbDirac --list: lists all versions available
    • SetupProject LHCbDirac <version>: sts up a named version of LHCbDirac
    • SetupProject LHCbDirac --dev: looks in $LHCBDEV rather than $LHCb_release_area (for tests)
    • SetupProject LHCbDirac [<version>] <external> -v <external_version>: overwrites the default version of <external> with the specified version. <external> can be for example gfal, lcgutils, lfc, Python. More than one external can be specified.
 

Create a user directory

Changed:
<
<
Use the command setenvDirac <version>. This will only work for a release installed in $LHCb_release_area unless you specify the --dev option if Dirac is installed in $LHCBDEV
>
>
Use the command setenvProject LHCbDirac <version>. This will only work for a release installed in $LHCb_release_area unless you specify the --dev option if LHCbDirac is installed in $LHCBDEV
 
Changed:
<
<

Check-out DIRAC packages

>
>

Check-out LHCbDIRAC packages

 This should be done only after the user directory has been created of course (see above)...
  1. source $LHCb_release_area/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.csh (you can put this in your login script, or better activate it adding a file .newLHCBLoginscript in your home directory)
Changed:
<
<
  1. cd ~/cmtuser/Dirac_<version>
>
>
  1. cd ~/cmtuser/LHCbDirac_<version>
 
  1. getpack <package> [head] where <package> must be preceded by the hat DIRAC/ if it is an actual DIRAC package
  2. ~/cmtuser/Dirac_<version>/DIRAC/dirac-make to create the scripts in the local area

Revision 212010-02-22 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

LHCbDirac client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 43 to 43
 Another guideline in this proposal is to avoid having to embed within the DIRAC distribution any external package but rather depend on external distribution of those as well as standard distribution with the LHCb Core Software tools. In particular the binary and library directories within DIRAC don't need to be distributed in the user tarball.

CMT projects and packages

Changed:
<
<
In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is as of v3r2. This consists in the following additions:
>
>
In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is as of v5r0. This consists in the following additions:
 
Changed:
<
<
  • File <DIRACBase>/cmt/project.cmt : contains the dependencies of DIRAC in terms of projects
  • Package DiracSys : this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
  • Package DiracPolicy : defines a number of policies for setting the environment. Also contains some utilities used by CMT.
  • Each DIRAC package listed above contains a cmt/requirements file listing dependencies (not used in a first instance) which is required for getpack. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-make) for a single package at once, in order to be able to (re)build a package in a developer's private area. DiracSys only contains dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.
>
>
  • File <DIRACBase>/cmt/project.cmt : contains the dependencies of LHCbDIRAC in terms of projects
  • Package LHCbDiracSys : this cmt package's requirements file defines the top dependencies of LHCbDIRAC
  • Package LHCbDiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
  • Package LHCbDiracPolicy : defines a number of policies for setting the environment. Also contains some utilities used by CMT.
  • Each LHCbDIRAC package listed above contains a cmt/requirements file listing dependencies (not used in a first instance) which is required for getpack. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-make) for a single package at once, in order to be able to (re)build a package in a developer's private area. LHCbDiracSys only contains dependencies on the LHCbDIRAC packages such that the whole of LHCbDIRAC can be built from the top.
 

External dependencies

Changed:
<
<
DIRAC has external dependencies on the following sets of packages provided by the LCG-AA:
>
>
LHCbDirac has external dependencies on the following sets of packages provided by the LCG-AA:
 
  • python
  • gLite client tools (gfal, lcg-utils, lfc, proxy handling)

Revision 202010-02-18 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Changed:
<
<

DIRAC3 client distribution and environment

>
>

LHCbDirac client distribution and environment

 This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.

Line: 119 to 119
 

User check out

Changed:
<
<
getpack and the CVS repository of DIRAC have been adapted to allow checking out a DIRAC package in a user directory (e.g. ~/cmtuser). Some actions must however be taken before running getpack (functionality of setenv<project>). They have been coded in the command setenvDirac <version>. This will create the appropriate structure in the cmtuser directory.
>
>
getpack and the SVN repository of LHCbDIRAC have been adapted to allow checking out a LHCbDIRAC package in a user directory (e.g. ~/cmtuser). Some actions must however be taken before running getpack (functionality of setenv<project>). They have been coded in the command setenvProject LHCbDirac <version>. This will create the appropriate structure in the cmtuser directory.
  Before using the getpack command you should issue the following commands:
> SetupProject LbScripts
Changed:
<
<
> cd ~/cmtuser/Dirac_=
>
>
> cd ~/cmtuser/LHCbDirac_=
 
Changed:
<
<
Using getpack -i allows to select the package in a list of available packages. The utility packages are located at the top level, e.g. getpack DiracConfig while the actual DIRAC packages have the hat DIRAC, e.g. getpack DIRAC/DataManagementSystem. Add the option head if you want to check out the head revision from CVS. One can check out the whole of DIRAC with the following command: getpack -r DiracSys
>
>
Using getpack -i allows to select the package in a list of available packages. The utility packages are located at the top level, e.g. getpack LHCbDiracConfig while the actual LHCbDIRAC packages have the hat LHCbDIRAC, e.g. getpack DIRAC/DataManagementSystem. Add the option head if you want to check out the head revision from SVN. One can check out the whole of LHCbDIRAC with the following command: getpack -r LHCbDiracSys
 
Changed:
<
<
Here are the actions performed by setenvDirac for reference:
  • Create a directory Dirac_<version> in the cmtuser directory (base, as all setenvProject)
  • Create a ./DIRAC directory
  • Copy __init__.py and dirac-make from the release area
  • Add the two magic lines to __init__.py:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
>
>
Here are the actions performed by setenvProject LHCbDirac for reference:
  • Create a directory LHCbDirac_<version> in the cmtuser directory (base, as all setenvProject)
 
  • Create a link to the ./etc directory of the release version of DIRAC.
  • Create a scripts directory and a link to it as ./InstallArea/scripts. This will automatically set the scripts on the PATH
  • Create a link to the base directory as ./InstallArea/python<version>. This will automatically set the PYTHONPATH as the base directory.

Building a local package

Changed:
<
<
Once a package has been checked out using getpack, it can be built running dirac-make in the DIRAC subdirectory. Eventually this utility will be made available from the cmt make command as for regular applications.
>
>
Once a package has been checked out using getpack, it can be built running make in the directory. Eventually this utility will be made available from the cmt make command as for regular applications.
  The build consists of the following steps (Ricardo should review this!):

Revision 192009-05-04 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 198 to 198
 
  1. dirac2AFS -v vArB -p $CMTCONFIG
  2. mkproject -p Dirac -v vArB -a tsrhw
  3. mkLHCbtar -p LHCbGrid -v vXrY -b $CMTCONFIG
Changed:
<
<
  1. mkLCGCMTtar -p DIRAC_vArB -b $CMTCONFIG --exclude="lib*.a"
>
>
  1. mkLCGCMTtar -n DIRAC_vArB -b $CMTCONFIG --exclude="lib*.a"
 
  1. mkLHCbtar -p Dirac -v vArB -b $CMTCONFIG

Revision 182009-04-14 - unknown

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Added:
>
>
 History:
  • First draft (PhC): 18-Nov-2008
  • Updated draft (PhC): 4-Dec-2008

Revision 172009-03-16 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 189 to 189
 
  1. copy the LCG_interfaces if necessary inside $LHCBRELEASE/LHCBGRID/LHCBGRID_vXrY/LCG_Interfaces
  2. mkproject -p LHCbGrid -v vXrY -a cd
  3. mkproject -p LHCbGrid -v vXrY -a rsrhw
Deleted:
<
<
  1. mkLCGCMTtar -p LCGGRID_vXrY -b $CMTCONFIG
  to create the distribution package for DIRAC:
  1. cd $LHCBRELEASE
Line: 197 to 196
 
  1. dirac2AFS -v vArB -p $CMTCONFIG
  2. mkproject -p Dirac -v vArB -a tsrhw
  3. mkLHCbtar -p LHCbGrid -v vXrY -b $CMTCONFIG
Added:
>
>
  1. mkLCGCMTtar -p DIRAC_vArB -b $CMTCONFIG --exclude="lib*.a"
 
  1. mkLHCbtar -p Dirac -v vArB -b $CMTCONFIG

Revision 162009-03-16 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 189 to 189
 
  1. copy the LCG_interfaces if necessary inside $LHCBRELEASE/LHCBGRID/LHCBGRID_vXrY/LCG_Interfaces
  2. mkproject -p LHCbGrid -v vXrY -a cd
  3. mkproject -p LHCbGrid -v vXrY -a rsrhw
Changed:
<
<
  1. mkLCGCMTtar -p LHCBGRID_vXrY -b $CMTCONFIG
>
>
  1. mkLCGCMTtar -p LCGGRID_vXrY -b $CMTCONFIG
  to create the distribution package for DIRAC:
  1. cd $LHCBRELEASE
  2. mkproject -p Dirac -v vArB -a n
  3. dirac2AFS -v vArB -p $CMTCONFIG
  4. mkproject -p Dirac -v vArB -a tsrhw
Changed:
<
<
  1. mkLCGCMTtar -p DIRAC_vArB -b $CMTCONFIG
>
>
  1. mkLHCbtar -p LHCbGrid -v vXrY -b $CMTCONFIG
 
  1. mkLHCbtar -p Dirac -v vArB -b $CMTCONFIG

Revision 152009-03-16 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 197 to 197
 
  1. dirac2AFS -v vArB -p $CMTCONFIG
  2. mkproject -p Dirac -v vArB -a tsrhw
  3. mkLCGCMTtar -p DIRAC_vArB -b $CMTCONFIG
Added:
>
>
  1. mkLHCbtar -p Dirac -v vArB -b $CMTCONFIG
 

-- PhilippeCharpentier - 18 Nov 2008

Revision 142009-03-12 - JoelClosier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 9 to 9
 
  • Proposal including discussions of Dec 5th (PhC): 8-Dec-2008
  • Added user guide on how to take benefit from it (PhC): 28-Jan-2009
  • Update user guide (PhC): 09-Mar-2009
Added:
>
>
 

Aim

The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly allow making tests with more recent (e.g bugfix) versions. The environment set up for the user should enable him/her to use most middleware commands (mainly Data Management) as well as their python bindings.
Line: 179 to 180
  You can modify your scripts/modules locally and then directly commit them to CVS after testing.

Added:
>
>

Instructions to create the DIRAC tarball

to create the distribution package for LHCbGrid:
  1. cd $LHCBRELEASE
  2. mkproject -p LHCbGrid -v vXrY -a ng
  3. copy the LCG_interfaces if necessary inside $LHCBRELEASE/LHCBGRID/LHCBGRID_vXrY/LCG_Interfaces
  4. mkproject -p LHCbGrid -v vXrY -a cd
  5. mkproject -p LHCbGrid -v vXrY -a rsrhw
  6. mkLCGCMTtar -p LHCBGRID_vXrY -b $CMTCONFIG

to create the distribution package for DIRAC:

  1. cd $LHCBRELEASE
  2. mkproject -p Dirac -v vArB -a n
  3. dirac2AFS -v vArB -p $CMTCONFIG
  4. mkproject -p Dirac -v vArB -a tsrhw
  5. mkLCGCMTtar -p DIRAC_vArB -b $CMTCONFIG
 -- PhilippeCharpentier - 18 Nov 2008

Revision 132009-03-09 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 8 to 8
 
  • Updated draft (PhC): 4-Dec-2008
  • Proposal including discussions of Dec 5th (PhC): 8-Dec-2008
  • Added user guide on how to take benefit from it (PhC): 28-Jan-2009
Added:
>
>
  • Update user guide (PhC): 09-Mar-2009
 

Aim

The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly allow making tests with more recent (e.g bugfix) versions. The environment set up for the user should enable him/her to use most middleware commands (mainly Data Management) as well as their python bindings.
Line: 165 to 166
 
    • SetupProject Dirac [<version>] <external> -v <external_version>: overwrites the default version of <external> with the specified version. <external> can be for example gfal, lcgutils, lfc, Python. More than one external can be specified.

Create a user directory

Changed:
<
<
Use the command setenvDirac <version>. This will only work for a release installed in $LHCb_release_area (for the timebeing).
>
>
Use the command setenvDirac <version>. This will only work for a release installed in $LHCb_release_area unless you specify the --dev option if Dirac is installed in $LHCBDEV
 

Check-out DIRAC packages

Changed:
<
<
This should be done only after the user directory has been created of course...
  1. source $LHCb_release_area/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.csh (provisional until this is the default LHCb login)
>
>
This should be done only after the user directory has been created of course (see above)...
  1. source $LHCb_release_area/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.csh (you can put this in your login script, or better activate it adding a file .newLHCBLoginscript in your home directory)
 
  1. cd ~/cmtuser/Dirac_<version>
  2. getpack <package> [head] where <package> must be preceded by the hat DIRAC/ if it is an actual DIRAC package
  3. ~/cmtuser/Dirac_<version>/DIRAC/dirac-make to create the scripts in the local area
Changed:
<
<
Note that after creating a local user directory for checking out modules, you should renew the SetupProject Dirac command is previously done in order to pick up the local installation.
>
>
Note that after creating a local user directory for checking out modules, you should renew the SetupProject Dirac command if previously done in order to pick up the local installation.
  You can modify your scripts/modules locally and then directly commit them to CVS after testing.

Revision 122009-01-30 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 169 to 169
 

Check-out DIRAC packages

This should be done only after the user directory has been created of course...
Changed:
<
<
  1. SetupProject LbLogin (provisional until this is the default LHCb login
>
>
  1. source $LHCb_release_area/LBSCRIPTS/prod/InstallArea/scripts/LbLogin.csh (provisional until this is the default LHCb login)
 
  1. cd ~/cmtuser/Dirac_<version>
  2. getpack <package> [head] where <package> must be preceded by the hat DIRAC/ if it is an actual DIRAC package
  3. ~/cmtuser/Dirac_<version>/DIRAC/dirac-make to create the scripts in the local area

Revision 112009-01-28 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 7 to 7
 
  • First draft (PhC): 18-Nov-2008
  • Updated draft (PhC): 4-Dec-2008
  • Proposal including discussions of Dec 5th (PhC): 8-Dec-2008
Changed:
<
<
  • Review by the proponents: to be done
>
>
  • Added user guide on how to take benefit from it (PhC): 28-Jan-2009
 

Aim

The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly allow making tests with more recent (e.g bugfix) versions. The environment set up for the user should enable him/her to use most middleware commands (mainly Data Management) as well as their python bindings.
Line: 44 to 44
 
  • File <DIRACBase>/cmt/project.cmt : contains the dependencies of DIRAC in terms of projects
  • Package DiracSys : this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
Changed:
<
<
  • Package DiracHome : temporary package used for finding the release area directory. It can also be used for setting some post-installation building rules in CMT if needed.
  • Each DIRAC package listed above will contain a cmt/requirements file listing dependencies (not used in a first instance) which is required for getpack. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-make) for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.
>
>
  • Package DiracPolicy : defines a number of policies for setting the environment. Also contains some utilities used by CMT.
  • Each DIRAC package listed above contains a cmt/requirements file listing dependencies (not used in a first instance) which is required for getpack. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-make) for a single package at once, in order to be able to (re)build a package in a developer's private area. DiracSys only contains dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.
 

External dependencies

DIRAC has external dependencies on the following sets of packages provided by the LCG-AA:
Line: 67 to 67
  Some modifications to the LCG externals have already been identified:
Changed:
<
<
  • The DM tools use a directory lib/python for their python binding. They should use lib/python<2digitVersion>. A link lib/python2.4 should be created pointing to the existing directory and the interface packages should be modified accordingly (Stefan, Hubert)
>
>
  • The DM tools use a directory lib/python for their python binding. They should use lib/python<2digitVersion>. A link lib/python2.4 should be created pointing to the existing directory and the interface packages should be modified accordingly (Stefan, Hubert). Done as of LCGCMT 55c.
 
  • It is noticed that pyQT has a dependency to the explicit 3-digit version of python for 2.4 but not for 2.5. We can probably survive like this unless we absolutely need 2.4.4 (Ricardo to check). In this case a 2-digit dependency should be used (can already be foreseen? Stefan)

Environment setting

Line: 77 to 77
 
  • InstallArea/scripts (link to scripts) : to be set on PATH
  • InstallArea/python (link to the top directory) : to be set on PYTHONPATH
Changed:
<
<
  • InstallArea/<CMTCONFIG>/python : to be set on PYTHONPATH
>
>
  • InstallArea/<CMTCONFIG>/python<2-digit-version> : to be set on PYTHONPATH
 
  • InstallArea/<CMTCONFIG>/lib : to be set on LD_LIBRARY_PATH (if needed)
  • InstallArea/<CMTCONFIG>/bin : to be set on PATH (if needed)
Line: 87 to 87
  As the python executables currently deployed are dynamically linked, the lib directory of python must be included in the LD_LIBRARY_PATH. Using any incompatible version of python without the library directory would not work, unless a static version is used.
Changed:
<
<
By default CMT doesn't check for the existence of directories included on the paths. There is a pattern in GaudiPolicy that allows that. It should eventually be moved to LCGPolicy, but in the mean time it will be moved to a DiracPolicy CMT package (Joel, Hubert). A clean up of the paths should also be added in DiracConfig (pattern path_remove).
>
>
By default CMT doesn't check for the existence of directories included on the paths. DiracPolicy ensures that all directories on the paths do exist (copied from GaudiPolicy)
 

Distribution packaging

Changed:
<
<
We propose to use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies from CMT. For DIRAC the dependencies are simple:
>
>
We use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies from CMT. For DIRAC the dependencies are simple:
 
  • LHCbGrid project (only interfaces and version definition)
  • gLite client middleware according to the version defined in LHCbGrid, including the file access client libraries (Castor, dCache, DPM). This tarball should contain a reduced list of libraries (limited to those that are used!)
Line: 100 to 100
  install_project uses a tarball that contains all platform-independent files (shared tarball), and another one with platform-dependent files. These tarballs can be created with a standard script. Like this one can install as many platforms as wanted on the same file system: the shared tarball is only installed once. None of these tarballs should contain the .pyc files as they are non-relocatable. After installation, it would be worth compiling the python modules (post-install action) and create the .pyc files. This is a post-installation for the shared tarball as the .pyc files are platform independent.
Changed:
<
<
By default, DIRAC should use the latest version of LHCbGrid, with a given major version (e.g. use LHCbGrid v1r*). In case one urgently needs to use specific versions of the middleware, one can fix them in DiracConfig if there is no LHCbGrid (yet) that contains these versions. Making a new release of LHCbGrid is the preferred solution for production versions. Superseding the versions is however very useful for testing them.
>
>
By default, DIRAC should use the latest version of LHCbGrid, with a given major version (e.g. use LHCbGrid v1r*). In case one urgently needs to use specific versions of the middleware, one can fix them in DiracConfig if there is no LHCbGrid (yet) that contains these versions. Making a new release of LHCbGrid is the preferred solution for production versions. Versions of externals can also be superseded from options in the SetupProject command (see later).

Currently, after a new release of DIRAC, it is installed on AFS at CERN using a dedicated script dirac2AFS that internally uses dirac_install and removes all unnecessary directories and files.

dirac2AFS.py <options>
  Installs DIRAC in $LHCb_release_area
  Options:
    -v, --version <version>: Manadatory, version to be installed
    -p, --platform <CMTCONFIG>: platform, default to $CMTCONFIG
    -t, --test <name>: name of the installed version for tests
    -d, --directory <dir>: directory where to install
    -D, --dev: install in $LHCBDEV
 

User check out

Changed:
<
<
getpack and the CVS repository of DIRAC have been adapted to allow checking out a DIRAC package in a user directory (e.g. ~/cmtuser). Some actions must however be taken before running getpack (functionality of setenv<project>).
>
>
getpack and the CVS repository of DIRAC have been adapted to allow checking out a DIRAC package in a user directory (e.g. ~/cmtuser). Some actions must however be taken before running getpack (functionality of setenv<project>). They have been coded in the command setenvDirac <version>. This will create the appropriate structure in the cmtuser directory.

Before using the getpack command you should issue the following commands:

> SetupProject LbScripts
> cd ~/cmtuser/Dirac_<version>=
 
Added:
>
>
Using getpack -i allows to select the package in a list of available packages. The utility packages are located at the top level, e.g. getpack DiracConfig while the actual DIRAC packages have the hat DIRAC, e.g. getpack DIRAC/DataManagementSystem. Add the option head if you want to check out the head revision from CVS. One can check out the whole of DIRAC with the following command: getpack -r DiracSys

Here are the actions performed by setenvDirac for reference:

 
  • Create a directory Dirac_<version> in the cmtuser directory (base, as all setenvProject)
  • Create a ./DIRAC directory
  • Copy __init__.py and dirac-make from the release area
Line: 115 to 136
 
  • Create a link to the ./etc directory of the release version of DIRAC.
  • Create a scripts directory and a link to it as ./InstallArea/scripts. This will automatically set the scripts on the PATH
Changed:
<
<
  • Create a link to the base directory as ./InstallArea/python. This will automatically set the PYTHONPATH as the base directory.
>
>
  • Create a link to the base directory as ./InstallArea/python<version>. This will automatically set the PYTHONPATH as the base directory.
 

Building a local package

Changed:
<
<
Once a package has been checked out using getpack, it can be built running dirac-make, which can also be done by CMT (cmt make). Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows). The build consists of the following steps (Ricardo should review this!):
>
>
Once a package has been checked out using getpack, it can be built running dirac-make in the DIRAC subdirectory. Eventually this utility will be made available from the cmt make command as for regular applications.

The build consists of the following steps (Ricardo should review this!):

 
  • Set up the scripts directory in order to only have one DIRAC directory on the PATH environment variable.
  • Compile the python modules (is this necessary?)
Added:
>
>
Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows).

User guide

This is a summary of commands to be used for taking advantage of the DIRAC installation using CMT.

Installing DIRAC locally

Refer to this link

Setup the DIRAC environment

  1. On a local installation, first execute source $MYSITEROOT/LbLogin.csh or . $MYSITEROOT/LbLogin.sh
  2. Use SetupProject to set the environment. Most useful forms:
    • SetupProject Dirac: sets up environment for the current CMTCONFIG and the latest version of Dirac
    • SetupProject Dirac --list: lists all versions available
    • SetupProject Dirac <version>: sts up a named version of Dirac
    • SetupProject Dirac --dev: looks in $LHCBDEV rather than $LHCb_release_area (for tests)
    • SetupProject Dirac [<version>] <external> -v <external_version>: overwrites the default version of <external> with the specified version. <external> can be for example gfal, lcgutils, lfc, Python. More than one external can be specified.

Create a user directory

Use the command setenvDirac <version>. This will only work for a release installed in $LHCb_release_area (for the timebeing).

Check-out DIRAC packages

This should be done only after the user directory has been created of course...
  1. SetupProject LbLogin (provisional until this is the default LHCb login
  2. cd ~/cmtuser/Dirac_<version>
  3. getpack <package> [head] where <package> must be preceded by the hat DIRAC/ if it is an actual DIRAC package
  4. ~/cmtuser/Dirac_<version>/DIRAC/dirac-make to create the scripts in the local area

Note that after creating a local user directory for checking out modules, you should renew the SetupProject Dirac command is previously done in order to pick up the local installation.

You can modify your scripts/modules locally and then directly commit them to CVS after testing.

 

-- PhilippeCharpentier - 18 Nov 2008

Revision 102008-12-08 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Added:
>
>
History:
  • First draft (PhC): 18-Nov-2008
  • Updated draft (PhC): 4-Dec-2008
  • Proposal including discussions of Dec 5th (PhC): 8-Dec-2008
  • Review by the proponents: to be done
 

Aim

Changed:
<
<
The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly make tests with newer versions. The environment set up for the user should enable him/her to use most middleware commands (mainly Data Management) as well as their python bindings.
>
>
The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly allow making tests with more recent (e.g bugfix) versions. The environment set up for the user should enable him/her to use most middleware commands (mainly Data Management) as well as their python bindings.
 
Changed:
<
<
The targeted tools are:
  • install_project : applies dependencies of CMT projects for downloading and installing packaged tarballs.
  • SetupProject : uses CMT for setting the run-time environment of the project.
  • getpack: allows to check out tagged versions of head of individual packages in a user area (created with SetupProject or a dedicated script)
>
>
The Core Software tools are:
  • install_project : applies dependencies of CMT projects for downloading and installing packaged tarballs.
  • SetupProject : uses CMT for setting the run-time environment of the project.
  • getpack : allows to check out tagged versions of head of individual packages in a user area (created with SetupProject or a dedicated script)
 
Changed:
<
<

Packages

>
>

DIRAC packages

 DIRAC is already split into packages: Framework, WMS, DMS etc... with well defined dependencies. The packages share a common base directory DIRAC. The full list is here:
  • AccountingSystem
  • BookkeepingSystem
Line: 28 to 34
  Inside each package, one can find standard directories: Agent, Client, Service and scripts. This overall structure is very similar to that of LHCb Gaudi applications and thus one can adapt easily the existing tools.
Changed:
<
<
Also similarly to LHCb applications, there is mainly one or two developer in charge of each package. The present proposal is trying to address the problem of testing software changes in one package, possibly against changes in other packages in a consistent way. This is understood not to be the current model but could potentially be very useful for pre-release testing, for example for exposing new releases of the bookkeeping GUI prior to the DIRAC releases, or test new versions of middleware.
>
>
Also similarly to LHCb applications, there are mainly one or two developers in charge of each package. The present proposal is also trying to address the problem of testing software changes in one package, possibly against changes in other packages in a consistent way. This is understood not to be the current model but could potentially be very useful for pre-release testing, for example for exposing new releases of the bookkeeping GUI prior to the DIRAC releases, or test new versions of middleware.
  Another guideline in this proposal is to avoid having to embed within the DIRAC distribution any external package but rather depend on external distribution of those as well as standard distribution with the LHCb Core Software tools. In particular the binary and library directories within DIRAC don't need to be distributed in the user tarball.

CMT projects and packages

Changed:
<
<
In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is. This consists in the following additions:
>
>
In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is as of v3r2. This consists in the following additions:
 
Changed:
<
<
  • File <DIRACBase>/cmt/project.cmt that contains the dependencies of DIRAC in terms of projects
>
>
  • File <DIRACBase>/cmt/project.cmt : contains the dependencies of DIRAC in terms of projects
 
  • Package DiracSys : this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
Changed:
<
<
  • Each package listed above will contain a cmt/requirements file listing dependencies (not used in a first instance) which is required for getpack. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-make), but for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.
>
>
  • Package DiracHome : temporary package used for finding the release area directory. It can also be used for setting some post-installation building rules in CMT if needed.
  • Each DIRAC package listed above will contain a cmt/requirements file listing dependencies (not used in a first instance) which is required for getpack. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-make) for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.
 

External dependencies

DIRAC has external dependencies on the following sets of packages provided by the LCG-AA:
Line: 47 to 54
 
  • gLite client tools (gfal, lcg-utils, lfc, proxy handling)
  • pyQt (for the bookkeeping GUI)
  • cx-oracle (for the BK service)
Changed:
<
<
  • mySQL, SQLite
>
>
  • mySQL, SQLite and more... (for services)
  Only the first 3 are needed for client distribution while the others (and many others) are needed for running services. The issue of installing services will be discussed separately but most of it can also benefit from this distribution toolkit. Additional software must be installed separately as addition to he OS.

All these packages are made available for the LCG-supported platforms in the LCG Applications Area repository (lcg/external). LCG provides CMT requirements files for setting properly the environment for these packages. Any request for new packages or for installing most recent versions can easily be made through the Architects' Forum (and even directly to the person responsible).

LHCbGrid project

Changed:
<
<
The LHCbGrid project is meant at gathering the information about which versions of the gLite middleware are to be used by LHCb, both for DIRAC and for Applications (they also depend on gfal and lfc, and in addition on Castor, dCache and possibly DPM client libraries). A coordination is needed for deciding on which versions have to be used. These versions can be defined within the requirements of the LHCbGrid project and can even be superseded in the DiracConfig requirements file (e.g. for testing or for quick move to a bugfix version).
>
>
The LHCbGrid project is meant at gathering the information about which versions of the gLite middleware and DM middleware are to be used by LHCb, both for DIRAC and for Applications (they also depend on gfal and lfc, and in addition on Castor, dCache and possibly DPM client libraries). A coordination is needed for deciding on which versions have to be used. These versions can be defined within the requirements of the LHCbGrid project and can even be superseded in the DiracConfig requirements file (e.g. for testing or for quick move to a bugfix version).
  As most requirements are coming from DIRAC, the changes in versions should be driven by DIRAC's needs. Requirements for getting any new version made available in the lcg/externals should be made by the DIRAC developers to the Architects' Forum via the LHCb representatives: Marco Cattaneo, Philippe Charpentier (in CC).

Some modifications to the LCG externals have already been identified:

Changed:
<
<
  • The DM tools use a directory lib/python for their python binding. They should use lib/python<2digitVersion>. A link lib/python2.4 should be created pointing to the existing directory and the interface packages should be modified accordingly (Stefan, Hubert)
>
>
  • The DM tools use a directory lib/python for their python binding. They should use lib/python<2digitVersion>. A link lib/python2.4 should be created pointing to the existing directory and the interface packages should be modified accordingly (Stefan, Hubert)
 
  • It is noticed that pyQT has a dependency to the explicit 3-digit version of python for 2.4 but not for 2.5. We can probably survive like this unless we absolutely need 2.4.4 (Ricardo to check). In this case a 2-digit dependency should be used (can already be foreseen? Stefan)

Environment setting

Changed:
<
<
CMT sets up the environment variables using either predefined patterns or running scripts. The LCGCMT interface package sets the PATH, LD_LIBRARY_PATH and PYTHONPATH such that all commands as well as standalone usage of e.g. python binding can be used. Obviously if this environment is broken accidentally of by setting another environment, these tools might no longer work. However it is assumed users understand what an environment means, which doesn't prevent having as much environment-free software as possible....
>
>
The command SetupProject Dirac [<version>] should be used to set up the environment, using CMT. CMT sets up the environment variables based on the DiracPolicy that is a copy of GaudiPolicy. It uses the environment variable CMTCONFIG for the platform-dependent part. As most of this dependency is in externals, most is done by LCGCMT and not directly in DIRAC. The only identified platform-dependent part in DIRAC is pyGSI that eventually could be sub-contracted to the LCG-AA (Adri to get in touch with Stefan)

CMT uses the InstallArea that may contain the following directories:

  • InstallArea/scripts (link to scripts) : to be set on PATH
  • InstallArea/python (link to the top directory) : to be set on PYTHONPATH
  • InstallArea/<CMTCONFIG>/python : to be set on PYTHONPATH
  • InstallArea/<CMTCONFIG>/lib : to be set on LD_LIBRARY_PATH (if needed)
  • InstallArea/<CMTCONFIG>/bin : to be set on PATH (if needed)

For middleware, the LCGCMT interface package also sets the PATH, LD_LIBRARY_PATH and PYTHONPATH such that all commands as well as standalone usage of e.g. python binding can be used. Obviously if this environment is broken accidentally of by setting another environment, these tools might no longer work. However it is assumed users understand what an environment means, which doesn't prevent having as much environment-free software as possible....

The supported platforms are the same as for LCG and the environment variable CMTCONFIG is defining it. A one-to-one correspondence exists between these platform names and the DIRAC ones (note that the LCG platform names are going to change as of LCGCMT 56 and become closer to DIRAC ones).

  As the python executables currently deployed are dynamically linked, the lib directory of python must be included in the LD_LIBRARY_PATH. Using any incompatible version of python without the library directory would not work, unless a static version is used.
Changed:
<
<
By default CMT doesn't check for the existence of directories included on the paths. There is a pattern in GaudiPolicy that allows that. It should eventually be moved to LCGPolicy, but in the mean time it will be moved to a DIRAC CMT package (Joel, Hubert). A clean up of the paths should also be added in DiracConfig (pattern path_remove)
>
>
By default CMT doesn't check for the existence of directories included on the paths. There is a pattern in GaudiPolicy that allows that. It should eventually be moved to LCGPolicy, but in the mean time it will be moved to a DiracPolicy CMT package (Joel, Hubert). A clean up of the paths should also be added in DiracConfig (pattern path_remove).

Distribution packaging

We propose to use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies from CMT. For DIRAC the dependencies are simple:

  • LHCbGrid project (only interfaces and version definition)
  • gLite client middleware according to the version defined in LHCbGrid, including the file access client libraries (Castor, dCache, DPM). This tarball should contain a reduced list of libraries (limited to those that are used!)
  • Other external packages: python, pyQt and its dependencies

Using install_project ensures that the necessary LHCb scripts are also installed and the configuration is properly defined (such as !SetupProject, LbLogin etc...). If they have already been installed when users have installed the LHCb applications, they will not need to be reinstalled. The use case is mostly for ganga users who anyway need to install the applications, and they should be advised to install DIRAC in the same area.

 
Changed:
<
<

Checking out

The DIRAC software is managed in CVS, and it is assumed that only the DIRAC source modules are stored in CVS. Therefore packages have to be built before being usable. Then the built DIRAC can be packaged as a tarball that can be used for distribution.
>
>
install_project uses a tarball that contains all platform-independent files (shared tarball), and another one with platform-dependent files. These tarballs can be created with a standard script. Like this one can install as many platforms as wanted on the same file system: the shared tarball is only installed once. None of these tarballs should contain the .pyc files as they are non-relocatable. After installation, it would be worth compiling the python modules (post-install action) and create the .pyc files. This is a post-installation for the shared tarball as the .pyc files are platform independent.
 
Changed:
<
<
getpack and the CVS repository of DIRAC have been adapted to allow checking out a DIRAC package with getpack. A actions must be taken before doing getpack (functionality of setenv).
>
>
By default, DIRAC should use the latest version of LHCbGrid, with a given major version (e.g. use LHCbGrid v1r*). In case one urgently needs to use specific versions of the middleware, one can fix them in DiracConfig if there is no LHCbGrid (yet) that contains these versions. Making a new release of LHCbGrid is the preferred solution for production versions. Superseding the versions is however very useful for testing them.

User check out

getpack and the CVS repository of DIRAC have been adapted to allow checking out a DIRAC package in a user directory (e.g. ~/cmtuser). Some actions must however be taken before running getpack (functionality of setenv<project>).
 
  • Create a directory Dirac_<version> in the cmtuser directory (base, as all setenvProject)
  • Create a ./DIRAC directory
  • Copy __init__.py and dirac-make from the release area
Changed:
<
<
  • Add the two magic line to __init__.py:
>
>
  • Add the two magic lines to __init__.py:
 
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
Changed:
<
<
  • Create a link to the released ./etc directory
>
>
  • Create a link to the ./etc directory of the release version of DIRAC.
 
  • Create a scripts directory and a link to it as ./InstallArea/scripts. This will automatically set the scripts on the PATH
  • Create a link to the base directory as ./InstallArea/python. This will automatically set the PYTHONPATH as the base directory.
Changed:
<
<

Building DIRAC

Once a package has been checked out, it can be built running dirac-make, which can be done by CMT. Building DIRAC is of course platform-dependent. The supported platforms are the same as for LCG. Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows). The build consists of the following steps (Ricardo should correct this!):
>
>

Building a local package

Once a package has been checked out using getpack, it can be built running dirac-make, which can also be done by CMT (cmt make). Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows). The build consists of the following steps (Ricardo should review this!):
 
Deleted:
<
<
  • Compile C/C++ code using the agreed compiler (is there any such compilation?)
 
  • Set up the scripts directory in order to only have one DIRAC directory on the PATH environment variable.
  • Compile the python modules (is this necessary?)
Deleted:
<
<

Distribution packaging

We propose to use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies (from CMT). For DIRAC the dependencies are simple:

  • LHCbGrid project (only interfaces and version definition)
  • gLite client middleware according to the version defined in LHCbGrid, including the file access client libraries (Castor, dCache, DPM). This tarball should contain a reduced list of libraries (limited to those that are used!)
  • Other external packages: python, pyQt and its dependencies

Using install_project ensures that the necessary scripts are also installed (such as SetupProject, lbLogin etc...). If they have already been installed when users have installed the LHCb applications, they will not need to be reinstalled. The use case is mostly for ganga users who anyway need to install the applications, and they should be advised to install DIRAC in the same area.

After installation of the python modules, it would be worth compiling them (post-install action) and create the .pyc files.

Unlike for applications where older versions need to use the latest version of LHCbGrid, DIRAC needs to fix those version, which can be done either using the latest version of LHCbGrid or event redefining the version if there is no LHCbGrid that contains these versions. Making a new release of LHCbGrid is however the preferred solution for production versions. Superseding the version is however very useful for testing them.

 

-- PhilippeCharpentier - 18 Nov 2008 \ No newline at end of file

Revision 92008-12-08 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
Line: 70 to 70
  By default CMT doesn't check for the existence of directories included on the paths. There is a pattern in GaudiPolicy that allows that. It should eventually be moved to LCGPolicy, but in the mean time it will be moved to a DIRAC CMT package (Joel, Hubert). A clean up of the paths should also be added in DiracConfig (pattern path_remove)
Changed:
<
<

Checking out and building DIRAC

>
>

Checking out

 The DIRAC software is managed in CVS, and it is assumed that only the DIRAC source modules are stored in CVS. Therefore packages have to be built before being usable. Then the built DIRAC can be packaged as a tarball that can be used for distribution.
Changed:
<
<
getpack and the CVS repository of DIRAC have been adapted to allow checking out a DIRAC package with getpack. A few local directories and links must have been created before doing getpack (functionality of setenv)
>
>
getpack and the CVS repository of DIRAC have been adapted to allow checking out a DIRAC package with getpack. A actions must be taken before doing getpack (functionality of setenv).
 
Changed:
<
<
Building DIRAC is of course platform-dependent. The supported platforms are the same as for LCG. Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows). The build consists of the following steps (Ricardo should correct this!):

  • Compile C/C++ code using the agreed compiler (is there any such compilation?)
  • set up the scripts directory in order to only have one DIRAC directory on the PATH environment variable.

CMT can easily be set up for building DIRAC packages, either using predefined patterns, new patterns or simply DIRAC scripts for more complex operations.

For checking out DIRAC from CVS, one can make use of the getpack general utility (to be slightly adapted) that is used for the application packages. It has the advantage of being able to use dependencies for checking out dependent packages. It also allows to set up the directory structure as required by CMT. This is an option, as dirac_install also has the checkout possibility. getpack would also allow to checkout single packages (e.g only the dependency definition packages for testing new versions of middleware). getpack would need to be instructed to also checkout the intermediate directories in order to have in particular the init.py files at all levels.

Using CMT for setting up the environment allows to use more than one installation of DIRAC using the CMTPROJECTPATH environment variable. This may allow to easily test a modified package against an existing release of DIRAC or against packages modified by another developer, or a development distribution (LHCBDEV). In order for python to accept having more than one instance of the DIRAC python package, two lines need to be inserted in DIRAC/__init__.py (can be done at build time, or be in the checked-in file):

>
>
  • Create a directory Dirac_<version> in the cmtuser directory (base, as all setenvProject)
  • Create a ./DIRAC directory
  • Copy __init__.py and dirac-make from the release area
  • Add the two magic line to __init__.py:
 
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
Added:
>
>
  • Create a link to the released ./etc directory
  • Create a scripts directory and a link to it as ./InstallArea/scripts. This will automatically set the scripts on the PATH
  • Create a link to the base directory as ./InstallArea/python. This will automatically set the PYTHONPATH as the base directory.

Building DIRAC

Once a package has been checked out, it can be built running dirac-make, which can be done by CMT. Building DIRAC is of course platform-dependent. The supported platforms are the same as for LCG. Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows). The build consists of the following steps (Ricardo should correct this!):

  • Compile C/C++ code using the agreed compiler (is there any such compilation?)
  • Set up the scripts directory in order to only have one DIRAC directory on the PATH environment variable.
  • Compile the python modules (is this necessary?)
 

Distribution packaging

We propose to use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies (from CMT). For DIRAC the dependencies are simple:

Revision 82008-12-08 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Changed:
<
<

DIRAC3 client distribution and environment (Draft)

>
>

DIRAC3 client distribution and environment

This document has been updated following a meeting held on December 5th with Ricardo, Hubert, Joel, Marco Cl and Philippe.
 

Aim

Changed:
<
<
The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly make tests with newer versions.
>
>
The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly make tests with newer versions. The environment set up for the user should enable him/her to use most middleware commands (mainly Data Management) as well as their python bindings.
  The targeted tools are:
  • install_project : applies dependencies of CMT projects for downloading and installing packaged tarballs.
  • SetupProject : uses CMT for setting the run-time environment of the project.
Added:
>
>
  • getpack: allows to check out tagged versions of head of individual packages in a user area (created with SetupProject or a dedicated script)
 

Packages

DIRAC is already split into packages: Framework, WMS, DMS etc... with well defined dependencies. The packages share a common base directory DIRAC. The full list is here:
Line: 26 to 28
  Inside each package, one can find standard directories: Agent, Client, Service and scripts. This overall structure is very similar to that of LHCb Gaudi applications and thus one can adapt easily the existing tools.
Changed:
<
<
Also similarly to LHCb applications, there is mainly one or two developer in charge of each package. The present proposal is trying to address the problem of testing software changes in one package, possibly against changes in other packages in a consistent way. This is understood not to be the current model but could potentially be very useful for pre-release testing.
>
>
Also similarly to LHCb applications, there is mainly one or two developer in charge of each package. The present proposal is trying to address the problem of testing software changes in one package, possibly against changes in other packages in a consistent way. This is understood not to be the current model but could potentially be very useful for pre-release testing, for example for exposing new releases of the bookkeeping GUI prior to the DIRAC releases, or test new versions of middleware.
 
Changed:
<
<
Another guideline in this proposal is to avoid having to embed within the DIRAC distribution any external package but rather depend on external distribution of those as well as standard distribution with the LHCb Core Software tools.
>
>
Another guideline in this proposal is to avoid having to embed within the DIRAC distribution any external package but rather depend on external distribution of those as well as standard distribution with the LHCb Core Software tools. In particular the binary and library directories within DIRAC don't need to be distributed in the user tarball.
 

CMT projects and packages

In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is. This consists in the following additions:
Line: 36 to 38
 
  • File <DIRACBase>/cmt/project.cmt that contains the dependencies of DIRAC in terms of projects
  • Package DiracSys : this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
Changed:
<
<
Similarly the DIRAC functional packages could be made CMT packages by adding a cmt/requirements file in each package. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-install), but for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.
>
>
  • Each package listed above will contain a cmt/requirements file listing dependencies (not used in a first instance) which is required for getpack. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-make), but for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.
 

External dependencies

Changed:
<
<
DIRAC has external dependencies on the following sets of packages:
>
>
DIRAC has external dependencies on the following sets of packages provided by the LCG-AA:
 
  • python
  • gLite client tools (gfal, lcg-utils, lfc, proxy handling)
  • pyQt (for the bookkeeping GUI)
  • cx-oracle (for the BK service)
Changed:
<
<
  • mySQL, SQLite?
>
>
  • mySQL, SQLite

Only the first 3 are needed for client distribution while the others (and many others) are needed for running services. The issue of installing services will be discussed separately but most of it can also benefit from this distribution toolkit. Additional software must be installed separately as addition to he OS.

  All these packages are made available for the LCG-supported platforms in the LCG Applications Area repository (lcg/external). LCG provides CMT requirements files for setting properly the environment for these packages. Any request for new packages or for installing most recent versions can easily be made through the Architects' Forum (and even directly to the person responsible).

LHCbGrid project

The LHCbGrid project is meant at gathering the information about which versions of the gLite middleware are to be used by LHCb, both for DIRAC and for Applications (they also depend on gfal and lfc, and in addition on Castor, dCache and possibly DPM client libraries). A coordination is needed for deciding on which versions have to be used. These versions can be defined within the requirements of the LHCbGrid project and can even be superseded in the DiracConfig requirements file (e.g. for testing or for quick move to a bugfix version).
Changed:
<
<

CMT environment setting

>
>
As most requirements are coming from DIRAC, the changes in versions should be driven by DIRAC's needs. Requirements for getting any new version made available in the lcg/externals should be made by the DIRAC developers to the Architects' Forum via the LHCb representatives: Marco Cattaneo, Philippe Charpentier (in CC).

Some modifications to the LCG externals have already been identified:

  • The DM tools use a directory lib/python for their python binding. They should use lib/python<2digitVersion>. A link lib/python2.4 should be created pointing to the existing directory and the interface packages should be modified accordingly (Stefan, Hubert)
  • It is noticed that pyQT has a dependency to the explicit 3-digit version of python for 2.4 but not for 2.5. We can probably survive like this unless we absolutely need 2.4.4 (Ricardo to check). In this case a 2-digit dependency should be used (can already be foreseen? Stefan)

Environment setting

 CMT sets up the environment variables using either predefined patterns or running scripts. The LCGCMT interface package sets the PATH, LD_LIBRARY_PATH and PYTHONPATH such that all commands as well as standalone usage of e.g. python binding can be used. Obviously if this environment is broken accidentally of by setting another environment, these tools might no longer work. However it is assumed users understand what an environment means, which doesn't prevent having as much environment-free software as possible....
Changed:
<
<

Building DIRAC

The DIRAC software is managed in CVS, and it is assumed that only the DIRAC source modules are stored in CVS. Therefore packages have to be built before being usable. Then the built DIRAC can be packaged as a tarball that can be used for distributing it.
>
>
As the python executables currently deployed are dynamically linked, the lib directory of python must be included in the LD_LIBRARY_PATH. Using any incompatible version of python without the library directory would not work, unless a static version is used.

By default CMT doesn't check for the existence of directories included on the paths. There is a pattern in GaudiPolicy that allows that. It should eventually be moved to LCGPolicy, but in the mean time it will be moved to a DIRAC CMT package (Joel, Hubert). A clean up of the paths should also be added in DiracConfig (pattern path_remove)

Checking out and building DIRAC

The DIRAC software is managed in CVS, and it is assumed that only the DIRAC source modules are stored in CVS. Therefore packages have to be built before being usable. Then the built DIRAC can be packaged as a tarball that can be used for distribution.

getpack and the CVS repository of DIRAC have been adapted to allow checking out a DIRAC package with getpack. A few local directories and links must have been created before doing getpack (functionality of setenv)

  Building DIRAC is of course platform-dependent. The supported platforms are the same as for LCG. Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows). The build consists of the following steps (Ricardo should correct this!):

Revision 72008-12-01 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment (Draft)

Line: 62 to 62
 Building DIRAC is of course platform-dependent. The supported platforms are the same as for LCG. Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows). The build consists of the following steps (Ricardo should correct this!):

  • Compile C/C++ code using the agreed compiler (is there any such compilation?)
Deleted:
<
<
  • Compile python modules (create .pyc) using the selected version of python
 
  • set up the scripts directory in order to only have one DIRAC directory on the PATH environment variable.

CMT can easily be set up for building DIRAC packages, either using predefined patterns, new patterns or simply DIRAC scripts for more complex operations.

Line: 84 to 83
  Using install_project ensures that the necessary scripts are also installed (such as SetupProject, lbLogin etc...). If they have already been installed when users have installed the LHCb applications, they will not need to be reinstalled. The use case is mostly for ganga users who anyway need to install the applications, and they should be advised to install DIRAC in the same area.
Added:
>
>
After installation of the python modules, it would be worth compiling them (post-install action) and create the .pyc files.
 Unlike for applications where older versions need to use the latest version of LHCbGrid, DIRAC needs to fix those version, which can be done either using the latest version of LHCbGrid or event redefining the version if there is no LHCbGrid that contains these versions. Making a new release of LHCbGrid is however the preferred solution for production versions. Superseding the version is however very useful for testing them.

Revision 62008-11-30 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment (Draft)

Line: 34 to 34
 In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is. This consists in the following additions:

  • File =<DIRACBase>/cmt/project.cmt=that contains the dependencies of DIRAC in terms of projects
Changed:
<
<
  • Package DiracSys: this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
>
>
  • Package DiracSys : this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
  Similarly the DIRAC functional packages could be made CMT packages by adding a cmt/requirements file in each package. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-install), but for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.

Revision 52008-11-28 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Changed:
<
<

DIRAC3 client distribution and environment

>
>

DIRAC3 client distribution and environment (Draft)

 

Aim

The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly make tests with newer versions.
Line: 78 to 78
 

Distribution packaging

We propose to use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies (from CMT). For DIRAC the dependencies are simple:
Changed:
<
<
  • LHCbGrid project (only interfaces and version definition)
  • gLite client middleware according to the version defined in LHCbGrid, including the file access client libraries (Castor, dCache, DPM). This tarball should contain a reduced list of libraries (limited to those that are used!)
  • Other external packages: python, pyQt and its dependencies
>
>
  • LHCbGrid project (only interfaces and version definition)
  • gLite client middleware according to the version defined in LHCbGrid, including the file access client libraries (Castor, dCache, DPM). This tarball should contain a reduced list of libraries (limited to those that are used!)
  • Other external packages: python, pyQt and its dependencies
 
Changed:
<
<
Using install_project ensures that the necessary scripts are also installed (such as SetupProject, lbLogin etc...). If they have already been installed when users have installed the LHCb applications, they will not need to be reinstalled. The use case is mostly for ganga users who anyway need to install the applications, and they should be advised to install DIRAC in the same area.
>
>
Using install_project ensures that the necessary scripts are also installed (such as SetupProject, lbLogin etc...). If they have already been installed when users have installed the LHCb applications, they will not need to be reinstalled. The use case is mostly for ganga users who anyway need to install the applications, and they should be advised to install DIRAC in the same area.
 
Changed:
<
<
Unlike for applications where older versions need to use the latest version of LHCbGrid, DIRAC needs to fix those version, which can be done either using the latest version of LHCbGrid or event redefining the version if there is no LHCbGrid that contains these versions. Making a new release of LHCbGrid is however the preferred solution for production versions. Superseding the version is however very useful for testing them.
>
>
Unlike for applications where older versions need to use the latest version of LHCbGrid, DIRAC needs to fix those version, which can be done either using the latest version of LHCbGrid or event redefining the version if there is no LHCbGrid that contains these versions. Making a new release of LHCbGrid is however the preferred solution for production versions. Superseding the version is however very useful for testing them.
 

-- PhilippeCharpentier - 18 Nov 2008

Revision 42008-11-27 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

Aim

Changed:
<
<
The aim is to allow the DIRAC3 client distribution to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly make tests with newer versions.
>
>
The aim is to allow the DIRAC3 distribution mechanism to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly make tests with newer versions.
  The targeted tools are:
  • install_project : applies dependencies of CMT projects for downloading and installing packaged tarballs.
Line: 24 to 24
 
  • StagerSystem
  • WorkloadManagementSystem
Changed:
<
<
Inside each package, one can find standard directories: Agent, Client, Service and scripts. This overall structure is very similar to that of LHCb Gaudi applications and thus one can adapt easily the existing methods.
>
>
Inside each package, one can find standard directories: Agent, Client, Service and scripts. This overall structure is very similar to that of LHCb Gaudi applications and thus one can adapt easily the existing tools.
  Also similarly to LHCb applications, there is mainly one or two developer in charge of each package. The present proposal is trying to address the problem of testing software changes in one package, possibly against changes in other packages in a consistent way. This is understood not to be the current model but could potentially be very useful for pre-release testing.
Changed:
<
<
Another guideline in this proposal is to avoid having to embed within the DIRAC packaging any external package but rather depend on external distribution of those as well as standard distribution with the LHCb Core Software tools.
>
>
Another guideline in this proposal is to avoid having to embed within the DIRAC distribution any external package but rather depend on external distribution of those as well as standard distribution with the LHCb Core Software tools.
 

CMT projects and packages

In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is. This consists in the following additions:
Line: 37 to 37
 
  • Package DiracSys: this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
Changed:
<
<
Similarly the DIRAC functional packages could be made CMT packages by adding a cmt/requirements file in each package. This requirements file could be use for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-install), but for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built fro the top.
>
>
Similarly the DIRAC functional packages could be made CMT packages by adding a cmt/requirements file in each package. This requirements file could be used for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-install), but for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built from the top.
 

External dependencies

DIRAC has external dependencies on the following sets of packages:
Line: 46 to 46
 
  • gLite client tools (gfal, lcg-utils, lfc, proxy handling)
  • pyQt (for the bookkeeping GUI)
  • cx-oracle (for the BK service)
Added:
>
>
  • mySQL, SQLite?
  All these packages are made available for the LCG-supported platforms in the LCG Applications Area repository (lcg/external). LCG provides CMT requirements files for setting properly the environment for these packages. Any request for new packages or for installing most recent versions can easily be made through the Architects' Forum (and even directly to the person responsible).
Line: 66 to 67
  CMT can easily be set up for building DIRAC packages, either using predefined patterns, new patterns or simply DIRAC scripts for more complex operations.
Added:
>
>
For checking out DIRAC from CVS, one can make use of the getpack general utility (to be slightly adapted) that is used for the application packages. It has the advantage of being able to use dependencies for checking out dependent packages. It also allows to set up the directory structure as required by CMT. This is an option, as dirac_install also has the checkout possibility. getpack would also allow to checkout single packages (e.g only the dependency definition packages for testing new versions of middleware). getpack would need to be instructed to also checkout the intermediate directories in order to have in particular the init.py files at all levels.

Using CMT for setting up the environment allows to use more than one installation of DIRAC using the CMTPROJECTPATH environment variable. This may allow to easily test a modified package against an existing release of DIRAC or against packages modified by another developer, or a development distribution (LHCBDEV). In order for python to accept having more than one instance of the DIRAC python package, two lines need to be inserted in DIRAC/__init__.py (can be done at build time, or be in the checked-in file):

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

Distribution packaging

We propose to use the standard mechanism that LHCb users are used to for installing locally the software. It is based on the install_project script that itself relies on a description of the dependencies (from CMT). For DIRAC the dependencies are simple:

  • LHCbGrid project (only interfaces and version definition)
  • gLite client middleware according to the version defined in LHCbGrid, including the file access client libraries (Castor, dCache, DPM). This tarball should contain a reduced list of libraries (limited to those that are used!)
  • Other external packages: python, pyQt and its dependencies

Using install_project ensures that the necessary scripts are also installed (such as SetupProject, lbLogin etc...). If they have already been installed when users have installed the LHCb applications, they will not need to be reinstalled. The use case is mostly for ganga users who anyway need to install the applications, and they should be advised to install DIRAC in the same area.

Unlike for applications where older versions need to use the latest version of LHCbGrid, DIRAC needs to fix those version, which can be done either using the latest version of LHCbGrid or event redefining the version if there is no LHCbGrid that contains these versions. Making a new release of LHCbGrid is however the preferred solution for production versions. Superseding the version is however very useful for testing them.

 -- PhilippeCharpentier - 18 Nov 2008

Revision 32008-11-27 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

Line: 26 to 26
  Inside each package, one can find standard directories: Agent, Client, Service and scripts. This overall structure is very similar to that of LHCb Gaudi applications and thus one can adapt easily the existing methods.
Added:
>
>
Also similarly to LHCb applications, there is mainly one or two developer in charge of each package. The present proposal is trying to address the problem of testing software changes in one package, possibly against changes in other packages in a consistent way. This is understood not to be the current model but could potentially be very useful for pre-release testing.

Another guideline in this proposal is to avoid having to embed within the DIRAC packaging any external package but rather depend on external distribution of those as well as standard distribution with the LHCb Core Software tools.

 

CMT projects and packages

In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is. This consists in the following additions:
Line: 33 to 37
 
  • Package DiracSys: this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
Added:
>
>
Similarly the DIRAC functional packages could be made CMT packages by adding a cmt/requirements file in each package. This requirements file could be use for building the package (i.e. creating the necessary file/links in for example the scripts directory, which is currently done by dirac-install), but for a single package at once, in order to be able to (re)build a package in a developer's private area. Then DiracSys would just contain dependencies on the DIRAC packages such that the whole of DIRAC can be built fro the top.

External dependencies

DIRAC has external dependencies on the following sets of packages:

  • python
  • gLite client tools (gfal, lcg-utils, lfc, proxy handling)
  • pyQt (for the bookkeeping GUI)
  • cx-oracle (for the BK service)

All these packages are made available for the LCG-supported platforms in the LCG Applications Area repository (lcg/external). LCG provides CMT requirements files for setting properly the environment for these packages. Any request for new packages or for installing most recent versions can easily be made through the Architects' Forum (and even directly to the person responsible).

LHCbGrid project

The LHCbGrid project is meant at gathering the information about which versions of the gLite middleware are to be used by LHCb, both for DIRAC and for Applications (they also depend on gfal and lfc, and in addition on Castor, dCache and possibly DPM client libraries). A coordination is needed for deciding on which versions have to be used. These versions can be defined within the requirements of the LHCbGrid project and can even be superseded in the DiracConfig requirements file (e.g. for testing or for quick move to a bugfix version).

CMT environment setting

CMT sets up the environment variables using either predefined patterns or running scripts. The LCGCMT interface package sets the PATH, LD_LIBRARY_PATH and PYTHONPATH such that all commands as well as standalone usage of e.g. python binding can be used. Obviously if this environment is broken accidentally of by setting another environment, these tools might no longer work. However it is assumed users understand what an environment means, which doesn't prevent having as much environment-free software as possible....

Building DIRAC

The DIRAC software is managed in CVS, and it is assumed that only the DIRAC source modules are stored in CVS. Therefore packages have to be built before being usable. Then the built DIRAC can be packaged as a tarball that can be used for distributing it.

Building DIRAC is of course platform-dependent. The supported platforms are the same as for LCG. Note that not all functionality might be available on all platforms (e.g. gLite on OSX and Windows). The build consists of the following steps (Ricardo should correct this!):

  • Compile C/C++ code using the agreed compiler (is there any such compilation?)
  • Compile python modules (create .pyc) using the selected version of python
  • set up the scripts directory in order to only have one DIRAC directory on the PATH environment variable.

CMT can easily be set up for building DIRAC packages, either using predefined patterns, new patterns or simply DIRAC scripts for more complex operations.

 -- PhilippeCharpentier - 18 Nov 2008

Revision 22008-11-18 - PhilippeCharpentier

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

Line: 10 to 10
 
  • SetupProject : uses CMT for setting the run-time environment of the project.

Packages

Changed:
<
<
DIRAC is already split into packages: Core, WMS, DMS etc... with well defined dependencies
>
>
DIRAC is already split into packages: Framework, WMS, DMS etc... with well defined dependencies. The packages share a common base directory DIRAC. The full list is here:
  • AccountingSystem
  • BookkeepingSystem
  • ConfigurationSystem
  • DataManagementSystem
  • FrameworkSystem
  • LHCbSystem
  • LoggingSystem
  • MonitoringSystem
  • ProductionManagementSystem
  • RequestManagementSystem
  • StagerSystem
  • WorkloadManagementSystem

Inside each package, one can find standard directories: Agent, Client, Service and scripts. This overall structure is very similar to that of LHCb Gaudi applications and thus one can adapt easily the existing methods.

CMT projects and packages

In order to be able to use install_project and SetupProject, DIRAC must be packaged as a CMT project, which it now is. This consists in the following additions:

  • File =<DIRACBase>/cmt/project.cmt=that contains the dependencies of DIRAC in terms of projects
  • Package DiracSys: this cmt package's requirements file defines the top dependencies of DIRAC
  • Package DiracConfig : this cmt package's requirements file defines the versions of external packages to be used, if needed (i.e. not using defaults)
  -- PhilippeCharpentier - 18 Nov 2008 \ No newline at end of file

Revision 12008-11-18 - PhilippeCharpentier

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbComputing"

DIRAC3 client distribution and environment

Aim

The aim is to allow the DIRAC3 client distribution to share the tools used by the LHCb applications, as well as benefit from the distribution of LCG clients in the LCG-AA for controlling the version and possibly make tests with newer versions.

The targeted tools are:

  • install_project : applies dependencies of CMT projects for downloading and installing packaged tarballs.
  • SetupProject : uses CMT for setting the run-time environment of the project.

Packages

DIRAC is already split into packages: Core, WMS, DMS etc... with well defined dependencies

-- PhilippeCharpentier - 18 Nov 2008

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback