Building DQ2 0.3

Firstly check out the required dashboard modules in the directory you want

cd dashboard/
cvs co arda.dashboard arda.dashboard.service-config arda.dashboard.cli arda.dashboard.common

Then DQ2

cd ../dq2-HEAD/
cvs co offline/DataManagement/DQ2

These are the connection parameters from outside cern, inside you can use eg

Set the PYTHONPATH to point to the destination stage directory in the DQ2 area, eg offline/DataManagement/DQ2/stage

Now build the dashboard in the correct order, staging to this dir

cd arda.dashboard/
python stage -s ../../dq2-HEAD/offline/DataManagement/DQ2/stage
cd ../arda.dashboard.common/
python stage -s ../../dq2-HEAD/offline/DataManagement/DQ2/stage
cd ../arda.dashboard.service-config/
python stage -s ../../dq2-HEAD/offline/DataManagement/DQ2/stage
cd ../arda.dashboard.cli/
python stage -s ../../dq2-HEAD/offline/DataManagement/DQ2/stage

A patch to dashboard's needs to be applied, this file should be copied from to the DQ2 stage dir

cp /afs/ stage/lib/dashboard/distutils/

The dq2.common package needs to be staged first with the -s option:

cd dq2.common
python stage -s ../stage

Then the other packages can be staged using the -l option to create symlinks, thus meaning no further staging is required after code changes, I use the following python script ( to stage all the packages:

#!/usr/bin/env python

import os
import commands

dirs = os.listdir('.')

for dir in dirs:
    if dir.find('dq2.') == 0 or dir.find('panda.') == 0:
        print 'staging %s...' % dir

        s,o = commands.getstatusoutput('cd %s; python clean; python stage -l; python stage -l' % dir)
        if s != 0:
            print o

Then run


Tests can be run from the stage area, for example

python stage/lib/panda/lrc/client/testcase/

Setting the PATH env var to stage/bin gives access to dq2-* and dashb-* commands.

Ubuntu install of RLS catalog service

This service serves as an interface to the Globus RLS used in NDGF for cataloging file replicas. In order that information on replicas in this catalog can be accessed outside NDGF without any dependencies, this http service was developed to handle query requests and contact the RLS service for information.

A test machine in Aalborg ( with the Ubuntu 6.0.6 OS is used as a test installation of the service. These are the instructions to get the service running.


Apache and mod_python

apt-get install apache2
apt-get install libapache2-mod-python

Grid tools

The Nordugrid standalone client provides all the necessary Globus tools. It was installed following these instructions

I used the 0.6.0 Debian tarball

Now there is a bug in Globus threading where it doesn't work properly on some systems, causing commands to hang instead of terminating (only kill -9 can stop them). This is worked around by setting the following environment variable

export LD_ASSUME_KERNEL=2.2.5


The default python version is 2.4.3

DQ2 installation

The version used was the current HEAD of DQ2 CVS (v0.3 from 16/4/07). To build DQ2 required the dashboard packages to be built. However it proved difficult to find some missing xml modules (eg libxml2) to make the dashboard compile so a tarball was copied over of a pre-compiled DQ2. This was compiled with python 2.3.4 but this did not seem to cause any problems running it with 2.4.3.

The configuration files go into /opt/dq2/etc. I put the following into /opt/dq2/etc/panda.lrc.server/panda.lrc.server.cfg


The rlscli implementation uses the globus command line version of the service. To use the mysql version which queries on the LRC views of the RLS tables, set the impl to rls and add the database connection parameters. I think this conf file is the only one required but I copied the whole tree of conf files from another machine so can't be sure.

Apache configuration

The structure of the Debian packaged Apache is slightly different from that of RH4, so the setup on had to be adjusted slightly. The following should go in /etc/apache2/sites-available/lrc.conf

Listen 8001


 DocumentRoot /var/www/apache2-default

 <Directory /var/www/apache2-default/lrc>

      Order allow,deny
      Allow from all

    # mod_python
    PassEnv             LD_LIBRARY_PATH PYTHONPATH
    PythonPath          "sys.path+['/opt/lrc/lib']"
    SetHandler          mod_python
    PythonHandler       mod_python.publisher
    PythonDebug         On



This runs the service on port 8001. The directory in the PythonPath setting is the stage area of DQ2. A symlink needs to be created to this in the sites-enabled dir

ln -s /etc/apache2/sites-available/lrc.conf /etc/apache2/sites-enabled/lrc.conf

To put the webservice into action create a symbolic link to the ws in the stage area:

mkdir /var/www/apache2-default/lrc
ln -s /opt/lrc/lib/panda/lrc/server/ /var/www/apache2-default/lrc/

Setup and running

A Grid proxy cert needs to be in place for the service to call globus-rls-cli, and this needs to be owned by the user www-data and have permissions 600. This is probably not the nicest way to do it...

grid-proxy-init -valid [long time]
sudo cp /tmp/x509up_u0000 /etc/apache2/
sudo chown www-data /etc/apache2/x509up_u0000

sudo does not take env variables such as LD_LIBRARY_PATH from the user's environment so to start and stop apache we need to use a small script (I called it to set the environment:


if [ ! $1 ]; then
  echo "must specify start|stop|restart"
  exit 1

export LD_ASSUME_KERNEL=2.2.5
export X509_USER_PROXY=/etc/apache2/x509up_u1054.apache
export PYTHONPATH=/user/dcameron/stage/lib:/var/www/apache2-default/dq2

cd nordugrid-arc-standalone-0.6.0

apache2ctl $1

To start/stop/restart apache:

sudo ./ start|stop|restart


The client package is the panda LRC client package which can be used for LRC and RLS. In the configuration file (eg ~/.dq2/etc/panda.lrc.client/panda.lrc.client.cfg) put


The RLS service is a service for querying only so there is no secure part to it.

-- DavidCameron - 18 Apr 2007

This topic: Sandbox > TWikiUsers > DavidCameron > DavidCameronSandbox
Topic revision: r10 - 2007-10-05 - DavidCameron
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback