LHCb Developments tricks

Unfortunately I spent a large amount of my time familiaizing myself with how this project works and how django/sql and web development work. This wasn't completely unfruitfull but the learning curve for a physicist who isn't a web developer can mean that progress can be slow on an actual coding front. Here I will try and summarise what I have learnt to try and speed things along for your work

Getting the code and setting up a dev environment

Get the source code:

git clone /afs/cern.ch/lhcb/software/GIT/LHCbPR.git

Initialize the environment vairables

first you (may) need to edit devel/setuptools such that line 4 goes from: CURRENT_DIR=$( cd "$( dirname "${0}" )" && pwd ) to, CURRENT_DIR=$( cd "$( dirname '${0}' )" && pwd )

then from the top level of the LHCbPR directory just created run:

source ./devel/getdjango source ./devel/setuptools

This will make sure that you have the correct version of django sourced and setup and some relavent sql tools configured for use.

Make sure you have the file ./django_apps/myconf.py

This file will contain a private key and the details for logging into the devdb11 database using the lhcb_pr account. If you are unsure of this contact someone associated with LHCbPR.

now try running (from within LHCbPR again)

python ./django_apps/manage.py

This should end in: Available subcommands: changepassword cleanup collectstatic ... syncdb test testserver unflag validate

If this is the case congratulations you've now got the dev environment setup.

Launching a private instance of LHCbPR for development.

LHCbPR makes extensive use of the devdb11 oracle database at CERN. This means that you CANNOT run remotely without reconfiguing the django to look for a different SQL server.

Option a)

Alternatively you can use lxplus as a dev environment and connect to the django instance remotely.

To do this you first need to start the django server for LHCbPR:

source ./devel/setuptools python django_apps/manage.py runserver

This will by default open up an instance of LHCbPR on port 8000 on this particular lxplus node. You will now need to setup an ssh tunnel between this port 8000 and another port on the machins you're working on. This can be done by adding the following to your ~/.ssh/config:

Host lxplus*.cern.ch
  ForwardAgent yes
  ControlMaster auto
  ControlPersist yes
  ControlPath ~/.ssh/socket-%r@%h:%p
  LocalForward 9000 localhost:8000
  KeepAlive yes
  ServerAliveInterval 30

Then following this close yout ssh connection and open a new one. This ssh configuration is intended to allow you to keep a single connection to a single lxplus node open through the use of network sockets. This also forwards all traffic from your machine at port 9000 to the lxplus node at port 8000. Basically, this means if you direct your browser to localhost:9000 you see what you would get from a browser on the lxplus machine on localhost:8000.

Option b)

Django can be made to look for the SQL server at CERN by changing the HOST option in django_apps/settings.py to be:

'HOST' : 'dbsrvg3305.cern.ch'

This requires that you have the relavent python SQL modules on your system and likely an instance of SQL installed. This is potentially quite involved.

Option c)

You can alternatively forward X11 traffic or use vnc to connect to the lxplus node in question. In that case I recommend you look at: http://lhcb-comp.web.cern.ch/lhcb-comp/Support/DevStudio/VNC.html

The advantage of this is that it's the quickest way to get setup to view the running test server. The disadvantage to this is that you will likely find this quite slow and frustrating.

Some general noted on development within the LHCbPR framework

Debugging your code

Not only is using print statements something that should be avoided it can cause the django server to fall over. In particular if you put print statements in the ROOT handler script. This is likely due to the way that the I/O streams are being redirected but I couldn't find anything obviously wrong with this code.

The best solution is to use python logging.

If you want to change the python logging level for part of the django server, i.e. you want to debug part of what's going on but you don't want to see the full debug output from the whole stack you will have to edit the setttings in django_apps/settings.py.

i.e. to debug the 'views_handler' part of the project you will need to edit the section for the 'views_handler' in this settings.py and change the level of logging which goes on:

'level' : 'DEBUG'

Try to remember to commit back changes of this file into the repo unless you know it's ok.

Debugging the generated webpage

When analysing the generated web page I suggest using Chrome or Firefox with FireBug installed. These tools allow you to interact with the elements that are rendered on the webpage and find/fix bugs much quicker.

Debugging in general

General idea(s) behind Django


The plan for the viewer side of LHCbPR was to first make use of DJango models: https://docs.djangoproject.com/en/dev/topics/db/models/ This significantly reduces the amount of direct SQL needed to be written for this project.

To see the possible models which match the database structure then I'd suggest running:

python ./django_apps/manage.py inspectdb

This will generate a set of templates matching what Django sees as the lhcb_pr user.


Most of the code at the moment in the LHCbPR framework is designed to generate the sql commands internally. This is then executed using the cursor.execute.

This command executes the custom SQL string and then returns the result. This has several advantages including knowing exactly what you're requesting to get back and the format. This places the requirement of optimisation and construction on the user.

Testing SQL queries directly against the database

In order to test any complex queries you may have to write I would advise trying them out on the sql command line. In order to do this you will need to setup SQLPlus to use at CERN.

SetupProject LCGCMT oracle pytools

and to login you need to type:

sqlplus lhcb_pr@devdb11

and enter the password when prompted. This will drop you to an sqlplus shell which will allow you to run requests interactively.

When this is running you should be able to run a command like:

SQL> select distinct apl.appversion as Version , opt.description as Options , plat.cmtconfig as Platform , rf. "FILE" from lhcbpr_job j , lhcbpr_jobresults r , lhcbpr_jobattribute att , lhcbpr_jobdescription jobdes , lhcbpr_application apl , lhcbpr_options opt , lhcbpr_resultfile rf , lhcbpr_platform plat where j.id = r.job_id and j.jobdescription_id = jobdes.id and jobdes.application_id = apl.id and jobdes.options_id = opt.id and r.jobattribute_id = att.id and rf.jobresults_ptr_id = r.id and j.success = 1 and j.platform_id = plat.id AND ( opt.id = 164 ) AND ( apl.id = 145 );


v45r3 30000000-Beam3500GeV-MagDown-2011-nu2-Gen-1000events x86_64-slc5-gcc46-opt 61/61/Gauss-30000000-1000ev-20131113-histos.root


v45r3 30000000-Beam3500GeV-MagDown-2011-nu2-Gen-1000events x86_64-slc5-gcc46-opt 61/61/Gauss-30000000-1000ev-20131113-histos_1.root


Database structure:

This is the part which confused me coming from primarily experience within the LHCb codebase recently and not having too much familiarity with SQL. If you're not too familiar with SQL I suggest getting some practice in before trying to code up some python to generate queries.

The current queries in the database are formed like most standard queries

Development procedure:

See PDF file in the docs/latex/ directory of the GIT repository

Django tools

See https://github.com/tomchristie/django-pdb

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2014-09-17 - RobCurrie
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback