-- RobCurrie - 04-May-2010

Welcome to PPE @ Edinburgh big grin


(I will make this a bit nicer when I learn the twiki system a bit better) (most of this will apply for both LHCb/ATLAS students as well I will mention when I think it doesn't)

I think I'll start by explaining what was needed for me to get up and running with 'stuff'.

This contains as much info that is sporadically around (somewhere) but now centrally located.

First of all, grab a desk in 5303, any that is going to be empty for more than a few months is fair game, (I assume you have done this)

Computing Accounts

You should have received your account information from Jane Patterson, if you didn't get your default username and password printed off.

You now have:

  • 1 ppe linux account
  • 1 windows account general access
  • 1 windows account for accessing the machines in ppe labs (separate windows account, same password as your default one)
  • 1 email account matric @sms.ed.ac.uk
  • 1 account with EASE, Edinburgh's online teaching system which you will probably only (if ever) use for TA (Teaching Assistant) duties

You will probably forget this password by the time you want to use a lab pc if ever, so keep the piece of paper with it on (although you can get this information online from somewhere within the Edinburgh internal websites)

Each of these accounts has the username of your matric number: i.e. s0123456 (that way you will have memorized it by 2nd week of logging in) ALL of them have the SAME default password.

Your normal windows account, EASE and your email have the same password associated with them i.e. this is one large account.

With ppe and labs being separate accounts if you want to change one remember your original password!

Computing Erratum

From the webmail page (www.sms.ed.ac.uk) you can register for a custom email address to be registered with your account i.e. custom@ed.ac.uk, or custom@ph.ed.ac.uk if you want one

You can ssh into the ppe machines remotely with:

ssh s0123456@ph-ppe.ph.ed.ac.uk

You can ssh into CERN remotely on the lxplus machines with:

ssh luser@lxplus.cern.ch

I recommend setting up your personal ssh file similar to below to avoid constantly using -X.

OS Diversity

In the group there are some die hard Mac Users for OSX specific things ask them how to do stuff on that. I gave up and installed linux on mine so I can only offer advice on how to do that :D.

There are quite a few l33t Debian users running Debian Unstable and window managers over desktop environments. They're normally quite friendly over general Linux issues.

You will need to register for the following:

wifi access


you WILL want to request the access to the vpn (which is automatically authorized) with the same password you choose here.

This makes it useful to watch iplayer/4OD, I mean access papers asif you were physically based on the edinburgh network...

This gives you 2 wireless accounts.

  • 1 on central-wpa Edinburgh's own wireless
  • 1 account on eduroam which is part of a larger inter-uni network.

You can login locally on all wireless networks with the username s0123456

When you are in another uni with eduroam this MUST be s0123456@ed.ac.uk


You will probably like access to ECDF at some point (Edinburgh's own supercomputer... although no replacement for the grid for HUGE jobs)


You probably won't need this until 2nd term at the earliest but doesn't hurt to get registered as you could learn to use things on the software better here,

(ECDF has local installs of all the groups tools and LHCb software to run local jobs without the grid, and without access to grid data)

Grid Certificate:

You can apply for your first grid certificate before you join LHCb/ATLAS officially. This is probably technically optional, BUT DO IT, you will discover a new part of JCMB at the least

https://ca.grid-support.ac.uk/cgi-bin/pub/pki?cmd=getStaticPage&name=index (more info @: http://www.ngs.ac.uk)

Once you're registered at CERN you can use your cern account to do this electronically which is more convenient: https://twiki.cern.ch/twiki/bin/view/LHCb/FAQ/Certificate

ALWAYS, request the new certificate BEFORE you remove your old one as full authorization can take up to 3 days (and you don't want to make my mistake of getting locked off the grid with work to be done...)


You now need your certificate to be affiliated with an experiment in order to be able to use the grid. I don't remember if you can do this before you register @ CERN. If you setup an external account for this then register they just remove your external account once you have full access to lxplus.

LHCb: https://lcg-voms.cern.ch:8443/vo/lhcb/vomrs

Registered Certificates

You can view the certificates registered with your account here: https://ca.cern.ch/ca/Certificates/managecadb.aspx


You will need to sort out your application to join LHCb/ATLAS. This NEEDS to be done BEFORE you visit CERN for the first time.

LHCb: http://lhcb.web.cern.ch/lhcb/Registration/lhcbregistration.htm

ATLAS: http://atlassec.web.cern.ch/atlassec/Registration.htm

(When you plan to go out find out what easyjet flight that someone else is getting, ask around in the office, you need to get past security to register, which you can't do without an invitation or an escort)


Make sure you do this incase something should happen to you... (it shouldn't but I'm sure we've all heard stories)

People tend to get irritated once your out here and they discover you don't have it.

Edinburgh University insures you whilst traveling on short academic related trips to CERN and weeks around the world.

To request insurance (approx 10 per trip debited directly from your RTSG):

Electronically: http://www.ed.ac.uk/schools-departments/finance/about/sections/insurance/online-forms/travel-insurance Speak to Jane to get the required account codes,

Email: email Jane Patterson of Misty Cooper and they will assist you with getting this insurance.

CERN Accounts

There are various levels of access to the computing facilities @ CERN. (a non-exhaustive list of access I am familiar with)

  • 1 Dumb external account with an email address - Useless
  • 2 Externally Registered by following application form for the group your with and requires sending copy of dept affiliation/passport - still restricts your access to MANY things
  • 3 Full Member - This will be you after the first visit to CERN and registering with the office

My personal advice is not to bother with an external account unless otherwise instructed that it's required. They just serve to show you what you don't yet have access to and are unlikely needed within your first term in Edinburgh.

Boost your AFS quota

To request extra space on AFS (upto 1Gb is automatically allocated)


CERN Email

You get a CERN email address given to you when you register. The fact that you have this is hidden in a long document somewhere and nearly cost me the chance to eat pizza.

Check it as soon as you are fully registered:


or for account details inc how to integrate it with your favourite client:


Indico Passwords (and occasionally elsewhere)

Most of the LHCb material on Indico and elsewhere is secured behind one of 3 usual passwords.

These are: _No I can't type them up here wink _

These are simple passwords which aren't normally sent out via email and just ask someone/anyone to tell you what they are, they aren't a secret within the collaboration but they help keep internal/unpublished things private.

Local Software

I will talk about LHCb software only here as I haven't used the ATLAS stuff. _Go and bug someone else about that wink _


Probably the most important tutorials you will ever complete!


A good place to start are the tutorials UK which have been running for 6 years.


Bug me rob.currie@edNOSPAMPLEASE.ac.uk if you can't access any of the material and it will be of any interest.

There should be more info on the twiki pages of LHCb Software weeks @ CERN.

Local Install


If you are unfamiliar with Linux sign yourself up to the transkills courses explaining it. www.transkills.ed.ac.uk

I also recommend the unix scripting courses within the Edinburgh computing dept as well.

Because of the wide support outside of CERN I recommend using bash quite strongly, but this is a personal opinion and the default shell @ CERN remains tcsh.

I highly recommend you consider installing and running Destop linux, if you're a die-hard windows at least try Ubuntu which has the ability to install itself atop of Windows as a program. Or try running a guest OS on a VM such as VMware.

Apparently most (all?) of the Linux things will run on the Mac as they're built on the same principles so you can just install some extra packages in Mac. (i.e. UNIX not NT)

The department has Scientific Linux (Red Hat Enterprise) on the machines in 5303 and supports it quite well. (Don't chose this for running at home unless you like pain)

Local LHCb Software

Locally installed software follows the LHCb install design.

To run LHCb software locally at Edinburgh:

Run (or add to the end of the file ~/.bashrc)

source /Disk/lochnagar0/lhcb/lhcb-soft/LbLogin.sh

If you don't yet have an LHCb account you can't access the software repositories as a registered user, however most repo's have readonly public access.

In order to be able to use just the public access you will need to override the getpacks instance that it use your username. create an executable text file in cmtuser or ~/bin called getpack.

In this file put the following line;

getpack --user=anonymous

or run:

mkdir -p ~/cmtuser
echo 'getpack --user=anonymous' >> ~/cmtuser/getpack
chmod +x ~/cmtuser/getpack

My .bashrc ~/.ssh/config

I have added quite a bit of customizations to my ~/.bashrc file which makes my life easier. I hope that what I show you will make yours easier as well

~/.bashrc :

#  This allows you to use your .bashrc with an interactive login and still use scp correctly!
# If not running interactively, don't do anything
[ -z "$PS1" ] && return

# Source global definitions
if [ -f /etc/bashrc ]; then
   . /etc/bashrc

#Setup 32-bit to push be able to push local binaries to the grid if needed   this is slowly getting phased out to be able to run both on the gid
export CMTCONFIG=slc4_ia32_gcc34

#Setup root                                                            ----required only at Edinburgh (I tested this version of root and it builds and runs RapiFit correctly!)

cd /Disk/lochnagar0/general/root/5.24.00_slc5_gcc34_bin
source bin/thisroot.sh

#Setup LHCb                                                          ----required only at Edinburgh
source /Disk/lochnagar0/lhcb/lhcb-soft/LbLogin.sh

#Fix backspace and various 'issues' with the lxplus bash setup    ---recommended whenever your ssh-ing into an account IMO
export TERM=linux

#Add $PWD and userspace executables with priority
export PATH=.:$HOME/bin:$PATH

#Highly related to personal taste but I **really** like a colorful terminal with a black/semi-transparent background
#This PS1 also has $PWD on the line above where you're working which is good for navigating through complex directory structures with a narrow screen should you end up on one.
#Also means I always know where to focus my eyes to see when a prog has finished output, but as I say personal taste
#export PS1="\[\e[1;31m\]\t\[\e[1;32m\]-CERN:\[\e[0;35m\]\w\n\[\e[1;33m\]$\l\u\[\e[1;32m\]@\[\e[1;34m\]\h\[\e[1;32m\]:->"

#some useful aliases I use to save me repeating an often used command and I can use tab to complete these as well to reduce typing out the full alias
alias ssh_ed="ssh ph-ppe.ph.ed.ac.uk"
alias ssh_cern="ssh lxplus.cern.ch"
alias ssh_eddie="ssh eddie.ecdf.ed.ac.uk"

export PATH=$PATH:./

to load your .bashrc profile on login when connecting remotely to ppe machines you need to create ~/.profile with the line source ~/.bashrc

~/.ssh/config :

this shortens the login command to:

ssh lxlpus.cern.ch

Host lxplus.cern.ch
User _username_
ForwardX11 yes
ForwardAgent yes
ForwardX11Trusted yes

Host ph-ppe.ph.ed.ac.uk
User s0123456
ForwardX11 yes
ForwardAgent yes
ForwardX11Trusted yes


FYI, if your wanting to transfer files (or use a graphical interface) over ssh over a fast connection turn the compression off to potentially gain a small performance boost.

I recommend using sshfs (linux) which allows you to mount userspace associated with an ssh account on your local system to view it transparently

(I don't use mac but try http://code.google.com/p/macfuse/ or http://pqrs.org/macosx/sshfs/ if you want sshfs)

alias disk_ed="sshfs -C ph-ppe.ph.ed.ac.uk: /some/path/ED-USER"
alias disk_cern="sshfs -C lxplus.cern.ch: /some/path/CERN-AFS"
alias disk_eddie="sshfs -C eddie.ecdf.ed.ac.uk: /some/path/EDDIE"
alias disks="disk_cern && disk_ed && disk_eddie"

Force close ssh as a client

If you have intermittent wifi or a bad connection you will end up with hung connections to CERN, to close these connections from the client (your) end type:

3 keys all at once: seperated by ---->

Return key ----> ~ ----> .

This will close the ssh connection and allow you to re-connect without having to close the shell and open a new one.

What they don't teach you in tutorials...

I found that these tutorials all seem to explain the basics and introduce you to LHCb from within particle physics, but very little explains well the 'complete' working environment for day to day use IMO.

Experience teaches you more than tutorials as I'm sure experience teaches you.

Sourcing the LHCb environment you setup a large amount of services/environmental variables.

This uses the CMT to manage local/global versions of programs/modules/libraries.

Large Programs in LHCb require further things to be setup in order to be run.

This requires the SetupProject Project v_x_r_yy_ command to set these things up.

In order to Run a program locally you need to install the required binaries in a local user area as well as have the environment setup.

Grab Source

To Install a program from CMT you need to get the sourcecode for the program


SetupProject Gauss --build-env
getpack Sim/Gauss


Now you have the source you will want to build the local binary:

cd Sim/Gauss/cmt
cmt gmake

Setup Environment

Finally you need to setup the Environmental variables so that you have the correct libraries/binaries pointed to.

SetupProject Gauss

When you issue SetupProject a lot of behind the scenes things are setup and when you are using multiple tools which do the same job they conflict a lot.


SetupProject LHCbDirac conflicts with SetupProject Ganga SetupProject Boole conflicts with SetupProject Panoptes (This doesn't always happen for me but sometimes has caused problems)

Local Execution

To run this Gauss locally you run something to the equivalent of:

gaudirun.py someGaussinstructions.py

In future you only need to configure the environment to run the program, ie:

SetupProject Gauss

To learn how to use a program I find it quickest to look in the src directory of the getpack from the project in order to see some simple example scripts.

If you are unsure what getpack you want just issue a getpack request with what you think you need and you will be provided with a list of options of packages it knows about if you select something bogus.

I recommend grabbing the Tutorial getpacks as they have solutions to the tutorial problems which allow you to see what's going on a lot quicker, although don't just glance at the code and assume you know what's going on at least muck around with it a bit and see what everything does.



LXPLUS uses the afs filesystem which is not the typical Unix fs type in the traditional sense of you can share a file based soley on chmod-ing the file and it's dirtectory.

It requires some more expert commands which are documented but hard to find and as such:

DO NOT DELETE ~/public ~/www etc.

You can fix this if you accidentally delete and recreate them you discover chmod doesn't work on AFS

You need to set ACLs according to what you want such as: https://espace.cern.ch/it-faqs/Lists/faqs/DispForm.aspx?ID=280


What can be done on the grid can be done many times, can be repeated, can be recreated and costs you nothing.

Where possible segment jobs into many, many subjobs, use the infrastructure to parallel compute, it's built for it.

_do it once, do it now, do it often_


ls, more, cd and every other tool useful for file management doesn't work on CASTOR, it's great for storing large files you don't necessarily want on the gid (but try to have a good reason why?)

important things to note:

Copying Files onto Castor:

rfcp some_local_file.ext    $CASTORY_HOME/Some_file_on_Castor.ext

Moving Files on Castor

Can't be done easily, I tried to hack together a rfmv script tying together the other programs for castor but this proved a pain so I abandoned it after 1 hour

# Rename Files
rfrename $CASTOR_HOME/some_file.ext $CASTOR_HOME/some_new_file.ext
# Copy Files - as above
# Deleting Files
rfrm $CASTOR_HOME/some_file.ext
# Making a new directory on CASTOR
rfmkdir $CASTOR_HOME/new_dir

Listing Contents of a Directory on Castor

# list your home directory on castor
# list your some subdir of your home on castor i.e. come ganga output
nsls 416/outputdata
# list some absolute path
nsls /castor/cern.ch/path/to/data

Editing files on Castor

Can't be done directly, copy them off and alter then move them back


To get the list of LFNs associated with your user account:


Uploading to the Grid

To Upload to the grid I find it easiest to use Ganga.

in ganga:

pf = PhysicalFile('$HOME/PATH/TO/FILE.dst')
in bash script form:
export PATH_TO_FILE=some_file.dst
mkdir -p /tmp/$USER
export push_script=/tmp/$USER/script.py
echo "pf = PhysicalFile('$PATH_TO_FILE')" >> $push_script
echo "pf.upload('$LFN','CERN-USER')" >> $push_script
and execute with:
SetupProject Ganga
ganga $push_script

I find it easier if I have multiple files to push them with a complex bash script creating simple python files to execute by Ganga as a tool.

If your uploading large files try using the command

dirac-dms-replicate-lfn /some/lfn
This places the file in more than one PFN on the grid and allows you to distribute the load/availability from more than one site for instance.

Downloading from the Grid

The output of grid jobs is best organized by ganga and python scripts to avoid too many confusions imo.

To download single files from the grid: This can be done through ganga but to just grab a file to work with; (remember large files >250Mb need to go to a scratch space on lxplus)

SetupProject LHCbDirac
dirac-dms-get-file /lhcb/user/l/luser/some/file

Deleting files from the Grid

I will warn users that removing a job from ganga doesn't remove the created LFNs. (I think this can be done with a separate command in ganga?) This is sensible behavior when you consider you may only need the LFNs as input in future jobs, however you don't necessarily want to remove them when you sort jobs in ganga.

But should you ever have a large amount of data on grid jobs (unlikely but nice to be friendly) remove the files with the command:

dirac-dms-remove-files /lhcb/l/luser/some/lfn

-- RobCurrie - 25-Oct-2010 "Here Endeth the Lesson."

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2014-11-28 - IwanThomasSmith
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback