CCNU LHCb computing farm
Access
Contact your supervisor to get an account. About Linux basics and how to log in, see
CcnuLhcbShell
Rules
Quota
- Usually each user is allocated ~ 1GB disk space for home directory.
- Bulk data should be put in your personal directory in /DATA
- There is currently no limit on the CPU, RAM, and data disk usage.
Software
ROOT
- ROOT 6.14.04: source /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/ROOT-env.sh
- ROOT 6.16.00: source /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/ROOT-env.sh
- ROOT 6.18.00: source /cvmfs/sft.cern.ch/lcg/releases/LCG_96/ROOT/6.18.00/x86_64-slc6-gcc8-opt/ROOT-env.sh
There are other versions, or the same version with different versions of compiler, available. But try to stick to those listed here, as it's cached on CVMFS, meaning: 1) each version will take some cache volume; 2) a new version will be very slow when it's used for the first time. Add your version here if it's not in the above list yet, so the cache can be shared with others.
A tutorial of ROOT can be found in
CcnuLhcbRoot
LHCb softwares
source /cvmfs/lhcb.cern.ch/etc/login.sh
How to use Dirac at hepfarm:
source /cvmfs/

lhcb.cern.ch/lib/LbLogin.sh
source /cvmfs/lhcb.cern.ch/lib/group_extra_settings.csh
lhcb-proxy-init
lb-run
LHCbDirac dirac-bookkeeping-get-files --BK /LHCb/Collision18/Beam6500GeV-VeloClosed-MagDown/RealData/Turbo05/94000000/LEPTONS.MDST --DQFlags=OK --Output=Collision18_MagDown_Turbo05_Leptons.txt
Another way: One can also just setup the LHCb environment by:
/cvmfs/lhcb.cern.ch/group_login.sh
instead of three .sh files. Then the lhcb-proxy-init command should work provided that the certificates are installed correctly.
Access BKK:
The web service of lhcb bookkeepping is not stable recently. There is another way to access it, via command lines, as:
source /cvmfs/
lhcb.cern.ch/lib/LbLogin.sh
source /cvmfs/lhcb.cern.ch/lib/group_extra_settings.csh
Then, try
lhcb-proxy-init
if you get problem in this step, which means the grid certification is not correct, therefore, you should download them from your cern account, as
scp -r YOUR.CERN.ACCOUNT@lxplusNOSPAMPLEASE.cern.ch:~/.globus ~/
The last step :
lhcb_bkk
it takes you ~ 1 mins to open it, depends on your internet speed.
Urania(v8r0)
The Urania package can be used to do the PID resampling
Link1 link2. The CERN cetificate is necessary for using
DataSets in CERN server, so please upload it and the precodures are same as that you've done for CERN server. Do not set any version of ROOT, then the package can be setup by
source /DATA/Install/setEnv/setEnv_Urania.sh
Geant4
using the version from LCG_96. Setup the environment:
source /cvmfs/sft.cern.ch/lcg/releases/LCG_96/CMake/3.14.3/x86_64-slc6-gcc8-opt/CMake-env.sh
source /cvmfs/sft.cern.ch/lcg/releases/LCG_96/Geant4/10.05.p01/x86_64-slc6-gcc8-opt/Geant4-env.sh
export G4DATADIR=/DATA/Install/Geant4_data
export G4ENSDFSTATEDATA=$G4DATADIR/G4ENSDFSTATE2.2
export G4ABLADATA=$G4DATADIR/G4ABLA3.1
export G4INCLDATA=$G4DATADIR/G4INCL1.0
export G4LEDATA=$G4DATADIR/G4EMLOW7.7
export G4LEVELGAMMADATA=$G4DATADIR/PhotonEvaporation5.3
export G4NEUTRONHPDATA=$G4DATADIR/G4NDL4.5
export G4PARTICLEXSDATA=$G4DATADIR/G4PARTICLEXS1.1
export G4PIIDATA=$G4DATADIR/G4PII1.3
export G4RADIOACTIVEDATA=$G4DATADIR/RadioactiveDecay5.3
export G4REALSURFACEDATA=$G4DATADIR/RealSurface2.1.1
export G4SAIDXSDATA=$G4DATADIR/G4SAIDDATA2.0
And compile one example:
mkdir test_example
cd test_example
cmake -DGeant4_DIR=/cvmfs/sft.cern.ch/lcg/releases/LCG_96/Geant4/10.05.p01/x86_64-slc6-gcc8-opt/ /cvmfs/sft.cern.ch/lcg/releases/LCG_96/Geant4/10.05.p01/x86_64-slc6-gcc8-opt/share/Geant4-10.5.1/examples/basic/B1/
make
And run the example:
./exampleB1 exampleB1.in
Alternatively
This works on lxplus and should also work on other machines as long as the needed CVMFS repositories are installed. You will need to change the versions of packages if you use a different version.
source /cvmfs/sft.cern.ch/lcg/contrib/gcc/8.3.0/x86_64-centos7/setup.sh
source /cvmfs/geant4.cern.ch/geant4/10.5.p01/x86_64-centos7-gcc8-opt/CMake-setup.sh
export CXX=`which g++`
export CC=`which gcc`
export GEANT4DIR=/cvmfs/geant4.cern.ch/share/data
mkdir B1_build
cd B1_build/
cmake -DGeant4_DIR=/cvmfs/geant4.cern.ch/geant4/10.5.p01/x86_64-centos7-gcc8-opt/lib64/Geant4-10.5.1 /cvmfs/geant4.cern.ch/geant4/10.5.p01/share/examples/basic/B1
make
./exampleB1
hep_ml
on hepfarm03 source /DATA/Install/setups/setup_hep_ml.sh
Package
The hep_ml package can be found by the
Github,hep_ml
Example of GB weight
The error occours when the GB reweight is performed on hepfarm03 :
import _tkinter # If this fails your Python may not be configured for Tk
ImportError: libtk8.5.so: cannot open shared object file: No such file or directory
It can be fixed by adding the following lines :
import matplotlib
matplotlib.use('pdf')
Two examples are attched at the bottom table, sel_mc_Jpsirho.py.txt and gbweight.tar,where the first one is a simple example and the second one needs more knowledge of Python but is much more convenient and useful.
Example of uBoost
Before you use the codes, please note that the signal and background training root files should be merged into one file and tagged by an extra variable, such as the variable named as 'labels' with value 0,1 for each sample.
Gauss simulation
Setup the environment & Install the Gauss package
Example for
GaussDev _v49r12
LbLogin -c x86_64-slc6-gcc49-opt
lb-dev Gauss/v49r12
cd ./GaussDev_v49r12
git lb-clone-pkg Gen/DecFiles
make install
About how to add a decay file. You can refer to this site:
https://gitlab.cern.ch/lhcb-datapkg/Gen/DecFiles/blob/master/CONTRIBUTING.md
Gauss simulation in the grid
cd GaussDev_v49r12
LbLogin -c x86_64-slc6-gcc49-opt
./run bash
cp $GAUSSOPTS/Gauss-Job.py .
vi ./Gauss-Job.py
Add:
importOptions('$GAUSSOPTS/Gauss-2016.py')
importOptions('$GAUSSOPTS/GenStandAlone.py')
importOptions('$DECFILESROOT/options/16264062.py')
importOptions('$LBPYTHIA8ROOT/options/Pythia8.py')
Now, you need a ganga script to submit it. Here I have an example: /afs/cern.ch/user/s/shiyang/public/Gauss
Latex
texlive
texlive2020 has been installed for the server,
comput .
To use it, edit the file
~/.bashrc by adding the following sentences :
PATH=/home/lhcb/softwares/texlive/2020/bin/x86_64-linux:$PATH; export PATH
MANPATH=/home/lhcb/softwares/texlive/2020/texmf-dist/doc/man:$MANPATH; export MANPATH
INFOPATH=/home/lhcb/softwares/texlive/2020/texmf-dist/doc/info:$INFOPATH; export INFOPATH
to be added
Known issues
None
Some words for students
Before you ask the questions, please learn to use and understand the codes by yourself, find and fix bugs of codes by yourself, and even rewrite the codes to make them more convinient to use. Then remember to update twiki by yourself. Always keep it in the mind that it's useless and helpless to improve our skills, thoughts and abilities if we only know how to use the codes rather than understand them.
Other useful computing resources
- IHEP cluster. Contact your supervisor for an account.
- CERN box
proved 1 TB for each registered user
- CERN GitLab
for code versioning. Github
and GitLab
are good alternatives if you don't have a CERN account.
- For instant messaging, Mattermost
is provided by CERN, and alternatives include gitter
. Both of them have markdown support allowing you to have pretty math formula. Skype is also popular in HEP community.
- The SWAN project
provides you a computing environment using Jupyter notebook with CERN box directly accessible.
- The WSL (Windows subsystem Linux) can enable you to run Ubuntu and a few other distributions as a subsystem in Windows 10. You can turn it on following these instructions
-- there are more than one way -- you probably can find more up to date instructions from a search engine. And for graphic server, Xming
could be used.
- CERN openstack service
can provide up to 10 vCPU for each user for "private" projects.
- Some examples of LHCb software
including EPM
, Davinci, Gauss
, PIDCalib and TMVA
for users
How to change your password
You can change your passwd on hepfarm02 by yppasswd
For administrator
How to configure the banner (on SLC6)
Edit the file: /etc/ssh/sshd_config to give the banner location:
# no default banner path
#Banner none
Banner /etc/mybanner
And put the content to /etc/mybanner. And then restart the sshd
service sshd restart
Source:
https://webhostinggeeks.com/howto/linux-login-banner/
How to add new users
Follow these steps:
- Login hepfarm02 as root;
- Run the command `useradd -g lhcb -c "Full Name" XXXX`, where XXXX is the desired login;
- Run `passwd XXXX` to have the new user set his/her password. If the passwd is not set by user. A temeray password could be used and one should use `passwd -e XXXX` to force the user to change his/her passwd when logging in for the first time.
- make -C /var/yp
- Might need to restart the services:
/etc/init.d/ypserv restart
/etc/init.d/yppasswdd restart
Notes:
- It seems impossible to create the user home directory from hepfarm03 -- to be confirmed;
- It seems the NIS system will automatically update the /etc/passwd file in hepfarm03 fairly fast, thus nothing need to do but wait for a while to be able to login hepfarm03. To be confirmed too.
Other:
- How to change one's default shell?
How to mount comput19 directories
Using sshfs:
sudo sshfs -o allow_other,default_permissions,IdentityFile=~/.ssh/id_rsa dzhang@xxx.xxx.xxx.20:/public/home/dzhang ~/links/ccnuC -p xxxx
umount:
fusermount -zu /home/dzhang/links/ccnuC
Links
CcnuTeaching CCNULHCbFarm2
--
DongliangZhang - 2019-04-16