CCNU LHCb computing farm

Access

Contact your supervisor to get an account. About Linux basics and how to log in, see CcnuLhcbShell

Rules

Quota

• Usually each user is allocated ~ 1GB disk space for home directory.
• Bulk data should be put in your personal directory in /DATA
• There is currently no limit on the CPU, RAM, and data disk usage.

Software

ROOT

• ROOT 6.14.04: source /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/ROOT-env.sh
• ROOT 6.16.00: source /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/ROOT-env.sh
• ROOT 6.18.00: source /cvmfs/sft.cern.ch/lcg/releases/LCG_96/ROOT/6.18.00/x86_64-slc6-gcc8-opt/ROOT-env.sh

There are other versions, or the same version with different versions of compiler, available. But try to stick to those listed here, as it's cached on CVMFS, meaning: 1) each version will take some cache volume; 2) a new version will be very slow when it's used for the first time. Add your version here if it's not in the above list yet, so the cache can be shared with others.

A tutorial of ROOT can be found in CcnuLhcbRoot

LHCb softwares

source /cvmfs/lhcb.cern.ch/lib/LbEnv

How to use Dirac at hepfarm:

source /cvmfs/lhcb.cern.ch/lib/LbEnv

source /cvmfs/lhcb.cern.ch/lib/group_extra_settings.csh

lhcb-proxy-init

lb-run LHCbDirac dirac-bookkeeping-get-files --BK /LHCb/Collision18/Beam6500GeV-VeloClosed-MagDown/RealData/Turbo05/94000000/LEPTONS.MDST --DQFlags=OK --Output=Collision18_MagDown_Turbo05_Leptons.txt

Another way: One can also just setup the LHCb environment by: /cvmfs/lhcb.cern.ch/group_login.sh instead of three .sh files. Then the lhcb-proxy-init command should work provided that the certificates are installed correctly.

Access BKK:

The web service of lhcb bookkeepping is not stable recently. There is another way to access it, via command lines, as:

source /cvmfs/lhcb.cern.ch/lib/LbEnv

source /cvmfs/lhcb.cern.ch/lib/group_extra_settings.csh

Then, try

lhcb-proxy-init

if you get problem in this step, which means the grid certification is not correct, therefore, you should download them from your cern account, as

scp -r YOUR.CERN.ACCOUNT@lxplusNOSPAMPLEASE.cern.ch:~/.globus ~/

The last step :

lhcb_bkk

it takes you ~ 1 mins to open it, depends on your internet speed.

Urania(v8r0)

The Urania package can be used to do the PID resampling Link1 link2. The CERN cetificate is necessary for using DataSets in CERN server, so please upload it and the precodures are same as that you've done for CERN server. Do not set any version of ROOT, then the package can be setup by

source /DATA/Install/setEnv/setEnv_Urania.sh

Geant4

A simple way to setup Geant4 is:

source /cvmfs/sft.cern.ch/lcg/views/LCG_101/x86_64-centos7-gcc11-opt/setup.sh

For testing,

mkdir B1_test
cd B1_test
cmake -DGeant4_DIR=/cvmfs/sft.cern.ch/lcg/releases/LCG_101/Geant4/10.07.p02/x86_64-centos7-gcc11-opt /cvmfs/sft.cern.ch/lcg/releases/LCG_101/Geant4/10.07.p02/x86_64-centos7-gcc11-opt/share/Geant4-10.7.2/examples/basic/B1
make
./exampleB1

One need to switch off the OGL option in vis.mac and turn on OI

In LCG_102, there is problem to load the qt5 xcb plugin.

Alternative 1

using the version from LCG_96. Setup the environment:

source /cvmfs/sft.cern.ch/lcg/releases/LCG_96/CMake/3.14.3/x86_64-slc6-gcc8-opt/CMake-env.sh
source /cvmfs/sft.cern.ch/lcg/releases/LCG_96/Geant4/10.05.p01/x86_64-slc6-gcc8-opt/Geant4-env.sh

export G4DATADIR=/DATA/Install/Geant4_data
export G4ENSDFSTATEDATA=$G4DATADIR/G4ENSDFSTATE2.2
export G4ABLADATA=$G4DATADIR/G4ABLA3.1
export G4INCLDATA=$G4DATADIR/G4INCL1.0
export G4LEDATA=$G4DATADIR/G4EMLOW7.7
export G4LEVELGAMMADATA=$G4DATADIR/PhotonEvaporation5.3
export G4NEUTRONHPDATA=$G4DATADIR/G4NDL4.5
export G4PARTICLEXSDATA=$G4DATADIR/G4PARTICLEXS1.1
export G4PIIDATA=$G4DATADIR/G4PII1.3
export G4RADIOACTIVEDATA=$G4DATADIR/RadioactiveDecay5.3
export G4REALSURFACEDATA=$G4DATADIR/RealSurface2.1.1
export G4SAIDXSDATA=$G4DATADIR/G4SAIDDATA2.0

And compile one example:

mkdir test_example
cd test_example
cmake -DGeant4_DIR=/cvmfs/sft.cern.ch/lcg/releases/LCG_96/Geant4/10.05.p01/x86_64-slc6-gcc8-opt/ /cvmfs/sft.cern.ch/lcg/releases/LCG_96/Geant4/10.05.p01/x86_64-slc6-gcc8-opt/share/Geant4-10.5.1/examples/basic/B1/
make

And run the example:

./exampleB1 exampleB1.in

Alternatively

This works on lxplus and should also work on other machines as long as the needed CVMFS repositories are installed. You will need to change the versions of packages if you use a different version.

source /cvmfs/sft.cern.ch/lcg/contrib/gcc/8.3.0/x86_64-centos7/setup.sh
source /cvmfs/geant4.cern.ch/geant4/10.5.p01/x86_64-centos7-gcc8-opt/CMake-setup.sh
export CXX=`which g++`
export CC=`which gcc`

export GEANT4DIR=/cvmfs/geant4.cern.ch/share/data
mkdir B1_build
cd B1_build/
cmake -DGeant4_DIR=/cvmfs/geant4.cern.ch/geant4/10.5.p01/x86_64-centos7-gcc8-opt/lib64/Geant4-10.5.1 /cvmfs/geant4.cern.ch/geant4/10.5.p01/share/examples/basic/B1
make

./exampleB1

hep_ml

on hepfarm03 source /DATA/Install/setups/setup_hep_ml.sh

Package

The hep_ml package can be found by the Github,hep_ml

Example of GB weight

The error occours when the GB reweight is performed on hepfarm03 :

import _tkinter # If this fails your Python may not be configured for Tk
ImportError: libtk8.5.so: cannot open shared object file: No such file or directory

It can be fixed by adding the following lines :

import matplotlib
matplotlib.use('pdf')

Two examples are attched at the bottom table, sel_mc_Jpsirho.py.txt and gbweight.tar,where the first one is a simple example and the second one needs more knowledge of Python but is much more convenient and useful.

Example of uBoost

Before you use the codes, please note that the signal and background training root files should be merged into one file and tagged by an extra variable, such as the variable named as 'labels' with value 0,1 for each sample.

Gauss simulation

Setup the environment & Install the Gauss package

Example for GaussDev _v49r12

LbLogin -c x86_64-slc6-gcc49-opt 
lb-dev Gauss/v49r12
cd ./GaussDev_v49r12
git lb-clone-pkg  Gen/DecFiles 
make install

About how to add a decay file. You can refer to this site: https://gitlab.cern.ch/lhcb-datapkg/Gen/DecFiles/blob/master/CONTRIBUTING.md

Gauss simulation in the grid

cd GaussDev_v49r12
LbLogin -c x86_64-slc6-gcc49-opt 
./run bash
cp $GAUSSOPTS/Gauss-Job.py .
vi ./Gauss-Job.py
Add:
importOptions('$GAUSSOPTS/Gauss-2016.py')
importOptions('$GAUSSOPTS/GenStandAlone.py')
importOptions('$DECFILESROOT/options/16264062.py')
importOptions('$LBPYTHIA8ROOT/options/Pythia8.py')

Now, you need a ganga script to submit it. Here I have an example: /afs/cern.ch/user/s/shiyang/public/Gauss

A Partial Wave Analysis program using Tensorflow(TFPWA)

This is a package and application for partial wave analysis (PWA) using TensorFlow. By using simple configuration file (and some scripts), PWA can be done fast and automatically.

GitHub: https://github.com/jiangyi15/tf-pwa

Document: https://jiangyi15.gitlab.io/tf-pwa

Environment setting

git clone https://github.com/jiangyi15/tf-pwa

GPU node(gpu6):

creat a new conda env
cd tf-pwa
conda config --add channels jiangyi15
conda install tf-pwa

CPU node(hepfarm,comput)

creat a new conda env
cd tf-pwa
conda config --add channels jiangyi15
conda install tf-pwa
conda install --file tensorflow_2_6_requirements.txt -c conda-forge

Mac M1 ROOT + Python(Gbreweight) + tensorflow(gpu) + TFPWA env set

Please check Command Line Tools for Xcode version <= 13.0, if not, please install Command_Line_Tools_for_Xcode_13.dmg Passwd:8iip

#install conda
download https://twiki.cern.ch/twiki/pub/Sandbox/CCNULHCbFarm/Miniforge3-MacOSX-arm64.sh
chmod +x ~/Downloads/Miniforge3-MacOSX-arm64.sh
sh ~/Downloads/Miniforge3-MacOSX-arm64.sh

#install M1 tensorflow (https://developer.apple.com/metal/tensorflow-plugin/)
creat a new conda env
conda install -c apple tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal

#install ROOT (https://root.cern/install/#conda)
conda install root -c conda-forge

#install TFPWA
git clone https://github.com/jiangyi15/tf-pwa
cd tf-pwa
cp tensorflow_2_6_requirements.txt aaa.txt
Delete the first three lines of aaa.txt
conda install --file aaa.txt -c conda-forge

#install gbreweight dependence
pip install root_numpy
conda install hep_ml
conda install ***package***

Latex

texlive

texlive2020 has been installed for the server, comput . To use it, edit the file ~/.bashrc by adding the following sentences ：

PATH=/home/lhcb/softwares/texlive/2020/bin/x86_64-linux:$PATH; export PATH
MANPATH=/home/lhcb/softwares/texlive/2020/texmf-dist/doc/man:$MANPATH; export MANPATH
INFOPATH=/home/lhcb/softwares/texlive/2020/texmf-dist/doc/info:$INFOPATH; export INFOPATH

texlive2022 has been installed for the server, hepfarm 04/05 . To use it, edit the file ~/.bashrc by adding the following sentences ：

export MANPATH=${MANPATH}:/DATA/Install/texlive/2022/texmf-dist/doc/man
export INFOPATH=${INFOPATH}:/DATA/Install/texlive/2022/texmf-dist/doc/info
export PATH=${PATH}:/DATA/Install/texlive/2022/bin/x86_64-linux

to be added

Known issues

None

Some words for students

Before you ask the questions, please learn to use and understand the codes by yourself, find and fix bugs of codes by yourself, and even rewrite the codes to make them more convinient to use. Then remember to update twiki by yourself. Always keep it in the mind that it's useless and helpless to improve our skills, thoughts and abilities if we only know how to use the codes rather than understand them.

Other useful computing resources

1. IHEP cluster. Contact your supervisor for an account.
2. CERN box proved 1 TB for each registered user
3. CERN GitLab for code versioning. Github and GitLab are good alternatives if you don't have a CERN account.
4. For instant messaging, Mattermost is provided by CERN, and alternatives include gitter. Both of them have markdown support allowing you to have pretty math formula. Skype is also popular in HEP community.
5. The SWAN project provides you a computing environment using Jupyter notebook with CERN box directly accessible.
6. The WSL (Windows subsystem Linux) can enable you to run Ubuntu and a few other distributions as a subsystem in Windows 10. You can turn it on following these instructions -- there are more than one way -- you probably can find more up to date instructions from a search engine. And for graphic server, Xming could be used.
7. CERN openstack service can provide up to 10 vCPU for each user for "private" projects.
8. Some examples of LHCb software including EPM, Davinci, Gauss, PIDCalib and TMVA

for users

How to change your password

You can change your passwd on hepfarm02 by yppasswd

For administrator

How to configure the banner (on SLC6)

Edit the file: /etc/ssh/sshd_config to give the banner location:

# no default banner path
#Banner none
Banner /etc/mybanner

And put the content to /etc/mybanner. And then restart the sshd

service sshd restart

Source: https://webhostinggeeks.com/howto/linux-login-banner/

How to add new users

Follow these steps:

1. Login hepfarm02 as root;
2. Run the command `useradd -g lhcb -c "Full Name" XXXX`, where XXXX is the desired login;
3. Run `passwd XXXX` to have the new user set his/her password. If the passwd is not set by user. A temeray password could be used and one should use `passwd -e XXXX` to force the user to change his/her passwd when logging in for the first time.
4. make -C /var/yp
5. Might need to restart the services:

/etc/init.d/ypserv restart
/etc/init.d/yppasswdd restart

Notes:

• It seems impossible to create the user home directory from hepfarm03 -- to be confirmed;
• It seems the NIS system will automatically update the /etc/passwd file in hepfarm03 fairly fast, thus nothing need to do but wait for a while to be able to login hepfarm03. To be confirmed too.

Other:

• How to change one's default shell?

How to disable user login

By changing the login shell of the user:

sudo chsh -s /sbin/nologin <username>

How to mount comput19 directories

Using sshfs:

sudo sshfs -o allow_other,default_permissions,IdentityFile=~/.ssh/id_rsa dzhang@xxx.xxx.xxx.20:/public/home/dzhang ~/links/ccnuC -p xxxx

umount:

fusermount -zu /home/dzhang/links/ccnuC

Links

CcnuTeaching CCNULHCbFarm2

-- DongliangZhang - 2019-04-16