How to run FEWZ at LXPLUS

(如何在LXPLUS上运行FEWZ)

This page is dedicate to give a detailed instruction on how to run FEWZ in lxplus. FEWZ is a generator with NNLO, which is the most feasible higher order calculation for single W, Z boson physics.

(在lxplus上面使用FEWZ来得到关于W与Z玻色子的理论预言值。FEWZ是一个次次领头阶的产生子,对于W、Z物理来说是最为可行的高阶计算。)

(1) install LHAPDF in lxplus :

(在lxplus上安装LHAPDF)

  • LHAPDF: LHAPDF is a interplator between PDF sets and different generators, which has been widely used in the high energy physics.
  • LHAPDF是一个部分子分布函数(PDF sets)跟不同事例产生子的接口软件,主要是用来把各种PDF引入到产生子中,在高能物理中被广泛使用。
wget https://lhapdf.hepforge.org/downloads/?f=LHAPDF-6.4.0.tar.gz

tar xvzf LHAPDF-6.4.0.tar.gz

cd LHAPDF-6.4.0

mkdir ../LHAPDF_640

source /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0/x86_64-centos7/setup.sh

sh /cvmfs/sft.cern.ch/lcg/releases//LCG_98python3/Python/3.7.6/x86_64-centos7-gcc8-opt/Python-env.sh

./configure --prefix=/afs/cern.ch/work/h/hyin/workspace/generators/WZ/LHAPDF_640

make -j4

make install 

Download PDF sets, take CT14NNLO as example, more PDF sets can be download here.(下载PDF文件,使用CT14NNLO作为例子,更多的PDF文件可以从 这里下载,注意,不需要全部下载,使用那个PDF才下载相应文件)


cd ../LHAPDF_640/share/LHAPDF/

wget http://www.hepforge.org/archive/lhapdf/pdfsets/6.2/CT14nnlo.tar.gz

tar xvzf CT14nnlo.tar.gz

wget http://www.hepforge.org/archive/lhapdf/pdfsets/6.2/CT14lo.tar.gz

tar xvzf CT14lo.tar.gz

<br />

(2) install FEWZ in lxplus

(在lxplus上安装FEWZ)

Reminder:please replace "/afs/cern.ch/PATH-TO-LHAPDF-XXX/LHAPDF_621" to the installed LHAPDF (Step-1)

注意:下面的脚本需要替换/afs/cern.ch/PATH-TO-LHAPDF-XXX/LHAPDF_621 成安装好的LHAPDF路径(上一步)


wget http://www.hep.anl.gov/fpetriello/FEWZ_3.1.rc.tar.gz

tar xvzf FEWZ_3.1.rc.tar.gz

cd FEWZ_3.1.rc 
  • modify the makefile and create a setup.sh (修改makefile,并创建setup.sh)
  • create setup.sh (创建setup.sh)
setenv LHAPDFSYS /afs/cern.ch/PATH-TO-LHAPDF-XXX/LHAPDF_621

setenv PATH $LHAPDFSYS/bin:${PATH}

setenv LD_LIBRARY_PATH $LHAPDFSYS/lib:${LD_LIBRARY_PATH}

setenv LHAPDF_DATA_PATH ${LHAPDFSYS}/share/LHAPDF

setenv PYTHONPATH ${LHAPDFSYS}/lib/python2.7/site-packages:${PYTHONPATH}

setenv LHAPATH $LHAPDFSYS/share/LHAPDF

source /cvmfs/sft.cern.ch/lcg/releases/LCG_87/gcc/4.9.3/x86_64-slc6/setup.csh

sh /cvmfs/sft.cern.ch/lcg/releases/LCG_87/Python/2.7.10/x86_64-slc6-gcc49-opt/Python-env.sh 

  • modify makefile in FEWZ_3.1.rc(修改makefile,在FEWZ_3.1.rc目录下面)
LHAPDF = on

LHADIR = /afs/cern.ch/PATH-TO-LHAPDF-XXX/LHAPDF_621/lib 

then comment out the following lines in the makefile, as line 38, 41, 112, 114, 121, 123 - 125.

需要将makefile中的第38,41,112,114,121,123-125行注释掉(删掉)。

  • compile, ~ 20 mins (编译,近20分钟)
source setup.sh

make 
  • Check bin/ directory, if the complie is succuessful, then you should have both fewzw and fewzz: (检查bin这个目录,需要有fewzw, fewzz两个文件)

(3) local test (本地测试)


cd bin

cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/z8_lo_c14.txt .

cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/histz.txt .

cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/phibins.txt .

cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/ptbins.txt .

source ../setup.sh

./local_run.sh z Z_LO z8_lo_c14.txt histz.txt txt .. &

<br />

It takes ~ 5 mins to get the results, you could check Z_LO/Z_LO0/ for the results.

In the directory, screen.out is a output for vegas integration, LO.txt is the calculated cross-section results.

Please check "Sigma (pb) = xxx" in the LO.txt, if it is a non-zero number, then, the FEWZ works well.

ps: the vegas integration is used to calculate the cross-section in differnt phase space, and then combine together.

(这次测试将用大约5分钟的时间,结果在Z_LO/Z_LO0/这个目录里面。

在这个目录里面,screen.out是vegas积分的进程结果,LO.txt是计算好的截面结果。

运行结束后,检查LO.txt,如果"Sigma (pb) = xxx" 是一个非0的数字,那么FEWZ是没有问题了。

另外:vegas积分主要是用来计算不同相空间里面的截面,结果合并起来就是最终结果)

(4) a quick test in the batch system (在batch系统下面的快速测试)。

Batch system can be used as a computer farm to running paraelle jobs, more details can be found here or there.

(Batch系统可以用来处理大量的并行工作,作为cern的一个计算机节点。更多有用的信息可以在 这里,与 这里找到。)


cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/runit_LO.sh .

cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/setup_submit.sh .

cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/local_run_nosubmit.sh . 

modify the runit_LO.sh according to what you want, (修改一下runit_LO.sh,根据你的需求)


WORKDIR=/afs/cern.ch/work/h/hyin/workspace/generators/WZ/FEWZ_3.1.rc/bin

OUTPUTNAME=LO_Z_8TeV

INPUTFILE=z8_lo_c14.txt

HISTFILE=histz.txt 

modify the setup_submit.sh, change the path of LHAPDFSYS. (修改一下setup_submit.sh,修改LHAPDFSYS到你自己的LHAPDF的路径)

export LHAPDFSYS=/afs/cern.ch/work/h/hyin/workspace/generators/WZ/LHAPDF_621_lxplus7


sh runit_LO.sh 

It will create a directory named 'qsub' to save the submitting script, and 'qsub_all.sh' is the command to submit all the batch jobs.

(自动生成一个qsub目录,存放工作提交的脚本。而qsub_all.sh是提交工作的总脚本。)

You could use bjobs to check the job status. (使用'bjobs'来查看提交工作的状态。)

It takes ~ 5 mins to get the output, as shown in LO_Z_8TeV/LO_Z_8TeV0/, and you will recieve an email to say it is done.

(大约要运行5分钟,会把结果拷贝回LO_Z_8TeV/LO_Z_8TeV0/,同时还会收到一封邮件,说明工作已经完成。)

(5) submit jobs for NNLO calculation (提交NNLO计算的jobs)


cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/runit.sh .

cp /afs/cern.ch/user/h/hyin/public/For_FEWZ/z8_nnlo_ct14.txt . 

modify the runit.sh according to what you want, (修改一下runit.sh,根据你的需求)


WORKDIR=/afs/cern.ch/work/h/hyin/workspace/generators/WZ/FEWZ_3.1.rc/bin

OUTPUTNAME=NNLO_Z_8TeV

INPUTFILE=z8_nnlo_c14.txt

HISTFILE=histz.txt 

submit your jobs, for Z boson cross section NNLO calculation, we have 127 sub-jobs.

(提交工作,对于Z玻色子的NNLO计算,总共有127个子工作)


sh runit.sh

sh qsub_all.sh

<br />

After each of sub-jobs is done (> 1 weeks), we can combine results as:(所有子工作结束后,超过一周,合并结果值)


./finish.sh Z_8TeV_NNLO NNLO.txt 

(6) Tips: (注意事项)

  • (a) These jobs are CPU time consuming, which may takes ~ 3 - 7 days to finish each sub-job, so be patient. (每个子工作可能需要3-7天来完成,耐心等待。)
  • (b) Mostlikely, the subjobs will be killed due to "CPU usage limit". In that case, you need submit the failed job by hand, as in 'qsub_all.sh' (大部分子工作会由于CPU 使用限制而杀掉,这种情况下,我们重新提交一下这个工作【比如 子工作-22】,具体命令见 qsub_all.sh)
bsub -c 100:00 -q 1nw -J fewzz-22 /afs/cern.ch/work/h/hyin/workspace/generators/WZ/FEWZ_3.1.rc/bin/qsub/Z_8TeV_NNLO_subjobs_22.sh 

As mentioned above, the vegas integration is used for FEWZ, which will start from the latest working point, even the job is failed. Like, if the vegas integration stopped at 40 iter, then after you resubmit the job, the FEWZ will begin to calculate the cross section from the 40 iter.

(在vegas积分中,FEWZ把每一步状态的相空间保留下来。所以,即使子工作失败了,也会从失败的vegas积分出发,继续计算,而不是从头开始。)

  • (c) When the sub-job failed, it will create file named core.xxx in the directory, with large size (> 1Gb). So you need to check your CERN email and clear them regularly (per 4 hours), otherwise, your directory will be filled with core.XXX, and the other results cannot write into the disk. (子工作失败后,会产生core.XXX文件,这个文件非常大(>1 Gb),很快就会把你的磁盘填满,所以每隔4小时,最好检查一下CERN邮箱,并清理一下各个子目录。)
rm -rf Z_8TeV_NNLO/Z_8TeV_NNLO*/core.* 
  • (d) For Z boson NNLO calculation, the vegas integration need perform 88 iterations. So please check this when one job is done. The last line of successful job should be something like this (in screen.out): (对于Z玻色子的NNLO计算,需要88步vegas积分,在成功的子工作screen.out最后几行)

So I used the command below to check which job is done (可以用下面的命令来检查那些工作结束了)


grep Thanks Z_8TeV_NNLO/Z_8TeV_NNLO*/screen.out 

combined with bjobs command, we can figure out which job is missing. (与bjobs命令结合起来,可以检查缺少那些文件)

-- HangYin - 2018-06-11

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng FEWZ_submit_test1.PNG r1 manage 6.3 K 2018-06-11 - 11:55 HangYin  
PNGpng FEWZ_submit_test2.PNG r1 manage 27.6 K 2018-06-11 - 11:55 HangYin  
PNGpng FEWZ_submit_test3.PNG r1 manage 150.6 K 2018-06-11 - 12:01 HangYin  
PNGpng FEWZ_submit_test4.PNG r1 manage 19.9 K 2018-06-11 - 14:21 HangYin  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2022-11-03 - HangYin
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback