Caroline's online analysis logbook

To do:

• DC5 efficiency for many 2015 runs (at least one for each subperiod) from BW
• Complete gitlab setup (escalade). Wait until coral is integrated.
• How does actually phastBatch.pl and phastjob.csh work? (lxplus)
• Install root on work laptop. https://root.cern.ch/phpBB3/viewtopic.php?t=6752
• Check for things to be fixed on this page.

Blue Waters logs

10x32 (5457258.bw) in more detail. Executed 33 commands on 31 minions for each of the following:

Duration [s]	Application	utime [s]	stime [s]	Rss	inblocks	outblocks
27622.453973	48521523	872457	214	403620	4083882	270973
37468.043909	48521520	1184396	213	408992	4002444	315629
37856.662393	48521462	1193732	229	410780	4005015	322550
39320.313733	48521458	1237791	230	407776	4057131	325569
42977.732554	48521526	1366276	314	411276	13579758	1346944
47150.122060	48521450	1495926	1274	412136	76887662	7878304
47649.446918	48521384	1514914	1063	413432	76888090	7942070
57461.700123	48521528	1823220	1530	411532	77186988	8055250
61907.220122	48521453	1963781	1565	413564	76889373	8157261
64911.669264	48521457	2054588	1526	412500	76889043	8040033

• aprun -n 32  ./pcp ${Period}_$i_${Nodes}times${Length}.cmdlist & ++ wait ++ sleep 100 with MyVariableDC5Eff_trafdic.2015.opt, ppn=32:xe
• qsub example: qsub MyVariable_10times32.pbs -N 10times32 -q high -l walltime=20:00:00 -l nodes=10:ppn=32:xe

(some of the jobs below appear also in the new table above)

October 2016

October 12, 2016: run coral on BW w/o MySQL database

• Need to do for all detector views and runs covered (!):

cd $CORAL/src/condb/mysqldb/Utils/ 
./getDBFilePath -r 260693 DC01X1

• Need: run list; list of detector views
• Then the idea would be to execute ./getDBFilePath in a loop for all runs and detector views and store the calibration files in a dedicated directory (can be group or user space for the time being)
• Then at the beginning of coral execution, link all those files to the execution directory and ...

September 2016

September 12, 2016: set up parallel command processor on BW ("pcp code")

• Use package bwapps-1816 compiled by Marco (copy them in BigData directory), and also looking at these files:
  • https://gitlab.cern.ch/illinois-and-friends-group/escalade-framework/blob/master/bin/bluewaters/phastjob-wrapper.pbs
  • https://gitlab.cern.ch/illinois-and-friends-group/escalade-framework/blob/master/bin/phastjob.sh
• cd MyRun/mDSTs/aprun/BigData
• cc parallel-command-processor.c -o pcp
• Write various scripts XXX add more info

• How to avoid ANSI escape codes when creating a chunk list: ls -f /scratch/sciteam/criedl/DATA/dy15W12-raw/ |grep cdr*| head -16 > Test_16.chunklist (they appear for me on BW because I have in my .bashrc : alias ls="ls --color=always")

September 9/10, 2016: run phast on BW

• cd /u/sciteam/criedl/COMPASS/coral_svn/trunk/src/condb/mysqldb/Utils/
• make
• MyRun/Phast/2015-W08
• ./phast -h MyOutput-2015-P02_DC05Y.11004-260061_U2117.root -u11 -U2117 -T DC05Y DC5/2016-09-06/phast_mem4gb.root :
  • "** mysqlDBEnv: Host computer name "h2ologin2" unknown, not matching either of: "lxplus", "pcco", "compass", "ccage""
    • Have to add BLUEWATERS to UserEvent.cc:

bool mysqlDBEnv(const char *&server, const char *&env, int &port)
{
  const char *hosts[] =   {"lxplus",    "pcco",    "compass",          "ccage",       "h2ologin1"};
  const char *servers[] = {"wwwcompass","pccodb00","compass.gridka.de","cccompassdb", "h2ologin1"};
  const char *envs[] =    {0,           "DAQ",     "GRIDKA",           "LYON",        "BLUEWATERS"};
  const int ports[] =     {0,           0,         0,                  23306,         0};
...

• • • Recompile phast libraries and phast&coral with usual procedure (see BW Twiki). OK then.
  • Interactive mode: Real time = 6.14613 [ms] per event, CPU time (utime+stime) = 5.83226 [ms] per event (33517 events)
• Batch job: qsub MyPhast_xk_packed_260061_11002.pbs  -N Phast_DC5Y_26006_11002 -l walltime=00:00:10
  • Host computer name "nid12351" unknown. I can't add all the nid computers to UserEvent11. Therefore I have two options:
    • run interactively only
    • or copy / link calibration files in directory with proper naming convention. success! 5439764.bw
• Process all three chunks (from Sept. 9) & extract plots:
  ./phast -h 2016-09-10/MyOutput-2015-P02_DC05Y.11002_11004-260061_U2117.root -u11 -U2117 -T DC05Y DC5/2016-09-09/phast_cdr1100*-260061_mem4GB_4cpus.root

September 6-9, 2016: testing coral on BW

• coral is slower on BW (0.81 sec/event) compared to lxplus (0.57 sec/event)!
• Try 4GB (same result): (what is the default memory assignment?)
  • qsub My_xk_packed.pbs -l walltime=24:00:00,mem=4gb [5425996.bw] 10.5h
• 1 node, 4 CPUs, mem=4GB, xk, walltime=24h . The 4 CPUs seem to make a difference!
  • qsub My_xk_packed_260061_11002.pbs -l walltime=24:00:00,mem=4gb [5437395.bw] 5h 27min
  • qsub My_xk_packed_260061_11003.pbs -l walltime=24:00:00,mem=4gb [5437396.bw] 4h 5min
  • qsub My_xk_packed_260061_11004.pbs -l walltime=24:00:00,mem=4gb [5437397.bw] 4h 7min

September 1/2, 2016: testing coral on BW

• MySQL data base is now accessible in batch mode, too (Marco fixed it)
• Process 1 chunk W08/cdr11002-260061.raw for DC05Y efficiency
  • qsub My_xk_packed.pbs -l walltime=10:00:00 : job exceeded wallclock limit.
  • qsub My_xk_packed.pbs -l walltime=24:00:00 [5367117.bw] 10.5h

August 2016

August 24-31, 2016: DC5 efficiencies on BW

• criedl@h2ologin1:~/COMPASS/MyRun/mDSTs> ./coral.exe TrafDic/MyDC5Eff_trafdic.2015.opt
• W08: run 260061, chunks 11002, 11003, 11004
• Interactively 50 events (chunk 11002): OK, coral output in Output/DC5/current/2016-08-25
• Simple batch:
  • Study the different batch example scripts. Differences are not yet clear to me. I start with the method:
  • Start job from my user directory MyRun, have it cd into $JOBID directory on scratch, where the $JOBID.out is stored.

           mkdir -p /scratch/sciteam/$USER/$PBS_JOBID
           cd /scratch/sciteam/$USER/$PBS_JOBID
           

  • In option file, have as many general pathes as possible. No local link, otherwise you have to create it in the scratch directory.
  • Have job put output into user directory (for the time being). Will probably have to change to scratch for output at some point.

• • In directory aprun: qsub My_xk_packed.pbs
  • 5335469.bw                 testjob          criedl                 0 Q normal

• • ppn=1 for only 1 chunk is enough. If I use the default of ppn=1024, the job is queued for a very long time.
  • What is PE (option -N)?
  • qsub -V -q M script (DESY grammar): -V is deprecated and queue M is not known... default is queue normal
  • qdel
  • Next submission, try to add in the batch script PBS_JOBNAME=DC5Y_26006_11002 (but isn't the jobname defined automatically...?). Or why is my job called "testjob"?
  • For loops over more than 1 chunk and other nice features, see eg. my RDdaql script...
  • Of course we will need a ManyRDs.csh script adapted to BW at the end of the day.

August 23/24/25, 2016: Continue to copy raw dy15W12 to BW

• Contacted CERN IT about many failures and they had some tips.

August 16/17, 2016: DC5 efficiency on lxplus

• phast execution works today, randomly...
• Run coral over cdr11007-261520.raw - the run I had all the time (261645) was somehow bad and had no beam tracks. Interactive mode is promising (1000 events). Then the histograms he1DC05 (1D eff), he2DC05 (2D eff) and hRTDC05 (RT relation) are filled.
  • Interactively: in MyRun/mDSTs ./coral.exe TrafDic/MyDC5Eff_trafdic.2015.opt
  • Send batch with entire chunk: ./ManyRDs.csh -d Output/DC5/ TrafDic/MyDC5Eff_trafdic.2015.opt 1
• Cross-check with Robert on P02 (W08): run 260061, chunks 11002, 11003, 11004
  • In MyRun/mDST: ./coral.exe TrafDic/MyDC5Eff_trafdic.2015-P02_DC05Y.opt
  • In MyRun/Phast-DC5/2015-P02-12: ./phast -h MyOutput_2117.root -u11 -U2117 -T DC05Y phast.root
    • Compare all four options from the phast option table below: Only difference is pVertex required or not. If yes, number of entries drops by a factor of 3 and efficiency is slightly higher (Y1: 77.5% instead of 75.3%)
  • DC05Y batch: ./ManyRDs.csh -d Output/DC5/P02/DC05Y -s 11002 TrafDic/MyDC5Eff_trafdic.2015-P02_DC05Y.opt 3
  • DC05V batch: ./ManyRDs.csh -d Output/DC5/P02/DC05V -s 11002 TrafDic/MyDC5Eff_trafdic.2015-P02_DC05V.opt 3
  • DC05U batch: ./ManyRDs.csh -d Output/DC5/P02/DC05U -s 11002 TrafDic/MyDC5Eff_trafdic.2015-P02_DC05U.opt 3
  • All coral jobs finished by the next morning. Details of running time see BW written logbook.
  • 1 coral output file (root) for 1 chunk, size ~100MB. Have 3 chunks for each operational detector view. Merge with hadd merge.root ntuple.0.root ntuple.1.root ntuple.2.root
    • Many Error in <TBufferFile::CheckByteCount>: object of class vector [...] read too few bytes: 6 instead of 8 Is this worrysome?
    • Merged file is huge, ~2GB GB, why? Compression not working?
  • ./phast -h MyOutput-2015-P02_DC05Y.11004-260061_U2117.root -u11 -U2117 -T DC05Y DC05Y/MyDC5Eff_trafdic.2015-P02_DC05Y.11004-260061.phast.root etc.
    • Should I better merge the phast output? file size = 1.6MB
    • hadd MyOutput-2015-P02_DC05Y.11002-11004-260061_U2117.root MyOutput-2015-P02_DC05Y.1100{2..4}-260061_U2117.root gives merged file with size of the sum of the three. But this merged file has some weird features (Y-axis screwed up etc.)
  • Best approach: run over all coral output files at the same time: ./phast -h MyOutput-2015-P02_DC05Y.11002_11004-260061_U2117.root -u11 -U2117 -T DC05Y DC05Y/MyDC5Eff_trafdic.2015-P02_DC05Y.*-260061.phast.root

• • It must be possible (and it will be necessary) to generalize this with options and scripts...:

sdiff -s MyDC5Eff_trafdic.2015-P02_DC05V.opt MyDC5Eff_trafdic.2015-P02_DC05U.opt
// DY-2015 data: DC05-V efficiency | // DY-2015 data: DC05-U efficiency
mDST hits DC05V // Output DC05V1/2 hits | mDST hits DC05U // Output DC05U1/2 hits
//TraF SmoothPos [0] 502.935 // Smooth @ (DC05U1+DC05U2) | TraF SmoothPos [0] 502.935 // Smooth @ (DC05U1+DC05U2)/2
TraF SmoothPos [0] 505.535 // Smooth @ (DC05V1+DC05V2)/2 | //TraF SmoothPos [0] 505.535 // Smooth @ (DC05V1+DC05V2)

• Useful docus about DC studies:
  • https://twiki.cern.ch/twiki/bin/view/Compass/DetectorStudies/DCDetectorStudies
  • https://twiki.cern.ch/twiki/bin/view/Compass/DetectorStudies/UserEvent11
  • https://twiki.cern.ch/twiki/bin/view/Compass/DetectorStudies/UserEvent11Analysis
  • https://twiki.cern.ch/twiki/bin/view/Compass/DetectorStudies/UserEvent11#Online_documentation
  • https://twiki.cern.ch/twiki/bin/view/Compass/DetectorStudies/UBExampleRT
• phast options: phast -Ucode , where code is a decimal number, which is in hex composed of the following bits:

bit	hex	meaning	U34881 = 0x8841	U34885 = 0x8845	U2113 = 0x0841	U2117 = 0x0845
1	0x0001	disable all other options	x	x	x	x
2	0x0040	neighboring hits	x	x	x	x
3	0x0800	correct hit time	x	x	x	x
4	0x8000	use detector.dat to correct for misalignment	x	x
5	0x0004	require pVertex in target		x		x

• The calibration files do NOT have to be linked in the execution directory if the default ones are used! If none are given, phast will go into the MySQL data base.
• A detector.dat in the execution directory is only needed if one wants to correct for misalignment.

August 10/11, 2016: continue with coral on BW and lxplus

• BW: Two improvements with environment: 1. can use again official setup.sh as it was corrected. 2. Put all COMPASS analysis paths into .bashrc (as it should be) so that I do not need to run a separate script.
• BW: Check out same coral version as on lxplus (revision 14211). Do exactly the same with the ChipSinica files as on lxplus: activating the 2015 version.
• BW: Now follow steps as described here: http://twiki.npl.illinois.edu/bin/view/COMPASS/Code#Caroline to compile coral.
• BW in MyRun, create Input directory. Move input for DC5 into Input directory, e.g. Input/DC5-2015-P04-1
• BW: Copy the Trafdic file I worked on in August 2 to the TrafDic directory: MyDC5Eff_trafdic.2015.opt. This will be the new master file. The special tracking options from trafdic.DC05.DCmode.opt still need to be implemented. (? or continue to have 2 files?)
• lxplus: pick up thread from June 28/29. Goal is to run coral & phast over one P4-1 (W10) chunk, W10/cdr14112-261645.raw , exclusing but writing the hits of DC05-Y.
  • Write improved option file MyDC5Eff_trafdic.2015.opt which calls trafdic.2015.opt . Use correct and latest detectors.dat and TraF SmoothPos.
  • . PreExeCoral.sh
  • Interactively: in MyRun/mDSTs ./coral.exe TrafDic/MyDC5Eff_trafdic.2015.opt "50 events" (those were requested in the option file)
  • Send batch: ./ManyRDs.csh -d Output/DC5/ TrafDic/MyDC5Eff_trafdic.2015.opt 1 "number of events: 68575, coraljob: coral successful"
  • Output was requested in MyRun/mDSTs/Output/DC5. mkdir 2016-08-10 and link current to it. In MyRun/Phast-DC5/2015-P04-1, current then points to the latest input for phast.
  • In MyRun/Phast-DC5/2015-P04-1, link DetectorsDat/2015/current/detectors.261513.transv.dat to detectors.dat .
  • In MyRun/Phast-DC5/2015-P04-1, ./phast -h MyOutput.root -u11 -U34881 -T DC05Y current/MyDC5Eff_trafdic.2015.14112-261645.phast.root
    • It seems [hast does not find the proper RT calibration files in this directory. I retrieve the proper files again for run 261645 (w/ getDBFilePath) and they are correct. ? ? Marco says this problem is know. Does not work always. Check source code of UserEvent11.cc. Chia-Yu says it is the naming of the calibration files. See Twiki or Mathieu's docu. (The name I give is indeed wrong)
  • Get calibrations: ./getDBFilePath -r 261645 DC05Y1.
    • To generate getDBFilePath (set up compilation environment): cd coral_svn/trunk/src/condb/mysqldb/Utils ; make

• DC05 in 2015 detectors.dat. For DC05 efficiency, need TraF SmoothPos, which is the average z-position of a given plane (i.e. average of un-primed and primed view). Files listed here were produced on June 24, 2016.

File	Z-Y1	Z-Y2	<Z-Y>	Z-V1	Z-V2	<Z-V>	Z-U1	Z-U2	<Z-U>
detectors.261513.transv.dat	511.1350	510.3350	510.735	505.9350	505.1350	505.535	503.3350	502.5350	502.935

August 8/10, 2016: update coral on lxplus and BW.

• Follow instructions as described here:
  • Check out new version of coral: revision 14211 (= August 10, was 14208 on August 8)
  • Link 2015 and 2016 DC05 decoding files (ChipSinica.cc and ChipSinica.h) into my coral_svn/trunk/src/DaqDataDecoding/src . For the time being, have the 2015 version activated:

mv ChipSinica.cc ChipSinica.cc_v14211
mv ChipSinica.h ChipSinica.h_v14211
ChipSinica.cc_2015 -> ../../../../../DC5/Decoding/ChipSinica.cc_2015
ChipSinica.cc_2016 -> ../../../../../DC5/Decoding/ChipSinica.cc_2016
ChipSinica.h_2016 -> ../../../../../DC5/Decoding/ChipSinica.h_2016
ChipSinica.h_2015 -> ../../../../../DC5/Decoding/ChipSinica.h_2015
ChipSinica.cc -> ChipSinica.cc_2015
ChipSinica.h -> ChipSinica.h_2015

• • Compile coral (DC05-2015) - August 10.

August 4-8, 2016: Use FTS3 to copy data from CERN to BW.

• After meeting with Xavi and Alejandro. See BW Twiki for details.

August 2, 2016: Copy more 2014 mDSTs (dy14T07t4) to BW

• Simple script that fetches, one volume at a time, data from CASTOR to my home afs, copies them (via GO CLI) to BW, using my personal GO endpoint, and deletes them in my home afs.
• Run 2 scripts in parallel, each is supposed to copy 162 volumes. Yesterday I copied 45 volumes. The total for dy14T07t4 is 1,182 volumes.

August 1, 2016: copy some 2014 mDSTs (dy14T07t4) to BW

• Unfortunately still have to use my afs home directory with <10 GB total volume.
• Activate BW GO endpoint (it is open only for ~11 days) on BW: ssh cli.globusonline.org , endpoint-activate ncsa#BlueWaters . Follow link.
• on lxplus: cd $HOME/COMPASS/GlobusOnline/globusconnectpersonal-2.3.1/ ; ./globusconnectpersonal -start &
• 2 at a time: xrdcp from Castor to my afs home directory; then use GO webinterface to copy files to BW. This is incredibly slow and ineffective. (details see table on BW data transfers, entry 3)

August 1/2, 2016: run coral on BW

• Set up environment (in $HOME): . MySetup.sh and . myup.sh. (setup.sh loads improper version of root, therefore I have my own setup script)
• Create directory of today's date in $HOME/MyRun/mDSTs/Output/DC5 and link current to it
• cd $HOME/MyRun/mDSTs/2015-P04-1
• ./coral.exe trafdic.DC05.DCmode.opt
• Reproduce Error 2 of July 5:

Error in <TMacro::ReadFile>: Cannot open file:
/projects/sciteam/badp/compass//geometry/2015/ROOTGeometry/detectors.C

Required file is in different path:

/projects/sciteam/badp/detector/geometry/2015/ROOTGeometry/detectors.C

Problem is environmental variable $COMPASS_FILES needed by trafdic.2015.opt (this default trafdic is also used as input even though I call coral with a specfic DC5 option file). On lxplus, export COMPASS_FILES=/afs/cern.ch/compass/detector in the setup.sh of coral_svn/trunk. On BW, it is defined in MySetup.sh (daughter of setup.sh) and is wrongly set to export COMPASS_FILES=$BADP/compass. Change it to $BADP/detector there (hoping it does not screw up at other places now).

• Error 3: Error in MySQLDB::ConnectDB(): can't connect to DB

Solution: edit $CORAL/src/user/trafdic.2015.opt. Outcomment one line and add the two following:

//CDB server    wwwcompass      
CDB server h2ologin1
CDB specialplace BLUEWATERS

• This needs to be done after each fresh checkout of coral (because the default trafdic.2015.opt will be checked out). Best is to write your own dedicated trafdic file.
• Marco has to stop the MySQL data base, add (allow) user anonymous and re-start.
• BLUEWATER is the key with that BW is registered on lxplus. Marco needs to add more lines that point to the directories with the actual calibration files on BW (M. Bodlak added only one line, which makes calibrations for many detectors unavailable)

July 2016

July 5, 2016: set up BW to run code and try to run coral

• On BW, in MyRun/Phast-DC5/2015-P04-1 directory (as "good example"), create links:
  • detectors.dat : for a P4 run, it is ../../../project/detector/geometry/2015/detectors.261513.transv.dat
  • RT calibration files: for a P4 run, the files are in ../../../project/detector/calibrations/MySQLDB_files13/ with date Sep23.
  • to mDSTs/Output/DC5/current
• On BW, in mDST directory: link to input coral file and link official trafdic file in subdirectory
• Set my personal environment directories: create file myup.sh, analog to PreExeCoral.sh on lxplus.
• I.e. before a new analysis session, set up your environment:

cd $HOME
. setup.sh
. myup.sh

myup.sh defines the paths to the personal directories MYDIR, CORAL, PHAST, mDST

• cd $HOME/MyRun/mDSTs/2015-P04-1
• Error 1 when running coral: ./coral.exe trafdic.DC05.DCmode.opt

./coral.exe: error while loading shared libraries: libGui.so.5.34:
cannot open shared object file: No such file or directory

Coral expects root 5.34 (as defined in its configure run before compilation), while the active root on BW is 6.06.

which root
/projects/sciteam/badp/opt/x86_64-suse-linux-gcc49/root/6.06.04/bin/root

Solution: in setup.sh script, change to root 5.34. Then this error does not occur anymore.

• Error 2 when running coral:

Error in <TMacro::ReadFile>: Cannot open file:
/projects/sciteam/badp/compass//geometry/2015/ROOTGeometry/detectors.C

July 4, 2016: try to copy some data to BW via personal Globus Connect

• Create my work directory $WORK on CERN afs: see link work-directory (100GB quota. Home directory has only 10GB.)
• Copy a test raw 2016 run into $WORK/COMPASS/DATA/2016 :

nsls -l /castor/cern.ch/compass/data/2016/raw/tmp | grep 271014
stager_get -M /castor/cern.ch/compass/data/2016/raw/tmp/cdr14060-271014.raw 
===> "/castor/cern.ch/compass/data/2016/raw/tmp/cdr14060-271014.raw SUBREQUEST_READY"
xrdcopy root://castorpublic.cern.ch//castor/cern.ch/compass/data/2016/raw/tmp/cdr14060-271014.raw .

• This is one chunk of run 271014 (born on the 4th of July).
  • 245 = nsls -l /castor/cern.ch/compass/data/2016/raw/tmp | grep 271014 | wc -l, with each about 1.1GB. I.e. I cannot even copy an entire run using this space.
  • Copy 90 chunks: nsls /castor/cern.ch/compass/data/2016/raw/tmp | grep 271014 | head -90 > 2016_271014_shortlist.txt . Edit with emacs and replace string cdr by root://castorpublic.cern.ch//castor/cern.ch/compass/data/2016/raw/tmp/cdr (shift 1 is bulk replace).
  • for i in `cat 2016_271014_shortlist.txt`; do xrdcopy $i . ; done (start at 10:55; end at 11:54, i.e. 1 hour for 100GB)
• Unfortunately, the $WORK directory is not accessible in the Globus Online web interface. Set up Globus Connect in $WORK directory, following the steps of July 1. But this endpoint also sees files in my home directory only. I delete it again.
• Transfer files via GO CLI: explored. See BW Illinois Twiki

July 1, 2016: install personal GO endpoint in my CERN user account.

• Details see BW Illinois Twiki.
• Now I can add the new CERN endpoint to my web interface.
• Copy of a test file from my CERN home directory to BW successful.
• /castor/cern.ch/compass directory however not accessible (not surprisingly) and also not /eos/compass and /eos/user/c/criedl. Need gateway between GridFTP and EOS?

June 2016

June 29, 2016: Set up MyRun directory structure on BW

Changes on lxplus:

• MyRun/DC5 ==> MyRun/Phast-DC5
• MyRun/Physics ==> MyRun/Example
• MyRun/mDSTs/MymDSTs ==> MyRun/mDSTs/Output
• In MyRun, move all trafdic files into new directory TrafDic
• Corresponding adjustment of environment scripts on lxplus:

Actions on BW:

• Copy MyRun directory to BW and adjust links (some links copied as directories)
• Link phast executable in MyRun/Phast-DC5/2015-P04-1 directory

June 28/29, 2016: coral and phast on lxplus

• Setup env. as described here (same as June 2)
• Vincent helps me to identify reason for phast crash on June 4: in my coral option file, the general coral_svn/trunk/src/user/trafdic.2015.opt is included. And there DC05 was still specified as IS_MWPC!Comment out these lines.
• Then I have to run coral again. Output will end up in MyRun/mDSTs/MymDSTs/DC5/2016-06-28 .
• Run coral: cd MyRun/mDSTs ; ./coral.exe trafdic.DC05.DCmode.opt
• Error: CsDriftChamberDetector.cc
  • DC05U2__: Not catered for => No propagation time correction.
  • Unlike I recommended to myself in Oct. 2015, "In coral_svn/trunk/src/geom/CsDriftChamberDetector.cc, the string DC5 has to appear in two places (not counting appearances in comments)", this is NOT the case.
  • In line 832, add: || strncmp(tbn,"DC05",4)==0 (I do not understand why this happened. I should have checked out a coral version where DC05 is taken care of already? Anyway will check out later a fresh version of coral.)
• Re-compile coral
  • Set up more env
  • Then proceed as described here. Steps 2, 3,4. Use "old personal setup" and "phast.7.148"
• Run coral (same run as on June 2)
  • Interactively: in MyRun/mDSTs ./coral.exe trafdic.DC05.DCmode.opt: results in send2nsd: NS002 - send error : No valid credential found (2x)
  • Send batch: ./ManyRDs.csh -d MymDSTs/DC5/ trafdic.DC05.DCmode.opt 1, finishes without output
  • Use instead of cdr11002-261645.raw: cdr14112-261645.raw, then interactive job produces something
  • In MyRun/DC5/2015-P04-1, ./phast -h MyOutput.root -u11 -U34881 -T DC05Y current/phast.root . There is something in the produced root file.
  • In MyRun/mDSTs, send batch: ./ManyRDs.csh -d MymDSTs/DC5/ trafdic.DC05.DCmode.opt 1 (runs 2-3 hours on Tuesday evening)
  • Wednesday: In MyRun/DC5/2015-P04-1, ./phast -h PhastOutput.DC05.DCmode.14112-261645.root -u11 -U34881 -T DC05Y current/mDST.DC05.DCmode.14112-261645.root
    • Some histograms are filled; but I don't see all the files I expect.
    • Next: run -U34885 (mv PhastOutput.DC05.DCmode.14112-261645.root PhastOutput.DC05.DCmode.14112-261645_U34881.root)

June 17/27, 2016: Set up Globus Online Command Line Interface

• Finally success - see docu on BW Illinois Twiki: http://twiki.npl.illinois.edu/bin/view/COMPASS/DataTransfer
• Was able to copy a test directory from BW-disk to BW-tape.

June 16, 2016: How to copy from CASTOR to EOS

• To have a gridftp endpoint at CERN.
• Note: this might not be needed - Damien's script copies from CASTOR with automatic stageing from what I understood

Access to EOS:

ssh -X lxplus.cern.ch
source /afs/cern.ch/project/eos/installation/user/etc/setup.sh 
eos ls -l /eos/user/c/criedl   
source /afs/cern.ch/project/eos/installation/compass/etc/setup.sh
eos ls -l /eos/compass

Access to CASTOR:

nsls -l /castor/cern.ch/compass/data/2016/raw/tmp |awk '$7==16'

Want to copy from CASTOR to EOS:

/castor/cern.ch/compass/data/2016/raw/tmp/cdr14052-270162.raw

• I have a user directory on EOS, /eos/user/c/criedl. eos quota is 2TB from what I see. To access this dir, need to source the first setup.sh.
• EOS tutorial
• EOS documentation

June 14/15, 2016: Setting up BW - part 1

• Compiling CORAL and PHAST: see http://twiki.npl.illinois.edu/bin/view/COMPASS/Code#CORAL_and_PHAST_compilation
• Set up gitlab:
  • Note: escalade-framework is now the main gitlab module for analysis, drell-yan is the submodule. Choose SSH option instead of KRB5 option from drop-down menu.
  • Setup ssh key

mkdir git
cd git
git clone ssh://git@gitlab.cern.ch:7999/illinois-and-friends-group/escalade-framework.git 
git clone ssh://git@gitlab.cern.ch:7999/illinois-and-friends-group/help-for-newcomers.git
git config --global user.name "Caroline Kathrin Riedl"
git config --global user.email "caroline.riedl@cern.ch"
export PHAST=/u/sciteam/criedl/COMPASS/phast
cd escalade-framework
. setup.sh
make

For the time being: wait until escalade and dy are ready. Then also remember git clone --recursive http://... Then set up gitlab framework on BW and re-compile software.

June 4, 2016: Getting started with phast on lxplus

• set up env. as on June 2

1. Compile UserEvent11 for DC efficiency
   • copy /afs/cern.ch/user/y/ybedfer/phast.utils/user/*User*11* to my phast/user/
   • goto git/dy-analysis, ./SetUpEnvironment.sh
   • make phast (i.e. re-compile phast in git framework. Problems with SetTresol: not declared in this scope. Check if I need newer version of phast.
   • Download Phast.tar.gz.7.149.
     • mv Phast.tar.gz.7.149 Phast.7.149.tar (it seems when I download the file, my computer automatically unzips it...?)
     • tar xvf Phast.7.149.tar
     • voila: phast.7.149 directory.
   • SetTresol is also not in phast.7.149. Continue using phast.7.148 for the time being. Instead, use older version (1.27) of UserEvent11.cc. Then phast compiles.
2. Prepare running of phast (following Mathieu's recipe)
   • Create new directory myRun/DC5, and in there, 2015-P01-S through 2015-P09-S, where S=1 or 2 if the period has 2 separate alignment files, or S=12 if it has 1 common alignment file (detectors.dat).
   • Link phast executable in each of these directories.
   • Link proper detectors.dat in each P-directory. Name string of link has to be called exactly detectors.dat. For example, the one chunk I produced with coral belongs to run 261645 = W10 = P4, sub-period 1, i.e. detectors.261513.transv.dat (for the time being, I use Chia-Yu's table). for 2015-P04-1.
   • In coral_svn/trunk/src/condb/mysqldb/Utils, compile with make to create getDBFilePath.
   • Run getDBFilePath -r 261645 DC05Y1 and similarly for Y2, U1, U2, V1, V2 to create proper calibration files.
   • Copy them to a subdirectory in DC5/RT-calibration, where the directory name indicates the creation date of the calibration file in the MySQL data base. In this example for run 261645: 2015-09-23
3. Run phast in 2015-P04-1 with ./phast -h MyOutput.root -u11 -U34881 -T DC05Y current/mDST.DC05.DCmode.11002-261645.root
   • error message ** UB: Mode 0x800 (i.e. correct for event time and signal propagation) requested while argument dets are not drift-like. I do not understand the line _ => Examining (option "IS_MWPC") detector plane(s):_ . The mDST was not produced in MWPC mode. Where does this come from?

June 2, 2016: Run coral on lxplus

Set up environment: (as described on main page)

cd coral_svn/trunk
. setup.sh 
cd ../../MyRun/mDSTs
./PreExeCoral.sh

./coral.exe trafdic.DC05.DCmode.opt Intercatively OK - 1 chunk produced as MymDSTs/phast.root (2.6MB) after I killed job by hand.

root -t phast.root 
TBrowser b

• Submit in batch mode: ./ManyRDs.csh -d MymDSTs/DC5/ trafdic.DC05.DCmode.opt 1
• Result (after ~ 6 hours for only 1 chunk?!): MymDSTs/DC5/mDST.DC05.DCmode.11002-261645.root (108MB).
• Move in directory MymDSTs/DC5/2016-06-02

June 1, 2016: set up git on lxplus

mkdir /afs/cern.ch/user/c/criedl/COMPASS/git
cd git
git clone ssh://git@gitlab.cern.ch:7999/illinois-and-friends-group/dy-analysis.git
export PHAST=/afs/cern.ch/user/c/criedl/COMPASS/phast
cd dy-analysis
. setup.sh
--> Add /afs/cern.ch/user/c/criedl/COMPASS/git/dy-analysis/lib to LD_LIBRARY_PATH and DYLD_LIBRARY_PATH (specific for MacOSX)
--> Define STAGE_HOST as castorpublic
--> Define STAGE_SVCCLASS as compassuser
--> Set the variable ROOTDIR as /afs/cern.ch/user/c/criedl/COMPASS/git/dy-analysis

Now typing make, he complains that ROOTSYS is not set. I have to type:
export ROOTSYS="/afs/cern.ch/sw/lcg/app/releases/ROOT/5.34.09/x86_64-slc6-gcc47-dbg/root"
Then I can compile.

Then : make depends

/afs/cern.ch/user/c/criedl/COMPASS/phast/phast: error while loading shared libraries: libGeom.so: cannot open shared object file: No such file or directory
--> Configuration selected :
   PHAST : /afs/cern.ch/user/c/criedl/COMPASS/phast
   PHAST Infos : 
   PHAST Arguments : -u93
   PHAST Output : /castor/cern.ch/user/c/criedl/prod/dy15W10t2_93
   PHAST Saved files : DST stored
--> Data configuration:
   Name : dy15W10t2_93
   Directory selected: /castor/cern.ch/compass/generalprod/testcoral/dy15W10t2/mDST
   Pattern of files : ".*DST.([0-9]*)-[0-9]-[0-9].root.*"
   Data output will be here: /afs/cern.ch/user/c/criedl/COMPASS/git/dy-analysis/src/dy15W10t2_93
--> Looking for data for dy15W10t2_93 matching with the following pattern ".*DST.([0-9]*)-[0-9]-[0-9].root.*"
--> Cleaning empty files in dy15W10t2_93..DONEcompass/generalprod/testcoral/dy15W10t2/mDST
--> Total 395 files found
--> Compiling library..

Generates runlists of the form src/dy15W10t2_93/261647.txt .

Then: compile PHAST. First need to set up more environment:

. /afs/cern.ch/sw/lcg/contrib/gcc/4.7/x86_64-slc6-gcc47-opt/setup.sh
export CERN=/afs/cern.ch/sw/lcg/external/cernlib/2006a
export CERN_LEVEL=x86_64-slc6-gcc47-opt
export ROOTSYS="/afs/cern.ch/sw/lcg/app/releases/ROOT/5.34.09/x86_64-slc6-gcc47-dbg/root"
export LD_LIBRARY_PATH=$ROOTSYS/lib:/afs/cern.ch/sw/lcg/contrib/gcc/4.7/x86_64-slc6-gcc47-opt/lib:${LD_LIBRARY_PATH}
. /afs/cern.ch/sw/lcg/app/releases/ROOT/5.34.09/x86_64-slc6-gcc47-dbg/root/bin/thisroot.sh

make phast

It takes all source files from my phast directory, and the local users directory.

Code crashes with

NicePlot.cc: In static member function ‘static TCanvas* NicePlot::Draw(TString, TH1*, TH1*, Bool_t, Double_t, Double_t)’:
NicePlot.cc:49:2: error: ‘SetExponentOffset’ is not a member of ‘TGaxis’

Codes compiles if in lib/NicePlot.cc the following line is outcommented:
//      TGaxis::SetExponentOffset(0.01, -0.05, "x");
Investigate. Not super urgent. Might be root version issue.

Submit a test phast job.

make try

with /castor/cern.ch/compass/generalprod/testcoral/dy15W10t2/mDST/mDST-261515-0-7.root. output.root (tree) and hist.root (histograms only) are written. Checked - OK.

October 2015

October 19, 2015

cd coral_svn/trunk . setup.sh cd ../../MyRun ./PreExeCoral.sh

• Adapt trafdic.DC05.DCmode.opt to run over run 261645 (1 chunk) with detectors.2015.W10.alon.ai3_forAlain.dat
• run interactively: ./coral.exe trafdic.DC05.DCmode.opt and work station 0. Looks OK - processes events.
• run on batch: ./ManyRDs.csh -d MymDSTs/DC5/ trafdic.DC05.DCmode.opt 1

* ManyRDs: Parsing and checking command line arguments and options file...
 * ManyRDs: All checks done. Proceeding to submit jobs..
 * Stageing run 261645, 11001 <= chunk < 11002
 * ManyRDs: Submitting chunk 11001-261645
Job <710260527> is submitted to queue <1nd>.

root -t phast.root 
TBrowser b

September 2015

September 2, 2015

Instead of MyEnvironmnent.sh, introduce 2 separate scripts:

PreCompilation.sh
PreExeCoral.sh

So that:

• Before compiling CORAL (which does not happen so often)

cd coral_svn/trunk
./PreCompilation.sh

• Before a normal analysis session,

cd coral_svn/trunk
. setup.sh 
cd ../../MyRun
./PreExeCoral.sh

Note: the 'export' lines in *.sh have to be pasted into the command line. Executing the script does not do the job. It's OK to do that for the time being, but why...

Next try:

• Use Yann's .tcshrc and try to run in bash. Is OK interactively after following above recipe.
• Runs in batch. But crashes after 8.5 hours (1 chunk only!) because file size exceeded...

September 1, 2015

Log of jobs: symbol lookup error: /afs/cern.ch/user/c/criedl/COMPASS/phast/coral/coral.exe: undefined symbol: gROOT ** coraljob: WARNING: Error in the execution of coral

But it works interactively! Do I have to set env. on castor?

Try with Yann's original ManyRDs.csh script. I had changed the following line: #set outDir = ~/w0/csub | set outDir = ~/w0/csub set outDir = /castor/cern.ch/user/c/criedl/DY2015/2015-08 <

Same error as before. The job does definitively run interactively - ./coral.exe trafdic.DC05.DCmode.opt and then "0" (choose no work station)

Try everything in csh ...

- log in - csh - in coral_svn/trunk: source setup.csh - setenv MYDIR /afs/cern.ch/user/c/criedl/COMPASS - setenv MYOUTPUTDIR ${MYDIR}/MyRun/mDSTs/MymDSTs - setenv PHAST ${MYDIR}/phast - in MyRun/mDSTs, ./coral.exe trafdic.DC05.DCmode.opt produces many events! - Now send batch job! ./ManyRDs.csh -d MymDSTs/DC5/ trafdic.DC05.DCmode.opt 1

Same problem.... OK. Get some help now. Now that I stripped down the environment it is easier to explain.

August 2015

August 31, 2015

Starting a fresh analysis session, code seems to run interactively. (Are my problems arising from losing X credentials?)

It is important to not only run the scripts as described on my Twiki, but also to run the following commands (not sure if all are needed):

. /afs/cern.ch/sw/lcg/app/releases/ROOT/5.34.09/x86_64-slc6-gcc47-dbg/root/bin/thisroot.sh export MYDIR=/afs/cern.ch/user/c/criedl/COMPASS export MYOUTPUTDIR=${MYDIR}/MyRun/mDSTs/MymDSTs export PHAST=${MYDIR}/phast

(why does this not work from script? Investigate.)

Then submit job: ./ManyRDs.csh -d MymDSTs/DC5/ trafdic.DC05.DCmode.opt 10

August 26, 2015

Checked out new version of coral (14063) and compiled everything successfully. Now CsDriftChamberDetector.cc is ready for DC5. Trying to run over /castor/cern.ch/compass/data/2015/raw/W09/cdr11002-260841.raw : kSigSegmentationViolation, seems to be a root problem. Try to fix it.

August 24, 2015

Root files couldn't be produced: /afs/cern.ch/user/c/criedl/COMPASS/phast/coral/coral.exe: symbol lookup error: /afs/cern.ch/user/c/criedl/COMPASS/phast/coral/coral.exe: undefined symbol: gROOT Debugging interactively.

ERROR, on Mon, 24/Aug/2015 19:24:11.168530 (GMT) from: CsEvent.cc 1576 `CsEvent::_decode(): exception: CsDriftChamberDetector::DecodeRawData(): Wrong digit type Decoding error for DC05V1__, detector may be empty during reconstruction.'

Same for V2, Y1, Y2 (but not for U1, U2!)

August 21, 2015

Sent first job ./ManyRDs.csh -d MymDSTs/DC5/ trafdic.DC05.DCmode.opt 1 But it had not good output.

-- CarolineRiedl - 2015-10-19