--
ParasNaik - 2017-05-16 --
ClaireProuve - 2015-07-18
Code for the Online RICH mirror alignment
RICH mirror alignment project and package overview
The alignment package for the online RICH mirror alignment (Rich/RichMirrorAlignmentOnline) is always kept and sourced from the LHCb project Panoptes git repository.
- code currently in the master branch of Panoptes:
https://gitlab.cern.ch/lhcb/Panoptes/tree/master/Rich/RichMirrorAlignmentOnline
At the pit, this package and others are drawn into our development directory of AlignmentOnline, in this directory here:
/group/rich/sw/cmtuser/AlignmentRelease
- NOTA BENE: AlignmentRelease is a soft link to
/group/rich/sw/cmtuser/AlignmentOnlineDev_vYYrX
-
vYYrX
when referenced in this TWiki should always be replaced with numbers for the current revision
- The current version being used by us online is v12r1. Experts, always keep this line updated.
- Also, update the soft link if the version changes!
This package relies on the projects:
-
AlignmentOnline
- This package relies on the
Online
project
-
Panoptes
whose packages are used as plug-ins
- This includes
Rich/RichMirrCombinFit
, the key package for fitting the reconstructed the RICH photons
- This includes
Rich/RichMirrAlign
, the package that performs the minimization required to determine the best alignment
-
Rec
whose packages are used as plug-ins
- This includes
Rec/Brunel
, the key package for the reconstructing the RICH photons
As you can see, the AlignmentOnline project relies on a lot to run our code. It is important then that we strictly follow the code workflows described in
LHCbRichMirrorAlignGitInfo and this document.
This TWiki page specifically discusses the
Rich/RichMirrorAlignmentOnline
package, and its special relationship to the
AlignmentOnline
project, that has been (or can be) installed at the pit.
More information on specific packages of interest and other projects is available on the main
LHCbRichMirrorAlign page.
The RichMirrorAlignmentOnline package
This package contains the "Iterator" and it's "Helper files", the "Analyzer", the Configuration for the alignment and the MirrPauseHandler. It also contains doc/release.notes with a full history of the code.
Most recent
overview talk
Key Definitions
Iterator = The script that drives the mirror alignment
Analyzer = The code that configures Brunel
analyzers = The HLT farm (a.k.a. worker) nodes that we use to process the data that comes from our RICH Mirrors HLT1 line
Configuration = The file which lists sets current state of every configurable option in the mirror alignment.
The Iterator (Iterator.py)
The Iterator drives the Rich mirror alignment. It sets up all necessary files, writes the xml files, puts them in the right place where the analyzers can pick it up for the next iteration, performs the fits an decides if the alignment has converged.
Everything in the iterator is steered by the
run(whichRich)
method, while most functionalities are outsourced into classes and methods.
- whichRich is provided by the choosing Rich1 or Rich2 in the run-control while running the alignment.
- The configuration is read in and variables are set.
- The while loop begins.
- The very first minor iteration is separate because the different command that will come from the run control, the rest of the iterations will be in the same place.
- At the start of the first iteration the Work-directory will get emptied out.
- The starting xml-file is gotten from '/group/online/alignment/Rich' + str(whichRich) + '/MirrorAlign/'
- The xml-files for the mirror-tilts for the magnification factors are made (unless we are running in predetermined magnification factors mode).
- The xml-file for the first iteration put in the place where the analyzers will pick it up.
- Set Iterator state to ready which will make the analyzers run their jobs.
- When the analyzers are ready the iterator will get the command "pause" and will check if one major iteration has been finished already:
- if not then it just copied the xml file for the next iteration to the place where the analyzers can pick it up
- if yes it will perform the fits and the alignment and check weather or not is has converged. If yes it's finished, if no it starts again at 7.
Iterator code (well documented) currently in the master branch of Panoptes:
https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/Iterator.py
Here a beautiful sketch of what the iterator does:
The Helper Files (*.py)
There are many helper files that contains classes that outsource all of the tasks that the Iterator needs to perform. These classes are contained within various
.py
files located in the same directory as the
RichAnalyzer and the Iterator. There is usually one class per file, except that
SaveSetRetriever and Histocopier are both in
HistoHelper.py. The classes contained within these helper files are described below:
RichAlignmentHelper class
• The methods in this file are used to run
RichMirrCombinFit and
RichMirrAlign
• checkFitQual just returns whether all bins of the deltaTheta vs. phi histogram have enough photons
• hasConverged checks to see if
RichMirrAlign thinks the alignment has converged
XMLFileHelper class
• This helper manipulates the XML files and helps prepare them for consumption by the Online DB or by the Brunel Configuration.
• It also prepares the final XML file.
tiltObj class
• This is Claire’s very nice python object that can
store the difference between one alignment and
another.
AlignMonitor class
• The methods in this file are used to perform the monitoring, and also to fit the deltaTheta plot that is shown in the final PDF summary file.
• It relies heavily on methods from tiltObj.py
UpdateHelper class
• This is the “DIM” service that we launch when our
alignment starts.When we decide what mirror alignment
we wish to apply to this fill, we use the updateDB method
to instruct Clara (via a program she wrote) to whether to
use the existing XML as our alignment for this fill, or an
new one.
SetupHelper class
•
SetupHelper determines the alignment name (i.e. the eventual directory name the working directory will be copied to when done). It also launches
the monitoring, finalizes the alignment when finishes, and writes summary.txt, emails, and ELOG entries.
trendHelper class
• trendHelper makes the long-term alignment trend plots. It can be run on its own manually, but also runs automatically with each automated mirror alignment.
fileFinder class
• Don’t worry too much about fileFinder.py
• It allows the trendHelper to find the historical XML files it needs to make the trend plots.
• You’ll probably never touch it.
Communicator class
• Don’t worry too much about Communicator.py
• It allows the Iterator to communicate with the run-control.
The key method is get_command()
• You’ll probably never touch it.
HistoHelper class
• Don’t worry too much about
HistoHelper.py
• It manages the retrieval of the
SaveSets (histograms) from the analyzers.
SaveSetRetriever class:The
SaveSets are not written by the Iterator but a very different part of the program. In order to know where to retrieve the save sets from, a "DIM" server method is used to retrieve the location where the histogram was written in real-time.
Histocopier class:
Encapsulates waiting for the
SaveSets being written, matching it to the correct iteration and copying it to the desired location.
• You'll probably never touch it.
These pieces of code as currently in the master branch of Panoptes:
https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/
The Analyzer is the entity that will configure the analyzers and execute the Brunel jobs and produce the histogram files.
The Analyzer code only contains the options for the Brunel jobs which will be read in the during the "configuration".
During run-time, the analyzers will pick up the rich mirror xml file for the reconstruction from the exact same place every time.
Analyzer code currently in the master branch of Panoptes:
https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/RichAnalyzer.py
The Configuration file (Configuration.py)
The Configuration file contains a lot of different settings for the alignment for Rich1 and Rich2 separately (this might be done smarter, but later). Some things in there should not be changed because they are part of the online-setup.
The Configurations will be read in by both the Iterator and the Analyzer.
Configuration file code currently in the master branch of Panoptes:
https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/python/RichMirrorAlignmentOnline/Configuration.py
The MirrPauseHandler contains the code that is executed when the analyzers go to 'paused' - which is after each iteration. It specifies (to a certain degree) when and where the histogram files are written. This code does (most probably) not need to be changed anymore.
MirrPauseHandler code currently in the master branch of Panoptes:
https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/src/MirrPauseHandler.cpp
Location and operation of the XML files [IMPORTANT]
The XML file the alignment is starting from is being picked up from here:
/group/online/alignment/Rich1/MirrorAlign/
/group/online/alignment/Rich2/MirrorAlign/
These XML files are colloquially called Vs (pronounced vees). When there is a new alignment, the Iterator produces a new V. Thus a particular V is simply a particular version of the mirror alignment compensations.
The Iterator will pick the XML file with the letter "=v=", followed by the biggest version number, followed by "=.xml=".
If you want to run starting with a different XML file, there's an option in the
Configuration.py
file.
The XML files in the above directories should not have a header, therefore when getting a starting XML file the header is being added by the Iterator, and when the finishing xml-file will be copied there it will have its header removed. If you add your own custom XML file to the list remove the header (= the first three lines; and the little </DDB> at the end).
We will run in one of three modes :
- Monitoring mode CURRENT MODE OF OPERATION
- in
Configuration.py
-
autoUpdate = False
-
testing = False
- If the mirror alignment decides no updated is needed, the same V remains in place e.g. v{N}.xml
- If the magnet polarity changed, for RICH1 and RICH2 each, the new V is written into
v{N+1}_maybe_YYYYMMDD_HHMMSS.xml
where YYYYMMDD_HHMMSS corresponds to the alignment name
- Consult with lhcb-rich-mirror-alignment-development@cernNOSPAMPLEASE.ch (Claire, Paras, and everyone else doing mirror alignment) about the = maybe = alignment, to make sure we think it's OK.
- If it is agreed, copy
v{N+1}_maybe_YYYYMMDD_HHMMSS.xml
to v{N+1}.xml
- Send a group email with the new v{N+1} for each RICH to:
- If the mirror alignment decides an update is needed, the new V is written into v{N+1}_maybe_YYYYMMDD_HHMMSS.xml where YYYYMMDD_HHMMSS corresponds to the alignment name
- If any of the alignment parameters is more than 2 times the {primary,secondary} {y,z} tolerance for any mirror combination, consider making an alignment update.
- When considering making the update, consult with the following people:
- If it is agreed, copy v{N+1}_maybe_YYYYMMDD_HHMMSS.xml to v{N+1}.xml
- The update will be picked up on the next fill.
- If for some reason the alignment did not run for one or both of the RICH detectors; instruct the HLT piquets to use the current V.
- If the problem seems to be with the farm, wait for the next fill and see what happens.
- If the problem definitely is something in our code, try to fix it.
- Any problems with the mirror alignment running at the pit, that the available mirror alignment experts cannot collectively solve immediately, send an group email to:
- Automatic mode (for later)
- in
Configuration.py
-
autoUpdate = True
-
testing = False
- If the mirror alignment decides no updated is needed, the same V remains in place e.g. v{N}.xml
- If the mirror alignment decides an update is needed, the new V is written into v{N+1}.xml
- The update will be picked up on the the current fill (the one on which the alignment was performed).
- Testing mode (if need be)
- in
Configuration.py
-
testing = False
(autoUpdate gets turned to False within the code if it happens to be True)
- If the mirror alignment decides no updated is needed, the V is written to v{N+1}_testing_YYYYMMDD_HHMMSS.xml
- If the mirror alignment decides an update is needed, the new V is written into v{N+1}_testing_maybe_YYYYMMDD_HHMMSS.xml
- The update is not picked up.
Other interesting places
'workDir', the Work-directory
This is the directory in which all the files created during an alignment are being written.
/group/online/AligWork/Rich1
/group/online/AligWork/Rich2
Savesets
Before the histogram files are being copied into the Work-directory they will be placed into
/hist/Savesets/2015/LHCbA/AligWrk_Rich1
/hist/Savesets/2015/LHCbA/AligWrk_Rich2
(check the year)
Driver scripts for the all online alignments
This is the script that holds everything together at the pit. It was made by the online people and we can only touch it if we are VERY careful!
/group/online/dataflow/cmtuser/OnlineDev_v6rX/Online/FarmConfig/job/AligDrv.sh
(for the Iterator)
/group/online/dataflow/cmtuser/OnlineDev_v6rX/Online/FarmConfig/job/AligWrk.sh
(for the
RichAnalyzer configuration of the farm nodes)
where "v6rX" is whatever version of Online is being accessed to run things at the pit (check if unsure!): This is NOT the version AlignmentOnline depends on, but should be the newest version of Online.
Logging staging area
When the alignment runs, we store the log of the machine that the Iterator runs on, which is always
hlt02
.
The output of hlt02 (including messages that aren't ours) during the last Rich1 alignment is stored in
/group/rich/AlignmentFiles/Logging/Rich1_hlt02.log
The output of hlt02 (including messages that aren't ours) during the last Rich2 alignment is stored in
/group/rich/AlignmentFiles/Logging/Rich2_hlt02.log
If an alignment is completed properly this is later moved to the output storage area. If not, and the Iterator makes it to the end, it is saved in this directory with a timestamp (you can safely delete old time stamped files, if the mirror alignment is operating OK).
- In either of these cases an "hlt02reduced" log is also created, this has a
grep
applied to it that filters out junk messages that show up on hlt02, that are unlikely to be from our mirror alignment iterator.
If the Iterator does not make it to the end, then it just remains in this directory until the next mirror alignment. However in this case the next alignment's output
appends to this file, instead of overwriting it as usual.
At the beginning of an alignment, the current hlt02 logs are copied to "_previous" to preserve the last run.
Some more information about looking at and inspecting these files can be found here:
LHCbRichMirrorAlignShiftInfo
Rolling log of Cherenkov angle resolutions
A rolling log of Cherenkov angle resolutions is kept in these files:
/group/rich/AlignmentFiles/Logging/Rich1_CKresTrend.txt
/group/rich/AlignmentFiles/Logging/Rich2_CKresTrend.txt
Do not touch these, unless you're editing them to remove multiple data points (e.g. the alignment was for some reason run twice on the same fill).
You can see
plots of this data
and
comparisons to the resolutions from OnlineBrunel
uses the data in these text files to provide a trend of the CK angle resolutions from the mirror alignment.
If
testing = True
in the Configuration for a particular mirror alignment, then the data is appended to this file with a "# TEST " designation on the line; this line is subsequently ignored.
Rolling log of Mirror Alignment updates
A rolling log of mirror alignment updates is kept in these files:
/group/rich/AlignmentFiles/Logging/Rich1_UpdateTrend.txt
/group/rich/AlignmentFiles/Logging/Rich2_UpdateTrend.txt
Do not touch these, unless you're editing them to remove multiple data points (e.g. the alignment was for some reason run twice on the same fill), OR the mirror alignment was a
_maybe
alignment that you decide to make the active alignment for a particular fill, in which case you need to adjust this file accordingly.
This file is used by trendHelper.py to automatically determine when there has been a magnet polarity change.
If
testing = True
in the Configuration for a particular mirror alignment, then the data is appended to this file with a "# TEST " designation on the line; this line is subsequently ignored.
Magnet polarity memory
The polarity of the last mirror alignment that converged, had no sanity problem, and was not a testing alignment, is stored here:
/group/rich/AlignmentFiles/Polarity/previousPolarity_RICH1.txt
/group/rich/AlignmentFiles/Polarity/previousPolarity_RICH2.txt
The code checks to see if the polarity of the current alignment is different from the previous alignment. If
autoUpdate = True
in the Configuration, then we will force an update of the mirror alignment (unless sanity checks are failed, of course).
Then at the end of the code, the previousPolarity file is overwritten with the current polarity information, unless
testing = True
.
Only the first line of this text file is actually compared. You may need to manually edit the first line back to indicating the previous polarity, if for some reason the alignment aborts during a polarity change.
Non-default MDCS corrections.
The
RICH1MDCS
is a system used to correct for distortions of the HPD photocathode images in the LHCb magnetic field.
The final corrections are are in the form of an XML file, which is obtained by the mirror alignment through the
CondDB.
- In the Configuration, for RICH1 (and for the heck of it, RICH2):
xmlMDCS = ""
However, if we want to use a different XML file we need to tell the mirror alignment through the configuration the location of the file
- for example, in the Configuration, for RICH1 (and for the heck of it, RICH2):
xmlMDCS = "/group/rich/AlignmentFiles/MDCS/Conditions/Rich1/Environment/HPD.xml"
See
/group/rich/AlignmentFiles/MDCS/README.txt
for details on what you should do if you are asked to use new MDCS corrections at the pit.
The subset of possible mirror combinations that are used for the mirror alignment are contained within the
RichMirrorAlignmentOnline package, and stored here at the pit:
/group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/files/Rich1CombAndMirrSubsets.txt
/group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/files/Rich2CombAndMirrSubsets.txt
Do not touch these files.
Magnification Factors
If
predetermined magnification factors are to be used (currently the default case), then they will be read from subdirectories which are contained within the
RichMirrorAlignmentOnline package, and stored inside these root directories at the pit:
/group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/files/MagnifFactors/RICH1/
/group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/files/MagnifFactors/RICH2/
Do not touch these files directly! If you are asked to add new magnification factors, then add them using the instructions below.
Recalculating Magnification Factors
Change
magnFactorsMode
from 0 to 2 in the
Configuration.py
and then recompile.
Run the mirror alignment. The mirror alignment will take 30 minutes per iteration instead of 3 minutes per iteration.
If the mirror alignment completes successfully, find the
savedir
in
summary.txt
or the RICH ELOG
If the alignment took N+1 iterations to complete (i0, ..., iN) and ZZ is the last two digits of the year the alignment data was taken, then the new magnification factors are in the files you can see if you go to the
savedir
and execute the command:
ls *CollisionZZ_*_iN.txt
Change
magnFactorsMode
back from 2 to 0 in the
Configuration.py
and then recompile.
Updating Predetermined Magnification Factors
Depending on which RICH detector you wish to update the predetermined magnification factors for,
check out the latest version of Panoptes from run2-patches and then go into one of the following directories:
Rich/RichMirrorAlignmentOnline/files/MagnifFactors/RICH1/
Rich/RichMirrorAlignmentOnline/files/MagnifFactors/RICH2/
then make a new directory with the same
YYYYMMDD_HHMMSS
timestamp as the alignment savedir that contains the recalculated magnification factors:
mkdir YYYYMMDD_HHMMSS
cd YYYYMMDD_HHMMSS
Then copy over the
*CollisionZZ_*_iN.txt
files from the savedir.
The last step is renaming. This is currently painstaking, but shouldn't take you too long.
the files you have will look like
Rich1MirrMagnFactors_Mp6Wi8.0Fm5Mm2Sm0_online_Collision18_pri_negYzerZ_i1.txt
you need to change these to files that look like
Rich1MirrMagnFactors_pri_negYzerZ_predefined.txt
Then when you are done also copy over
summary.txt
from the savedir
Lastly you will need to edit
Configuration.py
and change the
magnifDir
to point to the full directory path where the new predefined magnification factors are.
As soon as this is all done, commit the changes and push to a new branch.
At the pit, check out Rich/RichMirrorAlignmentOnline from your new branch and then recompile the code.
Ask for a merge request and make sure you merge into run2-patches.
Then ask Jordi if he could kindly also merge those commits into master.
Fresh install of code for new AlignmentOnline
version
If you are not asked to install a new
AlignmentOnline
version at the pit, then you may
skip this section (though it's worth a read once anyway, to see how the code comes together).
Login to a
plus
machine.
First, add these two lines to the end of your .bashrc:
echo "Check your current gid below. If it is not what you want, use newgrp to change your gid."
id
This will tell you your user ID (id), which group (rich, online, etc..) permissions are currently active (gid), and which groups your account can choose from (groups), in case you need to change your gid.
Here's what you need to do if you want to freshly install the mirror alignment on an online machine (e.g. pluscc, a machine running centos7). The following example is for
AlignmentOnline
v12r1.
DO NOT FOLLOW THIS EXAMPLE VERBATIM. Only for a newer version of
AlignmentOnline
, should you do this process, and then substitute the new version for v12r1 below.
echo "I have just set the User_release_area environment variable to:"
export User_release_area=/group/rich/sw/cmtuser/
echo $User_release_area
LbLogin
You could add these lines to the end of your .bashrc if you want.
Next, and this only has to be done once:
cd $User_release_area
lb-dev AlignmentOnline/v12r1
cd AlignmentOnlineDev_v12r1
Then find out what version of Online is newest. For example if it is v7r0, edit
CMakeLists.txt
and change:
gaudi_project(AlignmentOnlineDev v12r1
USE AlignmentOnline v12r1)
to
gaudi_project(AlignmentOnlineDev v12r1
# USE AlignmentOnline v12r1)
USE AlignmentOnline v12r1
OnlineDev v7r0 # Paras added this
Next, add these lines to the top of
build_env.csh
:
if ( $?CMTPROJECTPATH ) then
setenv CMTPROJECTPATH "/group/online/dataflow/cmtuser:${CMTPROJECTPATH}"
else
setenv CMTPROJECTPATH "/group/online/dataflow/cmtuser"
endif
if ( $?CMTPROJECTPATH ) then
setenv CMTPROJECTPATH "/group/rich/sw/dev-dir:${CMTPROJECTPATH}"
else
setenv CMTPROJECTPATH "/group/rich/sw/dev-dir"
endif
Then, add these lines to the top of
build_env.sh
:
export CMTPROJECTPATH="/group/online/dataflow/cmtuser${CMTPROJECTPATH:+:${CMTPROJECTPATH}}"
export CMTPROJECTPATH="/group/rich/sw/dev-dir${CMTPROJECTPATH:+:${CMTPROJECTPATH}}"
Next, add these lines to the top of
searchPath.cmake
:
list(INSERT CMAKE_PREFIX_PATH 0 "/group/online/dataflow/cmtuser")
list(INSERT CMAKE_PREFIX_PATH 0 "/group/rich/sw/dev-dir")
Then, edit
searchPath.py
by
carefully changing the
first part of the
path
only, as follows (and leaving the rest of it alone):
path = SearchPath([EnvSearchPathEntry('User_release_area', '/group/rich/sw/cmtuser/'),
to
path = SearchPath([EnvSearchPathEntry('User_release_area', '/group/rich/sw/cmtuser/'), SearchPathEntry('/group/rich/sw/dev-dir'), SearchPathEntry('/group/online/dataflow/cmtuser'),
AND later in the same line, also change:
EnvSearchPathEntry('CMTPROJECTPATH', '/group/online/dataflow/SwData:/group/rich/sw/cmtuser/:
to
EnvSearchPathEntry('CMTPROJECTPATH', '/group/rich/sw/cmtuser/:/group/rich/sw/dev-dir:/group/online/dataflow/SwData:
Our software is not currently stored in
AlignmentOnline
. If we use software from another project and need the other project's environment variables, we must also checkout packages from those projects. We definitely have to check out
Rich/RichMirrCombinFit
and
Rich/RichMirrAlign
, as the Iterator will want to run executables that are produced after we compile these. We think, but are not sure if it is necessary, that we should also check out
Rich/RichMirrorAlignmentOnline
in order for the local software to have access to the Iterator. We definitely need to get
Rec/Brunel
to be able to configure Brunel on the analyzer nodes.
So, to get the code do the following (change the branch from master, if need be):
cd $User_release_area/AlignmentOnlineDev_v12r1
git lb-use AlignmentOnline
git lb-use Panoptes
git lb-use Brunel
git fetch --all
git lb-checkout Panoptes/master Rich/RichMirrorAlignmentOnline
git lb-checkout Panoptes/masterRich/RichMirrAlign
git lb-checkout Panoptes/master Rich/RichMirrCombinFit
git lb-checkout Brunel/2018-patches Rec/Brunel
* NOTA BENE: When we get
Brunel
for now we must get it from
2018-patches
*
You may possibly want to use a tag or your own branch instead of the master branch, if you are testing code before you make a merge (to master) request.
Next,
cp ../AlignmentOnlineDev_v12r0/multiUserCompile.sh ./multiUserCompile.sh
OR create a new file called
multiUserCompile.sh
and copy in the contents of
multiUserCompile.sh. Make sure to change the version of
Online
to the current version (in this example v7r0).
Also,
cp ../AlignmentOnlineDev_v12r0/optCompileOnly.sh ./optCompileOnly.sh
OR create a new file called
optCompileOnly.sh
and copy in the contents of
optCompileOnly.sh. In this file make sure all of the lines starting with
git lb-checkout
are commented out for now, because you have already checked these out these packages.
Note in these last two lines, I am asking you to copy these
.sh
files from the
previous working version of
AlignmentOnline!
You need to
temporarily change the line in
./optCompileOnly.sh
from
source multiUserCompile.sh OPTONLY 2>&1 | tee compile.log
to
source multiUserCompile.sh 2>&1 | tee compile.log
Then compile!
source optCompileOnly.sh
(this command can help you organize lb-checkouts, and takes all standard and error output from multiUserCompile.sh and puts it into compile.log)
Hopefully you have no errors. If you do have errors check
compile.log
and try to solve the problem.
Common issues at this point
- "heptools" versions don't match between
AlignmentOnline
and Online
projects *Ask someone from Alignment group if they are aware of this, they may have to release a new version of AlignmentOnline
- Could not find a package configuration file provided by "=OnlineDev="
- This is likely because the compiler version you are using does not have a build made.
- echo $CMTOPT will tell you the version that you are compiling with=
- Check
ls /group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/
for the version of Online
that you are using, if there is no build matching the version that you are compiling with, then this is the problem. Edit your multiUserCompile.sh
file to use a compiler version that exists in the /group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/
directory.
- slc6 builds should also be compatible with centos7 machines, but not the other way around.
- for the mirror alignment, we now really only want to use centos7 (i.e. log into pluscc and not plus)
If you had no errors, you may change the line
source multiUserCompile.sh 2>&1 | tee compile.log
in
./optCompileOnly.sh
back to
source multiUserCompile.sh OPTONLY 2>&1 | tee compile.log
Then ONCE and only ONCE: run
source /group/online/dataflow/scripts/shell_macros.sh
cmsetup
Three files ending in
.vars
will be generated.
In all three files, Change the three
TCK lines which look something like this:
export HLTTCKROOT="${SW_LHCB}/DBASE/TCK/HltTCK/v3r18";
export L0TCKROOT="${SW_LHCB}/DBASE/TCK/L0TCK/v5r14";
export L0TCK="${SW_LHCB}/DBASE/TCK/L0TCK/v5r14/options";
to
export HLTTCKROOT="/group/online/dataflow/SwData/DBASE/TCK/HltTCK/v3r999";
export L0TCKROOT="/group/online/dataflow/SwData/DBASE/TCK/L0TCK/v5r999";
export L0TCK="/group/online/dataflow/SwData/DBASE/TCK/L0TCK/v5r999/options";
OR add these last three lines just before the final line of each
.vars
file.
After this, you may now make this version of
AlignmentOnlineDev
our release for operations by changing the softlink to
AlignmentRelease
as follows:
cd ..
rm AlignmentRelease
with no trailing forward slash!
ln -s AlignmentOnlineDev_v12r1 AlignmentRelease
now,
carefully edit
/group/online/dataflow/cmtuser/OnlineDev_v7r0/Online/FarmConfig/job/AligDrv.sh
(use the correct Online version of course)
There are
two lines that start with
. /group/rich/sw/cmtuser
and look like this:
. /group/rich/sw/cmtuser/AlignmentRelease/setup.x86_64-centos7-gcc62-opt.vars
change them
both so that you are definitely pointing to the correct compiler version. (i.e. these should now point to files that exist)
Lastly,
carefully edit
/group/online/dataflow/cmtuser/OnlineDev_v7r0/Online/FarmConfig/job/AligWrk.sh
(use the correct Online version of course)
There are
two lines that start with
. /group/rich/sw/cmtuser
and look like this:
. /group/rich/sw/cmtuser/AlignmentRelease/setup.x86_64-centos7-gcc62-opt.vars
change them
both so that you are definitely pointing to the correct compiler version.
...and you're done!
Though there may be other issues...
- In 2018, using Brunel
master
instead of 2018-patches
will cause the node configuration to fail, as the configurable Brunel.RichSequences
does not exist in the upgrade.
- Sometimes there are missing dependencies in the version of LCG needed by Roel's framework to publish plots in the presenter. To solve it, add those dependencies by hand in the python path. Add this line just before the final line in all of the above-mentioned
.vars
files:
export PYTHONPATH=$PYTHONPATH:/cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_93/chardet/3.0.4/x86_64-centos7-gcc62-opt/lib/python2.7/site-packages:/cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_93/idna/2.6/x86_64-centos7-gcc62-opt/lib/python2.7/site-packages;
Standard compiling and installing of code
Usually once a new release of AlignmentOnline is installed and soft linked to
AlignmentRelease
, we enter a long period of standard operations, where at the pit we only re- lb-checkout pieces of code from the git repository that need updating.
Say you want to install a new version of the code from the master branch, or test code from one of your branches. Then go into the
AlignmentRelease
directory. All the initial git stuff is set up, so all you need to do to bring a new piece of code in is:
git fetch --all
git lb-checkout
{project name}/{branch name} {full package name}
- example for {project name}/{branch name} {full package name} is
Panoptes/master Rich/RichMirrorAlignmentOnline
- If for some reason this doesn't work, and you are sure you got the {project name}/{branch name} {full package name} part correct, then try again replacing
git fetch --all
with git lb-use Panoptes
(if you don't want to bring in something from Project Panoptes
, then change it to Rec
, etc...).
Then perform the following in the
AlignmentRelease
directory:
source optCompileOnly.sh
Make sure you always check the contents before running
optCompileOnly.sh
in the
AlignmentRelease
area! (And feel free to edit it on
plus
if you need to achieve a different result)
- The main point here is that you do not want this script to lb-checkout things that haven't changed, which will save time.
That's it! Note that this script only compiles the OPT version and not the two debugger versions DBG and DO0. You don't need the latter two versions unless you need to do deep debugging of the mirror alignment. If you do need them,
mv optCompileOnly.sh CompileAll.sh
and remove the word
OPTONLY
from
CompileAll.sh
.
Then
source CompileAll.sh
Running the mirror alignment worker task on a single HLT Farm node
Sometimes when trying to solve an error with the mirror alignment, it is not enough to look at the error logs as discussed in
LHCbRichMirrorAlignRunOnline, especially if the problem has to do with our configuration of the analyzers.
To run our worker task on a node to test the processing of data on just one of the analyzers:
Make sure our code is checked out and compiled, e.g.:
cd /group/rich/sw/cmtuser/AlignmentOnlineDev_v12r1/ # change v12r1 to whatever version we are installed in!!!
source optCompileOnly.sh
Make sure you always check the contents before running
optCompileOnly.sh
in the
AlignmentRelease
area, to make sure you are checking out code from the correct branches! (And feel free to edit it on
plus
if you need to achieve a different result)
Next, via
ssh
, log into a HLT farm machine from
lbgw
, e.g.
hltf1119
There is a file called
/group/online/dataflow/options/LHCbA/HLT/OnlineEnvBase.py
which basically gets filled with all the useful info that you would normally set up in the FSM. So you can configure and set up the alignment through the FSM, then copy this file locally, or just grab a version of this someone else copied and edit it yourself if need be.
For the latter situation, I copied a
Rich1Alignment version of this file to:
/home/pnaik/singleNodeTest/OnlineEnvBase.py
which you could use.
In whatever directory your
OnlineEnvBase.py
is in (for me,
/home/pnaik/singleNodeTest/
) make a file called
singleNodeTest_RICH1.py
In this file place the following lines:
# copied straight from the exec task that is usually run in AligWrk.sh
import Gaudi,GaudiKernel.ProcessJobOptions
from Gaudi.Configuration import importOptions
GaudiKernel.ProcessJobOptions.printing_level=999
from PyMirrAlignOnline import RichAnalyzer
RichAnalyzer.doIt(filename = '/localdisk/Alignment/Rich', whichRich = 1) # Make sure whichRich matches your OnlineEnvBase.py!
# the extra code (from Roel) that actually steers the job
from GaudiPython.Bindings import AppMgr
gaudi = AppMgr()
gaudi.run(1000) # change the number of events if you want to run over more!
gaudi.stop()
gaudi.finalize()
You then run the following in your shell:
. /group/rich/sw/cmtuser/AlignmentRelease/setup.x86_64-centos7-gcc62-opt.vars # i.e. as in AligWrk.sh, get the environment for the mirror alignment
export PYTHONPATH=/group/online/dataflow/options/LHCbA/HLT:$PYTHONPATH # i.e. as in AligWrk.sh
python singleNodeTest_RICH1.py /group/online/dataflow/templates/options/Main.opts # basically, we run the python that would have been run via the ECS
If you do this, you see the printout is a lot more detailed than you would observe in the
LHCbA log.
If you want to make a change to the Analyzer Brunel configuration, edit:
/group/rich/sw/cmtuser/AlignmentOnlineDev_v12r1/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/RichAnalyzer.py
and then recompile. The simplest way to recompile (if you are using an fully established version of
AlignmentOnline [currently the case for v12r1]) is to run these commands in your bash shell:
cd /group/rich/sw/cmtuser/AlignmentOnlineDev_v12r1/
source optCompileOnly.sh
Make sure you always check the contents before running
optCompileOnly.sh
in the
AlignmentRelease
area! (And feel free to edit it on
plus
if you need to achieve a different result)
Useful scripts
PLEASE KEEP THESE UP TO DATE
multiUserCompile.sh
Here is the script (call it
multiUserCompile.sh
) you should put in /group/rich/sw/cmtuser/AlignmentRelease to automate the moving of compile files created by others and to automate the compiling of the configuration.
Make sure you change the Online version to the correct one for the version of AlignmentOnlineDev
that the softlink AlignmentRelease
is pointing to. Also make sure that you are using the correct
compiler version for the machine that you are on.
With the current version of this script you need to be compiling on a
centos7 plus (e.g.
ssh -Y pluscc
) machine. You may also compile on an
SLC6 plus machine but then you have to change centos7 to slc6 everywhere below, however now we should just stick to centos7 until the next big linux upgrade.
We use this script, because currently when you compile the code, the permissions to the files are owned only by you... this causes problems when others wish to recompile the code, due to a lack of overwrite permissions. This script moves other people's old compile files out of the way so that you can compile. This script also performs all of the compiling.
echo "Usage: Execute this script using 'source multiUserCompile.sh' to compile all versions and 'source multiUserCompile.sh OPTONLY' to compile only the OPT version."
echo "Positional Parameters"
echo '$0 = ' $0
echo '$1 = ' $1
echo '$2 = ' $2
echo '$3 = ' $3
echo "Now enforcing the use of centos7 gcc62 to these compiler versions (you should be compiling on a Centos7 machine):"
export MIRROPT=x86_64-centos7-gcc62-opt
export MIRRDBG=x86_64-centos7-gcc62-dbg
export MIRRDO0=x86_64-centos7-gcc62-do0
echo "opt: $MIRROPT"
echo "dbg: $MIRRDBG"
echo "db0: $MIRRDO0"
echo " "
echo "Moving all past build files to ./pastBuildFiles, if they exist."
if [ ! -d ./pastBuildFiles ]; then
mkdir -p ./pastBuildFiles
fi
if [ -d ./InstallArea ]; then
mv ./InstallArea pastBuildFiles/InstallArea.$(date +%s)
fi
if [ -d ./build.$MIRROPT ]; then
mv ./build.$MIRROPT pastBuildFiles/build.$MIRROPT.$(date +%s)
fi
if [ -d ./build.$MIRRDBG ]; then
mv ./build.$MIRRDBG pastBuildFiles/build.$MIRRDBG.$(date +%s)
fi
if [ -d ./build.$MIRRDO0 ]; then
mv ./build.$MIRRDO0 pastBuildFiles/build.$MIRRDO0.$(date +%s)
fi
echo "Deleting ./pastBuildFiles if we have the right permissions."
echo " (if not, that's fine, and the error messages are currently directed to /dev/null)"
rm -rf ./pastBuildFiles 2> /dev/null
echo " "
echo "Making OPT $MIRROPT."
LbLogin -c $MIRROPT
export OnlineDev_DIR=/group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/$MIRROPT
echo $OnlineDev_DIR
#env > envOPT.txt
make clean
make purge
make configure
make -j 8 install
if [ "$1" != "OPTONLY" ]; then
echo "Positional parameter 1 contains something besides OPTONLY."
echo "Compiling DBG and DB0."
echo "Making DBG $MIRRDBG."
LbLogin -c $MIRRDBG
export OnlineDev_DIR=/group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/$MIRRDBG
echo $OnlineDev_DIR
#env > envDBG.txt
make clean
make purge
make configure
make -j 8 install
echo " "
echo "Making DO0 $MIRRDO0."
LbLogin -c $MIRRDO0
export OnlineDev_DIR=/group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/$MIRRDO0
echo $OnlineDev_DIR
#env > envDO0.txt
make clean
make purge
make configure
make -j 8 install
else
echo "Positional parameter 1 contains OPTONLY."
echo "Not Compiling DBG and DB0, to produce them remove OPTONLY when calling this script."
fi
echo " "
echo "Compiling complete."
optCompileOnly.sh
git fetch --all # making this "git fetch Panoptes" can be a little faster if you are not touching any files from other projects
#git fetch Panoptes
# SELECT BRANCH TO USE FOR MIRRORALIGNMENTONLINE
# (leave commented if you wish to keep the code release already checked out fixed at the pit)
#git lb-checkout Panoptes/master Rich/RichMirrorAlignmentOnline
#git lb-checkout Panoptes/pnaik-MirrAlign-20170719 Rich/RichMirrorAlignmentOnline
# SELECT BRANCH TO USE FOR MIRRALIGN
# (leave commented if you wish to keep the code release already checked out fixed at the pit)
#git lb-checkout Panoptes/master Rich/RichMirrAlign
# SELECT BRANCH TO USE FOR MIRRCOMBINFIT
# (leave commented if you wish to keep the code release already checked out fixed at the pit)
#git lb-checkout Panoptes/master Rich/RichMirrCombinFit
source multiUserCompile.sh OPTONLY 2>&1 | tee compile.log
Back to main Mirror alignment TWiki