LHCb Web>LHCbComputing>RichSoftware>RichSoftwareCalib>LHCbRichMirrorAlign>LHCbRichMirrorAlignCodeOnline (2019-08-30, ParasNaik)

EditAttachPDF

-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18

Code for the Online RICH mirror alignment

RICH mirror alignment project and package overview
The RichMirrorAlignmentOnline package
Location and operation of the XML files [IMPORTANT]
Other interesting places
Fresh install of code for new AlignmentOnline version
Standard compiling and installing of code
Running the mirror alignment worker task on a single HLT Farm node
Useful scripts
- multiUserCompile.sh
- optCompileOnly.sh

RICH mirror alignment project and package overview

The alignment package for the online RICH mirror alignment (Rich/RichMirrorAlignmentOnline) is always kept and sourced from the LHCb project Panoptes git repository.

code currently in the master branch of Panoptes: https://gitlab.cern.ch/lhcb/Panoptes/tree/master/Rich/RichMirrorAlignmentOnline

At the pit, this package and others are drawn into our development directory of AlignmentOnline, in this directory here: /group/rich/sw/cmtuser/AlignmentRelease

NOTA BENE: AlignmentRelease is a soft link to /group/rich/sw/cmtuser/AlignmentOnlineDev_vYYrX
- vYYrX when referenced in this TWiki should always be replaced with numbers for the current revision
The current version being used by us online is v12r1. Experts, always keep this line updated.
- Also, update the soft link if the version changes!

This package relies on the projects:

AlignmentOnline
- This package relies on the Online project
Panoptes whose packages are used as plug-ins
- This includes Rich/RichMirrCombinFit, the key package for fitting the reconstructed the RICH photons
- This includes Rich/RichMirrAlign, the package that performs the minimization required to determine the best alignment
Rec whose packages are used as plug-ins
- This includes Rec/Brunel, the key package for the reconstructing the RICH photons

As you can see, the AlignmentOnline project relies on a lot to run our code. It is important then that we strictly follow the code workflows described in LHCbRichMirrorAlignGitInfo and this document.

This TWiki page specifically discusses the Rich/RichMirrorAlignmentOnline package, and its special relationship to the AlignmentOnline project, that has been (or can be) installed at the pit.

More information on specific packages of interest and other projects is available on the main LHCbRichMirrorAlign page.

The RichMirrorAlignmentOnline package

This package contains the "Iterator" and it's "Helper files", the "Analyzer", the Configuration for the alignment and the MirrPauseHandler. It also contains doc/release.notes with a full history of the code.

Most recent overview talk

Key Definitions

Iterator = The script that drives the mirror alignment

Analyzer = The code that configures Brunel

analyzers = The HLT farm (a.k.a. worker) nodes that we use to process the data that comes from our RICH Mirrors HLT1 line

Configuration = The file which lists sets current state of every configurable option in the mirror alignment.

The Iterator (Iterator.py)

The Iterator drives the Rich mirror alignment. It sets up all necessary files, writes the xml files, puts them in the right place where the analyzers can pick it up for the next iteration, performs the fits an decides if the alignment has converged.

Everything in the iterator is steered by the run(whichRich) method, while most functionalities are outsourced into classes and methods.

whichRich is provided by the choosing Rich1 or Rich2 in the run-control while running the alignment.
The configuration is read in and variables are set.
The while loop begins.
The very first minor iteration is separate because the different command that will come from the run control, the rest of the iterations will be in the same place.
At the start of the first iteration the Work-directory will get emptied out.
The starting xml-file is gotten from '/group/online/alignment/Rich' + str(whichRich) + '/MirrorAlign/'
The xml-files for the mirror-tilts for the magnification factors are made (unless we are running in predetermined magnification factors mode).
The xml-file for the first iteration put in the place where the analyzers will pick it up.
Set Iterator state to ready which will make the analyzers run their jobs.
When the analyzers are ready the iterator will get the command "pause" and will check if one major iteration has been finished already:
- if not then it just copied the xml file for the next iteration to the place where the analyzers can pick it up
- if yes it will perform the fits and the alignment and check weather or not is has converged. If yes it's finished, if no it starts again at 7.

Iterator code (well documented) currently in the master branch of Panoptes: https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/Iterator.py

Here a beautiful sketch of what the iterator does:

The Helper Files (*.py)

There are many helper files that contains classes that outsource all of the tasks that the Iterator needs to perform. These classes are contained within various .py files located in the same directory as the RichAnalyzer and the Iterator. There is usually one class per file, except that SaveSetRetriever and Histocopier are both in HistoHelper.py. The classes contained within these helper files are described below:

RichAlignmentHelper class
• The methods in this file are used to run RichMirrCombinFit and RichMirrAlign
• checkFitQual just returns whether all bins of the deltaTheta vs. phi histogram have enough photons
• hasConverged checks to see if RichMirrAlign thinks the alignment has converged

XMLFileHelper class
• This helper manipulates the XML files and helps prepare them for consumption by the Online DB or by the Brunel Configuration.
• It also prepares the final XML file.

tiltObj class
• This is Claire’s very nice python object that can store the difference between one alignment and another.

AlignMonitor class
• The methods in this file are used to perform the monitoring, and also to fit the deltaTheta plot that is shown in the final PDF summary file.
• It relies heavily on methods from tiltObj.py

UpdateHelper class
• This is the “DIM” service that we launch when our alignment starts.When we decide what mirror alignment we wish to apply to this fill, we use the updateDB method to instruct Clara (via a program she wrote) to whether to use the existing XML as our alignment for this fill, or an new one.

SetupHelper class
• SetupHelper determines the alignment name (i.e. the eventual directory name the working directory will be copied to when done). It also launches the monitoring, finalizes the alignment when finishes, and writes summary.txt, emails, and ELOG entries.

trendHelper class
• trendHelper makes the long-term alignment trend plots. It can be run on its own manually, but also runs automatically with each automated mirror alignment.

fileFinder class
• Don’t worry too much about fileFinder.py
• It allows the trendHelper to find the historical XML files it needs to make the trend plots.
• You’ll probably never touch it.

Communicator class
• Don’t worry too much about Communicator.py
• It allows the Iterator to communicate with the run-control.  The key method is get_command()
• You’ll probably never touch it.

HistoHelper class
• Don’t worry too much about HistoHelper.py
• It manages the retrieval of the SaveSets (histograms) from the analyzers. SaveSetRetriever class:
The SaveSets are not written by the Iterator but a very different part of the program. In order to know where to retrieve the save sets from, a "DIM" server method is used to retrieve the location where the histogram was written in real-time. Histocopier class:
Encapsulates waiting for the SaveSets being written, matching it to the correct iteration and copying it to the desired location.
• You'll probably never touch it.

These pieces of code as currently in the master branch of Panoptes: https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/

The Analyzer (RichAnalyzer.py)

The Analyzer is the entity that will configure the analyzers and execute the Brunel jobs and produce the histogram files.

The Analyzer code only contains the options for the Brunel jobs which will be read in the during the "configuration".
During run-time, the analyzers will pick up the rich mirror xml file for the reconstruction from the exact same place every time.

Analyzer code currently in the master branch of Panoptes: https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/RichAnalyzer.py

The Configuration file (Configuration.py)

The Configuration file contains a lot of different settings for the alignment for Rich1 and Rich2 separately (this might be done smarter, but later). Some things in there should not be changed because they are part of the online-setup.

The Configurations will be read in by both the Iterator and the Analyzer.

Configuration file code currently in the master branch of Panoptes: https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/python/RichMirrorAlignmentOnline/Configuration.py

The MirrPauseHandler (MirrPauseHandler.cpp)

The MirrPauseHandler contains the code that is executed when the analyzers go to 'paused' - which is after each iteration. It specifies (to a certain degree) when and where the histogram files are written. This code does (most probably) not need to be changed anymore.

MirrPauseHandler code currently in the master branch of Panoptes: https://gitlab.cern.ch/lhcb/Panoptes/blob/master/Rich/RichMirrorAlignmentOnline/src/MirrPauseHandler.cpp

Location and operation of the XML files [IMPORTANT]

The XML file the alignment is starting from is being picked up from here:

/group/online/alignment/Rich1/MirrorAlign/
/group/online/alignment/Rich2/MirrorAlign/

These XML files are colloquially called Vs (pronounced vees). When there is a new alignment, the Iterator produces a new V. Thus a particular V is simply a particular version of the mirror alignment compensations.

The Iterator will pick the XML file with the letter "=v=", followed by the biggest version number, followed by "=.xml=".
If you want to run starting with a different XML file, there's an option in the Configuration.py file.

The XML files in the above directories should not have a header, therefore when getting a starting XML file the header is being added by the Iterator, and when the finishing xml-file will be copied there it will have its header removed. If you add your own custom XML file to the list remove the header (= the first three lines; and the little </DDB> at the end).

We will run in one of three modes :

Monitoring mode CURRENT MODE OF OPERATION
- in Configuration.py
  - autoUpdate = False
  - testing = False
- If the mirror alignment decides no updated is needed, the same V remains in place e.g. v{N}.xml
- If the magnet polarity changed, for RICH1 and RICH2 each, the new V is written into v{N+1}_maybe_YYYYMMDD_HHMMSS.xml where YYYYMMDD_HHMMSS corresponds to the alignment name
  - Consult with lhcb-rich-mirror-alignment-development@cernNOSPAMPLEASE.ch (Claire, Paras, and everyone else doing mirror alignment) about the = maybe = alignment, to make sure we think it's OK.
    - If it is agreed, copy v{N+1}_maybe_YYYYMMDD_HHMMSS.xml to v{N+1}.xml
    - Send a group email with the new v{N+1} for each RICH to:
      - lhcb-rich-mirror-alignment-development@cernNOSPAMPLEASE.ch (Claire, Paras, and everyone else doing mirror alignment)
      - Clara.Gaspar@cernNOSPAMPLEASE.ch (Clara Gaspar, Online expert)
      - lhcb-hlt-piquet@cernNOSPAMPLEASE.ch (All HLT piquets)
- If the mirror alignment decides an update is needed, the new V is written into v{N+1}_maybe_YYYYMMDD_HHMMSS.xml where YYYYMMDD_HHMMSS corresponds to the alignment name
  - If any of the alignment parameters is more than 2 times the {primary,secondary} {y,z} tolerance for any mirror combination, consider making an alignment update.
    - When considering making the update, consult with the following people:
      - lhcb-rich-mirror-alignment-development@cernNOSPAMPLEASE.ch (Claire, Paras, and everyone else doing mirror alignment)
      - m.mccann@imperialNOSPAMPLEASE.ac.uk, silvia.gambetta@cernNOSPAMPLEASE.ch, jonesc@hepNOSPAMPLEASE.phy.cam.ac.uk (Mike McCann, Silvia Gambetta, and Chris Jones, from the RICH group)
      - silvia.borghi@cernNOSPAMPLEASE.ch (Silvia Borghi, the OPG coordinator)
        
        If need be (it appears to be a major problem) contact Silvia via CERN phone 160973. http://information-technology.web.cern.ch/book/mobile-operating-instructions/making-call).
    - If it is agreed, copy v{N+1}_maybe_YYYYMMDD_HHMMSS.xml to v{N+1}.xml
      - The update will be picked up on the next fill.
- If for some reason the alignment did not run for one or both of the RICH detectors; instruct the HLT piquets to use the current V.
  - If the problem seems to be with the farm, wait for the next fill and see what happens.
  - If the problem definitely is something in our code, try to fix it.
- Any problems with the mirror alignment running at the pit, that the available mirror alignment experts cannot collectively solve immediately, send an group email to:
  - The Alignment Piquet [see Shift DB for name [look up email in CERN phonebook], CERN phone 16 1487 (from outside CERN: +41 75 411 1487)]
  - Beat.Jost@cernNOSPAMPLEASE.ch, Clara.Gaspar@cernNOSPAMPLEASE.ch, Markus.Frank@cernNOSPAMPLEASE.ch (Online experts)
  - lhcb-rich-mirror-alignment-development@cernNOSPAMPLEASE.ch (Claire, Paras, and everyone else doing mirror alignment)

Automatic mode (for later)
- in Configuration.py
  - autoUpdate = True
  - testing = False
- If the mirror alignment decides no updated is needed, the same V remains in place e.g. v{N}.xml
- If the mirror alignment decides an update is needed, the new V is written into v{N+1}.xml
  - The update will be picked up on the the current fill (the one on which the alignment was performed).

Testing mode (if need be)
- in Configuration.py
  - testing = False (autoUpdate gets turned to False within the code if it happens to be True)
- If the mirror alignment decides no updated is needed, the V is written to v{N+1}_testing_YYYYMMDD_HHMMSS.xml
- If the mirror alignment decides an update is needed, the new V is written into v{N+1}_testing_maybe_YYYYMMDD_HHMMSS.xml
  - The update is not picked up.

Other interesting places

'workDir', the Work-directory

This is the directory in which all the files created during an alignment are being written.
/group/online/AligWork/Rich1
/group/online/AligWork/Rich2

Savesets

Before the histogram files are being copied into the Work-directory they will be placed into
/hist/Savesets/2015/LHCbA/AligWrk_Rich1
/hist/Savesets/2015/LHCbA/AligWrk_Rich2
(check the year)

Driver scripts for the all online alignments

This is the script that holds everything together at the pit. It was made by the online people and we can only touch it if we are VERY careful!

/group/online/dataflow/cmtuser/OnlineDev_v6rX/Online/FarmConfig/job/AligDrv.sh (for the Iterator)
/group/online/dataflow/cmtuser/OnlineDev_v6rX/Online/FarmConfig/job/AligWrk.sh (for the RichAnalyzer configuration of the farm nodes)

where "v6rX" is whatever version of Online is being accessed to run things at the pit (check if unsure!): This is NOT the version AlignmentOnline depends on, but should be the newest version of Online.

Logging staging area

When the alignment runs, we store the log of the machine that the Iterator runs on, which is always hlt02.

The output of hlt02 (including messages that aren't ours) during the last Rich1 alignment is stored in /group/rich/AlignmentFiles/Logging/Rich1_hlt02.log
The output of hlt02 (including messages that aren't ours) during the last Rich2 alignment is stored in /group/rich/AlignmentFiles/Logging/Rich2_hlt02.log

If an alignment is completed properly this is later moved to the output storage area. If not, and the Iterator makes it to the end, it is saved in this directory with a timestamp (you can safely delete old time stamped files, if the mirror alignment is operating OK).

In either of these cases an "hlt02reduced" log is also created, this has a grep applied to it that filters out junk messages that show up on hlt02, that are unlikely to be from our mirror alignment iterator.

If the Iterator does not make it to the end, then it just remains in this directory until the next mirror alignment. However in this case the next alignment's output appends to this file, instead of overwriting it as usual.

At the beginning of an alignment, the current hlt02 logs are copied to "_previous" to preserve the last run.

Some more information about looking at and inspecting these files can be found here: LHCbRichMirrorAlignShiftInfo

Rolling log of Cherenkov angle resolutions

A rolling log of Cherenkov angle resolutions is kept in these files:

/group/rich/AlignmentFiles/Logging/Rich1_CKresTrend.txt
/group/rich/AlignmentFiles/Logging/Rich2_CKresTrend.txt

Do not touch these, unless you're editing them to remove multiple data points (e.g. the alignment was for some reason run twice on the same fill).

You can see plots of this data and comparisons to the resolutions from OnlineBrunel uses the data in these text files to provide a trend of the CK angle resolutions from the mirror alignment.

If testing = True in the Configuration for a particular mirror alignment, then the data is appended to this file with a "# TEST " designation on the line; this line is subsequently ignored.

Rolling log of Mirror Alignment updates

A rolling log of mirror alignment updates is kept in these files:

/group/rich/AlignmentFiles/Logging/Rich1_UpdateTrend.txt
/group/rich/AlignmentFiles/Logging/Rich2_UpdateTrend.txt

Do not touch these, unless you're editing them to remove multiple data points (e.g. the alignment was for some reason run twice on the same fill), OR the mirror alignment was a _maybe alignment that you decide to make the active alignment for a particular fill, in which case you need to adjust this file accordingly.

This file is used by trendHelper.py to automatically determine when there has been a magnet polarity change.

If testing = True in the Configuration for a particular mirror alignment, then the data is appended to this file with a "# TEST " designation on the line; this line is subsequently ignored.

Magnet polarity memory

The polarity of the last mirror alignment that converged, had no sanity problem, and was not a testing alignment, is stored here:

/group/rich/AlignmentFiles/Polarity/previousPolarity_RICH1.txt
/group/rich/AlignmentFiles/Polarity/previousPolarity_RICH2.txt

The code checks to see if the polarity of the current alignment is different from the previous alignment. If autoUpdate = True in the Configuration, then we will force an update of the mirror alignment (unless sanity checks are failed, of course).

Then at the end of the code, the previousPolarity file is overwritten with the current polarity information, unless testing = True.

Only the first line of this text file is actually compared. You may need to manually edit the first line back to indicating the previous polarity, if for some reason the alignment aborts during a polarity change.

Non-default MDCS corrections.

The RICH1MDCS is a system used to correct for distortions of the HPD photocathode images in the LHCb magnetic field.

The final corrections are are in the form of an XML file, which is obtained by the mirror alignment through the CondDB.

In the Configuration, for RICH1 (and for the heck of it, RICH2): xmlMDCS = ""

However, if we want to use a different XML file we need to tell the mirror alignment through the configuration the location of the file

for example, in the Configuration, for RICH1 (and for the heck of it, RICH2): xmlMDCS = "/group/rich/AlignmentFiles/MDCS/Conditions/Rich1/Environment/HPD.xml"

See /group/rich/AlignmentFiles/MDCS/README.txt for details on what you should do if you are asked to use new MDCS corrections at the pit.

CombAndMirrSubsets.txt

The subset of possible mirror combinations that are used for the mirror alignment are contained within the RichMirrorAlignmentOnline package, and stored here at the pit:

/group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/files/Rich1CombAndMirrSubsets.txt
/group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/files/Rich2CombAndMirrSubsets.txt

Do not touch these files.

Magnification Factors

If predetermined magnification factors are to be used (currently the default case), then they will be read from subdirectories which are contained within the RichMirrorAlignmentOnline package, and stored inside these root directories at the pit:

/group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/files/MagnifFactors/RICH1/
/group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/files/MagnifFactors/RICH2/

Do not touch these files directly! If you are asked to add new magnification factors, then add them using the instructions below.

Recalculating Magnification Factors

Change magnFactorsMode from 0 to 2 in the Configuration.py and then recompile.

Run the mirror alignment. The mirror alignment will take 30 minutes per iteration instead of 3 minutes per iteration.

If the mirror alignment completes successfully, find the savedir in summary.txt or the RICH ELOG

If the alignment took N+1 iterations to complete (i0, ..., iN) and ZZ is the last two digits of the year the alignment data was taken, then the new magnification factors are in the files you can see if you go to the savedir and execute the command:

ls *CollisionZZ_*_iN.txt

Change magnFactorsMode back from 2 to 0 in the Configuration.py and then recompile.

Updating Predetermined Magnification Factors

Depending on which RICH detector you wish to update the predetermined magnification factors for, check out the latest version of Panoptes from run2-patches and then go into one of the following directories:

Rich/RichMirrorAlignmentOnline/files/MagnifFactors/RICH1/
Rich/RichMirrorAlignmentOnline/files/MagnifFactors/RICH2/

then make a new directory with the same YYYYMMDD_HHMMSS timestamp as the alignment savedir that contains the recalculated magnification factors:

mkdir YYYYMMDD_HHMMSS
cd YYYYMMDD_HHMMSS

Then copy over the

*CollisionZZ_*_iN.txt

files from the savedir.

The last step is renaming. This is currently painstaking, but shouldn't take you too long.

the files you have will look like Rich1MirrMagnFactors_Mp6Wi8.0Fm5Mm2Sm0_online_Collision18_pri_negYzerZ_i1.txt
you need to change these to files that look like Rich1MirrMagnFactors_pri_negYzerZ_predefined.txt

Then when you are done also copy over summary.txt from the savedir

Lastly you will need to edit Configuration.py and change the magnifDir to point to the full directory path where the new predefined magnification factors are.

As soon as this is all done, commit the changes and push to a new branch.

At the pit, check out Rich/RichMirrorAlignmentOnline from your new branch and then recompile the code.

Ask for a merge request and make sure you merge into run2-patches.

Then ask Jordi if he could kindly also merge those commits into master.

Fresh install of code for new `AlignmentOnline` version

If you are not asked to install a new AlignmentOnline version at the pit, then you may skip this section (though it's worth a read once anyway, to see how the code comes together).

First, add these two lines to the end of your .bashrc:

echo "Check your current gid below. If it is not what you want, use newgrp to change your gid."
id

This will tell you your user ID (id), which group (rich, online, etc..) permissions are currently active (gid), and which groups your account can choose from (groups), in case you need to change your gid.

Here's what you need to do if you want to freshly install the mirror alignment on an online machine (e.g. pluscc, a machine running centos7). The following example is for AlignmentOnline v12r1.

DO NOT FOLLOW THIS EXAMPLE VERBATIM. Only for a newer version of AlignmentOnline, should you do this process, and then substitute the new version for v12r1 below.

echo "I have just set the User_release_area environment variable to:"
export User_release_area=/group/rich/sw/cmtuser/
echo $User_release_area
LbLogin

You could add these lines to the end of your .bashrc if you want.

Next, and this only has to be done once:

cd $User_release_area
lb-dev AlignmentOnline/v12r1
cd AlignmentOnlineDev_v12r1

Then find out what version of Online is newest. For example if it is v7r0, edit CMakeLists.txt and change:

gaudi_project(AlignmentOnlineDev v12r1
               USE AlignmentOnline v12r1)

gaudi_project(AlignmentOnlineDev v12r1
 #              USE AlignmentOnline v12r1)
               USE AlignmentOnline v12r1
                   OnlineDev v7r0 # Paras added this

Next, add these lines to the top of build_env.csh:

if ( $?CMTPROJECTPATH ) then
  setenv CMTPROJECTPATH "/group/online/dataflow/cmtuser:${CMTPROJECTPATH}"
else
  setenv CMTPROJECTPATH "/group/online/dataflow/cmtuser"
endif

if ( $?CMTPROJECTPATH ) then
  setenv CMTPROJECTPATH "/group/rich/sw/dev-dir:${CMTPROJECTPATH}"
else
  setenv CMTPROJECTPATH "/group/rich/sw/dev-dir"
endif

Then, add these lines to the top of build_env.sh:

export CMTPROJECTPATH="/group/online/dataflow/cmtuser${CMTPROJECTPATH:+:${CMTPROJECTPATH}}"
export CMTPROJECTPATH="/group/rich/sw/dev-dir${CMTPROJECTPATH:+:${CMTPROJECTPATH}}"

Next, add these lines to the top of searchPath.cmake:

list(INSERT CMAKE_PREFIX_PATH 0 "/group/online/dataflow/cmtuser") 
list(INSERT CMAKE_PREFIX_PATH 0 "/group/rich/sw/dev-dir")

Then, edit searchPath.py by carefully changing the first part of the path only, as follows (and leaving the rest of it alone):

path = SearchPath([EnvSearchPathEntry('User_release_area', '/group/rich/sw/cmtuser/'),

path = SearchPath([EnvSearchPathEntry('User_release_area', '/group/rich/sw/cmtuser/'), SearchPathEntry('/group/rich/sw/dev-dir'), SearchPathEntry('/group/online/dataflow/cmtuser'),

AND later in the same line, also change:

EnvSearchPathEntry('CMTPROJECTPATH', '/group/online/dataflow/SwData:/group/rich/sw/cmtuser/:

EnvSearchPathEntry('CMTPROJECTPATH', '/group/rich/sw/cmtuser/:/group/rich/sw/dev-dir:/group/online/dataflow/SwData:

Our software is not currently stored in AlignmentOnline. If we use software from another project and need the other project's environment variables, we must also checkout packages from those projects. We definitely have to check out Rich/RichMirrCombinFit and Rich/RichMirrAlign, as the Iterator will want to run executables that are produced after we compile these. We think, but are not sure if it is necessary, that we should also check out Rich/RichMirrorAlignmentOnline in order for the local software to have access to the Iterator. We definitely need to get Rec/Brunel to be able to configure Brunel on the analyzer nodes.

So, to get the code do the following (change the branch from master, if need be):

cd $User_release_area/AlignmentOnlineDev_v12r1
git lb-use AlignmentOnline
git lb-use Panoptes
git lb-use Brunel
git fetch --all
git lb-checkout Panoptes/master Rich/RichMirrorAlignmentOnline 
git lb-checkout Panoptes/masterRich/RichMirrAlign  
git lb-checkout Panoptes/master Rich/RichMirrCombinFit 
git lb-checkout Brunel/2018-patches Rec/Brunel

* NOTA BENE: When we get Brunel for now we must get it from 2018-patches *

You may possibly want to use a tag or your own branch instead of the master branch, if you are testing code before you make a merge (to master) request.

Next, cp ../AlignmentOnlineDev_v12r0/multiUserCompile.sh ./multiUserCompile.sh OR create a new file called multiUserCompile.sh and copy in the contents of multiUserCompile.sh. Make sure to change the version of Online to the current version (in this example v7r0).

Also, cp ../AlignmentOnlineDev_v12r0/optCompileOnly.sh ./optCompileOnly.sh OR create a new file called optCompileOnly.sh and copy in the contents of optCompileOnly.sh. In this file make sure all of the lines starting with git lb-checkout are commented out for now, because you have already checked these out these packages.

Note in these last two lines, I am asking you to copy these .sh files from the previous working version of AlignmentOnline!

You need to temporarily change the line in ./optCompileOnly.sh from

source multiUserCompile.sh OPTONLY 2>&1 | tee compile.log

source multiUserCompile.sh 2>&1 | tee compile.log

Then compile!

source optCompileOnly.sh (this command can help you organize lb-checkouts, and takes all standard and error output from multiUserCompile.sh and puts it into compile.log)

Hopefully you have no errors. If you do have errors check compile.log and try to solve the problem.

Common issues at this point

"heptools" versions don't match between AlignmentOnline and Online projects *Ask someone from Alignment group if they are aware of this, they may have to release a new version of AlignmentOnline
Could not find a package configuration file provided by "=OnlineDev="
- This is likely because the compiler version you are using does not have a build made.
  - echo $CMTOPT will tell you the version that you are compiling with=
- Check ls /group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/ for the version of Online that you are using, if there is no build matching the version that you are compiling with, then this is the problem. Edit your multiUserCompile.sh file to use a compiler version that exists in the /group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/ directory.
  - slc6 builds should also be compatible with centos7 machines, but not the other way around.
  - for the mirror alignment, we now really only want to use centos7 (i.e. log into pluscc and not plus)

If you had no errors, you may change the line

source multiUserCompile.sh 2>&1 | tee compile.log

in ./optCompileOnly.sh back to

source multiUserCompile.sh OPTONLY 2>&1 | tee compile.log

Then ONCE and only ONCE: run

source /group/online/dataflow/scripts/shell_macros.sh
cmsetup

Three files ending in .vars will be generated. In all three files, Change the three TCK lines which look something like this:

export HLTTCKROOT="${SW_LHCB}/DBASE/TCK/HltTCK/v3r18";
export L0TCKROOT="${SW_LHCB}/DBASE/TCK/L0TCK/v5r14";
export L0TCK="${SW_LHCB}/DBASE/TCK/L0TCK/v5r14/options";

export HLTTCKROOT="/group/online/dataflow/SwData/DBASE/TCK/HltTCK/v3r999";
export L0TCKROOT="/group/online/dataflow/SwData/DBASE/TCK/L0TCK/v5r999";
export L0TCK="/group/online/dataflow/SwData/DBASE/TCK/L0TCK/v5r999/options";

OR add these last three lines just before the final line of each .vars file.

After this, you may now make this version of AlignmentOnlineDev our release for operations by changing the softlink to AlignmentRelease as follows:

cd ..

rm AlignmentRelease with no trailing forward slash!

ln -s AlignmentOnlineDev_v12r1 AlignmentRelease

now, carefully edit /group/online/dataflow/cmtuser/OnlineDev_v7r0/Online/FarmConfig/job/AligDrv.sh (use the correct Online version of course)

There are two lines that start with . /group/rich/sw/cmtuser and look like this:

. /group/rich/sw/cmtuser/AlignmentRelease/setup.x86_64-centos7-gcc62-opt.vars

change them both so that you are definitely pointing to the correct compiler version. (i.e. these should now point to files that exist)

Lastly, carefully edit /group/online/dataflow/cmtuser/OnlineDev_v7r0/Online/FarmConfig/job/AligWrk.sh (use the correct Online version of course)

There are two lines that start with . /group/rich/sw/cmtuser and look like this:

. /group/rich/sw/cmtuser/AlignmentRelease/setup.x86_64-centos7-gcc62-opt.vars

change them both so that you are definitely pointing to the correct compiler version.

...and you're done!

Though there may be other issues...

In 2018, using Brunel master instead of 2018-patches will cause the node configuration to fail, as the configurable Brunel.RichSequences does not exist in the upgrade.
Sometimes there are missing dependencies in the version of LCG needed by Roel's framework to publish plots in the presenter. To solve it, add those dependencies by hand in the python path. Add this line just before the final line in all of the above-mentioned .vars files:

export PYTHONPATH=$PYTHONPATH:/cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_93/chardet/3.0.4/x86_64-centos7-gcc62-opt/lib/python2.7/site-packages:/cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_93/idna/2.6/x86_64-centos7-gcc62-opt/lib/python2.7/site-packages;

Standard compiling and installing of code

Usually once a new release of AlignmentOnline is installed and soft linked to AlignmentRelease, we enter a long period of standard operations, where at the pit we only re- lb-checkout pieces of code from the git repository that need updating.

Say you want to install a new version of the code from the master branch, or test code from one of your branches. Then go into the AlignmentRelease directory. All the initial git stuff is set up, so all you need to do to bring a new piece of code in is:

git fetch --all
git lb-checkout {project name}/{branch name} {full package name}

example for {project name}/{branch name} {full package name} is Panoptes/master Rich/RichMirrorAlignmentOnline
If for some reason this doesn't work, and you are sure you got the {project name}/{branch name} {full package name} part correct, then try again replacing git fetch --all with git lb-use Panoptes (if you don't want to bring in something from Project Panoptes, then change it to Rec, etc...).

Then perform the following in the AlignmentRelease directory:

source optCompileOnly.sh

Make sure you always check the contents before running optCompileOnly.sh in the AlignmentRelease area! (And feel free to edit it on plus if you need to achieve a different result)

The main point here is that you do not want this script to lb-checkout things that haven't changed, which will save time.

That's it! Note that this script only compiles the OPT version and not the two debugger versions DBG and DO0. You don't need the latter two versions unless you need to do deep debugging of the mirror alignment. If you do need them,

mv optCompileOnly.sh CompileAll.sh

and remove the word OPTONLY from CompileAll.sh.

Then

source CompileAll.sh

Running the mirror alignment worker task on a single HLT Farm node

Sometimes when trying to solve an error with the mirror alignment, it is not enough to look at the error logs as discussed in LHCbRichMirrorAlignRunOnline, especially if the problem has to do with our configuration of the analyzers.

To run our worker task on a node to test the processing of data on just one of the analyzers:

Make sure our code is checked out and compiled, e.g.:

cd /group/rich/sw/cmtuser/AlignmentOnlineDev_v12r1/         # change v12r1 to whatever version we are installed in!!!
source optCompileOnly.sh

Make sure you always check the contents before running optCompileOnly.sh in the AlignmentRelease area, to make sure you are checking out code from the correct branches! (And feel free to edit it on plus if you need to achieve a different result)

Next, via ssh, log into a HLT farm machine from lbgw, e.g. hltf1119

There is a file called /group/online/dataflow/options/LHCbA/HLT/OnlineEnvBase.py which basically gets filled with all the useful info that you would normally set up in the FSM. So you can configure and set up the alignment through the FSM, then copy this file locally, or just grab a version of this someone else copied and edit it yourself if need be.

For the latter situation, I copied a Rich1Alignment version of this file to:

/home/pnaik/singleNodeTest/OnlineEnvBase.py

which you could use.

In whatever directory your OnlineEnvBase.py is in (for me, /home/pnaik/singleNodeTest/) make a file called singleNodeTest_RICH1.py

In this file place the following lines:

# copied straight from the exec task that is usually run in AligWrk.sh
import Gaudi,GaudiKernel.ProcessJobOptions
from Gaudi.Configuration import importOptions
GaudiKernel.ProcessJobOptions.printing_level=999
from PyMirrAlignOnline import RichAnalyzer
RichAnalyzer.doIt(filename = '/localdisk/Alignment/Rich', whichRich = 1)    # Make sure whichRich matches your OnlineEnvBase.py!
# the extra code (from Roel) that actually steers the job
from GaudiPython.Bindings import AppMgr
gaudi = AppMgr()
gaudi.run(1000)                                                                                             # change the number of events if you want to run over more!
gaudi.stop()
gaudi.finalize()

You then run the following in your shell:

. /group/rich/sw/cmtuser/AlignmentRelease/setup.x86_64-centos7-gcc62-opt.vars        # i.e. as in AligWrk.sh, get the environment for the mirror alignment
export PYTHONPATH=/group/online/dataflow/options/LHCbA/HLT:$PYTHONPATH      # i.e. as in AligWrk.sh
python singleNodeTest_RICH1.py /group/online/dataflow/templates/options/Main.opts  # basically, we run the python that would have been run via the ECS

If you do this, you see the printout is a lot more detailed than you would observe in the LHCbA log.

If you want to make a change to the Analyzer Brunel configuration, edit:

/group/rich/sw/cmtuser/AlignmentOnlineDev_v12r1/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/RichAnalyzer.py

and then recompile. The simplest way to recompile (if you are using an fully established version of AlignmentOnline [currently the case for v12r1]) is to run these commands in your bash shell:

cd /group/rich/sw/cmtuser/AlignmentOnlineDev_v12r1/
source optCompileOnly.sh

Make sure you always check the contents before running optCompileOnly.sh in the AlignmentRelease area! (And feel free to edit it on plus if you need to achieve a different result)

Useful scripts

PLEASE KEEP THESE UP TO DATE

multiUserCompile.sh

Here is the script (call it multiUserCompile.sh) you should put in /group/rich/sw/cmtuser/AlignmentRelease to automate the moving of compile files created by others and to automate the compiling of the configuration. Make sure you change the Online version to the correct one for the version of AlignmentOnlineDev that the softlink AlignmentRelease is pointing to. Also make sure that you are using the correct compiler version for the machine that you are on.

With the current version of this script you need to be compiling on a centos7 plus (e.g. ssh -Y pluscc) machine. You may also compile on an SLC6 plus machine but then you have to change centos7 to slc6 everywhere below, however now we should just stick to centos7 until the next big linux upgrade.

We use this script, because currently when you compile the code, the permissions to the files are owned only by you... this causes problems when others wish to recompile the code, due to a lack of overwrite permissions. This script moves other people's old compile files out of the way so that you can compile. This script also performs all of the compiling.

Show...

Hide

echo "Usage: Execute this script using 'source multiUserCompile.sh' to compile all versions and 'source multiUserCompile.sh OPTONLY' to compile only the OPT version." 

echo "Positional Parameters"
echo '$0 = ' $0
echo '$1 = ' $1
echo '$2 = ' $2
echo '$3 = ' $3

echo "Now enforcing the use of centos7 gcc62 to these compiler versions (you should be compiling on a Centos7 machine):"
export MIRROPT=x86_64-centos7-gcc62-opt
export MIRRDBG=x86_64-centos7-gcc62-dbg
export MIRRDO0=x86_64-centos7-gcc62-do0
echo "opt: $MIRROPT"
echo "dbg: $MIRRDBG"
echo "db0: $MIRRDO0"
echo " "

echo "Moving all past build files to ./pastBuildFiles, if they exist."
if [ ! -d ./pastBuildFiles ]; then
    mkdir -p ./pastBuildFiles
fi
if [ -d ./InstallArea ]; then
    mv ./InstallArea pastBuildFiles/InstallArea.$(date +%s)
fi
if [ -d ./build.$MIRROPT ]; then
    mv ./build.$MIRROPT pastBuildFiles/build.$MIRROPT.$(date +%s)
fi
if [ -d ./build.$MIRRDBG ]; then
    mv ./build.$MIRRDBG pastBuildFiles/build.$MIRRDBG.$(date +%s)
fi
if [ -d ./build.$MIRRDO0 ]; then
    mv ./build.$MIRRDO0 pastBuildFiles/build.$MIRRDO0.$(date +%s)
fi
echo "Deleting ./pastBuildFiles if we have the right permissions."
echo "  (if not, that's fine, and the error messages are currently directed to /dev/null)"
rm -rf ./pastBuildFiles 2> /dev/null
echo " "

echo "Making OPT $MIRROPT."
LbLogin -c $MIRROPT
export OnlineDev_DIR=/group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/$MIRROPT
echo $OnlineDev_DIR
#env > envOPT.txt
make clean
make purge
make configure
make -j 8 install

if [ "$1" != "OPTONLY" ]; then
    echo "Positional parameter 1 contains something besides OPTONLY."
    echo "Compiling DBG and DB0."
    echo "Making DBG $MIRRDBG."
    LbLogin -c $MIRRDBG
    export OnlineDev_DIR=/group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/$MIRRDBG
    echo $OnlineDev_DIR
    #env > envDBG.txt
    make clean
    make purge
    make configure
    make -j 8 install
    echo " "
    echo "Making DO0 $MIRRDO0."
    LbLogin -c $MIRRDO0
    export OnlineDev_DIR=/group/online/dataflow/cmtuser/OnlineDev_v7r0/InstallArea/$MIRRDO0
    echo $OnlineDev_DIR 
    #env > envDO0.txt
    make clean
    make purge
    make configure
    make -j 8 install
else
    echo "Positional parameter 1 contains OPTONLY."
    echo "Not Compiling DBG and DB0, to produce them remove OPTONLY when calling this script."
fi

echo " "
echo "Compiling complete."

optCompileOnly.sh

git fetch --all # making this "git fetch Panoptes" can be a little faster if you are not touching any files from other projects
#git fetch Panoptes

# SELECT BRANCH TO USE FOR MIRRORALIGNMENTONLINE
# (leave commented if you wish to keep the code release already checked out fixed at the pit)

#git lb-checkout Panoptes/master Rich/RichMirrorAlignmentOnline
#git lb-checkout Panoptes/pnaik-MirrAlign-20170719 Rich/RichMirrorAlignmentOnline

# SELECT BRANCH TO USE FOR MIRRALIGN
# (leave commented if you wish to keep the code release already checked out fixed at the pit)

#git lb-checkout Panoptes/master Rich/RichMirrAlign

# SELECT BRANCH TO USE FOR MIRRCOMBINFIT
# (leave commented if you wish to keep the code release already checked out fixed at the pit)

#git lb-checkout Panoptes/master Rich/RichMirrCombinFit

source multiUserCompile.sh OPTONLY 2>&1 | tee compile.log