-- ParasNaik - 2014-10-24 -- AnatolySolomin - 2009-07-16



RICH mirror alignment monitoring via software by the use of real data (Online, in real-time)


History, ideas, and implementations

The original idea of a "soft" alignment of RICH mirrors dates back to Alignment of the HERA-B RICH optical system with data. The approach and first results of the development of such a system at LHCb were done and published by Antonis Papanestis in LHCb-2001-141 , succeeded by several follow-ups, e.g. ( Antonis, June '05).

Anatoly Solomin later started to work on an automated facility which utilizes the approach described. This facility is built upon two packages hosted by the Panoptes project:

  • Rich/RichMirrCombinFit - finds out the total mirror tilts resulting from misalignments of both spherical and flat mirror segments,
  • Rich/RichMirrAlign - finds misalignment corrections for every individual mirror segment of both families (and outputs the XML version of the mirror alignment database "slice").

Originally, a Ganga-based automation was chosen and implemented. This above code was thus driven by Rich/RichMirrorAlignmentGanga , a do-everything steering Ganga script.

We were asked for Run II if it was possible to perform our alignment in 'real-time', by specially selecting events to be used for alignment and then processing them. Roughly speaking, without [various kinds of] optimizations, the procedure would require several thousand cores to be completed in time. Fortunately, we were offered the use of ~1700 nodes of the HLT (a.k.a "alignment") farm during Run II to perform the processing of those selected events. Still, the amount of time that we would be allowed to use the farm to process the data we use for the alignment would be very limited. Therefore, performance of the facility was a top priority.

Paras Naik and Anatoly fully upgraded the mirror alignment code in preparation for LHC Run II, also incorporating modifications and several improvements by Matt Coombes and Sam Harnew.

Paras also worked on the development of an HLT1 line to obtain the data we need to perform the mirror alignment in real-time.

Paras Naik and Claire Prouvé have now turned the Ganga-based automation into Online monitoring. This is also hosted in Panoptes:

  • Rich/RichMirrorAlignmentOnline

... but really could just as well have been a part of AlignmentOnline.

Claire's excellent work has been recognized with an LHCb Early Career Scientist Award: https://cds.cern.ch/journal/CERNBulletin/2016/38/News%20Articles/2215950

This work allowed us, from the start of Run II, to perform the alignment between the fills, i.e. it took only a matter of minutes (typically) to complete the procedure.

In 2017, Paras continued serious work on the mirror alignment again in earnest, implementing many improvements and bug fixes, including implementation of code to take advantage of a move to the Online database (allowing more frequent updates), and for the first time making the alignment of the LHCb RICH mirrors fully automated in both performance and decision-making. Also an active monitoring system was added to the mirror alignment, making it straightforward for Data Managers, RICH piquets, Alignment piquets, and RICH mirror alignment experts to identify any issues and resolve them.

Thought process on the evolution of the HLT line

Anatoly Solomin, Tom Hampson and Andrew Cook have worked on optimization in the past. Tom researched and implemented a selection method, proposed by Anatoly, that would filter events most likely to be useful for the RICH mirror alignment. The aim is to populate the outer mirrors of the RICH, in order to fill properly the phi (azimuthal angle) bins of a photon Cherenkov angle histogram [at least 16 of 20 bins must have minimum 300 photons for the mirror alignment procedure to work] for every mirror combination we use for alignment. But at the same time optimization is aimed at reducing redundant photon reconstruction (Brunel level) for more easily populatable central mirrors, as the reconstruction takes up most of the time. This was worked on further by Sam Harnew (though this work never became an official part of our offline code).

For maximal optimization Anatoly came up with the method of selecting at HLT level the events with tracks, whose photons are most likely to populate the histograms optimally during the subsequent off-line reconstruction. This requires preparation of a map for coordinates of such tracks in advance. Andrew made detailed study and worked out a first version of such a map to be used in HLT to identify these events and potentially select only the tracks that we need for the alignment. This was ideally to be implemented for Run 2.

The HLT map idea could still be implemented, but due to severe time constraints for Run II we leveraged Andrew's work by proceeding with an HLT line that selects whole events, where one track has very low pseudo rapidity and very high momentum. This populates the "outer mirrors" (i.e. least-populated mirrors combinations) quickly enough to ensure the mirror alignment can be performed per run/fill (depending on the allowed rate of the HLT line), but we get many more tracks than we need this way. So, we could still reduce the number of events we need to populate the mirrors with photons by two orders of magnitude, which would make the mirror alignment lightning-quick. However, since the current processing speed was sufficient we did not do this. It could be considered for Run III, but we will first prioritize getting our code working in the Run 3 Real-Time Analysis framework.

There is one more package Rich/RichAlignment, hosted by the Rec project. It is used by Brunel and the RichAlignmentMonitor.cc code fills the histograms necessary for the extraction of possible misalignments (see below).

Return to the Top of this TWIki

Basic description of how the mirror alignment works

Provide a settings script name (Run 1) or a python Configuration (Run 2)

Start the steering Ganga script (Run 1) or Online script (Run 2)

This script first:

  • prepares an XML version of the Conditions database
  • creates a local copy of the SQLite of that database

Then it starts a loop, each iteration of which consists of:

  • launch of highly parallelized Brunel job (Run 1: using DIRAC on lxplus, Run 2: using HLT farm at the pit) running over:
    • during first iteration - RAW data files from the Express stream and writing DSTs (RUN1) or RAW only (Run 2)
    • during any further iteration - DSTs produced at the first iteration (Run 1) or RAW only (Run 2),
  • and Brunel produces Δ$\theta$ vs. $\varphi$ distributions for every combination of one spherical and one flat mirror out of the list defined in the options of the Rich/RichAlignment package that is used by Brunel;

  • fit these histograms and extract the total tilts around Y and Z axes;

  • using calculated in advance magnification coefficients (for each segment) solve a system of linear equations and obtain individual tilt of of every segment;

  • subtract (algebraically) each tilt from the existing misalignment correction for the corresponding segment just used by Brunel during reconstruction at the current iteration;

  • if all the obtained corrections to the previous misalignment corrections are (by absolute value) less than 0.1*mrad, then just exit, but if not -
    • replace the numbers in the XML and subsequently re-create the the local copy of the SQLite Conditions database
    • proceed to the next iteration of the loop.

  • If a correction of the database needs to be made, we upload the local copy of the Conditions database (with a new Time Of Validity tag) into the the Online database. The decision is currently automatic, save several sanity checks; thus we choose when to update the DB. Thus the newest mirror alignment is always available for processing data in HLT2 (and for some analyses (Turbo), this data is then immediately ready for physics analysis!).

  • Full Online monitoring of our alignment is available and may be used to identify any issues with the mirror alignment.

  • The constants are eventually moved to the git CONDDB for use in an future Offline processing.

Return to the Top of this TWIki

CERN e-groups

  • Bristol-only RICH mirror alignment discussions lhcb-rich-mirror-alignment-development
    • Mailing list for software development (i.e. Bristol) only
      • Private list
    • Details of code modifications
    • Discussions of possible issues with the alignment
    • Discussions of future improvements
    • Post news of any new software releases here

  • LHCb RICH mirror alignment news lhcb-rich-mirror-alignment
    • General mirror alignment mailing list
      • lhcb-general members may ask to be added to this list
    • Advertise accomplishments
    • Discuss ongoing studies / questions that have to do with our integration into LHCb
    • Discuss any urgent issues
    • Inquiries

  • Also subscribe to the RICH Software mailing list

Return to the Top of this TWIki

Who's Who

  • Experts (or soon-to-be-experts) at present, particularly for the Run 3 Real-Time Analysis implementation of the RICH mirrors' alignment
    • Paras Naik - contact person Paras.Naik@bristolSPAMNOTNOSPAMPLEASE.ac.uk
      • Upgraded code for consistent operation Online and Offline, maintains the code
      • Designed the current HLT line
      • Automated the Online alignment
      • Alignment Piquet
    • Anatoly Solomin
      • Designed the entire framework, came up with idea for HLT and offline preselections
      • Provided several evolutions of the minimization and regularization schemes.
      • Now consulting, helps the expert(s) maintain the core software packages Rich/RichMirrAlign, Rich/RichMirrCombinFit

  • Key contacts for Run 3 (and Run 2 transition)
    • Claire Prouve
      • Performed the first and full implementation of the Online mirror alignment
      • Currently working on the Run 2 Mirror Alignment paper
    • Sneha Malde
      • Real-Time Analysis (RTA) (leader of Work Package 4: Alignment & Calibration)
    • Chris Jones
      • RICH software
    • (will need to talk to all sorts of people as time goes on....)

  • Others from Bristol involved in the Run 2 Implementation
    • Matt Chapman
      • Worked on analyzing 2016 trend plots to determine tolerances for 2017 running.
    • Jeremy Dalseno
      • Helped to investigate mirror alignment constant variation after MDMS calibration corrections
      • RICH piquet
    • Sam Harnew
      • Worked on 2012 preselection (still needs to be committed to the main code)
      • Updated code (RichMirrCombinFit and Ganga script), re-performed 2010 mirror alignment
    • Samuel Madrell-Mander
      • Worked on analyzing 2016 trend plots to determine tolerances for 2017 running.
      • Upgraded the trendPlotter for improved understanding of alignment updates, and automatic production of trend plots in the Mirror Alignment
    • Claire Prouve
      • Performed the first and full implementation of the Online mirror alignment
      • RICH piquet
    • Jonas Rademacker
      • Helping to maintain organization and provided support, including some key "emergency" decision making.

  • Previous Bristol Contributors who helped build the path to Run 2
    • Andrew Cook
      • Created HLT map to be used in forming the eventual, maximally efficient HLT line
    • Matt Coombes
    • Tom Hampson
      • Offline preselection

  • Helpful people outside of Bristol, who helped ensure the success of our Run 2 alignments (noted for thank-you purposes, and some possibly to be invited/considered as authors of our Run 2 paper)
    • OPG / Alignment
      • Silvia Borghi (OPG leader)
      • Barbara Sciascia (deputy OPG leader)
      • Giulio Dujany (Alignment leader)
      • Lucia Grillo (Former alignment leader)
      • Barbara Storaci (former OPG leader)
    • RICH
      • Chris Jones (Helped everywhere)
      • Antonis Papanestis (Proposed the implementation of the idea at LHCb, helped study Run II alignment stability)
      • Mike McCann (RICH expert)
      • Silvia Gambetta (RICH expert)
      • Jordi Garra Ticó (Panoptes release manager)
      • Jibo He
    • Online / Pit resources team
      • Roel Aaij
      • Beat Jost
      • Clara Gaspar
      • Markus Frank
    • HLT
      • Rosen Matev
      • Conor Fitzpatrick
      • Sascha Stahl
      • Vava Gligorov
      • Mika Vesterinen
      • Sebastian Neubert
      • Christian Linn
        • Helped to investigate mirror alignment constant variation after MDMS calibration corrections
    • Perhaps others who we did not mean to forget...

Return to the Top of this TWIki

Legacy Twiki


Incomplete List of Accomplishments for Run II


  • Anatoly developed new regularization scheme that accounts for differences in magnification factors across the detector
    • Substantially reduces RICH1 secondary mirror fluctuations
  • Sam M.-M. and Paras upgraded Claire's code to now produce alignment trend plots automatically on every fill


  • Matt Chapman and Sam M.-M. determined update tolerances for the mirror alignment based on 2016 data
  • Paras helped move our constants to the Online DB
  • Paras established full logging of the mirror alignments
  • Paras helped complete full monitoring of the mirror alignment, including in the Presenter/Monet
  • Paras created and automated several sanity checks
  • Paras fully automated the mirror alignment decision including automatic updates on change of magnet polarity, providing full documentation for Data Managers, Alignment Piquets, and RICH piquets.
  • Year-end summary talk: https://indico.cern.ch/event/683758/#18-mirror-alignment


  • Claire's excellent work has been recognized with an LHCb Early Career Scientist Award: https://cds.cern.ch/journal/CERNBulletin/2016/38/News%20Articles/2215950
  • This is a nice summary by Jibo He about what we have been able to accomplish as part of the RICH alignment and calibration group:
  • Claire established the creation of several monitoring plots for the alignment.
  • Claire has performed several studies of the RICH magnification coefficients and RICH1 and RICH2 alignments.
  • Claire has worked with Online to get the alignment to run automatically starting 28 May 2016 (We are in the alignment chain). Updates to the database, with new Intervals of Validity (IoVs), will still be done manually.
  • Running now with Anatoly's minuit method of doing alignment without starting from a single primary mirror, using regularization to prevent chaos.
  • Claire established "best" number of events to run each RICH mirror alignment upon
    • Currently 1M RICH1, 2M RICH2 with current HLT lines [*not sure this is the actual mix coming out from current prescales though*]
  • Increased deltaTheta window.
  • Using bigger calibration tilts for RICH1.
    • Was 0.3 in 2015; now 0.7 for RICH1 for 2016 running (double-check that it's 0.7 and not 0.8)
  • Paras ensured the code was up-to-date on SVN and verified the migration of the code to git


  • Paras, with invaluable help from Anatoly, Sam, and Jeremy has the mirror alignment code running again Offline. Many improvements to this code have been made.
  • HLT1 and HLT2 line designed and updated by Paras, based on information from Claire and Andrew, has been incorporated by Vava, updated by Roel, information below.
  • Paras introduced the Online code; it has been vastly improved and is now fully functioning entirely thanks to Claire.
  • Claire has the alignment being run by the FSM
  • Claire has produced several good Rich1 and Rich2 alignments with the 2015 data.
  • Claire documented the Online procedure on our TWiki

Return to the Top of this TWIki

Status of our Software and Documentation

Which code has actually been deployed online

It is absolutely critical for us to know what code has been deployed Online. This should always be kept up-to-date:

Project: AlignmentOnline *v12r1 [ /group/rich/sw/cmtuser/AlignmentOnlineDev_v12r1 on plus ]

  • Plug-in: Rich/RichMirrorAlignmentOnline v4r4 : Panoptes branch 2018-patches (commits also propagated to master)
  • Plug-in: Rich/RichMirrCombinFit v17r1 : Panoptes branch 2018-patches (commits also propagated to master)
  • Plug-in: Rich/RichMirrAlign v20r1 : Panoptes branch 2018-patches (commits also propagated to master)
  • Plug-in: Rec/Brunel v52r0 (always necessary) : Rec branch 2018-patches (master only works for the LHCb Upgrade)
  • All other packages come from default location or are equivalent to default packages in Panoptes and Rec

We need to keep our code current. List code that needs to be upgraded here:

Return to the Top of this TWIki

Detailed status of code and documentation that we are responsible for

Formal Written Documentation

Run 2 Mirror Alignment Paper draft

  • Plan
    • Let's get this into a circulatable state.
    • After local comments are accounted for, we should circulate it more widely
      • This will also motivate Silvia and Roel to finish up the alignment paper, if there is going to be one
    • Account for new comments
    • Post the paper to the arXiv OR post it as a public note to CDS (following LHCb rules, of course).
  • Is someone working on the note now: YES
    • Who: Claire, Paras (in communication)
    • For what purpose: In preparation for paper draft / public note release, provide material for the Run 2 real-time alignment paper
    • For how long: Until it is done

Run 1 Mirror Alignment Internal Note (SVN instructions)

  • Newest Stable Version: Release Notes PDF (2016-01-31)
  • Is someone working on the note now: Anatoly
    • Who: N/A
    • For what purpose: N/A
    • For how long: N/A
    • What should be done: "Revert" note to Run 1-only note (leaving in updated explanations), which is complemented by the existing RICH performance paper. Run 2 improvements will all be covered in the Run 2 public note.

Run 2 Mirror Alignment Paper Draft in git

Project page: https://gitlab.cern.ch/cprouve/RichAlignmentNote2

If you want to work on this, ask Claire to make you a Developer.

All instructions are here: https://gitlab.cern.ch/cprouve/RichAlignmentNote2/blob/master/README.md

STAY IN CONTACT when working on this; though if you push and pull OFTEN things will be OK. If not, then ask Claire to sort out a merge request to bring diverging versions back in line. Once they are back in line, use the command sequence:

git fetch --all

git reset --hard origin/master

locally to sync to the current version of the repository.

Run 2 Mirror Alignment Paper Draft in SVN (DEPRECATED!)

Run 1 Mirror Alignment Internal Note in SVN

ALL involved in the nitty-gritty of the RICH mirror alignment procedure should know where to find the note. The most recent version should be kept here on the TWiki at the "Current RICH Mirror Alignment Note" link above.

To view/edit the note on lxplus, you may find the LaTeX by the following command set:

cd [to whatever directory you want to check out the note in, preferably with an identifiable name like AlignmentNote ]
svn co svn+ssh://svn.cern.ch/reps/lhcbdocs/Notes/INT/2013/007
cd 007/latest/latex
evince SoftwareMirrorAlignment.pdf &          [or pick a saved/committed version, like SoftwareMirrorAlignment _2016-01-31.pdf]

If you do edit the note, please:

  • make sure it the LaTeX compiled, hopefully with no warnings (right now there is only one, which is explained in the release notes)
  • Update 007/latest/doc/release.notes
  • Copy 007/latest/latex/SoftwareMirrorAlignment.pdf to 007/latest/latex/SoftwareMirrorAlignment_YYYY-MM-DD.pdf (where YYYY-MM-DD is the date of commit e.g. 2016-02-31)
  • Commit the changes

Figure 3 in the RICH Mirror Alignment note probably has the wrong horizontal axis label. A correct scale plot is shown in https://indico.cern.ch/event/325126/contribution/1/material/slides/0.pdf Perhaps Sam can help in producing updated plots.

Panoptes (https://gitlab.cern.ch/lhcb/Panoptes)

NOTE: In git the formal idea of tagging packages is GONE. We currently have an informal system based on when the release.notes for a package is updated. We are working towards a better informal system.

Rich/RichMirrorAlignmentOnline (Project Panoptes, plug-in to AlignmentOnline)

  • Newest Stable Version: v4r4
  • Is someone working on updates to the MASTER version now: YES
    • Who: Paras
    • For what purpose: N/A
    • For how long: N/A

Rich/RichMirrCombinFit (Project Panoptes)

  • Newest Stable Version: v17r1
  • Is someone working on the MASTER version now: NO, please do not start work without authorization
    • Who: N/A
    • For what purpose: N/A
    • For how long: N/A

Rich/RichMirrAlign (Project Panoptes)

  • Newest Stable Version: v20r1
  • Is someone working on the MASTER version now: NO, please do not start work without authorization
    • Who: Anatoly
    • For what purpose: N/A
    • For how long: N/A

Rich/RichMirrorAlignmentGanga (Project Panoptes)

  • Newest Stable Version: v13r4p1
  • Is someone working on the MASTER version now: NO, please do not start work without authorization
    • Who: N/A
    • For what purpose: N/A
    • For how long: N/A

Rec (https://gitlab.cern.ch/lhcb/Rec)

Rich/RichAlignment (Project Rec, plug-in to Brunel) we are fully responsible for this; others working on the RICH may use this

  • Newest Known Stable Version: 2017-03-23 update to v1r12p1
  • Is someone working on the master version now: NO, please do not start work without authorization
    • Who: N/A
    • For what purpose: N/A
    • For how long: N/A

Rich/RichRecQC (Project Rec, plug-in to Brunel) we are fully responsible for this, since others use RichFuture; still, others working on the RICH may use this

  • Newest Known Stable Version: 2017-03-22 update to v2r74
  • Is someone working on the master version now: NO, please do not start work without authorization
    • Who: N/A
    • For what purpose: N/A
    • For how long: N/A

Hlt (https://gitlab.cern.ch/lhcb/Hlt)

We are responsible for our Mirrors HLT Line and the settings. We must communicate with the HLT Operations group if there is any change.

Packages no longer being worked with

DBASE (https://svnweb.cern.ch/trac/lhcb/browser/DBASE)

PRConfig (Project DBASE) we only work on python/AlignmentTests/TestModules/PythonMirrAlignOnline.py

  • Newest Known Stable Version: v1r20
  • Is someone working on the HEAD version now: NO, please do not start work without authorization
    • Who: N/A
    • For what purpose: N/A
    • For how long: N/A

Return to the Top of this TWIki

JIRA tasks


Basic Training

Code policies

  • Follow all LHCb coding conventions
  • Code "hacks" should never be left alone and forgotten, you should actively seek to make them part of the LHCb software in a proper way as soon as possible.


We are no longer using SVN for version control.

Please ignore anything on this page (or any linked page) that says SVN and instead use git (except for the Run II Public Note draft which is still on SVN, though it should be migrated to git).

The following resources will help you git started, and understand how we use git in the Mirror Alignment group:

git for LHCb Users

Return to the Top of this TWIki

Latest LHC Schedule

You should always know the LHC Schedule if you are working on Mirror Alignment. You may find it for this year and for future years here:

LHC Schedule

Return to the Top of this TWIki

Previous Mirror Alignment Training Sessions

Look here for slides from previous mirror alignment training sessions:

Return to the Top of this TWiki

Important talks

We've done a poor job of keeping track of these since early 2016. You'll just have to search for talks given by Claire, Anatoly, or Paras in Indico.

Claire's slides at LHCb-UK 2016 (2016 01 07)

Claire's slides at RICH Software Meeting (2015 )
Paras's slides at LHCb Week Bologna (2015 )
Claire's slides at Analysis and Software Week (2015 07 13) Paras's slides in Andrew's talk at the Bristol HEP group meeting (2015 07 07, Bristol only, check your email for the password) Claire's update at the Rich Software Meeting (2015 07 03) Paras's slides in Chris's talk at the Computing Week (2015 05 20) Paras's update at Tracking and Alignment Meeting (2015 04 30) Paras's update at Rich Operations Meeting (2015 04 22) Paras's update at Tracking and Alignment Meeting (2015 03 31) Paras's monitoring proposal (2015 03 27) Paras's LHCb-UK Talk (2015 01 05) Andrew's RICH Mirror Alignment Status and Plans Talk (2014 06 18) Andrew's HLT Selection Talk (2014 05 21) Beat Jost's Framework for Online Alignment Talk (2014 11 ) Advice from Tracking Alignment group (2014 11 ) Latest Run 2 Overall Alignment talk (2014 11 ) OPG Run II Readiness talk (2014 09 19, LHCb Week Orsay) Older Talks (Unorganized)

Return to the Top of this TWIki


TWIKI PAGE (this document):

Most Recent Stable RICH Mirror Alignment Note (Anatoly Solomin, others) Run 1 Mirror Alignment Code Flowchart (Sam Harnew) Run 2 Framework for Distributed Alignment (Beat Jost, others) HLT map description (from a draft of Andrew Cook's thesis) Tom Hampsons's thesis
  • https://cds.cern.ch/record/1670276?ln=en
  • Read p. 51 of Tom's thesis (CERN-THESIS-2013-291) for a clear description and derivation of the key angles used in the RICH mirror alignment. Noting that the small angle approximation is used, and the optical path (l) gets/is dropped on both sides of the equations, will make it easier to understand.
Software Architecture of the HLT

Return to the Top of this TWIki

Key Meetings

  • Regular RICH Mirror Alignment meeting (only held when needed)
    • 10:00 UK / 11:00 CERN, weekly on Tuesdays
  • LHCb Bristol meetings
    • 9:15 UK / 10:15 CERN scheduled for 105 minutes, weekly on Fridays
  • RICH Software meetings
    • 13:00 UK / 14:00 CERN scheduled for 120 minutes, weekly on Fridays
  • Operations Planning Group (OPG) meetings
    • 15:00 UK / 16:00 CERN scheduled for 135 minutes, weekly on Thursdays
  • LHCb Weeks, A&S Weeks, RICH Regular meetings (give an update when asked)
  • Tracking and Alignment Meeting (give an update when asked)
    • Tuesday 16:30-18 CERN time and Thursday 11-13 CERN time, in alternating weeks
  • RICH Operations meetings (give an update when asked)
    • RICH Ops meetings are typically Wednesday mornings
  • HLT Operations meetings (we do not usually attend)
    • 14:00 UK / 15:00 CERN scheduled for 180 minutes, weekly on Fridays
  • Various ad-hoc meetings

We need to attend as many of these as possible, to increase visibility and stay on target. Ad hoc meetings tend to be the most productive. If you need an ad-hoc meeting, call an ad-hoc meeting. We should be interacting and presenting our progress in the relevant meetings. Someone active on RICH mirror alignmnent should attend each required meeting.

Return to the Top of this TWIki

Basics and Information

LHCb Software Tutorials (VERY USEFUL!!!)

Using git to develop LHCb software (MUST READ!!!) LHCb Software Development (MUST READ!!!) Python Tutorial GitLab JIRA RICH Mirror Alignment component issues page The LHCb Nightly Builds system CondDB How-To page Convert degrees to pseudorapitity Embedding Python in another application HLT Tracking Twiki Online Twiki LHCb SVN Guidelines (With Tutorial for Developers... still a MUST READ!!!)

Return to the Top of this TWIki

Useful Shell Commands

Enable read access to anyone in LHCb (z5)

fs setacl -acl cern:z5 rl -dir `find . -type d`

Enable read access to a particular person (in this case, pnaik)

fs setacl -acl pnaik rl -dir `find . -type d`

Check permissions

fs listacl

Return to the Top of this TWIki

Useful SVN Commands

We are no longer using SVN; See LHCbRichMirrorAlignGitInfo for git commands.

Check if there are any updates

svn status -u


svn update [specific filename, or leave blank for everything in current subdirectory]


svn commit -m "Write a reasonably short but informative comment here. Put details on non-minor changes in the release notes." [specific filename, or leave blank for everything in current subdirectory]

perform a diff of your changes to the code to what you checked out from SVN that ignores whitespace differences and blank lines

svn diff --diff-cmd /usr/bin/diff -x "-wB" [specific filename, or leave blank for everything in current subdirectory]

perform a diff to the HEAD version of the code (i.e. if someone else committed changes you can check them using this) that ignores whitespace differences and blank lines

svn diff -rBASE:HEAD --diff-cmd /usr/bin/diff -x "-wB" [specific filename, or leave blank for everything in current subdirectory]

Useful Ganga Commands

This is left over information from the Offline alignment, will move eventually

Remove a bunch of jobs (note this only removes them from your gangadir, it does not delete the files on the grid). Make sure not to use 991 and 994 but the first job you want deleted and the last job you want deleted
j_Start = 991
j_End = 994
for j in range (j_Start, j_End + 1):
         print "Job %d doesn't exist, or trying to remove it threw an exception." % j
         print "Job %d doesn't exist, or force_status 'failed' threw an exception." % j
         print "Job %d doesn't exist, or trying to remove it threw an exception." % j

Example: When starting Ganga, use a different gangadir than your default gangadir (useful if you want to submit multiple alignments from different shells).

ganga -o"[Configuration]gangadir=/afs/cern.ch/work/p/pnaik/gangadir_2015"

ROOT Color Palette for nice "2D heat maps"

UInt_t Number = 3;
Double_t Red[Number]    = { 0.00, 0.00, 1.00};
Double_t Green[Number]  = { 0.00, 1.00, 0.00};
Double_t Blue[Number]   = { 1.00, 0.00, 0.00};
Double_t Length[Number] = { 0.00, 0.50, 1.00 };
Int_t nb=50;

Return to the Top of this TWIki

HLT1 Line

The HLT1 Line for the mirror alignment is absolutely crucial to getting the photons we need, specifically reflecting in the mirror combinations that we use to perform the mirror alignment.

Vava installed the skeleton line, which is based on the one track trigger. Paras provided and refined the selection criteria. Roel and others from the HLT group have been helping us with calibrating the line. Andrew consulted on using ideas from the HLT map work to make this possible.

Our HLT1 line is implemented into the HLT, and is called Hlt1CalibRICHMirror. ROUTING BIT 54 provides the data to be accessible Online.

More information on this dedicated TWiki: LHCbRichMirrorAlignHLT

Return to the Top of this TWIki

Online mirror alignment instructions

To run the rich mirror alignment online on the HLT farm, click this link and follow the instructions

  • Please follow Claire's good example and set System = "Alignment" whenever posting to the ELOG
  • All new alignments should be documented on the ELOG!

To learn about the alignment package and the online code click this link

Training Sessions

Timetable for the use of the HLT Farm (colloquially, the “Alignment” Farm) during 2017

If you want to use the Alignment Farm to manually run the alignment... you must reserve time here:


Return to the Top of this TWIki

AlignmentView (Online mirror alignment results)

Check it out, live streaming worldwide mirror alignments!


Location of the output files

Our most recent alignments are always here on plus:
Our past alignments are stored here:

Editing the AlignmentView PHP script

On 26/05/17 15:46, Michael McCann wrote:

The web pages are a bit more difficult to edit now we've moved to our own server, but not much. Thinking about it I could probably have made it identical, but I stuck with the method the admin used when he migrated us.

Anyway, all the files are hosted locally on lbrich, and in the directory /web/sites/lbrich/. The alignment php script is alignmentview.php. The machine doesn't have access to /home so some editors (like emacs) get unhappy, vim works fine though (although moans when you exit).


Each column in the table has an associated function write*Element($summary), which just writes an html table data element 'value' with a particular class and value, and returns a boolean of whether the value should be considered ok. $summary is a dictionary that contains anything in the summary text file parsed such that a line in the file "foo = bar" can be accessed $summary["foo"] = "bar" (everything as a string and external white space stripped). Basically all that's needed is to pick the class based on the value in whatever why you want.

The classes I've put in are "good", "bad", "expert", "unknown". "bad" and "expert" are identical at the moment, but are conceptually different (something definitely bad vs something to inform an expert about). "unknown" is used when the required info isn't in the summary (e.g. your old summary files don't contain all the fields).

The general form of the functions is

function writeSomeParameterElement($summary) { $style = 'unknown'; $value = $summary['Some Parameter']; if ($value) { //some code to select $style from $value, e.g if (intval($value) < 12345) $style = "good"; else $style = bad; } echo ''.$value.''; return ($style ='good'); }

but you're free to implement it however you like.

If you want to add a new column, look in the function writeAlignmentSummary($richfiles, $fills). If you want your column to appear in the autogenerated log entry, you will need to add a line of the form

$isgood['some parameter'] = writeSomeParameterElement($summary);

with the rest, where writeSomeParameterElement is a function you've written and obeys the specification above. You also need to add an element $badtext['some parameter'], of what you want written to the log if it isn't ok. For prettiness you should increment the "colspan" attribute to the summary panel a few lines below too. Don't forget to add a title to the table header.

Oh, if you make changes please run the script /group/rich/scripts/back-up.sh in the /web/sites/lbrich/ directory. It will update /group/rich/backup/lbrich/latest to reflect the changes.

Return to the Top of this TWiki

Making the trend plots manually

Trend plots are being made automatically every fill, and can be found in AlignmentView.

The automatic plots are made in the "hollow" style (points are hollow, unless an update was made during a particular fill, in which case the points are solid) and available with the LHCb fill numbers on the x-axis or the alignment date/time on the x-axis. The plots with fill numbers on the x-axis are also attached to the RICH ELOG alignment summaries.

To manually produce the trend plots do the following:

Create a new directory somewhere on plus (let's call this directory '/home/your_username/trendplots/' for now, but it could be anything)

cd into that '/home/your_username/trendplots/' directory

Then copy some files using these commands:

cp /group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/fileFinder.py .
cp /group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/LHCbStyle.py .
cp /group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/tiltObj.py .
cp /group/rich/sw/cmtuser/AlignmentRelease/Rich/RichMirrorAlignmentOnline/python/PyMirrAlignOnline/trendHelper.py .

Then edit your local version of trendHelper.py and check/change these things:

  • change the startDate and the endDate for RICH1 and RICH2 to fit whatever date ranges of mirror alignments you wish to plot
  • also, in the line that looks like
self.file_name = "/home/pnaik/2018trendplots/Rich"+str(self.whichRich)+"_allin1_"+str(self.plot_type)+"_"+str(self.trend_Xaxis)+"_current.pdf"
change /home/pnaik/2018trendplots/ to your '/home/your_username/trendplots/'

Then, just run:

source /group/rich/sw/cmtuser/AlignmentRelease/setup.x86_64-centos7-gcc62-opt.vars
python trendHelper.py > out.txt

and you will see that many plots are produced.

Trend plots will be produced for both RICH1 and RICH2 in combinations of the following formats:

  • plotType
    • "hollow" - points are hollow, unless an update was made during a particular fill, in which case the points are solid
    • "default" - points are solid, if an update was made then all points collapse to the zero line
  • trendXaxis
    • "Fills" - plotted by Fill Number
    • "Dates" - plotted by Date/Time
    • "Numbers" - plotted by Alignment Number (i.e. order in self.alignments)

You can also look at out.txt for the complete printout from the plotting code (for example, if you want to know the Alignment Name corresponding to an alignment number).

Return to the Top of this TWiki

Overall Online calibration and alignment procedures (for all detector alignments)

Calibration and Alignment Procedures

First calibration and alignment procedures for 2016 restart

Return to the Top of this TWIki

What all of the settings mean (also applicable to Offline)

Check Configuration.py for descriptions of the latest settings. This is historical information:

Some settings shown with example values. Note that some of the example values are strings. Please see an actual settings files for examples on actual implentations. These are just shown here to help elucidate the settings.

### workDir = /afs/cern.ch/user/p/pnaik/RichAlignment/2015/2015_R1_EM_Down_1250k_ev/

  • this is the alignment working directory, your settings file and all major output will go here

### dDDB_tag = dddb-20150703

  • together with the condDB_tag, specifies the detector conditions and alignments

### condDB_tag = cond-20150716

  • together with the dDDB_tag, specifies the detector conditions and alignments

### date = 1436546246000000000

  • this is an "epoch" date, in nanoseconds. It should fall within the range of dates of the data sample that you use for alignment. See http://www.epochconverter.com/ for more information

### localCondDB_tag =

  • we do not use this. not yet sure if this is obsolete. please leave this in the code

### setupProjectBrunelOpts =

  • options for setting up Brunel... don't need this right now

### eventSelectorInputFile = /afs/cern.ch/user/p/pnaik/ReduceRawFiles/EM/RICH1/10GeV/R1_10GeV_EM_Down.py

  • equivalent to ioHelperInputFile, this file contains the locations of the data files that we will run over to get the alignment

### LHCbVersion =

  • just use the default

### brunelVersion = v48r0

  • the Brunel version we want to use to process out data

### brunelEvtMax = 12376

  • the maximum number of events to run over per Ganga subjob. Offline, this is automatically calculated from the number of files and number of events you wish to use for alignment. See the settings file.

### brunelPrintFreq = 1000

  • frequency for printouts

### brunelInitialInputType = RAW

  • Type of data we are running over

### messageSvcOutputLevel = 3

  • standard, don't touch

### splitFilesPerJob = 5

  • Number of files per subjob. Offline, this is automatically calculated from the number of files you wish to use for alignment. See the settings file.

### splitMaxFiles = 99999999

  • standard, don't touch

### backEnd = dirac

  • standard, don't touch

### diracCPUTime = 100000

  • standard, don't touch

### mirrCombinSubset = Rich1CombAndMirrSubsets _20m16c_p0p1p2p3fix.txt

  • This is chosen automatically depending on richDetector. It defines the mirrors that are used in the alignment procedure.

### thisCase = 2015_R1_EM_Down_1250k_ev

  • A string used to identify the alignments. We usually make this the same as our working directory name.

### dataVariant = Collision15

  • For 2015, don't touch

### minAverageBinPop = 6.0

  • The minimum average population of the deltaTheta bins in a single bin of phi. Leave this as 6.0

### deltaThetaWindow = 4.0

  • Chosen automatically depending on richDetector. The window in deltaTheta.

### coeffCalibrTilt = 0.3

  • The size of the purposeful mirror tilts used in the alignment. Leave this as 0.3 for now

### useTruth =

  • The current version of the code has not been validated for use with Monte Carlo

### useOffsetsFromMC = false

  • Not sure what this is, again don't use with MC yet.

### alignHPDs = false

  • always false, do not touch

### AddedDB = []

  • a python list of strings that tell the alignment where to look for database slices that are newer than what is in the conddb and dddb. If you need them we will tell you.

### usePremisaligned = false

  • leave this alone for now.... have to come back to this later

### useSpecificAlignment = true

  • if this is true, then we want to substitute a specific mirror alignment for what is in the existing conddb.

### specificAlignment = /afs/cern.ch/user/p/pnaik/cmtuser/Panoptes_HEAD/Rich/RichMirrorAlignmentGanga/files/2015Alignments/2015_EM_MagDown/Rich1CondDBUpdate_Mp6Wi4.0Fv3Cm2Sm1_online_Collision15_i3.xml

  • if useSpecificAlignment is true, then this is the location of the alignment that we will use as our starting (zeroth iteration) alignment.

### combinFitVariant = 3

  • always 3, don't touch. FitSurface fitting technique for the 2D deltaTheta vs. phi histograms

### maximumNumberOfIterations = 10

  • we usually leave this alone. The alignment should converge well within 10 iterations if there is enough data

### richDetector = 1

  • IMPORTANT ... this is the rich (1 or 2) you wish to align

### solutionMethod = 1

  • always 1 for now. Algebraic method

### home = /afs/cern.ch/user/p/pnaik/

  • your home directory

### userReleaseArea = /afs/cern.ch/user/p/pnaik/cmtuser/

  • your cmtuser area

### pathToScripts = /afs/cern.ch/lhcb/software/releases/LBSCRIPTS/prod/InstallArea/scripts/

  • defined automatically

### RICHMIRRALIGNROOT = /afs/cern.ch/user/p/pnaik/cmtuser/Panoptes_HEAD/Rich/RichMirrAlign/

  • defined automatically

### RICHMIRRCOMBINFITROOT = /afs/cern.ch/user/p/pnaik/cmtuser/Panoptes_HEAD/Rich/RichMirrCombinFit/

  • defined automatically

### RICHMIRRORALIGNMENTGANGAROOT = /afs/cern.ch/user/p/pnaik/cmtuser/Panoptes_HEAD/Rich/RichMirrorAlignmentGanga/

  • defined automatically

### SQLITEDBPATH = /afs/cern.ch/lhcb/software/DEV/nightlies/DBASE/Det/SQLDDDB/db/

  • defined automatically

### previousJobNumbers = [23, 24, 25, 26, 27, 28, 29, 30, 31]

  • this is formed from information in the settings file, so refer to that. It is only needed if the first Reconstruction job is skipped

### startFromIteration = 0

  • you can start from any iteration. so if something broke you can start again from the beginning of that iteration by changing this number

### skipFirstRecoJob = 0

  • If the Reconstruction job in a broken iteration completed, or was completely submitted, then set this to 1

### skipFirstMergeJob = 0

  • If the Reconstruction job in a broken iteration completed and the files were merged but RichMirrCombinFit did not finish running, then set this to 1 as well as the previous

### skipFirstCombinFit = 0

  • If the Reconstruction job in a broken iteration completed and the files were merged and RichMirrCombinFit was run, then set this to 1 as well as the previous two

### justGlobalFit = 0

  • always leave this a zero... it's a legacy option that we'll get rid of later


  • we never use this... need to look into it

### thisAppConfigBrunelOpts = Default

  • we never use this... need to look into it

### stopTolerance = 0.1

  • RichMirrAlign will stop the alignment when all of the mirror tilts shift less than this amount from the previous iteration

### warningFactor = 20

  • A warning is given if any of the mirror tilts is greater than 20*stopTolerance. Right now there is no automatic mechanism to stop the alignment if this is the case, it's just a warning. This may change in the future.

### phiBinFactor = 3

  • Group the bins in phi, to reduce the total number of bins in phi by a factor of phiBinFactor. Brunel v47r9 uses 20 bins in phi. Brunel v48r0 and later uses 60 bins in phi. When that happens we set phiBinFactor to 3 to get 60/phiBinFactor = 20 bins in phi.

Return to the Top of this TWIki

We should be able to answer the following questions about every alignment we perform!

* (about the data)

  • what is the magnet polarity of the data?

* (from the HLT line)

  • what is the momentum cut on the trigger track of the line? (currently 20 GeV for Rich1 and 40 GeV for Rich2)

* (in the Brunel options)

  • What is the momentum cut on the tracks of the events? (currently 20 GeV for Rich1 and 40 GeV for Rich2)
  • What kind of tracks do we require? [for now it's still long tracks, we think]

* (in Rich/RichAlignment)

  • Which version of Rich/RichAlignment are we using? (hint: it's tied to the version of Rec that Brunel depends on)
  • What's the number of bins in phi produced by this version? (hint: for Rich/RichAlignment v1r12 and after it is 60 bins, otherwise assume it's 20 bins)
  • Are we using Chris's track selection to reduce photon background? (hint: for Rich/RichAlignment v1r12 and after, you are. Or if you copied Paras's Configuration.py [see below] to your cmtuser area if using Rich/RichAlignment v1r10p1)

* (in the settings of configuration file or printed in the startup of the alignment, so easiest to find)

  • Which version of Brunel are we using?
    • what is the number of files per job?
    • what is the maximum number of events brunel will reconstruct in each subjob?
    • what is the numberInputFiles?
    • what is the numberToProcess?
  • what added databases (i.e. detector conditions, including alignments that come before us in the chain) are we using, if any?
  • what are the CondDB and DDDB tags?
  • what is the epoch time?
  • what is the coeffCalibrTilt? [OFFLINE this is 0.3 for both RICH1 and RICH2 for now]
  • what is the minAverageBinPop? [OFFLINE this is 6.0 for both RICH1 and RICH2 for now]
  • what is the stopTolerance? (hint: if it's not in the settings file, then it's using the default of 0.1)
  • what is the phiBinFactor? (hint: if it's not in the settings file, then it's using the default of 1; you want it to be 3 if using Rich/RichAlignment v1r12 of later )

Return to the Top of this TWIki

Python script to compare two XML files

/afs/cern.ch/user/p/pnaik/cmtuser/Panoptes_HEAD/Rich/RichMirrorAlignmentGanga/scripts/XMLcompare.py on lxplus, and use python XMLcompare.py -h for help.
NOTE: This script creates temporary files, therefore you cannot run it from a directory where you do not have permissions!!!

For your convenience, here is the content of that script if you just want to copy it into a blank XMLcompare.py:

import os, re, argparse
parser = argparse.ArgumentParser(description='Compares two XML files or a single diff between XML files. Usage is "python XMLcompare.py " OR "python XMLcompare.py &lt;diff_between_two_xmls.txt&gt;". Designed only to compare Y and Z tilts for all alignment parameters in the XML file. Does not tell you the name of each parameter, only draws attention to those that differ by more than 0.1 mrad. Tilts are ROUNDED not truncated.')
parser.add_argument('files', metavar='FILE', type=str, nargs='+',
                   help='the file with the diff OR the two files you want to diff')
parser.add_argument('-w', action="store", default = 0.1, dest="warningPrecision", type=float, help='the precision of differences at which warnings are provided, default is 0.1')
parser.add_argument('-p', action="store", default = 2, dest="displayPrecision", type=int, help='the number of digits after the decimal point printed for output numbers, default is 2')
args = parser.parse_args()

warningPrecision = args.warningPrecision
displayPrecision = str(args.displayPrecision)

#Actually, the parser will tell you that you didn't provide enough arguments, so we don't need this next bit
# if (len(args.files)==0):
#     print "I need files."
#     raise SystemExit ("I need files.")
if (len(args.files)==1):
    file = args.files[0]
    print "pre-diffed input file: " + file
if (len(args.files)==2):
    file = args.files[0]
    file2 = args.files[1]
    tempfile = "temp_XMLcompare.txt"
    print "first XML file to diff: " + file
    print "second XML file to diff: " + file2
    os.system("diff "+file+" "+file2+" &gt; "+tempfile)
    file = tempfile

# what we're gonna do, is search through it line-by-line
# and parse out the numbers, using regular expressions

# what this basically does is, look for any number of characters
# that aren't digits or '-' [^-\d]  ^ means NOT
# then look for 0 or 1 dashes ('-') followed by one or more decimals
# and a dot and decimals again: [\-]{0,1}\d+\.\d+
# and then the same as first..
with open (file, "r") as myfile:
    data2 = myfile.read()

data = re.sub(r"([a-z]+)", r"\n \1", data2)

#print data

# use http://www.myezapp.com/apps/dev/regexp/show.ws to translate
pattern = re.compile(r"[^-\d]*([\-]{0,1}\d+\.\d+)[^-\d]*")

results = []
for line in data.split("\n"):
    match = pattern.match(line)
    if match:
# summary: looks for things like 0.3333 or -2.2344 and only keeps those

# print results

sets = []
i = 0
end = len(results)
while i &lt; end - 1:
    sets.append((results[i+0], results[i+1], results[i+2], results[i+3], results[i+4], results[i+5]))
    i += 6

print " Y and Z tilt differences for all parameters. Note that numbers are ROUNDED not truncated. "
for t in sets:
    #    print t
    a = float(t[4]) - float(t[1]) # Y
    b = float(t[5]) - float(t[2]) # Z
    str1 = "                                           "
    str2 = "                                           "
    for i in (0,int(displayPrecision)):
        str1 += " "
        str2 += " "
    if abs(a) &gt; warningPrecision:
        str1str = ", warning: this Y tilt difference is %7."+displayPrecision+"f mrad"
        str1 = str1str % a
    if abs(b) &gt; warningPrecision:
        str2str = ", warning: this Z tilt difference is %7."+displayPrecision+"f mrad"
        str2 = str2str % b
    printString = " %7."+displayPrecision+"f , %s, %7."+displayPrecision+"f , %s "
    print printString % (a, str1, b, str2)

if (len(args.files)==2):
    os.system("rm -rf "+tempfile)

Return to the Top of this TWIki

How to view our IOVs in the git ONLINE DB (since 2017)

In principle you can view these in the CondDBBrowser, but in Run 2 after 2017 there is a subtlety in the git ONLINE DB that makes it look like we keep reverting back from the assigned mirror alignment for a run to a "default" alignment in-between the runs. This "default" alignment is actually the (only) mirror alignment that was used throughout 2016. They are partly there to maintain compatibility with the old COOL DB.

Presuming Marco Clemencic doesn’t alter this “default” condition to something else, if you ever need to view the IOVs properly for:

Return to the Top of this TWIki


If you ever need this:

e.g. dirac-bookkeeping-file-path --File LHCbCollision16Beam6500GeV-VeloClosed-MagUpRealDataTurbo0394000000CHARMTWOBODYMDST.py --GroupBy RunNumber | grep RunNumber > MagUpRunsCollision16Turbo03.txt
  • then import into excel, plot them in a scatter plot, and use the cursor to pick off the run numbers

Get the dates from the ELOG

Return to the Top of this TWIki



Return to the Top of this TWIki


I---The following should be moved to the proper places above and/or to a separate TWiki---I


To Be Organized Above

Notes from Anatoly 6 February 2017

  • Monitoring
    • The main monitoring interface for the alignment should be a coloured schema of all mirrors (analogous to the one in the note) which will be updated after each alignment iteration/session with green mirrors that do not need corrections, yellow - when a correction is obtainable after a first iteration (without refreshing the coefficients) and red - if a second iteration is needed involving the refreshing of the coefficients. Whel all mirrors are aligned, and became green, those orange or red colours should remain on the picture as frames.
    • Monitoring of the our HLT line would be also a similar schema but already with mirror combinations in the boxes, and green - if there is "enough" photons there, orange - if on the verge (>85%), and red - if evidently not enough (<85%). Exact numbers can be refined, of course...

  • Storing magnification factors
    • Online format is the same as offline, as is.
    • We should re-assess the optimum subset of combinations with min bias or maybe b-hadron events (as Tom did). Naturally, can be done with MC.
    • Let's store all factors (including so-called cross-talk ones), and not averaged, because while calculating the chi-square, we can make use of the positive/negative related factors, as was already done long before, but commented out for now. Generally, it will improve convergence (simply will find a "milder" solution. The relative storage overhead will be entirely negligible...
    • Better to store them in a more XML-wise way: every number is a leaf, not a component of a vector. That will make parsing and updating more straightforward.
    • The task is untrivial, and needs careful planning

  • A corresponding MC sample at hand is needed
    • For re-determining of optimum subset of combinations.
    • For Measuring and applying the systematic error of the method (because in fact, MC also "needs" some alignment).

Here is a summary of some significant recent details which Anatoly thinks may be instructive in the current alignment context 5 June 2015

  • Tom's pre-selection:
    • Tom-slected events whatever year's, are entirely different, separate DST sets for RICH1 and for RICH2. Should be made sure that you run respectively, first thing.
    • The pre-selections were [previously] produced with 5/20 and such data-sets therefore definitely too small for running alignment over them with 20/40, especially if you additionally curtail at 80%. When Tom was running selections, he charged it for running over appropriate excess over was needed, to compensate for the job failures
    • Our previous pre-selections were run over the b-hadron-stream events, which have wider angular distribution, and hence more efficient for the periphery.

  • No pre-selection at all:
    • makes sense as an absolutely clean, unbiased solution good for any cases, including all kind of tests.
    • On the other note, If it works in the ultimately hard case, the Collaboration will be confident in our ability. And as to the performance improvements - they are always nice...
    • That said, the probability of some crash is proportional to the event number, and not only on the Grid, but the farm-wise too. One of the generally known bottlenecks is the merging of the root files.
    • And the speed up of ~20 times with Tom's offline pre-selection or even ~10K times for only offline iterations after the HLT pre-selection can't pass unnoticed by the Collaboration too.

  • RichMirrCombinFit:
    • One can't use slice-by-slice method: only the "functor" method is actual. Nothing ever converged until Anatoly "invented" that semi-surface method by the use of functor. Please make sure only this one is used.
    • The histogram-fitting part was justifiably rewritten some time ago (except the core of my implementation of the fitting procedure), although it was perhaps slightly overdone. Let me finish verifying it, and probably fixing it too. We can discuss the ETA.
  • RichMirrAlign: recently Anatoly did a few non-show-stopping changes:
    • the printouts are fixed: thay are extremely important for grasping at a glance the results of an iteration and the trend;
    • algebraic method is now simpler: two sides in case of RICH2 and four quadrants in case of RICH1 are solved in one go, with only the two most-central primary mirrors fixed at 0.0 for RICH2 and with all the four RICH1 primary mirrors fixed at 0.0, which is consistent in all senses.
    • Least-squares method is now entirely both side-agnostic and fixed-mirror-free. By definition, the results will be different from the algebraic ones. However, if one wants to probe consistency of results from the two methods, one can simply fix here the same mirrors as in the algebraic one. Alternatively and ultimately, we can regularize the algebraic method the same way as in the LS, but Anatoly does not think it would be worth the effort.
    • Particularly for RICH1, Anatoly dropped support of the Matt's averaging of the primary mirror corrections method (only during the first iteration) in the algebraic mode of operation, because it is superseded by the new regularization in the LS mode, which is also number-of-iteration-agnostic. Also the numbering schema is now undoubtedly correctly understood/used here.
    • Important detail about the LS method implementation: as Anatoly stressed some time ago in his chat with Paras, Anatoly deliberately switched off traditional "weighting" of the terms of the functional, because for us, "equal rights" of the mirrors in the alignment procedure is the priority, not by any means the dominance of the amply-populated mirror combinations. Ideally, when the "Map" project is implemented in full, all "weights" will be pretty much equal anyway.

  • Resolution function: finally, a few words about my opinion on the "integrated resolution".
    • The integrated "Cherenkov-angle resolution" distribution is not representative, if not to say stronger. This is clear to everyone.
    • Even summation of the normalized histograms (of the individual combinations) will not help that much, because so-to-say statistical smoothness of the strongly-populated histograms will dominate over the less regular shapes of the poorly-populated ones. While strictly speaking, all mirrors are largely equally important, the peripheral mirrors are of special importance for the most heavy particle decays and must not be "overshadowed" AMAP...
    • Of course, clean way - to select equal amounts of photons for each considered combination and then add up all such histograms. But this does not sound feasible right now, though.
    • Instead, Anatoly would suggest a more realistic thing: for each primary mirror (RICH2) select equal amount of photons reflected only off the most overlapping with it secondary mirror. Then fit the histograms and also their sum. The results will represent the overall quality of the alignment. In case of RICH1, Anatoly would instead dance from the secondary mirrors in a similar way, Just because primary mirrors here have almost no angular granularity, unlike the secondary ones.

How to estimate number of events per job? Anatoly 16 Apr 2015

nEventsPerJob = nFilesPerJob * nEventsTotal / nFiles

According to Andrew Cook's Thesis, 129523886 events need to pass through the HLT line to fill the histograms of the least populated mirror pair (p49s36 of RICH2). These events come from a sample that is approximately one-tenth of all events. Thus we need to filter through approximately 1.3 billion events to populate this mirror pair.

Chris Jones helped us to make major improvements to the speed that we can filter through these RAW events, such that we will no longer require preselection of events for Offline alignment. The main time saver is to immediately reject events that do not have any 40 GeV Forward (long) tracks, and reconstruct only those tracks that have enough momentum.

We determined that the new options only keep 1 in O(10000) events, maybe even more. This is assuming that event numbers in a run increment by 1. So if we filtered conservatively through 2 billion events, we would end up with about 200000 events (or parts thereof) needing to be reconstructed by the RICH.

Being a bit more conservative [I think the above calculation is wrong somewhere], and for convenience, in our 2012 re-alignment of the fourth Interval of Validitiy (IOV) we decided to collect 3M events. We believe this currently occurs across the sample as the RAW data files get split across the Grid (splitFilesPerJob) and Brunel runs over the first brunelEvtMax events in each subjob.

The Brunel specific settings that we use for the realignment of the 2012 fourth IOV are

splitMaxFiles                     =  750
splitFilesPerJob                  =  5
brunelEvtMax                      =  20000
We know for this sample there are 746 raw data files. So there are 150 subjobs to be run, each having 20000 events.

Hopefully this is enough for future RICH mirror alignments. We'll find out soon.


There were various not-completely-addressed email threads that have key information relevant to the mirror alignment. The key information from most of those emails has been extracted below, but still needs to be put in the proper places above. The name of the email thread is stated when it might be useful to know, if someone needs to go back and look at the thread.

"fits to Cherenkov resolution" thread, 25-26 April 2016 and "Tighter cuts in monitor for CKResolution: NO more bump", 28 April - 16 May 2016

An unfittable shoulder in our CK angle resolution plots was thought to have come from unsaturated Kaons. This was solved by increasing the momentum cut to get only saturated particles.

Chris Jones suggested to us the way to raise the momentum cut:

b.t.w. if you want you can define your own set of cuts, around the same place as line 257 
below 'Alignment' if you like, then at line 493 just add another line like

to create the histos with those cuts.

Looking at the tight cuts, I might change them a bit anyway as some things have changed a bit. 
For instance GhostProb < 0.8 is doing nothing, as now the track cuts at < 0.4. 
Might try lowering it to say < 0.2 or so...

At some point later Claire reported:

For monitoring we thought of using the same monitor that Chris uses for his refractive index calculation (PhotonMonitoring).
I included that one and increased the momentum cuts on the tracks (only the tracks that fill THIS monitor, 
not the ones that fill the alignment monitor) and actually managed to get rid of the "bump" in the Cherenkov angle resolution.

It's not clear if Claire changed the PhotonMonitoring or just adjusted the cuts. Worth having a look. The cuts only changed in this external monitor, our alignment still uses the HLT line cuts.

Paras thinks for the monitor we took the min track momentum in Rich2 from 40 to 60 GeV, and for Rich1 from 20 to 40 GeV. However Claire told us "Currently the external monitor can only be given one cut for both RICH1 and RICH2 (which is set to 60GeV)."

Claire did confirm in a separate email thread on 29 April: "I also included histograms made from the same monitor that Chris uses for the refractive index calibration but with a higher momentum threshold which seems to remove the "bump" in the Cherenkov angle resolution (I will run the fits on those histograms soon and report back)."

Claire updated to the new version of RichMirrAlign which means that now we can have different convergence criteria for primary and secondary mirrors in y and z respectively (a.k.a "quad" tolerances).

Anatoly introduced and implementd new "quad" convergence criteria in RichMirrAlign (the corrections finding step), accounted for magnification factors and nominal Cherenkov angle resolutions.

Anatoly introduced and implemented an alternative, also nominal-resolution-based convergence criteria, but at the histogram fitting (tilt finding) stage.

"Rich alignment update", 19 June 2016

IMPORTANT: this reminds us of the currently increased converge criteria... with the online DB we will be able to go back to normal criteria or future quad-tolerance criteria. We can then change constants with magnet polarity rather than having to inflate our tolerances.

Claire also added a piece of code that will calculate the difference of the resulting xml-file from another given xml-file. This means we can start the alignment from any arbitrary xml-file and still calculate the difference of the resulting alignment from the one that is currently in the database. This piece of code also adds another set of plots showing the mirror tilts for each mirror in y and z wrt the alignment in the current database, as can be seen here https://lbgroups.cern.ch/rich/alignmentview.php?date=2016%2F06%2F19 . These plots can be looked at by the piquet.

The current settings for the alignment is that it still starts from the alignment that is currently in the database while the convergence criteria for the RICH1 secondary mirrors in y is increased to 0.3mrad and in z to 0.2mrad. All other convergence criteria stay at 0.1mrad. This should make the alignments converge within one iteration. The configurations can easily be changed at any time.

"Re: Rich alignment and new tck ?", 11 June 2016 to 23 June 2016

>On 12/06/2016 00:20, Paras Naik wrote:
>Hi Claire,
>Could you please let me know the following? I would appreciate it:
>1) What was required to fix the problem (e.g., only an environment variable in 
>AlignmentOnline, or a change to our code?)

I had completely to update to AlignmentOnlineDev_v11r0, which means 
our code is in
For this I needed to make sure that:
- I got the right versions of all packages to be compatible with each other
- I put the right path variables in the setup-files for the L0TCK, HLTTCK, L0TCKROOT
- I got it all to properly build (not as easy as it sounds ;) )
- I updated the path the drivers for the Iterator and the Analysers picked up 
from "/group/rich/sw/cmtuser/AlignmentOnlineDev_v10r6" to "/group/rich/sw/cmtuser/AlignmentOnlineDev_v11r0"
- I think I did something else as well but I can't remember O_o

>2) How the Iterator managed to get to the MirrAlign step without making the subdirectories 
>for MirrCombinFit. Hopefully you know, as I don’t yet see in the code how this could happen...

This can't really happen...
As far as I can see, the output that you emailed around did not fit with what I saw when 
I looked into the workdir.
Your output tells me this: we could not find the right TCK-thingy and therefore no events 
were processed and the rootfiles were written with 0 events in them. Then RichMirrCombinFit 
ran on that but couldn't produce a result-file (everything else should exists, like the folder and the 
conf file and such, but when there are no events no result-file is written). That could be an ok 
mechanism to prevent against complete nonsense, but we have not guarded against that.
Does this make sense, not sure I explained it perfectly.

I also think we might wanna preface all outputs the alignment makes with "RICHX alignment:" 
cause it's a freaking nightmare trying to find the output in the clusterlogs :D


>On 14/06/2016 14:33, Paras Naik wrote:
>Hi Claire,
>I would like to know please if you checked out AlignmentOnlineDev_v11r0 and the packages 
>via git, svn/getpack, or you just copied things over from v10r6. This is just so I can properly 
>keep track of changes to the code with respect to the git versions. At some point we have 
>to change our process completely to git, as of course AlignmentOnline will probably 
>be moved to git, if it hasn’t been already.

Ohhhh, I do not remember since I started this process a while ago. I guess it would be
 good to recommit at least the richmirroralignmentonline at some point (maybe after 
making some small changes).

>That sounds like a reasonable explanation for what happened… except that the root
 file was actually 2 megabytes in size… but maybe that’s just the overhead >size for 
a bunch of empty histograms.

Yes, since these are histograms and not a tree the size of the rootfile will not change
 much depending on the number of entries (Chris was really angry last year when I 
made this mistake :D ).

>Lastly, suppose I wanted to incorporate minor changes to the RichMirrorAlignmentOnline 
>code that is actually in AlignmentOnlineDev_v11r0 now (I would parallel the changes in
> git of course). For example adding the printouts and the ELOG thing that Giulio has 
>done for the other alignment. Would it be OK for me to compile it when the alignment
> is not running? (i.e. say I did it 2 hours after the previous alignments were run). Does
> the “multiUserCompile” shell script [or whatever I called it] still work? If not could it be 
>updated with whatever commands you used to get it to build?

I think you should be able to change between the alignment running, you just have to
 make sure that it really runs etc. (sometimes it might compile but not run btw).
I dont see why the multiuser compile file should not work... I have not tested it though. 
The 'problem' is only the very first build.

>Ah one more thing, I guess I should take the list you just gave me and put it on the twiki 
>somewhere, under the heading “How to change Alignment Online versions” unless 
>you have already done this somewhere on the Online stuff TWiki.

I have not done this.


Return to the Top of this TWIki

Advice from Roel

Various words of wisdom from Roel

"Re: Issue with loading HPD QE" E-mail thread, 19 April 2017-ish

Advice 1

To look at the histograms being produced by one of the subfarms, pointing the presenter at one of the subfarms and see if you get anything:

$> presenter -P LHCbA -D hltf12

Tools -> Page Editor Online

Click on hltf12 in the top right to see if the adder (or other jobs) show up, open them, tick some histograms (if available), right-click and select "Add checked to page".

If you don't get any tasks with histograms or histograms, you can try seeing with $> HistTest.exe Adder hltf12 NB: Paras never got this command to work That gives a list of task names, select the one you want, and run with that as a first argument to see how many histograms there are.

Advice 2

At the level of the iterator, you can print what you want from the python.

If you want to be sure you see everything that is printed by a single process, you can log in to the host where it run (the iterator runs on hlt02) and start: $> errlog -m $HOST

If that command is now known, you'll have to: $> source /group/online/dataflow/scripts/shell_macros.sh

If you think the alignment worker is stalled, you can try to log in to the node as user online and start $> top press 'c' to see long command lines. If the letter in the column marked S is not R, but D, it's probably stuck on some cvmfs operation.

Remember the PID (process-id) of the process you are interested in from the listing in top, or use $> ps -lf -u online to find it.

If the letter is R, but no CPU is consumed, you can try to attach gdb to it. The gdb version that comes with lcg 88 is recent and has python support which is rather useful when debugging the iterator

$> lb-run LCG/88 bash --norc $> export PYTHONPATH=/home/raaij/software/Python-2.7.10/Tools/gdb:$PYTHONPATH $> /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/gdb/7.12.1/x86_64-centos7-gcc62-dbg/bin/gdb --args YOURPID (gdb) python import libpython press ctrl+d (gdb) py-bt

For the analyzers, this can be simplified to: $> lb-run LCG/88 bash --norc $> ps -lf -u online find the PID of the process $> /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_88/gdb/7.12.1/x86_64-centos7-gcc62-dbg/bin/gdb --args YOURPID (gdb) thread apply all bt

If the traceback shows that thread processing events (probably thread 1) is somewhere in the reconstruction sequence, do (gdb) continue wait a little bit and press ctrl+c (gdb) thread apply all bt

The processing thread should now be somewhere different. If not, try again, and if it's still in the same place, it's probably hanging there.

Advice 3 : (move to RUN troubleshooting)

To be practical, if it's just a few nodes, just exclude them and run with the rest.

If new nodes with problems pop up or it looks systematic to for example an entire subfarm, it helps the sysadmins if you report it with a mail to lbonsupp@cernNOSPAMPLEASE.ch, which will create a ticket. Don't do this for individual nodes, there are generally a lot of tickets and a few broken nodes is not urgent.

Cheers, Roel

Advice from Roel : (move to RUN troubleshooting)

If the worker jobs are in state RUNNING and still consuming CPU, they are still processing events. If most jobs are already paused for a while, and only few are still running, perhaps the maximum number of events is too high and there are a few nodes that happen to have a lot of events for the range of runs you have selected. If all of the jobs are PAUSED, but no histograms are arriving after a few minutes, there might be a problem with the adders.

pre-automization RICH mirror alignment plan

It has been decided that our full automatic update sequence (Rich mirror alignment decison, Decision sent to Clara, Clara sends to HLT2, and it is immediately used by HLT2) is postponed after middle of August.

It is agreed to run in the following way for the next 3 weeks:

(1) The Rich alignment will continue to run automatically, take a decision if needed or not to update. If the latter the new version of the alignment (new V) is given a temporary file name. (2) Regardless of the decision the current V will be sent to Clara. (3) The current V will thus be immediately propagated to HLT.

Rich alignment experts (Sam or Matt, maybe both when possible) should check the output of the alignment for each fill (~ every 12 hours). If the Cherenkov angle resolution changes more than the variation between the 2 magnet polarity (Rich1 >0.050 mrad; Rich2 >0.008 mrad) or the variation of the constants is > 2 times the set tolerances (i.e. the number on any mirror on the first page of AlignMonitor.pdf is 2 or higher), Sam and Matt should discuss with Claire and Rich experts (Chris J., Mike, Silvia Gambetta) and Silvia Borghi (phone 160973) the need for applying an update. From the mirror alignment side we should certainly update in either of these cases.

If an update should be applied, Sam or Matt should move the temporary V to v{N+1}.xml. It will be used from the next fill.

SPECIAL CASE: After the magnet polarity switch, HLT automatic will stay on, but Clara’s automatic reading of the update will be OFF. The alignment experts should provide the new version to be used for the new magnet polarity and re label it v{N+1}.xml. This should be provided within 1 day after the first fill (with at least 1500 b) with the new polarity. If this polarity switch is foreseen in advance, the rich mirror alignment experts should be informed. The new version numbers will be sent to Clara by email to be used by HLT. Please note, any change of polarity is not seen in the next 2 weeks. If possible, an email will be sent to the rich alignment expert and hlt piquet a bit in advance. Otherwise a polarity change warning will come up in the email sent at the start of an alignment.

In case of doubts for this procedure, please contact Silvia Borghi (Operations Coordinator) on the phone 160973.

More on the XML files and this procedure

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatdocx Framework_for_distributed_alignment_-_Oct_24_2014.docx r1 manage 177.7 K 2014-10-24 - 16:37 ParasNaik Run 2 Framework for Distributed Alignment
PDFpdf Framework_for_distributed_alignment_-_Oct_24_2014.pdf r1 manage 307.5 K 2023-01-14 - 13:54 ParasNaik Run 2 Framework for Distributed Alignment
JPEGjpg IMG_20141016_135607.jpg r1 manage 1846.0 K 2014-10-23 - 16:57 ParasNaik Run 1 RICH Mirror Alignment Software Flowchart (green arrows mean info comes straight from the CondDB) drawn by Sam Harnew
PDFpdf SoftwareMirrorAlignment_2013-08-06.pdf r1 manage 1127.5 K 2013-08-06 - 18:29 AnatolySolomin Old Mirror Alignment note (2013), the most recent Mirror Alignment note is linked to in the "Documentation" section above
PDFpdf Workflow_new.pdf r1 manage 28.2 K 2017-02-28 - 22:42 ParasNaik New proposed mirror alignment workflow for Run II
PDFpdf draftHLTmapchapter-AndrewCook.pdf r1 manage 3737.1 K 2014-10-30 - 22:30 ParasNaik Description of the HLT map intended for implementation into and HLT1 line designed to fill our histograms sufficiently to perform the alignment (from a draft of Andrew Cook's Thesis)
Texttxt new_printout.txt r1 manage 52.1 K 2017-03-14 - 15:19 ParasNaik  
Edit | Attach | Watch | Print version | History: r278 < r277 < r276 < r275 < r274 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r278 - 2023-01-14 - ParasNaik
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback