JIRA tasks

JIRA and Kanban Board

All of our tasks to be done with the mirror alignment should be posted to JIRA, software that we use for issue tracking and project management.

At-a-glance the status of our JIRA tasks progress can be viewed on our Kanban board.

List of all completed tasks

• Closed or Resolved
  • Move Detector conditions from CondDB to Online (List of constants) (LHCBCNDB-629)
  • Deploy first RICH1 and RICH2 mirror alignment updates for 2016 (LHCBCNDB-587)
  • Prepare steps to move RICH mirror alignment conditions to ONLINE DB (LHCBCNDB-584)
  • Update RICH mirror alignment (LHCBCNDB-560)
  • Preliminary RICH alignment for 2015 (LHCBCNDB-547)
  • Migrate 2017 processing to use new RICH ('RichFuture') reconstruction sequence (LBOPG-70)
  • RICH mirror alignment conditions: moving them to the online database (LBOPG-46)
  • Add XQilla to AlignmentOnline (LHCBALIGN-18)
  • HLT Line for RICH Mirror Alignment (LHCBPS-1371)
  • Fix hard coded path in RichMirrorAlignmentGanga (RICH-4)
  • Switch online mirror alignment software to use project AlignmentOnline v11r3 (RICH-7)
  • Prepare mirror alignment to deliver constants towards the Online DB (RICH-9)
    • subtask: Transfer the conditions from the xml files that HLT2 is using to the Offline CondDB in order to be used by Offline processing (RICH-15)
  • Remove some localdisk Alignment area data files (RICH-20)
  • Get magnet polarity and fill number into the mirror alignment (RICH-21)
  • Prepare mirror alignment to run on 2017 data / using info from git DB (RICH-22)
  • Store hlt02 and/or LHCbA log information with the mirror alignment results (RICH-23)
  • Determine early RICH1 alignment with first 2017 startup data (RICH-25)
  • Obtain the first RICH mirror alignments of 2017 (RICH-27)
  • Move combAndMirrSubsets file from /group/rich/AlignmentFiles to package Rich/RichMirrorAlignmentOnline (RICH-29)
  • Full automation of the mirror alignment procedure (RICH-30)
  • Use minimum histogram population criteria, instead of minimum number of events, to decide when not to automatically update the mirror alignment (RICH-34)
  • Auto-write in the RICH logbook (ELOG) and Auto-email from the alignment
  • Upgrade of Online code to use quad stop tolerances provided by RichMirrAlign v19r1
    • (though should double-check later that regularizationMode and stopToleranceMode are included)
  • Back propagate RichMirrorAlignmentOnline from the pit into git, then always work on the code remotely and pull/checkout the git version at the pit that you want to use
  • Add warnings if the “NGE" file (or other expected files) is missing
  • Change “9” to the actual number of processes instead of having it be hardcoded, as there is only one subprocess (not 9) when there are fixed magnification coefficients.
  • Put the time the alignment takes into the Summary file
  • Copy the files from the workDir to the MirrorAlignments directory in a way that preserves the original timestamps of the files
  • Unhacked RichRecQC; we now rely on the default version
  • Incorporate quad stopTolerance for Primary and Secondary mirrors Y and Z
  • Update to AlignmentOnline v11r3

We should also know what each v{N}.xml is in /group/online/alignment/Rich{N}/MirrorAlign, and then maintain a record of this somehow for future v{N}.

• We’ll start from v4.xml this year for both mirror alignments (Unless we can think of a reason to start with a “zeroed" xml this year).

List of tasks not on JIRA

Most recently active/open/closed tasks should be listed towards the top of each section.

• • GhostProb cut now at 0.5 in Rich/RichAlignment. Can we go down lower, even as far as to 0.1? Or do we lose too many photons? (can run tests)
  • Move from the simple HLT line to one based on Andrew's windows (HLT map) or a similar principle
  • Make this select individual tracks from the windows, rather than whole events.
  • Consider the possibility to use muons to perform the mirror alignment (https://indico.cern.ch/event/477525/)
  • Incorporate the required changes to make RichMirrorAlignmentOnline compatible with the newest MirrCombinFit v17r0
    • see https://gitlab.cern.ch/lhcb/Panoptes/commit/2eefbbe5d188d3317e087183b03f7e01365406f9 for list of changes
  • Move the whole mirror alignment to RichFuture
  • Parallelize the fitting histos part of the alignment, as it is currently all done on only one machine
  • Consider running the alignment on MC (Take any MC, say from B Hadron stream; see if it is possible to run HLT line on this and get enough data)
    • Important for determining best set of mirror combinations for the mirror alignment (and thus also which mirror combinations' magnification factors should be stored in the Online CondDB)

Studies that we should definitely be doing

• Test LS method with different regularizations
• Possibly try mixed sample alignments (combine mag up and mag down data) or other sorts of integrated alignments
• Proper toy studies to know how precise and accurate our method is (suggested by Claire).
  • We should really know the limitations of the method, also to know how to pick the convergence criteria and how to understand changes that we see.
  • Here are some questions that we should be able to answer (for a given number of events):
    • What is our accuracy/precision on the misalignments on the detector-plane?
    • What is our accuracy/precision on the magnification coefficients?
    • What is out accuracy/precision on the actual mirror tilts
    • How big does a change have to be to be detected by the new solution method?
    • How big of a mirror tilt would actually be "visible" in the magnification coefficients?
  • Not knowing the answers to these questions makes the more practical questions we have difficult to answer.
  • Do some of these studies already exist? If so, we must collect them in a certain place and summarize the results for everyone to know.

Things we should "always" be doing

• Improvements to the core packages, Rich/RichMirrAlign and Rich/RichMirrCombinFit
• Refine the Online (and not-Online) driver scripts, Rich/RichMirrorAlignmentGanga and Rich/RichMirrorAlignmentOnline
• Ongoing maintenance of all code

Novel work-flow based on resolution-oriented stop decision

Anatoly has designed a new, [next-to] optimal work-flow, that is based on the resolution-oriented stop decision (see the attached flow-chart). Please feel free to propose better texts in the boxes of the flowchart.

The main features are:

1. The decision to stop the iterations is taken when |Y/Z tilts| of all mirror combinations in use, are less than stopTolerance == a certain fraction of the nominal sigma, say 1/4. (Anatoly already wrote, presented and committed this earlier)

2. The work-flow also will make good use of storing the magnification factors in the Online DB.

3. One of the central ideas is to spare re-evaluation of the magnification factors by first using the existing ones, and only if the stopTolerance is not satisfied, refresh them.

Later on, we can determine the optimal stopFraction by setting it to very small value and then letting the framework to iterate until the "equilibrium" is reached, that is when the tilts only fluctuate around some level, but not improve. That level will be the basis for the stopFraction choice. Unfortunately, it will generally depend on the statistics, but as long as we use fixed number of events, it will be OK.

Implement full capability of (quad stop-tolerances) from RichMirrAlign v19r1

We implemented the capability of (quad stop-tolerances) RichMirrAlign v19r1. We haven't yet switched to using the newer regulariaztionMode (probably not necessary) and we are not using fractions of nominal sigma for the tolerances yet either.

This issue has been solved; click for details... Hide

The main things (more details are below in the release notes) are: 

1. The regularization in the LS method is accounted for the magnifications. 

2. According to my original suggestion, I added the ability to use four (not two, as Paras did) separate stop tolerances for primary and secondary mirrors and for Y, Z rotations. 
And they are calculated from nominal sigma, its tolerance fraction and magnifications. Alternatively, if desired, they can be set by hand as options. 

Four, not two, is because of this: 

RICH1: 

averaged magnification factor priY_Y =  1.88 
averaged magnification factor secY_Y = -0.56 
averaged magnification factor priZ_Z =  2.13 
averaged magnification factor secZ_Z =  0.84 

stopTolerancePriY =  0.21 
stopToleranceSecY =  0.71 
stopTolerancePriZ =  0.19 
stopToleranceSecZ =  0.47 

RICH2: 

averaged magnification factor priY_Y =  2.06 
averaged magnification factor secY_Y = -1.06 
averaged magnification factor priZ_Z =  1.85 
averaged magnification factor secZ_Z =  0.59 

stopTolerancePriY =  0.08 
stopToleranceSecY =  0.17 
stopTolerancePriZ =  0.09 
stopToleranceSecZ =  0.30 

3. Improved printouts. See:

4. Eliminated default values for RICH-dependent options. Otherwise it is easy to create a mess. If they are not found among the options, the program stops with a diagnostic. 

The changes on Claire's online code side will be minimal: just add the new options to the configurations

Here is the new entry in the release notes: 
----------------------------------------------------------------- 

2016-04-25 Rich/RichMirrAlign v19r0

This version to be released in PANOPTES v6r0. 
The regularization in LS method is accounted for the magnifications. 
Added the ability to use four separate stop tolerances for primary and secondary mirrors and Y, Z, calculated from nominal sigma, its tolerance fraction and magnifications, or set by hand. 
Improved printouts. 
Eliminated default values for RICH-dependent options.

2016-04-25 - Anatoly Solomin
- Incremented version to v19r0.

- In LS method the regularizing sum is now squares of "magnified"
  solution. Each component of the solution vector is multiplied by
  the respective magnification factor, averaged over all mirror
  combinations and calibrational tilts.

- Introduced option nominalResolutionSigma: RICH-dependent nominal
  Cherenkov angle distribution sigma; no default here.

- Introduced option stopSigmaFraction (default 0.5): tolerance
  for total tilt of mirror pair in terms of fraction of the
  nominal sigma.

- Introduced option printMode: results print style mode
  0 - Paras one-glance style
  1 - Anatoly detailed style
  2 - both: first Anatoly, then Paras
  -2.17 will be printed as: 1-> -2.1, 0-> -2, while
  -0.97 will be printed as: 1->  -.9, 0->  0

- Introduced four (instead of two) RICH-dependent stop tolerances:
  stopTolerancePriY
  stopTolerancePriZ
  stopToleranceSecY
  stopToleranceSecZ

- Introduced option stopToleranceMode: 0(default)->calculate them
  from nominal sigma, average magn factors and stopSigmaFraction, e.g.
  stopTolerancePriY = 0.5*nominalResolutionSigma*stopSigmaFraction/
  (averaged magnification factor priY_Y)
  1->use those set up by hand in the conf file.

- Respective changes in the conf files. Examles in <...>/files.

- Eliminated default values for RICH-dependent options.

- Fixed and improved the printouts.

- Fixed previous entries of these release notes. This text is "pre",
  and therefore it should be decently formatted by hand in the
  editor: Ctrl+j. Otherwise it looks ugly in the browser.
  Also convention is to summarize only per Panoptes release, not per
  version increment of a package.

2016-04-09 - Paras Naik
 - Added the ability to use separate stopTolerances for primary
   and secondary mirrors, If stopToleranceSec = 0, then all
   mirrors use stopTolerance to determine the threshold for
   alignment in this case, the printouts show the updated
   compensations in mrad, as used to be the case Otherwise the
   threshold for primary mirrors is stopTolerance and the
   threshold for secondary mirrors is stopToleranceSec in this new
   case, the printouts now show the updated compensations divided
   by the tolerance.

Return to the Top of this TWIki

Further information for tasks not yet in JIRA

Move to HLT Map

Things that could be done with regards to speed, that we should be thinking about even if we are already “fast enough":

• Some sort of pre-selection / post-selection related to bin counting in histograms
     (unlikely, hard to get analyzers to communicate when each histo bin is full)
• Full HLT map selecting single tracks
  • A HLT map for whole events

Regarding the last point, why not put an HLT map for whole events into the HLT line now. It would to save us such a factor of time [and, the related weighted selection mechanism would be a first step for an eventual HLT map anyway]. The key to doing this well is to perform a study with:

• (A) an unbiased L0-filtered sample… but it may not have enough events
• (B) the existing HLT line sample… have tons of events, just would need to run something over them to figure out the regions and weightings based on the event trigger track. (and of course, remember that our weighted selection gets performed on already HLT line selected data, not on unbiased data)

One might not ask why not go straight to the map… the only reason is that the map requires isolating and reconstructing individual tracks, and this does not. If along the way isolating tracks becomes possible we can always change gears. So the "whole events" version can possibly be a “simple” extension of the full HLT map.

RichFuture

We need to change all our mirror alignment options to use the RichFuture reconstruction.

Also, Rich/RichRecQC will eventually be completly retired and removed from the release stack. The replacement packages already exist in the future stack see Rich/RichFutureRecMonitors and Rich/RichFutureRecCheckers. These have the MC-free and MC based monitoring respectively. The equivalent of the resolution monitors are already there. Look at https://gitlab.cern.ch/lhcb/Rec/blob/master/Rich/RichFutureRecMonitors/python/RichFutureRecMonitors/ConfiguredRecoMonitors.py which is the equivalent python module to that from RichRecQC.

Parallelize the fitting histos part of the alignment

Parallelize the fitting histos part of the alignment, as it is currently all done on only one machine. [This has long been our list, and may start to make a difference if we do manage to bring the reconstruction time down. No one is yet assigned to this task.] Right now it takes quite some time especially for RICH2.

Claire knows the tracker people do something similar, where the analysers have 2 different tasks which are (1) reconstruction and (2) some Kalman stuff. Claire said she would try and have a chat with Roel about how feasible our case would be during A&S week. (I imagine that it's either fairly easy or impossible, but I might be wrong.) [WAS IT FEASIBLE????]

Ah that makes me think: so the tracker people perform their Kalman operation for each analyzer on the data that is stored on that particular analyzer and then when thats done mash it all together (they can do that). Now for us the analyzers would have to have access to the rootfiles... Claire will ask Roel if that is feasible. [WAS IT FEASIBLE????]

Make 1D plots of deltaTheta distributions

We already output PNG files of the 2D deltaTheta vs. phi distributions. We should make 1D projections as well.

> On 5 Jun 2017, at 17:47, Christopher Jones <jonesc@hep.phy.cam.ac.uk> wrote:
>
> Just a thought, but it would be really interesting if you could take those 2D plots and project them onto the y-axis, to recover a ‘normal’ resolution plot.
> Do you think this could be tried ?

Document the use of nm

On 18 Jun 2017, at 10:55 pm, Paras Naik <Paras.Naik@bristol.ac.uk> wrote:

Hi Chris,

WriteRichAlignmentsTool is here:

/group/online/dataflow/cmtuser/AlignmentRelease/Alignment/AlignTrTools/src/

I am not sure exactly where to look in the InstallArea though. There libAlignTrTools.so shows up in all cases, but that doesn’t mean WriteRichAlignmentsTool is in it

nm is your friend here…

Online:gw20 /group/online/dataflow/cmtuser/AlignmentOnlineDev_v11r3 > nm -s InstallArea/x86_64-centos7-gcc62-opt/lib/libAlignTrTools.so | c++filt | grep WriteRich 0000000000182290 t _GLOBAL__sub_I_WriteRichAlignmentsTool.cpp 000000000018b060 t _GLOBAL__sub_I_WriteRichCalibrationsTool.cpp 000000000018d3e0 t _GLOBAL__sub_I_WriteRichHPDQEsTool.cpp 00000000002571e0 W ZN11ToolFactoryI19WriteRichHPDQEsToolE6createIN5Gaudi13PluginService7FactoryIP8IAlgToolJRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESF_PK10IInterfaceEEEJSF_SF_SI_EEENT_10ReturnTypeEDpOT0 00000000001d47c0 W ZN11ToolFactoryI23WriteRichAlignmentsToolE6createIN5Gaudi13PluginService7FactoryIP8IAlgToolJRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESF_PK10IInterfaceEEEJSF_SF_SI_EEENT_10ReturnTypeEDpOT0 000000000023b950 W ZN11ToolFactoryI25WriteRichCalibrationsToolE6createIN5Gaudi13PluginService7FactoryIP8IAlgToolJRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESF_PK10IInterfaceEEEJSF_SF_SI_EEENT_10ReturnTypeEDpOT0 0000000000254940 T WriteRichHPDQEsTool::initialize()

So yes, its there.

I still suspect the issue is the CMTCONFIG Beat’s script is using. If its SLC6 then it will not be picking up anything from the checkout above, and only useing the released version of the v11r3 Alignment project.

Chris

Document that the safeguard to getting all of the calibrations available before running our alignment, is a 1 hour delay for the RICH1 alignment to start

Claire knows the whole history, but now we check that the calibrations exist within our alignment