Difference: LHCbRichMirrorAlignRunOnline (1 vs. 36)

Revision 362019-08-30 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18
Line: 294 to 294
  something complaining about some Configuration.py file in the OnlineDev something something directory ==> something in our own Configuration file is wrong
Added:
>
>

Specific errors

In LHCbA log, or when running on a single node, if you see Aug29-182253[ERROR]hltc0507: CTRL(HLTC0507_A_Controller): start: FAILED Summary:FINISHED:1 TIMEOUT:1 NodeAdder_0 ..... you may want to mention it to Beat and Clara, but it is usually not a problem

In LHCbA log, or when running on a single node, if you see something like Aug29-165429[ERROR]hltc0222: GaudiOnlineExe.exe(LHCbA_HLTC0222_AligWrk_0): RawDecoder: Rich::DAQ::RawDataFormatTool:: 'Unknown Level0 hardware ID 181' | L1HardID=10 Ingress=1 Input=1 [DecodeRawRichHLT] StatusCode=FAILURE This means that you are using the wrong DB tags.

In LHCbA log, or in a single node if you see Aug30-144215[WARN] hlt02: cp: cannot stat ‘/group/online/dataflow/options/LHCbA/HLT/LHCbA_HLT01_HLT.py’: No such file or directory Aug30-144215[WARN] hlt02: cp: cannot stat ‘/group/online/dataflow/options/LHCbA/LHCbA_HLT01_HLT.opts’: No such file or directory Just ignore it

 

Errors and warnings that are not a/our problem

There will be serveral harmless warnings from the analyzers during configuring and running such as:

Revision 352017-07-25 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18
Line: 192 to 192
  The "RunInfo" button on the left will change to "running". Wait a few seconds and see what happens to the "HLT" button. Sometimes it goes straight to running and sometimes it will be back at "Ready". In that case just select "start run" again.
Changed:
<
<
*10-11b. In lieu of the two steps (or three, if the farm is not allocated yet), if the runs and alignment are selected you can run the alignment in one step by switching the AUTOPILOT from OFF to ON. However if there is a bug in the code or some other problem this may cause weird loops to occur; if you see this happening switch AUTOPILOT from ON to OFF. And then try to diagnose and repair the problem following the guidelines below, and next time don't use the AUTOPILOT.
>
>
10-11b. Autopilot: In lieu of the two steps (or three, if the farm is not allocated yet), if the runs and alignment are selected you can run the alignment in one step by switching the AUTOPILOT from OFF to ON. However if there is a bug in the code or some other problem this may cause weird loops to occur; if you see this happening switch AUTOPILOT from ON to OFF. And then try to diagnose and repair the problem following the guidelines below, and next time don't use the AUTOPILOT.
  12. Now lean back and relax wink
Line: 204 to 204
 
  • You should have already done this, but if not: The alignment will have gone into READY, click on READY at the top and pick RESET
  • Wait until the alignment goes into NOT READY
Changed:
<
<
  • Now do whatever you need to do to prepare for the next alignment: it may be changing run numbers, changing configuration (REMEBER YOU MUST COMPILE), or changing the starting XML file
  • CONFIGURE the alignment
  • when it is all back in READY, START_RUN
>
>
  • Now do whatever you need to do to prepare for the next alignment: it may be changing run numbers, changing configuration and/or or changing the starting XML file (REMEMBER YOU MUST COMPILE if you change any of the code, including the Configuration)
  • Steps 10-11a or 10-11b
  14b. If you want to stop doing mirrror alignments, send a RESET, and free the alignment farm again
Line: 218 to 217
 

Monitoring the procedure

Changed:
<
<
There is no monitoring in place yet.
>
>
The results of each iteration is saved, this includes XML files and histograms produced at the end of each iteration (more details below).
 
Changed:
<
<
The results of each iterations are saved, this includes xml files and histograms (more details below). For the moment the monitoring should be performed manually looking at the output of the fits, especially looking at the plots!!!
>
>
For the moment the monitoring should be performed manually looking at the output of the fits, especially looking at the plots!!! See AlignmentView and you should be familiar with all of the information on LHCbRichMirrorAlignShiftInfo to help you evaluate an alignment.
  The output of each perfomed alignment will be saved under /group/online/AligWork/MirrorAlignments/Rich' + str(whichRich)/{YYMMDD_hhmmss} .
Added:
>
>
 

Xml files location

  • The xml files produced in the job you are running will be saved to /group/online/AligWork/MirrorAlignments/Rich' + str(whichRich)/{YYMMDD_hhmmss}
Line: 230 to 230
 

Histograms location

Changed:
<
<
The histograms are produced automatically when running the alignment. We have one root file for each iteration, they are at: the histograms are at /hist/Savesets/2015/LHCbA/ Nomenclature convention: ex. /hist/Savesets/2015/LHCbA/AligWork_Rich1/07/10/AligWrk_Rich1-1569160001-20150710T100630-EOR.root
  • Rich1 is the name of the activity (Rich1, Rich2, Muon Tracker or Velo)
>
>
The histograms are produced automatically when running the alignment. We have one root file for each iteration, they are at: the histograms are at /hist/Savesets/2015/LHCbA/ Nomenclature convention: e.g. /hist/Savesets/2015/LHCbA/AligWork_Rich1/07/10/AligWrk_Rich1-1569160001-20150710T100630-EOR.root
  • Rich1 is the name of the activity (Rich1, Rich2, Muon, Tracker or Velo)
 
  • 156916 is the first run number in the run list
  • 0001 is the number of iteration
  • 20150710T100630 is the time when the file has been generated.
Line: 241 to 241
 

Known issues and workarounds

Changed:
<
<
Everything takes ages. That's normal smile
>
>
Everything takes ages. That's normal smile
 

Actual errors

When taking the LHCbA portion, get a warning that says that PARTAlign is owned by ECS:LHCbARunControl (or someone else).

If you have to take LHCbA, and you get a warning like this chances are it LHCbA taken by someone else.

Changed:
<
<
Typically this partition should not be taken by someone else unless they are using it or the alignment is in automatic mode (otherwise you should have been warned that LHCbA is occupied and you shouldn't use it).
>
>
Typically this partition should not be taken by someone else unless they are using it or the alignment is in automatic mode (otherwise you should have been warned that LHCbA is occupied and you shouldn't use it).
 
Changed:
<
<
If none of the subfarms are able to be included and PARTAlign remains OFFLINE, you should try to force exclude PARTAlign, deallocate everything and then you can try to allocate yourself LHCbAlign from top level.
>
>
If none of the subfarms are able to be included and PARTAlign remains OFFLINE, you should try to force exclude PARTAlign, deallocate everything and then you can try to allocate yourself LHCbA from top level.
 To force exclude, you can right click on the lock to go into expert mode and then select the option.
Changed:
<
<

Alignment goes into state “READY,” but the alignment hasn’t converged (ALWAYS CHECK!)

>
>

Alignment goes into state “READY,” but the alignment hasn’t converged (ALWAYS CHECK! You should get an email if it converged, and in the ELOG there should be a summary report)

 
Changed:
<
<
  • Just click on “start run" again (no reconfiguring, etc.!!!)
>
>
  • In this case you probably follow Step 10a but not Step 10b. Just click on “start run" again (no reconfiguring, etc.!!!)
 

PARTAlign_Master READY, nodes READY

Line: 285 to 287
  Sometimes a node goes into error state without apparent reason. This will prevent the others to get to the next iteration. Just exclude the node and click "start run" again.
Changed:
<
<

Red herrings

>
>

Red herrings

  something with Options "../../jobs " something ==> a problem in your python-configurations
  • use tail -n 20 /clusterlogs/partitions/LHCbA/daq/LHCbA.log to see more details of the error (can increase the number 20 if need be)
Line: 304 to 306
  TrackBestTrackCreator.Fitter:TrackMasterFitter:: The WARNING message is suppressed: 'unable to fit the track' ...
Added:
>
>
and others discussed on LHCbRichMirrorAlignShiftInfo
 

After the alignment has run

Changed:
<
<
After the alignment has run check by hand in the working directory if it has converged and/or look at the output of the iterator in the hlt02 errorlogger.
>
>
After the alignment has run check by hand in the working directory if it has converged and/or look at the output of the iterator in the hlt02 error logger / logging files (discussed in LHCbRichMirrorAlignCodeOnline).
  In order to make sure that no nonsense happend in the alignment you should:
Line: 316 to 320
 

Log files

Changed:
<
<
There will be a log file produced each day that contains the output of all analyzers that ran from the LHCbA partition ( = the alignment farm).
>
>
There are the hlt02 error logger / logging files (discussed in LHCbRichMirrorAlignCodeOnline). They should show up in the YYYYMMDD_HHMMSS directory after a successful alignment, otherwise they are in the Logging directory.

There will be a log file produced each day (being written to all the time) that contains the output of all analyzers that ran from the LHCbA partition ( = the alignment farm).

  This log file will be at /clusterlogs/partitions/LHCbA/daq/LHCbA.log (under plus). Older log files are in clusterlogs/partitions/LHCbA/daq/old.
Changed:
<
<
Note that these log files contain the entire output from all analyzers of this day (including other alignments etc). They are extremely large and probably not very practial.
>
>
Note that these log files contain the entire output from all analyzers of this day (including other alignments etc). They are extremely large and probably not very practial. However you can filter the code using grep to isolate one node (e.g. hltf0119) to see how that node is configured and if everything ran (Brunel produces a table of processed events).
 

Explanation of files in the alignment folder (from Claire's email, 28 September 2015)

Changed:
<
<
For the moment all alignments and alignment-attempts are being stored in plus under /home/cprouve/RichAlignment/alignments

In there are folders for different dates which in turn contain alignments for Rich1 and/or Rich2.

>
>
In /group/online/AligWork/MirrorAlignments/ there are folders for different dates which in turn contain alignments for Rich1 and/or Rich2.
  The alignment procedure starts from the xml file like
Line: 336 to 340
  RichRecQCHistos_rich1_Mp6Wi4.0Fv3Cm2Sm1_online_Collision15_i0.root
Changed:
<
<
The the starting xml-file (Rich1CondDBUpdate _Mp6Wi4.0Fv3Cm2Sm1_online_Collision15_i0.xml) is taken and tilts of +0.3 and -0.3 are applied around the y- and the z-axis for the primary and secondary mirrors respectively which gives 8 new xml files with endings like (these are for the calculation of the magnification factors):
>
>
If not using predetermined magnification factors The the starting xml-file (Rich1CondDBUpdate _Mp6Wi4.0Fv3Cm2Sm1_online_Collision15_i0.xml) is taken and tilts of +0.3 and -0.3 (+/-0.7 for RICH1) are applied around the y- and the z-axis for the primary and secondary mirrors respectively which gives 8 new xml files with endings like (these are for the calculation of the magnification factors):
  Rich1CondDBUpdate_Mp6Wi4.0Fv3Cm2Sm1_online_Collision15_pri_negYzerZ_i0.xml
Line: 344 to 348
  RichRecQCHistos_rich1_Mp6Wi4.0Fv3Cm2Sm1_online_Collision15_pri_negYzerZ_i0.root
Changed:
<
<
The 2D fits are performed on each of the 9 root-files and the output stored in the folders which also contain the plots for each mirrorpair with the line of the 2D fit:
>
>
The 2D fits are performed on each of the 8 (+1 for untitled) root-files and the output stored in the folders which also contain the plots for each mirrorpair with the line of the 2D fit:
  Rich1MirrCombinFit_Mp6Wi4.0Fv3Cm2Sm1_online_Collision15_i0
Line: 356 to 360
  Rich1MirrMagnFactors_Mp6Wi4.0Fv3Cm2Sm1_online_Collision15_pri_negYzerZ_i0.txt
Added:
>
>
If using predetermined magnification factors Then the files like the one just above are already pre-made.
 After that the actual mirror-corrections are determined and summarised in the file (here only one per iteration):

Rich1MirrAlignOut_i0.txt

Line: 372 to 378
 

To figure out how many events were actually run over in the alignment

Added:
>
>
This info should be in AlignmentView now. You can also use the following:
 It can be very useful to figure out how many events were actually run over in the alignment for each RICH detector. It is hard to get this number directly, because our HLT Line provides both Rich1 and Rich2 samples to the online RICH mirror alignment together.

First, create a file called numTriggers.C

Revision 342017-07-24 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18
Line: 46 to 46
 
Creating a new directory for predetermined magnification factors

Choices for fixed magnification factors should always be stored in

Changed:
<
<
/group/rich/AlignmentFiles/MagnifFactors/Rich1/ and /group/rich/AlignmentFiles/MagnifFactors/Rich2/.
>
>
/group/rich/AlignmentFiles/MagnifFactors/Rich1/ and /group/rich/AlignmentFiles/MagnifFactors/Rich2/ and its subdirectories.
 
Changed:
<
<
If you want to use a new set of magnification factors make sure that the files have the same names as those that can currently be found within the substructure of those directories.
>
>
If you want to use a new set of magnification factors, make sure that the files have the same names as those that can currently be found within the substructure of those directories.
 
Changed:
<
<
Say you determine new magnification factors for RICH1, by allowing them to float. If you want to install them, follow the following procedure on a plus machine (substituting your alignment name for 20170612_004007, and the appropriate nameString instead of "_Mp6Wi8.0Fm5Mm2Sm0_online_Collision17"):
>
>
Say you determine new magnification factors for RICH1, by allowing them to float in a test alignment. If you want to install them, follow the following procedure on a plus machine (substituting your alignment name for 20170612_004007, and the appropriate nameString instead of "_Mp6Wi8.0Fm5Mm2Sm0_online_Collision17"):
 
  • cd /group/rich/AlignmentFiles/MagnifFactors/Rich1/
  • ls /group/online/AligWork/MirrorAlignments/Rich1/20170612_004007
Line: 71 to 71
  You will need to look at two different errorloggers:
Changed:
<
<
1. For the analyzers:
errorLog LHCbA
>
>
1. For the analyzers:
errorLog LHCbA OR in a separate terminal window less +F --follow-name  /clusterlogs/partitions/LHCbA/daq/LHCbA.log
 
Changed:
<
<
You can change the output level in the little settings window: navigate (with the arrow keys) up to "Severity for messages" and cycle though the options with the ">" key-combination. When you have found the one you want simply hit "enter". This can be done at any time during the running (the message window might need a while to catch up with the command though).
>
>
If using the errorLog, you can change the output level in the little settings window: navigate (with the arrow keys) up to "Severity for messages" and cycle though the options with the ">" key-combination. When you have found the one you want simply hit "enter". This can be done at any time during the running (the message window might need a while to catch up with the command though).
 
Line: 85 to 85
 
Changed:
<
<

Run the alignment

>
>
Alternatively, in a separate terminal window less +F --follow-name  /group/rich/AlignmentFiles/Logging/Rich1_hlt02.log should track the RICH1 mirror alignment when it is running, and pause otherwise. Change 'Rich1' to 'Rich2' for RICH2.

Run the mirror alignment

  0. Create an FSM launch script on ui. You only need to do this once (unless there is a new version of WinCC)
Line: 121 to 123
  source fsm.sh
Changed:
<
<
2. Right-click on LHCb_Align
>
>
2. Right-click on LHCb_Align when the FSM panel shows up
 
Changed:
<
<
3. Take the partion: Only take the partition if no one else is using the alignment!
>
>
3. Take the partition: Only take the partition if no one else is using the alignment!
  click "take" and then wait for a smaller new panel to show up (this may take a while, just be patient). In the new panel just click "*dismiss*".

Changed:
<
<
4. Choose your alignment: select the activity from the menu on the right of the Run Info panel.
>
>
When you have taken the partition, a complete loss of internet connection will log you off the FSM panel.

During data taking, you will not need to take the partition. The partition will be set up at the pit. The bad news is now you effectively share the partition with everyone else. The good news is if you lose internet connection, your alignments will still run (unless one of the automated alignments overrides your alignment, automated alignments have priority).

4. Choose your alignment: select the activity from the menu on the right of the Run Info panel. This will always be Alignment|Rich1 or Alignment|Rich2.

 
Changed:
<
<
5. Allocate:
>
>
5. If the state is NOT ALLOCATED, then Allocate:
 
Changed:
<
<
6. Reserve alignment farm: Put your name into the panel and click the button to reserve the alignment.
>
>
6. Reserve alignment farm: Put your name into the panel and click the button to reserve the alignment. This is so other people know you are there and that they shouldn't touch anything until you are done. Also, if they need the alignment they can contact you. During commissioning periods we use a TimeTable to let everyone know when we have booked the farm (the link may need to be updated each year).
 
Changed:
<
<
7. Select runs: Select the run range to use in the alignment. This procedure will be fully automated after an initial testing period. The runs to use are the first ones of each fill (if they have enough events). Click on "Choose runs for alignment".
>
>
7. Select runs: Select the run range to use in the alignment by clicking on "Choose runs for alignment". Note that the fill numbers are also displayed for your convenience. The runs to use are the first ones of each fill (if they have enough events), unless the event count is being cut off (currently 3M) in which case use all of them in a fill.
 
Changed:
<
<
You can look at which runs you want in the run database.
>
>
Information about all runs and fills can be found in the run database.
 
Changed:
<
<
Now select the runs you want to run over and click "Ok". Make sure to pick "Rich" from the list at the top! The number of events for each run listed in this window is the combination of events provided from the RICH1 and RICH2 mirror HLT lines. So it is not exact but should give an order-of-magnitude (approximate split was intended to be 50% RICH1 and 50% RICH2, but with substantial error bars). To get the number of events actually processed, use the numTriggers.C script.
>
>
Make sure to pick "Rich" from the list at the top! Now select the runs you want to run over and click "Ok". The number of events for each run listed in this window is the combination of events provided from the RICH1 and RICH2 mirror HLT lines. It is not quite the perfect mix, right now it is 57% RICH1 and 43% RICH2, but with substantial error bars [we aimed for 50-50, so not so bad]). To get the number of events actually processed, use the numTriggers.C script or just look in AlignmentView.
 
<!-- runsSelector.png -->
Changed:
<
<
8. Verify / select farms to run on: Click on "HLT" which will open up an overview of the HLT subfarms and nodes.
>
>
8. Verify / select farms to run on: You don't have to do this usually. However to see check the status of the HLT subfarms, click on "HLT" which will open up an overview of the HLT subfarms and nodes.
 
Changed:
<
<
The selection-panel on the left show the included/ excluded nodes and subfarms. In order to include/exclude subfarms/nodes select them in the panels and then click the button with the error pointing toward the left/right. The click the "Include"/"Remove" button. This might take a while, just be patient.
>
>
The selection-panel on the left show the included/excluded nodes and subfarms. In order to include/exclude subfarms/nodes select them in the panels and then click the button with the error pointing toward the left/right. Then also click the "Include"/"Remove" button. This might take a while, just be patient.
 
Changed:
<
<
The subfarm marked in red are not working right now (for whatever reasons) and can and shall not be included.
>
>
The subfarms marked in red are not working right now (for whatever reasons) and can and shall not be included.
 
Changed:
<
<
When all available nodes are included their number should be between 1200 and 1800. If it is significantly less (~500 or so) you need to deallocate and reallocate the partition. If the problem persists go and complain to an online expert.
>
>
When all available nodes are included their number should be between 1200 and 1800. If it is less than 1200, you need to DEALLOCATE and re-*ALLOCATE* the partition. This almost always solves the problem, and can also solve other problems with the mirror alignment (e.g. if LHCbA gets into a bad state and nothing seems to work exactly right). If the problem persists go and complain to an Online piquet (or expert [Beat Jost, Clara Gaspar, Markus Frank]) and/or the Alignment Piquet. See the ShiftDB for who's on shift.
 
Changed:
<
<
9. Look at the status of the subfarms and nodes: Click the "PARTAlign" button in the HLT panel
>
>
9. Look at the status of the subfarms and nodes: Also something you don't have to do usually. If need be though, click the "PARTAlign" button in the HLT panel.
 
Line: 172 to 178
 
Changed:
<
<
10. Start alignment by configuring: The configuring process might take a couple of minutes and some nodes might take significantly longer than others. That's just normal. If one node seems crazy slow or fails you can remove it as shown in step 7.
>
>
Choose steps 10a AND 11a if an alignment has not been performed in a while on the farm. Otherwise try step 10-11b.
 
Changed:
<
<
There is a time-limit set of how long the configuring is allowed to take. It is possible to take longer and if nodes take longer their status will appear as error. Give it time and verify in the errorlogger that something is still happening. If it takes to long or seems stuck just remove it.
>
>
10a. Start alignment by configuring: The configuring process might take a couple of minutes and some nodes might take significantly longer than others. That's just normal. If one node seems crazy slow or fails you can remove it as shown in step 7.
 
Changed:
<
<
Do not change the status of things anywhere else than in this specific panel (unless you are an expert). Don't give commands to subfarms or nodes directly!
>
>
There is a time-limit set of how long the configuring is allowed to take (6-8 minutes). It is possible this will take longer, and if nodes take longer their status will appear as ERROR, but other nodes in the HLT panel will already be READY. Give it time and verify in the LHCbA errorlogger that something is still happening. If some node(s) take(s) too long or seems stuck for some reason, just remove it.

Do not change the status of things anywhere else than in this specific panel (unless you are an expert). Don't give commands to subfarms or nodes directly! (But also don't worry, it's very unlikely a DEALLOCATE and ALLOCATE won't fix it)

 
Changed:
<
<
11. Start the run: From the same dropdown panel select "start run".
>
>
11a. Start the run: From the same dropdown panel select "start run".
  The "RunInfo" button on the left will change to "running". Wait a few seconds and see what happens to the "HLT" button. Sometimes it goes straight to running and sometimes it will be back at "Ready". In that case just select "start run" again.
Added:
>
>
*10-11b. In lieu of the two steps (or three, if the farm is not allocated yet), if the runs and alignment are selected you can run the alignment in one step by switching the AUTOPILOT from OFF to ON. However if there is a bug in the code or some other problem this may cause weird loops to occur; if you see this happening switch AUTOPILOT from ON to OFF. And then try to diagnose and repair the problem following the guidelines below, and next time don't use the AUTOPILOT.
 12. Now lean back and relax wink
Changed:
<
<
If you want to you can go into the working directory and see the files being written =)
>
>
If you want to you can go into the working directory and see the files being written, or watch the log files scroll by =)
 
Changed:
<
<
13a. If you want to stop aligning, free the alignment farm again
>
>
13. If the alignment is successful, everything goes into READY. However once this happens, it is very important that you execute a RESET command as soon as possible after the alignment goes into READY. We have no idea why this is not done automatically, but since it is not, you have to do it.
 
Changed:
<
<
>
>
14a. If and only if you want to perform another alignment, you MUST RECONFIGURE by following these steps:
 
Changed:
<
<
13b. If you want to perform another alignment, you MUST RECONFIGURE by following these steps

  • The alignment will have gone into READY, click on READY at the top and pick RESET
>
>
  • You should have already done this, but if not: The alignment will have gone into READY, click on READY at the top and pick RESET
 
  • Wait until the alignment goes into NOT READY
  • Now do whatever you need to do to prepare for the next alignment: it may be changing run numbers, changing configuration (REMEBER YOU MUST COMPILE), or changing the starting XML file
  • CONFIGURE the alignment
  • when it is all back in READY, START_RUN
Added:
>
>
14b. If you want to stop doing mirrror alignments, send a RESET, and free the alignment farm again

  • You should have already done this, but if not: The alignment will have gone into READY, click on READY at the top and pick RESET
  • DEALLOCATE the farm
  • If you took the farm, then you should click on the lock and release the farm.

 

Monitoring the procedure

There is no monitoring in place yet.

Line: 239 to 255
 

Alignment goes into state “READY,” but the alignment hasn’t converged (ALWAYS CHECK!)

Changed:
<
<
  • Just click on “start run” again (no reconfiguring, etc.!!!)
>
>
  • Just click on “start run" again (no reconfiguring, etc.!!!)
 

PARTAlign_Master READY, nodes READY

Revision 332017-07-24 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18
Line: 12 to 12
 

XML-file to start with

Changed:
<
<
If you want to start from a specific xml-file (i.e a specific mirror alignment) you need to copy it to the right location. Just follow the instructions here. Please remove the alignment afterwards or make sure the alignment that is currently in the database gets put back into the correct folder with the latest version number.
>
>
If you want to start from a specific xml-file (i.e a specific mirror alignment) you need to copy it to the right location. Just follow the instructions here.
<!-- Please remove the alignment afterwards or make sure the alignment that is currently in the database gets put back into the correct folder with the latest version number. -->
 
Changed:
<
<
The alignments that were put into the database should also be kept at this location: /group/rich/AlignmentFiles/databaseAlignments
where you can pick them up whenever you want to.
>
>
[Paras will remove later, as this is no longer needed, but first a backup may need to be made] The alignments that were put into the database should also be kept at this location: /group/rich/AlignmentFiles/databaseAlignments, where you can pick them up whenever you want to.
 

Check Configuration file

Changed:
<
<
There is only one confuration file, Configuration.py. Since everybody is using the same configuration file, we have created a subdirectory where the configuration file is located called SavedConfigurations, that contains .txt versions of various configurations that you can write over Configuration.py before you start your alignments. Please read README.txt. Always create new configurations in the SavedConfigurations directory, update SavedConfigurations/README.txt, then overwrite Configuration.py with your new configuration. It is necessary to check if all parameters (especially starting-iteration) are set correctly. Make sure all your variables are set consitently.
More info about the Configuration file.
After changing the Configuration file apply a do_configure and do_install for the changes to be picked up by the package.
>
>
There is only one configuration file, Configuration.py. If you want to use the same configuration as used in a previous alignment, since 2017 you can just go into that alignment's directory and retrieve the version of Configuration.py used. Of course, if you wish to repeat the alignment, you need to make sure that new Configurables have not been added since!

Since the Configuration file in the master branch of Panoptes is the Configuration file, where Configuration.py exists we have created a subdirectory called SavedConfigurations , that contains .txt versions of various configurations that you can write over Configuration.py before you start your alignments. Please read README.txt. Always create new configurations in the SavedConfigurations directory, update SavedConfigurations/README.txt, then overwrite Configuration.py with your new configuration.

It is necessary to check if all parameters (especially starting-iteration, though we almost always start from Iteration 0) are set correctly. Make sure all your variables are set consistently.

You can read more about the Configuration file.

NOTE after changing the Configuration file locally, you have to recompile the code (including a git fetch Panoptes and a git lb-checkout your branch of Rich/RichMirrorAlignmentOnline first) for the changes to be implemented in the next mirror alignment.

 

Magnification factors

Changed:
<
<
You can chose between calculating the magnification factors on-the-fly for each iteration or using predetermined ones. In order to chose you have to modify the value of magnifCoeffMode in the Configuration file.
>
>
You can chose between calculating the magnification factors on-the-fly for each iteration or using predetermined ones. In order to chose you have to modify the value of magnifCoeffMode in the Configuration file.

Find out more about using predetermined Magnification factors.

Magnification factors on-the-fly
 
Changed:
<
<
Magnification factors on the fly
>
>
Set the value of magnifCoeffMode in the Configuration file to 2.
 
Deleted:
<
<
Set the value of magnifCoeffMode in the Configuration file to 2.
 
Predetermined magnification factors
Changed:
<
<
Set the value of magnifCoeffMode in the Configuration file to 0.
Then you have to provide the files containing the predetermined magnification factors in a directory and point towards it in magnifDir in the Configuration file.
>
>
Set the value of magnifCoeffMode in the Configuration file to 0.
Then you have to provide the files containing the predetermined magnification factors in a directory and point towards it in magnifDir in the Configuration file.
 
Creating a new directory for predetermined magnification factors
Line: 52 to 63
 
    • for file in Rich1*; do mv "$file" ${file//_i4/_predefined}; done
  • for file in Rich1*; do mv "$file" ${file//_Mp6Wi8.0Fm5Mm2Sm0_online_Collision17/}; done
  • rm Rich1MirrMagnFactors_predefined.txt
Changed:
<
<
>
>
  For RICH2 of course you want to substitute "Rich2" for "Rich1" in the above procedure.

Revision 322017-07-09 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18
Line: 20 to 20
  There is only one confuration file, Configuration.py. Since everybody is using the same configuration file, we have created a subdirectory where the configuration file is located called SavedConfigurations, that contains .txt versions of various configurations that you can write over Configuration.py before you start your alignments. Please read README.txt. Always create new configurations in the SavedConfigurations directory, update SavedConfigurations/README.txt, then overwrite Configuration.py with your new configuration. It is necessary to check if all parameters (especially starting-iteration) are set correctly. Make sure all your variables are set consitently.
More info about the Configuration file.
After changing the Configuration file apply a do_configure and do_install for the changes to be picked up by the package.
Changed:
<
<

Magnification coefficients

>
>

Magnification factors

 
Changed:
<
<
You can chose between calculating the magnification coefficients on the fly for each iteration or using predetermined ones. In order to chose you have to modify the value of magnifCoeffMode in the Configuration file.
Magnification coefficients on the fly
>
>
You can chose between calculating the magnification factors on-the-fly for each iteration or using predetermined ones. In order to chose you have to modify the value of magnifCoeffMode in the Configuration file.

Magnification factors on the fly
  Set the value of magnifCoeffMode in the Configuration file to 2.
Changed:
<
<
Predetermined magnification coefficents
>
>
Predetermined magnification factors

Set the value of magnifCoeffMode in the Configuration file to 0.
Then you have to provide the files containing the predetermined magnification factors in a directory and point towards it in magnifDir in the Configuration file.

Creating a new directory for predetermined magnification factors

Choices for fixed magnification factors should always be stored in
/group/rich/AlignmentFiles/MagnifFactors/Rich1/ and /group/rich/AlignmentFiles/MagnifFactors/Rich2/.

If you want to use a new set of magnification factors make sure that the files have the same names as those that can currently be found within the substructure of those directories.

Say you determine new magnification factors for RICH1, by allowing them to float. If you want to install them, follow the following procedure on a plus machine (substituting your alignment name for 20170612_004007, and the appropriate nameString instead of "_Mp6Wi8.0Fm5Mm2Sm0_online_Collision17"):

  • cd /group/rich/AlignmentFiles/MagnifFactors/Rich1/
  • ls /group/online/AligWork/MirrorAlignments/Rich1/20170612_004007
    • Look at the last iteration number and remember it.
  • mkdir 20170612_004007
  • cd 20170612_004007
  • cp /group/online/AligWork/MirrorAlignments/Rich1/20170612_004007/summary.txt .
  • (substituting your alignment number for _i4)
    • cp /group/online/AligWork/MirrorAlignments/Rich1/20170612_004007/Rich1MirrMagnFactors*_i4.txt .
    • for file in Rich1*; do mv "$file" ${file//_i4/_predefined}; done
  • for file in Rich1*; do mv "$file" ${file//_Mp6Wi8.0Fm5Mm2Sm0_online_Collision17/}; done
  • rm Rich1MirrMagnFactors_predefined.txt
  • change the magnifDir in the Configuration file
 
Changed:
<
<
Set the value of magnifCoeffMode in the Configuration file to 0.
Then you have to provide the files containing the predetermined magnification coefficients in a directory and point towards it in magnifDir in Configuration file. There are some magnification coefficients already provided in a default folder at /group/rich/AlignmentFiles/MagnifFactors/Rich1/.
If you want to use your own magnification coefficents make sure that the files have the same names as those in the default directory.
>
>
For RICH2 of course you want to substitute "Rich2" for "Rich1" in the above procedure.
 

Errorloggers

Revision 312017-07-07 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18

Revision 292017-06-10 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18
Line: 194 to 194
 

Actual errors

Added:
>
>

When taking the LHCbA portion, get a warning that says that PARTAlign is owned by ECS:LHCbARunControl (or someone else).

If you have to take LHCbA, and you get a warning like this chances are it LHCbA taken by someone else. Typically this partition should not be taken by someone else unless they are using it or the alignment is in automatic mode (otherwise you should have been warned that LHCbA is occupied and you shouldn't use it).

If none of the subfarms are able to be included and PARTAlign remains OFFLINE, you should try to force exclude PARTAlign, deallocate everything and then you can try to allocate yourself LHCbAlign from top level. To force exclude, you can right click on the lock to go into expert mode and then select the option.

 

Alignment goes into state “READY,” but the alignment hasn’t converged (ALWAYS CHECK!)

  • Just click on “start run” again (no reconfiguring, etc.!!!)

Revision 282017-05-16 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
Changed:
<
<
-- ClaireProuve - 2015-07-18
>
>
-- ParasNaik - 2017-05-16 -- ClaireProuve - 2015-07-18
  LHCbRichMirrorAlign

Revision 272017-04-24 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ClaireProuve - 2015-07-18
Line: 52 to 52
 

Run the alignment

Changed:
<
<
0. Create an FSM launch script on ui. You only need to do this once
>
>
0. Create an FSM launch script on ui. You only need to do this once (unless there is a new version of WinCC)
  ssh -Y ui

Revision 262017-04-19 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ClaireProuve - 2015-07-18
Line: 42 to 42
 
Changed:
<
<
2. For the iterator:
ssh -Y hlt02
errlog -m hlt02
>
>
2. For the iterator:
ssh -Y hlt02
source /group/online/dataflow/scripts/shell_macros.sh
errlog -m hlt02

NOTE: here you want the errlog command and NOT the errorLog command!!!

  This panel is a bit moody and will sometimes not output the messages at the right time. It might take a while, or wait for the next iteration or wait til the program finished.

Revision 252017-04-18 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ClaireProuve - 2015-07-18
Line: 227 to 227
 

Red herrings

something with Options "../../jobs " something ==> a problem in your python-configurations

Added:
>
>
  • use tail -n 20 /clusterlogs/partitions/LHCbA/daq/LHCbA.log to see more details of the error (can increase the number 20 if need be)
  something complaining about some Configuration.py file in the OnlineDev something something directory ==> something in our own Configuration file is wrong

Revision 242017-04-14 - ParasNaik

Line: 1 to 1
 
META TOPICPARENT name="LHCbRichMirrorAlign"
-- ClaireProuve - 2015-07-18
Line: 40 to 40
  You can change the output level in the little settings window: navigate (with the arrow keys) up to "Severity for messages" and cycle though the options with the ">" key-combination. When you have found the one you want simply hit "enter". This can be done at any time during the running (the message window might need a while to catch up with the command though).
Changed:
<
<
>
>
  2. For the iterator:
ssh -Y hlt02
errlog -m hlt02

This panel is a bit moody and will sometimes not output the messages at the right time. It might take a while, or wait for the next iteration or wait til the program finished.

Changed:
<
<
>
>
 

Run the alignment

Changed:
<
<
1. Open the panel from an ui or plus machine:
>
>
0. Create an FSM launch script on ui. You only need to do this once
  ssh -Y ui
Changed:
<
<
/group/online/ecs/Shortcuts311/LHCb/ECS/ECS_UI_FSM.sh
>
>
then create a new file called fsm.sh and put the following into it:

<!--/twistyPlugin twikiMakeVisibleInline-->
<!-- SyntaxHighlightingPlugin -->
if [[ "$WCCOA_DIR" =~ "WinCC_OA/3.11" ]]; then
    echo "Starting the FSM on WinCC-OA 3.11"
    /group/online/ecs/Shortcuts311/LHCb/ECS/ECS_UI_FSM.sh 
elif [[ "$WCCOA_DIR" =~ "WinCC_OA/3.15" ]]; then
    echo "Starting the FSM on WinCC-OA 3.15"
    /group/online/ecs/Shortcuts315/LHCb/ECS/ECS_UI_FSM.sh 
else
    echo "You need to edit fsm.sh and add functionality for this WinCC version:"
    echo $WCCOA_DIR
fi
<!-- end SyntaxHighlightingPlugin -->
<!--/twistyPlugin-->

1. Open the panel from an ui or plus machine:

ssh -Y ui (if you haven't already)

 
Changed:
<
<
2. Right-click on LHCb_Align
>
>
source fsm.sh

2. Right-click on LHCb_Align

  3. Take the partion: Only take the partition if no one else is using the alignment!

click "take" and then wait for a smaller new panel to show up (this may take a while, just be patient). In the new panel just click "*dismiss*".

Changed:
<
<
>
>
  4. Choose your alignment: select the activity from the menu on the right of the Run Info panel.
Changed:
<
<