• A. FSM node for ALI_DCS\LHC\DCS_BCM is DEAD
    • 0. Log in on alidcscom042 as dcssys
    • 1. Restart FSM manager
    • 2. Verify in the Advanced FSM Control panel that nodes BCM_SIDE_A, BCM_LV_SIDE_A, BCM_SIDE_C, BCM_LV_SIDE_C are DEAD. If these nodes are not in DEAD then stop them using red buttons.
    • 3. Verify in the normal FSM Control panel that the BCM_SIDE_A and BCM_SIDE_C nodes are excluded from the LHC\DCS_BCM FSM node.
    • 4. If FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1 is in the state different from READY:
      • Send the FSM command GO_STBY_CONFIGURED and then GO_READY for the LHC\DCS_BCM\BCM_TELL1\myTell1_BCM node. Enter 2 as the value of parameter in the popup window. The state should change to READY.
    • 5. Go to the LHC\LHC_ALI_HANDSHAKE FSM node. If it is in state PROBLEM_REPORTED then send FSM command ACKOWLEDGE. The state should change to STANDBY.
    • 6. Verify there is no active errors/fatals from the lhc_exchange project on the alarm screen.
    • 7. Send an email to Mateusz with your comments about this procedure

  • B. Problems with control and monitoring of TELL1 (e.g. with operating the beam permits)
    • 1. Verify whether the operator recovery procedure was done properly
    • 2. Verify the status of DIM server directly on CCPC:
      • i. Login to alidcscom181 (this machine is also called alidcsexp)
      • ii. Start command prompt and type a command: permits. This will log you to the CCPC
      • iii. Execute sudo service ccserv status
      • iv. If the status is not OK execute sudo service ccserv restart
      • v. Each operation of restarting DIM server requires also restarting of PVSS man-ager
      • Background information: the actions on CCPC are performed on alidcsaux001 in the context of cc user

  • C. Manual restart of PVSS manager if restart procedure from the panel LHC\DCS_BCM\BCM_TELL1 fails:
    • 1. Login to alidcscom042
    • 2. Open pvss console and log
    • 3. Stop bcmMonitorSensors manager it should restart automatically
    • 4. Verify in the pvss log whether the transient recorder started successfully if not consult with Giacinto/Claudio/Mateusz restarting of the whole project

  • D. Checking the status of injection permits directly from the board via SSH:
    • 1. Go to LHC\DCS_BCM\BCM_TELL1 panel
    • 2. Click Generate report for section CCPC status. This operation last ~30 seconds. Please verify the timestamp of report.

  • E. Manual control of Tell1 (ONLY FOR DEBUGGING AND IN EMERGENCY):
    • The operation described below are not proven to be successful when they are ex-ecuted simultaneously with running CCSERV server. Nevertheless no problems were reported so far.
    • 1. Login to alidcscom181 (also called alidcsexp)
    • 2. Start command prompt and type a command: permits.
    • 3. After being transferred to the CCPC type following commands:
    • 4. cd cmdlnutils
    • 5. ls
    • 6. the list of scripts should be displayed (bcm_allowCirculation, bcm_allowInjection,..)
    • 7. ./bcm_checkStatus
    • 8. The view provides status of TELL1 and can used to monitoring
    • 9. You can stop this by pressing ctrl-c
    • 10. You can operate the TELL1 the most important commands:
      • a. ./bcm_allowCirculation
      • b. ./bcm_allowInjection
      • c. ./bcm_inhibitInjection
      • d. ./bcm_startTransientRecorder
      • Expert information: the actions on CCPC are performed on alidcsaux001 in the context of cc user

BCM basic recovery

IF

  • BCM rectangle in the Auxiliary Monitoring Zone in the left-bottom corner of the UI is not green
AND
  • you are not during the Post Mortem procedure (there is no Post Mortem popup panel and ALI_DCS\LHC\DCS_BCM FSM node is not in state DUMP)
THEN
  • BCM is failing

ACTIONS:

  • 1. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM and double click on it to open the panel:

  • 2. If the TELL1 on the top right side is in NO_CONTROL then
    • a. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1
    • b. Click the button A () to active the recovery buttons B and D
    • c. Click the button B (Restart all) and wait until C (Startup status) message switched to OK. Operation last ~120 seconds.
    • d. If C (Startup status) moves to ERROR press the button D (Restart PVSS manager) and wait until C (Startup status) message switched OK (operation last ~30 sec)
    • e. Open the FSM control panel (K) of the node ALI_DCS\LHC\DCS_BCM\BCM_TELL1 and send the command GO_STBY_CONFIGURED and then GO_READY and enter value 2 on the popup panel which appears.
    • f. The BCM_TELL1 node should move to state READY

If you do not succeed then call DCS On call.

Control of the ALICE permits, Post Mortem & BCM FSM configuration

News:

  • Please note that there is a new step: "Verify the BCM trends" in the "Post Mortem" and "Beam permit" procedures. The instructions are in the section: "How to verify the BCM trends"

Post Mortem procedure for DCS shifters

  • The diagram below presents the BCM system logic (upper half) and shifter's action (lower half) after Post Mortem occurs:

BCM_Processes.png

Manual control of the permits

  • Before arming the injection permits, make sure ALICE is SAFE/SUPERSAFE. Discuss with DCS On-call/ Shift Leader any exception.
  • Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1\myTell1_BCM ( A) and double click on it to open the panel. Click the button B (…) to open the popup panel with manual control of permits:

beam2.PNG.

  • Use the buttons C to operate the permits. Click the 'Enable' button to activate the permits removal buttons:

manual.PNG

Beam permit

If the beam background in ALICE is above a safe level then the BCM will dump the beam and make the beam permit FALSE (red).

IF

  • BEAM PERMIT IS FALSE (RED)

beam1.PNG

THEN

  • IT IS NECESSARY TO REARM THE BEAM PERMIT MANUALLY

ACTIONS:

  • 1. Inform Shift Leader

  • 2. If you are during the Post Mortem procedure (there is Post Mortem popup panel and ALI_DCS\LHC\DCS_BCM FSM node is in state DUMP) then wait until it finishes.

  • 3. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1\myTell1_BCM ( A) and double click on it to open the panel:
    beam2.PNG
  • 4. Click the button B (…) to open the popup panel with manual control of permits: beam3.PNG
  • 5. Press the button C. The beam (user) permit should move to TRUE (green). If you do not succeed then call DCS On call.
  • 6. Verify the BCM trends (instructions below).

How to verify the BCM trends:

  • 1. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1\myTell1_BCM ( A) and double click on it to open the panel: . Open the "Sensors" (B) tab. Use four buttons (C) to display the BCM trends one by one. nav2.PNG
    • a) For the RS32 trends: check if there is a peak of values on the plots that exceed the level of 20 nA (you should see a peak like D which corresponds to the time of the dump). Note the values of peaks
    • b) For the RS2 trends: check if there is a peak of values on the plots that exceed the level of 1000 nA. Note the values of peaks. plot_view.PNG

  • 2.
    • If there is NO peak above 20nA on the RS32 plots and NO peak above 1000nA on the RS2 plots and ALICE dumped the beam (the beam permit was removed) then call DCS Oncall.
    • If there is a peak above 20nA on RS32 plots OR a peak above 1000nA on RS2 and the beam permit was NOT removed then call DCS Oncall.
  • 3. Note in the shift report the values of the all the peaks.

Reminder of some BCM & Post Mortem procedures for DCS Shifters:

A. There are several situations when status of the ALICE permits may change:

  • LHC injection handshake,
  • CCC (via Shift Leader) asks for giving the permits manually,
  • Post Mortem
  • arming of the Beam Permit after Post Mortem triggered by ALICE
  • manual removing of the injection permits on Shift Leader request

Shifter should verify then on the alarm screen if there is no alarm related to lhc_exchange project. There are 2 obsolete alarms from lhc_exchange: FSM summary (from 2008 and 2009) - do not worry about them.

By default the manual rearming of the injection permits can be performed only if ALICE is SAFE/SUPERSAFE and CCC expects them. For some LHC tests when the injection is not planned the permits might be armed irrespective of the SAFE/SUPERSAFE state (only on Shift Leader explicit request!).

B. In case of communication problems (alarms, problem with giving the permits) the shifter can:

  • 1. Give the permits using emergency button if CCC is expecting them (TWIKI, FOLDER)
  • 2. Try to restore the communication using recovery button (DCS GUIDE, FOLDER)
  • 3. Check what is the true status of permits/transient recorder using manual report (on the panel ali_dcs\lhc\dcs_bcm\bcm_tell1) - upon Oncall request

DCS On call should be immediately informed.

POST MORTEM:

Post Mortem:

  • can occur anytime
  • it implicates the injection permits are removed automatically
  • if ALICE BCM caused the beam dump then also the Beam Permit is removed
  • Post Mortem last ~ 3 minutes

After the Post Mortem the shifter should check:

  • if permits were rearmed if necessary (i.e. ALICE is SAFE and ALICE is during injection handshake)
  • if there is no alarms related with LHC exchange project (up to 6 minutes after the Post Mortem procedure started)
  • the status of the Beam Permit. If it is red, then arm the Beam Permit manually, inform Shift Leader and DCS oncall (by email during the night)
  • peaks on the BCM trends

The injection permits are rearmed automatically after Post Mortem when:

  • ALICE is SAFE and
  • ALICE did not caused the beam dump and
  • LHC injection handshake is in 'injection' phase (the status is ‘LHC injection ready' or 'LHC injection imminent' after the command 'CONFIRM inject' has already been sent from ALICE)

Reminder of some BCM & Post Mortem procedures for DCS ON-CALLs:

In case of problems with communication of BCM (giving the permits, alarms)

1. Restart the communication:

  • Use recovery buttons (if shifter did not use them so far) (DCS GUIDE)
  • Restart of the PVSS manager bcmMonitorSensors with PVSS console on alidcscom042 (DCS GUIDE)
  • Restart the project - follow the instructions: Restart of the LHC_EXCHANGE project
  • Only if BCM experts are not available and it is absolutely critical: restart the VME with Tell1 (panel: ALI_DCS\LHC\DCS_BCM\BCM_INFRA\BCM_VME+ after few minutes restart the project + reupload thresholds table to Tell1
2. Write an email to Mateusz + Giacinto

The "CCPC is not accessible" alarm indicates communication problems. It can be removed manually in PARA on alidcscom042 - after successful recovery and when investigation is finished.

Configuration of BCM FSM

Please note that FSM nodes: BCM_SIDE_A and BCM_SIDE_C are excluded from LHC\DCS_BCM and should remain in state DEAD

bcm_dead.PNG

Emergency control of the permits

Use this procedure only if the normal BCM recovery fails and after DCS On-call approval

IF

  • There is a problem with giving the permits and LHC expects them (i.e. injection handshake is in underway), ALICE is SAFE/SUPERSAFE

THEN:

  • 1. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1\myTell1_BCM ( A).

  • 2. Click the button Emergency Control ( B).

  • 3. Follow the instructions from popup panel ( C).

emergency.PNG

  • go_stby_conf.PNG:
    go_stby_conf.PNG

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng go_stby_conf.PNG r1 manage 43.5 K 2010-11-08 - 16:39 MateuszLechman  

This topic: Sandbox > MateuszLechmanSandbox > TestCopy
Topic revision: r4 - 2010-11-09 - MateuszLechman
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback