DCS and SLIMOS quick reference

TIP Useful phone numbers

Reminder of some operational instructions

Hot Topics and Known Issues

This section shows all known issues which are not yet imported to the users guide or which will be valid only for a limited period of time. Check the topics before you start your shift. Please note the date of last update - the content might have changed since your last shift.

SubsystemTopicAddedModified
Hot Topics:
DCSLock RED in the handshake panel 16-NOV-2010
DCSSound generated by UI at startup or while clicking on objects15-NOV-2010
DCSSpecial care to be taken during Safe/Supersafe Transitions11-Nov-2010
DCSFSM Lock ownership issues3-Nov-201011-Nov-2010
DCSModified safety matrix panel25-Oct-2010
DCSSlight modification of handshake sequence03-Sep-201014-Sep-2010
Known Issues:
TPCTPC state is MOVING _READY for transition from SUPERSAFE to SAFE30-Oct-2010
SLIMOSHigh reading on RAMSES sensor05-Jul-2010
TOFTOF in DOWNLOADING while ALICE in SAFE11-Jun-2010
SLIMOSRecurrent DSS alarms09-Jun-2010
DCSWarnings from system Overview tool15-Apr-201006-Jun-2010
DCSRDB Freezing15-Apr-201014-Oct-2010
Operational instructions:
DCSControl of the ALICE permits, Post Mortem, BCM trends & BCM FSM configuration1-Sept-20104-Nov-2010
DCSHow to open a FSM lock if the owner is not you24-Oct-2010
DCSInstructions related to ALICE SAFE/SUPERSAFE06-Aug-201025-Oct-2010
HMPHMP HV trip cleaning instructions06-Aug-2010
DCSInstructions for magnet operation06-Aug-2010
DCSInstructions for setting-up the alert screens06-Jun-2010

Reminder of some operational instructions

ALERT! One of the most important tasks of the DCS operator is to make sure the experiment state is compatible (safe) with respect to the LHC activities. For the current operational procedures and instructions refer to the SafeSupersafe page and to the DCS Users Guide.

ALERT! No commands shall be given to any detector without explicit approval of that detector and/or Shift Leader/run-coordinator.

ALERT! The FSM control panel shall be closed after use, and no action shall be performed through this window other than taking or releasing detectors (on their request). When closing the FSM control panel, answer NO on the question in the popup if you want to release the hierarchy.

ALERT! Do not forget to warn the CTP shifter to change the clock (to LHC clock at the end of the handshake, as indicated on the panel, and to local clock at the dump or loss of beams)

DONE Do not leave the DCS console unattended for prolonged periods. Read through these instructions at the start of each of your shifts (at least check the revision date at the bottom...), as they might have been updated since your last shift.

DONE Keep track of all events and operations (timestamped) in the DCS shifter logbook: DCS ELOG (Accessible only from the DCS network). Create a single entry per shift under the tab 'Shifter Reports' (using 'New', then update it regularly with 'Edit; Submit').

DONE Keep track of all ALERTS in the DCS shifter logbook: DCS ELOG (Accessible only from the DCS network). Create a single entry per shift under the tab 'Alert Tracking' (using 'New' and Submit').

DONE Add all major events also in the ALICE electronic logbook (log in with your own cern account). Also add an 'end of shift' report (extract the most important events from the DCS ELOG).

DONE Keep the 'DCS Message' on the DCS monitoring page updated (see how)

DONE Make sure that all systems are taken, either by central DCS or (in exceptional cases, with permission of Shift Leader) locally by the detector. There should be no ‘orphan' systems. LHC, Services, Run Unit and TRI_DCS are always taken by central DCS.

DONE Do not acknowledge alarms (both DSS and Alarm Screen) ‘blindly', unless explicitly instructed by that detector to do so. Try to find the detector expert and check with him.

DONE As SLIMOS watch the DSS and CSAM status in the alarm box on the ‘big screen' in the ACR.If one of these turns red, look on the safety screens for details. If you are not the SLIMOS, inform the SLIMOS. In the rare event there is no SLIMOS assigned on your shift, call Andrι Augustinus (163534) or Arturo Tauro (162529).

* A. FSM node for ALI_DCS\LHC\DCS_BCM is DEAD

    • 0. Log in on alidcscom042 as dcssys
    • 1. Restart FSM manager
    • 2. Verify in the Advanced FSM Control panel that nodes BCM_SIDE_A, BCM_LV_SIDE_A, BCM_SIDE_C, BCM_LV_SIDE_C are DEAD. If these nodes are not in DEAD then stop them using red buttons.
    • 3. Verify in the normal FSM Control panel that the BCM_SIDE_A and BCM_SIDE_C nodes are excluded from the LHC\DCS_BCM FSM node.
    • 4. If FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1 is in the state different from READY:
      • Send the FSM command GO_STBY_CONFIGURED and then GO_READY for the LHC\DCS_BCM\BCM_TELL1\myTell1_BCM node. Enter “2” as the value of parameter in the popup window. The state should change to READY.
    • 5. Go to the LHC\LHC_ALI_HANDSHAKE FSM node. If it is in state PROBLEM_REPORTED then send FSM command ACKOWLEDGE. The state should change to STANDBY.
    • 6. Verify there is no active errors/fatals from the lhc_exchange project on the alarm screen.
    • 7. Send an email to Mateusz with your comments about this procedure

  • B. Problems with control and monitoring of TELL1 (e.g. with operating the beam permits)
    • 1. Verify whether the operator recovery procedure was done properly
    • 2. Verify the status of DIM server directly on CCPC:
      • i. Login to alidcscom181 (this machine is also called alidcsexp)
      • ii. Start command prompt and type a command: permits. This will log you to the CCPC
      • iii. Execute sudo service ccserv status
      • iv. If the status is not OK execute sudo service ccserv restart
      • v. Each operation of restarting DIM server requires also restarting of PVSS man-ager
      • Background information: the actions on CCPC are performed on alidcsaux001 in the context of cc user

  • C. Manual restart of PVSS manager if restart procedure from the panel LHC\DCS_BCM\BCM_TELL1 fails:
    • 1. Login to alidcscom042
    • 2. Open pvss console and log
    • 3. Stop bcmMonitorSensors manager – it should restart automatically
    • 4. Verify in the pvss log whether the transient recorder started successfully – if not consult with Giacinto/Claudio/Mateusz restarting of the whole project

  • D. Checking the status of injection permits directly from the board via SSH:
    • 1. Go to LHC\DCS_BCM\BCM_TELL1 panel
    • 2. Click Generate report for section CCPC status. This operation last ~30 seconds. Please verify the timestamp of report.

  • E. Manual control of Tell1 (ONLY FOR DEBUGGING AND IN EMERGENCY):
    • The operation described below are not proven to be successful when they are ex-ecuted simultaneously with running CCSERV server. Nevertheless no problems were reported so far.
    • 1. Login to alidcscom181 (also called alidcsexp)
    • 2. Start command prompt and type a command: permits.
    • 3. After being transferred to the CCPC type following commands:
    • 4. cd cmdlnutils
    • 5. ls
    • 6. the list of scripts should be displayed (bcm_allowCirculation, bcm_allowInjection,…..)
    • 7. ./bcm_checkStatus
    • 8. The view provides status of TELL1 and can used to monitoring
    • 9. You can stop this by pressing ctrl-c
    • 10. You can operate the TELL1 – the most important commands:
      • a. ./bcm_allowCirculation
      • b. ./bcm_allowInjection
      • c. ./bcm_inhibitInjection
      • d. ./bcm_startTransientRecorder
      • Expert information: the actions on CCPC are performed on alidcsaux001 in the context of cc user

BCM basic recovery

IF

  • BCM rectangle in the Auxiliary Monitoring Zone in the left-bottom corner of the UI is not green
AND
  • you are not during the Post Mortem procedure (there is no Post Mortem popup panel and ALI_DCS\LHC\DCS_BCM FSM node is not in state DUMP)
THEN
  • BCM is failing

ACTIONS:

  • 1. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM and double click on it to open the panel:

  • 2. If the TELL1 on the top right side is in NO_CONTROL then
    • a. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1
    • b. Click the button A (…) to active the recovery buttons B and D
    • c. Click the button B (Restart all) and wait until C (Startup status) message switched to OK. Operation last ~120 seconds.
    • d. If C (Startup status) moves to ERROR press the button D (Restart PVSS manager) and wait until C (Startup status) message switched OK (operation last ~30 sec)
    • e. Open the FSM control panel (K) of the node ALI_DCS\LHC\DCS_BCM\BCM_TELL1 and send the command GO_STBY_CONFIGURED and then GO_READY and enter value 2 on the popup panel which appears.
    • f. The BCM_TELL1 node should move to state READY

If you do not succeed then call DCS On call.

Control of the ALICE permits, Post Mortem & BCM FSM configuration

News:

  • Please note that there is a new step: "Verify the BCM trends" in the "Post Mortem" and "Beam permit" procedures. The instructions are in the section: "How to verify the BCM trends"

Post Mortem procedure for DCS shifters

  • The diagram below presents the BCM system logic (upper half) and shifter's action (lower half) after Post Mortem occurs:

BCM_Processes.png

Manual control of the permits

  • Before arming the injection permits, make sure ALICE is SAFE/SUPERSAFE. Discuss with DCS On-call/ Shift Leader any exception.
  • Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1\myTell1_BCM ( A) and double click on it to open the panel. Click the button B (…) to open the popup panel with manual control of permits:

beam2.PNG.

  • Use the buttons C to operate the permits. Click the 'Enable' button to activate the permits removal buttons:

manual.PNG

Beam permit

If the beam background in ALICE is above a safe level then the BCM will dump the beam and make the beam permit FALSE (red).

IF

  • BEAM PERMIT IS FALSE (RED)

beam1.PNG

THEN

  • IT IS NECESSARY TO REARM THE BEAM PERMIT MANUALLY

ACTIONS:

  • 1. Inform Shift Leader

  • 2. If you are during the Post Mortem procedure (there is Post Mortem popup panel and ALI_DCS\LHC\DCS_BCM FSM node is in state DUMP) then wait until it finishes.

  • 3. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1\myTell1_BCM ( A) and double click on it to open the panel:
    beam2.PNG
  • 4. Click the button B (…) to open the popup panel with manual control of permits: beam3.PNG
  • 5. Press the button C. The beam (user) permit should move to TRUE (green). If you do not succeed then call DCS On call.
  • 6. Verify the BCM trends (instructions below).

How to verify the BCM trends:

  • 1. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1\myTell1_BCM ( A) and double click on it to open the panel: . Open the "Sensors" (B) tab. Use four buttons (C) to display the BCM trends one by one. nav2.PNG
    • a) For the RS32 trends: check if there is a peak of values on the plots that exceed the level of 20 nA (you should see a peak like D which corresponds to the time of the dump). Note the values of peaks
    • b) For the RS2 trends: check if there is a peak of values on the plots that exceed the level of 1000 nA. Note the values of peaks. plot_view.PNG

  • 2.
    • If there is NO peak above 20nA on the RS32 plots and NO peak above 1000nA on the RS2 plots and ALICE dumped the beam (the beam permit was removed) then call DCS Oncall.
    • If there is a peak above 20nA on RS32 plots OR a peak above 1000nA on RS2 and the beam permit was NOT removed then call DCS Oncall.
  • 3. Note in the shift report the values of the all the peaks.

Reminder of some BCM & Post Mortem procedures for DCS Shifters:

A. There are several situations when status of the ALICE permits may change:

  • LHC injection handshake,
  • CCC (via Shift Leader) asks for giving the permits manually,
  • Post Mortem
  • arming of the Beam Permit after Post Mortem triggered by ALICE
  • manual removing of the injection permits on Shift Leader request

Shifter should verify then on the alarm screen if there is no alarm related to lhc_exchange project. There are 2 obsolete alarms from lhc_exchange: FSM summary (from 2008 and 2009) - do not worry about them.

By default the manual rearming of the injection permits can be performed only if ALICE is SAFE/SUPERSAFE and CCC expects them. For some LHC tests when the injection is not planned the permits might be armed irrespective of the SAFE/SUPERSAFE state (only on Shift Leader explicit request!).

B. In case of communication problems (alarms, problem with giving the permits) the shifter can:

  • 1. Give the permits using emergency button if CCC is expecting them (TWIKI, FOLDER)
  • 2. Try to restore the communication using recovery button (DCS GUIDE, FOLDER)
  • 3. Check what is the true status of permits/transient recorder using manual report (on the panel ali_dcs\lhc\dcs_bcm\bcm_tell1) - upon Oncall request

DCS On call should be immediately informed.

POST MORTEM:

Post Mortem:

  • can occur anytime
  • it implicates the injection permits are removed automatically
  • if ALICE BCM caused the beam dump then also the Beam Permit is removed
  • Post Mortem last ~ 3 minutes

After the Post Mortem the shifter should check:

  • if permits were rearmed if necessary (i.e. ALICE is SAFE and ALICE is during injection handshake)
  • if there is no alarms related with LHC exchange project (up to 6 minutes after the Post Mortem procedure started)
  • the status of the Beam Permit. If it is red, then arm the Beam Permit manually, inform Shift Leader and DCS oncall (by email during the night)
  • peaks on the BCM trends

The injection permits are rearmed automatically after Post Mortem when:

  • ALICE is SAFE and
  • ALICE did not caused the beam dump and
  • LHC injection handshake is in 'injection' phase (the status is ‘LHC injection ready' or 'LHC injection imminent' after the command 'CONFIRM inject' has already been sent from ALICE)

Reminder of some BCM & Post Mortem procedures for DCS ON-CALLs:

In case of problems with communication of BCM (giving the permits, alarms)

1. Restart the communication:

  • Use recovery buttons (if shifter did not use them so far) (DCS GUIDE)
  • Restart of the PVSS manager bcmMonitorSensors with PVSS console on alidcscom042 (DCS GUIDE)
  • Restart the project - follow the instructions: Restart of the LHC_EXCHANGE project
  • Only if BCM experts are not available and it is absolutely critical: restart the VME with Tell1 (panel: ALI_DCS\LHC\DCS_BCM\BCM_INFRA\BCM_VME+ after few minutes restart the project + reupload thresholds table to Tell1
2. Write an email to Mateusz + Giacinto

The "CCPC is not accessible" alarm indicates communication problems. It can be removed manually in PARA on alidcscom042 - after successful recovery and when investigation is finished.

Configuration of BCM FSM

Please note that FSM nodes: BCM_SIDE_A and BCM_SIDE_C are excluded from LHC\DCS_BCM and should remain in state DEAD

bcm_dead.PNG

Emergency control of the permits

Use this procedure only if the normal BCM recovery fails and after DCS On-call approval

IF

  • There is a problem with giving the permits and LHC expects them (i.e. injection handshake is in underway), ALICE is SAFE/SUPERSAFE

THEN:

  • 1. Navigate to the FSM node ALI_DCS\LHC\DCS_BCM\BCM_TELL1\myTell1_BCM ( A).

  • 2. Click the button Emergency Control ( B).

  • 3. Follow the instructions from popup panel ( C).

emergency.PNG

  • go_stby_conf.PNG:
    go_stby_conf.PNG
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng go_stby_conf.PNG r1 manage 43.5 K 2010-11-08 - 16:39 MateuszLechman  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2011-04-06 - MateuszLechman
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback