Shift Leader Duties and Instructions

Shift Leader Instructions feedback

If you have any feedback about this twiki page or documentation, don't hesitate to write us: cms-sldcs-shiftsupport@cernNOSPAMPLEASE.ch

Shift Leader Instructions during LS2, MWGR and COVID-19

During LS2, We have a reduced activities and sometimes different procedures at P5. Therefore, Please visit the following Twiki page.

Shift Leader Duties and Instructions during LS2, MWGR and COVID19

Shift Leader Instructions during normal data taking

Plan of the day at this link. Other short-term instructions are...

Links

GENERAL Type RC communications Date Valid CMS The SHIFT LEADER CHECKLIST for collisions has been updated recently. Please make sure you're aware of the latest changes and follow it carefully. Until further notice TRG The instructions for selecting prescale columns can be found in the Prescales Twiki, with a corresponding summary table. In cosmics, this depends on the random rate and in circulating the column is adjusted throughout the run such that the physics rate is always between 12 and 50 Hz. For collisions, the recommended column will change dependent on the LHC conditions (number of bunches, energy, PU) and the trigger shifter should confirm the correct column with the HLT DOC (165575 or 66278) if in doubt Until further notice STRIPS Tracker FEDs 353, 355, 434 should always be excluded from DAQ. Until further notice CMS If LS delay > 100 LS, stop and start the run. Try 2 times, if issue is still there, call DAQ doc. Until further notice RPC Call RPC DOC in the case of FED 793 into disconnected state at any moment Until further notice CMS SL should not call the CCC just after the beam dump (wait at least till ramp down starts since CCC shifter is busy in that moment). SL should check LHC Page 1 regularly to see the current status of LHC. Until further notice CT-PPS Include the CTPPS_TOT partition, with only FEDs 582 and 583, and also include the whole CTPPS partition. IF CTPPS starts to give softerrors, call the CTPPS DOC to inform them of this. And, if the softerrors are more frequent than 1 per 15 minutes, remove the CTPPS partition until we have had at least 3 hours of STABLE BEAMS. After 3 hours of STABLE BEAMS AND if the L1 rate is <60 kHz: Include FEDs 586 and 587 (but do not stop run only for this reason). Until further notice RPC TwinMux and CPPF are in error from time to time (RPC port error) because of hard resets by CSC. If the errors disappear within 2-3 minutes, no need to call, but please put this into the elog. Until further notice GEM Do not include GEMs in global. Until further notice CSC If CSC deadtime too high (>5%), switch to column without parking and call the HLT DOC to confirm. At PU ~37, switch back to column with parking. Until further notice HCAL If there is an audio DQM alarm due to the bad CapId rotation in HCAL (in HE more specifically), the data you are collecting at the moment is bad. Ask DAQ shifter to red recycle HCAL immediately and then call the HCAL DOC. Until further notice Injection, Ramp, Flat top, Squeeze, Adjust Type RFM communications Date Valid ECAL ECAL is sensitive to changes in clock frequency that often happen during LHC Ramp, and it is probable ECAL could go to Out-Of-Sync or show issues in DQM histograms. Please make sure ECAL DOC is not called for this, neither by you, DAQ DOC, or DQM DOC. This is understood and the only action that can be taken is halt the run and take the next action at Flat Top Until further notice

The Most used Contact phone list Plan of the day from Daily Run Meeting Checklist for Magnet Ramp Shift leader checklist for CRUZET/CRAFT SHIFT LEADER CHECKLIST for collisions OMS downtime tool Mattermost link Shift Leader instructions for 2022 beam commissioning

Expand all Collapse all

Part 1 : General Information

Part 2 : Safety and Technical Interventions

Part 3 : Instructions Associated with Data Taking

Part 4 : Instructions Associated to Magnet Operations

Part 5 : Tools (Elog, Run Time Logger, Online Monitoring Site)

Part 1 : General Information

Overview

You coordinate the implementation of the run plan and are responsible for the safety of the people and of the experiment (you are the Shift Leader In Matter of Safety, SLIMOS). When in doubt about anything, call the Run Field Manager. The plan for the day is usually set after the daily run meetings at 9:30am (10:00am) on Mon-Fri (Sat-Sun), indico, and posted on this web page above. If you find the table outdated, you can always check the plan using Run Coordination Tools. Just click on the date and look for Plan Of The Day. Try to chart an efficient way to implement this plan, use common sense, and "do no harm" i.e. take no unnecessary risks. The sort of interactions and decisions you need to make with your shift crew during data-taking are described below. If the shift crew looks like they are not making progress on a particular problem that is preventing implementation of the run plan or data-taking, do not hesitate to contact the appropriate experts, even in the middle of the night (see OnlineWBContactList).

When you arrive for your shift you should discuss with the outgoing shift leader the summary of the last shift, highlighting any problems encountered, any new special procedures, any changes to the run plan, etc. If in doubt about any aspects, please do not hesitate the call the Run Field Manager. At the end of your shift, please enter a shift summary into the Shift E-log. Many shift leaders organize this as a time line of activities. A summary of the problems encountered and the data-taking efficiency is very helpful. Also please verify the Run Registry filled in by the DQM shifter for runs taken during your shift.

Good and clear communication are a big part of the job. Requests from the subsystems for specific tests or configurations should channel through you, rather than directly to the other central shifters such as DAQ and Trigger. Moreover, you are the sole contact with the LHC operators at the CCC. Try to facilitate the communication between the subsystem and central shifters, and between CMS and the CCC, so that everyone is synchronized about the current status in order to avoid accidents and to operate efficiently. If there are any changes in the data-taking that could potentially affect the Tier-0 data processing (change in the online software for the HLT or for the Storage Manager, or special tests such as Tracker virgin raw), you should probably inform Data Operations (see on-call number in Contact List, ORM)

Part of the shift crew (Trigger, DAQ and DQM shifters) may be present on shift remotely (due to COVID-19). Communication between the shifters in the control room and on remote shift is organized via the dedicated Mattermost channel and Zoom meeting. For Mattermost, please make sure to :

• 1- Subscribe to the online ops Team (you can do so using this link).

• 2- then join MWGR: Shifters and DOCs channel.

ZOOM link and Mattermost link

If you have questions concerning these shift instructions, shift assignment or comments regarding shift procedures, please contact cms-sldcs-shiftsupport@cernNOSPAMPLEASE.ch

Shift Leader Training

The source of the Shift Leader Training information can be edited at this link.

Warning: Can't find topic Sandbox.CMSShiftLeaderTraining

Your Central Shift Crew and their Roles

Your station is at the left edge of the U-shaped central shift area next to the entrance from the central walkway. Duties are described in the Overview. You are responsible for making sure that your crew is awake and pay attention to their tasks.

The P5 DQM shifter is to your right and is responsible for running the DQM services and displays and for filling in the run quality information into a Run Registry page. You will review the data quality assignment made by the DQM shifter at the end of each run, as described further below.

There is also a remote DQM shifter who should be available on the Tandberg video connection above you to your right. You might have a live video connection to the Fermilab ROC (evening and night), DESY (day), and the Meyrin CMS Centre (all the time). From time-to-time someone on the link may ask you a question. If the video connection is dropped, as of this time someone else from the remote centers must reestablish the connection. The remote control currently only allows you to answer a call or to change the camera zoom.

In the event of some significant problems with a subdetector (e.g. power trips) that requires some intervention by experts, it is possible that problems may be seen only after some time has passed (for example, there may be a period without beam following the fix to the problem, and some issues may re-occur only once stable-beam collisions return). Be sure to check with the DQM shifter that DQM histograms for the particular subsystem (that has recovered from some problem) receive some additional scrutiny for a period after the fix.

The Trigger shifter is further to the right, and controls the “keys” (configurations) for the Level-1 Trigger. Depending what triggers you want deployed, the shifter needs to select or create the appropriate key. The shifter also has information about trigger rates, and about what subsystems may be exerting “back-pressure” to the DAQ (buffers are close to full).

The DAQ shifter is behind you and is the one who is in control of the run control system. This shifter will be the one who tells you if a subsystem FED goes into error.

In the other half of the central area of the control room there is the Detector Safety shifter (also known as Technical Shifter (ex DCS) shifter), who is responsible for monitoring the status of the Slow controls of the experiment, check for any alarms and for assisting in any technical interventions. During day time and on particular occasions this is also the area where the Beam Condition Monitoring people, Luminosity people and Magnet people will be sitting (and is the place where all the relevant monitors for these teams are located).

Sometimes during technical stops or dedicated machine development periods CMS will operate with a reduced shift crew, just the shift leader and the Technical Shifter (ex DCS) shifter. If we are trying to take data, e.g. cosmics, with a reduced shift crew this is done on a best effort basis by the crew at P5. If there are problems with the data taking that the shift leader can not resolve we should not rely on the on-call trigger and DAQ experts to restore running.

The Shift Tool webpage (how to select your shifts dates)

To sign-up as Shift Leader, follow the instructions at the shift tool instruction page

Computer Workstation

There are a computer and 3 screens for your use at the shift leader’s desk to get pertinent information (LHC, E-Log, etc.). The account to use is “leadershift” (ask for the passwd). You should generally have the online system open and the LHC Page 1 display open to keep abreast of developments. The LHC Logbook may also be helpful to gain an understanding of what is going on. Some instructions for how to tunnel out of the .cms network are available here.

If firefox becomes unresponsive (unable to connect to cmsonline.cern.ch), press "Ctrl-shift-Delete" in the browser window, choose "Everything" in the "time range to clear" dropdown list, select "cookies", "cache", and "site preferences", and "clear now". Then re-login to cmsonline.

Point 6 Compensator/Harmonic Filter

The BE-OP-TI (Beams Operations) group has agreed to alert us prior to switching harmonic compensator hardware at Point 6 in or out of the electrical network. This is only an alert for our information and does not require action on our part.

The alert will have the following form: "We are preparing for a compensator switch-over at Point 6 at time or in x minutes." The following acronyms may be substituted for the term "compensator"

• Harmonic Filter
• VAR Compensator
• Static Compensator
• SVC
• TCR

In the shift log, note the time the call was received and the expected time of the switch-over.

Phones

There are two phones at the shift leader station:

• There is a cordless phone (fixed land line 77111) designated as the primary phone to receive calls made to the CMS control room by CMS members and from which you should place calls (e.g. to the CCC).
• There is also a portable phone (169517) mapped to the virtual phone number 77705. This phone must always be answered as it is the phone number we give to people outside CMS to call in case of need to contact the shift leader. If you will be leaving the station far enough to not be able to hear the phone, please bring the portable GSM phone with you. Make sure that its battery is charged. To unlock, press "Unlock" and then "*".

Additonal info: The virtual phone number 77705 (every LHC experiment has a virtual phone 7770X where X is the IP number) is the one which LHC operation will call expecting to have an answer. In case of no response the call will be redirected in cascade to: 75140 (DCS phone), 161931 (Run Coordinator), 165341 (Deputy Run Coordinator) or 165000 (Technical Coordination on-call).

The 77111 phone can call internationally using pin code 1999. (To call Switzerland use: *4+1999+0+number, France: *4+1999+0+0033+number, rest of the world: *4+1999+0+00 (foreign country) + number)

Phone problems: if you have problems calling the 16XXXX or 77XXX numbers, try (with a private phone) to get in contact with CERN Technical Control Room (aka TCR): 72201

Control Room keys

Behind the Shift Leader station there is a grey cabinet containing the keys of most things in building 3562, such as lockers, First Aid materials, dosimeters, coffee... To open it, the SL has to pass their badge over the cabinet's card reader. Please note that the SL has only the permission to take some of them, the ones indicated by the green light. Also take care to put back each key in its correct place... each key has a unique spot.

Tandberg

If the videoconference connection to the CMS_Centre@CERN (Meyrin), Fermilab and other sites fails for any reason, first wait 10 minutes as the system should auto-restart itself in case of problems. If it still fails to start, contact CERN/IT videoconference support:

• Phone: Helpdesk (78888)
• Email: videoconference-support@cernNOSPAMPLEASE.ch (and cc: Lucas.Taylor@cernNOSPAMPLEASE.ch)

Do not attempt to restart it yourself -- it is remotely managed so this will not work. And definitely do not re-cable the device or otherwise fiddle with the hardware (as has happened several times in the past).

Underground Access

Access to underground areas (Underground Service Cavern (USC) or Underground eXperimental Cavern (UXC)) is subject to specific safety precautions...

• Everyone who goes underground must inform Shift Leader and fill the access list.
• Nobody should ever be underground alone. This applies particularly at night when probably no one is already there.
• Normally the PM54 lift is in "GENERAL mode". During long periods of access with no beam, PM54 is in "closed" mode 19:00-07:00 and during weekends.

When the magnet is on...

• No steel or other magnetic materials should be brought into UXC (however, steel toed safety shoes are OK)
• Do not bring in wallets and watches (so as not to kill your credit cards...)

When there is beam...

• For standard requests (not super urgent, working days and working hours only), subsystems get on a waiting list. Before submitting a request, first consider how long will it last and how important is it for CMS - every access request is counted as "LHC not available" and CMS is listed as the reason. We cannot afford to stop LHC for several hours in order to recover 0.01% of any detector. Please avoid unnecessary accesses - any accessed area has to be checked by Radioprotection before people can go in, which increases the total duration of the access and the "LHC non-available" time counted against CMS.
  • First send an e-mail to Dragoslav.Lazic@cernNOSPAMPLEASE.ch and cms-safety@cernNOSPAMPLEASE.ch explaining reasons for the access, importance, expected duration, limiting factors (presence of technicians, working hours, availability of experts etc).
  • Discuss details in Technical Coordination meeting on Tuesday or Friday 09:00
• When access is approved for another experiment it will be announced at the 08:30 LHC meeting…
  • Announcement via mail to extended TIG or Technical Coordination meeting
  • TC will delegate access preparation to RPE (Any access to UXC requires lifting of radiation veto and a sweep). Again - avoid going into UXC unless really needed.
  • During EYETS the procedure of taking the "Delegation" of UXC55 from CCC has been changed. The instructions on how to do it are available here
• If urgent request: Call 16-5000
  • Depending on urgency, go to 8:30 meeting or TC will interact directly with LHC
  • 16-5000 will take care of access procedure
• If LHC announces ad hoc access (i.e., not at 08:30 LHC meeting)
  • Shift Leader has to call 16-5000
  • If access will be used 16-5000 will take care

Control Room Etiquette

Traffic into the central shift area, and any long discussions, should be kept at a minimum in order not to distract the shift crew. Please ask the people to move to the coffee area, one of the overflow rooms, or outside. The subdetector shift area similarly should not get too crowded or noisy.

The shift areas should be kept tidy. The trash cans need periodic emptying. There should be bags in the cupboard at the entrance of the Control room where also office supplies are stored, and there are large bins on the other side of the SX5 building (near the PM54 lift).

Coffee

On the table of the shift leader there is a box that contains a coffee card. For people who need it the shift leader can lend them the card at the price of 1 CHF. The money will be collected by the Run Field Managers regularly. If the card needs to be refilled inform the RFM and use the spare card that is kept in the bottom of the Traka key cabinet.

Tidiness

The ONET cleaners come every Wednesday morning at 8:00am in order to clean the desktops in the control room. The desktops will need to be cleared of stuff. Please make a grand sweep up of the control room desks beforehand, on Wednesday morning, and then direct them to do this job. This includes the subsystem side, as well. If there is junk sitting around, please trash it. If there is stuff which clearly needs to be kept, could you please stack it.

Part 2 : Safety and Technical Interventions

First Aid Locker & Defibrillator

The medical materials for a first aid intervention are present inside the white cabinet in the Control Room‘s hallway.

You will find the key to open it inside the gray cabinet behind the SL Station.

The Defibrillator (for Cardiac Arrest) is located in the main entrance of the building (right to the sliding doors).

Short information about it could be find here : french version / english version

There you can find also a second First Aid locker.

Safety Questions

For any questions on matters of safety or infrastructure, call Technical Coordination on-call at 16-5000. If no answer call the CMS LEXGLIMOS, Niels Dupont, 16-5186.

Evacuation alarms, power outages, etc.

If there is an emergency such as an evacuation alarm in the tunnel or cavern, or an occurrence such as a power outage to Point 5, call 165000 (Technical Coordination on-call) and have someone come physically to the control room to take charge of the safety procedures of the experiment and of the personnel. However, some basic instructions are here:

If there is an evacuation alarm, the "evacuation" light on left side of the synoptique rack will become RED (not green). Check the DSS display to see what evacuation was triggered, and acknowledge the alarm by clicking on the appropriate row of the "PVSS Ack." column (puts an "X" for that entry). This does not clear the alarm, but allows it to be cleared later.

After the evacuation alarm is under control, the ability to access underground again must be established by the CERN Fire and Rescue service in coordination with 16-5000.

Local Emergency (e.g. subdetector) : Emergency Stop

In case a part of the detector or the underground needs to be switched off due to an emergency, take actions on the DSS panel (see picture below).

1. Inject the black key from the small blue box next to the DSS panel into the right side of the panel (see picture below). Turn it in order to enable actions on the panel and then switch off the corresponding part on the panel. (Some parts like the magnet are controlled with a separate key switch and/or might need a yellow key - both keys are in the blue key box next to the panel.) Note that the water mist system is activated only if the key is injected in the corresponding switch on the left hand side of the panel, turned and if the corresponding fire sensor has triggered. The key alone will not activate the water mist system.

• Figure 1: CMS Control room DSS panel and the key for enabling DSS actions:
  

Global Fire or other Emergency in the Control Room: Full Emergency Stop

Should CMS and the control room need to be evacuated during an emergency (e.g. fire):

1. Issue the evacuation alarm! (see above)
2. Inject the black "Synoptique Full Shutdown" key from the small blue box next to the DSS panel into the Full Shutdown via DSS (see picture below) and turn the key in order to switch off the experiment.
3. Calmly leave the building.

• Figure 1.1: DSS action for switching off both caverns:
  

Snow

Call 7-2201 to trigger the snow clearing operations. A vehicle dedicated to Point 5 will be on site within 20 minutes during the day.

During the night:

• please ask all your shifters to park in the parking next to SCX5 (the square one) instead of along the road as this will make it much easier to clear the snow.
• for the road and parking lot to be clean by 07:00 you should call around 03:00

Technical Interventions (cooling, ventilation, infrastructure, lighting, electrical)

Anyone who presents themselves requesting to perform an intervention on the infrastructure needs to be asked about the details.

In case a technical intervention is needed on basic services (elevator, etc.), call the technical control room (TCR) at 7-2201. If there is CMS personnel involved (e.g. somebody is trapped downstairs because of the elevator) then call 16-5000 as well.

It has been decided at the TIOC that the CCC would warn the CMS control room ahead of any technical intervention on the P5 infrastructure. This will be done by calling 7-7705 or if no reply 16-5000. The phone call should be placed minutes before the technical crew's arrival at P5.

The goal is for the technical crew to find somebody aware of their intervention when they arrive in the CMS control room to declare their activities. It may happen that the crews will ring at the door as they may not have access for the control room door. This new procedure will be further formalised...

Part 3 : Instructions Associated to LHC

Data Taking Procedure and CHECK LIST

Checklist for Physics Running and Preparing for Physics : Checklist for p-p , p-Pb, Pb-Pb collisions

During steady state operations the LHC will work with cycles of injection followed by data taking and dumping the beam. In the EDMS document 865811 you can find a description of the accelerator modes and beam modes allowed by the LHC, the modes in which an RF synchronization is allowed and therefore the LHC clock may be unstable, and a description on the handshake vs accelerator modes and beam modes.

This is illustrated in the figure below.

As the Shift Leader you coordinate the activities in the control room to effectively take data during this fill cycle. As indicated on the figure above, the data taking for collisions starts at when the LHC reaches full energy and we configure for collision data taking. Between the fills central DAQ configures for cosmics, some subsystems will go to local running for calibrations during the inter fill period.

Setting up for a Run

When we are running with collisions from the LHC the goal is to include all subdetectors in the run. Only if there are problems that prevent a subdetector from running should it be excluded from the run. Getting the full detector back to operation should be the highest priority of the relevant subsystem. If we have to take out any FED or subdetector from a collision run this should be discussed with the RFM or Run Coordinator.

Additionally, the Trigger shifter is going to want to know what trigger configuration you want to deploy. This will depend on whether we are aiming to log cosmics or whether we are set for beams (and whether that means single beams or collisions). It will also depend on what systems are ready to provide a trigger. For example, if the calorimeter trigger boards are not ready (RCT, GCT) you will not be able to run calorimeter triggers. If the DT system is not available, you will not be able to use DT muon triggers.

Check with the Trigger shifter that the TTCmi is running with beam 1 clock, regardless of what beam is in the machine (beam1, beam2 or both). This is not valid if LHC asks to go to internal clock. If you want to understand fully what is displayed on the LHC Page 1 check this page.

Ending a Run

Apart from irrecoverable errors in the DAQ, you may wish to end a run voluntarily for several reasons. For example, if a subsystem is ready to rejoin the global running, you may wish to stop and bring them in at a certain point. Or if enough data have been collected and you wish to move to another test, you may want to stop and reconfigure. Or if the LHC beam conditions may have changed, you may have to stop to power-down or power-up a detector. Keep in mind that stopping and restarting a run brings a certain degree of risk as you may encounter problems. You will need to balance the need against this risk, also keeping in mind that if it is the middle of the night it may be difficult to get the appropriate experts to diagnose the problems.

Problems during a Run

FED Problems

It may happen that a certain FED may go into error. Sometimes the DAQ shifter can clear this with a re-sync, hard reset, or reconfigure. If none of this works and the FED or subdetector needs to be excluded from a collision run the RFM and Run Coordinator should be informed.

Lights in cavern not off (Spikes in the trigger rate)

The RPC and EE trigger rates can display some spikes, which are caused by interferences with the lights in the Experimental Cavern. During a run, whenever the magnet is ON, the lights in the cavern should be turned off. To do this, you should take the elevator to level -3. After exiting the elevator go straight ahead through the door marked "US 54 Radio Protection Buffer Zone". To your right you will see electrical distribution cabinet with code EBD1/55X that has a handwritten note "UXC-LIGHT". Push all seven green buttons one after the other - you will hear contactors (dis-)engaging. The leftmost two take care of most of the lights, including the noisy floodlights. Similarly, when the lights are turned ON for an UXC Access, you should expect spikes in RPC and EE trigger rates - they should calm down in under 15 minutes.

Any problems seen in DQM ?

The Offline Run Manager should be contacted by the Shift Leader in case of problems spotted by the DQM shifter that may affect the quality of the data.

WBM DCS warning sound alarm

Whenever DCS status of any sub-detector partition goes off while Stable Beams are declared, the WBM DCS warning scripts will raise a sound alarm in the control room. Also, a small pop-up (one per subdetector) will appear at the shift leader console, informing about it. The alarm will sound only once, and then repeat after 30 minutes. The pop-up has a "resume alarm" button that the shift leader can push to re-enable the alarm after the problem has been fixed. More information at WbmDcsWatchdog page. In case of urgent issues, call WBM on-call.

Data Quality Monitoring (DQM) of Runs by the DQM shifter and shift leader

• Every run will automatically be classified as significant or not.
• Significant runs will automatically be classified as Cosmics, Collisions or Commissioning.
• The DQM shifter is asked to keep an eye on this process.

For each significant run:

• The DQM shifter visually inspects the DQM shift histograms in the DQM GUI and enters the results of the data quality assessment (GOOD/BAD for each detector component) in the Run Registry.
• You can edit the run registry from only the DQM work station using Edit Run Registry from DQM workstation.
• You can view the run registry (without editing) from any computer with a certificate using view run registry from other computer.
• The shift leader is responsible for the sign-off on the data quality results as assessed by the DQM shifter.

In practice, this means that for each run, and at least once per run, the shift leader takes note of the results produced by the DQM shifter and cross checks / confirms these results with his/her knowledge of the detector status and data taking conditions. This sign-off procedure helps catch any mistakes entered into the Run Registry and also keeps you and the DQM shifter on the same page with regards to the state of the experiment. The DQM results are input to the creation of the good run list for physics analyses.

The shift leader ensures that the DQM shifter marks the data of each sub-system as GOOD, BAD or STANDBY, appropriately. In case of doubt, consult the sub-system experts. If a run is potentially useful for analysis using that subsystem, the data should be marked as good. If the problems are severe enough to make the data unlikely to be useful, have the shifter mark it as BAD.

Shift leader task when DQM shift is cancelled: Registering Runs in DQM Run Registry

The DQM shift tasks become Shift Leader tasks for shifts with no DQM shifter (in case the central DQM shifts are cancelled).

In the past the shifter had to decide whether a run was significant or not, but now-a-days this goes automatically.

• The full instructions of the central Online DQM shift can be found here.
• But here is a shorter basic version for Shift Leaders:
  • You will need 4 tools:
    • The Online DQM GUI: http://dqm-prod-local:8030/dqm/online
    • The Online Run Registry: http://cern.ch/cmswbm2/runregistry/shift.jsp - Login with your afs account.
      • Important note: You have to use the DQM shift workstation to be able to edit anything in the Run Registry.
      • You can view the run registry (without editing) from any computer with a certificate using view run registry from other computer.

* The TWiki pages which describe the plots: DQMShiftHistograms#Online

• • • The ELog: This you already know. The list to report for DQM is Subsystems / Event Display and DQM
  • In the DQM GUI you will find a limited selection of plots that was prepared for you by each of the subsystems:
    • Click on Workspace and then select the Shift workspace in the first column. You will see a folder for each subsystem.
    • Check the plots for each sub-system and use the Twiki to understand what they should look like.
  • When you see a problem, you report this in the DQM ELog and you contact the DOC for the specific susystem.
  • In the Run Registry you will see two tables:
    • The top table shows all the runs.
    • The bottom table shows only the significant runs and shows an automatic classification (cosmic, collisions, commissioning)
  • Any run that appears in the bottom table is significant and needs to be edited and signed off:
    • You select the run and choose Manage > Edit. Then you fill in the stop reason and the flags GOOD, BAD or STANDBY (HV off) for each of the subsystems.
      • Write "Registered by shift leader" in the comment box on the right (exact spelling; any other comments can be added, if needed).
      • Click on Save to go back.
    • When the run is finished, you select the run and choose Move > to SIGNOFF. After that the run is ready for the offline certification process.

LHC Status

If the information on the LHC Page 1 is “stale” or otherwise unclear, and there is similarly not any illuminating information in the LHC logbook, you should call CCC for a status report, especially when it pertains to CMS run operation decisions. To understand the information on backgrounds displayed on the LHC Page 1 check this logbook entry and this logbook entry.

The CMS BRM page tends to be more up to date than the LHC Page 1 or the LHC Operations page. See below.

If you have trouble 'orienting' yourself with respect to which direction a given beam comes wrt to CMS and/or where CMS is in LHC you might want to check this.

If Beam is Imminent

You should be contacted by CCC beforehand, and CCC would ask you to lift the beam injection inhibit if it has been set on our side

Please announce to the shift crew and make a note in the Shift E-Log. In particular, sensitive detectors need to power-down or go to standby. You will have to judge the status of the beam backgrounds and monitors. In order to understand the info displayed in the BRM stations check this logbook entry.

Procedure for allowing injection again after accidental beam dumps

Instructions below provided by Richard Hall-Wilton on April 26, 2010.

• Alarm will appear on Technical Shifter (ex DCS) screen. Informing Technical Shifter (ex DCS) shifter about this.
• Phone the BRM on-call phone.
• Hold-off the injection re-enable until the BRM on-call has determined that the dump is understood sufficiently to continue.
• An official brief report should be available within a few days.

Filling scheme check

Check on the BRM desk (normally bottom left screen labeled 27 Screen 1) if the right filling scheme has been used. The plots called "Beam 1 Bunches" and "Beam 2 Bunches" show you the intensities and bunch numbers of colliding and non colliding bunches. The default view of these plots is too coarse to judge the exact bunch numbers. You can obtain exact numbers either by 1) zooming in on these plots, or 2) via DIP.

BPTX scalers display:

1. Find the correct mouse and click on the icon with the magnifying glass at the top of the BPTX bunches plot. Now drag-select a region of the trace to zoom in on. The scale will adjust and allow a determination of the number of bunches. Do this for each beam. (Right-click several times until the original scales are back and the whole figure is visible again.)
2. There should be a DIP browser scaler rates page, look for B1, B2, B1ANDB2 numbers.

How to compare the bunch numbers to the advertised filling scheme
The active filling scheme is advertised on the LHC VISTARS. Filling scheme naming follows the pattern: <spacing>_<Nb>b_<IP1/5>_<IP2>_<IP8>_<magic code>, where:

• <spacing> = Single, or 2025ns, or 525ns, or 75ns, or 50ns, or 25ns, etc.
• <Nb> = total number of bunches per beam
• <IP1/5> = expected number of colliding bunch pairs in IP1 and IP5
• <IP2> = expected number of colliding bunch pairs in IP2
• <IP8> = expected number of colliding bunch pairs in IP8

BPTX deltaT of IP (B1-B2) (aka cogging)

• The CMS value for "BPTX: deltaT of IP (B1-B2)" gives the result of the z vertex position as measured by the BPTX, in nanoseconds. A measurement of dT = 0.1 ns corresponds to 1.5 cm displacement in z. Normally, the value should be below 0.15 ns. Typically, the measurements for all the experiments should be compatible.
• To check the measurement look at the LHC Configuration Vistars page. This value should be checked towards the beginning of the fill for every physics fill and every preparatory fill for a physics fill. This should be done AFTER CMS has switched to LHC clock. This occurs when the DAQ shifter starts a physics run at FLAT TOP.
  • Additionally one can check at the Vistars page: LHC RF Timing
• To measure the cogging from the BPTX there must be at least one BX filled with both beams. The bunches must have intensity higher than 5x10⁹ protons per bunch. Even then, the measurement is only valid when CMS is on the LHC clock.
• The value of the CMS cogging measurement should be written in the shift elog. The acceptable range is within 0.0 +/- 0.15 ns. If it is outside this range, look at the ATLAS and LHCb values. If their values are OK, then by symmetry, we must be OK. Call the BRIL DOC to investigate the CMS measurement. If the ATLAS/LHCb measurements are also outside this range, call the Run Field Manager to figure out how to proceed.
  • If the Vistars shows a cogging number for CMS that deviates significantly from those of the other experiments, it's most likely a problem on the CMS side.
  • If the value published by CMS is -0.99, see notes below.
• The cogging should be rechecked once physics data taking has started and the tracker has been switched on. It also should be compared with the z location of the vertex from the tracker in online DQM.
• An easy way to look at the cogging history vs. time (e.g. to cross check the cogging histogram on the BRM display) is to look at the OMS LIVE.
• If the value published by CMS is -0.99, it represents not a real measurement but a default value published by the scripts. It means that BPTX timing scripts are running and, most likely, working properly. But there is nothing meaningful to measure, thus the default value is published. The reasons for this could be:
  • There is no beam
  • The beams are asynchronous, i.e. no beams are in the same bunch crossing
  • LHC status is RAMP: the scripts are disabled for that part and publish -0.99
  • The intensity of the bunches are too low, less than 5E9, hence not detected by BPTX

Background levels

As shift leader you are responsible for regularly - at least once every 30 minutes - monitoring the LHC background levels. Below is a summary of what to watch.

When the beam is circulating there are three numbers sent by CMS to the LHC that give an idea of the beam conditions in CMS. These numbers are:

• BKGD1: flux in the inner detector region, defined as 0.002*(sum(Hit rate of all 8 BCM1F detectors))(Hz)
• BKGD2: rate of background halo defined as 0.002*(sum(BSC-splash trigger beam{1,2})-1.8*BSC MinBias AllThreshold1) (Hz)
• BKGD3: fraction of ABORT defined as Maximum(Percentage Fraction of 40us, 5s, 83s ABORT thresholds (All 8 inner BCM2 diamonds)

The scale of BKGD1 is from 0 to 100:

• if any number is below 0.002, 0.002 will be sent, likewise if any number is greater than 99 , 99 will be sent.
• >20 means warning : the background level is detrimental for data taking and It should be reported to the Pixel and BRM oncall.
• >50 is alarm level and should lead to a call to CCC to ask if these conditions are understood, expected to continue etc...
• >100, if confirmed by the BRM oncall, means that conditions are so bad that LHC Operations should considering ABORTing the beam.

A detailed description can be found here https://edms.cern.ch/file/1051267/1/CMS_Background_Numbers.pdf

In addition to these three background measures propagated to the LHC, the strip and pixel detectors looks a background rates as described in https://twiki.cern.ch/twiki/pub/CMS/PixelOperations/PIXEL-HV-Procedures.pdf. You should verify that the conditions for keeping the strips and pixels on are satisfied.

In case the background conditions deteriorate contact the BRM on-call expert.

Subsystem operation

We no longer have subsystem shifters in the control room, so control of the subdetector power is normally kept by the central DCS shifter. Subsystems sometimes take control of their DCS for special tests. Be sure that control is returned to the central DCS shifter once the experts finish their work. The DCS shifter continually monitors the status of the subdetector control systems. If the shifter notices anything out of the ordinary, he or she should call the appropriate person (DOC or DCS on-call for that subdetector) without hesitation. The shift leader is responsible for approving the turn-on of the HV for the pixels and strip tracker. Below links to instructions for some subsystems are added in order for the shiftleaders to start to get familiar with the procedures.

Before turning OFF the Pixel/SiStrips, the corresponding DOC persons have to be called. If Pixel/SiStrips automatically turn OFF, e.g. LHC is in INJECT AND DUMP, call the corresponding DOC persons.

The Pixel/SiStrips LV cannot be turned On by the Technical Shifter (ex DCS) Shifter. If the reason for the OFF state is unknown, call the corresponding DOC persons and ask/inform them. If the Pixel/SiStrips decide to turn LV On, ask about the DAQ configuration status and the steps that cDAQ shifter has to follow once the LV is ON.

• Conditions to turn ON Pixels

Databases Monitoring

If for any reason you're in doubt about the correct functioning of the CMS Databases please look at this monitoring page. Everything should be in GREEN there, otherwise call the expert (see the Contact List). After contacting the expert, you should also inform the HLT DOC, just for their information. Database failures can cause an inability to take data, so if there are mysterious problems with the DAQ, check the monitoring page!

Part 4 : Instructions Associated to Magnet Operations

Ramp

First of all, the magnet operations are controlled by the magnet shifter in the central shift area. It is the area with the bank of displays to the right of the rack of DSS lights and buttons. If the magnet will be ramping imminently, the detector subsystems need to know as several systems want their high voltage off. The safe state for the sub detectors are described in the DCS instructions. Check with detector experts. If the detector shifter seems unsure, encourage him/her to contact their detector run coordinator. Once the magnet is stable at operating field strength, data-taking can resume.

The magnet can ramp from 0 to 3.8T in under 4 hours normally if there are no stops. It may be worth giving a 1 hour's notice to the detector subsystems (typically when the magnet is at 3T) to begin preparing for operations when the magnet reaches flat top in order not to waste too much time afterward.

Planned Discharge

The instructions depend if it will be a slow discharge or a fast dump. If a slow discharge is planned, then the instructions are similar to that for ramp-up since the rate of current change is similar. Again, check with the detector systems. If it will be a fast dump, all systems need to be turned off (at least high voltage and maybe low voltage – check with the subsystem experts).

Unintentional discharge

It can happen that the magnet goes into an unplanned discharge, for example due to a cooling failure or power outage. If that is the case, a flashing yellow light and siren should go off in the control room. The audio alarm can silenced by pushing the "stop siren" button on the DSS hardware panel located in the far left corner of the Technical Area next to the flashing yellow light. The flashing light will continue to flash for a few hours, unless it is acknowledged on the DSS screen. Systems which are sensitive to a magnet discharge should already be interlocked to power down upon any magnet ramp-down. SMS messages should be automatically sent to all field coordinators of the experiment as well.

If the magnet does go into an unplanned discharge, please inform immediately Technical coordination by calling 165000.

Part 5 : Tools (Elog, Run Time Logger, Online Monitoring Site)

Elog report

The shift leader is primarily responsible for entering information in the shift elog. He will coordinate with other shifters, particularly DAQ, Technical Shifter (ex DCS), and Trigger shfiter, to make sure that the relevant information is entered, see below. We should avoid duplication of information entered by the shift leader and DAQ shifter. Note that it is very important that the information is entered into the elog, it is used by experts to understand the flow of events during the shift and to debug problems. Below is a set of guidelines:

• The shift elog should not be one long list of all events that took place during the shift with manually entered times. Rather individual entries should be made for each event that is entered into the elog. (This will make searching easier, and will also allow for automated entries.)

All entries should have a type. The type simplifies the searching for information. The different entries that should be made into the elog based on the different types (put in bold) are listed below.

• Start of shift Shift leader
  A Brief entry to state that you have taken over the shift leader responsibility. List any special instructions or goals for the shift.

• LHC Fill Start Shift leader
  Report the start of fill; include fill number and fill pattern. (Should eventually be automated.)

• Run Start Shifter leader
  Start of run, include run number, initial intensities, detectors included. (Should eventually be automated.)

• Run End Shifter leader
  Include reasons for ending the run, number of L1A, integrated luminosity. In case of a problem causing the run to be terminated, please just note this here and also generate a Data Taking Problem entry. Please crosscheck DAQ problems with LHC-elog entries (LHC SEQ ...) and note correlations.

• Data Taking Condition Shift leader
  Note any changes in the run conditions from e.g. beam quality or detector changes.

• USC Access Shift leader
  Report access and reason for access in USC

• UXC Access Shift leader
  Report access and reason for access in UXC

• CCC Update Shift leader
  Report communication with CCC

• Data Taking Problem Shift leader with the help of relevant experts
  Report any issue that caused problems with the data taking. E.g. out of synch, HV trip, etc. If this problem is causing a run to end, please generate both this entry type and the Run End entry. The shift leader might find it useful to delegate the writing of this to the relevant shifter, e.g. DAQ, DQM, TRG, or Technical Shifter (ex DCS), as they might have the expertise to write a more meaning full report.

• Other Problem Shift leader
  E.g. problems with shuttle, elevator, coffee machine etc.

• Shift summary Shift leader
  Any observations or special notes for future shifters including the run plan.

• BRM summary Shift leader
  A screenshot of the 12-hour summary from BRM at the end of the shift (you need a tunnel to the .CMS network). New feature is currently active. The screenshots are be uploaded to the BRM snapshots webpage. Pick the one that says LHC 12Hour Activity. This feature is not very robust for now, so let BRM shifter know if you are getting the wrong page

Downtime logger (formerly known as Run Time Logger)

The downtime logger tool automatically creates new downtime entries for any period of > 1 seconds with no L1As. As shift leader, your job is to select a Group/Category and add a text comment for each automatic entry if needed. Corrections can be made by the OMS team together with the RFMs.

Link to downtime logger: OMS downtime logger

Detailed instructions here, tutorial.

Event Display

The event display should run automatically and pick up events for each new run within a few minutes after the start of the run, assuming data are being collected. The shift leader should check periodically to see that the event display is running. In case the event display is not updating or has other problems, contact the DQM DOC. They will diagnose the problem, and if necessary, they will contact the event display DOC to resolve the issue.

Online Monitoring Site

• CMS Online Monitoring System, OMS

• Vistar

• DAQ Status

• DCS Status

Miscellanea

In case the main door of the building is stuck open, here is the procedure. There is a "security button" on the right of the door (looking from inside). If someone pushes it, the door remains open. To move it back to normal conditions, there is a hanging "key" that must be inserted from below the button and it works.

%RESPONSIBLE% GregRakness