How to turn on DAQ equipment after a long shutdown or power cut

From the start of the recovery operation, do regularly post on slack what your intentions are and don't hesitate to ask for help!

Post an elog at the start and at the end of the recovery, indicating what difficulties you encountered.

Warning, important Before attempting to turn on anything, check with one of Roberto Acciarri, Karol Hennessy, Giovanna Lehmann Miotto, Filippo Resnati that it is OK to proceed.

Preliminaries

Make a visual inspection of the DAQ barrack:

  • all rack doors shall be closed. If they are not, close them;
  • look at the ceiling and verify that there is no sign of humidity/water leaks;
  • walk behind the barrack and check that there is no water on the floor;
  • all rack doors should have a blue light; if the light is off or red then you will have to clear any alarm and power on the racks from the DCS (in the NP04 control room).

Go to the NP04 control room and check on the DCS that the cooling circuit for the racks is operational (see CheckWaterCooling attachment below). If the DAQ room temperature is >24 C or the inlet water pressure is < 6 Bar it is NOT safe to turn on equipment without prior OK from an expert. Call the NP04DCS on call number.

If all is OK, go back to the DAQ barrack at EHN1 with a laptop. Verify that the network switches in racks 5 and 6 are on. Verify that the disk bays in rack 4 are powered on.

Turning on things

Turn on np04-srv-007 and np04-srv-008, These are the two NFS file servers.

When those are up, turn on np04-srv-023. This is the online monitoring computer.

  • np04_srv_001 to np04_srv_004 mount the OMoutput disk from np04-srv-023.

Then you can start bringing up other nodes. Do them in groups of four and take the time to check:

  • availability of file systems (the df command should show the mount points for nfs and the storage volumes for np04-srv-001..4);
  • correct functioning of supervisord (service supervisord status; if needed you can start the service with sudo service supervisord start ).

Once all hosts are on, from np04-srv-001 to np04-srv-024, you have to do the following:

  • login to np04-srv-010 and make sure that the WinCCOA project is stopped (see here).
  • login to np04-srv-024 and make sure that the WinCCOA project is running (see here).

Tip, idea If you suspect that something is not correctly configured, the ansible scripts may help you understanding what is missing. They are all kept here.

Now it's time to turn on specific DAQ hardware:

  • Turn on np04-onl-01 (in rack 6). This server allows you to re-program the CTB and timing units if needed.
  • Turn on the CTB (on/off switch on the back), the timing system units (on/off switches on the front) and the NIM crates (on/off switches on the front) in rack 8.
  • Re-program the timing systems following instructions here.
  • Check the status of the CTB following instructions here.
  • Go to the CRT rack close to the yellow fenced area and follow instructions listed [BringingCRTUpFromRackOff][here]] (possibly with a CRT expert following you on slack or at least a DCS expert being present). When the PC (np04-crt-001) is up you can program also the CRT timing (PDTS_PRIMARY).
  • Connect the power cords of the ATCA shelf (rack 6), if needed.

DAQ test

Now it's time to turn on our DAQ:
  • Ask the run coordinator or DCS expert to turn on low voltage for the TPC CE (WIBs) and the PD (SSPs);
  • Ping all SSPs to verify that network connectivity is OK (np04-ssp-001 ... np04-ssp-024);
  • Manually fix COB 8 as described here. If you are recovering from an unexpected power cut or you have turned on the ATCA shelf before the file servers were up, you will have to power cycle all COBs from the run control user interface;
  • Take partition 0, include all components, configure the system with the latest configuration advertised here and see how far you get!

-- GiovannaLehmannMiotto - 2018-12-14

Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r7 - 2018-12-20 - GiovannaLehmannMiotto
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CENF All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback