RichSSB2 < LHCb/RichOperations

LHCb/RichOperations Web>RichSSB2 (2007-11-23, StephenWotton) (raw view)
---+ The SSB2 wiki

The contents of these Wiki pages are based on the SSB2 lab manual version 0.2 that is attached to the bottom of this page. The figures have not yet been imported.

%TOC%

---+ SSB2 news

2007-08-09 [SW]: L0 board 5 on the SSB2 column appears to be malfunctioning (bad data transmission from both GOLs). The board is switched off and should be left off. HPDs 660 and 661 therefore cannot be read out at present.

2007-10-31 [SW]: pcukl105 firmware and PVSS controls have been updated.

---+ SSB2 infrastructure

It is mandatory to keep the He-free air flow to the SSB2 box always operational. The main valve at the bottle rack
outlet should be open and the flowmeter should indicate a minimal flow of ~10 l/h. Please have a regular look at
the flowmeter to check that the flow is OK.

In case the bottle rack is empty, or in case of problems, please call Erich (16-0183) or Didier (16-3846).

---+ Computing

We deal with a distributed system: please avoid any system-wide change without consulting everybody. In a distributed system non local effects might happen.

In order to avoid clashes/confusion the lhcbrich account should not be used for remote desktop, especially on machines running Win2003 Server (not "to steal" the local session).

Use your personal account for remote desktop.

Be careful when forcing logoff of any other user: improperly ended PVSS sessions give troubles.
Maximum number of users equal to one will be enforced on Win2003 Server machines to avoid concurrent sessions.

The =lhcbrich= generic user account exists on all the computers in the SSB2. 
The password is: {ask}.

Be careful that this account has, so far, administrative privileges on all the machines. 
As soon as the situation is more stable its privileges will be downgraded to the minimum required to just RUN the projects.

Each subsystem developer/responsible will have administrative privileges on its machines.
The desktop redirection for the =lhcbrich= account is disabled,  so that each PC desktop layout will be local only.
 
Do not change the =lhcbrich= account settings on NICE. 

In addition to the desktop, that is now not redirected anymore, it is essential that the roaming profile stay disabled.

The outlet of this laboratory is: =0156 R-0002 (0914/01)=. This plug connects a fan-out. All the information are accessible on http://network.cern.ch .

An Electronic Logbook (ELOG) has been created to upload in a live-time any important information available to anyone who is registered. Everybody can register himself on: http://lblogbook.cern.ch/RICH/.
Notice that it is possible to attach any kind of files (like graphs, photos, tables, data files, etc.).

---++ PC

The following machines are strictly reserved for development/running of the different sub-systems: 

|pclbcecs04|DCS / monitoring|Antonis / Ulrich|Win XP|
|lbh6lx01|ELMBs|Antonis|Win XP|
|pcphdt2006|ECS / DCS / UHV|Flavio / Alessandro|Win XP|
|lbriwico01|DAQ / L0 / Specs|Stig / Sean|Win 2003|
|pclbrich01|DAQ / L1|Steve / Hugh / Gareth|Win XP|
|lbt9lxco01|DAQ / event builder|Steve / Niko / Marco|SLC4|
|lbtbxp01|DAQ / ODIN|Steve / Marco|Win XP|
|pclhcb84|Beamer / Mag Distortion|Aleixandre / Barbara|Win XP|
|Pc161331|General use|All|Win XP|

All of these machines should be left typically free for their own use and that a lot of development work (as long as it does not involve any interaction with the hardware) can be done by the responsible remotely.

One PC is available in the SSB2 lab for personal use. 

---++ DIM Name Server

The "DIM Nameserver" (DIM DNS) is a system service in =pclbcecs04.cern.ch=. As long as this PC is up and running the server should be available, regardless of who is logged on, if any.

In a distributed PVSS system there must be a unique DIM DNS process (=dns=) addressed by all the components which need to communicate among each other.
 
Therefore the =dns= process must be running at all the times on a fixed machine, for the distributed system to run. It has therefore to be installed as a system service.

The DIM DNS running in =pclbcecs04.cern.ch= should not be manipulated.

---+ The Experiment Control System

---++ The RICH PVSS control processes (SSB2)

RICH2 was assigned the following PVSS project numbers: 170-179.

RICH2 DCS will have, tentatively, numbers 170-174.

RICH2 DAQ will have, tentatively, numbers 175-179.

RICH2 project names and numbers will be:

|!R2DCS1 LV and !SiBias plus top RICH FSM|170|
|!HPDVolt: UHV|171|
|!R2DCS2 (monitoring)|172|
|!R2DAQ1 (L0)|175|
|!R2DAQ2 (UKL1)|176|

---+++ General design rules

No potentially dangerous action (like switch on) should be triggerable by one "single-click".
Use standard tricks like: cascading two or more buttons.

---+ UHV control

The UHV control is done via CCPC.

The interaction vi the CCPC is via a PVSS interface project.

Please keep in mind that all the monitoring, control and configuration are done by a C program running on the CCPC; the PVSS interface is only an interface to that program.

---++ CCPC
The cold startup should proceed as follows:

The CCPC boots from the network It then shows the following prompt for login:
<verbatim>
pclbcc30 login:

Login name: rhv
Password: xxxxxxxx
</verbatim>
The following prompt appears:
<verbatim>
[pclbcc30] /home/rhv >
</verbatim>

At the prompt:
   * =tcsh=
   * Make sure the dns runs somewhere if PVSS is needed.
   * Make sure the environment variable DIM_DNS_NODE is defined accordingly (only if PVSS is needed) setenv DIM_DNS_NODE  pclbcecs04.cern.ch
   * Change to the following directory: =cd vhv=
   * Execute the following program: =./hvcont  --dns  -u=
 (Which will point to the most recent executable, currently =hvcont7= in folder =./hvcont7=)

Command line options
   * You can not anymore run 2 instances of the program, a lock mechanism avoids that, in case of a wrong lock (perhaps due to a previous crash) you may force
the running with the command line option =-u=  (for unlock)
   * Data is recorded on file only if you ask so, the default is not to do any logging, to log data please use the =--log= option  (beware it is a double dash:  =--log=)
   * To connect to PVSS you have to start the DIM server: =--dns= (with double dash again)

Note: when in the program, the backspace key is not properly emulated. 
It is replaced by =Ctrl-h=.

The screen that shows up is self-explanatory and it is updated every 3 s.

Commands are listed on the bottom line and are reproduced in the table attached. 

The normal HV ramp-up procedure should be as follows:

   * Set the maximum current limit by typing &#8220;i&#8221;, then choose the channel and enter the value  280.0 [µA]. 
NOTE: there is no default channel anymore, existing channels are 0, 1 and 2 but only channel 0 is connected to ISEG for the time being. 

   * Set the desired voltage by typing &#8220;v&#8221;, then the channel and the desired value, eg 5000 [V]. The ramp up starts automatically. The execution can be paused  only entering the actual voltage. 

If the terminal displays gets error messages it might show wrong things: in this case, or if you suspect your are reading wrong values, just quit and restart it


---++ List of available commands

|*Command name*|*Key*|*Default value (where applicable)*|*Unit (where applicable)*|
|Set V|V (or v)||V|
|Set I Max|I (or i)|350.0|µA|
|Set Ramp Up|U (or u)|50.0|V/s|
|Set Ramp Down|D (or d)|500.0|V/s|
|Kill one channel|K (or k)| | |		
|Trip time|T (or t)|20.0|s|
|Refresh  screen|R (or r)| | |		
|Help|H (or h)| | |		
|Quit|Q (or q)| | |		

---++ PVSS interface

Make sure that he process on the CCPC, =hvcontx=, is running before using the PVSS interface.

The PVSS project is a project called RICH_HPDVolt-x.y. It normally runs on =pcphdt2006.cern.ch=. The project with the higher version number (x.y) should be always started, unless explicitly required otherwise.

Setup after booting: none required.

The PVSS project is started via the PVSS project and administration icon: start PVSS by clicking on the desktop shortcut "PVSS Project Administration", select the line corresponding to RICH_HPDVolt-x.y and  then click on the green traffic-light.
 
Fig 1.

Give some time to PVSS to start all its processes (about one minute).

After the PVSS project is started, it should normally do itself all the required startup steps.


All the monitoring and control, for a 3 column SSB2 system, happen via the main panel which pops up at startup (see the figure 2).


There are only two mandatory operations to do before starting to work.

   * To click on the padlock near the UHV icon on the top-left of the panel and issue the Take command to take full control of the UHV tree; as a consequence of this operation all the State icons should move from a grey color to color depending on the current state. This operation cannot (and must not) be made automatic. In fact, taking the full control of the FSM tree implies that one can send commands to the system. For this same reason, it is a good practice, for security reasons, to click on the padlock and release the tree after finished working, even if a single click is not enough to issue a command. This ensures, among other things, that one cannot mistakenly issue commands.


   * To click on the UHV State button and issue either the command LOAD or CONFIGURE (they do the same, for the time being) in order to load the recipes from the database.

 
Fig 2.

A crucial issue is the connection to the CCPC. If the connection to the CCPC is not working, for any reason, the monitoring and control from PVSS have no meaning. For the time being the safest procedure to check that the connection is active is to watch the vMon icon in the control panel and check that the number is updated about every three seconds.

Common symptoms of lack of connections are: 
   * the icon connection to the CCPC red;
   * the colunn icon displays a number like -9000, -9001, -9002;

---++ General issues

For the time being, tentatively, the following voltage settings were introduced:
   * vSet = 0 Volt (OFF)
   * vSet = 200 Volt (STANDBY1)
   * vSet = 2000 Volt (STANDBY2)
   * vSet = 20000 Volt (READY)

---++ Control of all the channels via the FSM

When clicking on the UHV button a menu shows all the available commands in that state.

---++ Single channel detailed control

The two ovals at the top of each column indicate:
   * whether the column is in an emergency state (red) or not (green);
   * whether there is a link to the CCPC or not.

For each column the main panel shows the monitored data (updated from the CCPC every 3 seconds).

For each column you can set the voltage (=vSet=); after setting =vSet= you must click on switchOn to start ramping towards =vSet=.

The switchOff button sets the voltage to zero and starts ramping down at the rampDown speed.

The kill button sets the voltage to zero and start ramping down at the rampDown speed of 2000000 (i.e. it ramps down to zero in the shortest time allowed by software/hardware).

Expert configuration opens a small panel with some more configuration values (some defaults are present); to actually send them to the HW click on the configure button.

Plots shows a plot of =vMon=, =iMon= and a numerical code for the status flag; until the plot is kept open the data are kept in memory.

---+ The Low Voltages and Silicon Bias control

---++ LV hardware:

Cooling System MARATON:

   a. check the level of the water
   a. press the "switch on button" on the front
   a. In the Compatible Control CC1 to start the control system turn the knob (it is like a black button but you can also turn it) to select "ON" on the screen and then press it to confirm the action. You must see: the green light LED of pump ON, the blue LED of cooling flashing
   a. You can set the working temperature by pressing the &#8220;set button&#8221; and then turning the knob. To confirm the temperature press the knob.

When the temperature is near the working-point (to select it see 1.d), switch on the Primary Rectifier.

Switch on the MARATON.

The Remote Control Wiener:
   a. Switch on the power of the crate
   a. Check that in the screen there are setting: +5V and 3A

NOTE:
- On the front panel of the Wiener RCM the on/off state of each channel is indicated by a LED.

---++ Si Bias hardware:

   1. Check that the supply button at the back of the CAEN is ON. You must see that the "main LED" is ON (red light)
   1. Turn the key on the &#8220;local position&#8221;
   1. Switch on the screen. You must see that the operating system is starting
   1. Press "enter". A login screen must appear.
   1. Login with: username: admin, password: admin
   1. Access to the &#8220;main menu&#8221; and select &#8220;channels&#8221; from the list You must see all the channel status
   1. To change a value you can insert it in its field and then press the spacebar to confirm the action

NOTE:
   * if the status column is =Ext-Dis= it means that this channel is disable by the patch-panel. You can change the position of the channel lever on the patch panel to change the channel status.
   * to switch on/off each channel you have to access to the field and then press the spacebar to confirm the action 

---++ Location of the hardware in the racks:

---+ Detector Control System:

---++ PVSS ELMBs:

1.	Run the PVSS II Project Administrator (fig 1). It normally runs on =lbh6lx01.cern.ch=.
 
Fig 1.

2.	Select =R2DCS2= and then push the =start project= button.
 
Fig 2.

---++ PVSS DCS:

1.	Run the PVSS II Project Administrator (fig 1). It normally runs on =pclbcecs04.cern.ch=.

 
Fig 1


2.	Select the =R2DCS1= and then push the =start project= button. Some new windows will appear: the =console= (fig 2), the =Log Viewer=, =Graphical Editor= (fig 3) and the "RICH2 Columns Overview Window" (fig 4):

Fig 2

Fig 3

3.	Select the column to see information about the selected column: (In the SSB2 lab the column is =R2 Column 1=)

Fig 4

4.	Run the =Window User Interface= related to the =Device Editor Navigator= from the =console=. The =Device Editor & Navigator= window will appear

5.	Select the FSM tab and press =Start/Restart All= button

6.	Double click on =RICH2= and double click on =RICH2_DCS=

7.	Right click to the =R2A_DCSCOL1= and select =View=. A =R2A_DCSCOL1: !R2DCS1:Manager2= window will appear.

8.	Click on the padlock and then click the =take= button: the state field will become yellow like the =channel001= (channel referred to the Si Bias) field with the written =NOT_READY=. The =Channel2=, =channel3= and =HPDCOL_TEMP1= will become blue with the written =OFF= (fig 4). 

Fig 4

9.	In this step you have two options to start the system:
   a. Click on the =state= button and select =Switch on=
   a. Click on the =state= button of each sub-system 

NOTE: if you want to access to one sub-system manager you have to double click on the sub-system name.

---+ L0


See [[L0Controls][here]] for operating L0 instructions
---+ L1

See [[RichUkl1][here]] for general information about operating the UKL1 boards.

At the SSB2 lab the PVSS control is the same as described [[RichUkl1#L1Controls][here]] except that:
   * There is only one UKL1 board, =pcukl105=.
   * The project runs on =pclbrich01= under the =lhcbrich= account.
   * The GUI is running on the same PC as the project. 
      * The project is started by clicking start on the Windows start menu and selecting =Start PVSS 3.6=.
      * The project !R2DAQL1 should be started.
      * The panel fwUkl1 should open and once it has configured the FSM tree, the button =Open FSM= can be clicked and the project is usuable as described [[RichUkl1#UsingFsm][here]].

Note that the UKL1 boards may normally be left in the =RUNNING= state.

In case you think you need to do a complete reinitialisation all you have to do is:
   * Select =Stop= from the =pcukl105= top-level FSM menu.
   * Select =Reload= from the FSM menu.
   * Wait about 15s (the UKL1 FPGAs are being reinitialised).
   * Select =Configure= from the FSM menu (you are asked to select a recipe at this point).
   * Select =Start= from the FSM menu.

---+ The Event Builder

---++ Preparing the environment
   1. Log into the Windows PC =lbtbxp01= as user =lhcbrich= if not logged in already.
   1. Make sure "Exceed" is running. If it isn't, start it from the Hummingbird Connectivity folder on the Start Menu.
   1. Double click on the Putty icon on the desktop and double-click on =lhcbrich@lbt9lxco01= in the list of sessions.
   1. Use the password of the =lhcbrich= account to log in as that user.
   1. =source /group/rich/sw/scripts/Online_v3r5_Setup.sh=

---++ To START:
   1. =source /group/rich/scripts/hltrx.sh= when you want to start a run.
   1. Wait until the Buffer Manager Monitor window appears, and until a =HPDMoni= item is listed in it. Wait another minute (to allow =HPDMoni= fully to finish loading), and then the event builder is ready.
   1. =diskwr &= (Optional. Writes the MDF data to disk. If running, !DiskWR should appear in the buffer manager display.)
   1. You may now start sending triggers from the ODIN (see below).

Note: If the disk writer is running, data from the run will be stored in =/data/rich/mdf/rich{timestamp}.mdf=.

Automatic scripts archive data from the directory (and its subdirectories):
   * =/data/rich/CASTOR/=

The archived data are stored at =/castor/cern.ch/lhcb/testbeam/lhcbrich/SSB2/=.

Data are no longer exported from =/scratch/= but the directory =/scratch/RICH/= may be used for storing temporary data files. Note that this area is not backed up. You must ensure that the data you want to keep are archived.

At present, the disk writer is configured to write data to =/data/rich/mdf/=.
So to get the data to CASTOR you should first package together all the files for a given "run" from this directory "by hand" and move the packaged file to =/data/rich/CASTOR/=.

Don't forget to package the files. CASTOR is much more efficient for larger files. Generally it is also helpful to compress the packaged files as the subsequent file handling is quicker. Packaging and compressing can be done in one step using =tar= with the =z= option.

---++ To STOP:
   1. First stop sending triggers from ODIN (see below).
   1. Wait a few seconds to allow buffers to empty.
   1. =pkill Gaudi.exe=

---+ ODIN run control

---++ To START a run from ODIN:
   1. Open the window called =Partition_ODIN V2_21:TFC manager=
   1. Select =START_RUN= from the top-level FSM state menu.
   1. Set the run number and number of triggers desired (0=free running).
   1. The =Auxiliary Trigger= box should start counting at the first spill
   1. The green box shows =RUNNING=.
 
Fig 1

---++ To STOP run:
   1. Select =END_RUN= from the top-level FSM state menu.

---++ Odin setup for calibration runs

A new ODIN firmware and PVSS version is available to run locally in SSB2 useful for calibration runs

   1. =ssh tfc@odinv221= pass: =tfc-expert=
   1. =cd ODIN_v2=
   1. =source .setup_v2r50=
   1. =ps -aux=
   1. =sudo kill {rich TFC_server}=
   1. =TFC_server3.0 -s tfcodin21 {dim_dns_node of pc to run odin}=

sudo kill $(ps afx | grep "[T]FC_server" | gawk '{ print $1 };')

no 2nd argument necessary in line starting =TFC_server3.0= ???

dim_dns_node should be lbtbxp01

(or should dim_dns_node be lbontfc01 ?)

---+ Monitoring

---++ Online data monitoring
To start the online data monitoring display (testbeam version):
   1. Find =onlineMonitoring= window
   1. =bash=
   1. =cd /home/daq/test_beam_for_dummies/=
   1. =source start_hpd_viewer_on_lxplus.sh=
   1. keep typing =fg= until =GaudiOnlineMonitor= window will appear
   1. In the Gaudi Online Monitor window, click =connect=
   1. Find =HPDMoni= in the tree and select histograms from the list
   1. Select =colz= in =H2D= options
   1. Select =1D= in =stats= options
   1. Click =select=
   1. Click =start= 
NOTE: may need to redo =Select= and =Start= whenever options are changed.

 
Fig 2.

---++ Offline data monitoring
For new software (ONLINE_v3r3), works with both LHCb and ALICE modes:
   1. Log into =lbt9lxco01= as =lhcbrich=
   1. =. /group/rich/sw/scripts/Online_v3r3_Setup.sh=
   1. =. /group/rich/sw/scripts/offline_monitoring_setup_ssb2_2.sh=
   1. =nice /afs/cern.ch/lhcb/software/releases/BRUNEL/BRUNEL_v31r5/Rec/Brunel/v31r5/slc3_ia32_gcc323/Brunel.exe /home/daq/Online2/ssb2_hit_ntuple.opts=
(note: warnings about channels being suppressed are expected -- normally there should be 20 such messages in the first event, corresponding to (nLevel1Boards-nHPDs)=36-16)

For older software (but still post-testbeam: ONLINE_V2r2):
   1. Log into =lbt9lxco01= as =lhcbrich=
   1. =. /group/rich/sw/scripts/Online_v3r0_Setup.sh= (note the "v3r0" rather than "v2r2")
   1. =. /group/rich/sw/scripts/offline_monitoring_setup_ssb2.sh=
   1. =analysis2= (which will already have been created as an alias to =nice ../slc3_ia32_gcc323/TestBeam.exe ../options/ssb2_ntuple.opts= by the previous line)

Old version (from september 2006 testbeam):
   1. log into =lbt9lxco01= as user =adinolfi=
   1. =export CMTPATH=/home/daq/cmtuser=
   1. =. /home/daq/cmtuser=
   1. =cd /home/daq/cmtuser/Rich/TestBeam/r1r1/cmt=
   1. =. ./setup.sh=
   1. =../slc3_ia32_gcc323/TestBeam.exe ../options/run_on_current_output_file.opts=
   1. produces root file: =current-run.root=

Another old version (from september 2006 testbeam):
   1. On current run (or the last run, if data is not currently been taken)
      a. Log into =lbt9lxco01= as =adinolfi=
      a. =bash=
      a. =cd /home/daq/cmtuser/Rich/TestBeam/r1r1/cmt/=
      a. =export CMTPATH=/home/daq/cmtuser=
      a. =source setup.sh=
      a. =../slc3_ai32_gcc323/TestBeam.exe ../options/run_on_current_output_file.opts=
      a. File =current_run.root= is produced
      a. View histograms with root: =current_run.root=
   1. On an older run
      a. Steps a to e from above (on current run)
      a. Create a copy of =/home/daq/cmtuser/Rich/TestBeam/v1r1/options/run_on_current_output.opts=
      a. In your new copy, edit times containing:
         * =EventSelector.input=: for you to chosen input data file (e.g. =mdfOut.dat= using full path)
         * =HistogramPersistencySvc.OutputFile=: for you to chosen output file name (eg =file.root=)
      a. Run =../slc3_ia32_gcc323/TestBeam.exe {path to your new options}=
      a. Open your root file: =root {your .root file name}=

-- Main.wotton - 24 Jun 2007