EMF LTDB configuration at EMF

Jira for issues tracking:

If you have an issue during OpcUaLarLtdbServer (peripheral) operations please:

  1. Create a sub task on ATLLARONL-435 ticket
  2. Assigne Etienne FORTIN

Shell

All command are made for a bash shell

</>

IMPORTANT

  • NEVER RESTART OpcUaLarLtdbServer without having contacted CPPM experts before. This operation can put the system in bad state
  • Restart felixcore / opcuasca server only if there is an issue and the troubleshooter say to do it !

Experts

fw felix au 20 avril 2020 28-9-2019 at 16h41 This part is only for expert, it may put the system in bad state if used.

Change the FELIX firmware on pcemf-felix-01(if needed)

For using the peripheral it is needed to have a felix firmware different that tho one in use by Ka gui, to change it :

ssh -X pcemf-felix-01
source /data/FELIX/setup.sh
cd /data/ScriptInitFelix
./disable-minipod.sh    #If forgotten you may have to repower the FE crate
sudo /usr/local/bin/flxpci remove # DON'T FORGET THIS STEP, IF SO YOU MUST REBOOT THE COMPUTER
fflash -f 3 -L -b 1
rm fflash-tmp.txt # this file is created when you do fflash but if someone else did it before, he is the owner of this file and you can't do fflsh. So please clean it (remove) after fflash command
sudo shutdown -r now    ( # there is a bug in flxpci since we are in CC7 don't execute sudo /usr/local/bin/flxpci rescan)
./enable-minipod.sh

Felix configuration and felixcore start

Kill felixcore if running before you restart it.
ssh -X pcemf-felix-01
cd /data/ScriptInitFelix
./configurationFelixScript.sh

To check if felixcore is launched:

ssh -X pcemf-felix-01
cd /data/ScriptInitFelix/felixcore
./status.sh
</sticky>

To stop this server:

ssh -X pcemf-felix-01
cd /data/ScriptInitFelix/felixcore
./stop.sh #this will only work if you started felixcore yourself
sudo killall -3 felixcore #this will work if the server was started by someone else
</sticky>

Ping SCA

To test if all is good you can do a ping of all sca with this scan: The result have to be exactly the sames.

ssh -X lardcs@pcatllarltdbemf
source /opt/felix/setup.sh
cd /localdisk/tools/scanSca$
./scan.sh 

-------------------------LINK 1---------------------------
----------Scanning Sca for elink from 61 to 63
elink 3d found SCA, serial=30582
elink 3f found SCA, serial=30531
----------Scanning Sca for elink from 125 to 127
elink 7d found SCA, serial=30531
elink 7f found SCA, serial=31409
----------Scanning Sca for elink from 189 to 191
elink bd found SCA, serial=31409
elink bf found SCA, serial=30785
----------Scanning Sca for elink from 253 to 255
elink fd found SCA, serial=30785
elink ff found SCA, serial=30828
----------Scanning Sca for elink from 317 to 319
elink 13d found SCA, serial=30828
elink 13f found SCA, serial=30582
------- Result --------
Number of SCA found : 5

Launch OpcUaScaServer

The OpcUaScaServer should automatically be launched by a cron job. However, at the moment we have to start it manually:

ssh -X lardcs@pcatllarltdbemf 
cd /localdisk/OPC-PROJECTS/OpcUaScaServer
./start.sh configLTDB320-V2.xml
<

/!\ Take care of starting this server after felixcore

ssh -X lardcs@pcatllarltdbemf #Password provided by etienne fore some RC
cd /afs/cern.ch/user/l/lardcs/public/etfortin-peripheral/OpcUaScaConfig/
./start.sh configLTDB320-V2.xml # it create a log file with the actual date+time in logs/

The start script generate a script to see the last logs:

./tail_last.sh

The lasts log lines have to be:

***************************************************
 Server opened endpoints for following URLs:
     opc.tcp://pcatllartest.cern.ch:48020
***************************************************
2019-06-20 14:29.31.451179 [BaseQuasarServer.cpp:334, INF] ***************************************************
2019-06-20 14:29.31.451227 [BaseQuasarServer.cpp:335, INF] Press CTRL-C to shutdown server
2019-06-20 14:29.31.451281 [BaseQuasarServer.cpp:336, INF] ***************************************************

/!\ When you start this server it will reset the LTDB SCAs, it's a known issue of OpcUaScaServer Jira

To check if the server is launched :

ssh -X lardcs@pcatllarltdbemf
cd /afs/cern.ch/user/l/lardcs/public/etfortin-peripheral/OpcUaScaConfig/
./status.sh

To restart the server :

ssh -X lardcs@pcatllarltdbemf
cd /localdisk/OPC-PROJECTS/OpcUaScaConfig/
./stop.sh #'y' when it say that it don't have been launched by start script
#The watchdogs is configured to run every 2minutes so it may take 2 minute to restart, to check use status script

To see the logs the server :

ssh -X lardcs@pcatllarltdbemf
cd /localdisk/OPC-PROJECTS/OpcUaScaConfig/
./tail_last.sh

Launch OpcUaLarLtdbServer

The OpcUaLarLtdbServer should automatically be launched by a cron job. However, at the moment we have to start it manually:

ssh -X lardcs@pcatllarltdbemf 
cd /localdisk/OPC-PROJECTS/OpcUaLarLtdbServer
./start.sh config_EMF_ltdb_1c1.xml
<

On a different window

ssh -X lardcs@pcatllarltdbemf
cd /afs/cern.ch/user/l/lardcs/public/etfortin-peripheral/OpcUaLarLtdbServer
./start.sh config_EMF_ltdb_1c1.xml  # it create a log file with the actual date+time in logs/

The start script generate a script to see the last logs:

./tail_last.sh

The lasts log lines have to be:

***************************************************
 Server opened endpoints for following URLs:
     opc.tcp://pcatllarltdbemf:48030
***************************************************
2019-07-30 12:08.17.340425 [BaseQuasarServer.cpp:334, INF] ***************************************************
2019-07-30 12:08.17.340509 [BaseQuasarServer.cpp:335, INF] Press CTRL-C to shutdown server
2019-07-30 12:08.17.340552 [BaseQuasarServer.cpp:336, INF] ***************************************************

To check if the server is launched :

ssh -X lardcs@pcatllarltdbemf
cd /afs/cern.ch/user/l/lardcs/public/etfortin-peripheral/OpcUaLarLtdbServer
./status.sh

To stop this server:

ssh -X lardcs@pcatllarltdbemf
cd /afs/cern.ch/user/l/lardcs/public/etfortin-peripheral/OpcUaLarLtdbServer
./stop.sh
#The watchdogs is configured to run every 2minutes so it may take 2 minute to restart, to check use status script

To restart the server :

ssh -X lardcs@pcatllarltdbemf
cd /localdisk/OPC-PROJECTS/OpcUaLarLtdbServer
./stop.sh #'y' when it say that it don't have been launched by start script

To see the logs the server :

ssh -X lardcs@pcatllarltdbemf
cd /localdisk/OPC-PROJECTS/OpcUaLarLtdbServer
./tail_last.sh

User

Launch GUI

ssh -X lardcs@pcatllarltdbemf
WCCOAui -proj ATLLARLTDB -p main.pnl

Main panel

Select your setup: EMF or P1 with EMEC A, EMEC C, BARREL A, BARREL C
If one OpcUaLarLtdb Server is not connected (in red). Stop! and launch it

pannelPrincipalAvec_Erreur.jpg

For example, you choose EMF, an other panel appears

  1. Transition control
  2. Button to force check links: only when system is configured
  3. Copy the full state into the clipboard to paste it into elogs or mails to experts
  4. Status of two LTDB in one crate, Green is good, Red is Bad. You can click on it to have the crate details panels. (for troubleshooting)
  5. DBG MODE: RESERVED FOR GUI DEVELLOPER
  6. Indicate the global state and the time of the last transition
Be careful: when transition is running, don't care the state of the panel....status could be red during this step....

main-desc.png

This pannel shows the status of all boards and controls the transition for the full system (At EMF only one board for now). A green color on one board means that this board is in good state, a red one that there is an error. The differents states and transitions of LTDB are :

ltdb_state.png

To be ready for data taking the global state must be configured and LTDB green.

For stability issue, you can check the links status by pressing the corresponding button. If the LTDB stay green after that the system is ready ( automatically done after configure).

"Copy state" will copy into clipboard the full status of the system, you can paste it into elog or mail.

Now you click on the part 4, the green zone for E01 for example...appears le LTDB control panel of this E01 LTDB board. Description in the following chapter.

Ltdb monitoring/control panel

In the control panel, as the previous panel, you have the differents transitions: in grey, the transitions that have been yet done.
Every thing is ok if no red messages / no red zone.
The state of the LTDB you are looking should be coherent with the global state: a reminder at the top of the panel.

HOW TO SEE Monitoring: Clicking on the LTDB zone, you can have a look to differents parametres that are monitored: for the moment sca temperatures

Attention: After doing "CONFIG", it's better to click on the "Links Status" to update it.

* LTDB control panel:

ControlPanel.jpg

* monitoring LTDB panel:
Monitoringpanel.jpg

Errors messages:
1)
Erreur1.jpg
Follows instructions in the message: go back to the OFF state that means do transitions UNCONFIG and SHUTDOWN in this windows and after returns to the current Global state (here CONFIGURED) doing INIT and CONFIG in this windows (LTDB control panel)

2)

Erreur2.jpg
the status of the LTDB is RED= After CONFIGURE, "Links Status " is BAD => Expert can do a scan of the LOCx2 (Expert mode )


3) the log window: Liste of error code and recovery mesure: OpcUaLarLtdbServerTroubleshooter

-- EtienneMarieFortin - 2019-06-17

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf 08-10-19-SCA_RECOVERY.pdf r1 manage 244.9 K 2019-10-08 - 17:13 EtienneMarieFortin  
PNGpng Adc.png r1 manage 59.2 K 2019-06-26 - 10:21 EtienneMarieFortin  
PNGpng Capture_d’écran_de_2020-04-16_13-21-39.png r1 manage 48.5 K 2020-04-16 - 14:01 EtienneMarieFortin  
JPEGjpg ControlPanel.jpg r1 manage 75.4 K 2020-05-29 - 11:53 DanieleLaugier  
JPEGjpg EcranPP.jpg r1 manage 58.9 K 2020-05-13 - 11:31 DanieleLaugier  
JPEGjpg Erreur1.jpg r1 manage 38.8 K 2020-05-13 - 11:35 DanieleLaugier  
JPEGjpg Erreur2.jpg r1 manage 39.1 K 2020-05-13 - 11:35 DanieleLaugier  
JPEGjpg LTDB-panel-comments.jpg r1 manage 85.9 K 2020-05-13 - 11:33 DanieleLaugier  
JPEGjpg LTDB_panel.jpg r1 manage 49.0 K 2020-04-23 - 14:21 DanieleLaugier LTDB control panel
JPEGjpg Monitoringpanel.jpg r1 manage 42.2 K 2020-04-23 - 14:37 DanieleLaugier monitoring LTDB panel
PNGpng StatusPeripheral.png r2 r1 manage 39.1 K 2019-10-08 - 17:09 EtienneMarieFortin  
PNGpng config.png r2 r1 manage 24.0 K 2019-10-08 - 17:15 EtienneMarieFortin  
PNGpng configure_1c1.png r1 manage 36.9 K 2019-06-25 - 16:21 EtienneMarieFortin  
PNGpng elinkconfigSteps.png r1 manage 180.7 K 2019-06-17 - 11:35 EtienneMarieFortin  
PNGpng locx2_pannel.png r1 manage 101.4 K 2019-06-26 - 09:25 EtienneMarieFortin  
PNGpng ltdb_state.png r1 manage 14.6 K 2020-04-16 - 13:44 EtienneMarieFortin  
PNGpng main-desc.png r1 manage 48.5 K 2020-04-16 - 14:01 EtienneMarieFortin  
PNGpng main.png r1 manage 24.0 K 2019-07-03 - 22:30 EtienneMarieFortin  
JPEGjpg pannelPrincipalAvec_Erreur.jpg r1 manage 43.5 K 2020-05-14 - 16:09 DanieleLaugier  
PNGpng startProject.png r1 manage 156.6 K 2019-06-25 - 16:03 EtienneMarieFortin  
Edit | Attach | Watch | Print version | History: r73 < r72 < r71 < r70 < r69 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r73 - 2021-02-18 - DanieleLaugier
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback