Documents

MIT POCC

Monitoring_POCC_in_MIT.pdf

Asia POCC

AsiaPOCCNetworkTopology20130220.ppt

AMS Data Flow (photo)

AMS_Data_Flow_photo.jpg

Procedures

Lost data recovery (from Laptop, after signal loss)

Finding

Playback table.

Go to ams-backup.cerh.ch/playback (web intfc). Find records marked red.

Check for each day:

Look at:


t2nr = time 2 next run, shouldn't be > 00:22


lei = lost events

Check for the LEAD/DATA shift logs (at this time interval, 23min) for explanations,

- if none, then we'll need to playback this run (23min)

- if found, then add it as a comment

To playback, first, need to make find-files (which we are gonna playback):

- execute commands on the do_find page (Oleg's web face) on a console in POCC and on the AMS laptot

- kufwd_shell (not in the console's shell)

- don't forget to close forward tunnel to laptop

Mark in the logbook that file-finding-and-protection performed.

Playing back

Connect to DMC over DMC loop.

Ask for AMS switch to configuration Foxtrot (as we need to playback data).

Ask for XX Mbit/sec during YY minutes.

Confirm that it with some log monitor, check for "hrdl: ... , no signal illegal"

Specify the frames to playback and the speed 39.5Mbit/sec (errors if 40)

Start the playback with "W", check for no LOS and transfer speed (Mbit/s= 40),

Keep watching for errors

Check for additional streams

Look for errors when jumps between chunks (non-consecutive frames)

Mark in the logbook: started playback of the missing data...

(when finished) Mark in the logbook (and send a e-mail to baosong): playback completed, runs .. and ..

Mark as green (playback'ed) in the web interface.

Clean-up space on pcpos[p,c][0,1]

1. Receive a warning from nagios

2. Send an e-mail to Mike to ask for the FRAMES and BLOCKS numbers

3. Check those are BACK-UP'ed.

4. For pcposXX in pcpos[p,c][0,1] do:

4.0. Change user to ams: su ams

4.1. Execute: [apashnin@pcposp1 ~]$ /home/ams/bin/delete_frames_blocks_data.sh XXXX YYYY

where: XXX = frame number, YYY = block number

Graphana

Running Django+SQLite on ams-stats.cern.ch

Graphana for data from collectd on pcpos[p,c][0,1]

[root@pcposp0: /pocchome/apashnin ] ps aux| grep collectd
root 12274 0.0 0.0 6116 144 ? Ss Mar08 0:00 collectdmon -P /var/run/collectdmon.pid -c /usr/sbin/collectd -- -C /etc/collectd.conf
root 12275 0.8 0.0 806896 12424 ? SLl Mar08 1121:13 /usr/sbin/collectd -C /etc/collectd.conf -f

[root@pcposp0: /pocchome/apashnin ] vim /etc/collectd.conf
Hostname    "pcposp0.cern.ch"
##LoadPlugin dbi
LoadPlugin df
LoadPlugin disk
##LoadPlugin dns
<Plugin df>
 Device "/dev/sda1"
 Device "/dev/sda2"
 Device "/dev/sda3"


<Plugin processes>
 Process "HOSCfep"
 Process "HOSCfepRIC"
 #Process "mtr"
 Process "deframing"
 Process "datautil"
 Process "bbftpd"
</Plugin>
<Plugin write_graphite>
 <Node "example">
 Host "ams-stats.cern.ch"
 Port "2003"
 Protocol "tcp"
 LogSendErrors true
 Prefix "collectd."
# Postfix "collectd"
 StoreRates true
 AlwaysAppendDS false
 EscapeCharacter "_"
 </Node>
</Plugin>





The rest

-- AndreyPashnin - 2017-06-06

TestpageURL TEST Message

Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg AMS_Data_Flow_photo.jpg r1 manage 5509.7 K 2017-06-09 - 11:24 AndreyPashnin  
PowerPointppt AsiaPOCCNetworkTopology20130220.ppt r1 manage 269.0 K 2017-06-08 - 16:28 AndreyPashnin  
PDFpdf Monitoring_POCC_in_MIT.pdf r1 manage 1133.6 K 2017-06-08 - 16:27 AndreyPashnin  
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2017-06-12 - AndreyPashnin
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback