Timing systems advanced operations

On-Call Schedule

Link to on-call schedule: https://teamup.com/ks9fj1qdrba84uozmd On-call phone number: +41227665291

Repo links

Link to firmware repo: https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-firmware

Link to the software repo: https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-software

CERN System setup

The system is meant to be operated from the control pc np04-srv-012 with the exceptions of operations requiring a JTAG connection to the FPGA (i.e. firmware uploads).

Hardware

The JTAG identifiers for the systems at EHN1 are as follows:

Board IPBus ID IP address JTAG pc JTAG Cable Shortcut Full JTAG Cable ID
TLU PROD_MASTER 192.168.200.64 np04-onl-001 53FA Digilent/25163300153FA
Fanout 0 PROD_FANOUT_0 192.168.200.65 np04-onl-001 4E7A Digilent/2516330014E7A
Fanout 1 PROD_FANOUT_1 192.168.200.66 np04-onl-001 4D9A Digilent/2516330014D9A
Fanout 2 PROD_FANOUT_2 192.168.200.67 np04-onl-001 4E4A Digilent/2516330014E4A
TERTIARY PDTS_TERTIARY 192.168.200.32 np04-onl-002 F4CB Digilent/210299A1F4CB
PRIMARY PDTS_PRIMARY / CRT_ENDPOINT 192.168.200.16 np04-crt-001 c301 XiLinx/00001876c3c301
SECONDARY PDTS_SECONDARY 192.168.200.17 np04-onl-001 4001 XiLinx/000018af5f4001

Connections

Board Usage/Connection Firmware Image Connection file
TLU Cryostat and COBs 2-8 v5b0-1/overlord_tlu_relval-v5b0-1.tgz connections_v5a1.xml
Fanout 0 Cryostat and COBs 2-8 v5b0-1/fanout_0_pc059_relval-v5b0-1.tgz connections_v5a1.xml
Fanout 1 - v5b0-1/fanout_1_pc059_relval-v5b0-1.tgz connections_v5a1.xml
Fanout 2 Timing tests v5b0-1/fanout_2_pc059_relval-v5b0-1.tgz connections_v5a1.xml
TERTIARY VST and COB1 v5b0-1/ouroboros_pc059_relval-v5b0-1.tgz connections_v5a1.xml
CRT CRT endpoint /nfs/sw/timing/milestones/v4/firmware/v4b4/crt_fmc_180905_1239.tgz connections_v4b1-rc1.xml

Essential recovery guide

Updating/reloading timing firmware is an expert operation. Please contact a timing expert before attempting a system recovery.

PROM (MCS) files in np04-srv-007:/sw/timing/dev/firmware/promfiles

MCS files have been given the same name as the *.tgz archive containing the *.bit file they were generated from

Cold reset

Reload firmware
ssh np04-onl-001
source /nfs/sw/timing/tools/ipbb_env.sh
cd /nfs/sw/timing/dev/firmware/v5b0-1
ipb-prog vivado program 53FA overlord_tlu_relval-v5b0-1.tgz
ipb-prog vivado program 4E7A fanout_0_pc059_relval-v5b0-1.tgz
ipb-prog vivado program 4D9A fanout_1_pc059_relval-v5b0-1.tgz
ipb-prog vivado program 4E4A fanout_2_pc059_relval-v5b0-1.tgz
exit

Configure hardware layer

ssh np04-srv-012
source /nfs/sw/timing/env_dev.sh

# Hard resets
pdtbutler io PROD_MASTER reset
pdtbutler io PROD_FANOUT_0 reset
pdtbutler io PROD_FANOUT_1 reset
pdtbutler io PROD_FANOUT_2 reset

pdtbutler mst PROD_MASTER synctime
pdtbutler mst PROD_MASTER spillenable

# Trigger interface configuration
pdtbutler mst PROD_MASTER ext-trig ept reset
pdtbutler mst PROD_MASTER ext-trig enable --on

# Trigger interface monitor
pdtbutler mst PROD_MASTER ext-trig status

Trigger-Timing link recovery

# Trigger interface configuration
pdtbutler mst PROD_MASTER ext-trig ept reset

# Trigger interface monitor
pdtbutler mst PROD_MASTER ext-trig status

Command output

Control registers
+-------------+-----+
| ep_en       | 0x1 |
| ext_trig_en | 0x1 |
+-------------+-----+

Status registers
+---------+-----+
| ep_edge | 0x0 |
| ep_fdel | 0x2 |
| ep_rdy  | 0x1 |
| ep_stat | 0x8 |
+---------+-----+

...

CRT timing endpoint recovery

Set up the working environment
ssh np04-crt-001
cd /nfs/sw/crt/timing-board-software
source tests/env.sh

pdtbutler io CRT_EPT reset --force-pll-cfg $PDT_TESTS/etc/clock/devel/Si5344-PDTSCRT1NoZdm-RevD-400HzBW-Registers.txt

Check the endpoint status : ep_rdy=1, ep_stat=0x8

pdtbutler crt CRT_EPT status

Command output

Created crt device
+------------------+------------+
| csr              | 0x0        |
| csr.ctrl         | 0x0        |
| csr.ctrl.tgrp    | 0x0        |
| csr.stat         | 0x81       |
| csr.stat.ep_rdy  | 0x1        |
| csr.stat.ep_stat | 0x8        |
| pulse            | 0x0        |
| pulse.cnt        | 0x11       |
| pulse.ctrl       | 0x0        |
| pulse.ctrl.cmd   | 0x0        |
| pulse.ctrl.en    | 0x0        |
| pulse.ctrl.force | 0x0        |
| pulse.ts_h       | 0x116f765  |
| pulse.ts_l       | 0x90232afa |
+------------------+------------+

Configure the endpoint to emit a sync pulse at Run Start

pdtbutler crt CRT_EPT configure 0 RunStart

Timing system advanced user guide

Firmware

  • Reference project/tag: ouroboros (master + internal endpoint)/ v4b1
  • Firmware repository: https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-firmware
  • In-situ firmware image repository: /nfs/sw/timing/dev/firmware
  • Current firmware images: Check the Connections section.

Software

  • Software repository: https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-software
  • Reference installation: /nfs/sw/timing/pro/software/timing-board-software
  • Local address table repository: /nfs/sw/timing/pro/addrtab/[tag]

Version Compatibility Warnings

Please ensure you are running Python 2.7 with the below libraries installed, and that you have boost 1.53 (exactly). Both of them as standard packages available to Centos 7 installation. Additionally, the uhal package is required to communicate with the timing firmware. and uhal C++ library version 2.6 (exactly) should be installed. As a list:

Requirements:

  • uhal C++ library, version 2.6
  • python 2.7 with Click and Click_didyoumean packages
  • boost 1.53

All can be checked when you source env_dev.sh:

[jogreer@np04-srv-012 timing]$ source /nfs/sw/timing/env_dev.sh
uhal is installed
click is installed
click_didyoumean is installed
- LD_LIBRARY_PATH += /opt/cactus/lib
- PATH += /nfs/sw/timing/milestones/v4/software/v4b1-rc1/timing-board-software/core/bin
- PATH += /nfs/sw/timing/milestones/v4/software/v4b1-rc1/timing-board-software/tests/bin
- PATH += /nfs/sw/timing/milestones/v4/software/v4b1-rc1/timing-board-software/tests/scripts
- LD_LIBRARY_PATH += /nfs/sw/timing/milestones/v4/software/v4b1-rc1/timing-board-software/core/lib
- PYTHONPATH += /nfs/sw/timing/milestones/v4/software/v4b1-rc1/timing-board-software/python/pkg
- PYTHONPATH += /nfs/sw/timing/milestones/v4/software/v4b1-rc1/timing-board-software/tests/python

Upload timing firmware at np04

Updating/reloading timing firmware is an expert operation. Please contact a timing expert before doing it.

Connect to the appropriate JTAG host (see Hardware).

Source ipbb environment

source /nfs/sw/timing/tools/ipbb_env.sh

Once the ipbb environment is loaded

ipb-prog vivado program <jtag cable shortcut>:xc7a35t_0 <bitfile path>

The cable names and bitfiles paths are listed the Hardware and Firmware sections.

Example:

ipb-prog vivado program c301:xc7a35t_0 /nfs/sw/timing/pro/firmware/ouroboros_v3c/top.bit

Configure the physical layer

This step is only required after loading the FPGA with new firmware. Login to np04-srv-012 to implement pdtbutler commands. The reset command resets all ipbus registers in the master, loads the clock synthesizer chip configuration and starts the sync command generator.

pdtbutler io PDTS_TERTIARY reset

Reset and configure the clock infrastructure

Resets all master's ipbus registers, loads the SI clock chip configuration and starts the sync command generator. It takes option: --soft, which is a speedier reset that skips the clock chip configuration.

pdtbutler io PDTS_TERTIARY reset

Command output

Created device PDTS_TERTIARY
Master FW version: 0x40004
Resetting PDTS_TERTIARY
UID I2C Slaves
  FMC_UID_PROM: 0x53
  AX3_Switch: 0x21
  KC705_Switch: 0x74
I2C enable lines: 127
Timing Board UID: 0xd880395e5069
PLL version : 0x5344
Clock configuration to load SI5344/PDTS0000.txt
2018-03-12 10:28:17.239 pdt INFO     | Configuration read from file (445 entries). Starting upload...
Freq: 0 1 249.994377134
Freq: 1 1 249.994377134

--- Global status ---
tx_err: 0x0

Disabled command generator
Time-sync generator enabled

Help

   Usage: pdtbutler io reset [OPTIONS]

  Perform a hard reset on the timing master, including

  - ipbus registers
  - i2c buses
  - pll and pll configuration

Options:
  -s, --soft               Soft reset i.e. skip the clock chip configuration.
  -h, --help               Show this message and exit.

Installation and build instructions for standalone systems

Please ensure you are running Python 2.7, and that you have boost 1.53 (exactly) installed. Both of them as standard packages available to Centos 7 installation. Additionally, the uhal package is required to communicate with the timing firmware. The installation instructions of the uhal package are available at the following address

https://ipbus.web.cern.ch/ipbus/doc/user/html/software/installation.html

Download the v4a3 software tag

git clone -b v4a3 --depth 1 https://:@gitlab.cern.ch:8443/protoDUNE-SP-DAQ/timing-board-software.git

Compile the C++ layer

cd timing-board-software
make

Source the environment

source tests/env.sh

Ready to go!

Configure and start the timing master

The typical configuration & start sequence for stand-alone test setups is

pdtbutler io PDTS_TERTIARY reset --soft
pdtbutler mst PDTS_TERTIARY faketrig-conf 0 12
pdtbutler mst PDTS_TERTIARY part 0 configure
pdtbutler mst PDTS_TERTIARY part 0 start
pdtbutler mst PDTS_TERTIARY part 0 trig
pdtbutler mst PDTS_TERTIARY part 0 status

In this example the fake trigger rate is set to 12 Hz (faketrig-conf 12) and only partition 0 is enabled. Note that triggers need to be explicitly enabled for the chosen partition (part 0 trig) after start. Please note that the trigger command generator is a global entity. Trigger rate changes will affect all partitions.

If required, the fake trigger generator is disabled by running the command

pdtbutler mst PDTS_TERTIARY faketrig-clear 0

Periodic and poisson triggers

The fake triggers are typically set to periodic. To set them to a poisson (random) distribution use the --poisson flag
pdtbutler mst PDTS_TERTIARY faketrig-conf 0 12

Reference info

The status register for the board has the following definition:

State Description
0x0 Standing by
0x1 Waiting SFP for signal
0x2 Waiting CDR lock
0x3 Waiting for good frequency check
0x4 Waiting for comma alignment
0x5 Waiting for 8b10 decoder good packet
0x6 Waiting for time stamp initialisation
0x8 Ready
0xc Error in Rx
0xd Error in time stamp check
0xe Error in physical layer after lock

The 'sync' bus has the following commands at the moment:

TimeSync 0x0
Echo 0x1
SpillStart 0x2
SpillStop 0x3
RunStart 0x4
RunStop 0x5
WibCalib 0x6
SSPCalib 0x7
FakeTrig0 0x8
FakeTrig1 0x9
FakeTrig2 0xa
FakeTrig3 0xb
BeamTrig 0xc
NoBeamTrig 0xd
ExtFakeTrig 0xe

Fanout0 connections

Slot APA
0 CRT
1 US-RaS (3)
2 DS-DaS (4)
3 MS-DaS (6)
4 US-DaS (5)
5 DS-RaS (1)
6 MS-RaS (2)
7 COBs

Connections to TLU from SPS timing signals.

Signal Label in TLU Signal
WWE 6 (right hand side) 5
WE 5 4
EE 4 3

All signals NIM. Should be 2us wide. For definitions see, e.g., http://sba.web.cern.ch/sba/BeamsAndAreas/H2/H2_faq.html#What_is_the_timing_signal_that_I_receive_from_the_SPS__

White Rabbit

We have a WR-LEN module ( http://sevensols.com/index.php/products/wr-len/ ) that provides a 10MHz clock ( and IRIG-B time-code that we don't use at the moment ) . This clock is connected to the TLU via a Balun ( SMA at WR-LEN, 2-pole Lemo-00 at TLU ).

WR-LEN is monitored over USB Serial link ( currently connected to np04-onl-001 ). Can use e.g. minicom as a terminal ( e.g. , sudo minicom -D /dev/ttyUSB0 -b 115200 )

Use the "gui" command at the "wrc#" prompt to display status. e.g.

WR PTP Core Sync Monitor: PPSI - LEN board    
Esc = exit

TAI Time:                  Tue, Sep 4, 2018, 14:06:08

WR-LEN mode : WRC_SLAVE_WR0 
-----------------------
Link status:                                                                                                                                            
                                                                                                                                                        
 wr0 : Link up   (RX: 334697, TX: 102136), mode: WR Slave   Locked  Calibrated                                                                          
 IPv4: BOOTP running                                                                                                                                    

 wr1 : Link down 
-----------------------

Servo state:               TRACK_PHASE
Phase tracking:            ON
Synchronization source:    wr0

Timing parameters:

Round-trip time (mu):    1188624 ps
Master-slave delay:      598623 ps
Master PHY delays:       TX: 224603 ps, RX: 235091 ps
Slave PHY delays:        TX: 205019 ps, RX: 224093 ps
Total link asymmetry:        -8622 ps
Cable rtt delay:         299818 ps
Clock offset:                   -6 ps
Phase setpoint:               4926 ps
Skew:                            1 ps
Manual phase adjustment:         0 ps
Update counter:              89112 
--

... I think the things we are looking for are " WR Slave Locked Calibrated" and "Servo state: TRACK_PHASE"

Timing system on call Instructions

On call duties

What tasks are the on-call person expected to do? These bullets correspond to sections below.

  • Make sure you can access the np04.timing@cernNOSPAMPLEASE.ch Skype for Business account
  • Read the on-call section of this page at the start of your duties
  • Perform regular status checks
  • Respond to phone calls to the np04.timing@cernNOSPAMPLEASE.ch Skype for Business account
  • Attend meetings
  • Report back to PDTS group at the end of the week
  • Hand over to the next on-call person

When you encounter a problem

  • Get basic debug info
  • Check known issues
  • Know which expert to call

Problems that on-call person can fix without calling an expert

  • ...

Known issues are:

  • ...

Who to call for known issues:

  • ...

Contact details for experts:

  • ...

Instructions for on call duties

Basic overview of how PDTS works

...

Accessing the np04.timing@cernNOSPAMPLEASE.ch Skype for Business account

How to perform regular status checks

Meetings you need to attend (times+locations)

How to get basic debug info from the system

How to reload firmware

Instructions above in the Essential recovery guide (for full recovery of timing system) or Upload timing firmware at np04 sections (upload individual firmware files).
Topic attachments
I Attachment History Action Size Date Who Comment
Cascading Style Sheet filecss colors.css r1 manage 0.9 K 2018-08-28 - 10:25 AlessandroThea  
Cascading Style Sheet filecss style.css r1 manage 3.2 K 2018-08-28 - 10:25 AlessandroThea  
Edit | Attach | Watch | Print version | History: r34 < r33 < r32 < r31 < r30 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r34 - 2019-10-07 - AlessandroThea
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CENF All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback