PD-I timing system endpoint alignment

Details

Date 05/08/2020
Location University of Bristol lab
People Stoyan Trilov, David Cussans

Test objective

The alignment of local timestamps maintained by timing endpoints relies on each endpoint having an individualised delay applied to its incoming datastream in order to compensate for differences in fibre lengths and other factors. The endpoint delay can be applied in two different granularities, coarse and fine. Coarse adjustment is done in units of 20 ns, corresponding to the period of the 50 MHz used to maintain the endpoint timestamp. Fine adjustments are done in units of 4 ns, corresponding to the 250 MHz IO clock used to transmit the serial datastream. This test aims to demonstrate the successful application of endpoint delays in coarse and fine units. As well as studying the alignment of the timing messages between the two endpoints, the test also looked at the characteristics of the clocks recovered by the endpoints.

Test setup

The test was performed using an AIDA 2020 TLU, acting as the timing master, running the overlord firmware design. Two endpoints were used to test the application of delays. Both endpoints consist of a PD-I timing FMC, hosted by an Enclustra FPGA baseboard (PM3) with an AX3 FPGA module. The endpoints were running a modified version of the endpoint firmware design, which can be found at https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-firmware/-/commit/ffcbac4b9c3f7f4c29ac7f515c3b8ad385100d1a. The endpoint firmware was modified so that it emits an electrical "sync" pulse upon the reception of an "Echo" command. The signal path for the electrical pulse is as follows:
Endpoint signal -> FMC IO signal -> FPGA pin -> SO DIMM connector pin -> FMC connector pin -> FMC output
rx_pulse (endpoint_wrapper) -> gpout_0 -> L6/L5 -> 90/92 -> G9/G10 (FMC_LA13_P/FMC_LA13_N)  -> PX5 

The clock output from the timing FMC clock generator can be found on the PX4 connector.

The TLU transmits the timing datastream to the endpoints over single mode fibres, and a passive optical splitter. One of the endpoints, which will be referred to as "endpoint 1" throughout is connected using a 2 m fibre, whereas the other endpoint uses a 120 m fibre and will be referred to as "endpoint 2". A diagram of the experimental setup is shown below.

img1309f87c524f77c47306ed651fb5ff6c.png
Experimental setup of endpoint alignment test

To compensate for the different fibre lengths, endpoint 2 had no delay applied to it, whereas endpoint 1 had several different delay values applied. The delay value used for endpoint 1 are listed in the table below.

Coarse units Fine units Total delay [ns]
0 0 0
28 0 560
28 4 576

The time between the two pulses was measured using two different oscilloscopes PicoScope 6403C and LeCroy WR640. The PicoScope was used for single measurements of the pulse interval, whereas the LeCroy was used to collect a higher statistics sample.

The timing software used for the test can be found here: https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-software/-/commits/feature/fine_delay_adjustment. Installation instructions for the timing software can be found here: https://twiki.cern.ch/twiki/bin/view/CENF/TimingSystemAdvancedOp#Installation_and_build_instructi, where the git clone command should be changed to the one below.

git clone ssh://git@gitlab.cern.ch:7999/protoDUNE-SP-DAQ/timing-board-software.git -b feature/fine_delay_adjustment
In order to interact with the two endpoints, the following two entries will need to be added to the timing software connections.xml file.

 <connection id="EPT_0"             uri="ipbusudp-2.0://192.168.200.16:50001" address_table="file://${PDT_TESTS}/etc/addrtab/v5a2/endpoint_fmc/top.xml" />
 <connection id="EPT_1"             uri="ipbusudp-2.0://192.168.200.17:50001" address_table="file://${PDT_TESTS}/etc/addrtab/v5a2/endpoint_fmc/top.xml" />

N.B. The firmware as is from the git repository will build with an IP address of 192.168.200.16. In order to obtain a bitfile with the IP address 192.168.200.17, line 100 in the file projects/master/firmware/hdl/top_enclustra_ax3_pm3.vhd needs to be changed to:

addr <= X"1";

Test method

Timing master configuration

To configure the TLU, execute the following pdtbutler timing commands.
pdtbutler io PROD_MASTER reset
pdtbutler mst PROD_MASTER synctime
pdtbutler mst PROD_MASTER part 0 configure
pdtbutler mst PROD_MASTER part 0 status
The last command should yield something similar to the output below.
Created device PROD_MASTER
ID: design 'overlord' on board 'tlu' on carrier 'enclustra-a35'
Master FW rev: 0x50100, partitions: 4, channels: 5

-- Master state---

=> Cmd generator counters
----------------------------------------------------------------------------
|              |       Accept counters       |       Reject counters       |
----------------------------------------------------------------------------
|     Chan     |     cnts     |     hex      |     cnts     |     hex      |
----------------------------------------------------------------------------
|     0x0      |     9154     |    0x23c2    |      0       |     0x0      |
|     0x1      |      0       |     0x0      |      0       |     0x0      |
|     0x2      |      0       |     0x0      |      0       |     0x0      |
|     0x3      |      0       |     0x0      |      0       |     0x0      |
|     0x4      |      0       |     0x0      |      0       |     0x0      |
----------------------------------------------------------------------------

=> Partition 0
Control                   Status registers  
+---------------+------+  +----------+-----+
| buf_en        | 0x0  |  | buf_err  | 0x0 |
| frag_mask     | 0x0  |  | buf_warn | 0x0 |
| part_en       | 0x1  |  | in_run   | 0x0 |
| rate_ctrl_en  | 0x1  |  | in_spill | 0x0 |
| run_req       | 0x0  |  | part_up  | 0x1 |
| spill_gate_en | 0x1  |  | run_int  | 0x0 |
| trig_ctr_rst  | 0x0  |  +----------+-----+
| trig_en       | 0x0  |                    
| trig_mask     | 0xf1 |                    
+---------------+------+                    

Timestamp: 0x11b9eb8da5bcd74 -> Wed, 05 Aug 2020 15:45:03 +0000
EventCounter: 0
Buffer status: OK
Buffer occupancy: 0

----------------------------------------------------------------------------
|              |       Accept counters       |       Reject counters       |
----------------------------------------------------------------------------
|     Cmd      |     cnts     |     hex      |     cnts     |     hex      |
----------------------------------------------------------------------------
|   TimeSync   |    627255    |   0x99237    |      0       |     0x0      |
|     Echo     |     9217     |    0x2401    |      0       |     0x0      |
|  SpillStart  |      0       |     0x0      |      0       |     0x0      |
|  SpillStop   |      0       |     0x0      |      0       |     0x0      |
|   RunStart   |      0       |     0x0      |      0       |     0x0      |
|   RunStop    |      0       |     0x0      |      0       |     0x0      |
|   WibCalib   |      0       |     0x0      |      0       |     0x0      |
|   SSPCalib   |      0       |     0x0      |      0       |     0x0      |
|  FakeTrig0   |      0       |     0x0      |      0       |     0x0      |
|  FakeTrig1   |      0       |     0x0      |      0       |     0x0      |
|  FakeTrig2   |      0       |     0x0      |      0       |     0x0      |
|  FakeTrig3   |      0       |     0x0      |      0       |     0x0      |
|   BeamTrig   |      0       |     0x0      |      0       |     0x0      |
|  NoBeamTrig  |      0       |     0x0      |      0       |     0x0      |
| ExtFakeTrig  |      0       |     0x0      |      0       |     0x0      |
|     0xf      |      0       |     0x0      |      0       |     0x0      |
----------------------------------------------------------------------------

Endpoint configuration

To configure the endpoints, execute the following sequence of commands, where corresponds to the endpoint ID (as configured in connections.xml), and <adr> is the address assigned to the endpoint. For the test described here, the values used are in the table below.

Endpoint ID Endpoint address Endpoint IP address
0 1 192.168.200.16
1 2 192.168.200.17

pdtbutler io EPT_<id> reset --force-pll-cfg tests/etc/clock/devel/Si5344-PDTSCRT1NoZdm-RevD-400HzBW-Registers.txt
pdtbutler ept EPT_<id> 0 enable -a <adr>
pdtbutler ept EPT_<id> 0 status
The last command should yield something similar to the output below.
Created endpoint device EPT_0
+---------+
|    0    |
| 0x50100 |
+---------+
+------------------+--------------------------------------------+
|     Endpoint     |                     0                      |
+------------------+--------------------------------------------+
| State            | Waiting for phase adjustment command (0x6) |
| Partition        | 0                                          |
| Address          | 1                                          |
| Timestamp        | Wed, 05 Aug 2020 16:04:15 +0000            |
| Timestamp (hex)  | 0x11b9ec64384e0b6                          |
| EventCounter     | 0                                          |
| Buffer status    | Error                                      |
| Buffer occupancy | 0                                          |
+------------------+--------------------------------------------+

--- Endpoint state ---
+------------+----------+
|  Endpoint  |    0     |
+------------+----------+
| buf_err    |   0x1    |
| buf_warn   |   0x0    |
| ep_rdy     |   0x0    |
| ep_rsto    |   0x1    |
| ep_stat    |   0x6    |
| in_run     |   0x0    |
| in_spill   |   0x0    |
| sfp_tx_dis |   0x1    |
+------------+----------+

--- Command counters ---
+--------------+----------+
|   Endpoint   |    0     |
+--------------+----------+
| TimeSync     |   428    |
| Echo         |          |
| SpillStart   |          |
| SpillStop    |          |
| RunStart     |          |
| RunStop      |          |
| WibCalib     |          |
| SSPCalib     |          |
| FakeTrig0    |          |
| FakeTrig1    |          |
| FakeTrig2    |          |
| FakeTrig3    |          |
| BeamTrig     |          |
| NoBeamTrig   |          |
| ExtFakeTrig  |          |
| None         |          |
+--------------+----------+

Delay adjustment

Endpoint delays are applied via a timing command issued from the timing master, addressed to a particular endpoint. To apply a delay composed from <cdelay> coarse units and <cfdelay> fine units to an endpoint with address <adr>, issue the following command.

pdtbutler mst PROD_MASTER align apply-delay <adr> <cdelay> <fdelay> --force

After the configuration of the delay, the endpoint should transition to state 0x8 (Ready).

Echo command trigger

To send an "Echo" command to the timing endpoints, and hence trigger the pulses from the endpoints, issue the following command.

pdtbutler mst PROD_MASTER send-cmd Echo 0

If the command is issued successfully, an output similar to the one below should be received.

Created device PROD_MASTER
ID: design 'overlord' on board 'tlu' on carrier 'enclustra-a35'
Master FW rev: 0x50100, partitions: 4, channels: 5
Command sent Echo(1) from generator 0 @time 0x11b9ef6167cd29f 79832198185538207

Test results

Command synchronisation

0 ns delay

The time between the two pulses without delay adjustment for endpoint 1 was measured to be around 575.2 ns. This is compatible with the additional 118 m of fibre, as 1 m of fibre introduces approximately 4.9 ns of delay. A screenshot of the oscilloscope output is shown below.

img0ef10aeb42a8f6030eae619b3d727762.png
Screenshot of the PicoScope oscilloscope output measuring the time between the two endpoint pulses. Endpoint 1 delay: 0 ns

560 ns delay

The time between the two pulses, with a delay of 560 ns applied to endpoint 1 was measured to be around 15.7 ns, as it would be expected. A screenshot of the oscilloscope output is shown below.

img68440e7c4385ae5f3f81f9ce94cdb03b.png
Screenshot of the PicoScope oscilloscope output measuring the time between the two endpoint pulses. Endpoint 1 delay: 560 ns

576 ns delay

The time between the two pulses, with a delay of 576 ns applied to endpoint 1 was measured to be 267.2 ps, with a spread of 157.0 ps. A screenshot of the oscilloscope output and a plot of the pulse time differences are shown below.

img80a592d74d29f7f339d0b0014f2c1059.jpg
Screenshot of the LeCroy oscilloscope output measuring the time between the two endpoint pulses. Endpoint 1 delay: 576 ns
img755be8b2662e3e96919164f39e276e9a.png
Distribution of the time difference between arrival times of the endpoint pulse. Endpoint 1 delay: 576 ns

Clock properties

Clock edge alignment

The alignment between the clock edges coming from the endpoints was studied. The difference between the clock edges was found to have a mean of 2.6 ns, with a std. dev. of 76.2 ps. A screenshot of the oscilloscope output, along with a plot of the clock edge time difference can be found below.

img565156a6974692a67f517745231fc337.jpg
Screenshot of the LeCroy oscilloscope output measuring the time between the two endpoints' clock edges. Endpoint 1 delay: 576 ns
imgc373e38ead2db265f12b4eb367388464.png
Distribution of the time difference between arrival times of the endpoints' clock edges. Endpoint 1 delay: 576 ns

Clock jitter

The cycle-to-cycle jitter of the clock coming from an endpoint was found to be 9.6 ps. A screenshot of the oscilloscope output, as well as a plot of the cycle-to-cycle period variation can be found below.

img09d8e96507df0c720336371377323615.jpg
Screenshot of the LeCroy oscilloscope output measuring the cycle-to-cycle jitter of the endpoint clock. Endpoint 1 delay: 576 ns
img3f77458827d6f04f5d9eeefdbb00fa5c.png
Distribution of the cycle-to-cycle jitter of the endpoint clock. Endpoint 1 delay: 576 ns

-- StoyanMiroslavovTrilov - 2020-08-05

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng dt_cdelay_28.png r1 manage 544.8 K 2020-08-05 - 18:34 StoyanMiroslavovTrilov  
JPEGjpg dt_cdelay_28_fdelay_4.jpg r1 manage 184.6 K 2020-08-05 - 18:35 StoyanMiroslavovTrilov  
PNGpng dt_clock_edge.png r1 manage 45.2 K 2020-08-06 - 10:59 StoyanMiroslavovTrilov  
JPEGjpg dt_clock_edges_oscilloscope.jpg r1 manage 168.3 K 2020-08-06 - 10:59 StoyanMiroslavovTrilov  
PNGpng dt_clock_period.png r1 manage 48.6 K 2020-08-06 - 10:59 StoyanMiroslavovTrilov  
PNGpng dt_no_delays.png r2 r1 manage 604.9 K 2020-08-05 - 18:21 StoyanMiroslavovTrilov  
PNGpng dt_sync_pulses.png r1 manage 53.8 K 2020-08-05 - 18:36 StoyanMiroslavovTrilov  
PDFpdf endpoint_alignment_test_setup.pdf r1 manage 210.2 K 2020-08-05 - 16:33 StoyanMiroslavovTrilov  
PNGpng endpoint_alignment_test_setup.png r1 manage 290.4 K 2020-08-05 - 17:31 StoyanMiroslavovTrilov  
JPEGjpg ept_clocks_oscilloscope.jpg r1 manage 186.3 K 2020-08-06 - 11:00 StoyanMiroslavovTrilov  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2020-08-11 - StoyanMiroslavovTrilov
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CENF All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback