PD-I timing system endpoint alignment

Details

Date	05/08/2020
Location	University of Bristol lab
People	Stoyan Trilov, David Cussans

Test objective

The alignment of local timestamps maintained by timing endpoints relies on each endpoint having an individualised delay applied to its incoming datastream in order to compensate for differences in fibre lengths and other factors. The endpoint delay can be applied in two different granularities, coarse and fine. Coarse adjustment is done in units of 20 ns, corresponding to the period of the 50 MHz used to maintain the endpoint timestamp. Fine adjustments are done in units of 4 ns, corresponding to the 250 MHz IO clock used to transmit the serial datastream. This test aims to demonstrate the successful application of endpoint delays in coarse and fine units. As well as studying the alignment of the timing messages between the two endpoints, the test also looked at the characteristics of the clocks recovered by the endpoints.

Test setup

The test was performed using an AIDA 2020 TLU, acting as the timing master, running the overlord firmware design. Two endpoints were used to test the application of delays. Both endpoints consist of a PD-I timing FMC, hosted by an Enclustra FPGA baseboard (PM3) with an AX3 FPGA module. The endpoints were running a modified version of the endpoint firmware design, which can be found at https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-firmware/-/commit/ffcbac4b9c3f7f4c29ac7f515c3b8ad385100d1a. The endpoint firmware was modified so that it emits an electrical "sync" pulse upon the reception of an "Echo" command. The signal path for the electrical pulse is as follows:

Endpoint signal -> FMC IO signal -> FPGA pin -> SO DIMM connector pin -> FMC connector pin -> FMC output
rx_pulse (endpoint_wrapper) -> gpout_0 -> L6/L5 -> 90/92 -> G9/G10 (FMC_LA13_P/FMC_LA13_N)  -> PX5 

The clock output from the timing FMC clock generator can be found on the PX4 connector.

The TLU transmits the timing datastream to the endpoints over single mode fibres, and a passive optical splitter. One of the endpoints, which will be referred to as "endpoint 1" throughout is connected using a 2 m fibre, whereas the other endpoint uses a 120 m fibre and will be referred to as "endpoint 2". A diagram of the experimental setup is shown below.

To compensate for the different fibre lengths, endpoint 2 had no delay applied to it, whereas endpoint 1 had several different delay values applied. The delay value used for endpoint 1 are listed in the table below.

Coarse units	Fine units	Total delay [ns]
0	0	0
28	0	560
28	4	576

The time between the two pulses was measured using two different oscilloscopes PicoScope 6403C and LeCroy WR640. The PicoScope was used for single measurements of the pulse interval, whereas the LeCroy was used to collect a higher statistics sample.

The timing software used for the test can be found here: https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-software/-/commits/feature/fine_delay_adjustment. Installation instructions for the timing software can be found here: https://twiki.cern.ch/twiki/bin/view/CENF/TimingSystemAdvancedOp#Installation_and_build_instructi, where the git clone command should be changed to the one below.

git clone ssh://git@gitlab.cern.ch:7999/protoDUNE-SP-DAQ/timing-board-software.git -b feature/fine_delay_adjustment

In order to interact with the two endpoints, the following two entries will need to be added to the timing software connections.xml file.

 <connection id="EPT_0"             uri="ipbusudp-2.0://192.168.200.16:50001" address_table="file://${PDT_TESTS}/etc/addrtab/v5a2/endpoint_fmc/top.xml" />
 <connection id="EPT_1"             uri="ipbusudp-2.0://192.168.200.17:50001" address_table="file://${PDT_TESTS}/etc/addrtab/v5a2/endpoint_fmc/top.xml" />

N.B. The firmware as is from the git repository will build with an IP address of 192.168.200.16. In order to obtain a bitfile with the IP address 192.168.200.17, line 100 in the file projects/master/firmware/hdl/top_enclustra_ax3_pm3.vhd needs to be changed to:

addr <= X"1";

Test method

Timing master configuration

To configure the TLU, execute the following pdtbutler timing commands.

pdtbutler io PROD_MASTER reset
pdtbutler mst PROD_MASTER synctime
pdtbutler mst PROD_MASTER part 0 configure
pdtbutler mst PROD_MASTER part 0 status

The last command should yield something similar to the output below.

Created device PROD_MASTER
ID: design 'overlord' on board 'tlu' on carrier 'enclustra-a35'
Master FW rev: 0x50100, partitions: 4, channels: 5

-- Master state---

=> Cmd generator counters
----------------------------------------------------------------------------
|              |       Accept counters       |       Reject counters       |
----------------------------------------------------------------------------
|     Chan     |     cnts     |     hex      |     cnts     |     hex      |
----------------------------------------------------------------------------
|     0x0      |     9154     |    0x23c2    |      0       |     0x0      |
|     0x1      |      0       |     0x0      |      0       |     0x0      |
|     0x2      |      0       |     0x0      |      0       |     0x0      |
|     0x3      |      0       |     0x0      |      0       |     0x0      |
|     0x4      |      0       |     0x0      |      0       |     0x0      |
----------------------------------------------------------------------------

=> Partition 0
Control                   Status registers  
+---------------+------+  +----------+-----+
| buf_en        | 0x0  |  | buf_err  | 0x0 |
| frag_mask     | 0x0  |  | buf_warn | 0x0 |
| part_en       | 0x1  |  | in_run   | 0x0 |
| rate_ctrl_en  | 0x1  |  | in_spill | 0x0 |
| run_req       | 0x0  |  | part_up  | 0x1 |
| spill_gate_en | 0x1  |  | run_int  | 0x0 |
| trig_ctr_rst  | 0x0  |  +----------+-----+
| trig_en       | 0x0  |                    
| trig_mask     | 0xf1 |                    
+---------------+------+                    

Timestamp: 0x11b9eb8da5bcd74 -> Wed, 05 Aug 2020 15:45:03 +0000
EventCounter: 0
Buffer status: OK
Buffer occupancy: 0

----------------------------------------------------------------------------
|              |       Accept counters       |       Reject counters       |
----------------------------------------------------------------------------
|     Cmd      |     cnts     |     hex      |     cnts     |     hex      |
----------------------------------------------------------------------------
|   TimeSync   |    627255    |   0x99237    |      0       |     0x0      |
|     Echo     |     9217     |    0x2401    |      0       |     0x0      |
|  SpillStart  |      0       |     0x0      |      0       |     0x0      |
|  SpillStop   |      0       |     0x0      |      0       |     0x0      |
|   RunStart   |      0       |     0x0      |      0       |     0x0      |
|   RunStop    |      0       |     0x0      |      0       |     0x0      |
|   WibCalib   |      0       |     0x0      |      0       |     0x0      |
|   SSPCalib   |      0       |     0x0      |      0       |     0x0      |
|  FakeTrig0   |      0       |     0x0      |      0       |     0x0      |
|  FakeTrig1   |      0       |     0x0      |      0       |     0x0      |
|  FakeTrig2   |      0       |     0x0      |      0       |     0x0      |
|  FakeTrig3   |      0       |     0x0      |      0       |     0x0      |
|   BeamTrig   |      0       |     0x0      |      0       |     0x0      |
|  NoBeamTrig  |      0       |     0x0      |      0       |     0x0      |
| ExtFakeTrig  |      0       |     0x0      |      0       |     0x0      |
|     0xf      |      0       |     0x0      |      0       |     0x0      |
----------------------------------------------------------------------------

Endpoint configuration

To configure the endpoints, execute the following sequence of commands, where corresponds to the endpoint ID (as configured in connections.xml), and <adr> is the address assigned to the endpoint. For the test described here, the values used are in the table below.

Endpoint ID	Endpoint address	Endpoint IP address
0	1	192.168.200.16
1	2	192.168.200.17

pdtbutler io EPT_<id> reset --force-pll-cfg tests/etc/clock/devel/Si5344-PDTSCRT1NoZdm-RevD-400HzBW-Registers.txt
pdtbutler ept EPT_<id> 0 enable -a <adr>
pdtbutler ept EPT_<id> 0 status

The last command should yield something similar to the output below.

Created endpoint device EPT_0
+---------+
|    0    |
| 0x50100 |
+---------+
+------------------+--------------------------------------------+
|     Endpoint     |                     0                      |
+------------------+--------------------------------------------+
| State            | Waiting for phase adjustment command (0x6) |
| Partition        | 0                                          |
| Address          | 1                                          |
| Timestamp        | Wed, 05 Aug 2020 16:04:15 +0000            |
| Timestamp (hex)  | 0x11b9ec64384e0b6                          |
| EventCounter     | 0                                          |
| Buffer status    | Error                                      |
| Buffer occupancy | 0                                          |
+------------------+--------------------------------------------+

--- Endpoint state ---
+------------+----------+
|  Endpoint  |    0     |
+------------+----------+
| buf_err    |   0x1    |
| buf_warn   |   0x0    |
| ep_rdy     |   0x0    |
| ep_rsto    |   0x1    |
| ep_stat    |   0x6    |
| in_run     |   0x0    |
| in_spill   |   0x0    |
| sfp_tx_dis |   0x1    |
+------------+----------+

--- Command counters ---
+--------------+----------+
|   Endpoint   |    0     |
+--------------+----------+
| TimeSync     |   428    |
| Echo         |          |
| SpillStart   |          |
| SpillStop    |          |
| RunStart     |          |
| RunStop      |          |
| WibCalib     |          |
| SSPCalib     |          |
| FakeTrig0    |          |
| FakeTrig1    |          |
| FakeTrig2    |          |
| FakeTrig3    |          |
| BeamTrig     |          |
| NoBeamTrig   |          |
| ExtFakeTrig  |          |
| None         |          |
+--------------+----------+

Delay adjustment

Endpoint delays are applied via a timing command issued from the timing master, addressed to a particular endpoint. To apply a delay composed from <cdelay> coarse units and <cfdelay> fine units to an endpoint with address <adr>, issue the following command.

pdtbutler mst PROD_MASTER align apply-delay <adr> <cdelay> <fdelay> --force

After the configuration of the delay, the endpoint should transition to state 0x8 (Ready).

Echo command trigger

To send an "Echo" command to the timing endpoints, and hence trigger the pulses from the endpoints, issue the following command.

pdtbutler mst PROD_MASTER send-cmd Echo 0

If the command is issued successfully, an output similar to the one below should be received.

Created device PROD_MASTER
ID: design 'overlord' on board 'tlu' on carrier 'enclustra-a35'
Master FW rev: 0x50100, partitions: 4, channels: 5
Command sent Echo(1) from generator 0 @time 0x11b9ef6167cd29f 79832198185538207

Test results

Command synchronisation

0 ns delay

The time between the two pulses without delay adjustment for endpoint 1 was measured to be around 575.2 ns. This is compatible with the additional 118 m of fibre, as 1 m of fibre introduces approximately 4.9 ns of delay. A screenshot of the oscilloscope output is shown below.

Screenshot of the PicoScope oscilloscope output measuring the time between the two endpoint pulses. Endpoint 1 delay: 0 ns

560 ns delay

The time between the two pulses, with a delay of 560 ns applied to endpoint 1 was measured to be around 15.7 ns, as it would be expected. A screenshot of the oscilloscope output is shown below.

Screenshot of the PicoScope oscilloscope output measuring the time between the two endpoint pulses. Endpoint 1 delay: 560 ns

576 ns delay

The time between the two pulses, with a delay of 576 ns applied to endpoint 1 was measured to be 267.2 ps, with a spread of 157.0 ps. A screenshot of the oscilloscope output and a plot of the pulse time differences are shown below.

Screenshot of the LeCroy oscilloscope output measuring the time between the two endpoint pulses. Endpoint 1 delay: 576 ns

Distribution of the time difference between arrival times of the endpoint pulse. Endpoint 1 delay: 576 ns

Clock properties

Clock edge alignment

The alignment between the clock edges coming from the endpoints was studied. The difference between the clock edges was found to have a mean of 2.6 ns, with a std. dev. of 76.2 ps. A screenshot of the oscilloscope output, along with a plot of the clock edge time difference can be found below.

Screenshot of the LeCroy oscilloscope output measuring the time between the two endpoints' clock edges. Endpoint 1 delay: 576 ns

Distribution of the time difference between arrival times of the endpoints' clock edges. Endpoint 1 delay: 576 ns

Clock jitter

The cycle-to-cycle jitter of the clock coming from an endpoint was found to be 9.6 ps. A screenshot of the oscilloscope output, as well as a plot of the cycle-to-cycle period variation can be found below.

Screenshot of the LeCroy oscilloscope output measuring the cycle-to-cycle jitter of the endpoint clock. Endpoint 1 delay: 576 ns

Distribution of the cycle-to-cycle jitter of the endpoint clock. Endpoint 1 delay: 576 ns

-- StoyanMiroslavovTrilov - 2020-08-05