Verification of v5.4.0 firmware and software release

Details

Date	19/05/2021
Location	University of Bristol lab
People	Sudan

Test objective

Verify the operation of the software and firmware following the fixing of several bugfixes, and the addition of a new output port in endpoint firmware block.

The bugs fixed are documented below:

Test setup

Hardware

An AIDA 2020 TLU was configured to acts as master using the overlord fw design. An Enclustra AX3+PM3 with a timing FMC (pc053a) acted as an endpoint. The output from the TLU was fed into an 1:8 passive optical splitter via single mode optical fibre. The timing FMC was connected to one of the splitter outputs using a single mode optical fibre.

TLU UID: 0x5410ecbba408

FMC UID: 0x49162b675e3

Firmware

The bitfile used for the TLU was: https://pdts-fw.web.cern.ch/pdts-fw/mr/28/latest/overlord_tlu_mr28_sha-809ffc54_runner-slu9p8x4-project-19909-concurrent-17_210506_2040.tgz

The endpoint bitfile was https://pdts-fw.web.cern.ch/pdts-fw/mr/28/latest/endpoint_pc053d_fmc_mr28_sha-809ffc54_runner-6bb6b053-project-19909-concurrent-1_210506_2205.tgz

Software

The software version used for this test was: test/v5.4.0_1105

Device configuration

The TLU was configured using the following commands:

pdtbutler io PROD_MASTER reset  --force-pll-cfg DUNE_TLU-DUNTLU09-Registers_62.5_mhz.txt
pdtbutler mst PROD_MASTER synctime
pdtbutler mst PROD_MASTER part 0 configure

The endpoint was configured using the commands below:

pdtbutler io EPT_0 reset --force-pll-cfg Si5394-053endptr_62-5MHz_4kHz-Registers.txt
pdtbutler ept EPT_0 0 enable 
pdtbutler ept EPT_0 0 status

Test results

Following the configuration of the TLU and timing partition 0, the partition status command,

pdtbutler mst PROD_MASTER part 0 status

yielded the following output:

Created device PROD_MASTER
-----------Hardware info-----------
+----------------+----------------+
|   Board type   |       tlu      |
| Board revision |    kTLURev1    |
|    Board UID   | 0x5410ecbba408 |
|  Carrier type  |  enclustra-a35 |
|   Design type  |    overlord    |
+----------------+----------------+

Master FW rev: 0x50400, partitions: 4, channels: 5

-- Master state---

Timestamp: 0x120064b134692f4 -> Wed, 19 May 2021 15:24:16 +0000

--------------Cmd gen counters--------------
+------+-----------------+-----------------+
|      | Accept counters | Reject counters |
+------+-----------------+-----------------+
| Chan |  cnts  |   hex  |  cnts  |   hex  |
+------+--------+--------+--------+--------+
|  0x0 |    0   |    0   |    0   |    0   |
|  0x1 |    0   |    0   |    0   |    0   |
|  0x2 |    0   |    0   |    0   |    0   |
|  0x3 |    0   |    0   |    0   |    0   |
|  0x4 |    0   |    0   |    0   |    0   |
+------+--------+--------+--------+--------+


=> Partition 0

---------Controls--------
+---------------+-------+
|    Register   | Value |
+---------------+-------+
|     buf_en    |   0   |
|   frag_mask   |   0   |
|    part_en    |  0x1  |
|  rate_ctrl_en |  0x1  |
|    run_req    |   0   |
| spill_gate_en |  0x1  |
|  trig_ctr_rst |   0   |
|    trig_en    |   0   |
|   trig_mask   |  0xf1 |
+---------------+-------+

--------State-------
+----------+-------+
| Register | Value |
+----------+-------+
|  buf_err |   0   |
| buf_warn |   0   |
|  in_run  |   0   |
| in_spill |   0   |
|  part_up |  0x1  |
|  run_int |   0   |
+----------+-------+

Event Counter: 0
Buffer status: OK
Buffer occupancy: 0

+-------------+-----------------+-----------------+
|             | Accept counters | Reject counters |
+-------------+-----------------+-----------------+
|     Cmd     |  cnts  |   hex  |  cnts  |   hex  |
+-------------+--------+--------+--------+--------+
|   TimeSync  |   69   |  0x45  |    0   |    0   |
|     Echo    |    0   |    0   |    0   |    0   |
|  SpillStart |    0   |    0   |    0   |    0   |
|  SpillStop  |    0   |    0   |    0   |    0   |
|   RunStart  |    0   |    0   |    0   |    0   |
|   RunStop   |    0   |    0   |    0   |    0   |
|   WibCalib  |    0   |    0   |    0   |    0   |
|   SSPCalib  |    0   |    0   |    0   |    0   |
|  FakeTrig0  |    0   |    0   |    0   |    0   |
|  FakeTrig1  |    0   |    0   |    0   |    0   |
|  FakeTrig2  |    0   |    0   |    0   |    0   |
|  FakeTrig3  |    0   |    0   |    0   |    0   |
|   BeamTrig  |    0   |    0   |    0   |    0   |
|  NoBeamTrig |    0   |    0   |    0   |    0   |
| ExtFakeTrig |    0   |    0   |    0   |    0   |
+-------------+--------+--------+--------+--------+

The endpoint status command

pdtbutler ept EPT_0 0 status

gives the output below.

Created endpoint device EPT_0
-----------Hardware info----------
+----------------+---------------+
|   Board type   |      fmc      |
| Board revision |    kFMCRev3   |
|    Board UID   | 0x49162b675e3 |
|  Carrier type  | enclustra-a35 |
|   Design type  |    endpoint   |
+----------------+---------------+

+---------+
|    0    |
| 0x50100 |
+---------+
+------------------+--------------------------------------------+
|     Endpoint     |                     0                      |
+------------------+--------------------------------------------+
| State            | Waiting for phase adjustment command (0x6) |
| Partition        | 0                                          |
| Address          | 0                                          |
| Timestamp        | Wed, 19 May 2021 15:25:29 +0000            |
| Timestamp (hex)  | 0x120064bef16618d                          |
| EventCounter     | 0                                          |
| Buffer status    | OK                                         |
| Buffer occupancy | 0                                          |
+------------------+--------------------------------------------+

--- Endpoint state ---
+------------+----------+
|  Endpoint  |    0     |
+------------+----------+
| buf_err    |   0x0    |
| buf_warn   |   0x0    |
| ep_rdy     |   0x0    |
| ep_rsto    |   0x1    |
| ep_stat    |   0x6    |
| in_run     |   0x0    |
| in_spill   |   0x0    |
| sfp_tx_dis |   0x1    |
+------------+----------+

--- Command counters ---
+--------------+----------+
|   Endpoint   |    0     |
+--------------+----------+
| TimeSync     |    85    |
| Echo         |          |
| SpillStart   |          |
| SpillStop    |          |
| RunStart     |          |
| RunStop      |          |
| WibCalib     |          |
| SSPCalib     |          |
| FakeTrig0    |          |
| FakeTrig1    |          |
| FakeTrig2    |          |
| FakeTrig3    |          |
| BeamTrig     |          |
| NoBeamTrig   |          |
| ExtFakeTrig  |          |
| None         |          |
+--------------+----------+

Delay adjustment

Sending the delay adjustment command,

pdtbutler mst PROD_MASTER align apply-delay 0x1 0x3 0x7 --force

applies a coarse delay of 0x3, and a fine delay of 0x7 to endpoint address 0x1. The endpoint status after delay adjustment is: 

(dbt-pyvenv) -bash-4.2$ pdtbutler ept EPT_0 0 status
Created endpoint device EPT_0
-----------Hardware info----------
+----------------+---------------+
|   Board type   |      fmc      |
| Board revision |    kFMCRev3   |
|    Board UID   | 0x49162b675e3 |
|  Carrier type  | enclustra-a35 |
|   Design type  |    endpoint   |
+----------------+---------------+

+---------+
|    0    |
| 0x50100 |
+---------+
+------------------+---------------------------------+
|     Endpoint     |                0                |
+------------------+---------------------------------+
| State            | Ready (0x8)                     |
| Partition        | 0                               |
| Address          | 1                               |
| Timestamp        | Wed, 19 May 2021 16:55:42 +0000 |
| Timestamp (hex)  | 0x120068af2aad218               |
| EventCounter     | 0                               |
| Buffer status    | OK                              |
| Buffer occupancy | 0                               |
+------------------+---------------------------------+

--- Endpoint state ---
+------------+----------+
|  Endpoint  |    0     |
+------------+----------+
| buf_err    |   0x0    |
| buf_warn   |   0x0    |
| cdelay     |   0x3    |
| ep_rdy     |   0x1    |
| ep_rsto    |   0x0    |
| ep_stat    |   0x8    |
| fdelay     |   0x7    |
| in_run     |   0x0    |
| in_spill   |   0x0    |
| sfp_tx_dis |   0x0    |
+------------+----------+

--- Command counters ---
+--------------+----------+
|   Endpoint   |    0     |
+--------------+----------+
| TimeSync     |   843    |
| Echo         |    5     |
| SpillStart   |          |
| SpillStop    |          |
| RunStart     |          |
| RunStop      |          |
| WibCalib     |          |
| SSPCalib     |          |
| FakeTrig0    |          |
| FakeTrig1    |          |
| FakeTrig2    |          |
| FakeTrig3    |          |
| BeamTrig     |          |
| NoBeamTrig   |          |
| ExtFakeTrig  |          |
| None         |          |
+--------------+----------+

indicating the successful application of the coarse and fine delays. The following combinations of delay values were also verified.

Buffer reset error:

The functioning of the event buffer was verified by configuring a fake triggers generator, and enabling triggering for partition 0. The commands issued were:

pdtbutler mst PROD_MASTER faketrig-conf 0 1
pdtbutler mst PROD_MASTER part 0 configure --no-spill-gate
pdtbutler mst PROD_MASTER part 0 start
pdtbutler mst PROD_MASTER part 0 trig

The presence of events in the buffer was verified by the status command (output below).

Created device PROD_MASTER
-----------Hardware info-----------
+----------------+----------------+
|   Board type   |       tlu      |
| Board revision |    kTLURev1    |
|    Board UID   | 0x5410ecbba408 |
|  Carrier type  |  enclustra-a35 |
|   Design type  |    overlord    |
+----------------+----------------+

Master FW rev: 0x50400, partitions: 4, channels: 5

-- Master state---

Timestamp: 0x120065480c0a84c -> Wed, 19 May 2021 15:37:45 +0000

--------------Cmd gen counters--------------
+------+-----------------+-----------------+
|      | Accept counters | Reject counters |
+------+-----------------+-----------------+
| Chan |  cnts  |   hex  |  cnts  |   hex  |
+------+--------+--------+--------+--------+
|  0x0 |   24   |  0x18  |    0   |    0   |
|  0x1 |    0   |    0   |    0   |    0   |
|  0x2 |    0   |    0   |    0   |    0   |
|  0x3 |    0   |    0   |    0   |    0   |
|  0x4 |    0   |    0   |    0   |    0   |
+------+--------+--------+--------+--------+


=> Partition 0

---------Controls--------
+---------------+-------+
|    Register   | Value |
+---------------+-------+
|     buf_en    |  0x1  |
|   frag_mask   |   0   |
|    part_en    |  0x1  |
|  rate_ctrl_en |  0x1  |
|    run_req    |  0x1  |
| spill_gate_en |   0   |
|  trig_ctr_rst |   0   |
|    trig_en    |  0x1  |
|   trig_mask   |  0xf1 |
+---------------+-------+

--------State-------
+----------+-------+
| Register | Value |
+----------+-------+
|  buf_err |   0   |
| buf_warn |   0   |
|  in_run  |  0x1  |
| in_spill |   0   |
|  part_up |  0x1  |
|  run_int |  0x1  |
+----------+-------+

Event Counter: 8
Buffer status: OK
Buffer occupancy: 48

+-------------+-----------------+-----------------+
|             | Accept counters | Reject counters |
+-------------+-----------------+-----------------+
|     Cmd     |  cnts  |   hex  |  cnts  |   hex  |
+-------------+--------+--------+--------+--------+
|   TimeSync  |   14   |   0xe  |    0   |    0   |
|     Echo    |    0   |    0   |    0   |    0   |
|  SpillStart |    0   |    0   |    0   |    0   |
|  SpillStop  |    0   |    0   |    0   |    0   |
|   RunStart  |    1   |   0x1  |    0   |    0   |
|   RunStop   |    0   |    0   |    0   |    0   |
|   WibCalib  |    0   |    0   |    0   |    0   |
|   SSPCalib  |    0   |    0   |    0   |    0   |
|  FakeTrig0  |    8   |   0x8  |    4   |   0x4  |
|  FakeTrig1  |    0   |    0   |    0   |    0   |
|  FakeTrig2  |    0   |    0   |    0   |    0   |
|  FakeTrig3  |    0   |    0   |    0   |    0   |
|   BeamTrig  |    0   |    0   |    0   |    0   |
|  NoBeamTrig |    0   |    0   |    0   |    0   |
| ExtFakeTrig |    0   |    0   |    0   |    0   |
+-------------+--------+--------+--------+--------+

Checking application of delays

This is currently being debugged.

pdtbutler mst PROD_MASTER align apply-delay 0x1 0x3 0x7 --force

pdtbutler mst PROD_MASTER align toggle-tx 0 --on

(dbt-pyvenv) -bash-4.2$ pdtbutler mst PROD_MASTER align measure-delay 0
Created device PROD_MASTER
-----------Hardware info-----------
+----------------+----------------+
|   Board type   |       tlu      |
| Board revision |    kTLURev1    |
|    Board UID   | 0x5410ecbba408 |
|  Carrier type  |  enclustra-a35 |
|   Design type  |    overlord    |
+----------------+----------------+

Master FW rev: 0x50400, partitions: 4, channels: 5
2021-May-19 17:09:21,766 LOG [dunedaq::timing::GlobalNode::enable_upstream_endpoint(...) at /users/sp13109/v5.4.0_1105/testtiming/sourcecode/timing/src/GlobalNode.cpp:82] Upstream endpoint reset, waiting for lock
2021-May-19 17:09:21,877 LOG [dunedaq::timing::GlobalNode::enable_upstream_endpoint(...) at /users/sp13109/v5.4.0_1105/testtiming/sourcecode/timing/src/GlobalNode.cpp:104] Endpoint locked: state= 0xe
ERROR ('RuntimeError' exception): ' Timeout whilst waiting for echo. Timeout (ms): 500'

-- SudarshanParamesvaran - 2021-05-19