Timing system prototype for ProtoDUNE SP II

Design overview

Section giving a brief overview of the DUNE/!ProtoDUNE SP || system design Link to master design document

Prototype set-up

Section describing how to set up a timing system prototype

mTCA system

A functioning mTCA system is composed of several mandatory and optional components. These are listed below along with the models chosen for prototyping.

• mTCA crate, with a particular backplane topology; Schroff 11890-170; https://schroff.nvent.com/ProductDisplay?urlRequestType=Base&productId=3074457345616811168
• mTCA Carrier Hub (MCH); NAT-MCH; https://www.nateurope.com/products/NAT-MCH.html
• One or more power supplies; NAT-PM-AC600; https://www.nateurope.com/products/NAT-PM-AC600.html
• JTAG Switch Module (JSM) (optional); NAT-JSM; https://www.nateurope.com/products/NAT-JSM.html
• Advanced Mezzanine Card (AMC) (optional); AMC FMC Carrier (AFC); https://www.ohwr.org/project/afc/wikis/home

As per the timing system design described above, a custom mTCA Interface Board will sit in one of the MCH slots and distribute clock and data to the 12 AMCs using the mTCA backplane. The AMCs will then fan out these signals to the timing endpoints using optical fibres.

To set up the crate, plug in the MCH module in the secondary MCH slot, which is the most right-hand slot on the front of the crate. The power supplies can either be plugged in the two front slots above the MCH slots (PM1,PM2) or in two of the slots on the back (PM1A,PM2A). The JSM plugs in the back of the crate in the JSM slot. The location the aforementioned slots is shown in the following two images. For more information refer to the crate manual, https://schroff.nvent.com/wcsstore/ExtendedSitesCatalogAssetStore/Attachment/SchroffAttachments/Documents/63972-338.pdf.

Once all of the modules have been inserted, with their hot-swap handles fully in, connect the mains cables to the power supplies and flip the on/off switch to on (if not already in the on position). The LEDs of the power supplies and MCH should start blinking and the fans of the crate should start spinning. After a few seconds, several power supply and MCH LEDs should be solid green (see image below), indicating the MCH and power supplies have booted successfully. Any blue LEDs near the hot-swap handle either solidly on or flashing indicate a problem and therefore that particular module will not be receiving payload power.

mTCA Carrier Hub (MCH)

For the initial configuration of the MCH, one can use telnet over USB (a microUSB port is available on the front of the MCH) or network access. The parameters for the telnet session are listed below.

 Baud rate: 19200 Par/Bits: SN1

If the connection has been established successfully, the following prompt should appear:

nat>

To verify that the MCH is able to detect the power supples and any connected AMCs, type the command:

show_fru

The output should be similar to the one below.

FRU Information:
----------------
 FRU  Device   State  Name
==========================================
  0   MCH       M4    NAT-MCH-CM
  4   mcmc2     M4    NAT-MCH-MCMC
 13   AMC9      M4    AFC LOADE2
 40   CU1       M4    Schroff uTCA CU
 41   CU2       M4    Schroff uTCA CU
 50   PM1       M4    NAT-PM-AC600 
 51   PM2       M4    NAT-PM-AC600 
==========================================

The M4 state (hot-swap blue LED off) indicates that the FRU is receiving payload power.

Network configuration

Configure the network parameters as follows:

1. Enter ip at the command line to show the current configuration
2. Delete the current IP address using the backspace key. Enter the new address in dotted quad format, e.g. 192.168.1.42
3. Change all other parameters by repeating step 2
4. Reboot the MCH using the command reboot.

• If an IP address is configured to 0.0.0.0, it will be ignored by the MCH
• If you configure a non-zero gateway IP address the related routing configuration will be performed automatically when the MCH starts up

Current network configuration can be also displayed using the command

ifconfig

Bristol lab network parameters

IP Configuration Setup:
-----------------------
IP Address           : 192.168.121.99
IP Net Mask          : 255.255.255.0
IP Broadcast Addr    : 192.168.255.255
IP Gateway Addr      : 0.0.0.0
-----------------------

The MCH is accessible via bayban.phy.bris.ac.uk or nova.phy.bris.ac.uk.

Web interface access

Once the MCH has been correctly set up on the network, the web interface can be accessed by pointing a web browser to the IP address of the MCH. The default username and password are "root" and "nat" respectively.

Bristol lab web access

To access the web interface from the general Bristol network, the web traffic needs to be routed via bayban or nova. To set up the port forwarding, set up an ssh connection as follows:

ssh -L 8080:192.168.121.99:80 bayban.phy.bris.ac.uk

The web interface should then be available on

http://localhost:8080/

Or from bayban, simply navigate to http://192.168.121.99 using your web browser.

SSH access

SSH access is disabled by default. It can enabled by opening the web interface, and navigating to the "Base Configuration" page, and changing the "SSH access" configuration to "enabled". N.B. SSH access may not available immediately, as ssh keys need to be generated.

JTAG Switch Module (JSM)

If the JSM is installed successfully, its configuration should be available from the "JSM" page of the web interface. Through the JSM, one is able to access the backplane JTAG lines going to each AMC, and switch between them. The active AMC can be selected either through the rotary switch on the front-panel or though the JSM MCH web interface.

Rotary switch

A photo of the JSM front panel, with the rotary switch clearly visible, is shown below. It has 12 positions, labeled as 1-12 in hex, representing the 12 AMC slots.

MCH JSM interface

As mentioned above, the JSM web interface is available on the JSM page of the MCH web interface. There, the active AMC settings can be overridden selecting the desired AMC slot, and clicking the "Override" button. If the operation was successful, the override status should change to "Override enabled".

The JSM interface also allows the transmission of svf files, however before doing so, the svf files need to converted to NAT compliant svf files. This is achieved by downloading the tool from the JSM interface and executing as follows.

./svf_to_nsvf <svf source file>

N.B. This step is mandatory.

The newly created .nsvf file can then be sent over JTAG by following the steps outlined in the JSM web interface.

Power supplies

AMC FMC Carrier (AFC)

JTAG configuration

The AFC features a JTAG scan bridge, SCANSTA111, allowing JTAG access to the on-board FPGA as well as to the two FMC connectors. In order to be able to program the on-board FPGA using the Vivado tools, the SCANSTA111 needs to be put into transparent mode. This is achieved by transmitting the following svf script to the SCANSTA111 chip.

SIR 8 TDI (00); ! 00000000 Address ScanBridge
SIR 8 TDI (A0); ! 10100000 Load instruction to enable transparent mode for LSP0
SIR 8 TDI (a5); ! 10100101 Verify SIR
SDR 8 TDI (5a); ! 01011010 Verify SDR
SIR 8 TDI (C3); ! 11000011 Try to load GOTOWAIT in ScanBridge
SDR 8 TDI (5a); ! 01011010 Verify that ScanBridge did not recognize GOTOWAIT
! Now TDIB › lsp0 › TDOB

N.B. The script above enables transparent mode for LSP0, which is where the AFC on-board FPGA is connected. More information can be found here: https://www.ti.com/lit/an/snla068c/snla068c.pdf.

Loading firmware

Connect to the NAT MCH JSM web interface, http://192.168.121.99 (from bayban), see Web interface access. Check that the Xilinx Virtual Cable is enabled and configured as below. Note down the port number of the AMC slot which contains the AFC to be programmed.

On bayban, open the Hardware Manager, and then click on

Open target -> Open New Target... -> Next... -> (Connect to: Local server) Next... -> Add Xilinx Virtual Cable (XVC)

Enter the MCH IP address, 192.168.121.99, along with the port of your desired AMC slot, and click OK. You should now see the FPGA device on the AFC, xc7a200t_0, under the newly created hardware target. If you do not see the device, it is possible that the JTAG bridge on the AFC is not in transparent mode (see section below on how to put it in transparent mode). Click Next, and then Finish. Program the FPGA.

Putting AFC JTAG bridge into transparent mode

Connect to the NAT MCH JSM web interface, http://192.168.121.99 (from bayban), see Web interface access. Check that there are no connections to any of the XVC ports, (see screenshot below for an example of an active connection).

If there are, you will not be able to program the AFC JTAG bridge using the Web interface. The XVC connection will have to terminated before the web interface can be used for JTAG programming. The connection will most likely be coming from bayban, check if there are any instances of the hw_server process running on the machine. If so, it's likely that it's one of those making the connection. Ask the process owner to kill the process, and check that there are no active XVC connections (again via the JSM web interface).

Select the AMC slot you want to communicate with under "Step 3" of the section "Program FPGA with SVF File". Under "Step 5", select the .nsvf file to put the bridge in transparent mode. The file is attached to this page for convenience, https://twiki.cern.ch/twiki/pub/CENF/PDuneSPIITimingPrototype/scansta111_transparent_config.nsvf. Click "Upload".

N.B. Depending on your connection method, the web interface sometimes freezes after "Upload" has been clicked. In such a case, the MCH continues to function OK, and the file is transferred successfully. Rebooting the MCH resolves this web interface lock-up.

Timing system tests

ProtoDUNE SP I timing system tests

Tests involving ProtoDUNE SP I timing system hardware, i.e. AIDA 2020 TLU, timing FMC + Enclustra PM3/AX3

• Verification of the round trip time measurement
• Test of endpoint alignment via delay adjustment

ProtoDUNE SP II timing system tests

Tests with ProtoDUNE SP II/DUNE-like timing system, i.e. mTCA crate, GIB, MIB, FIB

Release tests

• Tests validating the November 2020 software and firmware release
• Tests validating relval/v5.3.0/b1 firmware
• Tests validating relval/v5.3.0/b2 firmware
• Tests validating v5.4.0 firmware, and v5.4.0/v1.1.0 timing/timinglibs software
• Tests validating v5.4.0 firmware and software
• Tests validating round trip measurements
• Tests validating MAC from PROM builds
• Tests validating the v2.8.0 firmware+software
• Tests validating the v2.8.0 firmware+software (continued)
• Tests validating the v2.8.0 firmware+software (continued) (continued)
• Tests validating the v6_0_0 b4 firmware
• Tests validating initial support for Digilent Nexsys Video firmware
• Tests validating the 2_8_1 release
• Tests validating the 2_10_0 release

* scansta111_transparent_config.nsvf: scansta111_transparent_config.nsvf