Timing system prototype for ProtoDUNE SP II

Design overview

Section giving a brief overview of the DUNE/!ProtoDUNE SP || system design Link to master design document

Prototype set-up

Section describing how to set up a timing system prototype

mTCA system

A functioning mTCA system is composed of several mandatory and optional components. These are listed below along with the models chosen for prototyping.

As per the timing system design described above, a custom mTCA Interface Board will sit in one of the MCH slots and distribute clock and data to the 12 AMCs using the mTCA backplane. The AMCs will then fan out these signals to the timing endpoints using optical fibres.

To set up the crate, plug in the MCH module in the secondary MCH slot, which is the most right-hand slot on the front of the crate. The power supplies can either be plugged in the two front slots above the MCH slots (PM1,PM2) or in two of the slots on the back (PM1A,PM2A). The JSM plugs in the back of the crate in the JSM slot. The location the aforementioned slots is shown in the following two images. For more information refer to the crate manual, https://schroff.nvent.com/wcsstore/ExtendedSitesCatalogAssetStore/Attachment/SchroffAttachments/Documents/63972-338.pdf.

mtca_crate_front_view.png mtca_crate_rear_view.png

Once all of the modules have been inserted, with their hot-swap handles fully in, connect the mains cables to the power supplies and flip the on/off switch to on (if not already in the on position). The LEDs of the power supplies and MCH should start blinking and the fans of the crate should start spinning. After a few seconds, several power supply and MCH LEDs should be solid green (see image below), indicating the MCH and power supplies have booted successfully. Any blue LEDs near the hot-swap handle either solidly on or flashing indicate a problem and therefore that particular module will not be receiving payload power.


mTCA Carrier Hub (MCH)

For the initial configuration of the MCH, one can use telnet over USB (a microUSB port is available on the front of the MCH) or network access. The parameters for the telnet session are listed below.
 Baud rate: 19200 Par/Bits: SN1
If the connection has been established successfully, the following prompt should appear:
To verify that the MCH is able to detect the power supples and any connected AMCs, type the command:
The output should be similar to the one below.
FRU Information:
 FRU  Device   State  Name
  0   MCH       M4    NAT-MCH-CM
  4   mcmc2     M4    NAT-MCH-MCMC
 13   AMC9      M4    AFC LOADE2
 40   CU1       M4    Schroff uTCA CU
 41   CU2       M4    Schroff uTCA CU
 50   PM1       M4    NAT-PM-AC600 
 51   PM2       M4    NAT-PM-AC600 
The M4 state (hot-swap blue LED off) indicates that the FRU is receiving payload power.

Network configuration
Configure the network parameters as follows:

  1. Enter ip at the command line to show the current configuration
  2. Delete the current IP address using the backspace key. Enter the new address in dotted quad format, e.g.
  3. Change all other parameters by repeating step 2
  4. Reboot the MCH using the command reboot.

  • If an IP address is configured to, it will be ignored by the MCH
  • If you configure a non-zero gateway IP address the related routing configuration will be performed automatically when the MCH starts up

Current network configuration can be also displayed using the command

Bristol lab network parameters
IP Configuration Setup:
IP Address           :
IP Net Mask          :
IP Broadcast Addr    :
IP Gateway Addr      :
The MCH is accessible via bayban.phy.bris.ac.uk or nova.phy.bris.ac.uk.

Web interface access
Once the MCH has been correctly set up on the network, the web interface can be accessed by pointing a web browser to the IP address of the MCH. The default username and password are "root" and "nat" respectively.

Bristol lab web access
To access the web interface from the general Bristol network, the web traffic needs to be routed via bayban or nova. To set up the port forwarding, set up an ssh connection as follows:
ssh -L 8080: bayban.phy.bris.ac.uk
The web interface should then be available on

Or from bayban, simply navigate to using your web browser.

SSH access
SSH access is disabled by default. It can enabled by opening the web interface, and navigating to the "Base Configuration" page, and changing the "SSH access" configuration to "enabled". N.B. SSH access may not available immediately, as ssh keys need to be generated.

JTAG Switch Module (JSM)

If the JSM is installed successfully, its configuration should be available from the "JSM" page of the web interface. Through the JSM, one is able to access the backplane JTAG lines going to each AMC, and switch between them. The active AMC can be selected either through the rotary switch on the front-panel or though the JSM MCH web interface.

Rotary switch
A photo of the JSM front panel, with the rotary switch clearly visible, is shown below. It has 12 positions, labeled as 1-12 in hex, representing the 12 AMC slots.


MCH JSM interface
As mentioned above, the JSM web interface is available on the JSM page of the MCH web interface. There, the active AMC settings can be overridden selecting the desired AMC slot, and clicking the "Override" button. If the operation was successful, the override status should change to "Override enabled".

The JSM interface also allows the transmission of svf files, however before doing so, the svf files need to converted to NAT compliant svf files. This is achieved by downloading the tool from the JSM interface and executing as follows.

./svf_to_nsvf <svf source file>
N.B. This step is mandatory.

The newly created .nsvf file can then be sent over JTAG by following the steps outlined in the JSM web interface.

Power supplies

AMC FMC Carrier (AFC)

JTAG configuration
The AFC features a JTAG scan bridge, SCANSTA111, allowing JTAG access to the on-board FPGA as well as to the two FMC connectors. In order to be able to program the on-board FPGA using the Vivado tools, the SCANSTA111 needs to be put into transparent mode. This is achieved by transmitting the following svf script to the SCANSTA111 chip.
SIR 8 TDI (00); ! 00000000 Address ScanBridge
SIR 8 TDI (A0); ! 10100000 Load instruction to enable transparent mode for LSP0
SIR 8 TDI (a5); ! 10100101 Verify SIR
SDR 8 TDI (5a); ! 01011010 Verify SDR
SIR 8 TDI (C3); ! 11000011 Try to load GOTOWAIT in ScanBridge
SDR 8 TDI (5a); ! 01011010 Verify that ScanBridge did not recognize GOTOWAIT
! Now TDIB  lsp0  TDOB
N.B. The script above enables transparent mode for LSP0, which is where the AFC on-board FPGA is connected. More information can be found here: https://www.ti.com/lit/an/snla068c/snla068c.pdf.

Loading firmware
Connect to the NAT MCH JSM web interface, (from bayban), see Web interface access. Check that the Xilinx Virtual Cable is enabled and configured as below. Note down the port number of the AMC slot which contains the AFC to be programmed.

JSM Xilinx Virtual Server

On bayban, open the Hardware Manager, and then click on

Open target -> Open New Target... -> Next... -> (Connect to: Local server) Next... -> Add Xilinx Virtual Cable (XVC)
Enter the MCH IP address,, along with the port of your desired AMC slot, and click OK. You should now see the FPGA device on the AFC, xc7a200t_0, under the newly created hardware target. If you do not see the device, it is possible that the JTAG bridge on the AFC is not in transparent mode (see section below on how to put it in transparent mode). Click Next, and then Finish. Program the FPGA.

Putting AFC JTAG bridge into transparent mode
Connect to the NAT MCH JSM web interface, (from bayban), see Web interface access. Check that there are no connections to any of the XVC ports, (see screenshot below for an example of an active connection).

JSM Xilinx Virtual Server

If there are, you will not be able to program the AFC JTAG bridge using the Web interface. The XVC connection will have to terminated before the web interface can be used for JTAG programming. The connection will most likely be coming from bayban, check if there are any instances of the hw_server process running on the machine. If so, it's likely that it's one of those making the connection. Ask the process owner to kill the process, and check that there are no active XVC connections (again via the JSM web interface).

Select the AMC slot you want to communicate with under "Step 3" of the section "Program FPGA with SVF File". Under "Step 5", select the .nsvf file to put the bridge in transparent mode. The file is attached to this page for convenience, https://twiki.cern.ch/twiki/pub/CENF/PDuneSPIITimingPrototype/scansta111_transparent_config.nsvf. Click "Upload".

N.B. Depending on your connection method, the web interface sometimes freezes after "Upload" has been clicked. In such a case, the MCH continues to function OK, and the file is transferred successfully. Rebooting the MCH resolves this web interface lock-up.

Timing system tests

ProtoDUNE SP I timing system tests

Tests involving ProtoDUNE SP I timing system hardware, i.e. AIDA 2020 TLU, timing FMC + Enclustra PM3/AX3

ProtoDUNE SP II timing system tests

Tests with ProtoDUNE SP II/DUNE-like timing system, i.e. mTCA crate, GIB, MIB, FIB

Release tests

* scansta111_transparent_config.nsvf: scansta111_transparent_config.nsvf

Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg jsm.JPG r1 manage 687.1 K 2020-02-19 - 15:31 StoyanMiroslavovTrilov  
PNGpng jsm_xvc.png r1 manage 607.0 K 2021-08-20 - 13:53 StoyanMiroslavovTrilov  
PNGpng jsm_xvc_active_connection.png r1 manage 580.6 K 2021-08-20 - 13:53 StoyanMiroslavovTrilov  
JPEGjpg mch+psu.JPG r2 r1 manage 1685.8 K 2020-02-19 - 15:44 StoyanMiroslavovTrilov  
PNGpng mtca_crate_front_view.png r1 manage 826.8 K 2020-02-18 - 16:42 StoyanMiroslavovTrilov  
PNGpng mtca_crate_rear_view.png r1 manage 531.1 K 2020-02-18 - 16:42 StoyanMiroslavovTrilov  
Unknown file formatnsvf scansta111_transparent_config.nsvf r1 manage 0.3 K 2021-08-20 - 13:57 StoyanMiroslavovTrilov  
Edit | Attach | Watch | Print version | History: r22 < r21 < r20 < r19 < r18 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r22 - 2022-03-18 - DavidCussans
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CENF All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback