This documents describe how to get started with RCE generation 3 development. The initial development is focusing on the deployment of an Arch-Linux based test infrastructure for ITk test stands beyond the existing RTEMS based infrastructure. This is part of the RCE readout development infrastructure in ATLAS, while some further general communications and specific test platform documentations can be found at the main RCEDevelopmentLab portal.

Hardware Setup

Hardware installation and setup

HSIO-II platform

The most common HSIO-II setup will be the pixel test stands with HSIO-II + pixel interface hosting 18 RJ45 connectors for pixel frontend as seen in picture below. The HSIO-II hardware details can be found in the HSIO-II manual and further design details in Dave Nelson's HSIO-II documentation page or the hardware documentation gallery in RCEDevelopmentLab. You can also inspect the functionalities of the various components from the annotated HSIO-2 picture. A typical test stand setup is illustrated below ( Twiki login at top left of page if picture below does not display ):

HSIO2-setup-diagram.png HSIO2_teststand.JPG
The HSIO-II is delivered with the stand-off posts as shown in the picture to elevate the board to prevent possible shorts on the bench. An AC->12V DC power supply is included, but due to the different AC power plugs in different countries, the recipient should still prepare a standard computer power cable that works with the local AC plug. A cooling fan and grounding tail adaptor is separately distributed with adapting Molex connector in the powering connection chain as shown in the picture above. The banana socket tail on the fan power adapter is to allow convenient grounding connection between HSIO-II and frontend/LV supply. The HSIO-II is delivered without the transceivers for the SFP/QSFP ports as the usage here may vary significantly and only relevant for advanced multi-Gb/s optical links when they become available. Please follow the HSIO-II manual for recommendation on transceivers. An important precaution is that only the channels connecting to P3 to the interface have protection of buffers on the interface board while most headers on the main HSIO-II are directly connected to the Artix FPGA I/O pins and vulnerable to potential static damage.

For ITk strip stave tests, HSIO-II is backward compatible wrt the P3 interface for the same usage as HSIO-I once the strip test firmware is ported from Virtex-4 to Artix-200. HSIO-II provides alternative options for more calibration software operating on the RCEs locally. The strip stave interface is currently evolving towards the new stave design. A new strip stave interface board for HSIO-II is planned to follow the new stave test interface once its spec is defined.

Hardware Set-up

  • Check capacitors C350,C351 and C377,C378 to verify they NOT loaded. The oscillators next to them already have built-in capacitors so that they are not needed. HSIO2 delivered before Apr/1/2016 had them loaded by mistake. The locations of these capacitors are labelled on this diagram.
  • Connect the HSIO2 to the pixel adapter board (if required).
  • Select which ethernet port to use on Linux machine. This will typically be a secondary network port not connected to the company network.
  • Connect Linux machine to HSIO2 port U32 (next to SFP without USB) using a standard (not crossover) ethernet cable.This port has a unique MAC address on the shipping box for the distributed boards. The other ethernet port U17 (double decker with USB) can also be used but needs precaution as it has a fixed Xilinx generic MAC 00:0a:35:00:00:00 which could clash if there are two such ports connected to the same DHCP server.
  • Mount the small cooling fan next to the DTM (see picture) as the DTM Zynq and Marvel transceiver chips can run pretty hot, and connect its Molex power adapter to HSIO-II power port. Some fan and Molex power adapter assemblies will be available at CERN SLAC pixel office (1-1-41) for HSIO-II owners to pick up if not yet have one.
  • Connect HSIO2 + fan adapter to power using the included AC/DC converter and a user supplied mains (wall plug) power cable.
  • Connect a microUSB cable to the DTM port J35 on the side of the HSIO2 board as the DTM UART console. When not using the console port, please jump a short microUSB-USB cable from the DTM console to the U17 USB port (see picture) as this can help to avoid an open/floating DTM console which is known to cause DTM boot glitches for some boards.
  • Upon power up, check the power on LEDs:
    • Front edge center 2x2 green LEDs: Bottom left PWR (Artix power) should be on. Bottom right DONE (Artix boot success) should be on. Top row LEDs are user defined which can vary depending on firmware program.
    • DTM back corner large green LED should be on to indicate power for DTM. Other smaller LEDs are user defined.
  • Front display screen shows Artix firmware version and served interface type upon successful boot.
  • For applications such as pixel calibrations, a good ground connection can make significant difference on noise performance and DC level stability, and ensure the LVDS transmission common mode reference level is controlled. The fan Molex adapter has a banana socket tail which is specifically aimed to serve such a robust ground connection from HSIO2 to frontend board or LV power supply ground. HSIO-II without the Molex fan adapter can use the TP5 DGND probe post on the HSIO2 just behind the display screen.

A list of the existing HSIO-II boards can be found here: HsioList. Descriptions of example reference platforms can be found at: CERN SR1, SLAC Lab1.

Artix Firmware Download

In order to program the Artix FPGA on the HSIO the Xilinx Vivado software is required. If your institute does not have a Vivado license you can use the Vivado lab edition which does not need a license and is sufficient to download the firmware. Also install the cable drivers. There are two alternatives. Either use the digilent drivers and connect a micro USB cable to the Digilent port (see image) or use the "Linux" drivers and connect a Xilinx dongle to the JTAG connector. The Digilent solution is usually easier to deal with and does not need a dongle. progartix.png
You can then either use the Vivado GUI to download the bitfile to the FPGA or the mcs-file to the PROM (28f00ap30t-bpi-x16) or you can use a script for this purpose:
   > source <> (exact path and name depend on the install path, version, and architecture)
   > daq/rce/scripts/ <mcsfile.mcs>

The mcs-files can be downloaded from the rceproject web page. It is also possible to use impact from the Xilinx ISE suite instead of Vivado. To activate the new firmware after a download to the PROM push the cold boot push button on the HSIO.|

DTM Firmware Download

To install new firmware on the DTM you just have to copy the bitfile to the DTM via ethernet while the DTM is running: > scp DtmHsio _xxxxxxxx.bit root@rce_x:/mnt/boot/fpga.bit Replace rce_x by your RCE's hostname or IP address. To boot into the new firmware log on to the DTM as root and type > sync > reboot The sync command is needed to finalize any file transactions before the reboot. Bitfiles are available on the rceproject web page. If for some reason you cannot do the programming in situ you can also remove the SD card from the DTM and use a card reader to copy the bitfile to the BOOT partition.

COB platform

The COB test stand hardware components and some infrastructure utilities are described in the 2015 RCE training workshop for the RTEMS infrastructure. The current Linux test environment development is targeting both RCE on COB DPM and DTM on HSIO-II. The immediately available COB test stand interface is through the SFP RTM to connect to single optical links.

COBs used in CSC system and the current ATLAS test stands are V8 with the original Fulcrum 10 GE embedded switch. As Intel acquired there has been an evolution that discontinued the original Fulcrum chip and issues a replacement chip with the same functionalities, but the register space and connectivity being significantly remapped. There are a few COBs at SLAC with this new replacement chip which required significant updates in DPM and DTM core code to branch accordingly based on switch type. DTM already has one branch with no switch when residing in HSIO2. These version switches will be hidden from the users as a core utility.

Host Computer Requirements

  • Linux server capable of running 64-bit Linux with two network interfaces. One interface for external networking to the lab, one interface to connect to the RCE. The Linux server - RCE interface is preferably 1GE. While regular pixel test stand calibration intense dataflow cycles are confined within RCEs, the large histogram output volume can benefit from a 1GE interface between the RCE and the Linux host computer. Other applications with heavier I/O between Linux server and RCE may also prefer the 1GE ethernet.
  • USB card reader
  • micro-USB cables to connect the RCE UART to the server
  • ethernet cable to connect one server network controller to the RCE

Setting up a Linux server for RCE development

Requirements for an RCE development server

  • Enterprise Linux version 7 (Scientific Linux CERN version 6 is recommended)
  • Ideally this server is dual-homed with in a public/general purpose network and a private network shared with the RCEs
  • Development workstation software selection installed
  • NFS server to distribute software to RCEs (optional)
  • DHCP server for RCE network
  • NTP server configured for RCE network
  • SELinux disabled
  • No firewall on the internal network interface

Additional packages:

  • bash> sudo yum install dhcp cmake pyparted lib # complete list hereDisabling the firewall:
bash> sudo /sbin/chkconfig iptables off
bash> sudo /sbin/service iptables stop

Disabling SELinux:

edit as superuser /etc/sysconfig/selinux

and set

bash>sudo echo 0 >> /selinux/enable

Atlas RCE SDK Installation

Major Development Projects

RCE Pixel Application

RCE pixel readout and calibration application is described here: RCEGen3PixelApplication

RCE GBT readout Application

GBT link related RCE readout application developments are documented at: RCEAppGBT


Can't connect to RCE

If you are unable to ssh into the RCE, try connecting using Minicom. For the RCE3 crate, use the provided RJ45 to 9pin cable in the shelf manager port labeled 'serial'. Open a terminal and type


The shmm500 login is


and the password field is empty (just press enter). From here, you can send commands to the shelf manager. Type 'clia' at the beginning of each line before typing the Pigin Point shelf manager commands. [Someone from SLAC - please put in a list of available commands.]

Major updates:
-- MatthiasWittgen - 2015-11-01 Responsible: MatthiasWittgen
Last reviewed by: Never reviewed

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf Cap_mod.pdf r1 manage 1687.5 K 2016-04-14 - 12:29 DongSu  
JPEGjpg HSIO2-PSU-small.jpg r1 manage 195.9 K 2015-12-26 - 10:10 DongSu  
JPEGjpg HSIO2-PSU.jpg r1 manage 641.7 K 2015-12-26 - 10:10 DongSu  
PNGpng HSIO2-annotated.png r1 manage 1596.1 K 2016-01-04 - 22:47 DongSu  
JPEGjpg HSIO2-fan-USB.JPG r1 manage 370.0 K 2016-05-25 - 02:39 DongSu  
JPEGjpg HSIO2-pixel-interface-small.jpg r1 manage 345.5 K 2015-12-26 - 10:09 DongSu  
JPEGjpg HSIO2-pixel-interface.jpg r1 manage 1725.7 K 2015-12-26 - 10:10 DongSu  
PNGpng HSIO2-setup-diagram.png r1 manage 120.1 K 2016-01-04 - 22:47 DongSu  
PNGtiff HSIO2-setup-diagram.tiff r1 manage 189.8 K 2015-12-26 - 15:37 DongSu  
JPEGjpg HSIO2_teststand.JPG r1 manage 849.2 K 2016-09-02 - 23:55 DongSu  
PNGpng progartix.png r1 manage 614.0 K 2016-05-17 - 17:28 MartinKocian HSIO 2 Artix programming ports.
Edit | Attach | Watch | Print version | History: r52 < r51 < r50 < r49 < r48 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r52 - 2018-12-16 - MatthiasWittgen
No permission to view Atlas.WebLeftBar
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Atlas All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback