Endcap Muon Track Finder Control Software

This page describes the software used to control the EMTF and in particular the Modular Track Finder (MTF7).

Problem Definition

We need a software that controls the upgrade endcap muon trigger system. The software needs to interact with the uTCA crates and all their boards (MTF7, miniPod/Avago, AMC13, and MCH) and in particular with the MTF7 and the optical boards. It has to be able to monitor and send commands to the system. It also need to define a easy to use user interface.

Requirement definition

The EMTF sw must fulfil the following requirements:

• it has to run on the computer installed in P5, and in particular within SLC6 (or later version).
• it must be fast
• It must be able to communicate via PCI express (and IPbus) with the boards
• It must be able to use DMA to write into the board memory, with or without a driver
• part of it must depends on the XDAQ framework
• it must be stored in a repository in cactus
• it must be compatible with nightly builds
• it must be able to produce RPM for centralised installation
• it must run in two basic modes:
  • one within the CMS trigger architecture during CMS and CSC running
  • one stand alone to run potentially non standard tests
• it must provide an easy to use and maintainable user interface
• it must be easy to expand, but it is not expected to be scalable
• it must interface the CMS database to read and write configuration keys and LUT

Design

Most of the design of the system can be inherited from the legacy system. The design will be adapted to the new hardware, but possibly most if it can be reused (nobody wants to reinvent the wheel!). The main difference is the communication protocol, so the HAL interface has to be changed (not VME anymore but PCIe). GP already started it: PCIExpressHALIntegration

List of Objects

First I would replicate real objects:

Real Objects: Generic objects (probably Abstract Factory pattern)

• Board
• Crate

• MTF7
• optical board
• MCH
• AMC13

Abstract objects

Generic

• Timing
• Database
• Configuration
• Test
• Errors Handliing - based on XDAQ except/Exception class
• Monitorables
• Command parser
• spy fifos
• raw data utils
• register settings
• misc sp function (redesign ?)
• trigModes (useful ?)
• sTTS_states (useful ?)

The bold ones are coming from the legacy system control software.

Specific (does it make sense to have specific objects for functionality?)

• Optical Links
• FPGA
• LookUpTable

In the old code there was a package just for firmware and LUT. Redesign?

Package design

Question: can svn accept two package with the same name?

Exercise check whether most of the functionality are following the Facade pattern.

The package can be factorised in three subpakcages:

SPBaseInteraction The very basic layer will provide base interaction between the boards: initialisation of the devices, board status/communication checks, reading and writing.

SPFunctions This package will contain the common functionalities of the boards. This package depends on SPBaseInteractions. In this packages it would be very useful to have a series of test executable to perform standard (and not) test of functionality.

LUT and Firmware - This comes from the old control software. Is it really useful to have separate? Can't it be on the base interaction? without firmware there is no base interaction... Maybe we can factorise the two and leave the LUT here and move the firmware on the base intersection? overhead or really useful?

SPValidation It is intended that each time a new module is produced, an old one is repaired or a new firmware set is uploaded that the basic tests would be run before any higher level processes incorporating the module are executed.

For the trigger supervisor package we must rely on SWATCH. There is no version of it yet so we have to wait for it.

Guidline for style

The style of the code is based as much as possible on google c++ style guideline.

Packages are lower case, and in general I will try to keep all files lower case to avoid incompatibility with different filesystems.

Handson instructions

Access the servers

How to access the CSCTF machines:

• First log in the node:
  • At P5: ssh -X -l yourUserName cmsusr
  • At 904: ssh -X -l yourUserName cms904usr
• Then log to the specific machine
  • At P5: srv-s2g18-26-01 - This is the final server machine that we'll use. Though at the moment it is not yet connected to the crate.
  • At P5: vmepc-s1d06-42-01 - This machine is sitting in the DAQ rack, close to the uTCA EMTF crate and it is connected via fibers cable to the MCH. This machine will probably be replaced with vmepc-s2g18-25-01.
  • At 904: vmepc-e1x06-32-01 - This machine is connected via fibers to the uTCA EMTF crate in b904.

All L1 PC information can be found in the central twiki L1TriggerPCs.

MTF7 driver

The driver for the boards can be found in svn:

https://svnweb.cern.ch/trac/cactus/browser/trunk/cactusprojects/emtf/mtf7_driver

Driver documentation pdf

Here the instruction to use it from the README.txt of Alex:

To install and run:

- recompile driver 'make clean; make'

- change if necessary parameters in utca_sp12.cfg file
DEBUG=1 # Enable debug output
SLOTS=1 # Number of installed physical cards in the system
OWNER=root # Owner of /dev/utca_sp12x device files
GROUP=users # Group owner of /dev/utca_sp12x device files
MODE=664 # Default permission of /dev/urca_sp12 device files
!!! Check if group name exists and that user actually belongs to it
- copy utca_sp12.cfg to /etc/ folder
- try to load driver under root with utca_sp12.local script
- check /sbin/lsmod if driver is loaded
- check /var/log/messages for any obvious initialization errors
- check ownership and permissions of /dev/utca_sp12x device files
- try to access device from your user program

Checkout HAL

First login in P5 gate machines and then as csctfdev. At the moment the boards communicates using a driver. For this reason you have to create an adequate busAdapter for the driver. - The possibility of getting rid of the driver is under evaluation. At 904, where another development board is installed, the workflow is very similar. The difference are noted inline

ssh [username]@cmsusr # for 904 use cms904usr
sudo -H -u csctfdev bash -login
cd $HOME
mkdir your-user-name # this avoid conflicts in case of parallel development
cd your-user-name
svn co svn+ssh://[your-username]@svn.cern.ch/reps/cmsos/releases/baseline12/trunk/daq/hal daq/hal # please check the latest release of HAL before checking out

The hal version has been modified to introduce a new busAdapter working with the MTF7 driver. The details of this modification are in PCIExpressHALIntegration twiki. You can find the hacked version of HAL for XDAQ12 in SVN (cactus): https://svnweb.cern.ch/trac/cactus/browser/trunk/cactusprojects/emtf/hal_pciexpr

Remember to modify the environment.sh file with your personal settings. Remember to keep the same sub packages structure for HAL. Get the environment script from https://twiki.cern.ch/twiki/pub/Main/PCIExpressHALIntegration/environment.sh, save it in daq/hal/environment.sh, then source it and compile.

#get an environment script from the PCIExpressHALIntegration twiki
wget  https://twiki.cern.ch/twiki/pub/Main/PCIExpressHALIntegration/environment.sh
#connect to the machine where the hardware is connected
ssh -Y vmepc-s1d06-42-01 # at 904 use vmepc-e1x06-32-01 
#connect as csctfdev
sudo -H -u csctfdev bash -login
#go into your working directory
cd EMTF/daq/hal
source environment.sh
#compile
make

All this gymnastic is needed because the gate machine is open to svn connection.

Short test of HAL

You can run a short test of the HAL integration with the MTF7 driver in

EMTF/daq/hal/examples/runnit.sh

This is a short loopback test on the board. To make the test work you need to have load the driver and created the devices.

In order to start the driver you can use the shell script

EMTF/mtf7_driver_start.sh
# if you change fibers or change the setup of the crate or the computer you can recover the driver without reboot the computer using this script
EMTF/mtf7_recover.sh

The driver is committed in svn (cactus) https://svnweb.cern.ch/trac/cactus/browser/trunk/cactusprojects/emtf/mtf7_driver

The driver is also stored in the local disk of vmepc-s1d06-42-01 (or vmepc-e1x06-32-01 in case of 904) in /opt/data/project/cpp/mtf7_driver. A copy of it has been zipped and placed in ~csctfdev/EMTF/mtf7_driver.zip.

You can check if and how many boards are mounted in the computer looking at the files in /dev/. The devices names are utca_sp12X where X is a recursive number from 0 to the maximum number of boards mounted. In case the boards are not mounted or they do not respnd there are two scripts you can use: one (mtf7_driver_start.sh) has to be used when the board are not mounted (not present in /dev/), and the other (mtf7_recover.sh) can be used when the crate has been power cycled and therefore lost communication with the computer, but the devices are still mounted in the computer.

Check out the code

Check out the code

svn co svn+ssh://[your user name]@svn.cern.ch/reps/cactus/branches/bortigno/emutf cactusprojects/emutf
svn co svn+ssh://[your user name]@svn.cern.ch/reps/cactus/trunk/cactusprojects/setup cactusprojects/setup # this will be useful for running the SW and comes from the legacy system code

In case you need to create a new branch

svn co -N https://svn.cern.ch/reps/cactus/branches
mkdir bortigno_new_branch
svn add bortigno_new_branch
svn ci bortigno_new_branch

Code description

Here follows the description of the packages. According to the trigger supervisor design the stricture of the packages has to follow specific rules. A dummy subpackage for reference can be found in https://svnweb.cern.ch/trac/cactus/browser/trunk/cactusprojects/subsystem The rules followed for emutf are

• each package has to go inside "cactusprojects" then
• cactusprojects/emutf
• the Makefile in emutf has to be based on the template provided in https://svnweb.cern.ch/trac/cactus/browser/trunk/cactusprojects/subsystem/Makefile
• each subpackage has this form: cactusporjects/emutf/subpackage
• header files of the subpackage have to be put in an include which replicates the same path of the package. Here an example: "cactusprojects/emutf/subpackage/include/emutf/subpackage/"
• in the same directory of the header files a file named "version.h" has to be created with the version of the package. Its structure has to be based on this one: https://svnweb.cern.ch/trac/cactus/browser/trunk/cactusprojects/subsystem/supervisor/include/subsystem/supervisor/version.h
• source files of the subpackage have to go in "src/common"

base-interactions

This is the basic package. Here I define all the objects that have a correspondent hardware.

tests

For testing.

ts

Inside this package has to go the trigger supervisor with the cell and the configuration.

SWATCH

A SWATCH project has been started by the central trigger software team to facilitate the implementation of the trigger board control software for the trigger upgrade. More information about the project can be found in the tutorial slides:

https://indico.cern.ch/event/436271/

Here you can find the SWATCH user guide twiki

https://twiki.cern.ch/twiki/bin/view/CMS/SWATCHUsersGuide

SWATCH in 904

In order to use SWATCH in 904 or in P5 is needed to have a specific puppet profile (trigger-upgrade). This profile has been installed in the machine at 904.

Check PCI Express connection status

you can log in to MCH and see that the PCIe link is actually present.

command is: ssh root@192NOSPAMPLEASE.168.2.4, pw: root

then in MCH, issue "pcie" command, it will show you boards connected via PCIe to the MCH's switch

In the PC, you need to use: lspci | grep 5546

this will show you MTF7 boards that this PC detected. However, if MTF7 was turned off and then on, it will still show up in lspci, but the connection would not work. Reboot is needed.

-- PierluigiBortignon - 16 Oct 2014