CarloTest

Introduction

This page describes the debugging procedures for FELIX systems. The first part of the twiki focuses on the hardware and is relevant for the on-call readout expert, the second is dedicated to the software and is relevant for on-call DAQ/HLT experts.

Note on logging: whenever an intervention is made on a system at P1 a dedicated elog entry should be made describing what was done and why, plus the expected impact on the system.

Note of jargon, cards and devices: one physical FELIX card is seen as two PCI devices by the computer. In many tools the card is selected via option -c, the device with -d. The enumeration starts at 0: card 0 includes devices 0 and 1. Card 1 corresponds to devices 2 and 3. Each device serves the e-links corresponding to one MTP connector (i.e. 12 GBT links per device on 24-channel cards). The TTC connector and BUSY are one per card.

Hardware

Diagnostic tools

Card identification, physical location

Some FELIX PCs are equipped with two cards. Card #0 is installed in the bottom slot of the FELIX PC, card #1 on the top slot. T

The location of the FELIX PCs is reported in the Table at the end of the page.

Optical power on GBT links

Each FELIX card is equipped with four optical transceivers called MiniPOD s. Two minipods emit light (TX), two receive light (RX). The emitted and received optical power can be visulised with the command

flx-info -c <card number>

The output is pasted below. The power is reported in the second table.

How to the read the table below:
# = FLX link endpoint OK     (no LOS)
- = FLX link endpoint not OK (LOS)
First letter:  Current channel status
Second letter: Latched channel status
Example: #(-) means channel had lost the signal in the past but the signal is present now.
 
Latched / current link status of channel:
        |   0  |   1  |   2  |   3  |   4  |   5  |   6  |   7  |   8  |   9  |   10  |   11  |
        |======|======|======|======|======|======|======|======|======|======|=======|=======|
1st TX | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) |  #(#) |  #(#) |
1st RX | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) |  -(-) |  -(-) |
2nd TX | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) |  #(#) |  #(#) |
2nd RX | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) |  -(-) |  -(-) |
 
 
Optical power (rx or tx) of channel in uW:
        |       0 |       1 |       2 |       3 |       4 |       5 |       6 |       7 |       8 |       9 |      10 |      11 |
        |=========|=========|=========|=========|=========|=========|=========|=========|=========|=========|=========|=========|
1st TX |  845.10 |  872.70 |  873.30 |  888.70 |  898.90 |  852.70 |  923.10 |  852.40 |  978.20 |  862.30 |  911.90 |  773.30 |
1st RX |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |
2nd TX |  911.40 |  964.50 |  899.80 | 1016.60 |  940.30 |  929.50 |  976.30 |  954.80 | 1065.40 |  974.00 |  965.20 |  902.70 |
2nd RX |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |

TTC fibre diagnostic

The status of the TTC optical connection can be checked with

flx-info -c <card number>

The output contains a section called "TTC (ADN2814) status" reporting one of the messages listed below

BUSY LEMO diagnostic

The BUSY state can be manually switched. To verify the correct functionality

1. assert BUSY with

fttcbusy -d <device number> -m 1 -i 0

2. measure the output of the LEMO connector with a voltmeter. The output must be 0V (BUSY on = logical zero)

3. de-assert busy with

fttcbusy -d <device number> -m 1 -i 0

4. use the voltmeter to check that you have a logical 1 (between 3.3 and 5 V)

If you do not get the expected logic levels something is wrong with the FLX-712. Replace the entire PC by a spare.

Issues and solutions

flx-init reports "Lock not found" (failed recovery of TTC clock)

The output of flx-init produces the

Card type: FLX-712
Configuring Si5345...
Si5345 hard reset
Si5345 configuration done
Enabling Si5345 output
Si5345: LOS register = 0x20
Si5345: Sticky LOS register = 0xf0
Si5345: LOL register = 0x02
Si5345: LOL register = 0x02
Si5345: LOL register = 0x02
Si5345: LOL register = 0x02
Si5345: LOL register = 0x02
Si5345 ERROR: Lock not found in 5 secs
Si5345: Sticky LOL register = 0x06

Follow the instructions of section "TTC fibre diagnostic"

FELIX card not detected

If a FELIX card disappears from the system query the driver with

cat /proc/flx

and look for errors such as

Monitoring via IS, ERS

Software

Diagnostic for the commissioning phase

List of FELIX nodes

The table below reports the list of all the FELIX nodes in USA15. The acronym HCA stands for Host Channel Adapter and indicated the type of Mellanox network card installed.

List of SW ROD nodes

Major updates:
-- MarkusJoos - 2020-05-25

%RESPONSIBLE% CarloAlbertoGottardo
%REVIEW% Never reviewed