CarloTest

Introduction

This page describes the debugging procedures for FELIX systems. The first part of the twiki focuses on the hardware and is relevant for the on-call readout expert, the second is dedicated to the software and is relevant for on-call DAQ/HLT experts.

Note on logging: whenever an intervention is made on a system at P1 a dedicated elog entry should be made describing what was done and why, plus the expected impact on the system.

Note of jargon: one physical FELIX card is seen as two PCI devices by the computer. In many tools the card is selected via option -c, the device with -d. The enumeration starts at 0: card 0 includes devices 0 and 1. Card 1 corresponds to devices 2 and 3. Each device serves the e-links corresponding to one MTP connector (i.e. 12 GBT links per device on 24-channel cards). The TTC connector and BUSY are one per card.

Hardware

Diagnostic tools

Optical power on GBT links

The

TTC fibre diagnositic

BUSY LEMO diagnositc

Issues

Software

GBT link diagnostic

Each FELIX-PC is equipped with one or two FLX-712 PCIe cards. For the purpose of the S/W tools they are enumerated with 0 and 1. This is the [X] parameter in e.g. flx-info.

Each card has two connectors for MTP fibres. Depending on the number (4 or 8) of MiniPOD devices installed on the FLX-712, these fibres have (together) 24 or 48 channels.

Before drawing any conclusions about the status of these channels run the "flx-init -c [X]" program to (re)initialize the links. Then run "flx-info -c [X] POD". You will see something like this:

How to the read the table below:
# = FLX link endpoint OK     (no LOS)
- = FLX link endpoint not OK (LOS)
First letter:  Current channel status
Second letter: Latched channel status
Example: #(-) means channel had lost the signal in the past but the signal is present now.
 
Latched / current link status of channel:
        |   0  |   1  |   2  |   3  |   4  |   5  |   6  |   7  |   8  |   9  |   10  |   11  |
        |======|======|======|======|======|======|======|======|======|======|=======|=======|
1st TX | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) |  #(#) |  #(#) |
1st RX | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) |  -(-) |  -(-) |
2nd TX | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) | #(#) |  #(#) |  #(#) |
2nd RX | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) | -(-) |  -(-) |  -(-) |
 
 
Optical power (rx or tx) of channel in uW:
        |       0 |       1 |       2 |       3 |       4 |       5 |       6 |       7 |       8 |       9 |      10 |      11 |
        |=========|=========|=========|=========|=========|=========|=========|=========|=========|=========|=========|=========|
1st TX |  845.10 |  872.70 |  873.30 |  888.70 |  898.90 |  852.70 |  923.10 |  852.40 |  978.20 |  862.30 |  911.90 |  773.30 |
1st RX |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |
2nd TX |  911.40 |  964.50 |  899.80 | 1016.60 |  940.30 |  929.50 |  976.30 |  954.80 | 1065.40 |  974.00 |  965.20 |  902.70 |
2nd RX |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |    0.00 |

How to interpret this and what to do:

TTC fibre diagnostic

1. Execute "flx-info -c [X] ADN2814" (with X = number of card. I.e. "0" or "1")
2. If you see the message "The TTC optical connection is up and working" you have got nothing to do
3. If you see the message "No light arriving. Check the fibre connection to the FLX-712" swap the TTC fibre with a fibre from a neighbouring FLX-712 and draw conclusions.
4. If you see the message "Light is arriving but the FLX-712 may have an internal problem":
   1. Swap the TTC fibre with a fibre from a neighbouring FLX-712
   2. Log on to the neighbouring FELIX (from where you have taken the fibre) and and run "flx-info -c [X] ADN2814"
   3. If on that FELIX you get "The TTC optical connection is up and working" you can conclude that the FLX-712 in the first FELIX has a problem. Replace the entire FELIX PC
   4. If you get the same error in the other FELIX as well, the problem is upstream in the TTC system. You cannot fix this.

Busy LEMO diagnostic

1. Execute "fttcbusy -m 1 -i 0"
2. Connect the voltmeter with the LEMO adapter to the Busy output and check that you have logical 0. This mean 0 Volt
3. Execute "fttcbusy -m 0 -i 1"
4. Use the voltmeter to check that you have a logical 1. This means 3.3V

List of FELIX nodes

The table below reports the list of all the FELIX nodes in USA15. The acronym HCA stands for Host Channel Adapter and indicated the type of Mellanox network card installed.

List of SW ROD nodes

Major updates:
-- MarkusJoos - 2020-05-25

%RESPONSIBLE% CarloAlbertoGottardo
%REVIEW% Never reviewed