Raw Data Format
In 2019, there was a restructuring and consolidation of the code, meaning that some of the format of the events has changed, hopefully for the better.
However, the old format does not completely represent the data format prior to 2019, this because there were some modification which were not documented.
Still, the changes are minor such as the IDs for the modules which are slightly different.
After 2019
The raw data format can be seen as a sequence of blocks.
In this section, the words are referred to with an offset, the offset is with respect to the start of the block.
The offset is in units of words, not bytes.
The raw file starts with a first block describing the run/file and is not documented here.
General description and block layout
The data from sequential events is separated by the so-called separator blocks.
Each separator block is typically followed by an event start block.
The event start block is followed by the readout module blocks.
The number and position of these blocks depends on what is configured to be read out; see the OKS database for details.
The readout module blocks share some layout in order to facilitate the reading of the data and of unknown readout types.
This is important and if you are implementing a new module, make sure to follow it.
After the readout module blocks there is the event end block, marking the end of the data.
Position |
Block |
0 |
Separator Block |
1 |
Event Start Block |
... |
Readout Module Block |
... |
Readout Module Block |
... |
Readout Module Block |
(N-1) |
Event End Block |
Separator Block
Position |
Explanation |
Example |
+0 |
Separator Marker (aka Header ID), always the same value |
0x1234cccc |
+1 |
Number of words in separator (aka size), includes the first word. In this version, should always be the same value |
0x00000004 |
+2 |
Number of blocks of data so far (in a simple setup, should match the event number +1) |
0x000000a3 |
+3 |
Size of all the event blocks in bytes (why?????) before the next separator, does not include the separator block |
0x0000014c |
Event Start Block
In the event start block, there seems to be a sub-block.
This sub block originates from the RCDApp and contains some run-related information.
This sub-block, just like the blocks from the readout modules has as the first word a source identifier.
The 2 high bytes of the source identifier are the sub-detector ID.
The 2 low bytes of the source identifier are the Module ID.
Both IDs can be configured in the OKS database, the Module ID (currently 0x0054) under the RCDApp and the sub-detector ID (currently 0x0051) from the General_TileSegment.data.xml file, since for
BL4S a simplified database is used with a single generic sub-detector.
Position |
Explanation |
Example |
+0 |
Event Start Marker (aka Event Start ID), always the same value |
0xee1234ee |
+1 |
Number of words in block (aka size), includes the first word. In this version, should always be the same value |
0x00000009 |
+2 |
The format version of the data |
0x03010000 |
+3 |
Start of the RCDApp sub-block. The source identifier |
0x00510054 |
+4 |
Run number |
0x5cfa80b6 |
+5 |
Level 1 ID (currently used as the event number) |
0x00000001 |
+6 |
Bunch crossing ID (in the BL4S setup, this seems to always match the Level 1 ID) |
0x00000001 |
+7 |
Level 1 trigger type (seems to be unused in BL4S) |
0x00000000 |
+8 |
Detector event type (seems to be unused in BL4S) |
0x00000000 |
Readout Module Blocks
Every readout module block should contain a header and a footer.
The data from the module should be placed after the header and before the footer.
The specific format of the data depends from module to module.
The first word of the header should always be the source identifier.
The 2 high bytes of the source identifier are the sub-detector ID.
The 2 low bytes of the source identifier are the Module ID.
Both IDs can be configured in the OKS database, the Module ID under the corresponding channel configuration and the sub-detector ID (currently 0x0051) from the General_TileSegment.data.xml file, since for
BL4S a simplified database is used with a single generic sub-detector.
The second word of the header is an ID to identify which type of module, called Model ID.
These IDs are in the source code of the plugins implementing the modules, see the rcd_**** under
RCDTDAQ
.
The third word of the header should be the size of the block in words.
This includes the 3 header words and 1 footer word.
The footer contains a single word which should always have the value 0xc0badebb.
For the specific modules, see below:
V792 (QDC)
The QDC traditionally used in
BL4S has 32 channels.
There does exist a V792 with only 16 channels, but it has not been used in the scope of
BL4S.
If it is ever used, some adaptations to the code might be required.
The V792 has a Model ID of 0x00000300.
Typically the V792 reads out all the channels, but it seems that it can on occasion miss a few, so the data packet itself will always be 32 words or smaller.
The 32 channels (1 word of data for each) plus the 3 words from the header, the one word from the footer and the 1 header and 1 trailer from the QDC data explain why the fragment size is typically 0x00000026, which is 32+6=38.
Position |
Explanation |
Example |
+0 |
The source identifier |
0x00510001 |
+1 |
Model ID, always the same value |
0x00000300 |
+2 |
Size, typically has the value 0x00000026 |
0x00000026 |
+3 |
Start of V792 data. The V792 defines a header word |
0xfa012000 |
... |
Typically 32 words with the V792 data format |
0xf8004037 |
N-2 |
End of V792 data. The V792 defines a trailer word |
0xfc000014 |
N-1 |
Fragment Footer |
0xc0badebb |
The V792 header word is a bitfield, see below.
The most relevant quantities are bits 26-24 which identify the word as being a header and bits 13-8 (CNT) the number of channel (i.e. words) that have been read.
The V792 data words are bitfields, see below.
The most relevant quantities are bits 26-24 which identify the word as being a data word, bits 20-16 which indicate what channel the data refers to and bits 11-0 which give the ADC reading.
Bits 13 and 12 might also be useful for indicating conditions that may have happened on the channel.
Note that for the 16 channel V792 (the V792 N) the CHANNEL field is one bit shorter.
The V792 trailer word is a bitfield, see below.
The most relevant quantities are bits 26-24 which identify the word as being a trailer and bits 23-0 which give an event counter, typically it should be the same as the event number.
V1290 (TDC)
V560 (Scaler)
Eudaq
(n.b. The current implementation/description assumes that the EUDAQ is only being used with the mimosa detectors. It is possible to use with some other detectors, but would require changes to the code at several levels)
The Eudaq readout plugin receives UDP packets from EUDAQ and stores them in the raw event data stream as a fragment.
The first word of the UDP packets from EUDAQ must be the value 0x0d15ea5e and the last word must be the value 0xd15ea5e5.
These two words are stripped and are not placed in the fragment inside the raw data stream.
This plugin is agnostic of the data that EUDAQ is actually sending, even though at the moment only the Mimosa detectors are used in this way.
This means that in the
RCDRawEvent class the EUDAQ fragment is automatically assumed to be for a mimosa detector and is decoded as such.
If new detectors are implemented from the EUDAQ side, this will most likely require adding a status word to identify which detector (most likely within the UDP packet itself) and then modifying the
RCDRawEvent class to correctly marshal the data.
The EUDAQ has a Model ID of 0x00000800.
Each UDP packet has its own header, with the first word being the IP from where the UDP packet was sent and the second word the total size of the UDP packet + header.
There may be more than one UDP packet in a EUDAQ fragment.
Typically one would expect 2 UDP packets, one for each readout frame of the mimosa detectors, however it is possible to configure EUDAQ to only send one frame or there might be network timeouts, meaning that the number of packets may vary between 0 and 2.
Please note however that timeouts during data taking most likely are a symptom of some other issue in the setup.
Position |
Explanation |
Example |
+0 |
The source identifier |
0x0051000a |
+1 |
Model ID, always the same value |
0x00000800 |
+2 |
Size of the fragment |
0x00000020 |
+3 |
Start of first UDP packet. The IP that sent the packet |
0x83a985d2 |
+4 |
The length of the UDP packet +2 |
0x0000000d |
... |
The UDP packet itself |
0x10000003 |
... |
The UDP packet itself |
0x00000001 |
... |
The UDP packet itself |
0x55515555 |
+M |
Start of second UDP packet. The IP that sent the packet |
0x83a985d2 |
+M+1 |
The length of the UDP packet +2 |
0x0000000f |
... |
The UDP packet itself |
0x11000003 |
... |
The UDP packet itself |
0x00000001 |
... |
The UDP packet itself |
0x55515555 |
N-1 |
Fragment Footer |
0xc0badebb |
MMFE8 (Micromegas)
Event End Block
For the event end block, the positions are counted from the end... for some reason.
Depending of the value of the last word, there are two possible layouts (although I have only seen the first one with
BL4S).
I have still not figured out where that status words come from nor how to configure how many exist.
Layout 1
Position |
Explanation |
Example |
... |
Status Word |
0x00000000 |
... |
Status Word |
0x00000000 |
-2 |
Number of Status Words |
0x00000002 |
-1 |
Size of all the readout module blocks in words (n.b. does not include the event start and event end blocks) |
0x00000045 |
+0 |
Status Position(0: Status -> Data, 1: Data -> Status) |
0x00000001 |
Layout 2
Position |
Explanation |
Example |
-(N-1) |
Size of all the readout module blocks in words (n.b. does not include the event start and event end blocks) |
0x00000045 |
... |
Status Word |
0x00000000 |
... |
Status Word |
0x00000000 |
-1 |
Number of Status Words |
0x00000002 |
+0 |
Status Position(0: Status -> Data, 1: Data -> Status) |
0x00000000 |
Old format
Word |
Explanation |
1234cccc |
Seperator Marker |
00000004 |
Number of words in seperator |
003d01a0 |
Number of following block of data within this file sequence |
00000198 |
Size of block before next seperator (there are 49 words, this marker shows 198 = 49*4, which means it is number of bytes) |
|
|
ee1234ee |
Event Start header marker |
00000009 |
Number of words in header |
03010000 |
Format version no (same with real atlas events) |
00510054 |
Source identifier (0051: sub det, 0054: Module ID) |
5418751b |
Run No (Repeated in whole file) |
003d019f |
Level 1 ID |
003d019f |
Bunch Crossing ID |
00000000 |
Level 1 Trig Type |
00000000 |
Detector Event Type |
|
|
00510002 |
Channel ID (v792) |
00000792 |
Model ID |
00000022 |
Number of words in next block (34: 32 data + 1 header + 1 EOB) |
fa012000 |
QDC Data words(Details in DataChannelV792.cpp) |
f8004036 |
... |
f810404f |
|
f8014054 |
|
f811406f |
|
f8024056 |
|
f812405c |
|
f803406a |
|
f8134075 |
|
f8044068 |
|
f814405c |
|
f8054052 |
|
f815405d |
|
f80640ab |
|
f8164077 |
|
f8074077 |
|
f8174068 |
|
f808405e |
|
f818408a |
|
f8094072 |
|
f8194072 |
|
f80a407f |
|
f81a407c |
|
f80b406b |
|
f81b406f |
|
f80c406c |
|
f81c406c |
|
f80d406e |
|
f81d4083 |
|
f80e4079 |
|
f81e4070 |
|
f80f4075 |
|
f81f4079 |
|
fc3d05ee |
QDC Trailer |
|
|
00510003 |
Channel ID(1st v1290) |
00001290 |
Model ID |
47a033ff |
TDC Data Words(Details in DataChannelV1290.cpp) |
0819f119 |
... |
00001333 |
|
00601309 |
|
0080173d |
|
00c011ac |
|
00201c12 |
|
00401758 |
|
008028e8 |
|
00e0203b |
|
000052af |
|
00a023fa |
|
1819f00c |
|
0919f119 |
|
010017c9 |
|
01600c99 |
|
01201c9e |
|
0140207a |
|
1919f006 |
|
8000029f |
Global TDC trailer |
|
|
00510006 |
Channel ID(2nd v1290) |
00001290 |
Model ID |
47a033ff |
TDC Data Words(Details in DataChannelV1290.cpp) |
0819f215 |
... |
0080069f |
|
00c00873 |
|
00a005ed |
|
1819f005 |
|
0919f215 |
|
01000691 |
|
1919f003 |
|
8000015f |
Global TDC trailer |
|
|
00510004 |
Channel ID(v560) |
00000560 |
Model ID |
00000010 |
Scaler Data Words(Details in DataChannelV560.cpp) |
003d01a0 |
|
00638db6 |
|
0221ec1e |
|
02178a1a |
|
02302813 |
|
01636ba4 |
|
060a8b03 |
|
00638dac |
|
00000000 |
|
00000000 |
|
00293f71 |
|
024cd812 |
|
00000000 |
|
00000000 |
|
00000000 |
|
00000000 |
|
|
|
00000000 |
Status Word 1 |
00000000 |
Status Word 1 |
00000000 |
Status Word 1 |
00000000 |
Status Word 1 |
00000004 |
Number of Status Words |
00000056 |
Data Words in the event (86 = 36(QDC)+21(TDC1)+11(TDC2)+18(Scaler) |
00000001 |
Status Position(0: Status -> Data, 1: Data -> Status) |