Raw Data Format

In 2019, there was a restructuring and consolidation of the code, meaning that some of the format of the events has changed, hopefully for the better. However, the old format does not completely represent the data format prior to 2019, this because there were some modification which were not documented. Still, the changes are minor such as the IDs for the modules which are slightly different.

After 2019

The raw data format can be seen as a sequence of blocks. In this section, the words are referred to with an offset, the offset is with respect to the start of the block. The offset is in units of words, not bytes. The raw file starts with a first block describing the run/file and is not documented here.

General description and block layout

The data from sequential events is separated by the so-called separator blocks. Each separator block is typically followed by an event start block. The event start block is followed by the readout module blocks. The number and position of these blocks depends on what is configured to be read out; see the OKS database for details. The readout module blocks share some layout in order to facilitate the reading of the data and of unknown readout types. This is important and if you are implementing a new module, make sure to follow it. After the readout module blocks there is the event end block, marking the end of the data.

Position	Block
0	Separator Block
1	Event Start Block
...	Readout Module Block
...	Readout Module Block
...	Readout Module Block
(N-1)	Event End Block

Separator Block

Position	Explanation	Example
+0	Separator Marker (aka Header ID), always the same value	0x1234cccc
+1	Number of words in separator (aka size), includes the first word. In this version, should always be the same value	0x00000004
+2	Number of blocks of data so far (in a simple setup, should match the event number +1)	0x000000a3
+3	Size of all the event blocks in bytes (why?????) before the next separator, does not include the separator block	0x0000014c

Event Start Block

In the event start block, there seems to be a sub-block. This sub block originates from the RCDApp and contains some run-related information. This sub-block, just like the blocks from the readout modules has as the first word a source identifier. The 2 high bytes of the source identifier are the sub-detector ID. The 2 low bytes of the source identifier are the Module ID. Both IDs can be configured in the OKS database, the Module ID (currently 0x0054) under the RCDApp and the sub-detector ID (currently 0x0051) from the General_TileSegment.data.xml file, since for BL4S a simplified database is used with a single generic sub-detector.

Position	Explanation	Example
+0	Event Start Marker (aka Event Start ID), always the same value	0xee1234ee
+1	Number of words in block (aka size), includes the first word. In this version, should always be the same value	0x00000009
+2	The format version of the data	0x03010000
+3	Start of the RCDApp sub-block. The source identifier	0x00510054
+4	Run number	0x5cfa80b6
+5	Level 1 ID (currently used as the event number)	0x00000001
+6	Bunch crossing ID (in the BL4S setup, this seems to always match the Level 1 ID)	0x00000001
+7	Level 1 trigger type (seems to be unused in BL4S)	0x00000000
+8	Detector event type (seems to be unused in BL4S)	0x00000000

Readout Module Blocks

Every readout module block should contain a header and a footer. The data from the module should be placed after the header and before the footer. The specific format of the data depends from module to module.

The first word of the header should always be the source identifier. The 2 high bytes of the source identifier are the sub-detector ID. The 2 low bytes of the source identifier are the Module ID. Both IDs can be configured in the OKS database, the Module ID under the corresponding channel configuration and the sub-detector ID (currently 0x0051) from the General_TileSegment.data.xml file, since for BL4S a simplified database is used with a single generic sub-detector.

The second word of the header is an ID to identify which type of module, called Model ID. These IDs are in the source code of the plugins implementing the modules, see the rcd_**** under RCDTDAQ.

The third word of the header should be the size of the block in words. This includes the 3 header words and 1 footer word.

The footer contains a single word which should always have the value 0xc0badebb.

For the specific modules, see below:

V792 (QDC)

The QDC traditionally used in BL4S has 32 channels. There does exist a V792 with only 16 channels, but it has not been used in the scope of BL4S. If it is ever used, some adaptations to the code might be required.

The V792 has a Model ID of 0x00000300. Typically the V792 reads out all the channels, but it seems that it can on occasion miss a few, so the data packet itself will always be 32 words or smaller. The 32 channels (1 word of data for each) plus the 3 words from the header, the one word from the footer and the 1 header and 1 trailer from the QDC data explain why the fragment size is typically 0x00000026, which is 32+6=38.

Position	Explanation	Example
+0	The source identifier	0x00510001
+1	Model ID, always the same value	0x00000300
+2	Size, typically has the value 0x00000026	0x00000026
+3	Start of V792 data. The V792 defines a header word	0xfa012000
...	Typically 32 words with the V792 data format	0xf8004037
N-2	End of V792 data. The V792 defines a trailer word	0xfc000014
N-1	Fragment Footer	0xc0badebb

The V792 header word is a bitfield, see below. The most relevant quantities are bits 26-24 which identify the word as being a header and bits 13-8 (CNT) the number of channel (i.e. words) that have been read.

31	30	29	28	27	26	25	24	23	22	21	20	19	18	17	16	15	14	13	12	11	10	9	8	7	6	5	4	3	2	1	0
GEO[4:0]					0	1	0	CRATE[7:0]								0	0	CNT[5:0]						NA

The V792 data words are bitfields, see below. The most relevant quantities are bits 26-24 which identify the word as being a data word, bits 20-16 which indicate what channel the data refers to and bits 11-0 which give the ADC reading. Bits 13 and 12 might also be useful for indicating conditions that may have happened on the channel. Note that for the 16 channel V792 (the V792 N) the CHANNEL field is one bit shorter.

Version	31	30	29	28	27	26	25	24	23	22	21	20	19	18	17	16	15	14	13	12	11	10	9	8	7	6	5	4	3	2	1	0
V792	GEO[4:0]					0	0	0	NA			CHANNEL[4:0]					NA		UN	OV	ADC[11:0]
V792 N	GEO[4:0]					0	0	0	NA			CHANNEL[3:0]				NA			UN	OV	ADC[11:0]

The V792 trailer word is a bitfield, see below. The most relevant quantities are bits 26-24 which identify the word as being a trailer and bits 23-0 which give an event counter, typically it should be the same as the event number.

31	30	29	28	27	26	25	24	23	22	21	20	19	18	17	16	15	14	13	12	11	10	9	8	7	6	5	4	3	2	1	0
GEO[4:0]					1	0	0	EVENT COUNTER[23:0]

V1290 (TDC)

V560 (Scaler)

Eudaq

(n.b. The current implementation/description assumes that the EUDAQ is only being used with the mimosa detectors. It is possible to use with some other detectors, but would require changes to the code at several levels)

The Eudaq readout plugin receives UDP packets from EUDAQ and stores them in the raw event data stream as a fragment. The first word of the UDP packets from EUDAQ must be the value 0x0d15ea5e and the last word must be the value 0xd15ea5e5. These two words are stripped and are not placed in the fragment inside the raw data stream. This plugin is agnostic of the data that EUDAQ is actually sending, even though at the moment only the Mimosa detectors are used in this way. This means that in the RCDRawEvent class the EUDAQ fragment is automatically assumed to be for a mimosa detector and is decoded as such. If new detectors are implemented from the EUDAQ side, this will most likely require adding a status word to identify which detector (most likely within the UDP packet itself) and then modifying the RCDRawEvent class to correctly marshal the data.

The EUDAQ has a Model ID of 0x00000800. Each UDP packet has its own header, with the first word being the IP from where the UDP packet was sent and the second word the total size of the UDP packet + header. There may be more than one UDP packet in a EUDAQ fragment. Typically one would expect 2 UDP packets, one for each readout frame of the mimosa detectors, however it is possible to configure EUDAQ to only send one frame or there might be network timeouts, meaning that the number of packets may vary between 0 and 2. Please note however that timeouts during data taking most likely are a symptom of some other issue in the setup.

Position	Explanation	Example
+0	The source identifier	0x0051000a
+1	Model ID, always the same value	0x00000800
+2	Size of the fragment	0x00000020
+3	Start of first UDP packet. The IP that sent the packet	0x83a985d2
+4	The length of the UDP packet +2	0x0000000d
...	The UDP packet itself	0x10000003
...	The UDP packet itself	0x00000001
...	The UDP packet itself	0x55515555
+M	Start of second UDP packet. The IP that sent the packet	0x83a985d2
+M+1	The length of the UDP packet +2	0x0000000f
...	The UDP packet itself	0x11000003
...	The UDP packet itself	0x00000001
...	The UDP packet itself	0x55515555
N-1	Fragment Footer	0xc0badebb

MMFE8 (Micromegas)

Event End Block

For the event end block, the positions are counted from the end... for some reason. Depending of the value of the last word, there are two possible layouts (although I have only seen the first one with BL4S). I have still not figured out where that status words come from nor how to configure how many exist.

Layout 1

Position	Explanation	Example
...	Status Word	0x00000000
...	Status Word	0x00000000
-2	Number of Status Words	0x00000002
-1	Size of all the readout module blocks in words (n.b. does not include the event start and event end blocks)	0x00000045
+0	Status Position(0: Status -> Data, 1: Data -> Status)	0x00000001

Layout 2

Position	Explanation	Example
-(N-1)	Size of all the readout module blocks in words (n.b. does not include the event start and event end blocks)	0x00000045
...	Status Word	0x00000000
...	Status Word	0x00000000
-1	Number of Status Words	0x00000002
+0	Status Position(0: Status -> Data, 1: Data -> Status)	0x00000000

Old format

Word	Explanation
1234cccc	Seperator Marker
00000004	Number of words in seperator
003d01a0	Number of following block of data within this file sequence
00000198	Size of block before next seperator (there are 49 words, this marker shows 198 = 49*4, which means it is number of bytes)
ee1234ee	Event Start header marker
00000009	Number of words in header
03010000	Format version no (same with real atlas events)
00510054	Source identifier (0051: sub det, 0054: Module ID)
5418751b	Run No (Repeated in whole file)
003d019f	Level 1 ID
003d019f	Bunch Crossing ID
00000000	Level 1 Trig Type
00000000	Detector Event Type
00510002	Channel ID (v792)
00000792	Model ID
00000022	Number of words in next block (34: 32 data + 1 header + 1 EOB)
fa012000	QDC Data words(Details in DataChannelV792.cpp)
f8004036	...
f810404f
f8014054
f811406f
f8024056
f812405c
f803406a
f8134075
f8044068
f814405c
f8054052
f815405d
f80640ab
f8164077
f8074077
f8174068
f808405e
f818408a
f8094072
f8194072
f80a407f
f81a407c
f80b406b
f81b406f
f80c406c
f81c406c
f80d406e
f81d4083
f80e4079
f81e4070
f80f4075
f81f4079
fc3d05ee	QDC Trailer
00510003	Channel ID(1st v1290)
00001290	Model ID
47a033ff	TDC Data Words(Details in DataChannelV1290.cpp)
0819f119	...
00001333
00601309
0080173d
00c011ac
00201c12
00401758
008028e8
00e0203b
000052af
00a023fa
1819f00c
0919f119
010017c9
01600c99
01201c9e
0140207a
1919f006
8000029f	Global TDC trailer
00510006	Channel ID(2nd v1290)
00001290	Model ID
47a033ff	TDC Data Words(Details in DataChannelV1290.cpp)
0819f215	...
0080069f
00c00873
00a005ed
1819f005
0919f215
01000691
1919f003
8000015f	Global TDC trailer
00510004	Channel ID(v560)
00000560	Model ID
00000010	Scaler Data Words(Details in DataChannelV560.cpp)
003d01a0
00638db6
0221ec1e
02178a1a
02302813
01636ba4
060a8b03
00638dac
00000000
00000000
00293f71
024cd812
00000000
00000000
00000000
00000000
00000000	Status Word 1
00000000	Status Word 1
00000000	Status Word 1
00000000	Status Word 1
00000004	Number of Status Words
00000056	Data Words in the event (86 = 36(QDC)+21(TDC1)+11(TDC2)+18(Scaler)
00000001	Status Position(0: Status -> Data, 1: Data -> Status)