Persistent Back-End for the Atlas Information System of TDAQ

I. Project official documentation

1. Initial project description

The ATLAS Data Acquisition provides a service to publish and share operational monitoring information among applications. Apart from a few of these data which are archived in the conditions database and the histograms which are archived in the MDA system, most of the information is transient and lost at the end of a data taking session. In order to be able to analyze the quality of data taking a posteriori it would be very useful to implement a generic mechanism to store operational monitoring data into a database.

The technical student will participate to the design of the database schema, with particular emphasis on its optimization in order to sustain potentially large data insertions and specific queries on retrieval. He then will develop the application(s) in charge of getting the data from the online system and archiving them. This project will allow him to gain experience with databases and also to learn about the interprocess communication (CORBA) used in the ATLAS online to share monitoring data. Knowledge about the basics of databases as well as object oriented programming (Java, C++ or Python) are required.

2. Official documents emerged from project work

3. Requirements Document

II. Project details

1. Dataset considerations

2. Top level architecture

3. Insertion Path

4. Retrieval Path

III. Software platform investigations

1. Long list of platforms taken into consideration

There are a vast number of database solutions available. After doing initial research browsing for the best, most optimized such platforms for time series data (and taking advice form various people from the DAQ group), just seven of these stood out:

Some of these have been ruled out due to not having certain features to satisfy our requirements. The platforms that were ruled out were:

  • RRDTool, Graphite, OpenTSDB they do not satisfy the generality requirement of the system (they only store numerical data)
  • KDB developed by a company called Kx which offers a non-commercial license (which is not open source) for the 32bit builds of kdb+. There is a google group for non-commercial users, which Kx does support, however the company focuses most of its efforts in developing its commercial solutions.

So the shortlist of platforms considered in more detail is:

2. Short listed platforms evaluation

2.1. HDF5 - Hierarchical Data Format 5

See HDF5.pdf for a summary of information found about this platform.

PyTables is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data. It is built on top of HDF5. See (courtesy of Ricardo)

2.2. Cassandra

See Cassandra.pdf for a summary of information found about this platform.

Explanation of how Cloudkick used Cassandra to store metrics from clouds:

2.3. MongoDB

See MongoDB.pdf for a summary of information found about this platform.

MongoDB used at CERN by CMS for data aggregation: see and

IV. Further work that needs to be accomplished

V. Attachments

-- AlexandruSicoe - 24-Apr-2012

-- AlexandruSicoe - 11-Feb-2011

  • HDF5.pdf: Information about the HDF5 platform

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf Cassandra.pdf r1 manage 95.2 K 2011-02-02 - 17:58 AlexandruSicoe Information about the Cassandra platform
PDFpdf HDF5.pdf r1 manage 191.3 K 2011-02-02 - 17:58 AlexandruSicoe Information about the HDF5 platform
PDFpdf MongoDB.pdf r1 manage 63.7 K 2011-02-02 - 17:59 AlexandruSicoe Information about the MongoDB platform
PDFpdf PBEAST_Report.pdf r1 manage 2189.6 K 2012-04-25 - 00:26 AlexandruSicoe Masters Thesis on P-BEAST
PDFpdf P_BEAST_Requirements_Draft_V2.3.pdf r1 manage 118.3 K 2011-02-24 - 09:43 AlexandruSicoe Requirements draft version 2.3
Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r13 - 2012-04-25 - AlexandruSicoe
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback