ATLAS Edinburgh GPU Computing

Porting the Z-finder algorithm to GPU (Chris Jones)

Overview

ATLAS is a general-purpose detector in the Large Hadron Collider (LHC), at CERN. Its purpose is to investigate the physics of the Standard Model, including the search for the Higgs boson, and possible new physics beyond this model.

With around a billion particle collisions occurring every second within the detector, it is impossible to store the data collected in its entirety. It is responsibility of the trigger to reduce this data, selecting out events of potential scientific interest for storage and further analysis. The works in three main stages: level 1 running on customised hardware, and levels 2 and 3 in software on server farms.

IDScan is a track-reconstruction program running in level 2 of the trigger, and taking its input from the inner detector. The first algorithm used by IDScan is the Z-finder algorithm, the purpose of which is to produce a good estimate for the z-coordinate of the collision event along the axis of the detector. This reduces the execution time of the track-reconstruction algorithms later in the chain.

The aim of this project will be to port the Z-finder algorithm to GPU using NVIDIA's C for CUDA, and investigate the performance.

Project Writeup

• GPUs and the ATLAS experiment (pdf)

SIMT design of the High Level Trigger Kalman Fitter (Maria Rovatsou)

Overview

The ATLAS detector is one of the two biggest detectors of the Large Hadron Collider (LHC) at the European Center of Nuclear Physics (CERN) dedicated for research of the origin of mass, extra dimensions, and dark matter. The detector at design performance works at luminosity 1034 cm^2 s^2 with beam bunch rate of 40MHz. The rate of the collisions taking place in the accelerator is the input rate of the software that stores and processes them, the ATLAS Trigger. The Trigger is divided into three levels, the first level is a custom electronics trigger and the other two levels compose the High-Level trigger that is responsible for combining the data from the detector and reconstructing the tracks of the particles in order to store interesting data. The track reconstruction is achieved with the utilization of the Kalman filter technique. The current implementation of the filter is a distributed sequential implementation. An acceleration of the filter would boost the whole process and would allow even more significant data to be available for processing. Apart from that, the proposed upgrade of the LHC in order to increase its performance, results in luminosity increase of one order of magnitude higher and would require a more efficient implementation of the Kalman filter. A possible way for acceleration of the filter is by porting it on a Graphics Processing Unit (GPU), which has immense computational power that can be exploited through programming models as CUDA. The proposed approach is a parallelization on thread level of the Kalman filter for execution on an NVIDIA GPU. The proposed programming model is CUDA and the analysis of its performance will be done with the CUDA profiling tools and with benchmarking tools.

Project Writeup

• Analysis Acceleration on GPUs for the ATLAS Experiment at CERN (pdf)

Resources

GPU-based Code

• Z-Finder GPU - Chris Jones (tar.gz)
• Kalman Fitter SIMT Design - Maria Rovatsou (tar.gz)
• Kalman Fitter for CUDA - Dimitry Emeliyanov (tar.gz)

Presentations

• Processing Petabytes per Second with the ATLAS Experiment at the Large Hadron Collider in CERN, Nvidia GPU Technology Conference September 2010 (stream, pdf)
• Algorithm Acceleration from GPGPUs for the ATLAS Upgrade, Computing in High Energy and Nuclear Physics October 2010 (pdf)
• GPU-based tracking for ATLAS Level 2 Trigger Parallelism in Experimental Nuclear Physics Workshop January 2011 (pdf)

ATLAS Trigger Software

• IDScan CERN wiki page
• IDScanGPU CERN wiki page
• IDScan Doxygen docs

HLT resources

• Determination of the z position of primary interactions in ATLAS prior to track reconstruction (pdf) Nikos Konstantinidis and Hans Drevermann ATLAS-DAQ-2002-014
• Tracking at level 2 for the ATLAS high level trigger (pdf) M.Sutton, Nuclear Instruments and Methods in Physics Research A 582 (2007)
• Fast SIMDized Kalman filter based track fit (pdf) S. Gorbunov, U. Kebschull, I. Kisel, V. Lindenstruth, W.F.J. Müller Computer Physics Communications 178 (2008)
• IDSCAN: LVL2 Tracking for the ATLAS Trigger (pdf) Janice Drohan UCL 1st Year Transfer Report June 20, 2003
• Fast Tracking for the ATLAS LVL2 Trigger (pdf) J. T. M. Baines, H. Drevermann, D. Emeliyanov, N. Konstantinidis, F. Parodi, C. Schiavi, M. Sutton, Computing in High Energy Physics and Nuclear Physics 2004, Interlaken, Switzerland, 27 Sep - 1 Oct 2004, pp.246
• HLT Track Reconstruction Performance (pdf) S. Ask, J.T. Baines, A. Coccaro, D. Emeliyanov, I. Grabowska-Bold, J. Kirk, N. Konstantinidis, J. Masik, E. Ozcan, F. Parodi, C. Schiavi, S. Sivoklokov, M. Sutton Detectors and Experimental Techniques, 17 Mar 2009. - 19 p.
• A Probabilistic Data Association Filter for fast tracking in the ATLAS Transition Radiation Tracker (pdf) D. Emeliyanov Nucl. Instrum. Meth. A566 (2006) 50-53.

Articles and Preprints

• ATLAS Technical Design Report: High-Level Trigger, Data Acquisition and Controls (pdf) ATLAS HLT/DAQ/DCS Group ATLAS-TDR-016 2 October, 2003
• Commissioning the ATLAS Inner Detector Trigger (pdf) Mark Sutton on behalf of the ATLAS Collaboration ATL-DAQ-PROC-2009-015
• Overview of the High-Level Trigger Electron and Photon Selection for the ATLAS Experiment at the LHC (pdf) IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 5, OCTOBER 2006
• Fast tracking in hadron collider experiments (pdf) Nikos Konstantinidis and Hans Drevermann
• Vertexing strategy and algorithms at ATLAS (pdf) F.Parodi, Nuclear Instruments and Methods in Physics Research A 560 (2006)
• ATLAS L1 track trigger for super-LHC (pdf) Nikos Konstantinidis ATL-UPGRADE-PROC-2010-003
• ATLAS Track Trigger for SLHC - Ideas & Plans (pdf) Nikos Konstantinidis ATL-UPGRADE-SLIDE-2009-243

CERN Articles and Preprints

• CERN Document Server search ATL-DAQ-PROC
• CERN Document Server search ATL-DAQ-SLIDE

Other Trigger Links

• Trigger Wiki
• Trigger Workshop 2009 Wiki