ATLAS Edinburgh GPU Computing

Porting the Z-finder algorithm to GPU (Chris Jones)


ATLAS is a general-purpose detector in the Large Hadron Collider (LHC), at CERN. Its purpose is to investigate the physics of the Standard Model, including the search for the Higgs boson, and possible new physics beyond this model.

With around a billion particle collisions occurring every second within the detector, it is impossible to store the data collected in its entirety. It is responsibility of the trigger to reduce this data, selecting out events of potential scientific interest for storage and further analysis. The works in three main stages: level 1 running on customised hardware, and levels 2 and 3 in software on server farms.

IDScan is a track-reconstruction program running in level 2 of the trigger, and taking its input from the inner detector. The first algorithm used by IDScan is the Z-finder algorithm, the purpose of which is to produce a good estimate for the z-coordinate of the collision event along the axis of the detector. This reduces the execution time of the track-reconstruction algorithms later in the chain.

The aim of this project will be to port the Z-finder algorithm to GPU using NVIDIA's C for CUDA, and investigate the performance.

Project Writeup

  • GPUs and the ATLAS experiment (pdf)

SIMT design of the High Level Trigger Kalman Fitter (Maria Rovatsou)


The ATLAS detector is one of the two biggest detectors of the Large Hadron Collider (LHC) at the European Center of Nuclear Physics (CERN) dedicated for research of the origin of mass, extra dimensions, and dark matter. The detector at design performance works at luminosity 1034 cm^2 s^2 with beam bunch rate of 40MHz. The rate of the collisions taking place in the accelerator is the input rate of the software that stores and processes them, the ATLAS Trigger. The Trigger is divided into three levels, the first level is a custom electronics trigger and the other two levels compose the High-Level trigger that is responsible for combining the data from the detector and reconstructing the tracks of the particles in order to store interesting data. The track reconstruction is achieved with the utilization of the Kalman filter technique. The current implementation of the filter is a distributed sequential implementation. An acceleration of the filter would boost the whole process and would allow even more significant data to be available for processing. Apart from that, the proposed upgrade of the LHC in order to increase its performance, results in luminosity increase of one order of magnitude higher and would require a more efficient implementation of the Kalman filter. A possible way for acceleration of the filter is by porting it on a Graphics Processing Unit (GPU), which has immense computational power that can be exploited through programming models as CUDA. The proposed approach is a parallelization on thread level of the Kalman filter for execution on an NVIDIA GPU. The proposed programming model is CUDA and the analysis of its performance will be done with the CUDA profiling tools and with benchmarking tools.

Project Writeup

  • Analysis Acceleration on GPUs for the ATLAS Experiment at CERN (pdf)


GPU-based Code

  • Z-Finder GPU - Chris Jones (tar.gz)
  • Kalman Fitter SIMT Design - Maria Rovatsou (tar.gz)
  • Kalman Fitter for CUDA - Dimitry Emeliyanov (tar.gz)


  • Processing Petabytes per Second with the ATLAS Experiment at the Large Hadron Collider in CERN, Nvidia GPU Technology Conference September 2010 (stream, pdf)
  • Algorithm Acceleration from GPGPUs for the ATLAS Upgrade, Computing in High Energy and Nuclear Physics October 2010 (pdf)
  • GPU-based tracking for ATLAS Level 2 Trigger Parallelism in Experimental Nuclear Physics Workshop January 2011 (pdf)

ATLAS Trigger Software

HLT resources

  • Determination of the z position of primary interactions in ATLAS prior to track reconstruction (pdf) Nikos Konstantinidis and Hans Drevermann ATLAS-DAQ-2002-014
  • Tracking at level 2 for the ATLAS high level trigger (pdf) M.Sutton, Nuclear Instruments and Methods in Physics Research A 582 (2007)
  • Fast SIMDized Kalman filter based track fit (pdf) S. Gorbunov, U. Kebschull, I. Kisel, V. Lindenstruth, W.F.J. Müller Computer Physics Communications 178 (2008)
  • IDSCAN: LVL2 Tracking for the ATLAS Trigger (pdf) Janice Drohan UCL 1st Year Transfer Report June 20, 2003
  • Fast Tracking for the ATLAS LVL2 Trigger (pdf) J. T. M. Baines, H. Drevermann, D. Emeliyanov, N. Konstantinidis, F. Parodi, C. Schiavi, M. Sutton, Computing in High Energy Physics and Nuclear Physics 2004, Interlaken, Switzerland, 27 Sep - 1 Oct 2004, pp.246
  • HLT Track Reconstruction Performance (pdf) S. Ask, J.T. Baines, A. Coccaro, D. Emeliyanov, I. Grabowska-Bold, J. Kirk, N. Konstantinidis, J. Masik, E. Ozcan, F. Parodi, C. Schiavi, S. Sivoklokov, M. Sutton Detectors and Experimental Techniques, 17 Mar 2009. - 19 p.
  • A Probabilistic Data Association Filter for fast tracking in the ATLAS Transition Radiation Tracker (pdf) D. Emeliyanov Nucl. Instrum. Meth. A566 (2006) 50-53.

Articles and Preprints

  • ATLAS Technical Design Report: High-Level Trigger, Data Acquisition and Controls (pdf) ATLAS HLT/DAQ/DCS Group ATLAS-TDR-016 2 October, 2003
  • Commissioning the ATLAS Inner Detector Trigger (pdf) Mark Sutton on behalf of the ATLAS Collaboration ATL-DAQ-PROC-2009-015
  • Overview of the High-Level Trigger Electron and Photon Selection for the ATLAS Experiment at the LHC (pdf) IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 5, OCTOBER 2006
  • Fast tracking in hadron collider experiments (pdf) Nikos Konstantinidis and Hans Drevermann
  • Vertexing strategy and algorithms at ATLAS (pdf) F.Parodi, Nuclear Instruments and Methods in Physics Research A 560 (2006)
  • ATLAS L1 track trigger for super-LHC (pdf) Nikos Konstantinidis ATL-UPGRADE-PROC-2010-003
  • ATLAS Track Trigger for SLHC - Ideas & Plans (pdf) Nikos Konstantinidis ATL-UPGRADE-SLIDE-2009-243

CERN Articles and Preprints

Other Trigger Links

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf ATL-PHYS-PUB-2009-026.pdf r1 manage 530.0 K 2010-09-28 - 18:19 AndrewWashbrook  
PDFpdf ATL-UPGRADE-PROC-2010-003.pdf r1 manage 304.4 K 2010-09-28 - 18:21 AndrewWashbrook  
PDFpdf ATL-UPGRADE-SLIDE-2009-243.pdf r1 manage 1213.7 K 2010-09-28 - 18:22 AndrewWashbrook  
PDFpdf ATLAS-Upgrade-GPGPU.pdf r1 manage 3985.8 K 2009-12-10 - 19:40 AndrewWashbrook GPGPU and HEP
PDFpdf ATLAS-zfinder.pdf r1 manage 123.6 K 2010-09-28 - 18:17 AndrewWashbrook  
PDFpdf AlanRichardsonDissertation.pdf r1 manage 581.4 K 2009-12-10 - 19:52 AndrewWashbrook Alan Richardson
PDFpdf Christopher0963432_REP_Submission.pdf r1 manage 340.5 K 2010-04-28 - 09:49 AndrewWashbrook  
Unknown file formatgz CudaFitter.tar.gz r1 manage 828.8 K 2011-01-28 - 12:50 AndrewWashbrook Dmitry CUDA code
PDFpdf GPU_Tracking-Emeliyanov.pdf r1 manage 1215.9 K 2011-01-27 - 18:24 AndrewWashbrook  
PDFpdf HLToverview.pdf r1 manage 156.8 K 2010-09-28 - 18:20 AndrewWashbrook  
PDFpdf IDSCAN-uclreport.pdf r1 manage 729.8 K 2010-09-28 - 18:18 AndrewWashbrook  
PDFpdf IRP.pdf r1 manage 447.7 K 2010-04-28 - 09:50 AndrewWashbrook  
PDFpdf Nvidia-GTC2010.pdf r1 manage 8604.7 K 2011-01-27 - 18:03 AndrewWashbrook NVIDIA GTC 2010 Presentation
PDFpdf Submission.pdf r1 manage 577.2 K 2010-09-28 - 18:14 AndrewWashbrook Chris Project
PDFpdf TDR-2up.pdf r1 manage 3463.6 K 2010-09-28 - 18:19 AndrewWashbrook  
Unknown file formatgz ZFinder-GPU.tar.gz r1 manage 1359.0 K 2011-01-27 - 18:04 AndrewWashbrook Z-finder GPU code
PDFpdf ajw-GPUAtlasUpgrade.pdf r1 manage 4019.2 K 2011-01-27 - 18:03 AndrewWashbrook CHEP 2010 Presentation
PDFpdf chris-presentation.pdf r1 manage 1305.9 K 2010-04-28 - 09:50 AndrewWashbrook  
PDFpdf commissioning-ATLAS-IDtrigger.pdf r1 manage 842.7 K 2010-09-28 - 18:20 AndrewWashbrook  
Unknown file formatgz cudaKalmanFitter.tar.gz r1 manage 13682.7 K 2011-01-27 - 18:04 AndrewWashbrook Kalman Fitter Code
PDFpdf infthesis.pdf r1 manage 1530.4 K 2010-09-28 - 18:12 AndrewWashbrook Maria Project
PDFpdf lazzaroCCR09.pdf r1 manage 2472.2 K 2009-12-10 - 20:07 AndrewWashbrook Lazzaro
PDFpdf p246.pdf r1 manage 424.3 K 2010-09-28 - 18:18 AndrewWashbrook  
PDFpdf probabilisticDataAssociationFilter.pdf r1 manage 220.8 K 2010-09-28 - 18:19 AndrewWashbrook  
PDFpdf simdkalman.pdf r1 manage 1223.5 K 2010-09-28 - 18:18 AndrewWashbrook  
PDFpdf trackingatL2-trigger.pdf r1 manage 414.0 K 2010-09-28 - 18:18 AndrewWashbrook  
PDFpdf ucl-idscan-hitfilter.pdf r1 manage 162.6 K 2010-09-28 - 18:21 AndrewWashbrook  
PDFpdf vertexstrategy.pdf r1 manage 131.8 K 2010-09-28 - 18:21 AndrewWashbrook  
Edit | Attach | Watch | Print version | History: r23 | r21 < r20 < r19 < r18 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r19 - 2011-01-28 - AndrewWashbrook
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback