ATLAS Analytics

Overview

The ATLAS Analytics effort is focused on creating systems which provide ATLAS Distributed Computing (ADC) with new capabilities for understanding distributed systems and overall operational performance. These capabilities include:

• Correlate information from multiple systems (PanDA, Rucio, FTS, Dashboards, Tier0, PilotFactory, ...)
• Predictive Analytics: Execute arbitrary data mining or machine learning algorithms over raw and aggregated data
• Ability to host new third party analytics services on a scalable compute platform
• Satisfy variety of use cases for different user roles for ad-hoc analytics
• Provide an open platform with documented collections and tools to broaden participation in ADC Analytics.

Overview New

• Correlate information from multiple systems (PanDA, Rucio, FTS, Dashboards, Tier0, PilotFactory, ...)
• Predictive Analytics: Execute arbitrary data mining or machine learning algorithms over raw and aggregated data
• Ability to host new third party analytics services on a scalable compute platform
• Satisfy variety of use cases for different user roles for ad-hoc analytics
• Provide an open platform with documented collections and tools to broaden participation in ADC Analytics.
• Mange the change process for log and monitoring data so that existing tool chains are not disrupted or can plan adaptation

Organisation

• Meeting: ADC Analytics Weekly (Indico), usually Wednesdays 16:00-18:00 CEST (40-R-402) - on a biweekly basis: one week is dedicated to Monitoring, the following week to Analytics
• Vidyo: ADC_Analytics
• General purpose e-group: atlas-adc-data-analytics
• Machine learning e-group: atlas-adc-machine-learning
• Ticketing: ATLASMINER JIRA
• Code repository: Github

Organisation NEW

• Meeting: ADC Analytics Coordination (Indico), usually Friday 15:00-16:00 CEST - on a biweekly basis: one week is dedicated to Monitoring, the following week to Analytics Coordination
• ZOOM, Passcode 57614002
• General purpose e-group: atlas-adc-data-analytics
• Machine learning e-group: atlas-adc-machine-learning
• Ticketing: ATLASMINER JIRA
• Code repository: Github

Clusters and Resources

• analytix cluster at CERN, access to hadoop: In order to get the access you have to be in ai-hadoop-users e-group which is managed by CERN-IT. To get access you need to open a Snow ticket to Hadoop Service, providing your lxplus user name.
• Elasticsearch at UChicago. Kibana can be accessed at https://atlas-kibana.mwt2.org:5601 . To get access, please send a preferred user name to Ilija Vukotic. A subset of Kibana spaces has open access and it is accessed at https://atlas-kibana.mwt2.org.

Data collection

• Sources
  • PanDA
  • PandaService
  • Rucio
  • xAOD monitoring
  • XrootD & FTS
  • PilotFactory
  • Data Knowledge Catalog
• Sinks
  • Hadoop
  • ElasticSearch

Data visualization

• UChicago Kibana
• CERN ITES Kibana
• UC ES visualizations - google doc with list of links and explanations of all the kibana dashboards and Jupyter notebooks at UC.

Data analysis

• Go here to see all notebooks of the ATLASMINER JIRA tickets
  • ADC Analytics Notebooks: http://uct3-lx2.mwt2.org:9999/tree/ADC%20Analytics
• Other available general purpose Jupyter servers
  • High-Performance Jupyter server http://uct3-lx2.mwt2.org:9999/ (master)
  • GPU-enabled Jupyter server https://shodan.cern.ch:7643/notebooks/ (playground)
  • SWAN Jupyter https://swan002.cern.ch/ (in beta, will become next master)

Dedicated projects

• Data Management Metrics
• Network Analytics Service
• Group Space Overview
• Reports for Scrutiny Group
• Distributed Analysis Analytics
• Data Knowledge Catalog
• Operational Analytics
• Dashboards for Sites: General, specific for LRZ, Storage element overview.

Tutorials and How-To's

• Getting the data into and out of Hadoop
• Correctly indexing data in ES
• ElasticSearch access using API example
• Recording of the Vidyo tutorial part 1, sqoop, flume, pig, ES indexing , ES+kibana.
• How to develop analytics notebooks with Jupyter

Related efforts

• WLCG Machine Learning Demonstrator Indico
• IT Analytics Working Group Indico, TWiki
• WLCG Analytics Platform TWiki

Associated efforts and contacts NEW

• Data popularity - Maria Grigoryeva <maria.grigorieva@cern.ch>
• Data placement - Kaushik De <kaushik@uta.edu>
• PanDA task execution time studies - FaHui Lin <fahui.lin@cern.ch>
• Rucio (traces, dumps, reporting) - Mario Lassnig <Mario.Lassnig@cern.ch>
• Frontier analytics - Nurcan Ozturk <Nurcan.Ozturk@cern.ch>
• Logs collection from ADC services - Aleksandr Alekseev <aleksandr.alekseev@cern.ch> (PanDA, JEDI, iDDS, Prodsys), Mario Lassnig <Mario.Lassnig@cern.ch> (Rucio)
• Operational intelligence - Alessandro Di Girolamo <Alessandro.Di.Girolamo@cern.ch>
• ADC ops (expert users creating their own dashboards) - Ivan Glushkov <Ivan.Glushkov@cern.ch>
• ElasticSearch University Chicago - Ilija Vukotic <ivukotic@cern.ch>
• Network projects: Shawn McKee <smckee@umich.edu>
  • Petya
  • ALTO
  • SENSE

Attic

Historical information goes into the ATLASAnalyticsAttic.

Major updates:
-- IlijaVukotic - 2022-11-30
-- MarioLassnig - 2016-06-08

Responsible: IlijaVukotic, MarioLassnig
Last reviewed by: Never reviewed