ATLAS Analytics

Overview

The ATLAS Analytics effort is focused on creating systems which provide ATLAS Distributed Computing (ADC) with new capabilities for understanding distributed systems and overall operational performance. These capabilities include:

• Correlate information from multiple systems (PanDA, Rucio, FTS, Dashboards, Tier0, PilotFactory, ...)
• Predictive Analytics: Execute arbitrary data mining or machine learning algorithms over raw and aggregated data
• Ability to host new third party analytics services on a scalable compute platform
• Satisfy variety of use cases for different user roles for ad-hoc analytics
• Provide an open platform with documented collections and tools to broaden participation in ADC Analytics.

Organisation

• Meeting: ADC Analytics Weekly (Indico), usually Wednesdays 17:00-18:00 CEST (40-R-402)
• Vidyo: ADC_Analytics
• General purpose e-group: atlas-adc-data-analytics
• Machine learning e-group: atlas-adc-machine-learning
• Ticketing: ATLASMINER JIRA
• Code repository: Github

Data collection

• Sources
  • PanDA
  • PandaService
  • Rucio
  • xAOD monitoring
  • XrootD & FTS
  • PilotFactory
  • Data Knowledge Catalog
• Sinks
  • Hadoop
  • ElasticSearch

Data visualization

• UChicago Kibana (current master, to be deprecated in favor of ITES)
• CERN ITES Kibana (in beta, will become next master)
• ADC Kibana (deprecated, don't use)
• UC ES visualizations - google doc with list of links and explanations of all the kibana dashboards and Jupyter notebooks at UC.

Data analysis

• Go here to see all notebooks of the ATLASMINER JIRA tickets
  • ADC Analytics Notebooks: http://uct3-lx2.mwt2.org:9999/tree/ADC%20Analytics
• Other available general purpose Jupyter servers
  • High-Performance Jupyter server http://uct3-lx2.mwt2.org:9999/ (master)
  • GPU-enabled Jupyter server https://shodan.cern.ch:7643/notebooks/ (playground)
  • SWAN Jupyter https://swan002.cern.ch/ (in beta, will become next master)

Completed notebooks

• For a given RSE, show the size vs. age distribution Link

Dedicated projects

• Data Management Metrics
• Network Analytics Service
• Group Space Overview
• Reports for Scrutiny Group
• Distributed Analysis Analytics
• Data Knowledge Catalog
• Dashboards for Sites: General, specific for LRZ, Storage element overview.

Tutorials and How-To's

• Getting the data into and out of Hadoop
• Correctly indexing data in ES
• ElasticSearch access using API example
• Recording of the Vidyo tutorial part 1, sqoop, flume, pig, ES indexing , ES+kibana.
• How to develop analytics notebooks with Jupyter

Related efforts

• WLCG Machine Learning Demonstrator Indico
• IT Analytics Working Group Indico, TWiki
• WLCG Analytics Platform TWiki

Attic

Historical information goes into the ATLASAnalyticsAttic.

Major updates:
-- IlijaVukotic - 2014-11-11
-- MarioLassnig - 2016-06-08

Responsible: IlijaVukotic, MarioLassnig
Last reviewed by: Never reviewed