ATLAS Analytics

Overview

The ATLAS Analytics effort is focused on creating systems which provide ATLAS Distributed Computing (ADC) with new capabilities for understanding distributed systems and overall operational performance. These capabilities include:

• Correlate information from multiple systems (PanDA, Rucio, FTS, Dashboards, Tier0, PilotFactory, ...)
• Predictive Analytics: Execute arbitrary data mining or machine learning algorithms over raw and aggregated data
• Ability to host new third party analytics services on a scalable compute platform
• Satisfy variety of use cases for different user roles for ad-hoc analytics
• Provide an open platform with documented collections and tools to broaden participation in ADC Analytics.

Organisation

• Meeting: ADC Analytics Weekly (Indico), usually Wednesdays 16:00-18:00 CEST (40-R-402) - on a biweekly basis: one week is dedicated to Monitoring, the following week to Analytics
• Vidyo: ADC_Analytics
• General purpose e-group: atlas-adc-data-analytics
• Machine learning e-group: atlas-adc-machine-learning
• Ticketing: ATLASMINER JIRA
• Code repository: Github

Clusters and Resources

• analytix cluster at CERN, access to hadoop: In order to get the access you have to be in ai-hadoop-users e-group which is managed by CERN-IT. To get access you need to open a Snow ticket to Hadoop Service, providing your lxplus user name.
• Elasticsearch at UChicago. Kibana can be accessed at https://atlas-kibana.mwt2.org:5601 . To get access, please send a preferred user name to Ilija Vukotic. A subset of Kibana spaces has open access and it is accessed at https://atlas-kibana.mwt2.org.

Data collection

• Sources
  • PanDA
  • PandaService
  • Rucio
  • xAOD monitoring
  • XrootD & FTS
  • PilotFactory
  • Data Knowledge Catalog
• Sinks
  • Hadoop
  • ElasticSearch

Data visualization

• UChicago Kibana
• CERN ITES Kibana
• UC ES visualizations - google doc with list of links and explanations of all the kibana dashboards and Jupyter notebooks at UC.

Data analysis

• Go here to see all notebooks of the ATLASMINER JIRA tickets
  • ADC Analytics Notebooks: http://uct3-lx2.mwt2.org:9999/tree/ADC%20Analytics
• Other available general purpose Jupyter servers
  • High-Performance Jupyter server http://uct3-lx2.mwt2.org:9999/ (master)
  • GPU-enabled Jupyter server https://shodan.cern.ch:7643/notebooks/ (playground)
  • SWAN Jupyter https://swan002.cern.ch/ (in beta, will become next master)

Completed notebooks

• For a given RSE, show the size vs. age distribution Link

Dedicated projects

• Data Management Metrics
• Network Analytics Service
• Group Space Overview
• Reports for Scrutiny Group
• Distributed Analysis Analytics
• Data Knowledge Catalog
• Dashboards for Sites: General, specific for LRZ, Storage element overview.

Tutorials and How-To's

• Getting the data into and out of Hadoop
• Correctly indexing data in ES
• ElasticSearch access using API example
• Recording of the Vidyo tutorial part 1, sqoop, flume, pig, ES indexing , ES+kibana.
• How to develop analytics notebooks with Jupyter

Related efforts

• WLCG Machine Learning Demonstrator Indico
• IT Analytics Working Group Indico, TWiki
• WLCG Analytics Platform TWiki

Attic

Historical information goes into the ATLASAnalyticsAttic.

Major updates:
-- IlijaVukotic - 2014-11-11
-- MarioLassnig - 2016-06-08

Responsible: IlijaVukotic, MarioLassnig
Last reviewed by: Never reviewed