ATLAS Analytics
Overview
The ATLAS Analytics effort is focused on creating systems which provide ATLAS Distributed Computing (ADC) with new capabilities for understanding distributed systems and overall operational performance. These capabilities include:
- Correlate information from multiple systems (PanDA, Rucio, FTS, Dashboards, Tier0, PilotFactory, ...)
- Predictive Analytics: Execute arbitrary data mining or machine learning algorithms over raw and aggregated data
- Ability to host new third party analytics services on a scalable compute platform
- Satisfy variety of use cases for different user roles for ad-hoc analytics
- Provide an open platform with documented collections and tools to broaden participation in ADC Analytics.
Overview New
- Correlate information from multiple systems (PanDA, Rucio, FTS, Dashboards, Tier0, PilotFactory, ...)
- Predictive Analytics: Execute arbitrary data mining or machine learning algorithms over raw and aggregated data
- Ability to host new third party analytics services on a scalable compute platform
- Satisfy variety of use cases for different user roles for ad-hoc analytics
- Provide an open platform with documented collections and tools to broaden participation in ADC Analytics.
- Mange the change process for log and monitoring data so that existing tool chains are not disrupted or can plan adaptation
Organisation
Organisation NEW
Clusters and Resources
- analytix cluster at CERN, access to hadoop: In order to get the access you have to be in ai-hadoop-users e-group which is managed by CERN-IT. To get access you need to open a Snow ticket to Hadoop Service, providing your lxplus user name.
- Elasticsearch at UChicago. Kibana can be accessed at https://atlas-kibana.mwt2.org:5601
. To get access, please send a preferred user name to Ilija Vukotic. A subset of Kibana spaces has open access and it is accessed at https://atlas-kibana.mwt2.org
.
Data collection
Data visualization
Data analysis
- Go here to see all notebooks of the ATLASMINER JIRA tickets
- Other available general purpose Jupyter servers
Dedicated projects
Tutorials and How-To's
Related efforts
- WLCG Machine Learning Demonstrator Indico
- IT Analytics Working Group Indico
, TWiki
- WLCG Analytics Platform TWiki
Associated efforts and contacts NEW
- Data popularity - Maria Grigoryeva <maria.grigorieva@cern.ch>
- Data placement - Kaushik De <kaushik@uta.edu>
- PanDA task execution time studies - FaHui Lin <fahui.lin@cern.ch>
- Rucio (traces, dumps, reporting) - Mario Lassnig <Mario.Lassnig@cern.ch>
- Frontier analytics - Nurcan Ozturk <Nurcan.Ozturk@cern.ch>
- Logs collection from ADC services - Aleksandr Alekseev <aleksandr.alekseev@cern.ch> (PanDA, JEDI, iDDS, Prodsys), Mario Lassnig <Mario.Lassnig@cern.ch> (Rucio)
- Operational intelligence - Alessandro Di Girolamo <Alessandro.Di.Girolamo@cern.ch>
- ADC ops (expert users creating their own dashboards) - Ivan Glushkov <Ivan.Glushkov@cern.ch>
- ElasticSearch University Chicago - Ilija Vukotic <ivukotic@cern.ch>
- Network projects: Shawn McKee <smckee@umich.edu>
Attic
Historical information goes into the
ATLASAnalyticsAttic.
Major updates:
--
IlijaVukotic - 2022-11-30
--
MarioLassnig - 2016-06-08
Responsible:
IlijaVukotic,
MarioLassnig
Last reviewed by:
Never reviewed