Project finished!

This project finished, and it was closed on the 5th of December 2014!

Thank you very much for your participation

Goal of the project

The goal of the project is the consolidation of the WLCG monitoring, including a critical analysis of what is monitored, the technology used and the deployment and support model. This should allow to:

• reduce the complexity of the system
• ensure simplified and more effective operations, support and service management
• encourage an efficient deployment strategy, with a common development process
• unify, where possible, the implementation of the monitoring components

The main objective of the project is to get to the point where the effort required for WLCG monitoring can be reduced to half of it's current level.

Tentative timetable

The first stage of the project (3 months) should include:

• Review of the current systems and metrics used for WLCG monitoring
• Collecting and summarizing requirements from the LHC computing community to understand the changed needs in monitoring.
• Propose a revised strategy and architecture for WLCG monitoring
• Where possible, suggest implementation technologies and approaches for the transitions
• Highlight areas that are expected to be problematic

Second stage: implementation of the new monitoring framework and components needed for the transition to the new toolchain.

A first prototype should be ready by the end of 2013. All monitoring should have transitioned to the new approach by summer 2014. There is a dedicated jira tracker to follow this task

Member list

Mailing list: wlcg-mon-consolidation

• Pablo -- project leader
• Edward
• Julia
• Lionel
• Luca
• Marian
• AI monitoring representative (Pedro)
• Experiment representatives:
  • ALICE (Costin, Latchezar, Maarten)
  • ATLAS (Alessandra (with Simone and Alessandro as consultants) )
  • CMS (Andrea, Nicolo, Stefano)
  • LHCb (Stefan)
• Operations representative (Maria, Pepe)

Some tasks of the project ( not a complete list)

1. Requirements:
   1. Understanding requirements of the experiments regarding remote testing of the distributed sites and services
   2. Review with the experiments existing Dashboard applications, define priorities and agree on the further development and support model for applications which would require development effort for customization
2. Understanding needs of the sites, famous question 'I like to see at a single display how my site is working for all LHC VOs'
3. Review of the test submission part. Should we continue to have several ways for test submission: one for stress testing, another one for regular tests (Hammer Cloud/Nagios/JobPilot )?
4. Overall architecture (strongly depends on previous point)
5. Common framework
   1. Common implementation of the messaging interfaces
6. Single web UI to WLCG monitoring info (not rewriting from scratch, based on myWLCG, and improving/extending functionality and adapting implementation to the design principles of the new framework).

Various technical topics

• NoSQLStorageResearch

Architecture reviews

• TransportReview
• StorageAggregationReview
• VisualisationReview
• RecurrentTaskReview
• DeploymentReview
• DocumentationReview

Work on simplified version of NCG

Status

• Applications as of 13-Jun-13
• Architecture as of 13-Jun-13
• Technologies as of 13-Jun-13
• Applications used of 5-Jul-13
• SAM usage in detail

Prototype status

• 19-Nov-13 First Evaluation of the prototype
• ValidationStatus
• Feedback on SAM3

Meetings

• 21-Jun-13 Kick off meeting
• 5-Jul-13 Requirements from Experiments
• 18-Jul-13 Input from Operations and Probes
• 2-Aug-13 Areas of an application
• 16-Aug-13 Proposal for the architecture
• 30-Aug-13 Storage review
• 12-Sep-13 Probe development
• 26-Sep-13 ATLAS probes
• 4-Oct-13 Deployment
• 11-Oct-13 GUI
• 25-Oct-13 Report
• 8-Nov-13 Demo of the prototype
• 22-Nov-13 Feedback on the prototype
• 6-Dec-13 SUM-like functionality
• 17-Dec-13 Condor-G probes
• 31-Jan-14 REBUS and VOFeed
• 14-Feb-14 SAM transition and SAM3 UI
• 28-Feb-14 Hammercloud Functional tests
• 14-Mar-14 Downtime
• 4-Apr-14 Test submission framework
• 11-Apr-14 Site Nagios and Availability comparison
• 25-Apr-14 PostgreSQL and SAM3 UI
• 9-May-14 MVC and Caching
• 23-May-14 Reports
• 6-Jun-14 CVMFS monitoring
• 20-Jun-14 MyWLCG trends
• 4-Jul-14 Availability recalculation
• 25-Ju6-14 Data transfers
• 15-Aug-14 Messaging
• 29-Aug-14 SAM3 feedback
• 19-Sep-14 SAM3 feedback
• 26-Sep-14 SAM3 feedback
• 10-Oct-14 WLCG transfers
• 31-Oct-14 Nagios configuration
• 14-Nov-14 SAM3 in production
• 5-Dec-14 Wrap up of WLCG Monitoring consolidation