ATLAS DDM Dashboard

Architecture

User interface

Collectors

  • Architecture
  • RPM packages
  • Configuration files
  • Log files

Data flows

Database

Accounts

Integration
  • Owner: atlas_dashboard_dm @ INT11R
  • Reader: atlas_dashboard_dm_reader @ INT11R
  • Writer: atlas_dashboard_dm_writer @ INT11R
Production
  • Owner: atlas_dashboard_ddm @ atlas_dashboard_dm
  • Reader: atlas_dashboard_ddm_reader @ atlas_dashboard_dm
  • Writer: atlas_dashboard_ddm_writer @ atlas_dashboard_dm
Schema atlas_dashboard_dm is a synnonym of LCGR.

Infrastructure

Production

Machines:
  • dashb-ai-641 - Web UI, collectors, database crons
  • dashb-ai-642 - Web UI, collectors

Integration

Machines:
  • dashb-rucio-01 - Web UI, collectors, database crons

New RPM release

  • Change code, commit on trunk...
  • Go to the local directory of the cosmic module to build
  • Make a release
     python setup.py release [release_type] 
    Release type could be:
    • -p : patch
    • -m : minor version
    • -M : major version
    • -c : release candidate
    • without type specified: stable release
      • Before making a stable release it might be required to build the package with one of keys mentioned above (please consult python setup.py release --help for details).

  • In case of issues:
    git commit --amend
    • then remove inserted line on RELEASE_NOTES and change back release number in setup.cfg
  • Promote package to QA

RPM update for message consumer

  • Stop the services
          su - dboard
          /usr/bin/python /usr/bin/simplevisor --conf /opt/dashboard/etc/dashboard-simplevisor/consumer-simplevisor.cfg stop
          dashb-agent-stop ddm.collector.dirq2db
          exit
        
    Ensure that collector is actually stopped:
          ps aux | grep dirq2db
        
  • Clean yum cache
    yum clean all
  • Update RPM package
    • On production machine
       yum update dashboard-service-collector-ddm 
    • On integration machine
       yum update dashboard-service-collector-ddm --enablerepo=ai6-testing 
  • Restart Puppet agent (it will finish the reconfiguration)
     puppet agent -t 
    • /opt/dashboard/doc/config/service-config/ddm.collector.dirq2db.xml_full is generated by Puppet from /opt/dashboard/doc/config/service-config/ddm.collector.dirq2db.xml
  • Start the services
          su - dboard
          /usr/bin/python /usr/bin/simplevisor --conf /opt/dashboard/etc/dashboard-simplevisor/consumer-simplevisor.cfg --daemon start
          dashb-agent-stop ddm.collector.dirq2db
          dashb-agent-start ddm.collector.dirq2db
          exit
        
  • Check the following log file
          /opt/dashboard/var/log/dashb-ddm.collector.dirq2db.log
        
    There should appear messages from all the message brokers (see corresponding section below).

Versions and features

Version Release date Main features and changes Related JIRA tickets RPMs versions
2.5 20 April 2015 Upload/download activity DASHB-2659 DASHB-2476 DDM-47  
Proper tagging of Staging activity ADCMONITOR-394 DDM-54
Introducing 'Deletion' activity instead of 'n/a' DASHB-2735
Recovering API used by DDM Sonar DDM-42
Fixing broken FTS links ADCMONITOR-383
2.5.1 5 May 2015 Activities merging DASHB-2735 ADCMONITOR-388 DDM-59 dashboard-dao-oracle-ddm-0.0.3-1
dashboard-service-collector-ddm-0.2.5-1
dashboard-service-monitor-ddm-0.0.0-1
dashboard-web-ddm-0.1.2-1
Activities list reorganization (groups for debug and not seen last month) DASHB-2735 ADCMONITOR-388
Bin size to be shown in Y-axis label for some kind of plots ADCMONITOR-68 DASHB-1899
Introduced 8 hours bins for plots DASHB-2169
Other minor fixes on Web UI DASHB-2220 DDM-65
2.5.2 20 August 2015 Support for proper distinguishing of staging DDM-83 DDM-94 dashboard-dao-oracle-ddm-0.0.7
dashboard-service-collector-ddm-0.2.9
dashboard-service-monitor-ddm-0.0.2
dashboard-web-ddm-0.1.6
dashboard-web-0.19.0
Activities menu improvements DDM-72 DDM-80
Upload/download activities splitted to six proper ones DDM-75
Source/destination urls and error details for upload/download activity DDM-67
Dataset name to transfer information and links to Rucio UI (machinery prepared, already collecting dataset for Pilot events) DDM-68
Bugfixes DDM-73 DDM-71 DASHB-2856 DASHB-2867 DASHB-2879
Improved stability of the service DDM-81 DDM-82 DDM-88 DDM-93 DDM-91 DDM-98 DASHB-2883 DASHB-2475
2.5.3 (planned)   Monitoring of deletion    
Search on datasets and files  
Dataset name from Rucio events - needed action to have name in events/from FTS DDM-68
Datasets monitoring: cross-links to the Rucio UI DDM-64
Support for the new time field in Rucio events DDM-97
Downloadable HighCharts plots in ATLAS DDM dashboard DASHB-2860

DDM Planning meeting

Minutes 22nd May 2015

DDM Dashboard:

  • 2.5.2 is the target for the next development Sprint, as by the table in https://twiki.cern.ch/twiki/bin/view/ArdaGrid/DDMDashboard#Versions_and_features
  • Dataset information will be available from pilots data, but postponed for FTS, as by the next point.
  • Discussion is needed with Rucio dev to understand why is not possible to forward dataset information directly from Rucio (which seems the preferred option from a DDM Dashboard perspective) rather than forward it to FTS as metadata.
  • Alessandra should ask for a slot in Rucio meeting on the first week of June to discuss this (and also the point on providing data for scrutiny every Monday, as below)
  • as future feature, Alessandro mentioned the possibility to show "going to be deleted" datasets somehow superimposed to current data occupancy plot. This requires investigation on how to get the dataset on deletion list and how to build visualization.

Scrutiny activity:

  • Sergey will work on automatize PIG jobs in the upcoming weeks. First result will be to make csv and static plots available every Monday. More advanced visualization may follow.
  • would be good to clarify the expected worflow from Rucio on how/when they commit to make the dataset available. Point to be raised at the meeting.
  • Investigation is needed to understand which information on tape access/usage is available from the current dataset structure. To be followed up by Sergey, with the help of Alessandro to identify specific information to retrieve.

Note:

  • Include Andrei in the loop for next meeting.

Minutes 7th April 2015

On the DDM Dashboard:

* DDM v.2.5 is ready: - Changelog: -- Upload/download activity -- Bug fixes (staging fix, deletion fix, minimize N/A, more: goo.gl/sH1VL8)

- By default, the upload/download activity will not be "ticked" in the view - Sergey aim at deploying to production by the end of this week

* Next DDM minor release v.2.5.1 in ~ 1 month, mainly bug fixes

* Next DDM feature release v.2.x : dataset events with Rucio - Sergey will interact with Rucio dev., Ale. etc. to define the requirements for dataset visualization in dashboard (search box, etc.) - Sergey will raise the topics at next Rucio-dev Thursday meeting - At the next meeting (~ mid May) Sergey will present an implementation plan following the requirements

On the Popularty Dashboard:

- Sergey's investigation shows the tool has still some potential - The goal is to cut the number plots, to have only few of known utility (derived from scrutiny work) which are populated automatically by new workflow (HDFS/PIG,etc) and displayed on the dashboard. - Next week Sergey, Ale and Thomas Kuba will meet and define (and share) the list of plots

On DQ2 old DDM dashboard:

- We can stop collectors, keep running 1 UI only, log accesses. - At the next meeting we can discuss the shutdown

Minutes 25th February 2015

Next DDM dashboard release (v.2.5) by 31 March with : - upload/download activity - several bug fixes from the shared list (if you have any preference let us know or we'll go in order)

Next priority for DDM dashboard (both need further design/discussion with Rucio guys) - deletion (Alessandra will summarize the current situation in a Jira ticket) - dataset events

On Rucio team interaction: - Sergey will act proactively - Mails and face to face discussion are both needed, but outcome should be summarized on Jira to be easily trucked.

Plan for analytics/popularity: - Going on with the effort of making scrutiny plots easily reproducible - Sergey will sit down with Jarka in the next days to study the current popularity dashboard. The outcome will be a report to understand: -- the list of features the dashboard provides -- the infrastructure is based on (DB tables, etc.) - the report will be used to discuss and decide (at the next meeting) if the tool is still interesting, which features are actually needed and if and how data produced by the new scrutiny code (pig jobs) can be fit in that architecture.

External services involved

Other documents on DDM Dashboard

-- LucaMagnoni - 02 Oct 2014

Edit | Attach | Watch | Print version | History: r27 < r26 < r25 < r24 < r23 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r27 - 2016-04-11 - SergeyBelov
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback