DDMDashboard < ArdaGrid

TWiki>

ArdaGrid Web>WebPreferences>DDMDashboard (2016-04-11, SergeyBelov)

EditAttachPDF

ATLAS DDM Dashboard

Architecture

User interface

Collectors

Architecture
RPM packages
Configuration files
Log files

Data flows

Database

Accounts

Integration

Owner: atlas_dashboard_dm @ INT11R
Reader: atlas_dashboard_dm_reader @ INT11R
Writer: atlas_dashboard_dm_writer @ INT11R

Production

Owner: atlas_dashboard_ddm @ atlas_dashboard_dm
Reader: atlas_dashboard_ddm_reader @ atlas_dashboard_dm
Writer: atlas_dashboard_ddm_writer @ atlas_dashboard_dm

Schema atlas_dashboard_dm is a synnonym of LCGR.

Infrastructure

Production

Machines:

dashb-ai-641 - Web UI, collectors, database crons
dashb-ai-642 - Web UI, collectors

Integration

Machines:

dashb-rucio-01 - Web UI, collectors, database crons

New RPM release

Change code, commit on trunk...
Go to the local directory of the cosmic module to build
Make a release
```
 python setup.py release [release_type] 
```
Release type could be:
- -p : patch
- -m : minor version
- -M : major version
- -c : release candidate
- without type specified: stable release
  - Before making a stable release it might be required to build the package with one of keys mentioned above (please consult python setup.py release --help for details).

In case of issues:
```
git commit --amend
```
- then remove inserted line on RELEASE_NOTES and change back release number in setup.cfg
Promote package to QA

RPM update for message consumer

Stop the services

      su - dboard
      /usr/bin/python /usr/bin/simplevisor --conf /opt/dashboard/etc/dashboard-simplevisor/consumer-simplevisor.cfg stop
      dashb-agent-stop ddm.collector.dirq2db
      exit

Ensure that collector is actually stopped:

      ps aux | grep dirq2db

Clean yum cache
```
yum clean all
```

Update RPM package

On production machine

 yum update dashboard-service-collector-ddm

On integration machine

 yum update dashboard-service-collector-ddm --enablerepo=ai6-testing

Restart Puppet agent (it will finish the reconfiguration)
```
 puppet agent -t 
```
- /opt/dashboard/doc/config/service-config/ddm.collector.dirq2db.xml_full is generated by Puppet from /opt/dashboard/doc/config/service-config/ddm.collector.dirq2db.xml

Start the services

      su - dboard
      /usr/bin/python /usr/bin/simplevisor --conf /opt/dashboard/etc/dashboard-simplevisor/consumer-simplevisor.cfg --daemon start
      dashb-agent-stop ddm.collector.dirq2db
      dashb-agent-start ddm.collector.dirq2db
      exit

Check the following log file
```
      /opt/dashboard/var/log/dashb-ddm.collector.dirq2db.log
    
```
There should appear messages from all the message brokers (see corresponding section below).

Versions and features

Version	Release date	Main features and changes	Related JIRA tickets	RPMs versions
2.5	20 April 2015	Upload/download activity	DASHB-2659 DASHB-2476 DDM-47
		Proper tagging of Staging activity	ADCMONITOR-394 DDM-54
		Introducing 'Deletion' activity instead of 'n/a'	DASHB-2735
		Recovering API used by DDM Sonar	DDM-42
		Fixing broken FTS links	ADCMONITOR-383
2.5.1	5 May 2015	Activities merging	DASHB-2735 ADCMONITOR-388 DDM-59	dashboard-dao-oracle-ddm-0.0.3-1 dashboard-service-collector-ddm-0.2.5-1 dashboard-service-monitor-ddm-0.0.0-1 dashboard-web-ddm-0.1.2-1
		Activities list reorganization (groups for debug and not seen last month)	DASHB-2735 ADCMONITOR-388
		Bin size to be shown in Y-axis label for some kind of plots	ADCMONITOR-68 DASHB-1899
		Introduced 8 hours bins for plots	DASHB-2169
		Other minor fixes on Web UI	DASHB-2220 DDM-65
2.5.2	20 August 2015	Support for proper distinguishing of staging	DDM-83 DDM-94	dashboard-dao-oracle-ddm-0.0.7 dashboard-service-collector-ddm-0.2.9 dashboard-service-monitor-ddm-0.0.2 dashboard-web-ddm-0.1.6 dashboard-web-0.19.0
		Activities menu improvements	DDM-72 DDM-80
		Upload/download activities splitted to six proper ones	DDM-75
		Source/destination urls and error details for upload/download activity	DDM-67
		Dataset name to transfer information and links to Rucio UI (machinery prepared, already collecting dataset for Pilot events)	DDM-68
		Bugfixes	DDM-73 DDM-71 DASHB-2856 DASHB-2867 DASHB-2879
		Improved stability of the service	DDM-81 DDM-82 DDM-88 DDM-93 DDM-91 DDM-98 DASHB-2883 DASHB-2475
2.5.3 (planned)		Monitoring of deletion
		Search on datasets and files
		Dataset name from Rucio events - needed action to have name in events/from FTS	DDM-68
		Datasets monitoring: cross-links to the Rucio UI	DDM-64
		Support for the new time field in Rucio events	DDM-97
		Downloadable HighCharts plots in ATLAS DDM dashboard	DASHB-2860

DDM Planning meeting

Minutes 22nd May 2015

DDM Dashboard:

2.5.2 is the target for the next development Sprint, as by the table in https://twiki.cern.ch/twiki/bin/view/ArdaGrid/DDMDashboard#Versions_and_features
Dataset information will be available from pilots data, but postponed for FTS, as by the next point.
Discussion is needed with Rucio dev to understand why is not possible to forward dataset information directly from Rucio (which seems the preferred option from a DDM Dashboard perspective) rather than forward it to FTS as metadata.
Alessandra should ask for a slot in Rucio meeting on the first week of June to discuss this (and also the point on providing data for scrutiny every Monday, as below)
as future feature, Alessandro mentioned the possibility to show "going to be deleted" datasets somehow superimposed to current data occupancy plot. This requires investigation on how to get the dataset on deletion list and how to build visualization.

Scrutiny activity:

Sergey will work on automatize PIG jobs in the upcoming weeks. First result will be to make csv and static plots available every Monday. More advanced visualization may follow.
would be good to clarify the expected worflow from Rucio on how/when they commit to make the dataset available. Point to be raised at the meeting.
Investigation is needed to understand which information on tape access/usage is available from the current dataset structure. To be followed up by Sergey, with the help of Alessandro to identify specific information to retrieve.

Note:

Include Andrei in the loop for next meeting.

Minutes 7th April 2015

On the DDM Dashboard:

* DDM v.2.5 is ready: - Changelog: -- Upload/download activity -- Bug fixes (staging fix, deletion fix, minimize N/A, more: goo.gl/sH1VL8)

- By default, the upload/download activity will not be "ticked" in the view - Sergey aim at deploying to production by the end of this week

* Next DDM minor release v.2.5.1 in ~ 1 month, mainly bug fixes

* Next DDM feature release v.2.x : dataset events with Rucio - Sergey will interact with Rucio dev., Ale. etc. to define the requirements for dataset visualization in dashboard (search box, etc.) - Sergey will raise the topics at next Rucio-dev Thursday meeting - At the next meeting (~ mid May) Sergey will present an implementation plan following the requirements

On the Popularty Dashboard:

- Sergey's investigation shows the tool has still some potential - The goal is to cut the number plots, to have only few of known utility (derived from scrutiny work) which are populated automatically by new workflow (HDFS/PIG,etc) and displayed on the dashboard. - Next week Sergey, Ale and Thomas Kuba will meet and define (and share) the list of plots

On DQ2 old DDM dashboard:

- We can stop collectors, keep running 1 UI only, log accesses. - At the next meeting we can discuss the shutdown

Minutes 25th February 2015

Next DDM dashboard release (v.2.5) by 31 March with : - upload/download activity - several bug fixes from the shared list (if you have any preference let us know or we'll go in order)

Next priority for DDM dashboard (both need further design/discussion with Rucio guys) - deletion (Alessandra will summarize the current situation in a Jira ticket) - dataset events

On Rucio team interaction: - Sergey will act proactively - Mails and face to face discussion are both needed, but outcome should be summarized on Jira to be easily trucked.

Plan for analytics/popularity: - Going on with the effort of making scrutiny plots easily reproducible - Sergey will sit down with Jarka in the next days to study the current popularity dashboard. The outcome will be a report to understand: -- the list of features the dashboard provides -- the infrastructure is based on (DB tables, etc.) - the report will be used to discuss and decide (at the next meeting) if the tool is still interesting, which features are actually needed and if and how data produced by the new scrutiny code (pig jobs) can be fit in that architecture.