FTSDashboard < LCG

---+ FTS Dashboard

 %TOC%

---++ *Views and APIs*
   * Matrix
      * URL: http://dashb-fts-transfers.cern.ch/ui/#p.grouping=server
      * API: http://dashb-fts-transfers.cern.ch/dashboard/request.py/transfer-matrix
   * Error samples: 
      * URL: http://dashb-fts-transfers.cern.ch/ui/#m.content=%28efficiency,errors%29&p.grouping=server 
      * API http://dashb-fts-transfers.cern.ch/dashboard/request.py/error-samples
   * Transfers monitoring: 
      * URL: http://dashb-fts-transfers.cern.ch/ui/#p.grouping=server&tab=transfer_plots
      * API http://dashb-fts-transfers.cern.ch/dashboard/request.py/transfer-bins
   * History of transfers: 
      * URL: http://dashb-fts-transfers.cern.ch/ui/#p.grouping=server&tab=history_plots
      * API: http://dashb-fts-transfers.cern.ch/dashboard/request.py/transfer-history
   * Files status monitoring: 
      * URL: http://dashb-fts-transfers.cern.ch/ui/#p.grouping=server&tab=fts_jobs_plots
      * API: http://dashb-fts-transfers.cern.ch/dashboard/request.py/fts-job-bins
   * Active transfers: 
      * URL: http://dashb-fts-transfers.cern.ch/ui/#p.grouping=server&tab=latency_plots
      * API: http://dashb-fts-transfers.cern.ch/dashboard/request.py/active-transfers
   * Transfers ranking: 
      * URL: http://dashb-fts-transfers.cern.ch/ui/#p.grouping=server&tab=ranking_plots
      * API: http://dashb-fts-transfers.cern.ch/dashboard/request.py/transfer-ranking
   * Transfers map: 
      * URL: http://dashb-fts-transfers.cern.ch/ui/#p.grouping=server&tab=map
      * API: http://dashb-fts-transfers.cern.ch/dashboard/request.py/transfer-map

---++ *Infrustructure*
   * PROD
      * machines : dashb-ai-578.cern.ch / dashb-ai-579.cern.ch 
      * alias : dashb-fts-transfers.cern.ch 
      * broker : dashb-mb.cern.ch
      * broker topics : transfer.fts_monitoring_start, transfer.fts_monitoring_complete, transfer.fts_monitoring_state
      * broker queues : Consumer.dashb-fts.
      * broker auth : dboard certificate (/home/dboard/.globus/*)
      * database : LCGR (DSN = lcg_dashboard_tfr)
      * database account : lcg_dashboard_tfr<_r,_w>
      * database password : Stored in tbag (dashboard_dao_fts_prod_owner/dashboard_dao_fts_prod_reader/dashboard_dao_fts_prod_writer)
      * puppet hostgroup : [[https://judy.cern.ch/hostgroups/1511-dashboard-web_server-fts-production/edit][dashboard/web_server/fts/production]] 

   * QA
      * machines : dashb-ai-552.cern.ch 
      * alias : - 
      * broker : dashb-test-mb.cern.ch
      * broker topics : transfer.fts_monitoring_start, transfer.fts_monitoring_complete, transfer.fts_monitoring_state
      * broker queues : Consumer.dashb-fts.
      * broker auth : dboard certificate (/home/dboard/.globus/*)
      * database : INT6R 
      * database account : lcg_transfers_test<_r,_w> 
      * database password : Stored in tbag (dashboard_dao_fts_int_owner/dashboard_dao_fts_int_reader/dashboard_dao_fts_int_writer)
      * puppet hostgroup : [[https://judy.cern.ch/hostgroups/1512-dashboard-web_server-fts-integration/edit][dashboard/web_server/fts/integration]]

   * DEV (UI only/ES eval.)
      * machines : dashb-ai-657.cern.ch/dashb-ai-613.cern.ch. 
      * alias : dashb-fts-transfers-dev.cern.ch
      * database : INT6R 
      * database account : lcg_transfers_test<_r,_w> 
      * database password : Stored in tbag (dashboard_dao_fts_int_owner/dashboard_dao_fts_int_reader/dashboard_dao_fts_int_writer)
      * puppet hostgroup : [[https://judy.cern.ch/hostgroups/1512-dashboard-web_server-fts-integration/edit][dashboard/web_server/fts/integration]]

   * Config.
      * Puppet manifest: 
         * https://gitlab.cern.ch/ai/it-puppet-hostgroup-dashboard/blob/master/code/manifests/init.pp
         * https://gitlab.cern.ch/ai/it-puppet-hostgroup-dashboard/blob/qa/code/manifests/init.pp

      * Foreman params. and modules manifests:
         * dashboard_consumer: 
            * https://gitlab.cern.ch/ai/it-puppet-hostgroup-dashboard/blob/master/code/manifests/modules/create_simplevisor_consumer_config.pp
            * https://gitlab.cern.ch/ai/it-puppet-hostgroup-dashboard/blob/qa/code/manifests/modules/create_simplevisor_consumer_config.pp
         * dashboard_dao: 
            * https://gitlab.cern.ch/ai/it-puppet-hostgroup-dashboard/blob/master/code/manifests/modules/database.pp
            * https://gitlab.cern.ch/ai/it-puppet-hostgroup-dashboard/blob/qa/code/manifests/modules/database.pp
         * dashboard_web_load: 
            * https://gitlab.cern.ch/ai/it-puppet-hostgroup-dashboard/blob/master/code/manifests/modules/load.pp
            * https://gitlab.cern.ch/ai/it-puppet-hostgroup-dashboard/blob/qa/code/manifests/modules/load.pp

---++ *Support*
   * FTS Dashboard issues: dashboard-support@cern.ch, SNOW FE: WLCG Data Management Monitoring. 
   * FTS issues (service admins and devels.): fts3-steering@cern.ch,  SNOW FE: FTS Development.

---++ *Service Ops*
   * UI
      * start/stop: <verbatim>[root@dashb-ai-552 ~]# service httpd restart</verbatim>
      * log: /opt/dashboard/var/log/dashboard
   * Collectors
      * start/stop: <verbatim>[dboard@dashb-ai-552 ~]$ /opt/dashboard/bin/dashb-agent-restart fts.collectors</verbatim>
      * log: /opt/dashboard/var/log/dashb-fts.collectors.log
   * Services simplevisor (collectors)
      * start/stop: <verbatim>[dboard@dashb-ai-552 ~]$simplevisor --conf /opt/dashboard/etc/dashboard-simplevisor/service-simplevisor.cfg --daemon stop
[dboard@dashb-ai-552 ~]$simplevisor --conf /opt/dashboard/etc/dashboard-simplevisor/service-simplevisor.cfg --daemon start </verbatim>
      * log: /opt/dashboard/var/log/services-simplevisor.log
      * config.: /opt/dashboard/etc/dashboard-simplevisor/services-simplevisor.cfg
   * Consumer simplevisor (stompclt)
      * start/stop: <verbatim>[dboard@dashb-ai-552 ~]$simplevisor --conf /opt/dashboard/etc/dashboard-simplevisor/consumer-simplevisor.cfg --daemon stop
[dboard@dashb-ai-552 ~]$simplevisor --conf /opt/dashboard/etc/dashboard-simplevisor/consumer-simplevisor.cfg --daemon start </verbatim>
      * log: /opt/dashboard/var/log/consumer-simplevisor.log 
      * config.: /opt/dashboard/etc/dashboard-simplevisor/consumer-simplevisor.cfg
   * Database access
      * config.: /opt/dashboard/etc/dashboard-dao/dashboard-dao.cfg

---++ *Database*
   * RAW tables:
      * t_transfer_start/t_transfer_complete/t_fts_state
   * STATS tables:
      * t_error_summaries
         * View: Errors categorization  
      * t_file_stats_detailled
         * View: FTS 3 jobs plots
      * t_tfrs_stats
         * Views: Matrix, Transfer plots, Ranking plots
      * t_transfers 
         * View: Latency plots

---++ *Topology*
   * Lookups:
      1. hostname (of the storage) ---> GOCDB name: statically using the t_topology table
      1. GOCDB name ---> vo site name: using the cached file of the following IS url for each vo
         * CMS: http://dashb-cms-vo-feed.cern.ch/dashboard/request.py/cmssitemapbdii
         * ATLAS: http://atlas-agis-api.cern.ch/request/atp/xml/
         * LHCb: http://lhcb-web-dirac.cern.ch/topology/lhcb_topology.xml
      1. GOCDB name ---> country: https://wlcg-rebus.cern.ch/apps/topology/all/json

   * t_topology table 
      * The SE in the following structure is the hostname of the storage as it is sent by FTS while SITE is the GOCDB name.    
<verbatim>
describe t_topology
Name Null     Type          
---- -------- ------------- 
SE   NOT NULL VARCHAR2(256) 
SITE          VARCHAR2(256) </verbatim>    
     

   * Flow
      1. The hostname of the storage is sent by FTS as src_hostname/dst_hostname and stored in the RAW tables
      1. The computation jobs run over the RAW tables and store the src_hostname/dst_hostname as they are sent by FTS
      1. At each invocation of the FTS Dasbhoard APIs pointed out above:
         *   Lookups.1 is performed and then depending on the requested grouping:
            1. By country: + Lookups.3
            1. By site: + Lookups.2 
            1. By host: no further lookup is needed

   * Metadata:
      1. GOCDB name
      1. VO site name
      1. Country
  
---++ *Modules & RPMs*
   * dashboard-transfers-collector: It includes the config and the code of the FTS collector (https://gitlab.cern.ch/exp_dashboard/cosmic/tree/master/dashboard.transfers-collector)
   * dashboard-transfers: APIs, template of simplevisors config., FTS DAOs, FTS specific UI, MVC actions config. for FTS (https://gitlab.cern.ch/exp_dashboard/cosmic/tree/master/dashboard.transfers)   
   * dashboard-transfers-topology: toplogy releated stuff - topology update cron, static topology files and code for the topology resolution (https://gitlab.cern.ch/exp_dashboard/cosmic/tree/master/dashboard.transfers-topology) 
   * The other RPMs from the dashboard framework that are needed and not specific to FTS: dashboard-common, dashboard-xbrowse, dashboard-util-url, dashboard-dao-oracle, dashboard-service-config, dashboard-cli, dashboard-dao, dashboard-web)
 


-- Main.HassenRiahi - 2016-02-08
This topic: LCG > WebPreferences > FTSDashboard
Topic revision: r10 - 2016-05-20 - HassenRiahi
Copyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback