Tools to detect problems:

  • RabbitMQ Web interface at: http://lbmessagingbroker.cern.ch:15672/#/queues
    • An accumulation of messages in BuildReadyCIAgent queue means that there is a problem with CIAgent
    • An accumulation of messages in CVMFSDevActions queue (> 1000 messages depending on the GC execution time) means that there is a problem with CVMFS Scheduler on CVMFS-Stratum 0
  • bin/lastCommands command on CVMFS-Stratum 0 displays the commands executed on the CVMFS Scheduler for the last 24h.
  • bin/report_generator on CVMFS-Stratum 0 generates a report for the status of todays installations. The reports are saved on on CVMFS-Stratum 0 in reports/ with the name reports.[txt or json]

On cvmfs-lhcbdev.cern.ch it is possible to list the commands currently in the queue by running:

$ viewCommands
1255   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-opt   AlignmentOnline
1254   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-opt   Lovell
1253   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-opt   LHCbIntegrationTests
1252   127   installNightlies   lhcb-head   1725   x86_64-slc6-gcc62-dbg   Geant4
1251   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc62-do0   Panoptes
1250   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-dbg   DaVinci
1249   127   installNightlies   lhcb-head   1725   x86_64-slc6-gcc62-dbg   Gauss
1248   127   installNightlies   lhcb-head   1725   x86_64-slc6-gcc62-dbg   Urania
1247   127   installNightlies   lhcb-head   1725   x86_64-slc6-gcc62-dbg   Erasmus
1246   127   installNightlies   lhcb-head   1725   x86_64-slc6-gcc62-dbg   Integration
1245   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc62-do0   Alignment
1244   127   installNightlies   lhcb-head   1725   x86_64-slc6-gcc62-dbg   Noether
1243   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc62-do0   Vetra
1242   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-dbg   Hlt
1241   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-dbg   Panoramix
1240   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-dbg   Bender
1239   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-dbg   Moore
1238   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-dbg   MooreOnline
1237   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-dbg   Panoptes
1236   127   installNightlies   lhcb-head   1725   x86_64-centos7-gcc7-dbg   Alignment
[...]
    

Debugging of CIAgent :

  • Demon resides on lbmessagingbroker
  • Start/Stop demon as root: systemctl start ciagent
  • Logs: journalctl -r  -u ciagent
  • Connect as lblocal via: sudo -s -u lblocal
  • Demon script: /build/lbciagent/lbciagentdaemon.sh
  • Environment: . /build/lbciagent/virtualenv/bin/activate
  • Python package: lbciagent at https://gitlab.cern.ch/lhcb-core/lbciagent
  • if you need to update the packages in case of a bug, please redeploy the package on the pypi server (refer to https://twiki.cern.ch/twiki/bin/view/LHCb/PythonPyPIServer), uninstall the package on CVMFS: pip uninstall and install the new version: pip install --extra-index-url https://lbmultipython03.cern.ch/simple/ --trusted-host lbmultipython03.cern.ch

Debugging of CVMFS scheduler on CVMFS-Stratum 0 :

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2018-01-16 - BenjaminCouturier
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback