Difference: LHCbPR (1 vs. 19)

Revision 192019-08-05 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 69 to 69
  There is an ongoing work on the new version of the web front-end available here.
Changed:
<
<
Results of the throughput tests are available here.
>
>
Results of the throughput tests are available here.
 

Example of plotting trend

Line: 119 to 119
 pyspark
Changed:
<
<
The notebook producing trend plots of the throughput is available here.
>
>
The notebook producing trend plots of the throughput is available here.
 

Setup of the throughput tests

Line: 148 to 148
 
Changed:
<
<
>
>
 

API service:

Revision 182019-06-17 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 119 to 119
 pyspark
Changed:
<
<
The notebook producing trend plots of the throughput is available here.
>
>
The notebook producing trend plots of the throughput is available here.

Setup of the throughput tests

The throughput tests are running making use of lhcb-benchmark-scripts repository. Please see readme for details.

You can find an example of the setup in https://lblhcbpr.cern.ch/api/options/90/. Please note that lhcb-benchmark-scripts repository is setup in LbNightlyTools.

 

Development

Revision 172019-06-17 - MarcoCattaneo

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 169 to 169
 
Added:
>
>
 
META FILEATTACHMENT attachment="BrunelTiming.pdf" attr="" comment="" date="1495462629" name="BrunelTiming.pdf" path="BrunelTiming.pdf" size="841538" user="maszyman" version="1"
META FILEATTACHMENT attachment="SequenceDiagram.pdf" attr="" comment="" date="1495462629" name="SequenceDiagram.pdf" path="SequenceDiagram.pdf" size="80965" user="maszyman" version="1"

Revision 162019-06-05 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 55 to 55
  Here are the machines used for running the periodic tests:
  • volhcb05 with SLC6 (8 executors)
Changed:
<
<
  • lblhcbpr5 with Centos7 (6 executors)
>
>
  • lblhcbpr6 with Centos7 (8 executors)
 
  • lblhcbpr1 with Centos7 (1 executor)
  • lbhltperf01 node devoted for throughput testing

Revision 152019-04-05 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 142 to 142
 
Changed:
<
<
>
>
 

API service:

Revision 142019-02-27 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 119 to 119
 pyspark
Changed:
<
<
The notebook producing trend plots of the throughput is available here.
>
>
The notebook producing trend plots of the throughput is available here.
 

Development

Revision 132019-02-22 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 48 to 48
 The urls to log files of the test (stored on EOS) and output of the jenkins job are provided.

Launching the tests

Changed:
<
<
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks (messaging infrastructure described here is used for this purpose) every 10 minutes if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test. If the messaging infrastructure is down, as a backup, the tests are started according to the time in the configuration.
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 10 minutes. Alternatively you can use command line:
    export RMQPWD=lhcbpr/lhcbpr
>
>
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks (messaging infrastructure described here is used for this purpose) every 5 minutes if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test. If the messaging infrastructure is down, as a backup, the tests may be started according to the time in the configuration when tests-poll Jenkins jobs will be enabled.
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 5 minutes. Alternatively you can use command line:
    export RMQPWD=lhcbpr/lhcbpr
 lbq-requesttest For example:
lbq-requesttest 1467 lhcb-sim09  Gauss x86_64-slc6-gcc49-opt "GAUSS-RADLENGTHSCAN" "lb-run|RadLengthHandler"
One can use also -l flag to specify the machine label in jenkins.

Machines

Changed:
<
<
Currently, two machines are used for running the periodic tests:
>
>
Here are the machines used for running the periodic tests:
 
  • volhcb05 with SLC6 (8 executors)
Changed:
<
<
  • lblhcbpr4 with Centos7 (8 executors)
>
>
  • lblhcbpr5 with Centos7 (6 executors)
 
  • lblhcbpr1 with Centos7 (1 executor)
Added:
>
>
  • lbhltperf01 node devoted for throughput testing
 

Handlers

Line: 66 to 67
  The results of the tests are automatically picked up by web front-end available here. For the development of the specific analysis module please see here. The generic tool to compare the plots can be found by going to LHCbPR Jobs and ROOT file viewer tabs. To perform the trend analysis see the example below.
Added:
>
>
There is an ongoing work on the new version of the web front-end available here.

Results of the throughput tests are available here.

 

Example of plotting trend

Letís assume we want to plot the time spent by EVENT_LOOP as a function of the software version. The option file can be found here, and the command used to run the test is:

Line: 93 to 98
  Owing to that, one can use SWAN notebooks to create custom reports on test results. Examples are available in /eos/user/m/maszyman/SWAN_projects/read_hdfs. The requirement to read from HDFS is to belong to ai-hadoop-users e-group (which can be granted by opening a SNOW ticket to the Hadoop and Spark Service to request access).
Added:
>
>
To be able to read from HDFS, go to SWAN, open a new notebook (you may need to create a new project first), click on a star (second to last in top row - Spark clusters connection). By default you should be directed to analytix cluster, when you click 'Connect'.

Alternatively, you can use docker container to run pyspark:

# get docker image
docker login gitlab-registry.cern.ch
docker pull gitlab-registry.cern.ch/db/cerndb-infra-hadoop-conf:qa

# run it
docker run -d -it -p 5000-5300:5000-5300 --hostname $HOSTNAME --name "lhcbpr-hadoop"  -v /cvmfs/sft.cern.ch:/cvmfs/sft.cern.ch:shared gitlab-registry.cern.ch/db/cerndb-infra-hadoop-conf:qa

# go into docker image
docker exec -it lhcbpr-hadoop bash

# get kerberos token
kinit ${USER}@CERN.CH

# run pyspark interactively (alternatively write a script and run it using spark-submit command)
pyspark 

The notebook producing trend plots of the throughput is available here.

 

Development

For the development of LHCbPR, please see here.

Line: 102 to 130
 
  • Ben Couturier
  • Alexander Mazurov
  • Maciej Szymański
Changed:
<
<
>
>
 

Resources

Dashboard:

Line: 110 to 139
 

Web application:

Added:
>
>
 
Added:
>
>
 

API service:

Revision 122019-02-22 - RosenMatev

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 29 to 29
  The meaning of the keys:
  • schedule type : you can specify week or month
Added:
>
>
  • schedule time : usually ignored, see LHCbPR#Launching_the_tests below.
  • schedule : the day of the week if type="week" (always respected).
 
  • slot, project, platform : you can use globbing, e.g. x86_64-slc6-gcc*-opt to run the test for all gcc versions
  • test runner : use lhcbpr
  • test group : description of the option file, please contact us to add it to LHCbPR database https://lblhcbpr.cern.ch/api/options/
Line: 46 to 48
 The urls to log files of the test (stored on EOS) and output of the jenkins job are provided.

Launching the tests

Changed:
<
<
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks (messaging infrastructure described here is used for this purpose) every hour if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test.
>
>
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks (messaging infrastructure described here is used for this purpose) every 10 minutes if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test. If the messaging infrastructure is down, as a backup, the tests are started according to the time in the configuration.
 
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 10 minutes. Alternatively you can use command line:
    export RMQPWD=lhcbpr/lhcbpr
    lbq-requesttest <slot> <buildid> <project> <config> <group> <env>
    For example:
    lbq-requesttest 1467 lhcb-sim09  Gauss x86_64-slc6-gcc49-opt "GAUSS-RADLENGTHSCAN" "lb-run|RadLengthHandler"
    One can use also -l flag to specify the machine label in jenkins.

Machines

Revision 112018-09-18 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 85 to 85
 
  • Handler to extract the relevant information from the test
  • Analysis module in case you are interested in specific presentation of results (other than trend analysis and generic comparison of plots using ROOT file browser)
Added:
>
>

Analysing data from LHCbPR in SWAN

Since May 2018, results of LHCbPR tests are copied to Hadoop Distributed File System (HDFS), see the user guide, twiki and knowledge base articles for reference. Details of the procedure can be found in https://gitlab.cern.ch/maszyman/lhcbpr-hadoop.

Owing to that, one can use SWAN notebooks to create custom reports on test results. Examples are available in /eos/user/m/maszyman/SWAN_projects/read_hdfs. The requirement to read from HDFS is to belong to ai-hadoop-users e-group (which can be granted by opening a SNOW ticket to the Hadoop and Spark Service to request access).

 

Development

For the development of LHCbPR, please see here.

Revision 102018-09-05 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 126 to 126
 
Added:
>
>
 
META FILEATTACHMENT attachment="BrunelTiming.pdf" attr="" comment="" date="1495462629" name="BrunelTiming.pdf" path="BrunelTiming.pdf" size="841538" user="maszyman" version="1"
META FILEATTACHMENT attachment="SequenceDiagram.pdf" attr="" comment="" date="1495462629" name="SequenceDiagram.pdf" path="SequenceDiagram.pdf" size="80965" user="maszyman" version="1"

Revision 92018-09-03 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 22 to 25
  perf 5
Changed:
<
<
>
>
 The meaning of the keys:
  • schedule type : you can specify week or month
  • slot, project, platform : you can use globbing, e.g. x86_64-slc6-gcc*-opt to run the test for all gcc versions
Line: 42 to 46
 The urls to log files of the test (stored on EOS) and output of the jenkins job are provided.

Launching the tests

Changed:
<
<
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks (messaging infrastructure described here is used for this purpose) every hour if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test.
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 10 minutes.
>
>
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks (messaging infrastructure described here is used for this purpose) every hour if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test.
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 10 minutes. Alternatively you can use command line:
    export RMQPWD=lhcbpr/lhcbpr
    lbq-requesttest <slot> <buildid> <project> <config> <group> <env>
    For example:
    lbq-requesttest 1467 lhcb-sim09  Gauss x86_64-slc6-gcc49-opt "GAUSS-RADLENGTHSCAN" "lb-run|RadLengthHandler"
    One can use also -l flag to specify the machine label in jenkins.
 

Machines

Added:
>
>
 Currently, two machines are used for running the periodic tests:
  • volhcb05 with SLC6 (8 executors)
  • lblhcbpr4 with Centos7 (8 executors)
Line: 82 to 93
 

Support:

  • Ben Couturier
  • Alexander Mazurov
Changed:
<
<
  • Maciej Szymański
>
>
  • Maciej Szymański
 

Resources

Revision 82018-06-26 - GiulioDujany

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 71 to 71
 

Requirements for participation

To add your project to LHCbPR the following information is needed:
  • Command to run the test
Changed:
<
<
  • The option file stored e.g. in PRConfig
>
>
  • The option file stored e.g. in PRConfig
 
  • Handler to extract the relevant information from the test
  • Analysis module in case you are interested in specific presentation of results (other than trend analysis and generic comparison of plots using ROOT file browser)

Revision 72017-11-13 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 28 to 28
 
  • schedule type : you can specify week or month
  • slot, project, platform : you can use globbing, e.g. x86_64-slc6-gcc*-opt to run the test for all gcc versions
  • test runner : use lhcbpr
Changed:
<
<
>
>
 
  • os_label : use perf for slc6 tests or perf-centos7 for centos7 or perf-centos7-timing for the timing tests on centos7 (the dedicated machine with the single executor in jenkins will be used)
  • count : specify number of runs for the test (you can run multiple tests to check statistics - standard deviation will be computed for the extracted metrics)
Line: 42 to 42
 The urls to log files of the test (stored on EOS) and output of the jenkins job are provided.

Launching the tests

Changed:
<
<
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks every hour if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test.
>
>
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks (messaging infrastructure described here is used for this purpose) every hour if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test.
 
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 10 minutes.

Machines

Line: 55 to 55
 Handlers are python scripts used to extract relevant information from the the output produced by the test runs. The BaseHandler class enables to save Int, Float, String, JSON and File. The LHCbPR framework produces the zip file with the collected results which is sent to database through Dirac Storage element (/lhcb/prdata/zips). The description how to create handler and test it can be found here.

Front-end

Changed:
<
<
The results of the tests are automatically picked up by web front-end available here. For the development of the specific analysis module please see here. The generic tool to compare the plots can be found by going to LHCbPR Jobs and ROOT file viewer tabs. To perform the trend analysis see the example below.
>
>
The results of the tests are automatically picked up by web front-end available here. For the development of the specific analysis module please see here. The generic tool to compare the plots can be found by going to LHCbPR Jobs and ROOT file viewer tabs. To perform the trend analysis see the example below.
 

Example of plotting trend

Letís assume we want to plot the time spent by EVENT_LOOP as a function of the software version. The option file can be found here, and the command used to run the test is:
Line: 89 to 89
 

Dashboard:

Web application:

Changed:
<
<
>
>
 

API service:

Changed:
<
<
>
>
 

ROOT HTTP service:

Revision 62017-08-09 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 105 to 105
 

Configuration of the periodic tests

Changed:
<
<
>
>
 

Collection of various talks given on the subject of LHCbPR

Revision 52017-06-30 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 30 to 30
 
Changed:
<
<
  • os_label : use perf for slc6 tests and perf-centos7 for centos7
>
>
  • os_label : use perf for slc6 tests or perf-centos7 for centos7 or perf-centos7-timing for the timing tests on centos7 (the dedicated machine with the single executor in jenkins will be used)
 
  • count : specify number of runs for the test (you can run multiple tests to check statistics - standard deviation will be computed for the extracted metrics)

Dashboard

Line: 42 to 42
 The urls to log files of the test (stored on EOS) and output of the jenkins job are provided.

Launching the tests

Changed:
<
<
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks every hour if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, it triggers another Jenkins job (the same which is used for tests of nightly builds) which actually runs the test.
>
>
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks every hour if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, another Jenkins job is triggered (the same which is used for tests of nightly builds) which actually runs the test.
 
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 10 minutes.

Machines

Currently, two machines are used for running the periodic tests:
  • volhcb05 with SLC6 (8 executors)
  • lblhcbpr4 with Centos7 (8 executors)
Added:
>
>
  • lblhcbpr1 with Centos7 (1 executor)
 

Handlers

Handlers are python scripts used to extract relevant information from the the output produced by the test runs. The BaseHandler class enables to save Int, Float, String, JSON and File. The LHCbPR framework produces the zip file with the collected results which is sent to database through Dirac Storage element (/lhcb/prdata/zips). The description how to create handler and test it can be found here.

Revision 42017-06-14 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCbPR - Performance and Regression Tests

Line: 42 to 42
 The urls to log files of the test (stored on EOS) and output of the jenkins job are provided.

Launching the tests

Changed:
<
<
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job which checks every 10 minutes if there any tests to be run. If yes, it triggers another Jenkins job (the same which is used for tests of nightly builds) which actually runs the test.
>
>
  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job. The job checks every hour if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, it triggers another Jenkins job (the same which is used for tests of nightly builds) which actually runs the test.
 
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 10 minutes.

Machines

Currently, two machines are used for running the periodic tests:
Changed:
<
<
  • volhcb05 with SLC6 (6 executors)
  • lblhcbpr3 with Centos7 (3 executors)
>
>
  • volhcb05 with SLC6 (8 executors)
  • lblhcbpr4 with Centos7 (8 executors)
 

Handlers

Handlers are python scripts used to extract relevant information from the the output produced by the test runs. The BaseHandler class enables to save Int, Float, String, JSON and File. The LHCbPR framework produces the zip file with the collected results which is sent to database through Dirac Storage element (/lhcb/prdata/zips). The description how to create handler and test it can be found here.

Revision 32017-05-22 - MaciejSzymanski

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCb PR - Performance and Regression Tests

Deleted:
<
<
 

Introduction

Added:
>
>
LHCbPR is responsible for systematically running the regression tests, collecting and comparing results of these tests so that any changes between different setups can be easily observed. The framework is based on a microservices architecture which breaks a project into loosely coupled modules communicating with each other through APIs. Most of the developed modules are generic which means that the proposed framework can be adopted for other experiments.

Architecture

The sequence diagram of the LHCbPR is shown in this figure: https://twiki.cern.ch/twiki/pub/LHCb/LHCbPR/SequenceDiagram.pdf.

Infrastructure

Configuration of the test

The test is configured via https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf/blob/master/test_schedule2.xml. The example of the configuration is the following:
 <periodictest>
    <schedule type="week" time="10:00">Mon,Tue,Wed,Thu,Fri</schedule>
    <slot>lhcb-future</slot>
    <project>Brunel</project>
    <platform>x86_64-slc6-gcc62-opt</platform>
    <test runner="lhcbpr" group="MiniBrunel" env="lb-run-gaudirun|TimeLineHandler"/>
    <os_label>perf</os_label>
    <count>5</count>
  </periodictest>
The meaning of the keys:
  • schedule type : you can specify week or month
  • slot, project, platform : you can use globbing, e.g. x86_64-slc6-gcc*-opt to run the test for all gcc versions
  • test runner : use lhcbpr
  • test group : description of the option file, please contact us to add it to LHCbPR database https://lblhcbpr2.cern.ch/api/options/
  • test env : description of the command to run the test and the name of the handler. Please contact us to add the executable to LHCbPR database https://lblhcbpr2.cern.ch/api/executables in case it is not already there. After | please specify the list of handlers separated by comma. The name should correspond to the file committed to https://gitlab.cern.ch/lhcb-core/LHCbPR2HD
  • os_label : use perf for slc6 tests and perf-centos7 for centos7
  • count : specify number of runs for the test (you can run multiple tests to check statistics - standard deviation will be computed for the extracted metrics)

Dashboard

Here you can find the dashboard for the periodic tests. One can verify here the status of the executed tests. The colour code is the following:
  • running tests in blue
  • successful tests in green
  • failed tests in red
  • tests which have been executed with success, but the handler failed (so that there is no output results for LHCbPR) in yellow
The urls to log files of the test (stored on EOS) and output of the jenkins job are provided.

Launching the tests

  • Automatic starting. Tests defined in configuration file are started automatically by the Jenkins job which checks every 10 minutes if there any tests to be run. If yes, it triggers another Jenkins job (the same which is used for tests of nightly builds) which actually runs the test.
  • Manual start on demand. After login to the dashboard, there is an orange button in top right, called Start new periodic test. After clicking it, you need to provide the same information which is in configuration file (except for scheduling). The message will be sent to the queue checked by Jenkins job every 10 minutes.

Machines

Currently, two machines are used for running the periodic tests:
  • volhcb05 with SLC6 (6 executors)
  • lblhcbpr3 with Centos7 (3 executors)

Handlers

Handlers are python scripts used to extract relevant information from the the output produced by the test runs. The BaseHandler class enables to save Int, Float, String, JSON and File. The LHCbPR framework produces the zip file with the collected results which is sent to database through Dirac Storage element (/lhcb/prdata/zips). The description how to create handler and test it can be found here.

Front-end

The results of the tests are automatically picked up by web front-end available here. For the development of the specific analysis module please see here. The generic tool to compare the plots can be found by going to LHCbPR Jobs and ROOT file viewer tabs. To perform the trend analysis see the example below.

Example of plotting trend

Letís assume we want to plot the time spent by EVENT_LOOP as a function of the software version. The option file can be found here, and the command used to run the test is:
lb-run --use=PRConfig -c x86_64-slc6-gcc62-opt --user-area=$(pwd)/../build Brunel/HEAD gaudirun.py \$PRCONFIGOPTS/Brunel/PRTEST-COLLISION15-1000evts.py 
To plot the trend:
  • go to Trends/Trends tab and select Brunel from the list of applications
  • select the option you are interested in, in this case PRTEST-COLLISION15-1000
  • tick Show Nightly versions and specify the number of versions to show, e.g. 50.
  • start typing the name of the algorithm in the field Filter attributes and click Show
You should see the plot: https://twiki.cern.ch/twiki/pub/LHCb/LHCbPR/BrunelTiming.pdf.
 
Deleted:
<
<
The project LHCb PR is the attempt to systematize profiling of all Gaudi frameworks and to help developers to evaluate how their recent changes behave in default test cases. It can also be used as indicator for worsen algorithms due to run-time or memory consumption and thus as knowledge basis for warning mechanism or for project leaders to track the development process from the optimization point of view. Everyone is welcome to participate.

Description

Structure

LHCb PR persists out of five important parts. Three which are important for users and two more for members of the LHCb PR project:

  1. Defining default use cases
  2. Start and monitor test jobs
  3. Analyze collected results
  4. Defining standard profiling scripts (restricted to members)
  5. Creating data handlers to hook information onto the database for web based analysis (restricted to members)

Defining default cases

A profiling job is determined by:

  • An option file to configure the Gaudi run
  • A very limited set of reference data
  • A selected profiler to collect profiling information
Everyone is oblieged to create default cases relevant to their development attempt. The current profilers of choice are: Many other profilers could be added by request. They should be mentioned on the Code Analysis page.

Monitoring and Submitting Test Jobs

To start and monitor test runs the nightly build system helps to define and run jobs on our dedicated host for profiling.

Analysis and Visualization

Web based analysis

The platform for analyzing collected data via a web front-end can be found here. Open the link and follow the instructions to see the results of successfully finished test jobs.

Detailed Analysis of profiling data on AFS

While profiling, information are stored onto the afs (/afs/cern.ch/lhcb/software/profiling). For more detailed analysis, e.g. with the GUI of Intel's VTune you can access data from there.

Project development

Framework development

Web front-end

LHCb PR is based on Django. The customized part can be checked out via GIT (git clone /afs/cern.ch/lhcb/software/GIT/LHCbPR.git). Some information about about how to develop them can be found here: LHCbPRTipsAndTricks

Profiler scripts

Each run creates a job on the monitoring platform jenkins from where it starts scripts to execute the profiling and to initiate data selection and to call the data handler for collecting information for the database. This project is available under GIT (git clone /afs/cern.ch/lhcb/software/GIT/profiling.git).

LHCb PR data handlers

Data handler to collect information in the end of each test run can be found on GIT (git clone /afs/cern.ch/lhcb/software/GIT/LHCbPRHandlers.git).

Ongoing development (ToDo)

The ongoing development can be obsereved via Jira.

 

Requirements for participation

Added:
>
>
To add your project to LHCbPR the following information is needed:
  • Command to run the test
  • The option file stored e.g. in PRConfig
  • Handler to extract the relevant information from the test
  • Analysis module in case you are interested in specific presentation of results (other than trend analysis and generic comparison of plots using ROOT file browser)

Development

For the development of LHCbPR, please see here.
 
Deleted:
<
<
To add your project for regular profiling you need to provide at least:
  • An optionfile for configuration
  • Reference data to process on
For specific profiling purposes please don't hesitate to contact us.
 

Contact and Infos

Support:

Deleted:
<
<
  • Stefan Lohn (Moore, Hlt, Brunel)
  • Ben Couturier (Brunel, DaVinci)
  • Gloria Corti (Gauss)

Developers:

  • Stefan Lohn
  • Emmanouil Kiagias
 \ No newline at end of file
Added:
>
>

Resources

Dashboard:

Web application:

API service:

ROOT HTTP service:

Testsí output handlers:

Project builder:

Jenkins configuration

Configuration of the periodic tests

Collection of various talks given on the subject of LHCbPR

META FILEATTACHMENT attachment="BrunelTiming.pdf" attr="" comment="" date="1495462629" name="BrunelTiming.pdf" path="BrunelTiming.pdf" size="841538" user="maszyman" version="1"
META FILEATTACHMENT attachment="SequenceDiagram.pdf" attr="" comment="" date="1495462629" name="SequenceDiagram.pdf" path="SequenceDiagram.pdf" size="80965" user="maszyman" version="1"

Revision 22013-06-28 - BenjaminCouturier

Line: 1 to 1
 
META TOPICPARENT name="CodeAnalysisTools"

LHCb PR - Performance and Regression Tests

Line: 42 to 42
 

Web front-end

LHCb PR is based on Django. The customized part can be checked out via GIT (git clone /afs/cern.ch/lhcb/software/GIT/LHCbPR.git).

Added:
>
>
Some information about about how to develop them can be found here: LHCbPRTipsAndTricks
 

Profiler scripts

Each run creates a job on the monitoring platform jenkins from where it starts scripts to execute the profiling and to initiate data selection and to call the data handler for collecting information for the database. This project is available under GIT (git clone /afs/cern.ch/lhcb/software/GIT/profiling.git).

Revision 12013-06-10 - StefanLohn

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="CodeAnalysisTools"

LHCb PR - Performance and Regression Tests

Introduction

The project LHCb PR is the attempt to systematize profiling of all Gaudi frameworks and to help developers to evaluate how their recent changes behave in default test cases. It can also be used as indicator for worsen algorithms due to run-time or memory consumption and thus as knowledge basis for warning mechanism or for project leaders to track the development process from the optimization point of view. Everyone is welcome to participate.

Description

Structure

LHCb PR persists out of five important parts. Three which are important for users and two more for members of the LHCb PR project:

  1. Defining default use cases
  2. Start and monitor test jobs
  3. Analyze collected results
  4. Defining standard profiling scripts (restricted to members)
  5. Creating data handlers to hook information onto the database for web based analysis (restricted to members)

Defining default cases

A profiling job is determined by:

  • An option file to configure the Gaudi run
  • A very limited set of reference data
  • A selected profiler to collect profiling information
Everyone is oblieged to create default cases relevant to their development attempt. The current profilers of choice are: Many other profilers could be added by request. They should be mentioned on the Code Analysis page.

Monitoring and Submitting Test Jobs

To start and monitor test runs the nightly build system helps to define and run jobs on our dedicated host for profiling.

Analysis and Visualization

Web based analysis

The platform for analyzing collected data via a web front-end can be found here. Open the link and follow the instructions to see the results of successfully finished test jobs.

Detailed Analysis of profiling data on AFS

While profiling, information are stored onto the afs (/afs/cern.ch/lhcb/software/profiling). For more detailed analysis, e.g. with the GUI of Intel's VTune you can access data from there.

Project development

Framework development

Web front-end

LHCb PR is based on Django. The customized part can be checked out via GIT (git clone /afs/cern.ch/lhcb/software/GIT/LHCbPR.git).

Profiler scripts

Each run creates a job on the monitoring platform jenkins from where it starts scripts to execute the profiling and to initiate data selection and to call the data handler for collecting information for the database. This project is available under GIT (git clone /afs/cern.ch/lhcb/software/GIT/profiling.git).

LHCb PR data handlers

Data handler to collect information in the end of each test run can be found on GIT (git clone /afs/cern.ch/lhcb/software/GIT/LHCbPRHandlers.git).

Ongoing development (ToDo)

The ongoing development can be obsereved via Jira.

Requirements for participation

To add your project for regular profiling you need to provide at least:

  • An optionfile for configuration
  • Reference data to process on
For specific profiling purposes please don't hesitate to contact us.

Contact and Infos

Support:

  • Stefan Lohn (Moore, Hlt, Brunel)
  • Ben Couturier (Brunel, DaVinci)
  • Gloria Corti (Gauss)

Developers:

  • Stefan Lohn
  • Emmanouil Kiagias
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback