TWiki
>
LHCb Web
>
LHCbComputing
>
CodeAnalysisTools
>
LHCbPR
(revision 16) (raw view)
Edit
Attach
PDF
---+ LHCbPR - Performance and Regression Tests ---++ Introduction LHCbPR is responsible for systematically running the regression tests, collecting and comparing results of these tests so that any changes between different setups can be easily observed. The framework is based on a microservices architecture which breaks a project into loosely coupled modules communicating with each other through APIs. Most of the developed modules are generic which means that the proposed framework can be adopted for other experiments. ---++ Architecture The sequence diagram of the LHCbPR is shown in this figure: %ATTACHURL%/SequenceDiagram.pdf. ---++ Infrastructure ---+++ Configuration of the test The test is configured via https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf/blob/master/test_schedule2.xml. The example of the configuration is the following: <verbatim> <periodictest> <schedule type="week" time="10:00">Mon,Tue,Wed,Thu,Fri</schedule> <slot>lhcb-future</slot> <project>Brunel</project> <platform>x86_64-slc6-gcc62-opt</platform> <test runner="lhcbpr" group="MiniBrunel" env="lb-run-gaudirun|TimeLineHandler"/> <os_label>perf</os_label> <count>5</count> </periodictest></verbatim> The meaning of the keys: * =schedule type= : you can specify =week= or =month= * =schedule time= : usually ignored, see LHCbPR#Launching_the_tests below. * =schedule= : the day of the week if =type="week"= (always respected). * =slot=, =project=, =platform= : you can use globbing, e.g. =x86_64-slc6-gcc*-opt= to run the test for all gcc versions * =test runner= : use =lhcbpr= * =test group= : description of the option file, please contact us to add it to LHCbPR database https://lblhcbpr.cern.ch/api/options/ * =test env= : description of the command to run the test and the name of the handler. Please contact us to add the executable to LHCbPR database https://lblhcbpr.cern.ch/api/executables in case it is not already there. After =|= please specify the list of handlers separated by comma. The name should correspond to the file committed to https://gitlab.cern.ch/lhcb-core/LHCbPR2HD * =os_label= : use =perf= for =slc6= tests or =perf-centos7= for =centos7= or =perf-centos7-timing= for the timing tests on =centos7= (the dedicated machine with the single executor in jenkins will be used) * =count= : specify number of runs for the test (you can run multiple tests to check statistics - standard deviation will be computed for the extracted metrics) ---+++ Dashboard [[https://lbnightlies.cern.ch/periodic/summary/][Here]] you can find the dashboard for the periodic tests. One can verify here the status of the executed tests. The colour code is the following: * running tests in blue * successful tests in green * failed tests in red * tests which have been executed with success, but the handler failed (so that there is no output results for LHCbPR) in yellow The urls to log files of the test (stored on EOS) and output of the jenkins job are provided. ---+++ Launching the tests * *Automatic starting*. Tests defined in [[https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf/blob/master/test_schedule2.xml][configuration file]] are started automatically by the [[https://jenkins-lhcb-nightlies.web.cern.ch/job/periodic-tests/job/tests-pollqueue/configure][Jenkins job]]. The job checks (messaging infrastructure described [[LHCbDevOps#RabbitMQ_messaging_engine][here]] is used for this purpose) every 5 minutes if there any new builds for the defined tests for the given day (the exact time from the configuration is ignored). If yes, [[https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/tests/][another Jenkins job]] is triggered (the same which is used for tests of nightly builds) which actually runs the test. If the messaging infrastructure is down, as a backup, the tests may be started according to the time in the configuration when [[https://jenkins-lhcb-nightlies.web.cern.ch/job/periodic-tests/job/tests-poll/][tests-poll Jenkins jobs]] will be enabled. * *Manual start on demand*. After login to the [[https://lbnightlies.cern.ch/periodic/summary/][dashboard]], there is an orange button in top right, called =Start new periodic test=. After clicking it, you need to provide the same information which is in [[https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf/blob/master/test_schedule2.xml][configuration file]] (except for scheduling). The message will be sent to the queue checked by [[https://jenkins-lhcb-nightlies.web.cern.ch/job/periodic-tests/job/tests-getteststorun/][Jenkins job]] every 5 minutes. Alternatively you can use command line: <verbatim>export RMQPWD=lhcbpr/lhcbpr lbq-requesttest <slot> <buildid> <project> <config> <group> <env></verbatim> For example:<br /> <verbatim>lbq-requesttest 1467 lhcb-sim09 Gauss x86_64-slc6-gcc49-opt "GAUSS-RADLENGTHSCAN" "lb-run|RadLengthHandler"</verbatim> One can use also -l flag to specify the machine label in jenkins. ---+++ Machines Here are the machines used for running the periodic tests: * =volhcb05= with SLC6 (8 executors) * =lblhcbpr6= with Centos7 (8 executors) * =lblhcbpr1= with Centos7 (1 executor) * =lbhltperf01= node devoted for throughput testing ---+++ Handlers Handlers are python scripts used to extract relevant information from the the output produced by the test runs. The =BaseHandler= class enables to save Int, Float, String, JSON and File. The LHCbPR framework produces the zip file with the collected results which is sent to database through Dirac Storage element (=/lhcb/prdata/zips=). The description how to create handler and test it can be found [[https://gitlab.cern.ch/lhcb-core/LHCbPR2HD/blob/master/README.md][here]]. ---+++ Front-end The results of the tests are automatically picked up by web front-end available [[https://lblhcbpr.cern.ch][here]]. For the development of the specific analysis module please [[https://gitlab.cern.ch/lhcb-core/LHCbPR2FE/blob/master/documentation/modules-guide.md][see here]]. The generic tool to compare the plots can be found by going to _LHCbPR Jobs_ and _ROOT file viewer_ tabs. To perform the trend analysis see the example below. There is an ongoing work on [[https://gitlab.cern.ch/lhcb-core/LHCbPR2FE2][the new version of the web front-end]] available [[https://lblhcbpr.cern.ch/fe2][here]]. Results of the throughput tests are available [[https://test-lhcbpr-hlt.web.cern.ch/test-lhcbpr-hlt/PerfTests/UpgradeVelo/][here]]. ---++ Example of plotting trend Lets assume we want to plot the time spent by =EVENT_LOOP= as a function of the software version. The option file can be found [[https://svnweb.cern.ch/trac/lhcb/browser/DBASE/trunk/PRConfig/options/Brunel/PRTEST-COLLISION15-1000evts.py][here]], and the command used to run the test is: <verbatim> lb-run --use=PRConfig -c x86_64-slc6-gcc62-opt --user-area=$(pwd)/../build Brunel/HEAD gaudirun.py \$PRCONFIGOPTS/Brunel/PRTEST-COLLISION15-1000evts.py </verbatim> To plot the trend: * go to _Trends/Trends_ tab and select =Brunel= from the list of applications * select the option you are interested in, in this case =PRTEST-COLLISION15-1000= * tick _Show Nightly versions_ and specify the number of versions to show, e.g. 50. * start typing the name of the algorithm in the field _Filter attributes_ and click _Show_ You should see the plot: %ATTACHURL%/BrunelTiming.pdf. ---++ Requirements for participation To add your project to LHCbPR the following information is needed: * Command to run the test * The option file stored e.g. in [[https://gitlab.cern.ch/lhcb-datapkg/PRConfig][PRConfig]] * Handler to extract the relevant information from the test * Analysis module in case you are interested in specific presentation of results (other than trend analysis and generic comparison of plots using ROOT file browser) ---++ Analysing data from LHCbPR in SWAN Since May 2018, results of LHCbPR tests are copied to Hadoop Distributed File System (HDFS), see the [[http://hadoop-user-guide.web.cern.ch/hadoop-user-guide][user guide]], [[https://twiki.cern.ch/twiki/bin/viewauth/IT/ItHadoop][twiki]] and [[https://cern.service-now.com/service-portal/faq.do?fe=Hadoop-Components][knowledge base articles]] for reference. Details of the procedure can be found in https://gitlab.cern.ch/maszyman/lhcbpr-hadoop. Owing to that, one can use [[https://swan.cern.ch][SWAN notebooks]] to create custom reports on test results. Examples are available in =/eos/user/m/maszyman/SWAN_projects/read_hdfs=. The requirement to read from HDFS is to belong to =ai-hadoop-users= e-group (which can be granted by opening a SNOW ticket to the Hadoop and Spark Service to request access). To be able to read from HDFS, go to [[swan.cern.ch][SWAN]], open a new notebook (you may need to create a new project first), click on a star (second to last in top row - Spark clusters connection). By default you should be directed to =analytix= cluster, when you click 'Connect'. Alternatively, you can use [[http://hadoop-user-guide.web.cern.ch/hadoop-user-guide/getstart/client_docker.html][docker container]] to run pyspark: <verbatim> # get docker image docker login gitlab-registry.cern.ch docker pull gitlab-registry.cern.ch/db/cerndb-infra-hadoop-conf:qa # run it docker run -d -it -p 5000-5300:5000-5300 --hostname $HOSTNAME --name "lhcbpr-hadoop" -v /cvmfs/sft.cern.ch:/cvmfs/sft.cern.ch:shared gitlab-registry.cern.ch/db/cerndb-infra-hadoop-conf:qa # go into docker image docker exec -it lhcbpr-hadoop bash # get kerberos token kinit ${USER}@CERN.CH # run pyspark interactively (alternatively write a script and run it using spark-submit command) pyspark </verbatim> The notebook producing [[http://test-lhcbpr-hlt.web.cern.ch/test-lhcbpr-hlt/PerfTests/UpgradeVelo/trend.html][trend plots of the throughput]] is available [[https://cernbox.cern.ch/index.php/s/kg0de63HrPyp3xy][here]]. ---++ Development For the development of LHCbPR, please see [[https://gitlab.cern.ch/amazurov/LHCbPR2/blob/master/README.md][here]]. ---++ Contact and Infos ---+++ Support: * Ben Couturier * Alexander Mazurov * Maciej Szymański * [[https://mattermost.web.cern.ch/lhcb/channels/lhcbpr2][Mattermost channel for discussion and announcements about LHCbPR]] * [[https://mattermost.web.cern.ch/lhcb/channels/lhcbpr-new-results][Mattermost channel for notifications about results of the throughput tests]] ---++ Resources ---+++ Dashboard: * https://lbnightlies.cern.ch/periodic/summary/ ---+++ Web application: * https://lblhcbpr.cern.ch * https://gitlab.cern.ch/lhcb-core/LHCbPR2FE * https://lblhcbpr.cern.ch/fe2 * https://gitlab.cern.ch/lhcb-core/LHCbPR2FE2 * https://lhcbpr-docs.web.cern.ch/lhcbpr-docs/ (Results of HLT Rate Tests) * https://test-lhcbpr-hlt.web.cern.ch/test-lhcbpr-hlt/PerfTests/UpgradeThroughput/ (Results of Throughput Tests) ---+++ API service: * https://lblhcbpr.cern.ch/api/ * https://gitlab.cern.ch/lhcb-core/LHCbPR2BE ---+++ ROOT HTTP service: * https://gitlab.cern.ch/lhcb-core/LHCbPR2ROOT ---+++ Tests output handlers: * https://gitlab.cern.ch/lhcb-core/LHCbPR2HD ---+++ Project builder: * https://gitlab.cern.ch/amazurov/LHCbPR2 ---+++ Jenkins configuration * https://jenkins-lhcb-nightlies.web.cern.ch/job/periodic-tests/ * https://gitlab.cern.ch/lhcb-core/LbNightlyTools ---+++ Configuration of the periodic tests * https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf/blob/master/test_schedule2.xml ---++ Collection of various talks given on the subject of LHCbPR * [[https://indico.cern.ch/event/464391/contributions/2187098/attachments/1284248/1909259/mazurov_LHCbPRV2_7th.pdf][Performance and Regression tests for Simulation]], Alexander Mazurov. 7th Computing Workshop * [[https://indico.cern.ch/event/505613/contributions/2228534/attachments/1346891/2031290/Poster-333.pdf][Microservices for systematic profiling and monitoring of the refactoring process at the LHCb experiment]], Alexander Mazurov, CHEP 2016 * [[https://indico.cern.ch/event/636995/#1-software-reliability-worksho][A dedicated infrastructure for performance and regression tests, LHCbPR2]], abstract and summary sent for NSS 2017, Alexander Mazurov * [[https://indico.cern.ch/event/603961/contributions/2531647/attachments/1439414/2215251/SimulationStatusLHCbPR2TimWilliams.pdf][LHCbPR2 and simulation checks]], Tim Williams, 67th A&S Week * [[https://indico.cern.ch/event/561981/contributions/2586868/attachments/1459298/2253567/Szymanski_ComputingWorkshop_20170515.pdf][Improvements to LHCbPR2]], Maciej Szymanski, 9th Computing Workshop * [[https://indico.cern.ch/event/561981/contributions/2578038/attachments/1461768/2258303/20170518-CompWs-Metrics.pdf][Performance metrics in PR2]], Stefan Roiser, 9th Computing Workshop * [[https://indico.cern.ch/event/587955/contributions/2938130/attachments/1678787/2697985/szymanski.pdf][Improvements to the LHCb software performance testing infrastructure using message queues and big data technologies]], Maciej Szymanski, CHEP 2018
Attachments
Attachments
Topic attachments
I
Attachment
History
Action
Size
Date
Who
Comment
pdf
BrunelTiming.pdf
r1
manage
821.8 K
2017-05-22 - 16:17
MaciejSzymanski
pdf
SequenceDiagram.pdf
r1
manage
79.1 K
2017-05-22 - 16:17
MaciejSzymanski
Edit
|
Attach
|
Watch
|
P
rint version
|
H
istory
:
r20
|
r18
<
r17
<
r16
<
r15
|
B
acklinks
|
V
iew topic
|
Raw edit
|
More topic actions...
Topic revision: r16 - 2019-06-05
-
MaciejSzymanski
Log In
LHCb
LHCb Web
LHCb Web Home
Changes
Index
Search
LHCb webs
LHCbComputing
LHCb FAQs
LHCbOnline
LHCbPhysics
LHCbVELO
LHCbST
LHCbOT
LHCbRICH
LHCbMuon
LHCbTrigger
LHCbDetectorAlignment
LHCbTechnicalCoordination
LHCbUpgrade
Public webs
Public webs
ABATBEA
ACPP
ADCgroup
AEGIS
AfricaMap
AgileInfrastructure
ALICE
AliceEbyE
AliceSPD
AliceSSD
AliceTOF
AliFemto
ALPHA
ArdaGrid
ASACUSA
AthenaFCalTBAna
Atlas
AtlasLBNL
AXIALPET
CAE
CALICE
CDS
CENF
CERNSearch
CLIC
Cloud
CloudServices
CMS
Controls
CTA
CvmFS
DB
DefaultWeb
DESgroup
DPHEP
DM-LHC
DSSGroup
EGEE
EgeePtf
ELFms
EMI
ETICS
FIOgroup
FlukaTeam
Frontier
Gaudi
GeneratorServices
GuidesInfo
HardwareLabs
HCC
HEPIX
ILCBDSColl
ILCTPC
IMWG
Inspire
IPv6
IT
ItCommTeam
ITCoord
ITdeptTechForum
ITDRP
ITGT
ITSDC
LAr
LCG
LCGAAWorkbook
Leade
LHCAccess
LHCAtHome
LHCb
LHCgas
LHCONE
LHCOPN
LinuxSupport
Main
Medipix
Messaging
MPGD
NA49
NA61
NA62
NTOF
Openlab
PDBService
Persistency
PESgroup
Plugins
PSAccess
PSBUpgrade
R2Eproject
RCTF
RD42
RFCond12
RFLowLevel
ROXIE
Sandbox
SocialActivities
SPI
SRMDev
SSM
Student
SuperComputing
Support
SwfCatalogue
TMVA
TOTEM
TWiki
UNOSAT
Virtualization
VOBox
WITCH
XTCA
Welcome Guest
Login
or
Register
Cern Search
TWiki Search
Google Search
LHCb
All webs
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use
Discourse
or
Send feedback