TWiki
>
LHCb Web
>
LHCbComputing
>
LHCbNightlies
>
LHCbNightliesTroubleshooting
(2015-07-07,
MarcoClemencic
)
(raw view)
E
dit
A
ttach
P
DF
---+!! LHCb Nightly Build System Troubleshooting and Operation %TOC% *It's responsibility of the Deployment Shifters to check that the nightly builds are functional.* ---++ Introduction First of all, be sure that you read LHCbNightliesImplementation to understand how the various bits and pieces stick together. It should not happen, but, as a matter of fact, Jenkins Jobs might fail for a number of reasons. Shifters will receive a mail for every (nightly build related) failed Jenkins job, as well as for the first successful job after a failure. In the most common cases the failure is due to a glitch of the infrastructure (communication between Jenkins and its slaves, connection with git/svn servers) and the automatic retry we use for most jobs will be enough to recover. ---++ How to Read Jenkins Mails There are two types of mails: _failure_ and _back to normal_. ---+++ Failed Builds In case of failure, the shifter will get a mail with a subject like: <verbatim> Build failed in Jenkins: <job name> <build name or id> </verbatim> where the <job name> is the string referring to the jobs described in LHCbNightliesImplementation and the <build name or id> could be a numeric id (if the job failed very early) or a human readable name for the job build, like <flavour>.<slot>.<id>. The body of the mail starts with a link to the failed build in the Jenkins web interface, followed by an excerpt of the console output of the build. ---+++ Build Back to Normal For the first successful build after a failure, shifter will receive a mail with a subject like: <verbatim> Jenkins build is back to normal : <job name> <build name or id> </verbatim> The body of the mail consists only of a link to the successful build in the Jenkins web interface. It is important to follow the link and check that the preceding build ("Previous Build" link) to check why it failed, because sometimes a build fails so early that the failure mail is not sent. ---++ Jenkins Build Farm Jenkins uses a pool of machines called _slaves_ to actually perform the tasks, plus one special machine called _master_. In our configuration, the _master_ is used for jobs that do not take CPU (e.g. waiting for LCG nightly builds, polling, etc.), and the slaves for builds and testing. In particular, we do not have a direct mapping machine-slave, but we have some special partitioning (see LHCbNightliesImplementation). ---+++ master The [[https://buildlhcb.cern.ch/jenkins/computer/(master)/][_master_ node]] is a crucial node which is always _up_. If the master node goes offline the whole system is blocked. ---+++ slaves -- Main.MarcoClemencic - 2015-07-06
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r2
<
r1
|
B
acklinks
|
V
iew topic
|
WYSIWYG
|
M
ore topic actions
Topic revision: r2 - 2015-07-07
-
MarcoClemencic
Log In
LHCb
LHCb Web
LHCb Web Home
Changes
Index
Search
LHCb webs
LHCbComputing
LHCb FAQs
LHCbOnline
LHCbPhysics
LHCbVELO
LHCbST
LHCbOT
LHCbRICH
LHCbMuon
LHCbTrigger
LHCbDetectorAlignment
LHCbTechnicalCoordination
LHCbUpgrade
Public webs
Public webs
ABATBEA
ACPP
ADCgroup
AEGIS
AfricaMap
AgileInfrastructure
ALICE
AliceEbyE
AliceSPD
AliceSSD
AliceTOF
AliFemto
ALPHA
ArdaGrid
ASACUSA
AthenaFCalTBAna
Atlas
AtlasLBNL
AXIALPET
CAE
CALICE
CDS
CENF
CERNSearch
CLIC
Cloud
CloudServices
CMS
Controls
CTA
CvmFS
DB
DefaultWeb
DESgroup
DPHEP
DM-LHC
DSSGroup
EGEE
EgeePtf
ELFms
EMI
ETICS
FIOgroup
FlukaTeam
Frontier
Gaudi
GeneratorServices
GuidesInfo
HardwareLabs
HCC
HEPIX
ILCBDSColl
ILCTPC
IMWG
Inspire
IPv6
IT
ItCommTeam
ITCoord
ITdeptTechForum
ITDRP
ITGT
ITSDC
LAr
LCG
LCGAAWorkbook
Leade
LHCAccess
LHCAtHome
LHCb
LHCgas
LHCONE
LHCOPN
LinuxSupport
Main
Medipix
Messaging
MPGD
NA49
NA61
NA62
NTOF
Openlab
PDBService
Persistency
PESgroup
Plugins
PSAccess
PSBUpgrade
R2Eproject
RCTF
RD42
RFCond12
RFLowLevel
ROXIE
Sandbox
SocialActivities
SPI
SRMDev
SSM
Student
SuperComputing
Support
SwfCatalogue
TMVA
TOTEM
TWiki
UNOSAT
Virtualization
VOBox
WITCH
XTCA
Welcome Guest
Login
or
Register
Cern Search
TWiki Search
Google Search
LHCb
All webs
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use
Discourse
or
Send feedback