TWiki
>
LCG Web
>
WLCGCommonComputingReadinessChallenges
>
WLCGDailyMeetingsWeek150302
(2015-03-05,
MaartenLitmaath
)
(raw view)
E
dit
A
ttach
P
DF
---+!! Week of 150302 %TOC% ---++ WLCG Operations Call details * At CERN the meeting room is [[https://maps.cern.ch/mapsearch/?centerX=2492565¢erY=1121070¢erScale=2500][513]] R-068. * For remote participation we use the Vidyo system. Instructions can be found [[https://indico.cern.ch/conferenceDisplay.py?confId=287280][here]]. ---++ General Information * The purpose of the meeting is: * to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting; * to announce or schedule interventions at Tier-1 sites; * to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites; * to provide important news about the middleware; * to communicate any other information considered interesting for WLCG operations. * The meeting should run from 15:00 until 15:20, exceptionally to 15:30. * The SCOD rota for the next few weeks is at ScodRota * General information about the WLCG Service can be accessed from the [[WLCGOperationsWeb][Operations Web]] * Whenever a particular topic needs to be discussed at the daily meeting requiring information from site or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOT.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting. ---++ Monday Attendance: * local: Alessandro (ATLAS), Jan (storage), Maarten (SCOD + ALICE), Steve (grid services), Zbyszek (databases) * remote: Christian (NDGF), Christoph (CMS), Di (TRIUMF), Felix (ASGC), Lisa (FNAL), Onno (NLT1), Pepe (PIC), Rob (OSG), Rolf (!IN2P3), Sang-Un (KISTI), Tiju (!RAL), Vladimir (LHCb) Experiments round table: * ATLAS [[AtlasComputing.ADCOperationsDailyReports2015][reports]] ( [[AtlasComputing.ADCOperationsDailyReports2015?raw=on][raw view]]) - * Central Services/T0/T1 * FTS3 servers upgrade: we would like that BNL, RAL and CERN to upgrade to the latest FTS server release to fix the activity share issue. * _Steve: the downtime will be less than 5 min; do the upgrade tomorrow (Tue)?_ * _Christoph: OK for CMS_ * _Vladimir: OK for LHCb_ * _Alessandro: we will only re-enable the shares once all 3 FTS for ATLAS have been upgraded; we will coordinate this through the FTS-3 mailing list_ * CMS [[CMS.FacOps_WLCGdailyreports][reports]] ( [[CMS.FacOps_WLCGdailyreports?raw=on][raw view]]) - * Business as usual * Only moderate activity * ALICE - * high activity * !RAL now allow up to 6k concurrent ALICE jobs when other VOs have little activity - thanks! * LHCb [[LHCb.ProductionOperationsWLCGdailyReports][reports]] ( [[LHCb.ProductionOperationsWLCGdailyReports?raw=on][raw view]]) - * Distributed computing dominated by Monte Carlo and user activities. * T0: NTR * T1: NTR * _Vladimir: there are fewer jobs than usual due to a minor issue that appeared after the migration to the DIRAC File Catalog and is being worked on_ Sites / Services round table: * ASGC: ntr * BNL: * CNAF: * FNAL: * the issue of GGUS alarm tickets paging the wrong number has been resolved * !GridPP: * !IN2P3: * reminder of the downtime tomorrow; the ATLAS tape buffer space will be more than sufficient during the intervention * JINR: * KISTI: ntr * KIT: * NDGF: * downtime tomorrow morning for dCache head nodes upgrade * there was a power supply problem at the Copenhagen site; the problem has disappeared, but the root cause has not been found yet * NL-T1: * on Mon March 9 the dCache upgrade from 2.6 to 2.10 will start; the downtime has been declared for 2 days because the DB upgrade might not finish in 1 day * NRC-KI: * OSG: ntr * PIC: * on Tue March 10 there will be a 5h downtime for network upgrades and maintenance as well as a dCache 2.10 patch * !RAL: ntr * RRC-KI: * TRIUMF: ntr * CERN batch and grid services: * the migration from VOMRS to !VOMS-Admin is proceeding as planned; the new service is expected to be available by 17:00 CET (and it is) * CERN storage services: * CASTOR-LHCb will be down for an upgrade this Wed * CASTOR-ALICE will be down for an upgrade next Mon * a bunch of rolling DB updates will soon be announced; they should be fairly transparent from the client side; some daemons may need to be restarted on the server side * Databases: * tomorrow there will be rolling updates of WLCG integration DBs (!INT6R and !INT11R) * the corresponding production DB updates will be done in 1 or 2 weeks * tomorrow there will be a full-day transparent intervention on back-end storage serving half of all DBs * CASTOR, WLCG and experiments are affected in principle * GGUS: * Grid Monitoring: * MW Officer: AOB: ---++ Thursday Attendance: * local: Alberto (grid services), Andrea M (MW Officer), Joel (LHCb), Maarten (SCOD + ALICE) * remote: Christian (NDGF), Dennis (NLT1), Di (TRIUMF), Felix (ASGC), John (!RAL), Jose (CMS), Lisa (FNAL), Michael (BNL), Pepe (PIC), Rob (OSG), Rolf (!IN2P3), Sang-Un (KISTI), Thomas (KIT), Vladimir (LHCb) Experiments round table: * ATLAS [[AtlasComputing.ADCOperationsDailyReports2015][reports]] ( [[AtlasComputing.ADCOperationsDailyReports2015?raw=on][raw view]]) - * Central Services/T0/T1 * NTR * CMS [[CMS.FacOps_WLCGdailyreports][reports]] ( [[CMS.FacOps_WLCGdailyreports?raw=on][raw view]]) - * Nothing to report. No significant problems since Monday. * ALICE - * NTR * LHCb [[LHCb.ProductionOperationsWLCGdailyReports][reports]] ( [[LHCb.ProductionOperationsWLCGdailyReports?raw=on][raw view]]) - * Distributed computing dominated by Monte Carlo and user activities. * T0: NTR * T1: * GRIDKA: problem with pilot submission; WARNING Downtime * _see KIT report below_ * IN2P3: Last downtime did not include SEs, but they were unavailable * _Rolf: the SE services should have been available as far as we could see. We did notice some LHCb disk space getting full during the downtime and alerted our LHCb contact._ * _Joel: the user space got full indeed, but our issue was not with that._ * _Vladimir: the SE remained affected after all supposedly relevant downtimes had finished. We did not open a ticket yet. It probably is OK now._ * _Rolf: we will look further into it._ * _Joel: the VOMRS to !VOMS-Admin migration went OK and the new service was working fine until Tue around lunchtime, when DIRAC was no longer able to talk to it._ * _The service had picked up a new version of Java that disables SSLv3 by default._ * _On the DIRAC side TLSv1 was tried, but that failed with another error._ * _Proposal: try downgrading Java or re-enabling SSLv3 on the old voms.cern.ch as a test?_ * _Update: re-enabling SSLv3 did the trick for now. The DIRAC developers will be asked to look into the SSLv3 dependency with some urgency._ Sites / Services round table: * ASGC: ntr * BNL: * a complex intervention on Tue was completed successfully: maintenance on dCache, network and SE HW * CNAF: * FNAL: ntr * !GridPP: * !IN2P3: * the scheduled downtime for batch took much longer than planned: * an unplanned emergency power cut interfered * the new version could not be made to work and had to be rolled back in the end * now the site is working again (still with the previous version) * JINR: * KISTI: * KISTI will have a scheduled downtime from 9 March 2015 00:00(UTC) to 10 March 2015 09:00 (UTC) for re-cabling work on nodes. Tier-1 services are supposed not to be affected by this intervention. * KIT: * there have been network issues lasting ~2 days: * a second 10G link to CERN was added * this caused intermittent routing issues * they were thought to be fixed yesterday evening, but further problems were seen this morning * for now the traffic only uses the first link * _Vladimir: your site looks OK now for LHCb_ * NDGF: * the dCache intervention on Tue went OK * still power problems in Copenhagen: * 1 phase is OK and the SE nodes only use that one * the other phases are used for CE services * NL-T1: ntr * OSG: ntr * PIC: ntr * !RAL: ntr * TRIUMF: ntr * CERN batch and grid services: * !VOMS-admin migration happened on Monday. * The migration went well, but a few groups of users lost certificates during the migration. We were able to recover most of them but there is still work in progress. * Please, report any strange behavior to GGUS:110227 * CERN storage services: * Databases: * GGUS: * Grid Monitoring: * MW Officer: ntr AOB:
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r10
<
r9
<
r8
<
r7
<
r6
|
B
acklinks
|
V
iew topic
|
WYSIWYG
|
M
ore topic actions
Topic revision: r10 - 2015-03-05
-
MaartenLitmaath
Log In
LCG
LCG Wiki Home
LCG Web Home
Changes
Index
Search
LCG Wikis
LCG Service
Coordination
LCG Grid
Deployment
LCG
Apps Area
Public webs
Public webs
ABATBEA
ACPP
ADCgroup
AEGIS
AfricaMap
AgileInfrastructure
ALICE
AliceEbyE
AliceSPD
AliceSSD
AliceTOF
AliFemto
ALPHA
Altair
ArdaGrid
ASACUSA
AthenaFCalTBAna
Atlas
AtlasLBNL
AXIALPET
CAE
CALICE
CDS
CENF
CERNSearch
CLIC
Cloud
CloudServices
CMS
Controls
CTA
CvmFS
DB
DefaultWeb
DESgroup
DPHEP
DM-LHC
DSSGroup
EGEE
EgeePtf
ELFms
EMI
ETICS
FIOgroup
FlukaTeam
Frontier
Gaudi
GeneratorServices
GuidesInfo
HardwareLabs
HCC
HEPIX
ILCBDSColl
ILCTPC
IMWG
Inspire
IPv6
IT
ItCommTeam
ITCoord
ITdeptTechForum
ITDRP
ITGT
ITSDC
LAr
LCG
LCGAAWorkbook
Leade
LHCAccess
LHCAtHome
LHCb
LHCgas
LHCONE
LHCOPN
LinuxSupport
Main
Medipix
Messaging
MPGD
NA49
NA61
NA62
NTOF
Openlab
PDBService
Persistency
PESgroup
Plugins
PSAccess
PSBUpgrade
R2Eproject
RCTF
RD42
RFCond12
RFLowLevel
ROXIE
Sandbox
SocialActivities
SPI
SRMDev
SSM
Student
SuperComputing
Support
SwfCatalogue
TMVA
TOTEM
TWiki
UNOSAT
Virtualization
VOBox
WITCH
XTCA
Welcome Guest
Login
or
Register
Cern Search
TWiki Search
Google Search
LCG
All webs
Copyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use
Discourse
or
Send feedback