WLCGDailyMeetingsWeek130812 < LCG

LCG Web>WLCGCommonComputingReadinessChallenges>WLCGOperationsMeetings>WLCGDailyMeetingsWeek130812 (2013-08-15, MaartenLitmaath) (raw view)
EditAttachPDF
---+!! Week of 130812

%TOC%

---++ Daily WLCG Operations Call details

To join the call, at 15.00 CE(S)T Monday to Friday inclusive (in CERN 513 R-068) do one of the following:

   1. Dial +41227676000 (Main) and enter access code 0119168, or
   2. To have the system call you, click [[https://audioconf.cern.ch/call/0119168][here]]

The scod rota for the next few weeks is at ScodRota

---++ WLCG Availability, Service Incidents, Broadcasts, Operations Web

| *VO Summaries of Site Usability* ||||*SIRs* |*Broadcasts* |*Operations Web* |
| [[http://dashb-alice-sum.cern.ch/dashboard/request.py/historicalsmryview-sum#view=siteavl&time%5B%5D=lastWeek&profile=ALICE_CRITICAL&group=all%2Bsites&site%5B%5D=CCIN2P3&site%5B%5D=CERN&site%5B%5D=CNAF&site%5B%5D=FZK&site%5B%5D=NIKHEF&site%5B%5D=RAL&site%5B%5D=SARA&type=quality][ALICE]] | [[http://dashb-atlas-sum.cern.ch/dashboard/request.py/historicalsmryview-sum#view=siteavl&time%5B%5D=lastWeek&profile=ATLAS_CRITICAL&group=All%2Bsites&site%5B%5D=BNL-ATLAS&site%5B%5D=CERN-PROD&site%5B%5D=FZK-LCG2&site%5B%5D=IN2P3-CC&site%5B%5D=INFN-T1&site%5B%5D=NDGF-T1&site%5B%5D=NIKHEF-ELPROD&site%5B%5D=pic&site%5B%5D=RAL-LCG2&site%5B%5D=SARA-MATRIX&site%5B%5D=Taiwan-LCG2&site%5B%5D=TRIUMF-LCG2&type=quality][ATLAS]] | [[http://dashb-cms-sum.cern.ch/dashboard/request.py/historicalsmryview-sum#view=siteavl&time%5B%5D=lastWeek&profile=CMS_CRITICAL_FULL&group=Tier1s%2B%252B%2BTier0&site%5B%5D=T0_CH_CERN&site%5B%5D=T1_CH_CERN&site%5B%5D=T1_DE_KIT&site%5B%5D=T1_ES_PIC&site%5B%5D=T1_FR_CCIN2P3&site%5B%5D=T1_IT_CNAF&site%5B%5D=T1_TW_ASGC&site%5B%5D=T1_UK_RAL&site%5B%5D=T1_US_FNAL&type=quality][CMS]] | [[http://dashb-lhcb-sum.cern.ch/dashboard/request.py/historicalsmryview-sum#view=siteavl&time%5B%5D=lastWeek&profile=LHCb_CRITICAL&group=Tier%2B0/1&site%5B%5D=LCG.CERN.ch&site%5B%5D=LCG.CNAF.it&site%5B%5D=LCG.GRIDKA.de&site%5B%5D=LCG.IN2P3.fr&site%5B%5D=LCG.NIKHEF.nl&site%5B%5D=LCG.PIC.es&site%5B%5D=LCG.RAL.uk&site%5B%5D=LCG.SARA.nl&type=quality][LHCb]] | [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGServiceIncidents][WLCG Service Incident Reports]] |  [[https://operations-portal.egi.eu/broadcast/archive][Broadcast archive]] | [[WLCGOperationsWeb][Operations Web]] |
---++ General Information

| *General Information* ||| *GGUS Information* | *LHC Machine Information* | 
| [[http://itssb.web.cern.ch/][CERN IT status board]] | [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions][WLCG Baseline Versions]] | [[http://cern.ch/planet-wlcg][WLCG Blogs]] | GgusInformation | [[https://espace.cern.ch/be-dep-op-lhc-machine-operation/default.aspx][Sharepoint site]] - [[http://op-webtools.web.cern.ch/op-webtools/vistar/vistars.php?usr=LHC1][LHC Page 1]] |


<HR>

---++ Monday

Attendance:

   * local: Simone (SCOD), Ivan (WLCG Monitoring), Luca (CERN Databases), Luca (CERN Storage)
   * remote: Dimitri (KIT), Sang-Un (KISTI), David (CMS), Michael (BNL), Tiju (RAL), Matteo (CNAF), Wei-Jen (ASGC), Onno (NL-T1), Pavol (ATLAS), Rob (OSG), Ulf (NDGF)

Experiments round table:

   * ATLAS [[https://twiki.cern.ch/twiki/bin/view/Atlas/ADCOperationsDailyReports2013][reports]] ([[https://twiki.cern.ch/twiki/bin/view/Atlas/ADCOperationsDailyReports2013?raw=on][raw view]]) -
      * T0
         * CERN-PROD GGUS:96524 FTS2 channel CERN->ASGC again stuck at 04:00 in the morning on Friday, after some investigation some spurious agent was removed in the afternoon, but it took around 48 hours, until backlog was cleared.
         * CERN EOS GGUS:96519 Atlas EOS instance had problems (memory issues), restarted.
      * T1
         * FZK (GGUS:96245) : Transfer problems from/to FZK with different sites affecting a fraction of transfers. Under investigation.

   * CMS [[https://twiki.cern.ch/twiki/bin/view/CMS/FacOps_WLCGdailyreports][reports]] ([[https://twiki.cern.ch/twiki/bin/view/CMS/FacOps_WLCGdailyreports?raw=on][raw view]]) -
      * MC production and rereconstruction continue
      * GGUS:96546/INC:356501 CMSEOS files not manifesting in global xrootd redirector, but are visible directly in eoscms.cern.ch -- possibly similar issue on May 30.
         * Luca: cmsd daemon had a problem  in reading a config file after a restart. Fixed.
      * GGUS:96504 User with possibly expired certificate
      * GGUS:96482 Transfers from Caltech to T1_UK_RAL -- investigation continues.
      * GGUS:96559 Hammercloud failures reading files at ASGC -- in progress
         * Wei-Jen: ASGC failed HC due to an expired host certificate. A new one has been requested. 

   * ALICE -
      * NTR

   * LHCb [[https://twiki.cern.ch/twiki/bin/view/LHCb/ProductionOperationsWLCGdailyReports][reports]] ([[https://twiki.cern.ch/twiki/bin/view/LHCb/ProductionOperationsWLCGdailyReports?raw=on][raw view]]) -
      * NTR

Sites / Services round table:

   * NL-T1: SARA had one pool node in HW maintenance this morning. Some files were unavailable. 
   * ASGC: scheduled downtime tonight for 1 day for network hardware upgrade. 
   * NDGF: problem with SRM during the weekend. 1/2 hour downtime between saturday and sunday

AOB: 

---++ Thursday

Attendance:
   * local: Simone (SCOD), Ivan (WLCG Monitoring), Luca (CERN Databases), Luca (CERN Storage), Alex (CERN Grid Services), Vitor (CERN Grid Services)  
   * remote: Michael (BNL), !WooJin (KIT), David (CMS), John (RAL), Jeremy (GridPP), Rob (OSG), Wei-Jen (ASGC), Ulf (NDGF)  

Experiments round table:

   * ATLAS [[https://twiki.cern.ch/twiki/bin/view/Atlas/ADCOperationsDailyReports2013][reports]] ([[https://twiki.cern.ch/twiki/bin/view/Atlas/ADCOperationsDailyReports2013?raw=on][raw view]]) -

   * CMS [[https://twiki.cern.ch/twiki/bin/view/CMS/FacOps_WLCGdailyreports][reports]] ([[https://twiki.cern.ch/twiki/bin/view/CMS/FacOps_WLCGdailyreports?raw=on][raw view]]) -
      * Relatively light activity -- primarily upgrade MC production
      * No new GGUS tickets -- GGUS:96482 (Caltech/RAL transfers) waiting for more info, CMS transfer team will follow up.

   * ALICE -
      * NTR

   * LHCb [[https://twiki.cern.ch/twiki/bin/view/LHCb/ProductionOperationsWLCGdailyReports][reports]] ([[https://twiki.cern.ch/twiki/bin/view/LHCb/ProductionOperationsWLCGdailyReports?raw=on][raw view]]) -
      * Mostly MC productions ongoing, tail of reprocessing and restripping campaign
      * T0:
         * CERN: NTR
      * T1:
         * Recovered from network interruptions at RAL earlier in the week
         * Local transfer failures at IN2P3 and SARA resolved (SRM overloads?)

Sites / Services round table:

   * WLCG Monitoring:
      * WLCG Transfers Dashboard: a new dashboard prototyping the future evolution of the WLCG Transfers Dashboard has been deployed. http://dashb-wlcg-transfers-new.cern.ch/
         * follows a hierarchical architecture designed to provide a common feature set independent of transfer protocol, while delegating to FTS and XRootD Dashboards for protocol-specific features
         * includes monitoring of ALICE XRootD data traffic. 
         * The current production WLCG Transfers Dashboard remains available. http://dashb-wlcg-transfers.cern.ch/
      * ATLAS DDM Dashboard: monitoring of on-demand transfers for ATLAS (dq2-get / dq2-put) has been deployed to the integration version of ATLAS DDM Dashboard (http://dashb-atlas-data-soup-tbed.cern.ch/ddm2/#activity=%288%29). This feature is currently undergoing validation by ATLAS before release to production.
   * CERN Storage: 
      * Tue 20 there will be the upgrade of CASTOR  oracle backend. Transparent. 
   * Grid services: 
      * the batch batch farm nodes will be reinstalled in a rolling fashion in the next days (transparent)

AOB:
Topic revision: r9 - 2013-08-15 - MaartenLitmaath
LCG Wikis
LCG Service
Coordination
LCG Grid
Deployment
LCG
Apps Area
Public webs
Welcome Guest
- Cern Search
- TWiki Search
- Google Search
LCG All webs
Copyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback