Week of 120227

Daily WLCG Operations Call details

To join the call, at 15.00 CE(S)T Monday to Friday inclusive (in CERN 513 R-068) do one of the following:

  1. Dial +41227676000 (Main) and enter access code 0119168, or
  2. To have the system call you, click here
  3. The scod rota for the next few weeks is at ScodRota

WLCG Service Incidents, Interventions and Availability, Change / Risk Assessments

VO Summaries of Site Usability SIRs, Open Issues & Broadcasts Change assessments
ALICE ATLAS CMS LHCb WLCG Service Incident Reports WLCG Service Open Issues Broadcast archive CASTOR Change Assessments

General Information

General Information GGUS Information LHC Machine Information
CERN IT status board M/W PPSCoordinationWorkLog WLCG Baseline Versions WLCG Blogs   GgusInformation Sharepoint site - Cooldown Status - News


Monday

Attendance: local(John, Michael, Dan, Ivan, Eva, Ale, MariaDZ, MariaG, Dirk);remote(Rob/OSG, Onno/NL-T1, Michael/BNL, Oliver/CMS, Rolf/IN2P3,Burt/FNAL, Roger/NDGF, Gonzalo, Jhen-Wei, Anonymous, Dimitri/KIT, Vladimir/LHCb, Tiju/RAL,Lorenzo/CNAF).

Experiments round table:

  • ATLAS reports -
    • T0
      • NTR
    • T1
      • FZK: GGUS:79653 Jobs failing with "dccp failed: [Errno 3] No such process".
      • pic: SRM problems GGUS:79677 pic experts are investigating: "it seems an overload on PNFS"

  • CMS reports -
    • Tier-0:
      • Cosmics data taking for the next 3 days
    • Processing activities:
      • 8 TeV MC simulation on Tier-1 sites, ramping up on Tier-2 sites
    • Issues:
      • GGUS:79648: T0_CH_CERN: Tier-0 node (probably) failing a lot of jobs, last updated 2012-02-27
      • GGUS:79259: T1_IT_CNAF: request to update FTSMonitor, will be handled within 2-4 weeks due to separation from lemon that runs on the same server, last update 2012-02-21
      • GGUS:79257: T1_DE_KIT: request to update FTSMonitor, will be looked at next week, last update 2012-02-16
      • GGUS:78872: ROC_CERN: REBUS gets OSG capacity numbers from bdii and it was agreed that they should get from MyOSG, trying to assign to REBUS developers via ROC_CERN, last update 2012-02-27
    • Next on shift and attending this call: Pepe Flix

  • LHCb reports -
    • Experiment activities:
      • User analysis, Validation productions for restripping
    • T0
      • CERN : CASTOR problem to access som efiles : (GGUS:79629)
    • T1
      • PIC : Request for space token migration (GGUS:79305)
      • GridKa : Request for space token migration (GGUS:79303) "nearly finished"
      • SARA : Request for space token migration (GGUS:79307)
    • T2
      • LAL (GGUS:79200).ongoing
      • MariaDZ - is the communication on this ticket satisfactory? Vladimir: yes, several updates also during weekend not yet fixed, but communication has been established.

Sites / Services round table:

  • Onno/NL-T1 - this morning a pool node in SRM was restarted after two hours. We will set parameter to restart automatically after kernel panic. This should become active in a few week from now, when it can be bundled with the next downtime.
  • Michael/BNL - ntr
  • Rolf/IN2P3 - ntr
  • Burt/FNAL - ntr
  • Roger/NDGF - network problem on friday will affect some Norwegian sites, but maybe too short to be noticeable (evening of tomorrow and wed).
  • Gonzalo/PIC - new ggus ticket from atlas: first investigations shows pnfs overload (also high load on weekend). Site is till looking into it and will report back via GGUS.
  • Jhen-Wei/ASGC - ntr
  • Dimitri/KIT - full site in maintenance 13 march 4am 15 march 11:00 am (UTC) already in gocdb. Oliver: why a two day outage? Dimitri: full site maintenance with various cluster related works.
  • Tiju/RAL - updated successfully CASTOR for LHCb to 2.1.11-8
  • Lorenzo/CNAF - ntr
  • Rob/OSG - GGUS upgrade happened today: OSG will upgrade for attachment problem (GGUS:78844) - should have fix in place by end of the day
  • Eva/CERN: short intervention on LHCBR went fine. Next Thu CMS online cluster will be moved back to CMS pit,
  • Mike: SAM did not receive any data since 9am this morning (GGUS:79683). Experiment and sites can now find sam info via newly created links on dashboard and SAM pages.

AOB: (MariaDZ) File ggus-tickets.xls is up-to-date and attached to page WLCGOperationsMeetings . We had 1 real ALARM GGUS:79623 last week by CMS on Castor SRM (CERN), now in status 'verified'. GGUS Release today. Please observe the Did you know?... text today about automatic ticket closure (a première for GGUS)! 13 test ALARMs opened so far to all European sites and ASGC, all in status 'solved'.

  • Ale: FTS 2.2.8 deployment: thanks to T1s we have all suggested dates for upgrades at the T1s(IN2P3 info expected later today). Plan to review all dates on next T1 service coord meeting.

  • MariaG: The next T1 coordination meeting is being planned for next Thu. Please send additional points for agenda as needed. The draft agenda will go out shortly.

Tuesday

Attendance: local(Ale, Michael, Ivan, Maarten, Eva, Steve, MariaDZ, Dirk);remote(Gonzalo/PIC, Paco/NL-T1, Lisa/FNAL, Michael/BNL,Stefano/CNAF, Rolf/IN2P3, Jhen-Wei/ASGC, Pepe/CMS, Roger/NDGF, Jeremy/GridPP,Xavier/KIT,Vladimir/LHCb ).

Experiments round table:

  • ATLAS reports -
    • T1:
      • pic LFC migration in progress: ES cloud offline in panda and blacklisted in DDM
  • CMS reports -
    • Tier-0 (plan):
      • Cosmics data taking, to stop tomorrow afternoon. Discussion if we want to run over the weekend... Monday we will start cosmics with magnetic field.
    • Processing activities:
      • 8 TeV MC simulation on Tier-1 sites, ramping up on Tier-2 sites
    • Tier-0/Tier-1 issues:
      • [in progress] GGUS:79648 (SAV:126644): T0_CH_CERN: Tier-0 node (probably lxbrl2316) failing a lot of jobs, last updated 2012-02-27
      • [in progress] GGUS:79259 (SAV:126400): T1_IT_CNAF: request to update FTSMonitor, will be handled within 2-4 weeks due to separation from lemon that runs on the same server, last update 2012-02-21
      • [in progress] GGUS:79257 (SAV:126398): T1_DE_KIT: request to update FTSMonitor, will be looked at next week, last update 2012-02-16
    • Services and Infrastructure issues:
      • [in progress] GGUS:78872: ROC_CERN: REBUS gets OSG capacity numbers from bdii and it was agreed that they should get from MyOSG. Today it was assigned to GStat 2nd Line Support, last update 2012-02-28
      • [new] GGUS:79719: Some CMS SAM tests not published in SAM for several days. Affected site: RWTH-Aachen, from 21/2 to 27/2 two CMS WN tests, org.cms.WN-analysis and org.cms.WN-basic, were not published. Assigned (SAM/Nagios), last update 2012-02-28
      • [new + phone call] Last night around 04:30 CMSR database was hanging. The problems were identified and service was restored in 1h. At the Tier-0 we got no permanent failures due to that, all the jobs went through in the retries.
    • Notes:
      • March 1st: proposed date to migrate CMSONR and CMSR databases to 11g. Will inform everyone and make sure that CMS systems are shut down before 10 AM on Thursday.
    • Current CRC shifter: Pepe Flix

  • ALICE reports -
    • CREAM nodes at CERN publish bad job counts, GGUS:79695 opened yesterday evening, no news as of 14:00 today; situation persists, not a big problem yet

  • LHCb reports -
    • Experiment activities:
      • User analysis, Validation productions for restripping
    • T0
      • CERN : CASTOR problem to access some files : (GGUS:79629)
      • CERN : setting the variable TMPDIR : (GGUS:79685)
    • T1
      • PIC : Last patches installed at EMI WMS
      • PIC : Request for space token migration (GGUS:79305)
      • GridKa : Request for space token migration (GGUS:79303) "nearly finished"
      • SARA : Request for space token migration (GGUS:79307)
    • T2

Sites / Services round table:

  • Gonzalo/PIC -ntr
  • Paco/NL-T1 - new ticket from LHCb in progress
  • Lisa/FNAL - ntr
  • Michael/BNL -ntr
  • Stefano/CNAF - ntr (fts upgrade schedule has been declared)
  • Rolf/IN2P3 - ntr
  • Jhen-Wei/ASGC - ntr
  • Roger/NDGF - ntr
  • Xavier/KIT - since 12am access to tape back-end got stuck - working on it
  • Jeremy/GridPP - ntr
  • Rob/OSG - are in bi-monthly maintenance - should be transperent to WLCG services
  • Eva/CERN - CMS prod and COMPASS DBs affected by failure on back-end storage caused by high load on NAS controller - being investigated with NetApp support

AOB: After a very long period of continuous effort, GGUS Internal Diary entries are now acceptable in SNOW as "Additional Comments". Please send GGUS ticket numbers which do not receive appropriate attention concerning experiments or Tier1s to MariaDZ by Wednesday 4pm CET for the usual presentation at the T1SCM.

  • Ale: meeting with sam team and experiments: results not published for seven days. Question to T1 sites: did you notice nagios processes after probe has finished? Please report back to Ale.

Wednesday

Attendance: local(Steve, Ale, Maarten, Ivan, Luca, Massimo, MariaDZ, Dirk);remote(Michael/BNL, Gonzalo/PIC, Xavier/KIT, Roger/NDGF, Pepe/CMS, Rolf/IN2P3, Tiju/RAL, Burt/FNAL, Ron/NL-T1, Vladimir/LHCb ).

Experiments round table:

  • ATLAS reports -
    • Nothing to report

  • CMS reports -
    • Tier-0 (plan):
      • Cosmics data taking stopped already. The rest of the afternoon for subsystem tests. On Friday magnet will perform a test ramp up to 1T and then back down to 0T, on Monday morning the complete ramp up to 3.8 T will take place (+Cosmics).
    • Processing activities:
      • 8 TeV MC simulation on Tier-1 sites, ramping up on Tier-2 sites
    • Tier-0/Tier-1 issues:
      • [in progress] GGUS:79648 (SAV:126644): T0_CH_CERN: Tier-0 node (probably lxbrl2316) failing a lot of jobs, last updated 2012-02-27
      • [in progress] GGUS:79259 (SAV:126400): T1_IT_CNAF: request to update FTSMonitor, will be handled within 2-4 weeks due to separation from lemon that runs on the same server, last update 2012-02-21
      • [in progress] GGUS:79257 (SAV:126398): T1_DE_KIT: request to update FTSMonitor, will be looked at next week, last update 2012-02-16
      • [new/closed]SAV:126744: T1_FR_CCIN2P3: Custodial Data Transfer request not approved since days, last update 2012-02-29
      • [new]GGUS:79754 (SAV:126742): T1_TW_ASGC: unmerged files not found, last update 2012-02-29
      • [new/closed]GGUS:79747 (SAV:126741): T1_IT_CNAF: Transfers to CNAF failing. The proxy renewal daemon was dead, last update 2012-02-29
    • Services and Infrastructure issues:
      • [in progress] GGUS:78872: ROC_CERN: REBUS gets OSG capacity numbers from bdii and it was agreed that they should get from MyOSG. Was assigned to GStat 2nd Line Support. Reported there is currently no GGUS Support Unit for REBUS. The issue about handling OSG shares is known and tracked in https://tomtools.cern.ch/jira/browse/APPS-130, a ticket created in Nov/2011, with no progress at all... last update 2012-02-29
      • [in progress] GGUS:79719: Some CMS SAM tests not published in SAM for several days. Affected site: RWTH-Aachen, from 21/2 to 27/2 two CMS WN tests, org.cms.WN-analysis and org.cms.WN-basic, were not published. Assigned (SAM/Nagios), last update 2012-02-28
    • Notes:
      • March 1st: proposed date to migrate CMSONR and CMSR databases to 11g. Everyone informe to make sure that CMS systems are shut down before 10 AM on Thursday.
    • Maarten: transfer problems to CNAF: is this affected node the phedex one? yes. Maarten: new version of this component is available which is successfully run already on the Alice vobox. Other sites with problems are invited to upgrade.
  • ALICE reports -
    • Yesterday's ticket GGUS:79695 has been fixed today.

  • LHCb reports -
    • Experiment activities:
      • User analysis, Validation productions for restripping
    • T0
      • CERN : CASTOR problem to access some files : (GGUS:79629)
      • CERN : setting the variable TMPDIR : (GGUS:79685)
    • T1
    • T2
    • Both Gonzalo/PIC and Ron/NL-T1 acknowledge the respective tickets and expect work on them to progress soon.

Sites / Services round table:

  • Michael/BNL - will have network intervention on Tue morning from 11:00 until 17:00 UTC with a border router replacement. This is expected to cause service outages. BNL is in contact already with Edoardo at CERN to coordinate. Need to explore how to implement policy based routing in cooperation with CERN. Will give more detail on twiki once available.

When the current router goes down, the CERN links and the dedicated end-to-end links connecting Tier-2 sites in the US will go down. At this point, all traffic will attempt to be routed over the generic IP internet circuits that are connected to the new Cisco Nexus we installed last week. Since there is no Policy Based Routing (PBR) available, the traffic that would normally use the LHCOPN links will most likely be dropped at the receiving end. Specifically IP traffic from BNL on 130.199.185.0, 130.199.48.0, and 192.12.15.0 might not be accepted at CERN since the traffic is being received over a different circuit. The Tier-2 end-to-end traffic should be delivered over the generic IP circuits since those links are not subject to any Policy Based Routing constraints. Atlas traffic that uses the generic Internet should also continue to flow unaffected. During the migration to the Nexus platform, our first priority is to verify the PBR and failover capabilities for the CERN links both the primary and backup. If the failover and PBR capabilities perform as expected, and we verify it with Edoardo at CERN, then we can complete the migration. We should have these results in a few hours since testing the CERN links is a manual process. Worst case expectation is a three hour outage for the CERN links until the testing is completed. Assuming all goes well with the CERN testing we can complete the rest of the migration. This would include attaching the secondary perimeter firewall, the secondary F5 loadbalancer, and reconnecting the Tier-2 end-to-end links. These last steps should not be disruptive and should be completed within the remainder of the six hour window.

  • Gonzalo/PIC - ntr
  • Xavier/KIT - tape access problems re-occured. Site has a repair receipt but don't yet understand root cause. The vendor has been contacted for further analysis.
  • Roger/NDGF - ntr
  • Rolf/IN2P3 - yesterday afternoon some jobs failed due to a cvmfs misconfiguration. fixed
  • Tiju/RAL - Castor instance for ALICE has been updated successfully
  • Burt/FNAL - ntr
  • Ron/NL-T1 - ntr
  • Rob/OSG - attachment problem is now solved (tested) after patches being in place.

AOB: (MariaDZ) Progress on the proposal to test all GGUS interfaces with external ticketing systems can be found in Savannah:126505 Rob (OSG) suggested that such tests be done on the training system BEFORE the release date. Please think about it and lets discuss at our next SL meeting.

Thursday

Attendance: local(MariaG, Ale, Doug, Ivan, Nicolo, Massimo, Andrea, Eva, Steve, Stefan, Dirk);remote(Michael/BNL, Gonzalo/PIC, Rolf/IN2P3, Kyle/OSG, Roger/NDGF, Jeremy/GridPP, Gareth/RAL, Vladimir/LHCb, Lisa/FNAL, Ronald/NL-T1, Andreas/KIT).

Experiments round table:

  • ATLAS reports -
    • T0/1:
      • pic LFC migration successfully finished.
      • FZK-LCG2 file access issue GGUS:79801 (just few files not accessible)
      • CERN-PROD one tape with some problems GGUS:79788
      • TRIUMF-LCG2 GGUS:79742 one disk server problematic, now fixed, thanks.
    • general for all the ATLAS sites: in order to automate the exlcusion/inclusion of resources, the Production Functional Test have been activated for CA, FR, DE, IT, NL, ES, TW and UK

  • CMS reports -
    • Tier-0 (plan):
      • Monday morning the complete ramp up to 3.8 T will take place. Then, Cosmics.
    • Processing activities:
      • 8 TeV MC simulation on Tier-1 sites, ramping up on Tier-2 sites
    • Tier-0/Tier-1 issues:
      • [in progress+] GGUS:79648 (SAV:126644): T0_CH_CERN: Tier-0 node (probably lxbrl2316) failing a lot of jobs. Apparently it has some sort of memory allocation problem. last updated 2012-02-29
      • [in progress] GGUS:79259 (SAV:126400): T1_IT_CNAF: request to update FTSMonitor, will be handled within 2-4 weeks due to separation from lemon that runs on the same server, last update 2012-02-21
      • [in progress+] GGUS:79257 (SAV:126398): T1_DE_KIT: request to update FTSMonitor, experts already planning the upgrade, last update 2012-02-29
      • [closed+] GGUS:79754 (SAV:126742): T1_TW_ASGC: unmerged files not found. The disk server suffered network problem during the night. Resubmission went through, last update 2012-02-29
      • [new/closed+] SAV:126770: T1_TW_ASGC: JobRobot failures, caused by a disk server problem, last update 2012-02-29
      • [new+] GGUS:79799 (SAV:126762): T1_FR_CCIN2P3: JobRobot failures and SAM failures yesterday from 15h to 23h, last update 2012-02-29
      • [new/closed+] SAV:126759: T1_UK_RAL: JobRobot failures. Some jobs became stuck in the IDLE state, known bug in CREAM CEs?, last update 2012-02-29
    • Services and Infrastructure issues:
      • [in progress+] GGUS:78872: ROC_CERN: REBUS gets OSG capacity numbers from bdii and it was agreed that they should get from MyOSG. Was assigned to GStat 2nd Line Support. Reported there is currently no GGUS Support Unit for REBUS. The issue about handling OSG shares is known and tracked in https://tomtools.cern.ch/jira/browse/APPS-130, a ticket created in Nov/2011, with no progress at all... The GGUS was set to 'unsolved', last update 2012-02-29
      • [in progress] GGUS:79719: Some CMS SAM tests not published in SAM for several days. Affected site: RWTH-Aachen, from 21/2 to 27/2 two CMS WN tests, org.cms.WN-analysis and org.cms.WN-basic, were not published. Assigned (SAM/Nagios), last update 2012-02-28
    • Notes:
      • Today's migration of CMSONR and CMSR databases to 11g went fine from CMS p.o.v. No complains so far...
    • Rolf/IN2P3: CMS hit submission limit, which can be configured. Problem should be fixed… BTW: Ticket has only be created today? Pepe: yes, the shifter did a mistake in forwarding.

  • LHCb reports -
    • Experiment activities:
      • MC, User analysis, Validation productions for restripping
    • T0
      • CERN : CASTOR problem to access some files : (GGUS:79629) Solved
      • CERN : setting the variable TMPDIR : (GGUS:79685)
    • T1
    • T2
    • Ronald/NL-T1 - still investigating reason for CVMFS problems: had problems with squid servers this morning, but this does not explain earlier LHCb problems. Investigation is continuing.

Sites / Services round table:

  • Michael/BNL - ntr
  • Gonzalo/PIC - ntr
  • Rolf/IN2P3 - ntr
  • Kyle/OSG - ntr
  • Roger/NDGF - ntr
  • Jeremy/GridPP - ntr
  • Gareth/RAL - ntr
  • Lisa/FNAL - ntr
  • Ronald/NL-T1 - ntr
  • Andreas/KIT - ntr
  • Massimo/CERN - incident during upgrade on CMS EOS: some files were unavailable from 11:00-12:00.

AOB:

Friday

Attendance: local(Massimo/Storage, Alessandro/ATLAS, Ivan/Dashboard, Steve/GridService, Eva/DB); remote(Ulf/NDGF, Alexandre/NLT1, Gonzalo/PIC, Lisa/FNAL, John/RAL, Pepe/CMS, Lorenzo/CNAF, Pavel/KIT, Rob/OSG, Rolf/IN2P3, Michael/BNL, Vladimir/LHCb).

Experiments round table:

  • ATLAS reports -
    • T0/1:
      • FZK-LCG2 storage issue GGUS:79816 "the storage is under heavy load"
      • CERN-PROD about GGUS:79788 "one file not accessible on one tape": should we declare that file lost?
      • NDGF-T1 GGUS:79811 " We missed an announced network service window affecting the Swedish and Finnish parts of the NDGF-T1 storage. The estimated outage was 15 minutes for OTDR measurements in a 4 hour window. The network has been down for 4.5 hours now and the service window closed 30 minutes ago. We have no idea of what's going on. Contacting supplier. "

  • CMS reports -
    • Tier-0 (plan):
      • Monday morning the complete ramp up to 3.8 T will take place. Then, Cosmics.
    • Processing activities:
      • 8 TeV MC simulation on Tier-1 sites, ramping up on Tier-2 sites
    • Tier-0/Tier-1 issues:
      • [in progress] GGUS:79648 (SAV:126644): T0_CH_CERN: Tier-0 node (probably lxbrl2316) failing a lot of jobs. Apparently it has some sort of memory allocation problem. last updated 2012-02-29
      • [in progress] GGUS:79259 (SAV:126400): T1_IT_CNAF: request to update FTSMonitor, will be handled within 2-4 weeks due to separation from lemon that runs on the same server, last update 2012-02-21
      • [in progress] GGUS:79257 (SAV:126398): T1_DE_KIT: request to update FTSMonitor, experts already planning the upgrade, last update 2012-02-29
      • [in progress+] GGUS:79799 (SAV:126762): T1_FR_CCIN2P3: Misbehaving CREAM-CE, needs an upgrade. CMS drained jobs to that CE, they can proceed, ast update 2012-03-02
      • [new+] GGUS:79719 (SAV:126799): T1_FR_CCIN2P3: Files not stagged, cannot be accessed. Apparently, all files in a directory, last update 2012-03-02
      • [unnoticed to me!+] SAV:126368: T1_TW_ASGC: Backfill dataset (not meant to go to tape) was accepted to be transferred to Tape. The migration is stuck. See ticket, to solve this incident we might delete the dataset, which will clear the request and the queue... last update 2012-02-15(!)
    • Services and Infrastructure issues:
      • [unsolved+] GGUS:78872: ROC_CERN: REBUS gets OSG capacity numbers from bdii and it was agreed that they should get from MyOSG. Was assigned to GStat 2nd Line Support. Reported there is currently no GGUS Support Unit for REBUS. The issue about handling OSG shares is known and tracked in https://tomtools.cern.ch/jira/browse/APPS-130, a ticket created in Nov/2011, with no progress at all... The GGUS was set to 'unsolved'. Add GGUS Support Unit for REBUS - does this need a GGUS to GGUS?, last update 2012-03-02
      • [in progress+] GGUS:79719: Some CMS SAM tests not published in SAM for several days. Affected site: RWTH-Aachen, from 21/2 to 27/2 two CMS WN tests, org.cms.WN-analysis and org.cms.WN-basic, were not published. Assigned (SAM/Nagios), but they do not provide suppor (EMI Product Teams do). CMS ask more: "if the code of a WN test gets stuck, what happens? Is Nagios going to kill the process and report the output as CRITICAL, or the test simply does not get published?", last update 2012-03-02
    • Notes:
      • none

  • LHCb reports -
    • Experiment activities
      • MC, User analysis, Validation productions for restripping
    • New GGUS (or RT) tickets
    • T0
      • CERN : setting the variable TMPDIR : (GGUS:79685)
    • T1
    • T2

Sites / Services round table:

  • NDGF: Network intervention in the next weeks. Hope the network provider will stick in the announced downtime (15')
  • IN2P3: dCache ticket solved. During a service change some files could not staged-in anylonger. Fixed.
  • RAL: Downtime next Tuesday: DB and FTS interventions
  • Grid services: looking into the TMPDIR problem. Yesterday FNAL-CERN moved to a different usage (overload). Since the activity eventually dropped difficult to investigate the root cause but inciedent closed
  • Other sites and central services: NTR
AOB:

-- JamieShiers - 18-Jan-2012

Edit | Attach | Watch | Print version | History: r26 < r25 < r24 < r23 < r22 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r26 - 2012-03-21 - MariaDimou
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback