December 2014 Reports

18 December 2014 (Thursday)

• "Legacy Run1 Stripping" campaign running full steam and progressing well, prestaging of input data was restarted + MC and user jobs. Note, the stripping campaign will continue over the xmas break.
• T0:
• T1:
  • CNAF: some debugging ongoing on mostly SRM protocol access for different kinds of activities to the storage elements, no conclusion/solution yet.

15 December 2014 (Monday)

• "Legacy Run1 Stripping" campaign running full steam and progressing well, prestaging of input data was restarted + MC and user jobs.
• T0:
• T1:
  • CNAF: hiccup with SRM on Friday, fixed promptly
  • RAL: WARNING DT on the week-end went bad, ask sites to declare OUTAGE in case a the site is affected -> automatic handling by Dirac

11 December 2014 (Thursday)

• "Legacy Run1 Stripping" campaign running full steam and progressing well + MC and user jobs.
• T0: Two GGUS tickets open and not answered yet
  • GGUS:110604 - investigation of higher failure rate for stripping jobs at CERN
  • GGUS:110583 - SAM probes were timing out at CERN queues
• T1:
  • IN2P3:

8 December 2014 (Monday)

• MC and user jobs. "Legacy Run1 Stripping" campaign running full steam and progressing well
• T0: Very low number of stripping job errors but CERN still shows a multiple error rate compared to other T1 sites, would like to understand this. Resubmission of the same file usually works (GGUS:110604)
• T1:
  • IN2P3: DT tomorrow, draining tonight
  • RAL: Again, low total number of errors but saw several jobs failing with "bus error" during the week-end for stripping

4 December 2014 (Thursday)

• MC and user jobs. Validation for "Legacy Run1 stripping campaign" finished. Plan to launch the full campaign this evening.
• T0: Problem with lbvoboxes authenticating to LFC without proxy (GGUS:110469). Still not informed in the ticket about getting all the lbvoboxes whitelisted for read-only access.
• T1: NTR
• Others : "DOS" issue - large number of queries hitting LHCb configuration services starting on morning of 3 December 2014. Three ip addresses banned in the LHCb configuration (2 from Romania and 1 from FZK). The FZK address was un-banned this morning and jobs are running again at GridKa. email sent to Romanian site admin and we are waiting for response from him.

1 December 2014 (Monday)

• MC and user jobs. Validation for "Legacy Run1 stripping campaign" almost finished. Now waiting for responses from Physics groups before getting the greenlight for launching the full campaign.
• T0: Problem with lbvoboxes authenticating to LFC without proxy starting Friday night (GGUS:110469). Significant effect on productions over the weekend as new jobs were not being created automatically. Did not affect user jobs.
• T1: NTR

-- JoelClosier - 31 Mar 2014