https://indico.cern.ch/conferenceDisplay.py?confId=254677

Attending:

Personel:
Oct 22 --> Oct 29 Julian (+Adli)
Oct 29 --> Nov 4 Adli
  • Adlis shift (let's keep an eye on him)
    • By the way, thanks for the help during the week (Julian)
  • Edgar is on vacations until October 23
  • John is gone Oct 23-Nov 11
  • FNAL at normal operations, last week's crisis was averted :-).
Infrastructure
  • vomcs202 (reprocessing agent): if not works by tuesday, consider adding another agent to repro team.
    • Why FNAL is not getting jobs for reproc?
    • Glidin issues, doesn't appear to be completely solved at end of day here.
    • Aparently problem on condor side.
  • Vocms201 Keep an eye on disk space: periodically redeploy wmagent to clear space?
    • confirm that disk space is being properly cleared out after upgrades.
    • we need to find some view, that can be deleted.
    • log flushing?
  • WMAgent issues:
    • v0.9.82 deployed in vocms202 and vocms235. Next: vocms216, vocms234 and cmssrv98 are in drain (almost done).
    • vocms201 and vocms216 full disk at /data1/
    • 2 jobs with the same output. Partially fixed through Diego's script. Take care when do ACDC's.
  • Workflow issues:
    • Stuck workflows:
      • Unnoticed site problem stuck LogCollect and Cleanup jobs
        • Adding information to the stuck requests script as we discover the problems to make it easier to debug
      • Aborted workflow has many jobs to archive (vocms85)
        • Consider changing Mysql to Oracle (Ask Edgar).
      • Rejected batch due to request setup.
        • Aborted workflows are not moving to aborted-complete
        • Workflows souldn't go from aborted to abort-completed
  • Workflow that was causing massive failed jobs:
  • Oli's plot about worfkows rejected, datasets invalidated, vs total requests.

Site Problems
cms-comp-ops-site-support-team (Site Support Team) <cms-comp-ops-site-support-team@cern.ch>

Sites with PhEDEx agents running:

CMS SiteSorted ascending PhEDEx agents state
T2_CN_Beijing Running
T3_CH_CERN_OpenData Running
T3_RU_MEPhI Running

XRootD restarting pending tickets:

TicketSorted ascending State CMS Site
149071 waiting for reply T2_DE_DESY

Sites currently not enable LifeStatus state:

SITESorted ascending Status Duration Reason
T2_PK_NCP Waiting Room 1+ month 2+ HC evals
T2_PL_Swierk Waiting Room 1 week(s) SAM evals continously failing
T2_PT_NCG_Lisbon Waiting Room 1 week(s) 2+ HC evals
T2_RU_ITEP Downtime 1 week(s) Went from morgue to DT
T2_RU_SINP Morgue 1+ month No change
T2_UK_SGrid_Bristol Waiting Room 1 week(s) FTS evals continously failing

Warsaw to be decommissioned

Ticket journal of last week:

TicketSorted ascending CMS Site Creation Date Status Subject
149475 T2_BE_UCL Monday, 11/16 solved Transfers and VOPut failing at
149476 T2_IT_Rome Monday, 11/16 solved CEs are not executing SAM test
149477 T2_FI_HIP Monday, 11/16 solved Few SAM tests running at T2_FI
149486 T2_UK_SGrid_Bristol Monday, 11/16 assigned Unable to create loadtest file
149488 T2_US_MIT Tuesday, 11/17 solved Transfers to MIT failing: Syst
149491 T2_TR_METU Tuesday, 11/17 solved JobSubmit failing at T2_TR_MET
149493 T0_CH_CERN Tuesday, 11/17 in progress Files unavailable at CERN
149501 T2_KR_KISTI Tuesday, 11/17 verified Pilots at T2_KR_KISTI
149503 T2_UK_London_Brunel Tuesday, 11/17 solved SAM tests failing at T2_UK_Lon
149508 T1_US_FNAL Wednesday, 11/18 assigned Transfers failing from T1_US_F
149523 T2_BR_SPRACE Thursday, 11/19 solved Traffic from T2_BR_SPRACE not
149524 T2_PT_NCG_Lisbon Thursday, 11/19 solved HC tests failing at T2_PT_NCG_
149525 T2_RU_IHEP Thursday, 11/19 solved CE holding SAM tests at T2_RU_
149529 T2_DE_RWTH Thursday, 11/19 in progress Transfers failing from/to T2_D
149533 T0_CH_CSCS_HPC Thursday, 11/19 solved site PhEDEx agents at (T0_CH_
149534 T1_ES_PIC Thursday, 11/19 solved site PhEDEx agents at (T1_ES_
149535 T1_FR_CCIN2P3 Thursday, 11/19 solved site PhEDEx agents at (T1_FR_
149536 T1_RU_JINR Thursday, 11/19 solved site PhEDEx agents at (T1_RU_
149537 T1_US_FNAL Thursday, 11/19 solved site PhEDEx agents at (T1_US_
149538 T2_AT_Vienna Thursday, 11/19 solved site PhEDEx agents at (T2_AT_
149539 T2_BE_IIHE Thursday, 11/19 solved site PhEDEx agents at (T2_BE_
149540 T2_BE_UCL Thursday, 11/19 solved site PhEDEx agents at (T2_BE_
149541 T2_BR_UERJ Thursday, 11/19 solved site PhEDEx agents at (T2_BR_
149542 T2_CN_Beijing Thursday, 11/19 assigned site PhEDEx agents at (T2_CN_
149543 T2_DE_DESY Thursday, 11/19 solved site PhEDEx agents at (T2_DE_
149544 T2_DE_RWTH Thursday, 11/19 solved site PhEDEx agents at (T2_DE_
149545 T2_EE_Estonia Thursday, 11/19 solved site PhEDEx agents at (T2_EE_
149546 T2_ES_IFCA Thursday, 11/19 solved site PhEDEx agents at (T2_ES_
149547 T2_FR_CCIN2P3 Thursday, 11/19 solved site PhEDEx agents at (T2_FR_
149548 T2_GR_Ioannina Thursday, 11/19 assigned site PhEDEx agents at (T2_GR_
149549 T2_HU_Budapest Thursday, 11/19 solved site PhEDEx agents at (T2_HU_
149550 T2_IN_TIFR Thursday, 11/19 solved site PhEDEx agents at (T2_IN_
149551 T2_IT_Legnaro Thursday, 11/19 solved site PhEDEx agents at (T2_IT_
149552 T2_KR_KISTI Thursday, 11/19 solved site PhEDEx agents at (T2_KR_
149553 T2_PL_Swierk Thursday, 11/19 solved site PhEDEx agents at (T2_PL_
149554 T2_RU_IHEP Thursday, 11/19 solved site PhEDEx agents at (T2_RU_
149555 T2_RU_INR Thursday, 11/19 solved site PhEDEx agents at (T2_RU_
149556 T2_RU_ITEP Thursday, 11/19 in progress site PhEDEx agents at (T2_RU_
149557 T2_TR_METU Thursday, 11/19 solved site PhEDEx agents at (T2_TR_
149558 T2_TW_NCHC Thursday, 11/19 assigned site PhEDEx agents at (T2_TW_
149559 T2_UK_London_Brunel Thursday, 11/19 solved site PhEDEx agents at (T2_UK_
149560 T2_UK_London_IC Thursday, 11/19 solved site PhEDEx agents at (T2_UK_
149561 T2_US_MIT Thursday, 11/19 solved site PhEDEx agents at (T2_US_
149562 T2_US_Vanderbilt Thursday, 11/19 assigned site PhEDEx agents at (T2_US_
149563 T2_US_Wisconsin Thursday, 11/19 solved site PhEDEx agents at (T2_US_
149564 T3_CH_PSI Thursday, 11/19 solved site PhEDEx agents at (T3_CH_
149565 T3_FR_IPNL Thursday, 11/19 solved site PhEDEx agents at (T3_FR_
149566 T3_KR_KNU Thursday, 11/19 solved site PhEDEx agents at (T3_KR_
149567 T3_US_Baylor Thursday, 11/19 solved site PhEDEx agents at (T3_US_
149568 T3_US_Colorado Thursday, 11/19 solved site PhEDEx agents at (T3_US_
149569 T3_US_NotreDame Thursday, 11/19 solved site PhEDEx agents at (T3_US_
149570 T3_US_Princeton_ICSE Thursday, 11/19 solved site PhEDEx agents at (T3_US_
149571 T3_US_PuertoRico Thursday, 11/19 solved site PhEDEx agents at (T3_US_
149572 T3_US_Rutgers Thursday, 11/19 assigned site PhEDEx agents at (T3_US_
149573 T3_US_UMD Thursday, 11/19 solved site PhEDEx agents at (T3_US_
149574 T3_US_UMiss Thursday, 11/19 solved site PhEDEx agents at (T3_US_
149576 T2_IN_TIFR Thursday, 11/19 assigned Pilots at T2_IN_TIFR
149577 T2_US_UCSD Thursday, 11/19 assigned Pilots at T2_US_UCSD
149588 T2_UK_SGrid_RALPP Friday, 11/20 solved SAM tests failing at T2_UK_SGr

Generated on 23/Nov/2020 (GMT)

Sites with open GGUS tickets:

CMS SiteSorted ascending Number of Tickets Tickets
Generated on 23/Nov/2020 13:52:36 (GMT), Total number of tickets: 36
T0_CH_CERN 1 148790
T0_CH_CSCS_HPC 2 149166 147830
T1_IT_CNAF 1 149215
T2_AT_Vienna 1 148309
T2_BE_UCL 1 148354
T2_BR_SPRACE 1 149430
T2_CN_Beijing 4 148997 149321 148780 148688
T2_DE_DESY 1 149071
T2_FI_HIP 1 148708
T2_FR_GRIF_IRFU 2 149150 149272
T2_IN_TIFR 1 147831
T2_IT_Bari 1 148735
T2_IT_Legnaro 1 149019
T2_IT_Pisa 1 149184
T2_PK_NCP 1 148185
T2_PL_Swierk 3 149392 149274 147006
T2_PT_NCG_Lisbon 1 149275
T2_RU_ITEP 2 149046 148568
T2_TR_METU 2 149344 149276
T2_UK_SGrid_Bristol 1 149425
T2_US_UCSD 1 149443
T2_US_Vanderbilt 2 148148 144936
T3_US_Minnesota 2 139435 140562
T3_US_Rice 1 149315
T3_US_UMD 1 148604

Updated on: 2020-11-23 at 10:55:18 by HectorCamiloZambranoHernandez

  • Any problem, email site support team list (while John is in vacation).

AOB
Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r7 - 2016-07-22 - StephanLammel
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback