Workflow Team Meeting - July 30rd 4PM CERN, 9 FNAL time

Vidyo Link

• https://indico.cern.ch/event/434917/

Attending

• US: Jen, Jorge, Luis, Gaston Eliana, John, Ajit and Matteo
• EU: Dima, Julian, Alan

Personel

• Jen off Aug 8-26
• Julian Sept 14-30
• Luis is leaving us Sept 5
• John will be leaving us Aug 28
• Jorge Aug 23 off
• Gaston (New FNAL site support person) and Eliana (New FNAL T0 Operator) both now at FNAL

News

• Eliana and Gaston have arrived
• Dima -
  • We need to redo all the GEN_SIM, all digi RECO and all data
  • hopefully all upgrades will stop
  • We need to find better ways of monitoring things to determine that we are fully using resources
    • currently using glideinWMS monitoring that Brian gave them
    • https://twiki.cern.ch/twiki/bin/view/CMSPublic/CompOpsWorkflowOperatorResponsibilities Dima will add what he is currently doing to the operator twiki and will send a note to the mailing list.

3 top issues affecting production

• cleaning up all the failed submission workflows

Site support - John

Transfer - Jorge

• Xfers are failing from/to T2_US_Wisconsin for last 6-8 hours. The error is "Couldn't submit to FTS ..". Please take a look (requested by Ajit).
• /GluGluToHToGG_M-125_14TeV-powheg-pythia6/TP2023HGCALNoTRKExtGS-DES23_62_V1-v1/GEN-SIM dataset xfer to T2_CH_CERN is stuck at 72% for last few days. Please take a look.

Workflows

• High priority campaigns, per Dima we are cutting these at 90% :
  • TP2023HGCALDR
  • TP2023HGCALNoTRKExtGS
  • TP2023HGCALGS
  • TP2023HGCALNoTRKExtDR

ReDigi

• cleaning up workflows with timeouts and submission failures, making slow but steady progress

TaskChains

• also affected by the submission failure

MonteCarlo

• A few workflows with big lumis:

ReReco's

• NTR

Store Results

• NTR

Agent Issues

Redeployment Plan

RelVal Andrew

L3 discussion - Ajit, Jean-Roch, Matteo

Opportunistic Resources

Automatic Assignment And Unified Software

AOB