Workflow Team Meeting - March 3 4PM CERN, 9 FNAL time

Vidyo Link

• https://indico.cern.ch/event/505823/

Attending

• FNAL: Gaston, SeangChan,
• US:
• CERN : Jen, Paola, Alan, Andrew, Dima

Personnel

• Jen to CERN Feb 29-March 4 - tickets are booked!
• Jorge to Columbia April 15-May 2, Talk on April 27
• Alison will at FNAL March 7-8
• Korean Shifters are on shift work

News - Dima

• One big premix request coming, input dataset 100TB only available at KIT right now, so we need to make copies,

3 top issues affecting production

• Generally quiet week, getting caught up, training and testing
  • Caltech is having issues so it will
• most ACDC/clone/stuck workflows were due to sites being put into drain while the workflow was in flight. What can we do to catch this and recover faster?
  • Looks like most of them are stuck at T0_CH_CERN, Alan will attempt to clean all of these up tomorrow and then we will see what is still stuck tomorrow afternoon.
• Couch replication
• Harvesting workflows are not closing, they are not being seen so we can't close the workflows,
  • it's a missing api, Alan knows what is going on and will fix it

Site support - Gaston

• Why were the US T2's put into drain? Causing lots of clones and stuck work.

• Current Waiting Room : T2_RU_SINP, T2_RU_INR , T2_ES_IFCA,T2_IN_TIFR , T2_PK_NCP, T2_EE_Estonia, T2_BE_IIHE, T2_IT_Rome.
• Current Morgue: T2_RU_RRC_KI, T2_RU_ITEP, T2_MY_UPM_BIRUNI, T2_TR_METU, T2_RU_PNPI, T2_TH_CUNSTDA, T2_PL_Warsaw

Transfers - Jorge

Workflows

ReDigi

MiniAOD

TaskChains

StepChain

Rereco

Store Results

• https://ggus.eu/index.php?mode=ticket_info&ticket_id=119273

MonteCarlo

• jen_a_HIG-RunIIWinter15GenOnly-00043_00004_v0__160215_221920_8367
  • errors we can not recover from https://cms-logbook.cern.ch/elog/Workflow+processing/23153
  • there is no useful information in the logs

Agent Issues

Agent redeployment

• reboot of 309, 310, 311 on Tues, took a while for replication to catch up
• new version of the agents in March, should - mid March, plan is to have everything redeployed by the beginning of the April Campaign
• who is going to write up the redeployment schedule? Alan & Paola
• cmsweb update - most of the crab experts won't be available next week due to Barcelona workshop the deployment will be March 8
  • the next one would be ~First Tues of the month in general

ReqestMgr2 Migration

• we are not dropping request manager1 anytime soon
• Jen and Matteo have started testing

RelVal Andrew

L3 discussion - Ajit, Jean-Roch, Matteo

Opportunistic Resources

Automatic Assignment And Unified Software

• Alli will be in charge of documentation

AOB

-- JenniferAdelmanMcCarthy - 2016-03-02