Reprocessing and Production Team Meeting - May 12 4PM CERN, 9 FNAL time

Vidyo Link

• https://indico.cern.ch/event/515148/

Attending

• FNAL: Jen, Jorge, Jesus, Matteo, SeangChan
• US: Allie
• CERN : Alan, Paola, Andrew, JeanRoch
• Korea :

Personnel

• Gaston to Colombia Early May 12-28, talk on the 13
• Paola May 13
• Jen 1/2 day May 13 and 20

News - Dima

• New Shifters! Mykola and Svenja from DESY University.
  • Paola & Alan will do some training with them next week

3 top issues affecting production

• Lots of Submit failures, FNAL, KIT, RWTH, PISA needing manual cleanup
• Where are we in testing merge issues at T0_CH_CERN
• Script has been integrated, I downloaded it Wed and it is working perfectly! Thanks Paola!
• filemistach : delay in uploading is bad for book keeping. pdmvserv_SUS-RunIISummer15GS-00131_00284_v0__160323_032949_5198 in the list since a couple of weeks
  • running* => completed transition only when all files are in ? is there an api to check this is all done ? any ways to have this automated out ? please provide the script that check in the agents for files pending injection.
• T0 merge : how comes permission issues come and go ?
• /reqmgr2/data/request?status= failing repeatedly bringing unified to errors.

Site support - Gaston is on his way to Colombia

• please find a way to report (Monday meeting ?) when a site is known bad for xrootd until we have an adequate monitoring : https://hypernews.cern.ch/HyperNews/CMS/get/comp-ops/2860.html

• T2_CH_CSCS - where are we in the testing?
  • Last week's test failed but this week is ok so Paola will answer her findings in a ggus tiket

Transfers - Jorge

• there is a GenSim dataset stuck in transfers that Jorge is working on

Workflows

• lots of workflow are completing, lots of recoveries pending
  • lots of smaller workflows that just finished at rate of 25% into recovey, it would be interesting to have and keep track of this number over time.

ReDigi

MiniAOD

TaskChains

StepChain

• NA

Rereco

Store Results

• https://ggus.eu/index.php?mode=ticket_info&ticket_id=119273

MonteCarlo

Agent Issues

Agent redeployment

RequestMgr2 Migration

Merging Scripts

• The merging changes in this pull request https://github.com/CMSCompOps/WmAgentScripts/pull/139.
• reject.py/resubmit.py tested OK.
• Do we wanna merge resubmitUnprocessedBlocks.py and resubmitWithBlockBlacklist.py? Has Unified picked up the below functionality?
• assignProdTaskChain.py was changed according to https://github.com/CMSCompOps/WmAgentScripts/pull/140, script tested ok. Are this changes propagated to the new merged assign.py?
• What is left?
• Automatic ACDC, step 1: start with a script that drill down through the various wmstats calls for a given workflow and expose what are the main issues, error code, sites, ecc. Paola will be on top of that.

RelVal Andrew

AOB

-- JenniferAdelmanMcCarthy - 2016-05-12