Reprocessing and Production Team Meeting - June 30 4PM CERN, 9 FNAL time

Vidyo Link

• https://indico.cern.ch/event/548593/

Attending

• FNAL: Matteo, Jorge, Jen, Gastón
• US: Andrew
• CERN : Dima, Paola, Sebastian, Alan

Personnel

• don't forget July 4 is a US Holiday, US people will not be around
• Jen Vacation July 25-29, 1/2 day July 1???
• SeangChan Jun 2-July 5

News - Dima

• Workshop date are going to be discuss on the Computing Operations Chat. Matteo agreed to propose August 15th.
• All future campaigns are going to use premixing samples needs to be read remotely, we need to rely on AAA. We need to run some tests for sure!!

Top issues affecting production

• There was a backlog accumulated these days when unified stopped running during the weekend.
  • I have seen things moving, we are overcoming this quickly.

• We are going through the oldest workflows in the system.
  • JR has closed-out some of them because we have enough statistics for the RAWAOD.
  • Some of them have been announced.
  • Another ones are running some little ACDCs, it will no take longer.
  • But, we have a problem, there are some stuck ACDCs at 0304 agent.
  • Alan knows about this problem; the solution is going to take a while.

• Event splitting on ACDC and duplicates
  • There was an elog discussion about duplicates generated by ACDCs (https://cms-logbook.cern.ch/elog/Workflow+processing/24800).
  • The conclusion was, for event based splitting workflows, only timeouts and memory can be changed. Otherwise we need to start from scratch.
  • Jen updated the documentation yesterday (https://twiki.cern.ch/twiki/bin/view/CMSPublic/CompOpsWorkflowTeamLeaderResponsibilities#Creating_ACDC_s_Cloning_Resubmit).
  • We won’t make that mistake again

• Of course we need to pay attention to stuck ACDC or even clones. For example https://cms-logbook.cern.ch/elog/Workflow+processing/24747. We need to do it more often, indeed.

Site support

• Vienna having merge issues, probably it is better to reopen this GGUS ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122241

Transfers - Sebastian

• We need to take a look for the files at /store/backfill/1. Jorge is going to invalidate those once we give a green light.

Workflows

ReDigi

MiniAOD

TaskChains

StepChain

• NA

Rereco

Store Results

• https://ggus.eu/index.php?mode=ticket_info&ticket_id=119273

MonteCarlo

Agent Issues

• VOCMS0304 having some stuck ACDC. Alan is going to take a look.

Agent redeployment

• Alan is going to redeploy all agent during this and the next week. 218 today.

RequestMgr2 Migration

• RequestMgr2 is going to be up next week on testbed, who is going to do the tests?

Merging Scripts

• In maintenance.

RelVal Andrew

AOB

-- JenniferAdelmanMcCarthy - 2016-06-30