Reprocessing and Production Team Meeting - Aug 10 4PM CERN, 9 FNAL time

Vidyo Link

• https://indico.cern.ch/event/558448/

Attending

• US: Jorge, Gastón, Matteo, Jesus, Alli, David Mason
• CERN : Alan, Dima, Sebastian, Paola
• EU: Svenja

Personnel

• Workshop Week of Aug 16-19
• Jen off 23-24
• Paola will be off 25, 26 and maybe 29 of Aug

News - Dima

• Workshop Agenda: https://indico.cern.ch/event/557895/
• Rerecos: We can track missing lumisections for rereco, it is not a trivial task.

Top issues affecting production

• Vanderbilt is back up, we have been sending jobs that have to run there again.
• SubmitFailed "error message". We should report the site the jobs want to run. https://github.com/dmwm/WMCore/issues/6868
  • it does not match with any site information. No possible site, there is no clear print out.
  • When we see an example we shuould report it to the developers.
• The release usage in merge tasks for taskchain is not coherent.
  • Some merge task are using a diffent version of CMSSW.
  • When we see that issue, we just kill and clone, and it works.
  • https://github.com/dmwm/WMCore/issues/7012
• Unmerged files for very long workflow: We have unmerged files been deleted before the ACDC for the workflow is created.
  • This is an usual issue.
  • Where this situation is happenning?

Site support -

• Waiting Room: T2_ES_IFCA, T2_UK_London_Brunel
• Out the Waiting Room: T2_GR_Ioannina, T2_BE_UCL, T2_UK_SGrid_Bristol, T2_US_Vanderbilt

Sites in Morgue: 9

• The team is working on the tests of CSCS. We sent the backfill wfs to vocms0231, but there was a problem with puppet on that machine.
  • The firewall rules were messed up and caused the schedd got missconfigured. Farrukh fixed puppet, and we have jobs running right now.
  • I hope to get the results for the test today.

Transfer Team

• https://docs.google.com/spreadsheets/d/1aB7Aym2wCzP3j8QV5Whcl_KFEuBhW15Jn457AaQYm20/edit#gid=0

Workflows

• 2 ReReco WF's with 0 events in input:
  • https://cms-logbook.cern.ch/elog/Workflow+processing/25325
  • what should we do with these?
  • we can reject those wfs, is up to us.

Agent Issues

• Couch Is going down over and over again on vocms026, what should we do?
  • if happens again, we should redeploy.

Agent redeployment

• In the middle of august we are going to start redploying everything
• The next week we are going to get a new release of te agent.
• Tier0 back to production next week.

RequestMgr2 Migration

• We should create some ACDCs on testbed using the web interface.

Scripts

• We need to work on the MakeAllACDC script: ACDC workflow should be created only if the desired site is currently up. https://github.com/CMSCompOps/WmAgentScripts/issues/166

RelVal Andrew

AOB

-- JenniferAdelmanMcCarthy - 2016-08-10