Workflow Team Meeting - Nov 5 4PM CERN, 9 FNAL time

Vidyo Link

• https://indico.cern.ch/event/460195/

Attending

• FNAL:
• US:
• CERN :
• EU:

Personnel

• Jorge - Nov 9-13
• SeangChan off Nov 3-10
• Julian to Colombia Dec 14, and then contract ends, will be working remotely from Colombia until the First of Jan
• Ajit out till Nov
• JR unavailable until Nov 23d

News - Dima

3 top issues effecting production

• See ReReco notes
• StepChain -
• cmsweb upgrades - finished already https://cms-logbook.cern.ch/elog/Workflow+processing/22103
  • Watch out for PNN's
• do we need to revive recovery script for ReReco
• Lots of WF;s in status failed, with no reason given in file why they failed. Is something broken? I found several of them that had been reset, or cloned but the origioinal was still stuck in status failed. There was nothing in elog about the workflows being cloned or reset, the only way I knew what happened was to look for output datasets with higher version numbers. Is it really that hard to post an elog when you clone/reset workflows? Is there a better way to check on the workflows in failed?
  • (JR) https://cms-logbook.cern.ch/elog/Workflow+processing/21775. Several were reset in McM and "rejected" automatically. tip for finding out faster https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=B2G-RunIISummer15GS-00315

Site support - Gaston

No changes in morgue.

News & Issues

• (JR) Several sites over dataops DDM quota : reasons are
  • heavy gen-sim in production or in use
  • 4 or 5 opened campaign with secondary input : 40+40+25+10+10 TB = 125TB of secondary on disk
• (JR) Large TaskChain with large job blow-up ratio : now assigning to >4k slots sites if ratio >5

Workflows

ReDigi

TaskChains

• Workflows with None AcqEra - https://cms-logbook.cern.ch/elog/Workflow+processing/22100
• We have already discussed why this happened, and outlined the procedure to fix it
  • this is a problem of communication and documentation, Alan and Julian knew what to do, I sort of knew what to do but nothing was documented anywhere!
  • What other exceptions to regular acdc production and assigning are there?
• (JR) what can be done about the very long running taskchains for now

StepChain

Rereco

• Need to start returning WF's that are not 100%
  • need to update the get missing run/lumi info
    • https://twiki.cern.ch/twiki/bin/view/CMSPublic/CompOpsWorkflowTeamLeaderResponsibilities#Finding_Missing_Run_Lumi_informa
    • for many of these workflows, this information is missing under the "status" column because it says that the only errors are in LogCollect, sometimes "blind acdc" is working, but now always, the recovery script was written take care of this
    • is there an easy way for Matteo to get this info from WMStats?
• How much have we talked to the developers on why we are getting missing lumi's but they are not showing up in WmStats as errors? We need to find a better way of recovering this!

Store Results

MonteCarlo

Agent Issues

Redeployment Plan

• everything is up to date - redeploy in mid-November, and then we'll have major changes to iron out before Christmas Production
• all sites need proper local site node name info or they won't work - reposting last weeks list here to see where we are in getting things done!
  • before site config returned node names , but se name and phedex node name need to be returned or they will fail. Gaston needs to verify that the CERN Local configs are right

* as discussed in the meeting, some sites are not providing the phedex-node value in the site-local-config.xml. For example, T2_CH_CERN is Ok:

• • https://cmsweb.cern.ch/gitweb/?p=siteconf/.git;a=blob;f=T2_CH_CERN/JobConfig/site-local-config.xml;h=e3867b547ece409eccc3f1ed4ddd793d803e13ec;hb=HEAD

• but T2_CH_CERN_HLT is missing that info:
  • https://cmsweb.cern.ch/gitweb/?p=siteconf/.git;a=blob;f=T2_CH_CERN_HLT/JobConfig/site-local-config.xml;h=78225ad2ede5b7f1c70d6637b10e4f68602b2bc0;hb=HEAD

• We need all the processing sites to properly provide this info otherwise we'll start having problems in the next WMAgent release.
• Also the site we are using has to be listed here. If not we either update siteDB or get the site name from other sources.
  • https://cmsweb.cern.ch/sitedb/data/prod/data-processing

RelVal Andrew

L3 discussion - Ajit, Jean-Roch, Matteo

Opportunistic Resources

Automatic Assignment And Unified Software

• In the past Julian has "extended MC workflows" that didn't meet statistics, is this something Unified can take over?

AOB

• PNN change - have Gaston check sitedb list that maps

-- JenniferAdelmanMcCarthy - 2015-10-28