Dynamic Data Management

A few notes on DDM for our codefest...

CRAB2/3 popularity packet structure:

From Brian and Marco, I get the following dashboard popularity report example:
  'BasenameParent': '',

This was taken from http://glidemon.web.cern.ch/glidemon/show.php?log=http://vocms95.cern.ch/mon/cms1925/140527_122818_crab3test-2:mmascher_crab_skim_200_preprod1/job_out.1.0.txt. Presumably other examples can be found there too.

The CRAB3 code that fills this structure is at https://github.com/dmwm/CRABServer/blob/master/scripts/CMSRunAnalysis.py#L145.

The formats are covered somewhat in the comments:

    # Now, compute the strings about the input files.  Each input file has the following format in the report:
    # %(name)s::%(report_type)d::%(file_type)s::%(access_type)s::%(counter)d
    # Where each variable has the following values:
    #  - name: the portion of the LFN after the common prefix
    #  - report_type: 1 for a file successfully processed, 0 for a failed file with an error message, 2 otherwise
    #  - file_type: EDM or Unknown
    #  - access_type: Remote or Local
    #  - counter: monotonically increasing counter

Multiple files are semi-colon separated:

    inputString = ';'.join(["%s::1::%s::%s::%d" % tuple(value) for value in inputInfo.values()])

Note - in CRAB3, due to current WMCore limitations, only has report_type 1. Also, all access types are currently listed as "Local", and parent information is never filled in.

The basename is the common part of the path. Say we processed /store/foo and /store/bar; then basename would be /store/ and the %(name)s referenced above would be "foo" and "bar", respectively. -- TonyWildish - 06 Jun 2014

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2014-06-06 - TonyWildish
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback