Difference: SWGuideConfigurationValidationAndHelp (1 vs. 50)

Revision 502019-08-05 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 359 to 359
 one exception. When one is describing a ParameterSet nested inside a ParameterSet, one must always provide two arguments.
Changed:
<
<
If no default is provided, then there are two consequences:
>
>
If no default is provided, then there are three consequences:
 
  • During validation, if a required parameter is missing and there is no default then an exception is thrown. If a default was provided, then it is inserted.
Changed:
<
<
  • For both required and optional parameters, if no default is provided, then the parameter is not written into the generated CFI file. If a required parameter has no default, then someone using the CFI will be required to explicitly add the missing parameter in one of the python files that imports the CFI.
>
>
  • This is indicated in the output of edmPluginHelp.
  • The parameter will be written into the CFI file in a different way. It will appear as follows depending on whether the parameter is tracked or untracked and whether it is required or optional.

p1 = cms.required.int32,
p2 = cms.optional.int32,
p3 = cms.required.untracked.int32,
p4 = cms.optional.untracked.int32

Most of this TWIKI is discussing a level of validation that occurs when C++ code is being executed. There is a separate level of validation that occurs earlier while the Python code is being executed. While the Python is being executed and just before Python data structures are converted into C++ data structures the parameters are checked. A Python exception is thrown if required parameters as shown above have not been set to a value. Optional parameters that have not been set to a value are simply dropped.

If the parameters were set after the import of the generated CFI file, then they will be passed through to the C++ data structures as if they had been declared as follows:

p1 = cms.int32(value1),
p2 = cms.int32(value2),
p3 = cms.untracked.int32(value3),
p4 = cms.untracked.int32(value4)

In addition to "required" and "optional", there is another special Python type called "obsolete". This type will never be written into generated CFI files. During the Python level validation, those are simply ignored and not passed into the C++ ParameterSet objects.

Note that the behavior described above is new with release series 11_0_X. Prior to that parameters with no defaults were not written into CFI files at all.

 

Adding Comments

Line: 1516 to 1549
  output just print out as if the parameter were a ParameterSet and not the python SecSource type. This is incorrect behavior.
  • Defaults for wildcards.
Deleted:
<
<
  • Placeholders in the python to allow replacing parameter values in cfg files which do not exist in the CFI files.
 
  • Some discussion of eliminating CFI files entirely in favor of inserting missing parameters. (Probably very far in the future if ever)

Revision 492019-08-05 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 134 to 134
 vector<ParameterSet> are slightly different than in the example above. These are described in the following sections.
Added:
>
>
Note that the Framework will automatically add extra parameters in special cases. For example "mightGet" is added for producer modules and "appendToDataLabel" is added for EventSetup producer modules. Even though not explicitly appearing in fillDescriptions functions, these special parameters have the same effect as if they were, they affect validation and show up in generated cfi files and the output of edmPluginHelp.
 One point above is worth repeating. If any parameters are missing from the python configuration, then during validation the parameters will be inserted into the ParameterSet before it is passed to the plugin

Revision 482019-08-05 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 201 to 201
 vector InputTag vector
Added:
>
>
ESInputTag vector
 FileInPath LuminosityBlockRange vector

Revision 472018-10-06 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 414 to 414
 

NEW A Plugin Module Using Another Helper Plugin NEW

Changed:
<
<
WARNING THIS SECTION IS CURRENTLY BEING EDITED AND CREATED. EXPECT IT TO CHANGE FREQUENTLY OVER THE NEXT FEW DAYS AND IT MAY CONTAIN ERRONEOUS INFORMATION IN THE PRELIMINARY VERSIONS
>
>
This is a new feature first available in CMSSW_10_3_X_2018-10-04-1100 which should be included in CMSSW_10_3_0 when it is released.

Sometimes a module plugin may load and use one or more helper plugins. In this case, it is desirable that the helper plugin itself contains the description of its configuration parameters. This is good for several reasons. First of all there is just the general principal that a module plugin should not depend on the details of the implementation of a helper plugin it uses. The module plugin should only depend on the interface specified in the helper plugin's base class. Second, there may be many different helper plugin types that can be used and module plugin does not know in advance which will be used. Also, it should be possible to use a completely new type of helper plugin without modifying the module plugin. Third, a helper plugin might be used by many different module plugins and it would be wasteful to duplicate the description in all those different module plugins.

Putting the parameter description in the helper plugin requires special handling, because the compiled libraries containing the helper plugins will not be loaded when validation is started. And the executable should only load the plugin libraries that are actually used. This is not known at compile time. It only known at run time, because it depends on configuration parameters.

There are 6 places where you will need to do something special to use validation for a helper plugin.

  1. in the configuration
  2. in the helper plugin
  3. in fillDescriptions in the module plugin
  4. at factory registration
  5. at plugin registration
  6. in the module plugin constructor

The details of what needs to be done are discussed below.

1. The configuration of the module plugin would look something like the following:

foo = cms.EDProducer(“FooProd”,
    plugin = cms.PSet(
      type = cms.string(“BarHelper”),
      value = cms.int32(5)
    )
    # other module parameters allowed outside the PSet
    ...
)

It is important that there is a separate PSet defined that is contained inside the module's configuration. All the parameters used to configure the helper plugin should be in this separate PSet. This separate PSet should also contain a special parameter of type string whose value is the plugin name of the helper plugin (usually this is the type of the helper plugin or a name associated with that type).

Many of the names used in the example above are just made up names that would be appropriately replaced in an actual configuration. "foo" is the module name. "FooProd" is plugin name of the module. This will work for types other than EDProducer. "BarHelper" is an example plugin name for the helper type. The value parameter would be replaced with the parameters needed to configure the helper. The names "plugin" and "type" could be replaced with different names, but the names used in the configuration must match the names used in the C++ code defining the modules constructor and fillDescriptions function. If the module loaded more than one helper plugin, then the module would need to use a name other than "plugin" so the names would not conflict.

2. The following function must be defined inside the plugin helper class. It has to be a static function returning void named "fillPSetDescription" that takes a reference to a ParameterSetDescription as an argument. The content of the function would be replaced with code adding whatever parameters are appropriate for that helper plugin (the "value" parameter in the example should be replaced).

There is one addition important requirement. The string parameter that contains the plugin name of the helper type should not be included. It gets added to the description automatically.

static void
fillPSetDescription(edm::ParameterSetDescription& iDesc) {
    iDesc.add<int>(“value”,5);
}

3. The fillDescriptions function would need to be defined in the module as follows. If there is a helper plugin, then one needs to create a ParameterSetDescription devoted solely to that helper plugin. The ParameterSetDescription should have exactly one node added to it as its content. The node should be of type PluginDescription. The template parameter of PluginDescription should be the type of the factory that creates the plugin. The first argument of its constructor is the name of the parameter that contains the plugin name of the helper plugin type. The second argument indicates whether or not that helper plugin name is stored in a tracked parameter (false would indicate untracked).

void FooProd::fillDescriptions(edm:ConfigurationDescriptions& oDesc) {

    edm::ParameterSetDescription pluginDesc;   
    pluginDesc.addNode(edm::PluginDescription<HelperFactory>(“type”, true));

    edm::ParameterSetDescription top;
    top.add<edm::ParameterSetDescription>(“plugin”, pluginDesc);
    # other parameters can be added to top
    ...
    oDesc.addDefault(top);
}

The ParameterDescription class has an alternative contructor that takes 3 arguments. The first and last arguments are the same as above. There is an additional argument added in between the other two that allows specifying a default helper plugin name.

4. The factory that creates the creates the helper plugin objects needs to be registered with a special macro. If you are converting an existing helper plugin that does not support validation to support validation, then the macro listed below would replace the macro currently being used to register the factory.

#include “FWCore/ParameterSet/interface/ValidatedPluginFactoryMacros.h”
...
EDM_REGISTER_VALIDATED_PLUGINFACTORY(HelperFactory, “HelperFactory”);

5. The helper plugin type also needs to be registered with a special macro. If you are converting an existing helper plugin that does not support validation to support validation, then the macro listed below would replace the macro currently being used to register the helper plugin.

#include “FWCore/ParameterSet/interface/ValidatedPluginMacros.h”
...
DEFINE_EDM_VALIDATED_PLUGIN(HelperFactory,BarHelper, ”BarHelper”);

6. And finally the helper plugin would need to be created in the constructor of the module plugin in a certain way. Even without validation, helper plugins might be created this way. But to be able to validate the helper plugin's configuration, the helper must be created this way.

auto pluginPSet = pset.getParameter<edm::ParameterSet>("plugin");
   
//  helper_ is a data member, probably a unique_ptr<BarHelperBase>
helper_.reset(HelperFactory::get()->create(pluginPSet.getParameter<std::string>("type"), pluginPSet));

In the example above, a contained ParameterSet is retrieved from the module's ParameterSet. The type of the helper plugin is retrieved from the contained ParameterSet. It must be retrieved from there. Then both the helper plugin name and the contained ParameterSet are passed to the create method of the factory. The ParameterSet then gets passed through to the constructor of the helper plugin C++ class.

One final comment, the above prescription also works with a vector of ParameterSets in a straightforward way.

 

Let the ParameterSetDescription Allow Anything

Line: 1350 to 1513
  of inserting missing parameters. (Probably very far in the future if ever)
  • Support the MessageLogger special case
Deleted:
<
<
  • Extend to support lower level plugins
 
  • Implement a reset function for the value of a single parameter description that makes it easier to define multiple descriptions for the same plugin
  • Improve error checking and messages printed after exceptions
Changed:
<
<
  • Implement the fillDescriptions function in many plugins. Currently very few plugins have this function.
>
>
  • Implement the fillDescriptions function in more plugins. Many plugins already have it, but not all.
 
  • Continue to improve this documentation
  • Add the ability for the validation to check if the value of a parameter is in a certain range, or is one of a limited number
Line: 1368 to 1529
 It started supporting sources and services in release 3_5_0. Support for ESSources and ESProducers has been implemented and checked into CVS and will probably be available starting in release 3_11_0.
Added:
>
>
Support for module plugins loading helper plugins should be available in release 10_3_0.
 

An example

Here is an example of a job that will detect a parameter that

Changed:
<
<
is not allowed (works in 3_1_0_pre7):
>
>
is not allowed:
 
  cmsRun FWCore/Integration/test/testParameterSet_cfg.py
Line: 1389 to 1552
 
Reviewer/Editor and Date (copy from screen) Comments
DavidDagenhart - 18 May 2009 created page
DavidDagenhart - 22 April 2013 reviewed
Added:
>
>
DavidDagenhart - 5 October 2018 added section on helper plugins
 
<!-- In the following line, be sure to put a blank space AFTER your name; otherwise the Summary doesn't come out right. -->

Revision 462018-10-05 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 412 to 412
 ...
Added:
>
>

NEW A Plugin Module Using Another Helper Plugin NEW

WARNING THIS SECTION IS CURRENTLY BEING EDITED AND CREATED. EXPECT IT TO CHANGE FREQUENTLY OVER THE NEXT FEW DAYS AND IT MAY CONTAIN ERRONEOUS INFORMATION IN THE PRELIMINARY VERSIONS

 

Let the ParameterSetDescription Allow Anything

It is possible to tell a ParameterSetDescription to allow anything.

Revision 452018-10-01 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 88 to 88
 would stop if any parameters other than "algorithm", "ptCut", or "debugLevel" were in the configuration. This protects the user against spelling errors and misuse of parameters in the configuration.
Changed:
<
<
An exception would also be thrown if the parameter had the wrong
>
>
An exception would also be thrown if a parameter had the wrong
 type or was tracked when it should have been untracked or vice versa. If any of the three parameters above were missing in the configuration, then the validation algorithm would insert them into the ParameterSet that is passed to the plugin constructor with default values which are provided in the second argument to the "add" function above.
Changed:
<
<
During the build a CFI file would be automatically generated with this path: "cfipython/slc5_ia32_gcc434/SubsystemName/PackageName/produceMuons_cfi.py".
>
>
During the build, a CFI file would be automatically generated with the following path (substituting something like slc6_amd64_gcc700 for CurrentArchitecture, and the actual directories holding the associated code for the SubsystemName/PackageName with the filename based on the module label passed as an argument in the last line of the code above):
cfipython/currentArchitecture/SubsystemName/PackageName/produceMuons_cfi.py
 This file would contain the following:

Revision 442018-02-27 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 604 to 604
  The addWithDefaultLabel() works otherwise exactly as regular add().
Added:
>
>

Fixed size arrays

If you know the size of a vector of parameters at compile time there is a utility function that will check the size and return a std::array object containing the contents of the std::vector. If the size does not match the expected size, then an exception will be thrown. This function does not run as part of the validation. It runs later when getting the parameter. See FWCore/ParameterSet/interface/getFixedSizeArray.h. This was added in the 10_1_X release series.

 

More Complex Features Available for Use in a Description

The next several sections introduce several interesting features that

Revision 432018-02-16 - MattiKortelainen

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 592 to 592
 not have a value equal to the plugin name. This is all hidden from the user.
Added:
>
>

Automatic module labels from plugin class names

In `10_0_0_pre3` it became possible to use a module label that is automatically derived from the plugin class name. This is useful mostly for templated plugins where a typedef is used as the plugin name. The automatic label can be used with

void MuonProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
   edm::ParameterSetDescription desc;
   desc.add<int>("algorithm", 1);
   descriptions.addWithDefaultLabel(desc); // in this case the label would be "muonProducer"
}
The addWithDefaultLabel() works otherwise exactly as regular add().
 

More Complex Features Available for Use in a Description

The next several sections introduce several interesting features that

Revision 422017-08-16 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 394 to 394
 hard coded will not be fast. The correction will not reach users until the next software release is distributed, which could be a long time.
Added:
>
>
In release 9_3_X another alternative way to set comments will be added. Later sections will describe how a ParameterDescription can be explicitly constructed. In only those cases, one can add the comment as an additional argument to the constructor.

...
edm::ParameterDescription<std::string>("name", "defaultValue", true, edm::Comment("your comment"))
...
 

Let the ParameterSetDescription Allow Anything

It is possible to tell a ParameterSetDescription to allow anything.

Revision 412017-07-24 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 134 to 134
 plugin's code will no longer be used for anything. We recommend these other defaults be deleted because they might cause confusion.
Added:
>
>

Converting an Existing CFI file into a fillDescriptions Function

There is a Python executable that will convert an existing CFI file into a fillDescriptions function. Run it as follows:

  edmPythonConfigToCppValidation [configuration file name]

This is very useful if you already have CFI file defined and want to create a fillDescriptions function. It could also be useful if you are more familiar with writing CFI functions in Python than writing the syntax used in fillDescriptions functions in C++.

In most simple cases, the output will be exactly what you need. You only need to cut and paste it into the C++ code of your module. But the fillDescriptions functions can describe requirements and relationships between parameters that cannot be expressed in a cfi file. In complex cases, the output may need additional editing to get the desired fillDescriptions function.

Note when you use a CFI file in a configuration passed to cmsRun, a manually written CFI file in the python subdirectory will override a generated CFI file with the same file name, so you probably want to remove the existing CFI file from the CMSSW repository when you create a fillDescriptions function that generates a CFI file with the same name.

 

Types Allowed for Arguments

In the examples on this TWIKI, a literal string is shown for the argument

Revision 402017-07-13 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 431 to 431
 The module label is used in both the file name and the content of the file. A CFI file is not generated for a default description.
Added:
>
>

Uniqueness of CFI Filenames

Within a package, the filenames of all the generated CFI files must be unique because the CFI files are kept in the same directory. These filenames are based on the module labels passed to the ConfigurationDescriptions:add function. So these module labels must be unique within a package across the fillDescriptions functions for all the plugins in the package.

For plugins based on template classes, it is necessary to create a function specialization of the fillDescriptions function if a CFI file is generated. In this case, the CFI filenames will conflict if all the template instantiations share the same fillDescriptions function definition.

For plugins based on an inheritance hierarchy and generating CFI files, all derived classes will need to define their own fillDescriptions function. If they share that function from a base class and generate CFI files, then the filenames will conflict.

 

Sources and Services

Defining a ParameterSetDescription for a service or a source is almost the

Revision 392013-04-22 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 1298 to 1298
 
Reviewer/Editor and Date (copy from screen) Comments
DavidDagenhart - 18 May 2009 created page
Added:
>
>
DavidDagenhart - 22 April 2013 reviewed
 
<!-- In the following line, be sure to put a blank space AFTER your name; otherwise the Summary doesn't come out right. -->

Revision 382011-12-08 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 1302 to 1302
 
<!-- In the following line, be sure to put a blank space AFTER your name; otherwise the Summary doesn't come out right. -->

Responsible: DavidDagenhart

Deleted:
<
<
Last reviewed by:
 \ No newline at end of file
Added:
>
>
Last reviewed by: DavidDagenhart - 8 December 2011
 \ No newline at end of file

Revision 372011-09-07 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 383 to 383
 will be injected and other types of configuration errors can result in exceptions being thrown.
Changed:
<
<
(Note that prior to release cycle 5_0_X, the behavior when setAllowAnything
>
>
(Note that prior to release CMSSW_4_4_0, the behavior when setAllowAnything
 was called was different. In those earlier releases, when this was called
Changed:
<
<
validation was skipped entirely and injection was not done. There is a chance the new behavior will be backported to 4_2_X and later releases although as of this writing this has not been done yet)
>
>
validation was skipped entirely and missing parameter injection was not done.)
 

Multiple Descriptions for One Plugin

Revision 362011-09-02 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 20 to 20
  validation will throw if a parameter is in the configuration that is not in the description (protection against spelling errors).
Changed:
<
<
  • The description of the parameters that a plugins requires or allows will be accessible
>
>
  • The description of the parameters that a plugin requires or allows will be accessible
  from the executable edmPluginHelp in a human readable format.
  • CFI python files will be generated automatically after giving the command "scram b". The python files are generated along with the libraries and other things
Line: 376 to 376
  desc.setAllowAnything();
Changed:
<
<
When this function has been called no validation occurs for the corresponding ParameterSet in the configuration. This means no exceptions will be thrown and missing parameters will not be injected. I'm not sure if it would be useful, but one could add parameters even with this set. It does not affect the output of CFI generation and the only affect on edmPluginHelp is a couple additional lines that indicate the function was called.
>
>
When this function has been called illegal parameters that are in the configuration but not in the description will be ignored instead of an exception being thrown. Other than these exceptions not being thrown, validation will proceed normally. Missing parameters will be injected and other types of configuration errors can result in exceptions being thrown.

(Note that prior to release cycle 5_0_X, the behavior when setAllowAnything was called was different. In those earlier releases, when this was called validation was skipped entirely and injection was not done. There is a chance the new behavior will be backported to 4_2_X and later releases although as of this writing this has not been done yet)

 

Multiple Descriptions for One Plugin

Revision 352011-01-27 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 469 to 469
 

ESSource and ESProducer

Changed:
<
<
ESSource's and ESProducer's are handled almost identically to the other types of modules. There is one very minor difference.
>
>
Support for parameter set validation for ESSource's and ESProducer's was added in release 3_11_0. ESSource's and ESProducer's are handled almost identically to the other types of modules. There are two differences.

First, for every description for an ESSource or ESProducer, the parameter named "appendToDataLabel" will be automatically added to the description unless it was already explicitly added. If added automatically, it will have type string, be required, be tracked, and have for a default value an empty string. This automatic parameter addition did not get included in release 3_11_0, although it should be included in releases very soon after that.

The other difference is very minor. Most people can just ignore it and will never know the difference, but just to be complete I'll describe it here (I advise most readers to skip the rest of this subsection and not waste their time reading it).

 In the old configuration language that was used before we started using python to configure cmsRun, an ESSource or ESProducer could be labeled or unlabeled. This was not true for other modules

Revision 342011-01-13 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 59 to 59
  The class name MuonProducer is just used as an example. It needs to be replaced with the C++ class name for the plugin.
Changed:
<
<
The declaration of the function is the same for EDFilter's, EDAnalyzer's, OutputModule's, InputSource's and Services.
>
>
The declaration of the function is the same for EDProducer's, EDFilter's, EDAnalyzer's, OutputModule's, InputSource's, ESSource's, ESProducer's and Service's.
 

An Example

Line: 359 to 360
 have a setComment function. All of these comments will show up in different places in the output of edmPluginHelp.
Deleted:
<
<
It is important to not include newlines inside comments. edmPluginHelp will automatically break lines to adjust them to fit within terminal width or 80 characters. And this width is adjustable by using the -w option of edmPluginHelp. Currently newlines in the comments will interact badly with the automatic formatting and make a mess. We might improve this at some point to allow newlines.
 There is a lot of flexibility in how these comments can be used. Our current thought is that it is best if they are kept brief, saving long explanations for TWIKI pages or other document formats.
Line: 393 to 387
 

Multiple Descriptions for One Plugin

It is possible to add multiple ParameterSetDescriptions for

Changed:
<
<
one plugin if it is an EDProducer, EDFilter, EDAnalyzer, or OutputModule.
>
>
one plugin if it is an EDProducer, EDFilter, EDAnalyzer, OutputModule, ESSource, or ESProducer.
 
#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
Line: 473 to 467
 or a service to have a module label. That is why one is not allowed in the description either.
Added:
>
>

ESSource and ESProducer

ESSource's and ESProducer's are handled almost identically to the other types of modules. There is one very minor difference. In the old configuration language that was used before we started using python to configure cmsRun, an ESSource or ESProducer could be labeled or unlabeled. This was not true for other modules like EDAnalyzer. Even in the old configuration language, they were always labeled. So there are two cases to consider, labeled and unlabeled.

The labeled case is easy. In this case, validation and parameter set descriptions are handled exactly the same for ESSource's and ESProducer's as for other modules.

The unlabeled ESSource's and ESProducer's really had no label in the old configuration language. In python, every object must have a name so the unlabeled ESSource's and ESProducer's actually have a label in python configuration files. We identify this special case by giving the module label the same value as the plugin name. For example,

process.DoodadESSource = cms.ESSource("DoodadESSource")

That is an 'unlabeled' ESSource. So you can see that 'unlabeled' is now a misnomer. Internally and in its interface the parameter set validation system treats this 'unlabeled' case as having module label equal to its plugin name.

The one place this affects the interface is when adding a ParameterSetDescription to a ConfigurationDescriptions object. Here is an example:

...
descriptions.add("DoodadESSource", desc1);
...
descriptions.add("myModuleLabel", desc2);

The first ParameterSetDescription added above has the same name as the plugin and so will be used for validation in the 'unlabeled' case. It is legal to add other descriptions with other labels.

As far as I know the user is never faced with an object that actually does not have a label. You have to look into the internal workings of the ParameterSet class to find anyplace where the label does not have a value equal to the plugin name. This is all hidden from the user.

 

More Complex Features Available for Use in a Description

The next several sections introduce several interesting features that

Line: 1188 to 1235
  purposes only, but when you do this the cfi files and edmPluginHelp output just print out as if the parameter were a ParameterSet and not the python SecSource type. This is incorrect behavior.
Deleted:
<
<
  • Currently this supports modules (Producers, Analyzers, Filters, and OutputModules). It also supports sources and services starting in release 3_5_0. We may add support ESSources and ESProducers.
 
  • Defaults for wildcards.
  • Placeholders in the python to allow replacing parameter values in cfg files which do not exist in the CFI files.
Line: 1210 to 1254
  of allowed values.
  • In the case of parameters that are vectors, the validation might also check that the vector is of a certain size.
Changed:
<
<
  • Fix it so it is possible to put newlines in comments.
>
>
Note that the validation system has supported EDProducers, EDAnalyzers, EDFilters, and OutputModules since release 3_1_0. It started supporting sources and services in release 3_5_0. Support for ESSources and ESProducers has been implemented and checked into CVS and will probably be available starting in release 3_11_0.
 

An example

Revision 332010-10-02 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 359 to 359
 have a setComment function. All of these comments will show up in different places in the output of edmPluginHelp.
Added:
>
>
It is important to not include newlines inside comments. edmPluginHelp will automatically break lines to adjust them to fit within terminal width or 80 characters. And this width is adjustable by using the -w option of edmPluginHelp. Currently newlines in the comments will interact badly with the automatic formatting and make a mess. We might improve this at some point to allow newlines.
 There is a lot of flexibility in how these comments can be used. Our current thought is that it is best if they are kept brief, saving long explanations for TWIKI pages or other document formats.
Line: 1203 to 1210
  of allowed values.
  • In the case of parameters that are vectors, the validation might also check that the vector is of a certain size.
Added:
>
>
  • Fix it so it is possible to put newlines in comments.
 

An example

Revision 322010-02-10 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 1124 to 1124
  all parameters not marked "untracked" are "tracked". "tracked" implies the parameter value will be saved in the output file of a cmsRun job in the provenance.
Added:
>
>
(There is one exception to this. ParameterSet's used to configure services are not saved in the provenance at all, even if they contain tracked parameters.)
 
  1. Definition of "optional": When the validation encounters a required parameter that is missing, it either inserts a default value for

Revision 312010-02-08 - ChrisDJones

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Added:
>
>
Goal: We need a way for users to determine the allowed parameters for a module and to enforce that only those allowed parameters are passed to the module (no more accidently mistyping parameter names and not noticing). The configuration validation system described on this page solves these problems.
 

Introduction

Revision 302010-02-05 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 684 to 684
 
Operators

The "xor", "and", and "or" operators described above are

Changed:
<
<
genuine overloaded C++ operators. You can use the symbol
>
>
overloaded C++ operators. You can use the symbol
 "^" instead of "xor". You can use "||" instead of "or". You can use "&&" instead of "and". They have the precedence of those operators and one can use parentheses to group them.

When filling the description these operators are binary operators.

Changed:
<
<
They take two arguments of type ParameterDescriptionNode and return a value of type ParameterDescriptionNode. This means that
>
>
They take two arguments of type "ParameterDescriptionNode const&" or "auto_ptr<ParameterDescriptionNode>" and return a value of type "auto_ptr<ParameterDescriptionNode>". This means that
 the result of any of the three operators can be fed as an argument
Changed:
<
<
to any of the three operators. The only limit to the complexity of expressions that can be created is system memory. It is probably a bad idea to try to make these expressions too complex, because beyond some point no user will be able to understand how to configure the module.

The ParameterDescriptionNode that is returned from these operators contains pointers to the two ParameterDescriptionNode's that are its arguments. When an expression is built that contains several of these operators a tree of nodes is created. The final operator evaluated returns a top level node with two pointers to other nodes and recursively following all the pointers will include all the nodes in the expression.

When filling the description, these operators are actually binary operators in every sense of the word. Validation of a ParameterSet is a different operation that occurs later than filling the description. For the "xor" operator, validation is not a binary operation at all. (The analogy between the behavior of the operator and its use in this case is not perfect.)

>
>
to any of the three operators. This allows one to build long expressions. The node created by the operator contains two smart pointers to the nodes passed to it as arguments. Filling the description with a long expression creates a node tree in memory. ParameterDescriptionNode is a base class. The actual class of the object which is pointed to by the auto_ptr returned by the operator is either XORGroupDescription, ORGroupDescription, and ANDGroupDescription. The type of the returned node is the only difference in the behavior of the three operators while filling the description. The difference in the logic for "and", "or", and "xor" does make a difference until the node is used to validate a ParameterSet.

Validation of a ParameterSet is a different operation that occurs later than filling the description. For the "xor" operator, validation is not a binary operation at all. (The analogy between the standard behavior of the operator and its use in this case is not perfect.)

 Here what happens when an "xor" node is validated.
  • The validation algorithm starts at the top of the node tree and works down.
  • When a node is encountered that is an "xor" node, the algorithm searches down the node tree following both pointers until it encounters a node that is not an "xor" node on every branch. Then all of these nodes are dealt with as group. These could be simple ParameterDescription type nodes
Changed:
<
<
or "switch" nodes or "and" nodes or other types of nodes.
>
>
or "switch" nodes or "and" nodes or any type of node other than an "xor" node.
 
  • Then it checks whether each of those nodes evaluates to true
Changed:
<
<
or false, without inserting any parameters or validating any of those nodes. (You might ask, what does it mean without validating any nodes? The entire validation process for a ParameterSet records the name of each
>
>
or false considering only the parameters that already exist in the ParameterSet and without validating any of those nodes. (You might ask, what does it mean without validating any nodes? It means two things. First, without inserting any parameters. Second, the entire validation process for a ParameterSet records the name of each
  validated parameter as it is validated. Then at the end the names in the ParameterSet are compared to the list of validated names and if there are extra names an exception is thrown. Without validating a node
Line: 762 to 764
  Both of these expressions have the same identical meaning. The node tree they create is the same. Particularly note that both evaluate false and an exception
Changed:
<
<
will get thrown if all 3 parameters exist in the ParameterSet. The validation
>
>
will get thrown if all 3 parameters exist in the ParameterSet being validated using this node. The validation
 algorithm would not evaluate B^C as (True^True = False) and then A^False as True^False which is True. The algorithm will evaluate based on all three parameters at the same time and only 1 of the three is allowed.

The "and" operator behaves in a similar way. The difference is that

Changed:
<
<
if in the validation procedure any of the nodes tested evaluates as true (for a ParameterDescription this means the parameter exists in the ParameterSet), then the validate method is called on all the nodes. For simple parameters, this means that they are inserted or an exception is thrown.
>
>
the validation method is called on all the subnodes if the top level "and" node has its validation method called. Note that the testing procedure mentioned above is actually only relevant if the "and" node is optional (discussed later). In that case if all the nodes evaluate to false then none are validated.
  The "or" operator also behaves in a similar way. The difference is that in the validation procedure only the nodes tested that evaluated as true have their validation procedure called, unless there are none. If there are none, the first node gets its validation procedure called.
Deleted:
<
<
For reference purposes, here are the names of the classes that represent the operators in the node tree: XORGroupDescription, ORGroupDescription, and ANDGroupDescription. All three classes inherit from ParameterDescriptionNode. It is possible to use these directly in a fillDescription function, but probably they should never be used directly because the operator notation is so much easier to read and understand.
 
A "Switch" - More Details

Consider this example again

Line: 1062 to 1058
  Nodes that are contained in different cases of a ParameterSwitch or different subnodes of an "exclusive or" are allowed to use the same names.

Changed:
<
<
  • Another potential problem is that a parameter insertion related to one ParameterDescription could match unrelated wildcards causing other validation requirements to change from being passing to failing or vice versa. This makes it almost impossible to determine if a ParameterSet passes validation. Each time you try to loop through and check, the result of validation could change. To avoid this problem, a list is maintained of the type for all wildcards. Another list is maintained for the type of all parameters. As new items are added we check for collisions. The function that builds the ParameterSetDescription, will throw if this rule is violated. At the moment, the criteria for a collision is matching types between a parameter and a wildcard. (This criteria is overrestrictive. With some additional CPU and code development the restriction could be loosened to parameters that might be injected cannot match the type, trackiness, and wildcard label pattern of any wildcard that requires a match. And further this could not apply to wildcards on different branches of a ParameterSwitch or "exclusive or".)
>
>
  • Within a single ParameterSetDescription a wildcard and a ParameterDescription cannot have the same type.
  There is one related problem that can occur with the current code. If insertion is necessary to make an "exclusive or" node pass
Line: 1097 to 1078
 

edmPluginHelp

The executable named edmPluginHelp will print out the configuration

Changed:
<
<
descriptions in a human readable format. This executable appeared for the first time in release CMSSW_3_1_0_pre7.

For the most part, the output of edmPluginHelp is self explanatory. The command "edmPluginHelp -h" explains the available options and gives a little information about the

>
>
descriptions in a human readable format. For the most part, the output of edmPluginHelp is self explanatory. The command "edmPluginHelp -h" explains the available options and gives a little information about the
 format. Here are some more details.

  1. The output of edmPluginHelp includes comments that describe
Changed:
<
<
the output format. In addition to these generic comments, it is also possible for a plugin to add comments into its description that will appear in the edmPluginHelp output. These are optional, but could be used to describe the plugin and its parameters in great detail.
>
>
the output format. In addition to the generic comments edmPluginHelp automatically inserts, it is also possible for a plugin to add comments into its description that will appear in the edmPluginHelp output. These comments are optional, but could help to significantly improve the clarity of the output. We recommend such comments be brief saving long explanations for other formats.
 
  1. One option to the command will cause it to produce a brief format. This suppresses most comments and produces a more compact format. Other than the comments the information is the same. For simple parameters,
Line: 1197 to 1177
  output just print out as if the parameter were a ParameterSet and not the python SecSource type. This is incorrect behavior.
  • Currently this supports modules (Producers, Analyzers, Filters,
Changed:
<
<
and OutputModules). It also supports sources starting in release 3_5_0_pre3. Support for services has been checked into CVS, but is not in any release yet. We may add support ESSources and ESProducers.
>
>
and OutputModules). It also supports sources and services starting in release 3_5_0. We may add support ESSources and ESProducers.
 
  • Defaults for wildcards.
  • Placeholders in the python to allow replacing parameter values in cfg files which do not exist in the CFI files.
Line: 1211 to 1189
 
  • Extend to support lower level plugins
  • Implement a reset function for the value of a single parameter description that makes it easier to define multiple descriptions for the same plugin
Changed:
<
<
  • Improve error checking and messages
>
>
  • Improve error checking and messages printed after exceptions
 
  • Implement the fillDescriptions function in many plugins. Currently very few plugins have this function.
  • Continue to improve this documentation

Revision 292010-02-05 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 88 to 88
 An exception would also be thrown if the parameter had the wrong type or was tracked when it should have been untracked or vice versa. If any of the three parameters above were missing in the configuration,
Changed:
<
<
then the validation would insert them into the ParameterSet that is passed
>
>
then the validation algorithm would insert them into the ParameterSet that is passed
 to the plugin constructor with default values which are provided in the second argument to the "add" function above.
Line: 139 to 139
 a "std::string const&".

In the examples on this TWIKI, some simple type like "int" is used for the parameter

Changed:
<
<
type improve readability, but the description code already fully supports all the
>
>
type to improve readability, but the description code already fully supports all the
 types listed below. The only one allowed in a python configuration that is not supported at the moment is "SecSource" (and we will probably add support for that as soon as we find the time).
Line: 507 to 507
  This creates a group of 3 parameters named "x", "y", and "z". When validating a ParameterSet this requires that all three parameters must exist in the
Changed:
<
<
ParameterSet. If some of them are missing, they will be inserted when
>
>
ParameterSet. If some or all of them are missing, they will be inserted when
 a ParameterSet is validated.

The "Or" Operator
Line: 752 to 752
 Exactly one of the parameters A and B is required to be in the ParameterSet and exactly one of the parameters C,D, and E is required to be in the ParameterSet.
Changed:
<
<
A ^ B ^ (C && D) ^ E)
>
>
A ^ B ^ (C && D) ^ E
  Exactly one of the parameters A, B, and E are required to be in the ParameterSet, except that if none of those are then both parameters C and D are required to be in the ParameterSet.

Revision 282010-02-05 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 614 to 614
 

This is something that requires nothing and allows nothing.

Added:
>
>
When evaluated during the course of validation it always has the value true.
 By itself it is meaningless, but it can be nested in and combined together with many of the features described above. In that context, it can be useful.

Details Related to the More Complex Features

Deleted:
<
<
WARNING The rest of this section and the ones that follow are still being written. It is certainly incomplete and may be in a state of change as I write new things. Some of it probably wrong.
 
ParameterDescription

ParameterDescription is the C++ class name for objects that hold the description

Line: 991 to 986
 factors allow it, then the node will get written into a CFI file.

In addition, the following functions are members of the ParameterSetDescription

Changed:
<
<
class and allow one to add optional nodes. The only difference between them and the corresponding function already described.
>
>
class and allow one to add optional nodes.
 
Changed:
<
<
>
>
ifValueOptional ifExistsOptional labelsFromOptional labelsFromOptionalUntracked

Here is how the behavior of the more complex types of nodes is affected if they are optional.

  • "and", "or", and "xor" nodes - If the parameters that are their subnodes (the arguments to their constructors) all do not exist or evaluate to false without parameter insertion, then nothing is done. Nothing is inserted, nothing is thrown, and the validation method is not called for their subnodes.
  • For a wildcard node, optional means nothing is thrown if nothing matches no matter what the criteria parameter is set to.
  • For a switch, if the switch parameter is missing, then it is not inserted, nothing is thrown, and none of the cases get validated.
  • If the ifExists node is optional, then the existence of the first node implies the second node is optional instead of implying that it is required.
  • For the AllowedLabelsDescription, it means the parameter containing the vector of names is optional. If it is not there, nothing is done during validation for that node.

An observant reader might notice that the interface only allows the top level nodes in a ParameterSetDescription to be optional. Nodes nested in other nodes cannot be optional. (Try imagining what it would mean for node inside an "xor" node to be optional ... That is only the beginning of the problems allowing optional nodes in nodes would bring. The logic would be insane, both for the user and implementer.)

Writing CFIs

Here is the behavior of the more complex nodes, when the CFI file is being generated.

  • For "and" nodes, both subnodes are written to the CFI file.
  • For "or" and "xor" nodes, only the left subnode is written to the CFI file.
  • For a switch node, the switch parameter is written to the CFI file. If that parameter has a default, it is used to select a case and the node associated with that default case is written to the CFI file.
  • For a wildcard node, nothing is written to the CFI file.
  • For an "AllowedLabels" node, the parameter that is the vector of allowed labels is written to the CFI file, but the vector is empty.
  • For an "IfExists" node, both subnodes are written to the CFI file.

Restrictions

The ParameterSetDescription system puts some restrictions on what is allowed in a description. The motivation for the restriction is that these rules prevent a problem that can occur when validating a ParameterSet. The problem is related to parameters that get injected into a ParameterSet. When this injection occurs one must worry that nodes other than the one being validated could be affected by the parameters that are being validated. In particular, nodes that have already passed validation could be put in a state where they no longer pass validation. With the flexibility the system allows, a vicious circle could be entered with great complications for the implementation. It might not even be possible to come up with a validation algorithm that would work in all cases. So there are restrictions that avoid the problem.

These restrictions have the additional benefit that the things they prohibit would tend to confuse a user trying to configure a module or a module developer writing the code to extract the parameters from a ParameterSet. These rules tend to prohibit bad design. Even if the original motivation for them did not exist, they may be a good set of rules to enforce.

Here are the restrictions:

  • The same parameter names cannot occur in different nodes of the same ParameterSetDescription. There are two exceptions to this. Nodes that are contained in different cases of a ParameterSwitch or different subnodes of an "exclusive or" are allowed to use the same names.

  • Another potential problem is that a parameter insertion related to one ParameterDescription could match unrelated wildcards causing other validation requirements to change from being passing to failing or vice versa. This makes it almost impossible to determine if a ParameterSet passes validation. Each time you try to loop through and check, the result of validation could change. To avoid this problem, a list is maintained of the type for all wildcards. Another list is maintained for the type of all parameters. As new items are added we check for collisions. The function that builds the ParameterSetDescription, will throw if this rule is violated. At the moment, the criteria for a collision is matching types between a parameter and a wildcard. (This criteria is overrestrictive. With some additional CPU and code development the restriction could be loosened to parameters that might be injected cannot match the type, trackiness, and wildcard label pattern of any wildcard that requires a match. And further this could not apply to wildcards on different branches of a ParameterSwitch or "exclusive or".)

There is one related problem that can occur with the current code. If insertion is necessary to make an "exclusive or" node pass validation, then the insertion could make more than one of the possibilities evaluate true. This must be checked for after the insertions occur. The current behavior is to throw a Configuration exception if this problem is encountered. (Example: (A && B) ^ (A && C) where C already exists in the ParameterSet but A and B do not. A and B get inserted by the algorithm, because it tries to make the first possibility true when all fail without insertion. Then both parts of the "exclusive or" pass, which is a validation failure). We are discussing improvements which may affect which subnode of an "xor" node is selected which may reduce problems like this, although probably not completely eliminate them.

 

edmPluginHelp

Revision 272010-02-04 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 121 to 121
 

Note that the interfaces for adding the types ParameterSet and

Changed:
<
<
vector< ParameterSets > are slightly different than in the example
>
>
vector<ParameterSet> are slightly different than in the example
 above. These are described in the following sections.

One point above is worth repeating. If any parameters are missing

Line: 135 to 135
  In the examples on this TWIKI, a literal string is shown for the argument for the parameter name and some other kinds of names.
Changed:
<
<
In almost all these cases, the function will accept either a "const char*" or a "std::string".
>
>
In almost all these cases, the function will accept either a "char const*" or a "std::string const&".
  In the examples on this TWIKI, some simple type like "int" is used for the parameter type improve readability, but the description code already fully supports all the
Line: 226 to 226
  A ParameterSet can contain a vector of ParameterSet's. This is implemented in the configuration by making the type of a parameter be
Changed:
<
<
vector< ParameterSet >.
>
>
vector<ParameterSet>.
 One adds one of these parameters to the description using a function named "addVPSet" instead of "add". Here is an example.
Line: 265 to 265
 default value 7. The other element of the default vector already contains "x" so it will pass through validation unmodified.
Added:
>
>
There is an "addVPSetUntracked" function also that should be used instead of "addVPSet" if the parameter is untracked.
 

Optional Parameters

All the parameters described in the sections above are required parameters.

Line: 315 to 318
  exception is thrown. If a default was provided, then it is inserted.
  • For both required and optional parameters, if no default is provided, then the parameter is not written into the generated
Changed:
<
<
CFI file.
>
>
CFI file. If a required parameter has no default, then someone using the CFI will be required to explicitly add the missing parameter in one of the python files that imports the CFI.
 

Adding Comments

Line: 452 to 455
 void Tracer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) { edm::ParameterSetDescription desc;
Changed:
<
<
desc.addUntracked<std::string>("indention", "++");
>
>
... // add parameters as usual
  descriptions.add("Tracer", desc); }
Line: 552 to 555
 desc.addWildcard("*");
Changed:
<
<
A ParameterSet is allowed to have zero or more parameters of type "int" and any name.
>
>
A ParameterSet is allowed to have zero or more tracked parameters of type "int" and any name. There is also an "addWildcardUntracked" function if the allowed parameters are untracked instead of tracked.
 
Allowed Labels
Line: 565 to 569
 

A ParameterSet is required to have a parameter whose

Changed:
<
<
name is "allowedLabels" (the argument above). This parameter is a vector of strings. The strings are names of other parameters
>
>
name is "allowedLabels" (the argument above). This parameter is a tracked vector of strings. The strings are names of other parameters
 that are allowed to be in the ParameterSet. These other parameters must have type "int" (the template
Changed:
<
<
parameter above).
>
>
parameter above) and be tracked.

There is also a "labelsFromUntracked" function if the allowed parameters and vector are untracked instead of tracked.

 
IfExists

Consider the following description.

Added:
>
>
#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h" #include "FWCore/ParameterSet/interface/ParameterSetDescription.h" #include "FWCore/ParameterSet/interface/IfExistsDescription.h" ...
 edm::ParameterSetDescription desc; desc.ifExists( edm::ParameterDescription("x", 11U, true), edm::ParameterDescription<std::string>("y", "11", true) );
Line: 796 to 808
 Each case is constructed using operator>> whose left argument identifies the case and matches the type of the switch variable and whose right argument is any kind of a ParameterDescriptionNode.
Changed:
<
<
Note the second case above the node is an "and" node containing two
>
>
Note that for the second case above, the right argument is an "and" node containing two
 parameters. These cases are connected with an "or" operator whose
Changed:
<
<
arguments are the cases (not the same "or" operator discussed in the previous section). If the value of the switch variable does not match one of the case, an exception is thrown. If the entire thing is required, then the switch variable will be inserted if it is missing. When this construction is evaluated as part of a logical expression, it evaluates as true if the switch variable exists in the ParameterSet.
>
>
arguments are the cases. This is not the same "or" operator discussed in the previous section. It has different types for its arguments and return type. The type use to represent a case does not inherit from ParameterDescriptionNode. The only use of this "or" operator is to collect the cases together to pass them into the switch.

The "ifValue" function shown above is used to insert the switch directly into the ParameterSetDescription. The ParameterSetDescription contains a vector whose elements contain a pointer to a ParameterDescriptionNode. When added directly the switch node goes into that vector. It is also possible to add a switch node to some other type of node, for example as an argument to an "and" operator. In this case, the switch must be constructed explicitly. The arguments to the constructor are the same.

edm::ParameterSetDescription desc;
edm::ParameterSwitch<std::string> switchNode(
              edm::ParameterDescription<std::string>("sswitch", "b", true),
              "a" >> edm::ParameterDescription<int>("x", 100, true) or
              "b" >> (edm::ParameterDescription<double>("y1", 101.0, true) and 
                      edm::ParameterDescription<double>("y2", 101.0, true)) or
              "c" >> edm::ParameterDescription<std::string>("z", "102", true) );
desc.addNode(switchNode);
 
Changed:
<
<
A ParameterSetDescription contains a vector of nodes. If the switch node is in that vector then it should be added with the "ifValue" function shown above. This is not possible when the switch node is nested inside of some other kind of node. Then it has to constructed directly.
>
>
When validating a ParameterSet, if the value of the switch variable does not match one of the cases in the description, an exception is thrown.
 
Added:
>
>
During validation, a node sometimes need to be evaluated for a true or false value without actually validating it. For a ParameterSwitch, it will evaluate to true if its switch variable exists in the ParameterSet. When a required ParameterSwitch is validated and the switch parameter is missing, then it will be inserted. If that parameter has no default and cannot be inserted, then an exception is thrown. Whether it was inserted or not, the node corresponding to the selected case gets validated and nothing is done with the other nodes.
 
Added:
>
>
Wildcards - More Details
 
Added:
>
>
Consider this example again:
 
Added:
>
>
edm::ParameterSetDescription desc;
desc.addWildcard<int>("*");

The wildcard pattern used here is "*". This will match any parameter name in a ParameterSet. Currently, that is the only pattern allowed. We have discussed extending this to support regular expressions or globbing. If there were actually requests for this feature, it would motivate us to implement this faster. At the moment it is fairly low on the priority list.

 
Changed:
<
<
Other random stuff being edited
>
>
The function above supports adding the node directly to the ParameterSetDescription, but one can also construct the node directly and add it to some other kind of node, for example as an operator argument or switch case.

edm::ParameterSetDescription desc;
edm::ParameterWildcard<int> wnode("*", edm::RequireExactlyOne, true);
desc.addNode(wnode);
 
Added:
>
>
The class ParameterWildcard inherits from ParameterDescriptionNode. In the example above, the template argument is the type which parameter types must match. The first argument is the pattern which parameter names must match. The second argument in an enumeration and there are 3 allowed values.
  • edm::RequireZeroOrMore
  • edm::RequireAtLeastOne,
  • edm::RequireExactlyOne
These are requirements on the number of matches that must exist for the wildcard node to pass validation. Also note that currently, the class ParameterWildcard does not allow defaults. There are never any parameters inserted for a wildcard. If the enum requires matches and they do not exist, an exception is thrown. Finally, the third argument is a boolean. If it is true then parameters must be tracked to match the wildcard. Otherwise they must be untracked.
 
Changed:
<
<
Adding a parameter to a ParameterSetDescription can be viewed as adding a statement that the parameter will exist in a ParameterSet when it is validated. If the parameter exists, then the statement is true. If the parameter does not exist, then the statement is false. If we view these as statements that are either true or false, we can then use logical operators to create more complicated statements. For example, we could say that either parameter "x" exists or parameter "y" exists but not both.
>
>
If the type that the parameter must match is a ParameterSet or vector<ParameterSet> then the constructor is defined as follows. Notice the template parameter is different and there is extra argument which will be used to validate any ParameterSet's which match the wildcard. The extra argument can be omitted and if it is omitted the matching ParameterSet's are not validated.
 
edm::ParameterSetDescription desc;
Changed:
<
<
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true) xor edm::ParameterDescription("y", 11U, true) );
>
>
edm::ParameterSetDescription val; ... add content to val ... edm::ParameterWildcard<edm::ParameterSetDescription> wnode("*", edm::RequireExactlyOne, true, val); desc.addNode(wnode);
 
Deleted:
<
<
When validating a ParameterSet using this description, an exception will be thrown if both "x" and "y" exist in the ParameterSet. If only one exists, then the ParameterSet will pass validation and the statement made by the "xor" expression will evaluate as true. If neither "x" nor "y" exists in the ParameterSet, then the validation algorithm will insert the first parameter, which is "x" in this case. After the insertion the ParameterSet will pass validation and the statement made by the "xor" statement will evaluate to true. (In the more general case, the insertion might fail and also after the insertion the entire "xor" expression would need to be reevaluated.)

When generating a CFI file, the first argument of an "xor" expression is written to the CFI file. The other argument is not. In the example above, "x" is written to the CFI file and "y" is not.

The operator "xor" in the above example is the C++ operator "^" and the symbol "^" can be used in place of "xor" above. It behaves with the precedence of that operator.

You can replace the function "addNode" in the above example with the function "addOptionalNode". The fact that it is optional changes the behavior during validation only in the case where neither parameter is present. In that case, nothing is done. Nothing gets injected and an exception is not thrown. In addition to making the expression optional, there is an extra boolean argument that controls whether anything will be written into a CFI file for this expression. If it is false, neither "x" nor "y" would be written to the CFI file.

 
edm::ParameterSetDescription desc;
Changed:
<
<
desc.addOptionalNode( edm::ParameterDescription<std::string>("x", "11", true) xor edm::ParameterDescription("y", 11U, true), true );
>
>
edm::ParameterSetDescription val; ... add content to val ... edm::ParameterWildcard<std::vector<edm::ParameterSet> > wnode("*", edm::RequireExactlyOne, true, val); desc.addNode(wnode);
 
Changed:
<
<
The examples in this section use two names that have not been defined yet, node and ParameterDescription. These are discussed in the following section. The next section also explains the behavior of "xor" during validation when two or more are used together.

ParameterDescriptionNode's and ParameterDescription's

It is of some importance that both the arguments and return type of the "xor" operator have type ParameterDescriptionNode. That means any type of node can be an operand and the expression itself can be used as an operand in other expressions. The following sections describe several other types of nodes: "and" operators, "or" operators, a switch operator, and others. The "xor" expression can be used in those expressions, for for example as a case in a switch statement. And also an "and" expression can be used as the operand in an "xor" expression. There is no limit to level of nesting allowed other than available system memory of your machine.

There is a node created for each binary operator. Each of these nodes points to two other nodes. These nodes could be two simple parameters or they could be other binary operators or other types of nodes. The normal rules of precedence are followed when creating the node tree for binary operators.

>
>
Allowed Labels - More Details

The labelsFrom function also creates a node that is added directly to the ParameterSetDescription.

 
Deleted:
<
<
There is one point that I glossed over above that a very perceptive reader might have noticed. The expression:
 
Changed:
<
<
A ^ B ^ C
>
>
edm::ParameterSetDescription desc; desc.labelsFrom("allowedLabels");
 
Changed:
<
<
would evaluate to true if all three statements A, B, and C were true. But this is not the behavior in the example above. Here is how it works. A ParameterSetDescription contains a vector of nodes. When the a ParameterSet is validated using that description, it loops over the top level nodes of the description and calls the validate method of each node.
  • For single parameter nodes, the parameter is validated if it exists or is successfully inserted. The names of validated names are saved.
  • More complicated nodes like "xor" nodes can contain other nodes in a tree structure. When an "xor" node is encountered the node tree is searched downward and the search continues to nodes contained in nodes when other "xor" nodes are encountered. So a group of subnodes of "xor" nodes is found. Then the logical requirement is then applied to the entire group. The requirement is that only one of the nodes can be true (which means exist for simple parameters). Only the one node that exists is actually validated and the associated name or names added to the list of validated

In the description language, only one of the 3 statements would be allowed to be true. On validation, the node tree is searched downward until something other than an "xor" type node is found on all the branches. And then the logical condition is imposed on all the nodes that are found that only one of them can be true. The operation "xor" is binary when the node tree is being built, but it is not a binary operation when the tree is being used to evaluate whether a particular ParameterSet will pass validation.

Wildcards

>
>
This node can also be explicitly constructed so that it can be added to other nodes. The class AllowedLabelsDescription inherits from ParameterDescriptionNode.
 
Added:
>
>
edm::ParameterSetDescription desc; edm::AllowedLabelsDescription node("allowedLabels", true); desc.addNode(node);

AllowedLabelsDescription inherits from ParameterDescriptionNode. In the above example, the template parameter gives the type of the parameters that are allowed to exist. The first argument is the name of the parameter that holds the vector of strings which are the names of the parameters that are allowed to exist. The second argument is a boolean that is true if the allowed parameters must be tracked, otherwise they must be untracked.

These following two should be used in the special case that the allowed parameter type would be a ParameterSet or vector<ParameterSet>. The added second argument would be used validate the allowed ParameterSet's. That argument can be omitted and the nested ParameterSet's would not be validated.

edm::ParameterSetDescription desc;
edm::ParameterSetDescription val;
edm::AllowedLabelsDescription<edm::ParameterSetDescription> node("allowedLabels", val, true);
desc.addNode(node);

edm::ParameterSetDescription desc;
edm::ParameterSetDescription val;
edm::AllowedLabelsDescription<std::vector<edm::ParameterSet> > node("allowedLabels", val, true);
desc.addNode(node);

IfExists - More Details

The ifExists function also creates a node that is added directly to the ParameterSetDescription.

edm::ParameterSetDescription desc;
desc.ifExists( edm::ParameterDescription<unsigned int>("x", 11U, true),
               edm::ParameterDescription<std::string>("y", "11", true) );
 
Changed:
<
<
edm::ParameterSetDescription wildcardPset;

wildcardPset.addWildcard("*"); wildcardPset.addWildcardUntracked(std::string("*");

std::auto_ptr<edm::ParameterDescriptionNode> wnode(new edm::ParameterWildcard<edm::ParameterSetDescription>("*", edm::RequireExactlyOne, true)); wildcardPset.addOptionalNode(wnode, false);

edm::ParameterSetDescription wSet1; wSet1.add("Drinks", 5);

std::auto_ptr<edm::ParameterDescriptionNode> wnode2(new edm::ParameterWildcard<edm::ParameterSetDescription>("*", edm::RequireAtLeastOne, true, wSet1)); wildcardPset.addOptionalNode(wnode2, false);

std::auto_ptr<edm::ParameterDescriptionNode> wnode3(new edm::ParameterWildcard<std::vector<edm::ParameterSet> >("*", edm::RequireExactlyOne, true)); wildcardPset.addOptionalNode(wnode3, false);

>
>
This node can also be explicitly constructed so that it can be added to other nodes. The class IfExistsDescription inherits from ParameterDescriptionNode.
 
Changed:
<
<
wSet1.add("Drinks2", 11);
>
>
edm::ParameterSetDescription desc;
IfExistsDescription node( edm::ParameterDescription<unsigned int>("x", 11U, true),
                          edm::ParameterDescription<std::string>("y", "11", true) );
desc.addNode(node);

Optional Nodes
 
Changed:
<
<
std::auto_ptr<edm::ParameterDescriptionNode> wnode4(new edm::ParameterWildcard<std::vector<edm::ParameterSet> >("*", edm::RequireAtLeastOne, true, wSet1)); wildcardPset.addOptionalNode(wnode4, false);
>
>
In a much earlier section above we discussed optional parameters. These are optional nodes where the node has type ParameterDescription. There is another way one can add these to a ParameterSetDescription.
 
Changed:
<
<
iDesc.add("wildcardPset", wildcardPset);
>
>
edm::ParameterSetDescription desc;
desc.addOptionalNode( edm::ParameterDescription<int>("algorithm", 1, true),
                      true);
 
Added:
>
>
The first argument is the node. It does not have to be a ParameterDescription node, it can be any type inheriting from ParameterDescriptionNode. The second argument is something not available in the addNode function. It is a boolean. If it is false the node will not be written into the generated CFI file. If it is true and other factors allow it, then the node will get written into a CFI file.

In addition, the following functions are members of the ParameterSetDescription class and allow one to add optional nodes. The only difference between them and the corresponding function already described.

 

edmPluginHelp

Revision 262010-02-04 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 543 to 543
 "y2" are both required. If "sswitch" has the value "c", then parameter "z" is required.
Deleted:
<
<
The switch parameter is allowed to have one of three types: string, int, or bool.
 
Wildcards

Consider the following description.

Line: 681 to 679
 The "xor", "and", and "or" operators described above are genuine overloaded C++ operators. You can use the symbol "^" instead of "xor". You can use "||" instead of "or". You can
Changed:
<
<
use "&&" instead of "and". The have the precedence of those
>
>
use "&&" instead of "and". They have the precedence of those
 operators and one can use parentheses to group them.

When filling the description these operators are binary operators.

Line: 702 to 700
 following all the pointers will include all the nodes in the expression.

When filling the description, these operators are actually binary operators

Changed:
<
<
in every sense of the word. Validation is a different operation. For the "xor" operator it is not a binary operation at all. Here what happens when an "xor" node is validated.
  • If one were to search down the node tree, following both pointers at every step, until hitting a node that is not an "xor" node on every path.
  • Then it checks whether each of those evaluates to true or false, without inserting any parameters or validating any of those nodes. For node that is just a simple parameter, true means that the parameter exists in the ParameterSet being validated. (This means asking the question would this validate as true without inserting any parameters without actually validating its contents)
>
>
in every sense of the word. Validation of a ParameterSet is a different operation that occurs later than filling the description. For the "xor" operator, validation is not a binary operation at all. (The analogy between the behavior of the operator and its use in this case is not perfect.) Here what happens when an "xor" node is validated.
  • The validation algorithm starts at the top of the node tree and works down.
  • When a node is encountered that is an "xor" node, the algorithm searches down the node tree following both pointers until it encounters a node that is not an "xor" node on every branch. Then all of these nodes are dealt with as group. These could be simple ParameterDescription type nodes or "switch" nodes or "and" nodes or other types of nodes.
  • Then it checks whether each of those nodes evaluates to true or false, without inserting any parameters or validating any of those nodes. (You might ask, what does it mean without validating any nodes? The entire validation process for a ParameterSet records the name of each validated parameter as it is validated. Then at the end the names in the ParameterSet are compared to the list of validated names and if there are extra names an exception is thrown. Without validating a node means the names in that node are not recorded on the list of validated names yet).
  • Then it examines how many nodes evaluated to true.
 
  • If it finds more than one that is true, it just throws an exception.
  • If it finds one that is true, then the validation method is called on that one node. All the other nodes and whatever parameters they might contain
Line: 720 to 728
  selected node validates them also. *If it finds that none are true, then calls the validation method on the first node. If this were a single parameter and it had a default, it would be inserted.
Changed:
<
<
This previous paragraph is cryptic enough that most people probably will not follow what it said. Here are some examples.
>
>
In this case, after the validation is complete it goes back and tests that the entire set of "xor" related nodes evaluates to true because validating one node might have had side effects on other nodes. Here are some examples that might make some of the consequences of this procedure more clear. For clarity, I've simplified the notation and not given the full C++ syntax. Let A, B, C, D, and E represent some kind of node, they could be a single parameter in the description.
 
A ^ B ^ C ^ D ^ E
Exactly one of the parameters A,B,C,D, and E is required to be in the ParameterSet.
Changed:
<
<
(A ^ B ^ C) && (D ^ E)
>
>
(A ^ B) && (C ^ D ^ E)
  Exactly one of the parameters A and B is required to be in the ParameterSet and exactly one of the parameters C,D, and E is required to be in the ParameterSet.
Line: 748 to 760
 as True^False which is True. The algorithm will evaluate based on all three parameters at the same time and only 1 of the three is allowed.
Added:
>
>
The "and" operator behaves in a similar way. The difference is that if in the validation procedure any of the nodes tested evaluates as true (for a ParameterDescription this means the parameter exists in the ParameterSet), then the validate method is called on all the nodes. For simple parameters, this means that they are inserted or an exception is thrown.

The "or" operator also behaves in a similar way. The difference is that in the validation procedure only the nodes tested that evaluated as true have their validation procedure called, unless there are none. If there are none, the first node gets its validation procedure called.

For reference purposes, here are the names of the classes that represent the operators in the node tree: XORGroupDescription, ORGroupDescription, and ANDGroupDescription. All three classes inherit from ParameterDescriptionNode. It is possible to use these directly in a fillDescription function, but probably they should never be used directly because the operator notation is so much easier to read and understand.

 
Added:
>
>
A "Switch" - More Details
 
Added:
>
>
Consider this example again
 
Added:
>
>
edm::ParameterSetDescription desc;
desc.ifValue( edm::ParameterDescription<std::string>("sswitch", "b", true),
              "a" >> edm::ParameterDescription<int>("x", 100, true) or
              "b" >> (edm::ParameterDescription<double>("y1", 101.0, true) and 
                      edm::ParameterDescription<double>("y2", 101.0, true)) or
              "c" >> edm::ParameterDescription<std::string>("z", "102", true) );
 
Added:
>
>
The second argument to ifValue function is a fairly complicated thing. Each case is constructed using operator>> whose left argument identifies the case and matches the type of the switch variable and whose right argument is any kind of a ParameterDescriptionNode. Note the second case above the node is an "and" node containing two parameters. These cases are connected with an "or" operator whose arguments are the cases (not the same "or" operator discussed in the previous section). If the value of the switch variable does not match one of the case, an exception is thrown. If the entire thing is required, then the switch variable will be inserted if it is missing. When this construction is evaluated as part of a logical expression, it evaluates as true if the switch variable exists in the ParameterSet.

A ParameterSetDescription contains a vector of nodes. If the switch node is in that vector then it should be added with the "ifValue" function shown above. This is not possible when the switch node is nested inside of some other kind of node. Then it has to constructed directly.

 

Revision 252010-02-04 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 641 to 641
 
edm::ParameterSetDescription desc;
... content gets added to desc
Changed:
<
<
edm::ParameterDescription<edm::ParameterSetDescription>("aDesc", desc, true)
>
>
edm::ParameterDescription<edm::ParameterSetDescription>("x", desc, true)
  In this case, the template argument and second argument are ParameterSetDescription instead of being ParameterSet, which
Line: 654 to 654
 ... content gets added to desc std::vector<edm::ParameterSet> vectorOfPSets; ... content gets added to vectorOfPSets
Changed:
<
<
edm::ParameterDescription<std::vector<edm::ParameterSet> >("aVectorOfPSets", desc, true, vectorOfPSets);
>
>
edm::ParameterDescription<std::vector<edm::ParameterSet> >("x", desc, true, vectorOfPSets);
  In this case, the second argument is used to validate every element of the vector of ParameterSet's. The fourth argument is the default
Line: 676 to 676
 desc.add("algorithm", 1);
Added:
>
>
Operators

The "xor", "and", and "or" operators described above are genuine overloaded C++ operators. You can use the symbol "^" instead of "xor". You can use "||" instead of "or". You can use "&&" instead of "and". The have the precedence of those operators and one can use parentheses to group them.

When filling the description these operators are binary operators. They take two arguments of type ParameterDescriptionNode and return a value of type ParameterDescriptionNode. This means that the result of any of the three operators can be fed as an argument to any of the three operators. The only limit to the complexity of expressions that can be created is system memory. It is probably a bad idea to try to make these expressions too complex, because beyond some point no user will be able to understand how to configure the module.

The ParameterDescriptionNode that is returned from these operators contains pointers to the two ParameterDescriptionNode's that are its arguments. When an expression is built that contains several of these operators a tree of nodes is created. The final operator evaluated returns a top level node with two pointers to other nodes and recursively following all the pointers will include all the nodes in the expression.

When filling the description, these operators are actually binary operators in every sense of the word. Validation is a different operation. For the "xor" operator it is not a binary operation at all. Here what happens when an "xor" node is validated.

  • If one were to search down the node tree, following both pointers at every step, until hitting a node that is not an "xor" node on every path.
  • Then it checks whether each of those evaluates to true or false, without inserting any parameters or validating any of those nodes. For node that is just a simple parameter, true means that the parameter exists in the ParameterSet being validated. (This means asking the question would this validate as true without inserting any parameters without actually validating its contents)
  • If it finds more than one that is true, it just throws an exception.
  • If it finds one that is true, then the validation method is called on that one node. All the other nodes and whatever parameters they might contain are not validated and should not appear in the ParameterSet unless the selected node validates them also. *If it finds that none are true, then calls the validation method on the first node. If this were a single parameter and it had a default, it would be inserted.

This previous paragraph is cryptic enough that most people probably will not follow what it said. Here are some examples.

A ^ B ^ C ^ D ^ E
Exactly one of the parameters A,B,C,D, and E is required to be in the ParameterSet.
(A ^ B ^ C) && (D ^ E)
Exactly one of the parameters A and B is required to be in the ParameterSet and exactly one of the parameters C,D, and E is required to be in the ParameterSet.
A ^ B ^ (C && D) ^ E)
Exactly one of the parameters A, B, and E are required to be in the ParameterSet, except that if none of those are then both parameters C and D are required to be in the ParameterSet.
A ^ B ^ C
A ^ (B ^ C)
Both of these expressions have the same identical meaning. The node tree they create is the same. Particularly note that both evaluate false and an exception will get thrown if all 3 parameters exist in the ParameterSet. The validation algorithm would not evaluate B^C as (True^True = False) and then A^False as True^False which is True. The algorithm will evaluate based on all three parameters at the same time and only 1 of the three is allowed.

 
Other random stuff being edited

Revision 242010-02-03 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 461 to 461
 or a service to have a module label. That is why one is not allowed in the description either.
Changed:
<
<

Logical Exclusive Or Operator

>
>

More Complex Features Available for Use in a Description

The next several sections introduce several interesting features that can be used when writing a fillDescriptions function. These sections are intentionally brief and they do not explain all the details related to these features. With your intuition, you might be able to use the features without reading any additional documentation. After the brief sections there is additional documentation that explains in detail how these features work and can be used together.

The "Exclusive Or" Operator

Consider the following description.

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true) xor 
              edm::ParameterDescription<unsigned int>("y", 11U, true) xor
              edm::ParameterDescription<double>("z", 11.0, true ));

This creates a group of 3 parameters named "x", "y", and "z". When validating a ParameterSet, this requires that exactly one parameter from that group is in the ParameterSet. If more than one of them exists in the ParameterSet, then an exception would be thrown. If none of them exist, then it would attempt to insert the first parameter in the group into the ParameterSet. In this particular case this would always succeed and the ParameterSet would pass validation. If exactly one of these parameters already existed in the ParameterSet, then it would pass validation without anything being inserted.

The "And" Operator

Consider the following description.

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true) and
              edm::ParameterDescription<unsigned int>("y", 11U, true) and
              edm::ParameterDescription<double>("z", 11.0, true ));

This creates a group of 3 parameters named "x", "y", and "z". When validating a ParameterSet this requires that all three parameters must exist in the ParameterSet. If some of them are missing, they will be inserted when a ParameterSet is validated.

The "Or" Operator

Consider the following description.

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true) or
              edm::ParameterDescription<unsigned int>("y", 11U, true) or
              edm::ParameterDescription<double>("z", 11.0, true ));

This creates a group of 3 parameters named "x", "y", and "z". When validating a ParameterSet this requires that at least one of the three parameters must exist in the ParameterSet. If one or more of them exists, then the ParameterSet will pass validation. If all three are not present, then the first one, "x", will be inserted into the ParameterSet.

A "Switch"

Consider the following description.

edm::ParameterSetDescription desc;
desc.ifValue( edm::ParameterDescription<std::string>("sswitch", "b", true),
              "a" >> edm::ParameterDescription<int>("x", 100, true) or
              "b" >> (edm::ParameterDescription<double>("y1", 101.0, true) and 
                      edm::ParameterDescription<double>("y2", 101.0, true)) or
              "c" >> edm::ParameterDescription<std::string>("z", "102", true) );

The parameter "sswitch" controls which other parameters are required to be in a ParameterSet. If "sswitch" has the value "a", then the parameter "x" is required. If "sswitch" has the value "b", then parameters "y1" and "y2" are both required. If "sswitch" has the value "c", then parameter "z" is required.

The switch parameter is allowed to have one of three types: string, int, or bool.

Wildcards

Consider the following description.

edm::ParameterSetDescription desc;
desc.addWildcard<int>("*");

A ParameterSet is allowed to have zero or more parameters of type "int" and any name.

Allowed Labels

Consider the following description.

edm::ParameterSetDescription desc;
desc.labelsFrom<int>("allowedLabels");

A ParameterSet is required to have a parameter whose name is "allowedLabels" (the argument above). This parameter is a vector of strings. The strings are names of other parameters that are allowed to be in the ParameterSet. These other parameters must have type "int" (the template parameter above).

IfExists

Consider the following description.

edm::ParameterSetDescription desc;
desc.ifExists( edm::ParameterDescription<unsigned int>("x", 11U, true),
               edm::ParameterDescription<std::string>("y", "11", true) );

If parameter "x" exists in a ParameterSet, then parameter "y" is required to also exist. If both do not exist it is OK and this will pass validation without any insertion of missing variables. If parameter "y" exists and parameter "x" does not exist, then "x" will be inserted into the ParameterSet.

Empty Description

Consider the following description.

#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"
#include "FWCore/ParameterSet/interface/EmptyGroupDescription.h"
...
edm::ParameterSetDescription desc;
edm::EmptyGroupDescription emptyGroup;
desc.addNode(emptyGroup);

This is something that requires nothing and allows nothing. By itself it is meaningless, but it can be nested in and combined together with many of the features described above. In that context, it can be useful.

Details Related to the More Complex Features

  WARNING
Line: 470 to 617
 as I write new things. Some of it probably wrong.
Added:
>
>
ParameterDescription

ParameterDescription is the C++ class name for objects that hold the description of a single parameter. Here is how one is created.

edm::ParameterDescription<std::string>("x", "11", true)
The template argument is the type of the parameter. The first function argument of the constructor is the parameter name. The second argument is the default value of the parameter and has the same type as the template argument. The third argument is a boolean that is true if the parameter is "tracked" and false if it is "untracked". There is another constructor that is the same except the argument providing the default value is omitted. When the constructor without a default is used, the parameter does not get written into CFI files and also cannot be inserted into ParameterSet's during validation. If the parameter is required and missing, then an exception gets thrown.

There are two types of parameters where the constructor is different. First, when the parameter type is ParameterSet:

edm::ParameterSetDescription desc;
... content gets added to desc
edm::ParameterDescription<edm::ParameterSetDescription>("aDesc", desc, true)
In this case, the template argument and second argument are ParameterSetDescription instead of being ParameterSet, which is the actual type of the parameter. The second argument is also a more than a default, it is also used to validate the nested ParameterSet and it must always be present. The other special case occurs when the type of the parameter is vector of ParameterSet.
edm::ParameterSetDescription desc;
... content gets added to desc
std::vector<edm::ParameterSet> vectorOfPSets;
... content gets added to vectorOfPSets
edm::ParameterDescription<std::vector<edm::ParameterSet> >("aVectorOfPSets", desc, true, vectorOfPSets);
In this case, the second argument is used to validate every element of the vector of ParameterSet's. The fourth argument is the default and there is another constructor with the default omitted.

The base class of ParameterDescription is ParameterDescriptionNode. A single parameter is a ParameterDescriptionNode, but it is not the only kind of ParameterDescriptionNode. Because it is a node this works:

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<int>("algorithm", 1, true));

This is equivalent to

edm::ParameterSetDescription desc;
desc.add<int>("algorithm", 1);

Other random stuff being edited
 Adding a parameter to a ParameterSetDescription can be viewed as adding a statement that the parameter will exist in a ParameterSet when it is validated. If the parameter exists, then the statement is true. If the parameter
Line: 524 to 733
 

ParameterDescriptionNode's and ParameterDescription's

Deleted:
<
<
A ParameterDescription is the class name for objects that hold the description of a single parameter. The template argument is the type of the parameter. The first argument of its constructor is the name of the parameter. The second argument is the default value of the parameter. The third argument specifies whether the parameter is tracked. It also possible to use a different constructor that has only two arguments by dropping the second argument that gives a default value. If this is done, the parameter cannot be inserted if missing or written into CFI files.

The base class of ParameterDescription is ParameterDescriptionNode. A single parameter is a ParameterDescriptionNode, but it is not the only kind of ParameterDescriptionNode. The "xor" operator returns a value whose type also inherits from ParameterDescriptionNode (XORGroupDescription). The operands of the "xor" operator can be any type of ParameterDescriptionNode. This allows one to extend the example from the previous section to 3 or more parameters. For example, we could say that either "x" or "y" or "z" exists in the ParameterSet being validated, but only 1 of them.

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true) xor 
              edm::ParameterDescription<unsigned int>("y", 11U, true) xor
              edm::ParameterDescription<double>("z", 11.0, true ));

Also note that the function that adds this expression to the ParameterSetDescription is called "addNode". It can take as an argument any type of ParameterDescriptionNode. This will work also:

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true));

There are two special cases where the constructor of the ParameterDescription is a little different. Here is how it looks in those cases.

edm::ParameterSetDescription desc;

edm::ParameterSetDescription desc1;
edm::ParameterSetDescription desc2;
std::vector<edm::ParameterSet> vectorOfPSets;

desc.addNode( edm::ParameterDescription<edm::ParameterSetDescription>("aDesc", desc1, true) xor
              edm::ParameterDescription<std::vector<edm::ParameterSet> >("aVectorOfPSets1", desc2, true, vectorOfPSets) xor
              edm::ParameterDescription<std::vector<edm::ParameterSet> >("aVectorOfPSets2", desc2, true));

These behave as described in the sections above titled "!ParameterSets that Contain ParameterSets" and "!ParameterSets that Contain Vectors of ParameterSets". The first argument is the parameter name. The second argument is the ParameterSetDescription used to validate the ParameterSet or all the ParameterSet's in the vector. The third argument is a boolean that is true if the parameter is a tracked parameter. In the vector case, there is allowed to be one additional argument that specifies the default.

In the example of above the ParameterSetDescription's and vector inside the expression are empty only to make the example simple. Normally, they would have contents and those contents are not constrained by the fact that they are inside the expression. Also note that the evaluation of the "xor" expression only depends on the existence of the parameters, it does not depend on their contents in any way. If it validates as true, then the one option that exists is validated and that includes includes validation of all ParameterSet's it contains.

  It is of some importance that both the arguments and return type of the "xor" operator have type ParameterDescriptionNode. That means
Line: 672 to 815
  iDesc.add("wildcardPset", wildcardPset);
Deleted:
<
<

Other Things

Here are the topics that still need to be explained:

wildcards
AND
OR
XOR
ifValue
ifExists
allowedLabels
 

edmPluginHelp

Revision 232010-02-03 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 457 to 457
 }
Changed:
<
<
You cannot supply a module label for a source or service because they are not allowed to have a module label in a python configuration.
>
>
The python configuration syntax for cmsRun does not allow a source or a service to have a module label. That is why one is not allowed in the description either.
 

Logical Exclusive Or Operator

Added:
>
>
WARNING The rest of this section and the ones that follow are still being written. It is certainly incomplete and may be in a state of change as I write new things. Some of it probably wrong.
 Adding a parameter to a ParameterSetDescription can be viewed as adding a statement that the parameter will exist in a ParameterSet when it is validated. If the parameter exists, then the statement is true. If the parameter
Changed:
<
<
does not exist, then the statement is false. When we view these as statements
>
>
does not exist, then the statement is false. If we view these as statements
 that are either true or false, we can then use logical operators to
Changed:
<
<
create more complicated statements. For example, we could say that either parameter "x" exists or parameter "y" exists
>
>
create more complicated statements. For example, we could say that either parameter "x" exists or parameter "y" exists
 but not both.
Line: 481 to 487
  When validating a ParameterSet using this description, an exception will be thrown if both "x" and "y" exist in the ParameterSet. If only one exists, then
Changed:
<
<
validation will pass and the node created by the "xor" expression will evaluate as true. If neither "x" nor "y" exists in the configuration, then the validation algorithm will try to insert the first parameter in the expression into the ParameterSet. If successful, then the expression will still evaluate as true and validation will pass. Otherwise an exception will be thrown.

When generating the CFI file, the first argument of this expression is written to the CFI file. The other argument of the expression is not.

>
>
the ParameterSet will pass validation and the statement made by the "xor" expression will evaluate as true. If neither "x" nor "y" exists in the ParameterSet, then the validation algorithm will insert the first parameter, which is "x" in this case. After the insertion the ParameterSet will pass validation and the statement made by the "xor" statement will evaluate to true. (In the more general case, the insertion might fail and also after the insertion the entire "xor" expression would need to be reevaluated.)

When generating a CFI file, the first argument of an "xor" expression is written to the CFI file. The other argument is not. In the example above, "x" is written to the CFI file and "y" is not.

  The operator "xor" in the above example is the C++ operator "^" and the symbol "^" can be used in place of "xor" above. It behaves with the precedence of that operator.
Changed:
<
<
Optionally, one can replace the function "addNode" in the above example with the function "addOptionalNode". There is an extra boolean arguement in this case that controls whether something will be written into a CFI file for this expression. The fact that it is optional changes the behavior during validation only in the case where neither parameter is present. In that case, nothing is done. Nothing gets injected and an exception is not thrown.
>
>
You can replace the function "addNode" in the above example with the function "addOptionalNode". The fact that it is optional changes the behavior during validation only in the case where neither parameter is present. In that case, nothing is done. Nothing gets injected and an exception is not thrown. In addition to making the expression optional, there is an extra boolean argument that controls whether anything will be written into a CFI file for this expression. If it is false, neither "x" nor "y" would be written to the CFI file.
 
edm::ParameterSetDescription desc;
desc.addOptionalNode( edm::ParameterDescription<std::string>("x", "11", true) xor 
Line: 509 to 519
  The examples in this section use two names that have not been defined yet, node and ParameterDescription. These are discussed in the following section.
Added:
>
>
The next section also explains the behavior of "xor" during validation when two or more are used together.
 

ParameterDescriptionNode's and ParameterDescription's

A ParameterDescription is the class name for objects that hold the description of a single parameter. The template argument is the type of the parameter.

Changed:
<
<
The first function argument is the name of the parameter. The second argument
>
>
The first argument of its constructor is the name of the parameter. The second argument
 is the default value of the parameter. The third argument specifies whether the parameter is tracked. It also possible to use a different constructor that has only two arguments by dropping the second argument that gives a default
Line: 546 to 558
 desc.addNode( edm::ParameterDescription<std::string>("x", "11", true));
Deleted:
<
<
Internally, this is what the "add" function described in previous sections is doing.
 There are two special cases where the constructor of the ParameterDescription is a little different. Here is how it looks in those cases.
Line: 592 to 602
 "xor" expression. There is no limit to level of nesting allowed other than available system memory of your machine.
Added:
>
>
There is a node created for each binary operator. Each of these nodes points to two other nodes. These nodes could be two simple parameters or they could be other binary operators or other types of nodes. The normal rules of precedence are followed when creating the node tree for binary operators.

There is one point that I glossed over above that a very perceptive reader might have noticed. The expression:

A ^ B ^ C
would evaluate to true if all three statements A, B, and C were true. But this is not the behavior in the example above. Here is how it works. A ParameterSetDescription contains a vector of nodes. When the a ParameterSet is validated using that description, it loops over the top level nodes of the description and calls the validate method of each node.
  • For single parameter nodes, the parameter is validated if it exists or is successfully inserted. The names of validated names are saved.
  • More complicated nodes like "xor" nodes can contain other nodes in a tree structure. When an "xor" node is encountered the node tree is searched downward and the search continues to nodes contained in nodes when other "xor" nodes are encountered. So a group of subnodes of "xor" nodes is found. Then the logical requirement is then applied to the entire group. The requirement is that only one of the nodes can be true (which means exist for simple parameters). Only the one node that exists is actually validated and the associated name or names added to the list of validated

In the description language, only one of the 3 statements would be allowed to be true. On validation, the node tree is searched downward until something other than an "xor" type node is found on all the branches. And then the logical condition is imposed on all the nodes that are found that only one of them can be true. The operation "xor" is binary when the node tree is being built, but it is not a binary operation when the tree is being used to evaluate whether a particular ParameterSet will pass validation.

 

Wildcards

Line: 623 to 674
 

Other Things

Deleted:
<
<
WARNING The rest of this section is still being written. It is certainly incomplete and may be in a state of change as I write new things.
  Here are the topics that still need to be explained:

Revision 222010-02-02 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 464 to 464
 

Logical Exclusive Or Operator

Changed:
<
<
Adding a parameter to a ParameterSetDescription can be viewed as a statement that the parameter is present in the ParameterSet being validated. If the parameter is present, then the statement is true. If the parameter is not present then the statement is false. When we view these as statements
>
>
Adding a parameter to a ParameterSetDescription can be viewed as adding a statement that the parameter will exist in a ParameterSet when it is validated. If the parameter exists, then the statement is true. If the parameter does not exist, then the statement is false. When we view these as statements
 that are either true or false, we can then use logical operators to create more complicated statements. For example, we could say
Changed:
<
<
that either parameter "x" is present or parameter "y" is present
>
>
that either parameter "x" exists or parameter "y" exists
 but not both.
Line: 481 to 481
  When validating a ParameterSet using this description, an exception will be thrown if both "x" and "y" exist in the ParameterSet. If only one exists, then
Changed:
<
<
validation will pass and this xor expression will evaluate as true. If neither "x" nor "y" exists in the configuration, then the validation algorithm will try
>
>
validation will pass and the node created by the "xor" expression will evaluate as true. If neither "x" nor "y" exists in the configuration, then the validation algorithm will try
 to insert the first parameter in the expression into the ParameterSet.
Changed:
<
<
If successful then the expression will still evaluate as true and
>
>
If successful, then the expression will still evaluate as true and
 validation will pass. Otherwise an exception will be thrown.
Added:
>
>
When generating the CFI file, the first argument of this expression is written to the CFI file. The other argument of the expression is not.
 The operator "xor" in the above example is the C++ operator "^" and
Changed:
<
<
the "^" can be used in place of "xor" above. It behaves with the precedence and of that operator.
>
>
the symbol "^" can be used in place of "xor" above. It behaves with the precedence of that operator.

Optionally, one can replace the function "addNode" in the above example with the function "addOptionalNode". There is an extra boolean arguement in this case that controls whether something will be written into a CFI file for this expression. The fact that it is optional changes the behavior during validation only in the case where neither parameter is present. In that case, nothing is done. Nothing gets injected and an exception is not thrown.

edm::ParameterSetDescription desc;
desc.addOptionalNode( edm::ParameterDescription<std::string>("x", "11", true) xor 
                      edm::ParameterDescription<unsigned int>("y", 11U, true),
                      true );

The examples in this section use two names that have not been defined yet, node and ParameterDescription. These are discussed in the following section.

ParameterDescriptionNode's and ParameterDescription's

  A ParameterDescription is the class name for objects that hold the description of a single parameter. The template argument is the type of the parameter.
Line: 502 to 523
  The base class of ParameterDescription is ParameterDescriptionNode. A single parameter is a ParameterDescriptionNode, but it is not the only kind
Changed:
<
<
of ParameterDescriptionNode. The "xor" operator returns a value of a type that also inherits from ParameterDescriptionNode (XORGroupDescription).
>
>
of ParameterDescriptionNode. The "xor" operator returns a value whose type also inherits from ParameterDescriptionNode (XORGroupDescription).
 The operands of the "xor" operator can be any type of ParameterDescriptionNode.
Changed:
<
<
This allows one to make similar statements with 3 or more parameters. For example, we could say that either "x" or "y" or "z" exists in the ParameterSet being validated, but only 1 of them.
>
>
This allows one to extend the example from the previous section to 3 or more parameters. For example, we could say that either "x" or "y" or "z" exists in the ParameterSet being validated, but only 1 of them.
 
edm::ParameterSetDescription desc;
Line: 528 to 549
 Internally, this is what the "add" function described in previous sections is doing.

There are two special cases where the constructor of the ParameterDescription

Changed:
<
<
is a little different. When the type of a single parameter is ParameterSet, then the description for that parameter is created with one of these two constructors.
>
>
is a little different. Here is how it looks in those cases.
 
edm::ParameterSetDescription desc;
Line: 541 to 559
 std::vector<edm::ParameterSet> vectorOfPSets;

desc.addNode( edm::ParameterDescription<edm::ParameterSetDescription>("aDesc", desc1, true) xor

Changed:
<
<
edm::ParameterDescription<std::vector<edm::ParameterSet>("aVectorOfPSets", desc2, true, vectorOfPSets) );

ParameterDescription(char const* iLabel,
                                  ParameterSetDescription const& value,
                                  bool isTracked
);

When the type of a parameter is vector&lt ParameterSet &gt, then the description for that parameter is created with one of these constructors.

ParameterDescription(std::string const& iLabel,
                                  ParameterSetDescription const& psetDesc,
                                  bool isTracked,
                                  std::vector<ParameterSet> const& vPset
);
 
ParameterDescription(char const* iLabel,
                                  ParameterSetDescription const& psetDesc,
                                  bool isTracked,
                                  std::vector<ParameterSet> const& vPset
); 

ParameterDescription(std::string const& iLabel,
                                   ParameterSetDescription const& psetDesc,
                                   bool isTracked
);
 
ParameterDescription(char const* iLabel,
                                   ParameterSetDescription const& psetDesc,
                                   bool isTracked
);
>
>
edm::ParameterDescription<std::vector<edm::ParameterSet> >("aVectorOfPSets1", desc2, true, vectorOfPSets) xor edm::ParameterDescription<std::vector<edm::ParameterSet> >("aVectorOfPSets2", desc2, true));
 

These behave as described in the sections above titled "!ParameterSets that Contain ParameterSets" and

Changed:
<
<
"!ParameterSets that Contain Vectors of ParameterSets"
>
>
"!ParameterSets that Contain Vectors of ParameterSets". The first argument is the parameter name. The second argument is the ParameterSetDescription used to validate the ParameterSet or all the ParameterSet's in the vector. The third argument is a boolean that is true if the parameter is a tracked parameter. In the vector case, there is allowed to be one additional argument that specifies the default.

In the example of above the ParameterSetDescription's and vector inside the expression are empty only to make the example simple. Normally, they would have contents and those contents are not constrained by the fact that they are inside the expression. Also note that the evaluation of the "xor" expression only depends on the existence of the parameters, it does not depend on their contents in any way. If it validates as true, then the one option that exists is validated and that includes includes validation of all ParameterSet's it contains.

It is of some importance that both the arguments and return type of the "xor" operator have type ParameterDescriptionNode. That means any type of node can be an operand and the expression itself can be used as an operand in other expressions. The following sections describe several other types of nodes: "and" operators, "or" operators, a switch operator, and others. The "xor" expression can be used in those expressions, for for example as a case in a switch statement. And also an "and" expression can be used as the operand in an "xor" expression. There is no limit to level of nesting allowed other than available system memory of your machine.

 

Wildcards

Revision 212010-02-02 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 18 to 18
  validation will throw if a parameter is in the configuration that is not in the description (protection against spelling errors).
Changed:
<
<
  • The plugins configuration will be accessible from the executable edmPluginHelp in a human readable format.
>
>
  • The description of the parameters that a plugins requires or allows will be accessible from the executable edmPluginHelp in a human readable format.
 
  • CFI python files will be generated automatically after giving the command "scram b". The python files are generated along with the libraries and other things created as part of the build. These python files are distributed as part
Line: 30 to 30
  applications to access the details of the description.

This validation system is designed so that it can be deployed incrementally,

Changed:
<
<
one plugin at a time. If the fillDescription function for a plugin has not been
>
>
one plugin at a time. If the "fillDescriptions" function for a plugin has not been
 defined, then validation will do nothing for that plugin.

The next two sections contain all the information needed for the majority

Line: 92 to 92
 to the plugin constructor with default values which are provided in the second argument to the "add" function above.
Changed:
<
<
During the build a CFI file would be automatically generated with this path
>
>
During the build a CFI file would be automatically generated with this path:
 "cfipython/slc5_ia32_gcc434/SubsystemName/PackageName/produceMuons_cfi.py". This file would contain the following:
Line: 120 to 120
  debugLevel untracked uint32 1
Changed:
<
<
Note that there is support for all types of parameters that are allowed in python configuration files, with the exception of the SecSource type. The interface for the types ParameterSet and vector of ParameterSets is slightly different than in the example above. This is described in one of the following sections.
>
>
Note that the interfaces for adding the types ParameterSet and vector< ParameterSets > are slightly different than in the example above. These are described in the following sections.
  One point above is worth repeating. If any parameters are missing from the python configuration, then during validation the parameters
Line: 140 to 138
 In almost all these cases, the function will accept either a "const char*" or a "std::string".
Changed:
<
<
In the examples on the TWIKI, some type like "int" is used for the parameter type to keep things simple, but the description code already supports all the
>
>
In the examples on this TWIKI, some simple type like "int" is used for the parameter type improve readability, but the description code already fully supports all the
 types listed below. The only one allowed in a python configuration that is not supported at the moment is "SecSource" (and we will probably add support for that as soon as we find the time).

Revision 202010-02-02 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 133 to 133
 plugin's code will no longer be used for anything. We recommend these other defaults be deleted because they might cause confusion.
Added:
>
>

Types Allowed for Arguments

In the examples on this TWIKI, a literal string is shown for the argument for the parameter name and some other kinds of names. In almost all these cases, the function will accept either a "const char*" or a "std::string".

In the examples on the TWIKI, some type like "int" is used for the parameter type to keep things simple, but the description code already supports all the types listed below. The only one allowed in a python configuration that is not supported at the moment is "SecSource" (and we will probably add support for that as soon as we find the time).

int32
vector<int32>
uint32
vector<uint32>
int64
vector<int64>
uint64
vector<uint64>
double
vector<double>
bool
string
vector<string>
EventID
vector<EventID>
LuminosityBlockID
vector<LuminosityBlockID>
InputTag
vector<InputTag>
FileInPath
LuminosityBlockRange
vector<LuminosityBlockRange>
EventRange
vector<EventRange>
ParameterSet
vector<ParameterSet>
 

A Plugin Which Allows No Parameters

The simplest case is a plugin that does have any configurable parameters.

Line: 421 to 463
 they are not allowed to have a module label in a python configuration.
Added:
>
>

Logical Exclusive Or Operator

Adding a parameter to a ParameterSetDescription can be viewed as a statement that the parameter is present in the ParameterSet being validated. If the parameter is present, then the statement is true. If the parameter is not present then the statement is false. When we view these as statements that are either true or false, we can then use logical operators to create more complicated statements. For example, we could say that either parameter "x" is present or parameter "y" is present but not both.

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true) xor 
              edm::ParameterDescription<unsigned int>("y", 11U, true) );

When validating a ParameterSet using this description, an exception will be thrown if both "x" and "y" exist in the ParameterSet. If only one exists, then validation will pass and this xor expression will evaluate as true. If neither "x" nor "y" exists in the configuration, then the validation algorithm will try to insert the first parameter in the expression into the ParameterSet. If successful then the expression will still evaluate as true and validation will pass. Otherwise an exception will be thrown.

The operator "xor" in the above example is the C++ operator "^" and the "^" can be used in place of "xor" above. It behaves with the precedence and of that operator.

A ParameterDescription is the class name for objects that hold the description of a single parameter. The template argument is the type of the parameter. The first function argument is the name of the parameter. The second argument is the default value of the parameter. The third argument specifies whether the parameter is tracked. It also possible to use a different constructor that has only two arguments by dropping the second argument that gives a default value. If this is done, the parameter cannot be inserted if missing or written into CFI files.

The base class of ParameterDescription is ParameterDescriptionNode. A single parameter is a ParameterDescriptionNode, but it is not the only kind of ParameterDescriptionNode. The "xor" operator returns a value of a type that also inherits from ParameterDescriptionNode (XORGroupDescription). The operands of the "xor" operator can be any type of ParameterDescriptionNode. This allows one to make similar statements with 3 or more parameters. For example, we could say that either "x" or "y" or "z" exists in the ParameterSet being validated, but only 1 of them.

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true) xor 
                         edm::ParameterDescription<unsigned int>("y", 11U, true) xor
                         edm::ParameterDescription<double>("z", 11.0, true ));

Also note that the function that adds this expression to the ParameterSetDescription is called "addNode". It can take as an argument any type of ParameterDescriptionNode. This will work also:

edm::ParameterSetDescription desc;
desc.addNode( edm::ParameterDescription<std::string>("x", "11", true));

Internally, this is what the "add" function described in previous sections is doing.

There are two special cases where the constructor of the ParameterDescription is a little different. When the type of a single parameter is ParameterSet, then the description for that parameter is created with one of these two constructors.

edm::ParameterSetDescription desc;

edm::ParameterSetDescription desc1;
edm::ParameterSetDescription desc2;
std::vector<edm::ParameterSet> vectorOfPSets;

desc.addNode( edm::ParameterDescription<edm::ParameterSetDescription>("aDesc", desc1, true) xor 
              edm::ParameterDescription<std::vector<edm::ParameterSet>("aVectorOfPSets", desc2, true, vectorOfPSets) );

ParameterDescription(char const* iLabel,
                                  ParameterSetDescription const& value,
                                  bool isTracked
);

When the type of a parameter is vector&lt ParameterSet &gt, then the description for that parameter is created with one of these constructors.

ParameterDescription(std::string const& iLabel,
                                  ParameterSetDescription const& psetDesc,
                                  bool isTracked,
                                  std::vector<ParameterSet> const& vPset
);
 
ParameterDescription(char const* iLabel,
                                  ParameterSetDescription const& psetDesc,
                                  bool isTracked,
                                  std::vector<ParameterSet> const& vPset
); 

ParameterDescription(std::string const& iLabel,
                                   ParameterSetDescription const& psetDesc,
                                   bool isTracked
);
 
ParameterDescription(char const* iLabel,
                                   ParameterSetDescription const& psetDesc,
                                   bool isTracked
);

These behave as described in the sections above titled "!ParameterSets that Contain ParameterSets" and "!ParameterSets that Contain Vectors of ParameterSets"

 

Wildcards

Revision 192010-02-02 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 421 to 421
 they are not allowed to have a module label in a python configuration.
Added:
>
>

Wildcards


 edm::ParameterSetDescription wildcardPset;

  wildcardPset.addWildcard<int>("*");
  wildcardPset.addWildcardUntracked<double>(std::string("*");

  std::auto_ptr<edm::ParameterDescriptionNode> wnode(new edm::ParameterWildcard<edm::ParameterSetDescription>("*", edm::RequireExactlyOne, true));
  wildcardPset.addOptionalNode(wnode, false);

  edm::ParameterSetDescription wSet1;
  wSet1.add<unsigned int>("Drinks", 5);

  std::auto_ptr<edm::ParameterDescriptionNode> wnode2(new edm::ParameterWildcard<edm::ParameterSetDescription>("*", edm::RequireAtLeastOne, true, wSet1));
  wildcardPset.addOptionalNode(wnode2, false);

  std::auto_ptr<edm::ParameterDescriptionNode> wnode3(new edm::ParameterWildcard<std::vector<edm::ParameterSet> >("*", edm::RequireExactlyOne, true));
  wildcardPset.addOptionalNode(wnode3, false);

  wSet1.add<unsigned int>("Drinks2", 11);

  std::auto_ptr<edm::ParameterDescriptionNode> wnode4(new edm::ParameterWildcard<std::vector<edm::ParameterSet> >("*", edm::RequireAtLeastOne, true, wSet1));
  wildcardPset.addOptionalNode(wnode4, false);

  iDesc.add("wildcardPset", wildcardPset);
 

Other Things

Revision 182010-02-01 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 20 to 20
  against spelling errors).
  • The plugins configuration will be accessible from the executable edmPluginHelp in a human readable format.
Changed:
<
<
  • CFI python files will be generated based on the information in the description. In addition to running automatically this can also be run standalone with the edmWriteConfig executable.
>
>
  • CFI python files will be generated automatically after giving the command "scram b". The python files are generated along with the libraries and other things created as part of the build. These python files are distributed as part of standard releases and will be built in local areas if a package is checked out. Also you can generate the python files manually with the edmWriteConfigs executable.
 
  • This part is not implemented yet, but we imagine a C++ interface that allows other applications to access the details of the description.
Line: 333 to 336
 CFI generation and the only affect on edmPluginHelp is a couple additional lines that indicate the function was called.
Added:
>
>

Multiple Descriptions for One Plugin

It is possible to add multiple ParameterSetDescriptions for one plugin if it is an EDProducer, EDFilter, EDAnalyzer, or OutputModule.

#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"

void MuonProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {

   edm::ParameterSetDescription desc;
   desc.add<int>("algorithm", 1);
   descriptions.add("produceMuons", desc);

   edm::ParameterSetDescription desc1;
   desc1.add<int>("debugLevel", 7);
   descriptions.add("debugProduceMuons", desc1);

   edm::ParameterSetDescription desc2;
   desc2.add<double>("ptCut", 100.0);
   descriptions.addDefault(desc2);
}

Each description must be associated with a unique module label that is passed as an argument to the add function. Other than that the descriptions are completely independent. In addition, a default description can be added with the addDefault function. There can only be one default description per plugin.

When validating a plugin's configuration, a description is selected using the following steps.

  • If there is a description whose label exactly matches the label in the configuration, then that description is used for validation.
  • If there is not an exact match and there is a default description, then the default is used.
  • If there is no match and no default, the first description added in the fillDescriptions function is used.
  • If there are not any descriptions, then silently no validation is performed.

A CFI file will be generated for each description that was added with a module label. The module label is used in both the file name and the content of the file. A CFI file is not generated for a default description.

Sources and Services

Defining a ParameterSetDescription for a service or a source is almost the same as defining one for a plugin that is a module. There are a few minor differences. First of all, sources and services can only have one description associated with them. It can be one description with a name or one default description, but not both. If a name is associated with the description, then a CFI file will be generated. A CFI file will not be generated if it is a default description.

For sources the name passed to the add function must always be "source". Here is an example of how it is done for the PoolSource.

void
PoolSource::fillDescriptions(ConfigurationDescriptions & descriptions) {
  edm::ParameterSetDescription desc;

  ... // add parameters as usual
 
  descriptions.add("source", desc);
}

For services the name passed to the add function must always be the name of the plugin. Here is an example of how it is done for the Tracer plugin.

void
Tracer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) { 
  edm::ParameterSetDescription desc;
  desc.addUntracked<std::string>("indention", "++");
  descriptions.add("Tracer", desc);
}

You cannot supply a module label for a source or service because they are not allowed to have a module label in a python configuration.

 

Other Things

Line: 345 to 433
 Here are the topics that still need to be explained:
Deleted:
<
<
Multiple module labels and defaults sources and services
 wildcards AND OR

Revision 172010-01-30 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 30 to 30
 one plugin at a time. If the fillDescription function for a plugin has not been defined, then validation will do nothing for that plugin.
Added:
>
>
The next two sections contain all the information needed for the majority of CMS plugins. Some readers may want to stop there and only use the rest of this TWIKI page as a reference when needed.
 

Describing the configuration in C++

You can describe the configuration parameters a plugin can use by

Line: 248 to 252
 and the default value should be deleted from the getUntrackedParameter function call.
Added:
>
>

Parameters Without Default Values

You can add a parameter without a default value as follows:

edm::ParameterSetDescription desc;
desc.add<int>("x");

The only difference between this and the case where a default is provided is eliminating the argument that provides the default. This works in every case described in the sections above, with one exception. When one is describing a ParameterSet nested inside a ParameterSet, one must always provide two arguments.

If no default is provided, then there are two consequences:

  • During validation, if a required parameter is missing and there is no default then an exception is thrown. If a default was provided, then it is inserted.
  • For both required and optional parameters, if no default is provided, then the parameter is not written into the generated CFI file.

Adding Comments

You can add comments to the description. These comments show up in the output of the edmPluginHelp utility. They do not affect validation or the CFI files. Comments are not necessary and not required. Here is an example:

     ParameterSetDescription desc;
     desc.addUntracked<bool>("verbose", false)->
     setComment("If true, the contents of products are printed using Reflex.");

And this is part of the output from edmPluginHelp related to the above code:

% edmPluginHelp -p EventContentAnalyzer
...
     verbose
                        type: untracked bool
                        default: False
                        If true, the contents of products are printed using Reflex.
...

Note that the addUntracked function above returns a pointer to a ParameterDescriptionNode. This is the base class for the class that hold the description of a single parameter. But it is also a base class for other things like the logical expressions which are the subject of the following sections. It is possible to call setComment on all of those objects. In addition, the classes ParameterSetDescription and ConfigurationDescriptions also have a setComment function. All of these comments will show up in different places in the output of edmPluginHelp.

There is a lot of flexibility in how these comments can be used. Our current thought is that it is best if they are kept brief, saving long explanations for TWIKI pages or other document formats. Also remember that correcting an error in documentation that is hard coded will not be fast. The correction will not reach users until the next software release is distributed, which could be a long time.

Let the ParameterSetDescription Allow Anything

It is possible to tell a ParameterSetDescription to allow anything.

  edm::ParameterSetDescription desc;
  desc.setAllowAnything();

When this function has been called no validation occurs for the corresponding ParameterSet in the configuration. This means no exceptions will be thrown and missing parameters will not be injected. I'm not sure if it would be useful, but one could add parameters even with this set. It does not affect the output of CFI generation and the only affect on edmPluginHelp is a couple additional lines that indicate the function was called.

 

Other Things

Line: 262 to 347
 
Multiple module labels and defaults
sources and services
Deleted:
<
<
no default value parameters comments
 wildcards AND OR

Revision 162010-01-29 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 218 to 218
 default value 7. The other element of the default vector already contains "x" so it will pass through validation unmodified.
Added:
>
>

Optional Parameters

All the parameters described in the sections above are required parameters. In addition, there are functions analogous to the functions shown above that can be used to add optional parameters to a ParameterSetDescription. They are:

addOptional
addOptionalUntracked
addVPSetOptional
addVPSetOptionalUntracked

These are used in the same way as the functions above. There is only one difference in behavior when these are used. During validation if a required parameter is missing, then it is either inserted with its default value or an exception is thrown. During validation if an optional parameter is missing, then it is just ignored. Nothing is inserted and nothing is thrown. There are no other differences. Note that both optional and required parameters are written into CFI files in exactly the same manner.

It is the intent of the designers that optional parameters only be used in the case where the behavior of a plugin is different depending on whether a parameter exists in the ParameterSet used to configure the plugin. It is not intended to be used in places where defaults were provided in the code as a second argument to a getUntrackedParameter call. Those should be added to the description as required parameters and the default value should be deleted from the getUntrackedParameter function call.

 

Other Things

Line: 233 to 263
 Multiple module labels and defaults sources and services no default value parameters
Deleted:
<
<
optional parameters
 comments wildcards AND

Revision 152010-01-29 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 175 to 175
 the configuration. If a required nested ParameterSet is missing, first the validation will insert an empty ParameterSet and then use the description to validate that.
Added:
>
>

ParameterSets that Contain Vectors of ParameterSets

A ParameterSet can contain a vector of ParameterSet's. This is implemented in the configuration by making the type of a parameter be vector< ParameterSet >. One adds one of these parameters to the description using a function named "addVPSet" instead of "add". Here is an example.

#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"
#include "FWCore/ParameterSet/interface/ParameterSet.h"
#include <vector>
...
edm::ParameterSetDescription desc;

edm::ParameterSetDescription validator;
validator.add<int>("x", 7);

std::vector<edm::ParameterSet> vDefaults;
edm::ParameterSet vDefaults0;
vDefaults.push_back(vDefaults0);
edm::ParameterSet vDefaults1;
vDefaults1.addParameter<int>("x", 100);
vDefaults.push_back(vDefaults1);

desc.addVPSet("nameForVPSet", validator, vDefaults);

The second argument to addVPSet is a ParameterSetDescription. This description is used to validate all ParameterSet's that are in the vector being validated. The third argument is a default that is used if the vector is missing and also when writing the CFI file. Note that validation will be run on each element of the default vector if it is ever used, so the second and third arguments must be consistent. In the example above, the 0th element of the default vector is an empty ParameterSet and validation will insert the parameter named "x" with the default value 7. The other element of the default vector already contains "x" so it will pass through validation unmodified.

 

Other Things

Line: 187 to 230
 Here are the topics that still need to be explained:
Deleted:
<
<
vector
 Multiple module labels and defaults sources and services no default value parameters

Revision 142010-01-29 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 143 to 143
 }
Added:
>
>

ParameterSets that Contain ParameterSets

A ParameterSet can contain other ParameterSet's. This is implemented in the configuration by making the type of a parameter be ParameterSet. One adds one of these parameters to the description in almost the same way as other types of parameters, but there are a few subtle differences. Here is an example:

edm::ParameterSetDescription desc;
 
edm::ParameterSetDescription descNested;
descNested.add<unsigned int>("x", 5);

desc.add<edm::ParameterSetDescription>("nameForDescNested", descNested);

The template parameter of the add function is ParameterSetDescription. It is not ParameterSet. The second argument of the add function is also of type ParameterSetDescription. It describes the nested ParameterSet. This description may include the defaults for the parameters it contains, but it is more than just a default value. This second argument must always be present when adding a parameter of type ParameterSet to the description. For other types of parameters, it is possible to add them without defaults (described below in another section). This is not possible for parameters of type ParameterSet.

Validation will recursively descend into nested ParameterSet's using the provided ParameterSetDescription to validate the contents of the nested ParameterSet from the configuration. If a required nested ParameterSet is missing, first the validation will insert an empty ParameterSet and then use the description to validate that.

 

Other Things

Line: 155 to 187
 Here are the topics that still need to be explained:
Deleted:
<
<
Nested ParameterSet
 vector Multiple module labels and defaults sources and services
Line: 292 to 323
 
  • Implement the fillDescriptions function in many plugins. Currently very few plugins have this function.
  • Continue to improve this documentation
Added:
>
>
  • Add the ability for the validation to check if the value of a parameter is in a certain range, or is one of a limited number of allowed values.
  • In the case of parameters that are vectors, the validation might also check that the vector is of a certain size.
 

An example

Revision 132010-01-26 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 9 to 9
 Each plugin can describe the parameters it allows or requires to be in its configuration. This description will appear in a function named fillDescriptions in the plugin's C++ code. When this function has been implemented, several
Changed:
<
<
things will be possible.
>
>
things will happen automatically.
 
  • cmsRun will validate the configuration for each plugin at the beginning of a job. If the validation determines that a parameter required by the description is missing,
Line: 18 to 18
  validation will throw if a parameter is in the configuration that is not in the description (protection against spelling errors).
Changed:
<
<
  • The executable edmPluginHelp will printout human readable documentation about the configuration of a plugin.
  • CFI python files can be automatically generated based on the information in the description. This is now run automatically as part of the build and can also be run
>
>
  • The plugins configuration will be accessible from the executable edmPluginHelp in a human readable format.
  • CFI python files will be generated based on the information in the description. In addition to running automatically this can also be run
  standalone with the edmWriteConfig executable.
Changed:
<
<
  • This part is not implemented yet, but we imagine a C++ API that allows other
>
>
  • This part is not implemented yet, but we imagine a C++ interface that allows other
  applications to access the details of the description.

This validation system is designed so that it can be deployed incrementally, one plugin at a time. If the fillDescription function for a plugin has not been

Changed:
<
<
defined, then the validation will do nothing for that plugin.
>
>
defined, then validation will do nothing for that plugin.
 

Describing the configuration in C++

You can describe the configuration parameters a plugin can use by

Changed:
<
<
adding a single function to the C++ code of the plugin. If this function is defined, then the parameters passed to the plugin in a cmsRun process will be validated automatically, CFI files will be generated automatically, and documentation will be available through the edmPluginHelp utility. The declaration of this function in the header file of the class will always look like this:
>
>
adding a single function to the C++ code of the plugin. The declaration of this function in the header file of the class will always look like this:
 
namespace edm {
Line: 126 to 122
 One point above is worth repeating. If any parameters are missing from the python configuration, then during validation the parameters will be inserted into the ParameterSet before it is passed to the plugin
Changed:
<
<
constructor. This means that any defaults that reside other parts of the
>
>
constructor. This means that any defaults that reside in other parts of the
 plugin's code will no longer be used for anything. We recommend these other defaults be deleted because they might cause confusion.
Line: 147 to 143
 }
Added:
>
>

Other Things

  WARNING
Changed:
<
<
The rest of this section is being actively edited at the moment. It may contain things that make no sense and are wrong and unreadable.
>
>
The rest of this section is still being written. It is certainly incomplete and may be in a state of change as I write new things.
 
Changed:
<
<
Here what I've still got left to describe
>
>
Here are the topics that still need to be explained:
 
Nested ParameterSet

Revision 122010-01-26 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 30 to 30
 one plugin at a time. If the fillDescription function for a plugin has not been defined, then the validation will do nothing for that plugin.
Added:
>
>

Describing the configuration in C++

You can describe the configuration parameters a plugin can use by adding a single function to the C++ code of the plugin. If this function is defined, then the parameters passed to the plugin in a cmsRun process will be validated automatically, CFI files will be generated automatically, and documentation will be available through the edmPluginHelp utility. The declaration of this function in the header file of the class will always look like this:

namespace edm {
   class ConfigurationDescriptions;
}
class MuonProducer : public edm::EDProducer {
...
public:
   static void fillDescriptions(edm::ConfigurationDescriptions & descriptions);
...
};

The class name MuonProducer is just used as an example. It needs to be replaced with the C++ class name for the plugin. The declaration of the function is the same for EDFilter's, EDAnalyzer's, OutputModule's, InputSource's and Services.

An Example

Let's assume this hypothetical MuonProducer module could use 3 parameters. The first parameter is named "algorithm", has type int, and is tracked. The second parameter is named "ptCut", has type double, and is tracked. The third parameter is named "debugLevel", has type unsigned int, and is untracked. In this case, the fillDescriptions function should be defined as follows:

#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"

void MuonProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
   edm::ParameterSetDescription desc;
   desc.add<int>("algorithm", 1);
   desc.add<double>("ptCut", 20.0);
   desc.addUntracked<unsigned int>("debugLevel", 1U);
   descriptions.add("produceMuons", desc);
}

During validation, an exception would be thrown and the process would stop if any parameters other than "algorithm", "ptCut", or "debugLevel" were in the configuration. This protects the user against spelling errors and misuse of parameters in the configuration. An exception would also be thrown if the parameter had the wrong type or was tracked when it should have been untracked or vice versa. If any of the three parameters above were missing in the configuration, then the validation would insert them into the ParameterSet that is passed to the plugin constructor with default values which are provided in the second argument to the "add" function above.

During the build a CFI file would be automatically generated with this path "cfipython/slc5_ia32_gcc434/SubsystemName/PackageName/produceMuons_cfi.py". This file would contain the following:

import FWCore.ParameterSet.Config as cms

produceMuons = cms.EDProducer('MuonProducer',
  algorithm = cms.int32(1),
  ptCut = cms.double(20),
  debugLevel = cms.untracked.uint32(1)
)

If you gave the following command at the UNIX prompt in a CMSSW working release area, you would get the following output.

% edmPluginHelp -b -p MuonProducer
1  MuonProducer  pluginTestMuonProducer.so
  1.1 produceMuons
    algorithm  int32             1
    ptCut      double            20
    debugLevel untracked uint32  1

Note that there is support for all types of parameters that are allowed in python configuration files, with the exception of the SecSource type. The interface for the types ParameterSet and vector of ParameterSets is slightly different than in the example above. This is described in one of the following sections.

One point above is worth repeating. If any parameters are missing from the python configuration, then during validation the parameters will be inserted into the ParameterSet before it is passed to the plugin constructor. This means that any defaults that reside other parts of the plugin's code will no longer be used for anything. We recommend these other defaults be deleted because they might cause confusion.

A Plugin Which Allows No Parameters

The simplest case is a plugin that does have any configurable parameters. The fillDescriptions function is implemented as shown below. In this case, the validation will throw an exception if the user tries to include any parameters in the python configuration for this plugin.

#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"

void MuonProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
   edm::ParameterSetDescription desc;
   descriptions.add("aModuleLabel", desc);
}

WARNING The rest of this section is being actively edited at the moment. It may contain things that make no sense and are wrong and unreadable.

Here what I've still got left to describe

Nested ParameterSet
vector<ParameterSet>
Multiple module labels and defaults
sources and services
no default value parameters
optional parameters
comments
wildcards
AND
OR
XOR
ifValue
ifExists
allowedLabels
 

edmPluginHelp

The executable named edmPluginHelp will print out the configuration

Line: 119 to 261
  is matched against the parameter label. The algorithm only searches for matching parameters of a single specified type.
Deleted:
<
<

Validation

Describing the configuration in C++

WARNING This section is being actively edited at the moment. It may contain things that make no sense and are wrong and unreadable.

You can describe the configuration parameters a plugin can use by adding a single function to the C++ code of the plugin. If this function is defined, then the parameters passed to the plugin in a cmsRun process will be validated automatically, CFI files will be generated automatically, and documentation will be available through the edmPluginHelp utility. The declaration of this function in the header file of the class will always look like this:

namespace edm {
   class ConfigurationDescriptions;
}
class MuonProducer : public edm::EDProducer {
...
public:
   static void fillDescriptions(edm::ConfigurationDescriptions & descriptions);
...
};

The class name MuonProducer is just used as an example. It needs to be replaced with the C++ class name for the plugin. The declaration of the function is the same for EDFilter's, EDAnalyzer's, OutputModule's, InputSource's and Services.

A Simple Example

Let's assume this hypothetical MuonProducer module could use 3 parameters. The first parameter is named "algorithm" which has type int, the second parameter is named "ptCut" and has type double, and the third parameter is named "debugLevel" and has type unsigned int. In this case, the fillDescriptions function should be defined as follows:

#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"

void MuonProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
   edm::ParameterSetDescription desc;
   desc.add<int>("algorithm", 1);
   desc.add<double>("ptCut", 20.0);
   desc.addUntracked<unsigned int>("debugLevel", 1U);
   descriptions.add("produceMuons", desc);
}

During validation, an exception would be thrown and the process would stop if any parameters other than "algorithm", "ptCut", or "debugLevel" were in the configuration. This protects the user against spelling errors and misuse of parameters in the configuration. If any of the three parameters above were missing in the configuration, then the validation would insert them into the ParameterSet that is passed to the module with default values which are provided in the second argument to the "add" function.

During the build a CFI file would be automatically generated with this path cfipython/slc5_ia32_gcc434/SubsystemName/PackageName/produceMuons_cfi.py. This file would contain the following:

import FWCore.ParameterSet.Config as cms

produceMuons = cms.EDProducer('MuonProducer',
  algorithm = cms.int32(1),
  ptCut = cms.double(20),
  debugLevel = cms.untracked.uint32(1)
)

A Plugin Which Allows No Parameters

The simplest case is a plugin that does have any configurable parameters. The fillDescriptions function is implemented as shown below. In this case the validation will throw an exception if the user tries to include any parameters in the python configuration. A file named aModuleLabel_cfi.py will be generated during the build that contains no parameters. One will be able to use edmPluginHelp to see that the description is empty and no parameters are allowed.

#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"

void MuonProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
   edm::ParameterSetDescription desc;
   descriptions.add("aModuleLabel", desc);
}

A Plugin with One Tracked and One Untracked

There is not much documentation about writing the C++ fillDescriptions functions yet. This section needs to be improved and expanded. The original proposal that Chris Jones wrote is here:

https://twiki.cern.ch/twiki/bin/view/CMS/FwEdmParameterSetValidationProposal

This is out of date, but gives a flavor for how one creates a description. The following two functions can be looked up in the code in the CVS head. The first is a very simple example. We expect the configuration for most modules to be this simple. The second tests all the possible features of a description and is much more complicated than we expect any real description to be.

  • EventContentAnalyzer::fillDescriptions
  • ProducerWithPSetDesc::fillDescriptions
 

Features that still need to implemented

We are discussing whether and when to implement these features.

Revision 112010-01-25 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 128 to 128
 things that make no sense and are wrong and unreadable.
Changed:
<
<
The description of the configuration of a plugin is defined by adding a single function to the C++ code of the plugin. If this function is defined, then validation will occur automatically, CFI files will be generated automatically, and documentation will be available through the edmPluginHelp utility. The declaration of this new function in the header file of the class will always look like this:
>
>
You can describe the configuration parameters a plugin can use by adding a single function to the C++ code of the plugin. If this function is defined, then the parameters passed to the plugin in a cmsRun process will be validated automatically, CFI files will be generated automatically, and documentation will be available through the edmPluginHelp utility. The declaration of this function in the header file of the class will always look like this:
 
namespace edm {
   class ConfigurationDescriptions;
}
Changed:
<
<
class MuonProducer {
>
>
class MuonProducer : public edm::EDProducer {
 ... public: static void fillDescriptions(edm::ConfigurationDescriptions & descriptions);
Line: 150 to 150
  The class name MuonProducer is just used as an example. It needs to be replaced with the C++ class name for the plugin.
Added:
>
>
The declaration of the function is the same for EDFilter's, EDAnalyzer's, OutputModule's, InputSource's and Services.
 

A Simple Example

Added:
>
>
Let's assume this hypothetical MuonProducer module could use 3 parameters. The first parameter is named "algorithm" which has type int, the second parameter is named "ptCut" and has type double, and the third parameter is named "debugLevel" and has type unsigned int. In this case, the fillDescriptions function should be defined as follows:
 
#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"
Changed:
<
<
void ThingProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
>
>
void MuonProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
  edm::ParameterSetDescription desc; desc.add("algorithm", 1); desc.add("ptCut", 20.0);
Changed:
<
<
desc.addUntracked("debugLevel", 1U); descriptions.add("aModuleLabel", desc);
>
>
desc.addUntracked("debugLevel", 1U); descriptions.add("produceMuons", desc);
 }
Added:
>
>
During validation, an exception would be thrown and the process would stop if any parameters other than "algorithm", "ptCut", or "debugLevel" were in the configuration. This protects the user against spelling errors and misuse of parameters in the configuration. If any of the three parameters above were missing in the configuration, then the validation would insert them into the ParameterSet that is passed to the module with default values which are provided in the second argument to the "add" function.

During the build a CFI file would be automatically generated with this path cfipython/slc5_ia32_gcc434/SubsystemName/PackageName/produceMuons_cfi.py. This file would contain the following:

import FWCore.ParameterSet.Config as cms

produceMuons = cms.EDProducer('MuonProducer',
  algorithm = cms.int32(1),
  ptCut = cms.double(20),
  debugLevel = cms.untracked.uint32(1)
)

 

A Plugin Which Allows No Parameters

The simplest case is a plugin that does have any configurable parameters.

Line: 179 to 211
 #include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h" #include "FWCore/ParameterSet/interface/ParameterSetDescription.h"
Changed:
<
<
void XYZ::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
>
>
void MuonProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
  edm::ParameterSetDescription desc; descriptions.add("aModuleLabel", desc); }

Revision 102010-01-23 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 140 to 140
 namespace edm { class ConfigurationDescriptions; }
Changed:
<
<
class XYZ {
>
>
class MuonProducer {
 ... public: static void fillDescriptions(edm::ConfigurationDescriptions & descriptions);
Line: 148 to 148
 };
Changed:
<
<
For the vast majority of plugins the configuration is simple. There is a list of a limited number of parameters that can be assigned values. These parameters have types, may be tracked or untracked, and may have defaults. The first section below describes how to implement the description in this simple case. This is all most people will need to know and I would suggest most readers stop there. But the descriptions can be much more complex and the subsequent sections describe the more complicated features available.
>
>
The class name MuonProducer is just used as an example. It needs to be replaced with the C++ class name for the plugin.
 
Changed:
<
<

Describing the Configuration in C++ for Simple Cases

>
>

A Simple Example

 
Added:
>
>
#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"

void ThingProducer::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
   edm::ParameterSetDescription desc;
   desc.add<int>("algorithm", 1);
   desc.add<double>("ptCut", 20.0);
   desc.addUntracked<int>("debugLevel", 1U);
   descriptions.add("aModuleLabel", desc);
}
 
Added:
>
>

A Plugin Which Allows No Parameters

 
Added:
>
>
The simplest case is a plugin that does have any configurable parameters. The fillDescriptions function is implemented as shown below. In this case the validation will throw an exception if the user tries to include any parameters in the python configuration. A file named aModuleLabel_cfi.py will be generated during the build that contains no parameters. One will be able to use edmPluginHelp to see that the description is empty and no parameters are allowed.
 
Added:
>
>
#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"

void XYZ::fillDescriptions(edm::ConfigurationDescriptions & descriptions) {
   edm::ParameterSetDescription desc;
   descriptions.add("aModuleLabel", desc);
}
 
Added:
>
>

A Plugin with One Tracked and One Untracked

  There is not much documentation about writing the C++ fillDescriptions functions yet. This section needs to be improved and expanded. The

Revision 92010-01-22 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 130 to 130
 

The description of the configuration of a plugin is defined by adding a

Changed:
<
<
single function to the C++ code of the plugin. Once this function is defined validation will occur automatically. CFI files will be generated automatically and documentation will be available through the edmPluginHelp utility. For the vast majority of plugins the configuration is simple. There is a list of a limited number of parameters that can be assigned values. These parameters have types, may be tracked or untracked, and may have defaults. The first section below describes how to implement the description in this simple case. This is all most people will need to know and I would suggest most readers stop there. But the descriptions can be much more complex and the following sections describe the more complicated features available.
>
>
single function to the C++ code of the plugin. If this function is defined, then validation will occur automatically, CFI files will be generated automatically, and documentation will be available through the edmPluginHelp utility. The declaration of this new function in the header file of the class will always look like this:
 
namespace edm {
Line: 154 to 148
 };
Added:
>
>
For the vast majority of plugins the configuration is simple. There is a list of a limited number of parameters that can be assigned values. These parameters have types, may be tracked or untracked, and may have defaults. The first section below describes how to implement the description in this simple case. This is all most people will need to know and I would suggest most readers stop there. But the descriptions can be much more complex and the subsequent sections describe the more complicated features available.
 

Describing the Configuration in C++ for Simple Cases

Line: 193 to 196
  and OutputModules). It also supports sources starting in release 3_5_0_pre3. Support for services has been checked into CVS, but is not in any release yet. We may add support
Changed:
<
<
ESSources, and ESProducers.
>
>
ESSources and ESProducers.
 
  • Defaults for wildcards.
  • Placeholders in the python to allow replacing parameter values in cfg files which do not exist in the CFI files.

Revision 82010-01-21 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 122 to 122
 

Validation

Describing the configuration in C++

Added:
>
>
WARNING This section is being actively edited at the moment. It may contain things that make no sense and are wrong and unreadable.

The description of the configuration of a plugin is defined by adding a single function to the C++ code of the plugin. Once this function is defined validation will occur automatically. CFI files will be generated automatically and documentation will be available through the edmPluginHelp utility. For the vast majority of plugins the configuration is simple. There is a list of a limited number of parameters that can be assigned values. These parameters have types, may be tracked or untracked, and may have defaults. The first section below describes how to implement the description in this simple case. This is all most people will need to know and I would suggest most readers stop there. But the descriptions can be much more complex and the following sections describe the more complicated features available.

namespace edm {
   class ConfigurationDescriptions;
}
class XYZ {
...
public:
      static void fillDescriptions(edm::ConfigurationDescriptions & descriptions);
...
};

Describing the Configuration in C++ for Simple Cases

  There is not much documentation about writing the C++ fillDescriptions functions yet. This section needs to be improved and expanded. The

Revision 72010-01-19 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 21 to 21
 
  • The executable edmPluginHelp will printout human readable documentation about the configuration of a plugin.
  • CFI python files can be automatically generated based on the information in the
Changed:
<
<
description. This will be part of the build and can also be run standalone with the edmWriteConfig executable.
>
>
description. This is now run automatically as part of the build and can also be run standalone with the edmWriteConfig executable.
 
  • This part is not implemented yet, but we imagine a C++ API that allows other applications to access the details of the description.
Line: 140 to 140
 

Features that still need to implemented

Deleted:
<
<
We plan to implement these features soon.

  • Modify SCRAM to automatically create CFI files during the build
 We are discussing whether and when to implement these features.

  • A C++ interface to allow other applications to access
Line: 155 to 151
  purposes only, but when you do this the cfi files and edmPluginHelp output just print out as if the parameter were a ParameterSet and not the python SecSource type. This is incorrect behavior.
Changed:
<
<
  • Currently this only supports modules (Producers, Analyzers, Filters, and OutputModules). We may add support for services, sources, ESSources, and ESProducers.
>
>
  • Currently this supports modules (Producers, Analyzers, Filters, and OutputModules). It also supports sources starting in release 3_5_0_pre3. Support for services has been checked into CVS, but is not in any release yet. We may add support ESSources, and ESProducers.
 
  • Defaults for wildcards.
  • Placeholders in the python to allow replacing parameter values in cfg files which do not exist in the CFI files.

Revision 62009-12-22 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 150 to 150
  the description. For example, ConfDB might be populated this way. The description is already capable of storing the information. The main issue is defining the interface and its requirements.
Added:
>
>
  • Add support for the python parameter type SecSource. Currently one can treat this type as if it were a nested ParameterSet for validation purposes only, but when you do this the cfi files and edmPluginHelp output just print out as if the parameter were a ParameterSet and not the python SecSource type. This is incorrect behavior.
 
  • Currently this only supports modules (Producers, Analyzers, Filters, and OutputModules). We may add support for services, sources, ESSources, and ESProducers.

Revision 52009-05-19 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"

Configuration Validation and Help

Line: 140 to 140
 

Features that still need to implemented

Changed:
<
<
We plan to implement these features.
>
>
We plan to implement these features soon.
 
Deleted:
<
<
  • Descriptions should support required parameters without defaults
 
  • Modify SCRAM to automatically create CFI files during the build
Changed:
<
<
  • Implement the fillDescriptions function in many plugins. Currently very few plugins have this function.
  • Continue to improve this documentation
>
>
 We are discussing whether and when to implement these features.

  • A C++ interface to allow other applications to access
Line: 167 to 164
 
  • Implement a reset function for the value of a single parameter description that makes it easier to define multiple descriptions for the same plugin
  • Improve error checking and messages
Added:
>
>
  • Implement the fillDescriptions function in many plugins. Currently very few plugins have this function.
  • Continue to improve this documentation
 

An example

Revision 42009-05-18 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"
Added:
>
>

Configuration Validation and Help

 
Changed:
<
<
>
>
 
Changed:
<
<

Configuration Validation and Help

>
>

Introduction

  Each plugin can describe the parameters it allows or requires to be in its configuration. This description will appear in a function named fillDescriptions
Line: 179 to 180
 It will fail because the configuration has a parameter not in the description of the allowed parameters.
Deleted:
<
<
-- DavidDagenhart - 14 May 2009
 \ No newline at end of file
Added:
>
>

Review status

<!-- Add your review status in this table structure with 2 columns delineated by three vertical bars -->

Reviewer/Editor and Date (copy from screen) Comments
DavidDagenhart - 18 May 2009 created page

<!-- In the following line, be sure to put a blank space AFTER your name; otherwise the Summary doesn't come out right. -->

Responsible: DavidDagenhart
Last reviewed by:

 \ No newline at end of file

Revision 32009-05-15 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"
Deleted:
<
<

This page and the system it describes are only partially functional and still under active development.

This is in such a preliminary form that I do NOT recommend reading it yet! At the moment it is bits and pieces cut and pasted from other places that I am in the middle of editing.

 
Line: 33 to 26
  applications to access the details of the description.

This validation system is designed so that it can be deployed incrementally,

Changed:
<
<
one module at a time. If the fillDescription function for a plugin has not been
>
>
one plugin at a time. If the fillDescription function for a plugin has not been
 defined, then the validation will do nothing for that plugin.

edmPluginHelp

The executable named edmPluginHelp will print out the configuration

Changed:
<
<
descriptions in a human readable format. As I am writing this, this executable is not in any release yet (3_1_0_pre6 is the latest at the moment), but should appear soon. It is in CVS, tagged, and in the queue for 3_1_X.
>
>
descriptions in a human readable format. This executable appeared for the first time in release CMSSW_3_1_0_pre7.
  For the most part, the output of edmPluginHelp is self explanatory. The command "edmPluginHelp -h" explains the available options and gives a little information about the
Changed:
<
<
format. Here are some more details in no particular order.

  1. Some people get confused and think the output is a particular configuration. It is not. It describes the parameters that can be in a configuration.
>
>
format. Here are some more details.
 
Changed:
<
<
  1. There is a long format and a brief format. For simple parameters,
>
>
  1. The output of edmPluginHelp includes comments that describe the output format. In addition to these generic comments, it is also possible for a plugin to add comments into its description that will appear in the edmPluginHelp output. These are optional, but could be used to describe the plugin and its parameters in great detail.
  2. One option to the command will cause it to produce a brief format. This suppresses most comments and produces a more compact format. Other than the comments the information is the same. For simple parameters,
  the columns in the brief format are:
    • parameter name
    • type (possibly preceded by "untracked")
    • possibly the word "optional"
    • default values
Deleted:
<
<
The compact format omits all comments inserted by the plugin itself and also most comments in the output that describe the generic printout format. If you are confused by the compact format, then look at the longer format. Other than comments, both formats contain the same information.
 
  1. The printout for each PSet description will start with a series of top level parameters, references to groups of parameters, or wildcards that could represent many parameters. Below the top level list there can be
Changed:
<
<
a series of sections describing groups. The sections are numbered and the reference to the section in the top level
>
>
a series of sections describing groups or default contents of large vectors. The sections are numbered and the reference to the section in the top level
  group will contain that number. In complex cases, one might find the search function on an editor useful to jump back and forth between a section and the reference to it.
Line: 91 to 77
  all parameters not marked "untracked" are "tracked". "tracked" implies the parameter value will be saved in the output file of a cmsRun job in the provenance.
Changed:
<
<
  1. Definition of "optional" - When the validation encounters a required parameter
>
>
  1. Definition of "optional": When the validation encounters a required parameter
  that is missing, it either inserts a default value for the parameter into the PSet or throws an exception. Parameters are either required or optional.
Line: 119 to 103
  validation is concerned "tracked" and "optional" are independent concepts. If a missing parameter will cause a module to throw, the module's description
Changed:
<
<
SHOULD declare that the parameter is required.
>
>
should declare that the parameter is required.
  However, the validation itself will be perfectly happy with tracked parameters that are missing if the module's description declares them optional.
Line: 123 to 107
  However, the validation itself will be perfectly happy with tracked parameters that are missing if the module's description declares them optional.
Deleted:
<
<
 
  1. Values in vectors are preceded by the index in square
Changed:
<
<
brackets. For example, "[0] " for the first element of the
>
>
brackets. For example, "[0] " precedes the first element of the
  vector.
Deleted:
<
<
 
  1. At the moment, the only allowed wildcard expression is simply "*", which matches any parameter label. We are thinking of extending this to either regular expressions
Line: 170 to 150
 
  • A C++ interface to allow other applications to access the description. For example, ConfDB might be populated
Changed:
<
<
this way. The data is already stored in the description. The main
>
>
this way. The description is already capable of storing the information. The main
  issue is defining the interface and its requirements.
  • Currently this only supports modules (Producers, Analyzers, Filters, and OutputModules). We may add support
Line: 190 to 170
 

An example

Here is an example of a job that will detect a parameter that

Changed:
<
<
is not allowed (works in 3_1_0_pre6):
>
>
is not allowed (works in 3_1_0_pre7):
 
  cmsRun FWCore/Integration/test/testParameterSet_cfg.py

Revision 22009-05-15 - DavidDagenhart

Line: 1 to 1
 
META TOPICPARENT name="SWGuideFrameWork"
Deleted:
<
<
 
Changed:
<
<

This page and the system it describes are only partially functional and still under active development.

>
>

This page and the system it describes are only partially functional and still under active development.

 This is in such a preliminary form that I do NOT recommend reading it yet! At the moment it is bits and pieces cut and pasted from other places that I am in the middle of editing.

Added:
>
>
 

Configuration Validation and Help

Each plugin can describe the parameters it allows or requires to be in

Changed:
<
<
its configuration in the plugin's C++ code. When this description has been implemented, several things will be possible.
  • When cmsRun creates a plugin it passes it a configuration. This configuration
will be validated against the description. Validation will throw an exception if certain errors occur. For example, it will throw if a parameter is in the configuration that is not in the description (or just mispelled). Required missing parameters can be inserted into the configuration if the description provides suitable defaults.
  • The executable edmPluginHelp can be used and will printout human readable
documentation about the configuration description of the module.
>
>
its configuration. This description will appear in a function named fillDescriptions in the plugin's C++ code. When this function has been implemented, several things will be possible.

  • cmsRun will validate the configuration for each plugin at the beginning of a job. If the validation determines that a parameter required by the description is missing, then it will insert the parameter into the configuration if it has a default. For other types of configuration errors, validation will throw an exception. For example, validation will throw if a parameter is in the configuration that is not in the description (protection against spelling errors).
  • The executable edmPluginHelp will printout human readable documentation about the configuration of a plugin.
 
  • CFI python files can be automatically generated based on the information in the
Changed:
<
<
description.
>
>
description. This will be part of the build and can also be run standalone with the edmWriteConfig executable.
 
  • This part is not implemented yet, but we imagine a C++ API that allows other
applications to access the details of the description.
Changed:
<
<

>
>
This validation system is designed so that it can be deployed incrementally, one module at a time. If the fillDescription function for a plugin has not been defined, then the validation will do nothing for that plugin.

edmPluginHelp

The executable named edmPluginHelp will print out the configuration descriptions in a human readable format. As I am writing this, this executable is not in any release yet (3_1_0_pre6 is the latest at the moment), but should appear soon. It is in CVS, tagged, and in the queue for 3_1_X.

  For the most part, the output of edmPluginHelp is self
Changed:
<
<
explanatory. "edmPluginHelp -h" also explains a few things. Here are some more details in no particular order.

1. For simple parameters, the columns in the brief format are: -- parameter name -- type (possibly preceded by "untracked") -- possibly the word "optional" -- default values

>
>
explanatory. The command "edmPluginHelp -h" explains the available options and gives a little information about the format. Here are some more details in no particular order.

  1. Some people get confused and think the output is a particular configuration. It is not. It describes the parameters that can be in a configuration.

  1. There is a long format and a brief format. For simple parameters, the columns in the brief format are:
    • parameter name
    • type (possibly preceded by "untracked")
    • possibly the word "optional"
    • default values
 The compact format omits all comments inserted by the plugin itself and also most comments in the output that describe the generic printout format. If you are
Line: 46 to 66
 format. Other than comments, both formats contain the same information.
Changed:
<
<
2. The printout for each PSet description will start
>
>
  1. The printout for each PSet description will start
 with a series of top level parameters, references to groups of parameters, or wildcards that could represent many parameters. Below the top level list there can be
Line: 55 to 75
 group will contain that number. In complex cases, one might find the search function on an editor useful to jump back and forth between a section and the reference to it.
Deleted:
<
<
(Although we expect many modules will be simple and only have a list of parameters and no sections).
 
Changed:
<
<
3. Nested ParameterSetDescriptions and vectors of them are analogous to the nested structures in ParameterSets.
>
>
  1. Nested ParameterSetDescriptions and vectors of them are analogous to the nested structures in ParameterSets.
 But other nested structures like AND Groups and switches
Changed:
<
<
will NOT appear in a PSet or python configuration
>
>
will NOT appear in a PSet or python configuration
 files. They only help to describe which combinations of parameters are allowed or required to be in the configuration.
Changed:
<
<
4. Parameters can be "tracked" or "untracked". edmPluginHelp
>
>
  1. Parameters can be "tracked" or "untracked". edmPluginHelp
 will explicitly print the word "untracked" just before the type for "untracked" parameters. It does NOT print "tracked" for the others. You just have to assume that all parameters not marked "untracked" are "tracked". "tracked" implies the parameter value will be saved
Changed:
<
<
in the output file of a cmsRun job in the provenance.
>
>
in the output file of a cmsRun job in the provenance.
 
Changed:
<
<
5. Definition of "optional"
>
>
  1. Definition of "optional" -
 When the validation encounters a required parameter that is missing, it either inserts a default value for
Changed:
<
<
the parameter into the PSet or throws an exception.
>
>
the parameter into the PSet or throws an exception.
 Parameters are either required or optional. When the validation encounters an optional missing parameter, it does nothing. It does not insert the missing parameter, nor does it throw. It just silently ignores it and proceeds to validate the next parameter in the description. In the printed output of
Changed:
<
<
edmPluginHelp the word "optional" is printed following
>
>
edmPluginHelp the word "optional" is printed following
 the type for optional parameters. The word "required" is not printed. One can assume anything not marked optional is required. Note that only the top level
Changed:
<
<
entries in a PSet description can be optional.
>
>
entries in a PSet description can be optional.
 
Changed:
<
<
6. Historically (before the validation and PSet description code existed), tracked parameters
>
>
  1. Historically (before the validation and PSet description code existed), tracked parameters
 were supposed to always be required parameters. The most common function used to get a tracked parameter
Changed:
<
<
from a ParameterSet would and still will throw if
>
>
from a ParameterSet would and still will throw if
 the parameter is not there. But for a long time there have been ways around this used in many places
Changed:
<
<
in CMSSW. As far as the PSet description and
>
>
in CMSSW. As far as the PSet description and
 validation is concerned "tracked" and "optional" are independent concepts. If a missing parameter will cause a module to throw, the module's description
Changed:
<
<
SHOULD declare that the parameter is required.
>
>
SHOULD declare that the parameter is required.
 However, the validation itself will be perfectly happy with tracked parameters that are missing if the module's description declares them optional.
Changed:
<
<
7. Values in vectors are preceded by the index in square
>
>
  1. Values in vectors are preceded by the index in square
 brackets. For example, "[0] " for the first element of the vector.
Changed:
<
<
8. At the moment, the only allowed wildcard expression is
>
>
  1. At the moment, the only allowed wildcard expression is
 simply "*", which matches any parameter label. We are thinking of extending this to either regular expressions or globbing. (If users requested this, this would be more
Line: 118 to 136
 is matched against the parameter label. The algorithm only searches for matching parameters of a single specified type.
Deleted:
<
<

Another section

 
Deleted:
<
<
There are two modules with these methods already implemented. See this function:
 
Changed:
<
<
EventContentAnalyzer::fillDescriptions(ConfigurationDescriptions & descriptions)
>
>

Validation

 
Changed:
<
<
There is also a test module that tests all the features of the ParameterSetDescription code, but it is hopefully much more complicated than any real module will ever be.

This is designed so that it can be deployed incrementally, one module at a time. I do not think management has decided what the pace of deployment will be yet or whether all modules will be required to implement a description or just certain modules. I think a decision whether or not to deploy at all still needs to be made. Not many people have looked at this yet and it is still under development.

Trying to retrieve a parameter that is not in the description will result in a configuration exception being thrown that ends the process. But this only happens if the module has a description. No validation is done if the description has not been implemented yet for a particular module.

There are various pieces to this system. - validation that will throw for certain configuration errors - validation can also insert missing parameters in ParameterSets - I am currently working on an executable that will print human readable documentation from these descriptions. - cfi files can be automatically generated from the description and this is being added to scram as part of the build

>
>

Describing the configuration in C++

 
Changed:
<
<
Large parts of this are already working. For example, try this is 3_1_0_pre6:
>
>
There is not much documentation about writing the C++ fillDescriptions functions yet. This section needs to be improved and expanded. The original proposal that Chris Jones wrote is here:
 
Changed:
<
<
cmsRun FWCore/Integration/test/testParameterSet_cfg.py
>
>
https://twiki.cern.ch/twiki/bin/view/CMS/FwEdmParameterSetValidationProposal
 
Changed:
<
<
It will fail because the configuration has a parameter not in the description of the allowed parameters.
>
>
This is out of date, but gives a flavor for how one creates a description. The following two functions can be looked up in the code in the CVS head. The first is a very simple example. We expect the configuration for most modules to be this simple. The second tests all the possible features of a description and is much more complicated than we expect any real description to be.

  • EventContentAnalyzer::fillDescriptions
  • ProducerWithPSetDesc::fillDescriptions

Features that still need to implemented

We plan to implement these features.

  • Descriptions should support required parameters without defaults
  • Modify SCRAM to automatically create CFI files during the build
  • Implement the fillDescriptions function in many plugins. Currently very few plugins have this function.
  • Continue to improve this documentation
We are discussing whether and when to implement these features.

  • A C++ interface to allow other applications to access the description. For example, ConfDB might be populated this way. The data is already stored in the description. The main issue is defining the interface and its requirements.
  • Currently this only supports modules (Producers, Analyzers, Filters, and OutputModules). We may add support for services, sources, ESSources, and ESProducers.
  • Defaults for wildcards.
  • Placeholders in the python to allow replacing parameter values in cfg files which do not exist in the CFI files.
  • Some discussion of eliminating CFI files entirely in favor of inserting missing parameters. (Probably very far in the future if ever)
  • Support the MessageLogger special case
  • Extend to support lower level plugins
  • Implement a reset function for the value of a single parameter description that makes it easier to define multiple descriptions for the same plugin
  • Improve error checking and messages
 
Changed:
<
<
There is not much documentation on this yet and that is on my list of things to do. This is the original proposal that Chris Jones wrote:
>
>

An example

 
Changed:
<
<
https://twiki.cern.ch/twiki/bin/view/CMS/FwEdmParameterSetValidationProposal
>
>
Here is an example of a job that will detect a parameter that is not allowed (works in 3_1_0_pre6):
 
Changed:
<
<
It is a bit out of date but give a flavor for how the system works. I am doing the work to implement Chris's design with some help from Chris.
>
>
  cmsRun FWCore/Integration/test/testParameterSet_cfg.py
 
Added:
>
>
It will fail because the configuration has a parameter not in the description of the allowed parameters.
  -- DavidDagenhart - 14 May 2009 \ No newline at end of file

Revision 12009-05-14 - DavidDagenhart

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="SWGuideFrameWork"

This page and the system it describes are only partially functional and still under active development.

This is in such a preliminary form that I do NOT recommend reading it yet! At the moment it is bits and pieces cut and pasted from other places that I am in the middle of editing.

Configuration Validation and Help

Each plugin can describe the parameters it allows or requires to be in its configuration in the plugin's C++ code. When this description has been implemented, several things will be possible.

  • When cmsRun creates a plugin it passes it a configuration. This configuration
will be validated against the description. Validation will throw an exception if certain errors occur. For example, it will throw if a parameter is in the configuration that is not in the description (or just mispelled). Required missing parameters can be inserted into the configuration if the description provides suitable defaults.
  • The executable edmPluginHelp can be used and will printout human readable
documentation about the configuration description of the module.
  • CFI python files can be automatically generated based on the information in the
description.
  • This part is not implemented yet, but we imagine a C++ API that allows other
applications to access the details of the description.

For the most part, the output of edmPluginHelp is self explanatory. "edmPluginHelp -h" also explains a few things. Here are some more details in no particular order.

1. For simple parameters, the columns in the brief format are: -- parameter name -- type (possibly preceded by "untracked") -- possibly the word "optional" -- default values The compact format omits all comments inserted by the plugin itself and also most comments in the output that describe the generic printout format. If you are confused by the compact format, then look at the longer format. Other than comments, both formats contain the same information.

2. The printout for each PSet description will start with a series of top level parameters, references to groups of parameters, or wildcards that could represent many parameters. Below the top level list there can be a series of sections describing groups. The sections are numbered and the reference to the section in the top level group will contain that number. In complex cases, one might find the search function on an editor useful to jump back and forth between a section and the reference to it. (Although we expect many modules will be simple and only have a list of parameters and no sections).

3. Nested ParameterSetDescriptions and vectors of them are analogous to the nested structures in ParameterSets. But other nested structures like AND Groups and switches will NOT appear in a PSet or python configuration files. They only help to describe which combinations of parameters are allowed or required to be in the configuration.

4. Parameters can be "tracked" or "untracked". edmPluginHelp will explicitly print the word "untracked" just before the type for "untracked" parameters. It does NOT print "tracked" for the others. You just have to assume that all parameters not marked "untracked" are "tracked". "tracked" implies the parameter value will be saved in the output file of a cmsRun job in the provenance.

5. Definition of "optional" When the validation encounters a required parameter that is missing, it either inserts a default value for the parameter into the PSet or throws an exception. Parameters are either required or optional. When the validation encounters an optional missing parameter, it does nothing. It does not insert the missing parameter, nor does it throw. It just silently ignores it and proceeds to validate the next parameter in the description. In the printed output of edmPluginHelp the word "optional" is printed following the type for optional parameters. The word "required" is not printed. One can assume anything not marked optional is required. Note that only the top level entries in a PSet description can be optional.

6. Historically (before the validation and PSet description code existed), tracked parameters were supposed to always be required parameters. The most common function used to get a tracked parameter from a ParameterSet would and still will throw if the parameter is not there. But for a long time there have been ways around this used in many places in CMSSW. As far as the PSet description and validation is concerned "tracked" and "optional" are independent concepts. If a missing parameter will cause a module to throw, the module's description SHOULD declare that the parameter is required. However, the validation itself will be perfectly happy with tracked parameters that are missing if the module's description declares them optional.

7. Values in vectors are preceded by the index in square brackets. For example, "[0] " for the first element of the vector.

8. At the moment, the only allowed wildcard expression is simply "*", which matches any parameter label. We are thinking of extending this to either regular expressions or globbing. (If users requested this, this would be more likely to happen or happen sooner.) The wildcard pattern is matched against the parameter label. The algorithm only searches for matching parameters of a single specified type.

Another section

There are two modules with these methods already implemented. See this function:

EventContentAnalyzer::fillDescriptions(ConfigurationDescriptions & descriptions)

There is also a test module that tests all the features of the ParameterSetDescription code, but it is hopefully much more complicated than any real module will ever be.

This is designed so that it can be deployed incrementally, one module at a time. I do not think management has decided what the pace of deployment will be yet or whether all modules will be required to implement a description or just certain modules. I think a decision whether or not to deploy at all still needs to be made. Not many people have looked at this yet and it is still under development.

Trying to retrieve a parameter that is not in the description will result in a configuration exception being thrown that ends the process. But this only happens if the module has a description. No validation is done if the description has not been implemented yet for a particular module.

There are various pieces to this system. - validation that will throw for certain configuration errors - validation can also insert missing parameters in ParameterSets - I am currently working on an executable that will print human readable documentation from these descriptions. - cfi files can be automatically generated from the description and this is being added to scram as part of the build

Large parts of this are already working. For example, try this is 3_1_0_pre6:

cmsRun FWCore/Integration/test/testParameterSet_cfg.py

It will fail because the configuration has a parameter not in the description of the allowed parameters.

There is not much documentation on this yet and that is on my list of things to do. This is the original proposal that Chris Jones wrote:

https://twiki.cern.ch/twiki/bin/view/CMS/FwEdmParameterSetValidationProposal

It is a bit out of date but give a flavor for how the system works. I am doing the work to implement Chris's design with some help from Chris.

-- DavidDagenhart - 14 May 2009

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback