How to write conditions data - PoolDBOutputService

Complete: 3


Parameters and public interfaces described in this page are compatible wit CMSSW_12_1_X and higher. For using older release versions, read older version of the page.

In CondCore system, edm service module PoolDBOutputService is responsible for persistify conditions data as well as managing the interval of validity, it takes care of all the interactions with the database. A documentation about how to use an edm service in general can be found here. In software version higher than CMSSW_1_8_0, logging feature is added. One can choose to log transaction status into a log DB which may or may not be in the same account as the real data. This feature is the building block of the CMS.PopCon application.

Writing conditions data into database consists of two parts: writing the bulk payload data and assigning the interval of validity to the payload. The payload data must be defined as objects by the user. User is required to provide the dictionary information for each object. The user does not need and should not write any database specific code in his/her analysis module nor in the object definition. Guidelines and procedures for creating new persistent-capable objects and the dictionary are described in details here.

A detailed description of the concept of interval of validity(IOV), IOV metadata and IOV tagging policy can be found here. In the current implementation, an IOV index is always closed with the end-of-time.

To resolve the PopConSourceHandler leaks memory issue, new policy is advice to keep the ownership of the payload object at the client side. The new reference based method, writeOneIOV, is introduced and encouraged to migrate.

Service callback functions

The following public functions of PoolDBOutputService can be called by the client module :
writeOneIOV : write one object, either create or append. The template function stores also the given payload object and takes the reference of the payload object as the first parameter.

writeOne : Obsolete use writeOneIOV write one object, either create or append.

createOneIOV : this is the new reference based method to replace the Obsolete createNewIOV.

createNewIOV : Obsolete use writeOneIOV create a new IOV sequence. There are two forms of the function. The template function stores also the given payload object. Note: the function takes the ownership of the payload object pointer. The non-templated function attaches IOV to existing payload objects. Till-time of the first payload is required. If since-time is not specified, global begin-of-time is assumed.

appendOneIOV : this is the new reference based method to replace the Obsolete appendSinceTime and appendTillTime.

appendSinceTime : Obsolete use writeOneIOV append since time. There are templated and non-templated forms of the function.

isNewTagRequest : returns true if the tag name specified in the configuration is new to the given EventSetup record name

currentTime : returns the current time of the event according to the "timetype" parameter specified in the configuration

endOfTime : returns the end-of-time value according to the "timetype" parameter specified in the configuration

tag : returns the tag name attached to the given EventSetup record name

setLogHeaderForRecord : set log entry header for a given record

tagInfo : retrieve the tag information associated with a record

queryLog : get the DB log interface

Configuration Parameters for PoolDBOutputService

name type description default allowed
connect string database connection string None POOLORA connection string format
timetype untracked string the type of time variable "runnumber" "runnumber", "timestamp"
BlobStreamerName untracked string blobStreaming service to use "DefaultBlobStreamingService" "TBufferBlobStreamingService", "DefaultBlobStreamingService"
toPut(see below) VPSet records to put into the database None comma separated parameter pairs
DBParameters(see below) named Parameter Set common database connection parameters default values provided by the common .cfi file  

Parameters expected in toPut VPSet

name type description default allowed
record string EventSetupRecord the data belong(required) None existing records
tag string the tag to the iov(required) None non-empty strings

Named Parameter Set DBParameters sets the connection parameters. The default values are provided by the framework in the following files CondCore/DBCommon/data/CondDBCommon.cfi or CondCore/DBCommon/data/CondDBSetup.cfi.

The common practice is to include one of the above default .cfi in job configuration and replace the parameters you want to customise. Parameters in named parameter set DBParameters

name type description default allowed
authenticationPath untracked string directory path to the authentication file "." current dir empty string or a valid path. If empty string, environment variables CORAL_AUTH_USER, CORAL_AUTH_PASSWORD should be set by the user
messageLevel untracked int32 verbosity level 0 0(Error),1(Warning), 2(Info), 3(Debug)
enableConnectionSharing untracked bool connection sharing switch true true, false
connectionTimeOut untracked int32 connection timeout value 600(sec)  
connectionRetrialPeriod untracked int32 connection retry time interval 30(sec)  
connectionRetrialTimeOut untracked int32 connection retry timeout 180(sec)  

Hint: If in one job you use more than one Cond DB modules and connect to more than one database instance, include CondDBSetup.cfi instead of CondDBCommon.cfi and define the connect and catalog parameters for each DB instance.

Note: parameter DBParameters.authenticationPath should point to the location of your authentication.xml. For example, at CERN one of the locations on AFS is "/afs/".

How to create a new IOV sequence

In your algorithm module EDAnalyzer, get hold an instance of cond::service::PoolDBOutputService:
edm::Service<cond::service::PoolDBOutputService> poolDbService;
if( poolDbService.isAvailable() ){ //use it to write new data}
Then assign the interval of validity to the new payload object using the public callback function writeOneIOV. A snippet of a EDAnalyzer module MyCalibAlgo is shown below as example:
    edm::Service<cond::service::PoolDBOutputService> mydbservice;
    if( !mydbservice.isAvailable() )  return;
    Pedestals myped; // Create a normal instance of Pedestals object
    //fillup myped
    //create new infinite IOV
    cond::Time_t firstSinceTime=mydbservice->beginOfTime();
    mydbservice->writeOneIOV(myped, firstSinceTime, "PedestalsRcd");
Note: the payload object, e.g. Pedestals, should not be created with the new operator. A normal object instance or a smart pointer is recommended. The user takes the ownership of the object and should delete it when no more needed.

Hint: in the simple case of writing a payload with infinite IOV, you can put the DB service related calls in the function EDAnalyzer::endJob since you do not rely on the time information. If you need event time information for assigning IOV, put Cond DB calls in EDAnalyzer::analyze method and ask for currentTime from the PoolDBOutputService.

Each new IOV sequence must associate to a tag. The tag should be unique for your task, it should not already exist in the database you are writing into. The tag should be specified in the configuration parameter toPut of PoolDBOutputService. Shown below is an example job configuration:


   process MYTEST={
        include "CondCore/DBCommon/data/CondDBCommon.cfi"
        replace CondDBCommon.connect = "sqlite_file:test.db"
        replace CondDBCommon.DBParameters.messageLevel = 3
        replace CondDBCommon.DBParameters.authenticationPath = "/afs/"
        source = EmptyIOVSource {
           string timetype = "runnumber"
           untracked uint32 firstRun = 1
           untracked uint32 lastRun = 1
           uint32 interval = 1    
        service = PoolDBOutputService{
           using CondDBCommon
           untracked string timetype="runnumber"
            VPSet toPut = {
              {string record = "PedestalsRcd" string tag = "mytest" }
        path p = {mytest}
        module mytest = MyCalibAlgo{
              string record = "PedestalsRcd"

How to append IOV to an existing tag

There is no difference in job configuration for appending new data to existing tag and for creating new IOV. However, the PoolDBOutputService call logic inside your EDAnalyzer may become slightly more complicated if you insist that the first entry shall start from beginOfTime. We modify MyCalibAlgo example shown above to include the append usecase:

    cond::Time_t sinceTime =  mydbservice->isNewTagRequest("PedestalsRcd") ? 
            mydbservice->beginOfTime() : mydbservice->currentTime();
    mydbservice->writeOneIOV(myped, sinceTime,"PedestalsRcd");

More examples can be found in CMSSW/CondCore/DBOutputService/test/stubs and data directories.

How to log status into log db

This is the machinary behind the CMS.PopCon applications. Though PoolDBOutput users can use the logging feature as it is, it is mainly expected to be used by the CMS.PopCon package. In the service configuration file, set the untracked string parameter "logconnect" to the log db connection string, the logging will be switched on.
untracked string logconnect = "sqlite_file:log.db"
If you want to set appropriate metadata to identify your log entry, in your source code, call the service method setLogHeaderForRecord.

Migration Status

Please refer to this google Doc:

Edit | Attach | Watch | Print version | History: r16 < r15 < r14 < r13 < r12 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r16 - 2021-11-04 - YuanChao

    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback