O2O Framework HOWTO

Complete: 3

Goal of this page

This page describes the use of the Online-to-Offline (O2O) Framework for the O2O administrator. The O2O Framework is a collection of scripts that setup the offline database (ORCON) to receive data from the online database (OMDS) via an SQL query over a database link, and then make usable POOL-ORA objects from that data using POOL command-line tools. The O2O Framework is a simplification to the many steps required to go from online relational data to offline POOL-ORA objects.

Data that will undergo O2O transfer require some expertise in both the online data format and the offline data format. A HOWTO on preparing data for O2O by subdetector experts is here.

Setup of the offline database

In order to do O2O transfer, the offline database must be prepared.

Offline Schema Setup

General setup of the offline database is described here. In this step, the offline schema is prepared but no O2O-specific structures are prepared. This must be done before any of the following steps.

Setup Offline DB for O2O

O2O transfer requires that the transfer procedure be written into the subdetector's offline schema and the procedure registered in the O2O_SETUP table on the general schema. To do this follow the steps below.

Setup Work Area

Check out CondTools/OracleDBA and CondTools/O2OFramework into your CMSSW project area. The relevant directories are described below:

CondTools/OracleDBA/
scripts/ Offline DBA scripts
dbconfig/ Offline DB configuration XML
xml/ POOL-ORA object mapping and database setup XML
sql/ Additional offline table creation SQL and utility procedures

CondTools/O2OFramework
scripts/ O2O DBA scripts
dbconfig/ O2O configuration XML
sql/ O2O Framework creation SQL and subdetector object transfer procedures

Edit o2oconfiguration.xml

CondTools/O2OFramework/dbsetup/o2oconfiguration.xml contains the configuration of O2O for all CMS subdetectors and objects. This configuration file should be committed to CVS on updates and follow the CMSSW release schedule so that the configuration history can be traced.

An example file is provided below along with comments describing the purpose of the elements and attributes.

<?xml version="1.0" ?>

<o2oconfiguration>

  <general online_db='omds'>
    <dbobject name='o2o_setup' sqlfile='create_o2o_setup.sql' />
    <dbobject name='o2o_log' sqlfile='create_o2o_log.sql' />
    <dbobject name='master_payload_o2o' sqlfile='master_payload_o2o.sql' />
  </general>

  <detector name='ECAL' online_schema='CMS_ECAL'>
    <object name='EcalPedestals' table='ECALPEDESTALS' timetype='runnumber' tagsuffix='from_online' />
  </detector>

  ... more detectors ...

</o2oconfiguration>

<general> Element

This is the general configuration section, of which there should be only one element of type <general>. The 'online_db' attribute specifies the name of the online DB in the setup, which is used for creating the database link to the online DB. The nested <dbobject> elements represent tables and procedures that are created in the general schema of the offline DB. These should never be edited unless the O2OFramework is being changed.

<detector> Elements

One or more <detector> elements represent detector schemas. Each one is giving a 'name' and 'online_schema' attribute. Nested within the <detector> element is one or more <object> elements which represent objects that will undergo O2O. The attributes available for this element are:

<object> Element Attributes
name (required) Name of the C++ object that will be created in the offline DB
table (required) The top-level-table for this object, which contains the IOV_VALUE_ID and TIME columns
timetype One of 'runnumber' or 'timestamp', representing the type of time that is used for the IOV. The default is 'runnumber'.
tagsuffix (required) Used in creation of the tag for the object. The prefix is the object name, so tag = name_tagsuffix
infiniteiov One of 'true' or 'false', whether or not to use infinite IOV mode, which assigns an infinite IOV to the object transferred.
query Optional SQL WHERE clause to use when selecting objects from the 'table' to include in the tag.

Note that each <detector> should have a corresponding entry in the CondTools/OracleDBA/dbconfig/dbconfiguration.xml file. Also, each <object> should have the payload transfer procedure in the sql/ directory with the name <<objectName>_payload_o2o.sql. This is an assumption made by the setup script.

Edit authentication.xml

Ensure entries to CondTools/OracleDBA/dbsetup/authentication.xml describe the login for each of the following:

  1. The general schema of the offline DB described in CondTools/OracleDBA/dbconfig/dbconfiguration.xml
  2. The offline detector schema for each <detector> described in CondTools/OracleDBA/dbconfig/dbconfiguration.xml
  3. The online detector schema for each <detector> described in CondTools/OracleDBA/dbconfig/o2oconfiguration.xml

Execute cmscond_bootstrap_o2o.pl

The script cmscond_bootstrap_o2o.pl reads the configuration XML files in your project area (or files specified in command line arguments) and sets up O2O on the offline database according to their configuration. The script checks for the existence of needed files before beginning its work and stops on any errors. Here is the help information from the script:

cmscond_bootstrap_o2o.pl [options] [detector1 detector2 ...]
Options:
--general    Setup the general offline schema for O2O
--dbaconfig  Offline DB configuration file (hardcoded default in project area)
--o2oconfig  O2O configuration file (hardcoded default in project area)
--auth       DB connection file (hardcoded default in project area)
--online_db  Name of online DB (default in o2oconfig)
--offline_db Name of offline DB (default in dbaconfig)
--all        Setup all detectors in O2O configuration file
--fake       Don't actually do anything, only print commands
--debug      Print additional debug information
--log        Log file
--help, -h   Print this message and exit

The --general option must be done once on the offline DB to set up the general schema. This writes the bookeeping tables to the general schema as well as the master_payload_o2o() procedure.

The positional arguments [detector1 detector2 ...] allow you to specify which detector's schema (as described in o2oconfiguration.xml) you wish to set up. If you wish to set up everything described in o2oconfiguration.xml use the --all option.

The --fake option will check the connections specified in the configuration XMLs but will not actually modify the offline DBs. Use this in conjunction with --debug to get verbose output of what the script will do without attempting to modify the database.

Execute O2O

Once the offline database has been prepared for O2O with cmscond_bootstrap_o2o.pl O2O can be executed using cmscond_do_o2o.pl. The same configuration files are used for both programs. The help information for this script is as follows:

cmscond_do_o2o.pl [options] [[--all] | [object1 object2 ...]]
Options:
--dbaconfig  Offline DB configuration file (hardcoded default in project area)
--o2oconfig  O2O configuration file (hardcoded default in project area)
--auth       DB connection file (hardcoded default in project area)
--offline_db Name of offline DB (default in dbaconfig)
--all        Setup all objects in O2O configuration file
--fake       Don't actually do anything, only print commands
--debug      Print additional debug information
--log        Log file
--help, -h   Print this message and exit

Instead of specifying detectors as in the bootstrap script, this time you specify object names, e.g. EcalPedestals, or use the --all option for all objects described in o2oconfiguration.xml. The options for this program are identical to the options available for the bootstrapping program as described above (minus the --general option).

Adding O2O to a cron job for automated O2O

Since the execution of the cmscond_do_o2o.pl script requires the CMSSW environment to be set up, a small shell script will have to be executed from the cron daemon that does this before calling the program. Here is an example:

do-o2o.sh

#!/bin/sh

cd path/to/CMSSW/project
eval `scramv1 -sh`
cmscond_bootstrap_o2o.pl --log path/to/log.txt --all

Then this program can be added to the crontab (edit with 'crontab -e'). As an example, to run O2O every hour:

crontab

0 * * * * /path/to/do-o2o.sh

Review Status

Reviewer/Editor and date Comments
Main.egeland - 11 Aug 2006 page author
Main.tomalini - 09 Oct 2006 page content last edited
JennyWilliams - 03 Apr 2007 moved page into swguide


Responsible: FrancescaCavallari
Last reviewed by: Reviewer
Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2010-03-19 - unknown



 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback