Difference: CondDBManagement (1 vs. 29)

Revision 292018-10-11 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 24 to 24
 
  • each merge request should target the branch for the correct data type ("dt-datatype" branches)
  • if a merge request should be applied to other data types, the required data types should be specified as labels (will be propagated by CondDB managers via Gitlab cherry-pick)
  • if a merge request include several commits, it should be accepted ticking the "squash" checkbox, unless the requester explicitly asks not to
Changed:
<
<
  • when appropriate, a tag should be created for the data type, recording in the tag message the data types the tag applies to in the format:
>
>
  • when appropriate, a tag should be created for the data type (and optionally simulation type and reconstruction type), recording in the tag message the data types the tag applies to in the format:
 
---
datatypes: [2017, 2016]
Added:
>
>
simtypes: [Sim10, Sim09] recotypes: [Reco14]
 
Line: 36 to 38
 

Creation of a Global Tag

Once all required changes have been merged in the data type branch, a new global tag can be created via the "create new tag" link in the Gitlab project.
Changed:
<
<
The tag message must contain a section enclosed between two lines made of only 3 dashes (---). This section must contain valid YAML code declaring the named field datatypes containing a list of numbers or strings, one for each data type the tag is meant for.
>
>
The tag message must contain a section enclosed between two lines made of only 3 dashes (---). This section must contain valid YAML code declaring the named field datatypes and optionally the fields simtypes and recotypes, containing lists of numbers or strings, one for each data/sim/reco type the tag is meant for.
 For example:
---
datatypes: [2017, 2016]
Added:
>
>
simtypes: [Sim10, Sim09] recotypes: [Reco14]
 
Line: 272 to 276
  This has to be done only for those CondDB global tags, which introduce changes important for HLT. The latter knowledge, typically, is provided by the patch requester. In any case, this action has to be discussed with the LHCb Computing project leader and the LHCb HLT coordinator. It may happen that, even though a new tag has changes relevant for HLT, putting this new tag in action for HLT may be postponed for whatever reasons.
Deleted:
<
<

Adding compatibility information to Ariadne

In this step, the new tag has to be registered in the Ariadne system. The procedure has to be done from LXPLUS, or any SLC machine with Kerberized CERN login and 'cern-get-sso-cookie' package installed (note that access to Ariadne from SLC5 and earlier is not supported any more).

In particular, insertion of the new tag and of all its compatibility relationships to other metadata entities, tracked by Ariadne (see the flavor list of all tracked entities at the bottom of the section), into the Ariadne system has to be performed. Below some typical use cases are listed:

  1. Adding a tag WITHOUT compatibility relationships (i.e, incompatible with everything):
    This covers the simplest case of adding a new tag, which is incompatible with everything known by that time. See below the command line on an example of tag TAG_NAME located in DDDB partition:
          ariadne add tag:DDDB:TAG_NAME
    The latter command line will add a standalone 'tag' node to the knowledge graph of the Ariadne system which is not related in any way to other Ariadne nodes. This, though, doesn't prevent to add any relationships to it later on.
  2. Adding a tag WITH compatibility relationships, IDENTICAL to those of another tag:
    If a new tag has to inherit all relationships from an existent tag (which is what is wanted in most of the cases) the following command line will do the job:
          ariadne add tag:DDDB:TAG_NAME =tag:DDDB:ANOTHER_TAG_NAME
    The instruction in the second argument, uses the '=' operator to specify the node where relationships have to be cloned from.
  3. Adding a tag with NEW set of compatibility relationships:
    This is the most non-trivial use case. Let's say one needs to clone all relationships from an existent node, then add relationship to, e.g., Reco20 reco type, and remove another relationship to 2012 detector type. To accomplish this task two extra instructions, using '+' and '-' operators, have to be given:
          ariadne add tag:DDDB:TAG_NAME =tag:DDDB:ANOTHER_TAG_NAME +reco_type:Reco20 -detector_type:2012

Tip, idea To check what are the current relationships of a tag use the following command:

  ariadne q[uery] tag:DDDB:TAG_NAME

or, in batch:

  ariadne q -r tag:DDDB:TAG_NAME | grep WHATEVER


Vision ATTENTION :
One has to always devote extra attention to creating realistic relationships of a new tag with detector-, reco- and/or sim-types. In particular, the following relationships have to be ensured:

  • For DDDB tags: relationships with detector-, reco-, sim-types.
  • For LHCBCOND tags: relationships with detector- and reco-types.
  • For SIMCOND tags: relationships with detector-, reco-, sim-types.
  • For DQFLAGS tags: relationships with detector-types (relationships to other entities are not tracked).
These relationships have to be verified even in the quite regular use case of cloning compatibilities from the tag the new one is based on, since, even though the neighboring tags are usually quite similar by content, the new tag can have significantly different appropriation.


Info The following metadata entities are tracked by Ariadne:

      application:NAME:VERSION
      tag:PARTITION:TAGNAME
      reco_type:NAME
      sim_type:NAME
      detector_type:NAME

Tip, idea It is usually useful to add a -d option to all the command lines above to see what exactly the tool is performing. To see the command-line tool help issue ariadne add -h.

 

Adding new CondDB tags to the BookKeeping database

\ No newline at end of file

Revision 282018-05-30 - GloriaCorti

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 69 to 69
 
Added:
>
>

Porting changes for Upgrade to MagnetUp

A branch for magnet up is setup upgrade/magup matching the contents of upgrade/master but for the magnetic field. When ready to port changes added to upgrade/master to make a corresponding mag up tag, one should * create a Merge Request with upgrade/master as the source branch and upgrade/magup as the target branch * double check the only changes will be those applied to master * accept the merge request * create a tag in upgrade/magup corresponding to the one in upgrade/master

 

COOL Conditions Database

CondDB operation assistance systems

Revision 272017-10-12 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 179 to 179
 fi lb-run --dev LHCb/latest \$GITENTITYRESOLVERROOT/utils/make_git_conddb.py --no-head \$SQLITEDBPATH/${part}.db \$SQLITEDBPATH/../doc/release_notes.xml ${part} # the option --dev to lb-run is needed to get tags not published yet
Added:
>
>
# For Upgrade use the following lb-run --dev LHCb/latest \$GITENTITYRESOLVERROOT/utils/make_git_conddb.py --no-head --tag-prefix=upgrade/ \$SQLITEUPGRADEDBPATH/${part}.db \$SQLITEUPGRADEDBPATH/../doc/release_notes.xml ${part}
 %ENDCODE%

Check the generated tags (and branches) and push them. On top of the new tags, the script generate some internal branches (branch-N) that should be ignored, and some data type branches (dt-TYPE) that should be pushed to the main repository too.

Revision 262017-08-13 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 21 to 21
  The management of contributions to Conditions Databases in Git (https://gitlab.cern.ch/lhcb-conddb) is not much different from the workflow of any regular software project:
Changed:
<
<
  • each merge request should target the branch for the correct data type ("dt-_datatype_" branches)
>
>
  • each merge request should target the branch for the correct data type ("dt-datatype" branches)
 
  • if a merge request should be applied to other data types, the required data types should be specified as labels (will be propagated by CondDB managers via Gitlab cherry-pick)
  • if a merge request include several commits, it should be accepted ticking the "squash" checkbox, unless the requester explicitly asks not to
Changed:
<
<
  • when appropriate, a tag should be created for the appropriate data type, and the data types a tag applies to must be specified in the tag message (not release notes) in the format:
>
>
  • when appropriate, a tag should be created for the data type, recording in the tag message the data types the tag applies to in the format:
 
---
datatypes: [2017, 2016]
Line: 163 to 163
 lb-run --dev LHCb/latest CondDBAdmin _GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831 %ENDCODE%
Added:
>
>
Warning, important new global tags must be cloned to Git before the SQLite files are published.

Porting changes from COOL to GitCondDB

When new global tags are created in COOL databases, they have to be ported to the Git copies.

The tool used originally to clone SQLite COOL databases to Git can be used to update the global tags:

<!-- SyntaxHighlightingPlugin -->
part=LHCBCOND
if [ ! -e ${part} ] ; then
  git clone --reference /cvmfs/lhcb.cern.ch/lib/lhcb/git-conddb/${part}.git ssh://git@gitlab.cern.ch:7999/lhcb-conddb/${part}.git
else
  (cd ${part} && git checkout master && git pull)
fi
lb-run --dev LHCb/latest \$GITENTITYRESOLVERROOT/utils/make_git_conddb.py --no-head \$SQLITEDBPATH/${part}.db \$SQLITEDBPATH/../doc/release_notes.xml ${part}
# the option =--dev= to lb-run is needed to get tags not published yet
<!-- end SyntaxHighlightingPlugin -->

Check the generated tags (and branches) and push them. On top of the new tags, the script generate some internal branches (branch-N) that should be ignored, and some data type branches (dt-TYPE) that should be pushed to the main repository too. For example:

<!-- SyntaxHighlightingPlugin -->
cd ${part}
git push origin tag dddb-12345678 dt-2017 master
<!-- end SyntaxHighlightingPlugin -->
 

Releasing CondDB

To make a CondDB release a set of procedures has to be completed in the order stated below.

Revision 252017-07-24 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 23 to 23
 
  • developers should create merge requests to one of the projects in https://gitlab.cern.ch/lhcb-conddb
  • each merge request should target the branch for the correct data type ("dt-_datatype_" branches)
  • if a merge request should be applied to other data types, the required data types should be specified as labels (will be propagated by CondDB managers via Gitlab cherry-pick)
Added:
>
>
  • if a merge request include several commits, it should be accepted ticking the "squash" checkbox, unless the requester explicitly asks not to
 
  • when appropriate, a tag should be created for the appropriate data type, and the data types a tag applies to must be specified in the tag message (not release notes) in the format:
    ---
Line: 43 to 44
 
Changed:
<
<
Was the tag has been created, it should be ported to the COOL database.
>
>
Once the tag has been created, it should be ported to the COOL database.
 

Porting changes from Git to COOL

A GitCondDB version (tag, commit) can be converted to COOL easily by cloning the GitCondDB repository, checking out the tag and using the usual COOL procedure on the resulting directory.
Line: 57 to 58
 lb-run LHCb/latest CondDBAdmin_Commit.py ... %ENDCODE%
Added:
>
>

Changes for Upgrade

Upgrade related changes are kept in branches and tags prefixed with upgrade, so the master branch for Upgrade is upgrade/master, etc.

For Upgrade tags, the only allowed data type is Upgrade so the tag message must always be:

---
datatypes: [Upgrade]
---
 

COOL Conditions Database

CondDB operation assistance systems

Revision 242017-07-23 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Line: 28 to 28
 
datatypes: [2017, 2016]
Added:
>
>

Changes in gitlab are automatically propagated to the Conditions Databases copies on CVMFS.

Creation of a Global Tag

Once all required changes have been merged in the data type branch, a new global tag can be created via the "create new tag" link in the Gitlab project.

The tag message must contain a section enclosed between two lines made of only 3 dashes (---). This section must contain valid YAML code declaring the named field datatypes containing a list of numbers or strings, one for each data type the tag is meant for. For example:

---
datatypes: [2017, 2016]
---

Was the tag has been created, it should be ported to the COOL database.

Porting changes from Git to COOL

A GitCondDB version (tag, commit) can be converted to COOL easily by cloning the GitCondDB repository, checking out the tag and using the usual COOL procedure on the resulting directory.

As an example:

<!-- SyntaxHighlightingPlugin -->
git clone https://gitlab.cern.ch/lhcb-conddb/LHCBCOND.git
cd LHCBCOND
git checkout cond-12345678
cd ..
lb-run LHCb/latest CondDBAdmin_Commit.py  ...
<!-- end SyntaxHighlightingPlugin -->
 

COOL Conditions Database

CondDB operation assistance systems

Line: 41 to 69
 

CondDB release preparation procedures

Committing CondDB patches

Changed:
<
<
In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin _Commit.py script, available under the LHCb Project environment:
<!-- SyntaxHighlightingPlugin -->
lb-run LHCb/latest CondDBAdmin _Commit.py -h <br />
<!-- end SyntaxHighlightingPlugin -->
The script can take an input to be committed in two interchangeable forms:
>
>
In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin _Commit.py script, available under the LHCb Project environment:
<!-- SyntaxHighlightingPlugin -->
lb-run LHCb/latest CondDBAdmin _Commit.py -h
<!-- end SyntaxHighlightingPlugin -->

The script can take an input to be committed in two interchangeable forms:

 
  • XML files
  • SQLite database slice
If there are no completely new files in a patch then either the XML input file tree, or the SQLite DB slice internal nodes tree, has to reproduce the internal tree of the destination CondDB SQLite database. Otherwise, the CondDB node which has a wrong/incomplete path will be added as a new node.
Line: 54 to 87
  The first one is the standard SQLite CondDB location which you get when requesting the development environment ( lb-run --dev LHCb/latest). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).
Changed:
<
<
An example of a patch commitment looks like:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest bash --norc echo $SQLITEDBPATH CondDBAdmin _Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME <br />
<!-- end SyntaxHighlightingPlugin -->
where
>
>
An example of a patch commitment looks like:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest bash --norc
echo $SQLITEDBPATH
CondDBAdmin _Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME
<!-- end SyntaxHighlightingPlugin -->
where
 
  • NAME OF REQUESTOR - full name of the person submitted the patch
  • PATCH_NUMBER - numeric part of JIRA ticket (e.g. 589 for the ticket LHCBCNDB-589)
  • DATATYPES_LIST - list of data types the patch is meant for (if many then one has to use comma separated list without spaces. E.g. 2012,2011,test)
Line: 65 to 104
 
  • LOCAL_TAG_NAME - a name of a local tag (e.g. it-20120101)
After an execution of this command line the script will diff the SOURCE with the cumulative database content (at the HEAD of the database in this example), and will schedule the real changes for the commitment (a difference in only one space in a file is enough for the script to think it is really different). The commitment is not done immediately upon the command line execution. The script will provide a commitment summary to a user. At this point one has to cross-check all the settings (especially, the SOURCE and DESTINATION values), and confirm or reject the commitment.
Changed:
<
<
After committing a patch check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser PARTITION <br />
<!-- end SyntaxHighlightingPlugin -->
If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:
<!-- SyntaxHighlightingPlugin -->
pbzip2 -vkc $SQLITEDBPATH/PARTITION.db &gt; SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2 <br />
<!-- end SyntaxHighlightingPlugin -->
This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.
>
>
After committing a patch check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser PARTITION
<!-- end SyntaxHighlightingPlugin -->
If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:
<!-- SyntaxHighlightingPlugin -->
pbzip2 -vkc $SQLITEDBPATH/PARTITION.db &gt; SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2
<!-- end SyntaxHighlightingPlugin -->
This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.
 

Committing CondDB Upgrade patches

Changed:
<
<
The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh):
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest bash --norc setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db <br />
<!-- end SyntaxHighlightingPlugin -->
>
>
The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh):
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest bash --norc
setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db
<!-- end SyntaxHighlightingPlugin -->
  From now on the procedure is the same as in previous section.

Committing CondDB DQFlags patches

Changed:
<
<
Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _DQ_Commit.py -h <br />
<!-- end SyntaxHighlightingPlugin -->
An example of how to commit a DQ patch is as follows:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907 <br />
<!-- end SyntaxHighlightingPlugin -->
Check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBBrowser DQFLAGS <br />
<!-- end SyntaxHighlightingPlugin -->
>
>
Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _DQ_Commit.py -h
<!-- end SyntaxHighlightingPlugin -->
An example of how to commit a DQ patch is as follows:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907
<!-- end SyntaxHighlightingPlugin -->
Check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBBrowser DQFLAGS
<!-- end SyntaxHighlightingPlugin -->
 

Creating a global tag in CondDB

Changed:
<
<
Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _GlobalTag.py -h <br />
<!-- end SyntaxHighlightingPlugin -->
An example of global tagging is as follows:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831 <br />
<!-- end SyntaxHighlightingPlugin -->
>
>
Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _GlobalTag.py -h
<!-- end SyntaxHighlightingPlugin -->
An example of global tagging is as follows:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831
<!-- end SyntaxHighlightingPlugin -->
 

Releasing CondDB

Line: 88 to 158
  The release procedure is simple and involves copying the new SQLite CondDB files to the CondDB (or CondDB Upgrade) gateway which is regularly analyzed by dedicated acron jobs. The publishing frequency is "every time the LHCb Online run is finished" but not more frequently than once per 10 minutes. If that acron job finds the CondDB gateway state changed it publishes the changed files to the web server based public repository. From that moment another machinery comes into action deploying the new file to all release locations: AFS, CVMFS, LHCb Pit.
Changed:
<
<
So, to make the CondDB release set the correct gateway path
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB <br />
<!-- end SyntaxHighlightingPlugin -->
OR, for the case of CondDB Upgrade
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade <br />
<!-- end SyntaxHighlightingPlugin -->
and just copy the files in a safe manner to the gateway:
<!-- SyntaxHighlightingPlugin -->
cp -pfv PATH2NEWSQLITEFILE /PARTITION.db $gateway/db/PARTITON.db-tmp~ diff PATH2NEWSQLITEFILE /PARTITION.db $gateway/db/PARTITON.db-tmp~ cp -pfv PATH2NEWSQLITEFILE /../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~ mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml <br />
<!-- end SyntaxHighlightingPlugin -->
>
>
So, to make the CondDB release set the correct gateway path
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB
<!-- end SyntaxHighlightingPlugin -->
OR, for the case of CondDB Upgrade
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade
<!-- end SyntaxHighlightingPlugin -->
and just copy the files in a safe manner to the gateway:
<!-- SyntaxHighlightingPlugin -->
cp -pfv PATH2NEWSQLITEFILE /PARTITION.db $gateway/db/PARTITON.db-tmp~
diff PATH2NEWSQLITEFILE /PARTITION.db $gateway/db/PARTITON.db-tmp~
cp -pfv PATH2NEWSQLITEFILE /../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~
mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db
mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml
<!-- end SyntaxHighlightingPlugin -->
  As already mentioned above, the publishing acron job is executed every 10 minutes but the actual check whether the gateway state has changed is started only if there is a new LHCb run.
Line: 106 to 191
 

Releasing Oracle CondDB: synchronizing Oracle CondDB from SQLite CondDB

Changed:
<
<
From lxplus5 (still to be validated on lxplus6), prepare the environment with
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb <br />
<!-- end SyntaxHighlightingPlugin -->
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the those partitions (DDDB, LHCBCOND or SIMCOND) which have been modified:
<!-- SyntaxHighlightingPlugin -->
coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB" coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND" coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND" <br />
<!-- end SyntaxHighlightingPlugin -->
If there are new files (nodes) in the replicated changes, then also access permissions have to be updated:
<!-- SyntaxHighlightingPlugin -->
coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader <br />
<!-- end SyntaxHighlightingPlugin -->
>
>
From lxplus5 (still to be validated on lxplus6), prepare the environment with
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
<!-- end SyntaxHighlightingPlugin -->
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the those partitions (DDDB, LHCBCOND or SIMCOND) which have been modified:
<!-- SyntaxHighlightingPlugin -->
coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB"
coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND"
coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND"
<!-- end SyntaxHighlightingPlugin -->
If there are new files (nodes) in the replicated changes, then also access permissions have to be updated:
<!-- SyntaxHighlightingPlugin -->
coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader
coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader
coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader
<!-- end SyntaxHighlightingPlugin -->
  NO-GOs:
  • Never, ever replicate to Oracle the same SQLite database more than once: this may be done by mistake if, e.g., the $SQLITEDBPATH points to previously replicated SQLite files, and will result in corrupted Oracle state.

Revision 232017-07-23 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
Deleted:
<
<
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

 
Changed:
<
<

General information

CondDB Project Management Team

>
>

General information

CondDB Project Management Team

  Current CondDB and CondDB Upgrade management team consists of the following people: Marco Clemencic, Liang Sun.

Limited patches management responsibility for the CondDB Upgrade project is conferred on Sajan Easo.

Changed:
<
<

CondDB operation assistance systems

>
>

Migration to the new Conditions Database Backend (GitCondDB)

For 2017 data taking we introduced the new backend for conditions data (GitCondDB), but kept the old backend (COOL) to ease the migration.

The decommissioning of the COOL backend is planned for the end of 2017, so until then we must keep to two systems in sync, meaning that any new global tag created in any of the two systems must be propagated to the other one.

In this page we describe the procedure for both integrating changes in each system and how to port changes from one system to the other.

Git Conditions Database

The management of contributions to Conditions Databases in Git (https://gitlab.cern.ch/lhcb-conddb) is not much different from the workflow of any regular software project:

  • developers should create merge requests to one of the projects in https://gitlab.cern.ch/lhcb-conddb
  • each merge request should target the branch for the correct data type ("dt-_datatype_" branches)
  • if a merge request should be applied to other data types, the required data types should be specified as labels (will be propagated by CondDB managers via Gitlab cherry-pick)
  • when appropriate, a tag should be created for the appropriate data type, and the data types a tag applies to must be specified in the tag message (not release notes) in the format:
    ---
    datatypes: [2017, 2016]
    ---

COOL Conditions Database

CondDB operation assistance systems

  DBS - CondDB deployment and backup system;
Line: 23 to 41
 

CondDB release preparation procedures

Committing CondDB patches

Changed:
<
<
In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment:
<!-- SyntaxHighlightingPlugin -->
lb-run LHCb/latest CondDBAdmin_Commit.py -h 
<!-- end SyntaxHighlightingPlugin -->
The script can take an input to be committed in two interchangeable forms:
>
>
In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin _Commit.py script, available under the LHCb Project environment:
<!-- SyntaxHighlightingPlugin -->
lb-run LHCb/latest CondDBAdmin _Commit.py -h <br />
<!-- end SyntaxHighlightingPlugin -->
The script can take an input to be committed in two interchangeable forms:
 
  • XML files
  • SQLite database slice
If there are no completely new files in a patch then either the XML input file tree, or the SQLite DB slice internal nodes tree, has to reproduce the internal tree of the destination CondDB SQLite database. Otherwise, the CondDB node which has a wrong/incomplete path will be added as a new node.
Changed:
<
<
The lookup of the destination SQLite database is performed by the script at the $SQLITEDBPATH location. So it is usually a good idea to do echo $SQLITEDBPATH before committing to cross-check the destination. Currently, there are two locations commonly used to commit and test CondDB patches at:
>
>
The lookup of the destination SQLite database is performed by the script at the $SQLITEDBPATH location. So it is usually a good idea to do echo $SQLITEDBPATH before committing to cross-check the destination. Currently, there are two locations commonly used to commit and test CondDB patches at:
 
$LHCBDEV/DBASE/Det/SQLDDDB/db/
Changed:
<
<
$LHCBDEV/DBASE/Det/SQLDDDB_Upgrade/db/
>
>
$LHCBDEV/DBASE/Det/SQLDDDB_Upgrade/db/
 
Changed:
<
<
The first one is the standard SQLite CondDB location which you get when requesting the development environment ( lb-run --dev LHCb/latest). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).
>
>
The first one is the standard SQLite CondDB location which you get when requesting the development environment ( lb-run --dev LHCb/latest). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).
 
Changed:
<
<
An example of a patch commitment looks like:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest bash --norc
echo $SQLITEDBPATH
CondDBAdmin_Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME 
<!-- end SyntaxHighlightingPlugin -->
where
>
>
An example of a patch commitment looks like:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest bash --norc echo $SQLITEDBPATH CondDBAdmin _Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME <br />
<!-- end SyntaxHighlightingPlugin -->
where
 
  • NAME OF REQUESTOR - full name of the person submitted the patch
  • PATCH_NUMBER - numeric part of JIRA ticket (e.g. 589 for the ticket LHCBCNDB-589)
  • DATATYPES_LIST - list of data types the patch is meant for (if many then one has to use comma separated list without spaces. E.g. 2012,2011,test)
  • PATCH DESCRIPTION - short, but meaningful, patch description which will appear on the web CondDB Release Notes portal
  • SOURCE - path to the source of changes containing either XML files, or SQLite DB slice.
Changed:
<
<
  • DESTINATION_PARTITION - a CondDB destination partition. Currently the following ones are available: DDDB, LHCBCOND, SIMCOND (there is also fourth one - DQFLAGS, but it is STRONGLY FORBIDDEN to commit anything to it with the CondDBAdmin_Commit.py script, there is another one to do that)
>
>
  • DESTINATION_PARTITION - a CondDB destination partition. Currently the following ones are available: DDDB, LHCBCOND, SIMCOND (there is also fourth one - DQFLAGS, but it is STRONGLY FORBIDDEN to commit anything to it with the CondDBAdmin _Commit.py script, there is another one to do that)
 
  • HEAD - has to stay the same, which means that the script will compare the new changes provided in SOURCE to the latest previously committed to CondDB content.
  • LOCAL_TAG_NAME - a name of a local tag (e.g. it-20120101)
After an execution of this command line the script will diff the SOURCE with the cumulative database content (at the HEAD of the database in this example), and will schedule the real changes for the commitment (a difference in only one space in a file is enough for the script to think it is really different). The commitment is not done immediately upon the command line execution. The script will provide a commitment summary to a user. At this point one has to cross-check all the settings (especially, the SOURCE and DESTINATION values), and confirm or reject the commitment.
Changed:
<
<
After committing a patch check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser PARTITION 
<!-- end SyntaxHighlightingPlugin -->
If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:
<!-- SyntaxHighlightingPlugin -->
pbzip2 -vkc $SQLITEDBPATH/PARTITION.db > SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2 
<!-- end SyntaxHighlightingPlugin -->
This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.
>
>
After committing a patch check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser PARTITION <br />
<!-- end SyntaxHighlightingPlugin -->
If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:
<!-- SyntaxHighlightingPlugin -->
pbzip2 -vkc $SQLITEDBPATH/PARTITION.db &gt; SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2 <br />
<!-- end SyntaxHighlightingPlugin -->
This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.
 

Committing CondDB Upgrade patches

Changed:
<
<
The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh):
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest bash --norc
setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db 
<!-- end SyntaxHighlightingPlugin -->
>
>
The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh):
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest bash --norc setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db <br />
<!-- end SyntaxHighlightingPlugin -->
  From now on the procedure is the same as in previous section.

Committing CondDB DQFlags patches

Changed:
<
<
Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin_DQ_Commit.py -h 
<!-- end SyntaxHighlightingPlugin -->
An example of how to commit a DQ patch is as follows:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin_DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907 
<!-- end SyntaxHighlightingPlugin -->
Check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBBrowser DQFLAGS 
<!-- end SyntaxHighlightingPlugin -->
>
>
Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _DQ_Commit.py -h <br />
<!-- end SyntaxHighlightingPlugin -->
An example of how to commit a DQ patch is as follows:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907 <br />
<!-- end SyntaxHighlightingPlugin -->
Check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBBrowser DQFLAGS <br />
<!-- end SyntaxHighlightingPlugin -->
 

Creating a global tag in CondDB

Changed:
<
<
Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin_GlobalTag.py -h 
<!-- end SyntaxHighlightingPlugin -->
An example of global tagging is as follows:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin_GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831 
<!-- end SyntaxHighlightingPlugin -->
>
>
Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _GlobalTag.py -h <br />
<!-- end SyntaxHighlightingPlugin -->
An example of global tagging is as follows:
<!-- SyntaxHighlightingPlugin -->
lb-run --dev LHCb/latest CondDBAdmin _GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831 <br />
<!-- end SyntaxHighlightingPlugin -->
 

Releasing CondDB

Line: 102 to 88
  The release procedure is simple and involves copying the new SQLite CondDB files to the CondDB (or CondDB Upgrade) gateway which is regularly analyzed by dedicated acron jobs. The publishing frequency is "every time the LHCb Online run is finished" but not more frequently than once per 10 minutes. If that acron job finds the CondDB gateway state changed it publishes the changed files to the web server based public repository. From that moment another machinery comes into action deploying the new file to all release locations: AFS, CVMFS, LHCb Pit.
Changed:
<
<
So, to make the CondDB release set the correct gateway path
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB 
<!-- end SyntaxHighlightingPlugin -->
OR, for the case of CondDB Upgrade
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade 
<!-- end SyntaxHighlightingPlugin -->
and just copy the files in a safe manner to the gateway:
<!-- SyntaxHighlightingPlugin -->
cp -pfv PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~
diff PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~
cp -pfv PATH2NEWSQLITEFILE/../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~
mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db
mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml 
<!-- end SyntaxHighlightingPlugin -->
>
>
So, to make the CondDB release set the correct gateway path
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB <br />
<!-- end SyntaxHighlightingPlugin -->
OR, for the case of CondDB Upgrade
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade <br />
<!-- end SyntaxHighlightingPlugin -->
and just copy the files in a safe manner to the gateway:
<!-- SyntaxHighlightingPlugin -->
cp -pfv PATH2NEWSQLITEFILE /PARTITION.db $gateway/db/PARTITON.db-tmp~ diff PATH2NEWSQLITEFILE /PARTITION.db $gateway/db/PARTITON.db-tmp~ cp -pfv PATH2NEWSQLITEFILE /../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~ mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml <br />
<!-- end SyntaxHighlightingPlugin -->
  As already mentioned above, the publishing acron job is executed every 10 minutes but the actual check whether the gateway state has changed is started only if there is a new LHCb run.
Line: 133 to 106
 

Releasing Oracle CondDB: synchronizing Oracle CondDB from SQLite CondDB

Changed:
<
<
From lxplus5 (still to be validated on lxplus6), prepare the environment with
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb 
<!-- end SyntaxHighlightingPlugin -->
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the those partitions (DDDB, LHCBCOND or SIMCOND) which have been modified:
<!-- SyntaxHighlightingPlugin -->
coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB"
coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND"
coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND" 
<!-- end SyntaxHighlightingPlugin -->
If there are new files (nodes) in the replicated changes, then also access permissions have to be updated:
<!-- SyntaxHighlightingPlugin -->
coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader
coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader
coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader 
<!-- end SyntaxHighlightingPlugin -->
>
>
From lxplus5 (still to be validated on lxplus6), prepare the environment with
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb <br />
<!-- end SyntaxHighlightingPlugin -->
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the those partitions (DDDB, LHCBCOND or SIMCOND) which have been modified:
<!-- SyntaxHighlightingPlugin -->
coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB" coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND" coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND" <br />
<!-- end SyntaxHighlightingPlugin -->
If there are new files (nodes) in the replicated changes, then also access permissions have to be updated:
<!-- SyntaxHighlightingPlugin -->
coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader <br />
<!-- end SyntaxHighlightingPlugin -->
  NO-GOs:
  • Never, ever replicate to Oracle the same SQLite database more than once: this may be done by mistake if, e.g., the $SQLITEDBPATH points to previously replicated SQLite files, and will result in corrupted Oracle state.
Line: 167 to 127
  In particular, insertion of the new tag and of all its compatibility relationships to other metadata entities, tracked by Ariadne (see the flavor list of all tracked entities at the bottom of the section), into the Ariadne system has to be performed. Below some typical use cases are listed:
Changed:
<
<
  1. Adding a tag WITHOUT compatibility relationships (i.e, incompatible with everything):
    This covers the simplest case of adding a new tag, which is incompatible with everything known by that time. See below the command line on an example of tag TAG_NAME located in DDDB partition:
          ariadne add tag:DDDB:TAG_NAME
    The latter command line will add a standalone 'tag' node to the knowledge graph of the Ariadne system which is not related in any way to other Ariadne nodes. This, though, doesn't prevent to add any relationships to it later on.
  2. Adding a tag WITH compatibility relationships, IDENTICAL to those of another tag:
    If a new tag has to inherit all relationships from an existent tag (which is what is wanted in most of the cases) the following command line will do the job:
          ariadne add tag:DDDB:TAG_NAME =tag:DDDB:ANOTHER_TAG_NAME
    The instruction in the second argument, uses the '=' operator to specify the node where relationships have to be cloned from.
  3. Adding a tag with NEW set of compatibility relationships:
    This is the most non-trivial use case. Let's say one needs to clone all relationships from an existent node, then add relationship to, e.g., Reco20 reco type, and remove another relationship to 2012 detector type. To accomplish this task two extra instructions, using '+' and '-' operators, have to be given:

>
>
  1. Adding a tag WITHOUT compatibility relationships (i.e, incompatible with everything):
    This covers the simplest case of adding a new tag, which is incompatible with everything known by that time. See below the command line on an example of tag TAG_NAME located in DDDB partition:
          ariadne add tag:DDDB:TAG_NAME
    The latter command line will add a standalone 'tag' node to the knowledge graph of the Ariadne system which is not related in any way to other Ariadne nodes. This, though, doesn't prevent to add any relationships to it later on.
  2. Adding a tag WITH compatibility relationships, IDENTICAL to those of another tag:
    If a new tag has to inherit all relationships from an existent tag (which is what is wanted in most of the cases) the following command line will do the job:
          ariadne add tag:DDDB:TAG_NAME =tag:DDDB:ANOTHER_TAG_NAME
    The instruction in the second argument, uses the '=' operator to specify the node where relationships have to be cloned from.
  3. Adding a tag with NEW set of compatibility relationships:
    This is the most non-trivial use case. Let's say one needs to clone all relationships from an existent node, then add relationship to, e.g., Reco20 reco type, and remove another relationship to 2012 detector type. To accomplish this task two extra instructions, using '+' and '-' operators, have to be given:
  ariadne add tag:DDDB:TAG_NAME =tag:DDDB:ANOTHER_TAG_NAME +reco_type:Reco20 -detector_type:2012

Tip, idea To check what are the current relationships of a tag use the following command:

Line: 190 to 144
 
  ariadne q -r tag:DDDB:TAG_NAME | grep WHATEVER
Changed:
<
<
Vision ATTENTION :
One has to always devote extra attention to creating realistic relationships of a new tag with detector-, reco- and/or sim-types. In particular, the following relationships have to be ensured:
>
>

Vision ATTENTION :
One has to always devote extra attention to creating realistic relationships of a new tag with detector-, reco- and/or sim-types. In particular, the following relationships have to be ensured:
 
  • For DDDB tags: relationships with detector-, reco-, sim-types.
  • For LHCBCOND tags: relationships with detector- and reco-types.
  • For SIMCOND tags: relationships with detector-, reco-, sim-types.
  • For DQFLAGS tags: relationships with detector-types (relationships to other entities are not tracked).
These relationships have to be verified even in the quite regular use case of cloning compatibilities from the tag the new one is based on, since, even though the neighboring tags are usually quite similar by content, the new tag can have significantly different appropriation.
Changed:
<
<
Info The following metadata entities are tracked by Ariadne:
>
>

Info The following metadata entities are tracked by Ariadne:
 
      application:NAME:VERSION
      tag:PARTITION:TAGNAME
Line: 206 to 160
  sim_type:NAME detector_type:NAME
Changed:
<
<
Tip, idea It is usually useful to add a -d option to all the command lines above to see what exactly the tool is performing. To see the command-line tool help issue ariadne add -h.
>
>
Tip, idea It is usually useful to add a -d option to all the command lines above to see what exactly the tool is performing. To see the command-line tool help issue ariadne add -h.
 

Adding new CondDB tags to the BookKeeping database

\ No newline at end of file

Revision 222017-06-30 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Line: 8 to 8
 

General information

CondDB Project Management Team

Changed:
<
<
Current CondDB and CondDB Upgrade management team consists of the following people: Marco Clemencic, Illya Shapoval.
>
>
Current CondDB and CondDB Upgrade management team consists of the following people: Marco Clemencic, Liang Sun.
  Limited patches management responsibility for the CondDB Upgrade project is conferred on Sajan Easo.

Revision 212016-05-17 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Line: 23 to 23
 

CondDB release preparation procedures

Committing CondDB patches

Changed:
<
<
In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment:
>
>
In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment:
 
<!-- SyntaxHighlightingPlugin -->
lb-run LHCb/latest CondDBAdmin_Commit.py -h 
<!-- end SyntaxHighlightingPlugin -->
The script can take an input to be committed in two interchangeable forms:
Line: 37 to 37
 $LHCBDEV/DBASE/Det/SQLDDDB_Upgrade/db/
Changed:
<
<
The first one is the standard SQLite CondDB location which you get when requesting the development environment ( SetupProject --dev LHCb). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).
>
>
The first one is the standard SQLite CondDB location which you get when requesting the development environment ( lb-run --dev LHCb/latest). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).
  An example of a patch commitment looks like: %CODE{ "tcsh" }%
Line: 46 to 46
 CondDBAdmin_Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME %ENDCODE% where
  • NAME OF REQUESTOR - full name of the person submitted the patch
Changed:
<
<
  • PATCH_NUMBER - Savannah patch number (e.g. 5493)
>
>
  • PATCH_NUMBER - numeric part of JIRA ticket (e.g. 589 for the ticket LHCBCNDB-589)
 
  • DATATYPES_LIST - list of data types the patch is meant for (if many then one has to use comma separated list without spaces. E.g. 2012,2011,test)
  • PATCH DESCRIPTION - short, but meaningful, patch description which will appear on the web CondDB Release Notes portal
  • SOURCE - path to the source of changes containing either XML files, or SQLite DB slice.
Line: 67 to 67
  The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh): %CODE{ "tcsh" }%
Changed:
<
<
SetupProject LHCb
>
>
lb-run --dev LHCb/latest bash --norc
 setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db %ENDCODE%
Line: 77 to 77
  Prepare the LHCb project environment and check the script %CODE{"tcsh"}%
Changed:
<
<
SetupProject LHCb CondDBAdmin_DQ_Commit.py -h
>
>
lb-run --dev LHCb/latest CondDBAdmin_DQ_Commit.py -h
 %ENDCODE% An example of how to commit a DQ patch is as follows: %CODE{"tcsh"}%
Changed:
<
<
SetupProject --dev LHCb CondDBAdmin_DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907
>
>
lb-run --dev LHCb/latest CondDBAdmin_DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907
 %ENDCODE% Check the result with CondDBBrowser:
Changed:
<
<
%CODE{"tcsh"}% CondDBBrowser DQFLAGS
>
>
%CODE{"tcsh"}% lb-run --dev LHCb/latest CondDBBrowser DQFLAGS
 %ENDCODE%

Creating a global tag in CondDB

Prepare the LHCb project environment and check the script %CODE{"tcsh"}%

Changed:
<
<
SetupProject LHCb CondDBAdmin_GlobalTag.py -h
>
>
lb-run --dev LHCb/latest CondDBAdmin_GlobalTag.py -h
 %ENDCODE% An example of global tagging is as follows: %CODE{"tcsh"}%
Changed:
<
<
SetupProject --dev LHCb CondDBAdmin_GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831
>
>
lb-run --dev LHCb/latest CondDBAdmin_GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831
 %ENDCODE%

Releasing CondDB

Revision 202016-05-17 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Line: 25 to 25
  In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment: %CODE{ "tcsh" }%
Changed:
<
<
SetupProject LHCb CondDBAdmin_Commit.py -h
>
>
lb-run LHCb/latest CondDBAdmin_Commit.py -h
 %ENDCODE% The script can take an input to be committed in two interchangeable forms:
  • XML files
  • SQLite database slice
Line: 42 to 41
  An example of a patch commitment looks like: %CODE{ "tcsh" }%
Changed:
<
<
SetupProject --dev LHCb
>
>
lb-run --dev LHCb/latest bash --norc
 echo $SQLITEDBPATH CondDBAdmin_Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME %ENDCODE% where

Revision 192014-06-04 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Line: 196 to 196
  ariadne q -r tag:DDDB:TAG_NAME | grep WHATEVER
Changed:
<
<
Vision ATTENTION :
Unless there is an intention to create some specific compatibilities of a new tag with applications and other tags one can come to nothing more than cloning the compatibilities from the tag the new one is based on (see use case B). But one has to devote extra attention to creating realistic relationships of a new tag with detector-, reco- and sim-types. In particular, the following relationships have to be ensured:
>
>
Vision ATTENTION :
One has to always devote extra attention to creating realistic relationships of a new tag with detector-, reco- and/or sim-types. In particular, the following relationships have to be ensured:
 
  • For DDDB tags: relationships with detector-, reco-, sim-types.
  • For LHCBCOND tags: relationships with detector- and reco-types.
  • For SIMCOND tags: relationships with detector-, reco-, sim-types.
  • For DQFLAGS tags: relationships with detector-types (relationships to other entities are not tracked).
Added:
>
>
These relationships have to be verified even in the quite regular use case of cloning compatibilities from the tag the new one is based on, since, even though the neighboring tags are usually quite similar by content, the new tag can have significantly different appropriation.
  Info The following metadata entities are tracked by Ariadne:

Revision 182014-06-03 - MarcoClemencic

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Line: 24 to 24
 

Committing CondDB patches

In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment:

Changed:
<
<
%CODE{ "tcsh" }% SetupProject LHCb CondDBAdmin_Commit.py -h
>
>
%CODE{ "tcsh" }% SetupProject LHCb CondDBAdmin_Commit.py -h
 %ENDCODE% The script can take an input to be committed in two interchangeable forms:
  • XML files
  • SQLite database slice
Line: 36 to 38
 $LHCBDEV/DBASE/Det/SQLDDDB_Upgrade/db/
Changed:
<
<
The first one is the standard SQLite CondDB location which you get when requesting the development environment ( SetupProject LHCb --dev). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).
>
>
The first one is the standard SQLite CondDB location which you get when requesting the development environment ( SetupProject --dev LHCb). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).
  An example of a patch commitment looks like:
Changed:
<
<
%CODE{ "tcsh" }% SetupProject LHCb --dev echo $SQLITEDBPATH CondDBAdmin _Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME
>
>
%CODE{ "tcsh" }% SetupProject --dev LHCb echo $SQLITEDBPATH CondDBAdmin_Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME
 %ENDCODE% where
  • NAME OF REQUESTOR - full name of the person submitted the patch
  • PATCH_NUMBER - Savannah patch number (e.g. 5493)
Line: 52 to 57
 After an execution of this command line the script will diff the SOURCE with the cumulative database content (at the HEAD of the database in this example), and will schedule the real changes for the commitment (a difference in only one space in a file is enough for the script to think it is really different). The commitment is not done immediately upon the command line execution. The script will provide a commitment summary to a user. At this point one has to cross-check all the settings (especially, the SOURCE and DESTINATION values), and confirm or reject the commitment.

After committing a patch check the result with CondDBBrowser:

Changed:
<
<
%CODE{"tcsh"}% CondDBBrowser PARTITION
>
>
%CODE{"tcsh"}% CondDBBrowser PARTITION
 %ENDCODE% If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:
Changed:
<
<
%CODE{ "tcsh" }% pbzip2 -vkc $SQLITEDBPATH/PARTITION.db > SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2
>
>
%CODE{ "tcsh" }% pbzip2 -vkc $SQLITEDBPATH/PARTITION.db > SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2
 %ENDCODE% This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.

Committing CondDB Upgrade patches

The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh):

Changed:
<
<
%CODE{ "tcsh" }% SetupProject LHCb setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db
>
>
%CODE{ "tcsh" }% SetupProject LHCb setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db
 %ENDCODE%

From now on the procedure is the same as in previous section.

Line: 68 to 77
 

Committing CondDB DQFlags patches

Prepare the LHCb project environment and check the script

Changed:
<
<
%CODE{"tcsh"}% SetupProject LHCb CondDBAdmin _DQ_Commit.py -h
>
>
%CODE{"tcsh"}% SetupProject LHCb CondDBAdmin_DQ_Commit.py -h
 %ENDCODE% An example of how to commit a DQ patch is as follows:
Changed:
<
<
%CODE{"tcsh"}% SetupProject LHCb --dev CondDBAdmin _DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907
>
>
%CODE{"tcsh"}% SetupProject --dev LHCb CondDBAdmin_DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907
 %ENDCODE% Check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser DQFLAGS 
<!-- end SyntaxHighlightingPlugin -->

Creating a global tag in CondDB

Prepare the LHCb project environment and check the script

Changed:
<
<
%CODE{"tcsh"}% SetupProject LHCb CondDBAdmin _GlobalTag.py -h
>
>
%CODE{"tcsh"}% SetupProject LHCb CondDBAdmin_GlobalTag.py -h
 %ENDCODE% An example of global tagging is as follows:
Changed:
<
<
%CODE{"tcsh"}% SetupProject LHCb --dev CondDBAdmin _GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831
>
>
%CODE{"tcsh"}% SetupProject --dev LHCb CondDBAdmin_GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831
 %ENDCODE%

Releasing CondDB

Line: 91 to 108
 The release procedure is simple and involves copying the new SQLite CondDB files to the CondDB (or CondDB Upgrade) gateway which is regularly analyzed by dedicated acron jobs. The publishing frequency is "every time the LHCb Online run is finished" but not more frequently than once per 10 minutes. If that acron job finds the CondDB gateway state changed it publishes the changed files to the web server based public repository. From that moment another machinery comes into action deploying the new file to all release locations: AFS, CVMFS, LHCb Pit.

So, to make the CondDB release set the correct gateway path

Changed:
<
<
%CODE{ "tcsh" }% set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB
>
>
%CODE{ "tcsh" }% set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB
 %ENDCODE% OR, for the case of CondDB Upgrade
Changed:
<
<
%CODE{ "tcsh" }% set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade
>
>
%CODE{ "tcsh" }% set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade
 %ENDCODE% and just copy the files in a safe manner to the gateway:
Changed:
<
<
%CODE{ "tcsh" }% cp -pfv PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ diff PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ cp -pfv PATH2NEWSQLITEFILE/../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~ mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml
>
>
%CODE{ "tcsh" }% cp -pfv PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ diff PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ cp -pfv PATH2NEWSQLITEFILE/../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~ mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml
 %ENDCODE%

As already mentioned above, the publishing acron job is executed every 10 minutes but the actual check whether the gateway state has changed is started only if there is a new LHCb run.

Line: 115 to 139
 

Releasing Oracle CondDB: synchronizing Oracle CondDB from SQLite CondDB

From lxplus5 (still to be validated on lxplus6), prepare the environment with

Changed:
<
<
%CODE{ "tcsh" }% SetupProject LHCb
>
>
%CODE{ "tcsh" }% SetupProject LHCb
 %ENDCODE% Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the those partitions ( DDDB, LHCBCOND or SIMCOND) which have been modified:
Changed:
<
<
%CODE{ "tcsh" }% coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB" coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND" coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND"
>
>
%CODE{ "tcsh" }% coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB" coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND" coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND"
 %ENDCODE% If there are new files (nodes) in the replicated changes, then also access permissions have to be updated:
Changed:
<
<
%CODE{ "tcsh" }% coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader
>
>
%CODE{ "tcsh" }% coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader
 %ENDCODE%

NO-GOs:

Revision 172014-05-30 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Line: 137 to 137
 

Adding compatibility information to Ariadne

Changed:
<
<
The procedure has to be done from LXPLUS, or any SLC machine with Kerberized CERN login and 'cern-get-sso-cookie' package installed (note that access to Ariadne from SLC5 and earlier is not supported any more). In this step insertion of a new tag and of all its relationships to other Ariadne metadata entities (applications, tags in other partitions, data- and reco-types) into the Ariadne system.
>
>
In this step, the new tag has to be registered in the Ariadne system. The procedure has to be done from LXPLUS, or any SLC machine with Kerberized CERN login and 'cern-get-sso-cookie' package installed (note that access to Ariadne from SLC5 and earlier is not supported any more).
 
Changed:
<
<
Below the most frequent use cases are listed:
>
>
In particular, insertion of the new tag and of all its compatibility relationships to other metadata entities, tracked by Ariadne (see the flavor list of all tracked entities at the bottom of the section), into the Ariadne system has to be performed. Below some typical use cases are listed:
 
  1. Adding a tag WITHOUT compatibility relationships (i.e, incompatible with everything):
Changed:
<
<
This covers the simplest case of adding a new tag, which is incompatible with everything known by that time (see the flavor list of tracked entities at the bottom of the section). See below the command line on an example of tag TAG_NAME located in DDDB partition:
>
>
This covers the simplest case of adding a new tag, which is incompatible with everything known by that time. See below the command line on an example of tag TAG_NAME located in DDDB partition:
 
      ariadne add tag:DDDB:TAG_NAME
The latter command line will add a standalone 'tag' node to the knowledge graph of the Ariadne system which is not related in any way to other Ariadne nodes. This, though, doesn't prevent to add any relationships to it later on.
  1. Adding a tag WITH compatibility relationships, IDENTICAL to those of another tag:
    If a new tag has to inherit all relationships from an existent tag (which is what is wanted in most of the cases) the following command line will do the job:

Changed:
<
<
ariadne add tag:DDDB:TAGNAME =tag:DDDB:ANOTHER_TAG_NAME
>
>
ariadne add tag:DDDB:TAG_NAME =tag:DDDB:ANOTHER_TAG_NAME
  The instruction in the second argument, uses the '=' operator to specify the node where relationships have to be cloned from.
  1. Adding a tag with NEW set of compatibility relationships:
    This is the most non-trivial use case. Let's say one needs to clone all relationships from an existent node, then add relationship to, e.g., Reco20 reco type, and remove another relationship to 2012 detector type. To accomplish this task two extra instructions, using '+' and '-' operators, have to be given:

Changed:
<
<
ariadne add tag:DDDB:TAGNAME =tag:DDDB:ANOTHER_TAG_NAME +reco_type:Reco20 -detector_type:2012
>
>
ariadne add tag:DDDB:TAG_NAME =tag:DDDB:ANOTHER_TAG_NAME +reco_type:Reco20 -detector_type:2012
 
Changed:
<
<
In all the above mentioned examples the following metadata entities may be specified:
>
>
Tip, idea To check what are the current relationships of a tag use the following command:
  ariadne q[uery] tag:DDDB:TAG_NAME

or, in batch:

  ariadne q -r tag:DDDB:TAG_NAME | grep WHATEVER

Vision ATTENTION :
Unless there is an intention to create some specific compatibilities of a new tag with applications and other tags one can come to nothing more than cloning the compatibilities from the tag the new one is based on (see use case B). But one has to devote extra attention to creating realistic relationships of a new tag with detector-, reco- and sim-types. In particular, the following relationships have to be ensured:

  • For DDDB tags: relationships with detector-, reco-, sim-types.
  • For LHCBCOND tags: relationships with detector- and reco-types.
  • For SIMCOND tags: relationships with detector-, reco-, sim-types.
  • For DQFLAGS tags: relationships with detector-types (relationships to other entities are not tracked).

Info The following metadata entities are tracked by Ariadne:

 
      application:NAME:VERSION
      tag:PARTITION:TAGNAME

Revision 162014-05-29 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Line: 135 to 135
  This has to be done only for those CondDB global tags, which introduce changes important for HLT. The latter knowledge, typically, is provided by the patch requester. In any case, this action has to be discussed with the LHCb Computing project leader and the LHCb HLT coordinator. It may happen that, even though a new tag has changes relevant for HLT, putting this new tag in action for HLT may be postponed for whatever reasons.
Changed:
<
<

Adding compatibility relations for the new CondDB tags to the CTS

>
>

Adding compatibility information to Ariadne

The procedure has to be done from LXPLUS, or any SLC machine with Kerberized CERN login and 'cern-get-sso-cookie' package installed (note that access to Ariadne from SLC5 and earlier is not supported any more). In this step insertion of a new tag and of all its relationships to other Ariadne metadata entities (applications, tags in other partitions, data- and reco-types) into the Ariadne system.

Below the most frequent use cases are listed:

  1. Adding a tag WITHOUT compatibility relationships (i.e, incompatible with everything):
    This covers the simplest case of adding a new tag, which is incompatible with everything known by that time (see the flavor list of tracked entities at the bottom of the section). See below the command line on an example of tag TAG_NAME located in DDDB partition:
          ariadne add tag:DDDB:TAG_NAME
    The latter command line will add a standalone 'tag' node to the knowledge graph of the Ariadne system which is not related in any way to other Ariadne nodes. This, though, doesn't prevent to add any relationships to it later on.
  2. Adding a tag WITH compatibility relationships, IDENTICAL to those of another tag:
    If a new tag has to inherit all relationships from an existent tag (which is what is wanted in most of the cases) the following command line will do the job:
          ariadne add tag:DDDB:TAGNAME =tag:DDDB:ANOTHER_TAG_NAME
    The instruction in the second argument, uses the '=' operator to specify the node where relationships have to be cloned from.
  3. Adding a tag with NEW set of compatibility relationships:
    This is the most non-trivial use case. Let's say one needs to clone all relationships from an existent node, then add relationship to, e.g., Reco20 reco type, and remove another relationship to 2012 detector type. To accomplish this task two extra instructions, using '+' and '-' operators, have to be given:
          ariadne add tag:DDDB:TAGNAME =tag:DDDB:ANOTHER_TAG_NAME +reco_type:Reco20 -detector_type:2012

In all the above mentioned examples the following metadata entities may be specified:

      application:NAME:VERSION
      tag:PARTITION:TAGNAME
      reco_type:NAME
      sim_type:NAME
      detector_type:NAME

Tip, idea It is usually useful to add a -d option to all the command lines above to see what exactly the tool is performing. To see the command-line tool help issue ariadne add -h.

 

Adding new CondDB tags to the BookKeeping database

Revision 152013-03-28 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Line: 104 to 104
 
  • Use rsync with caution to copy files to the gateway. There were a couple of silent problems with it for unknown reasons (most probably due to AFS glitches) for large files when destination was not identical to source in the end.
  • Once the new files are ready to be released try to transmit all of them to the gateway in-between the publishing cron sessions such that when cron detects that the gateway state has changed it pushes all of new files in one go. Otherwise, it may happen that, e.g., the release notes file is published alone with the new tag entries but the SQLite file does not contain those tags yet.
Changed:
<
<

Creating flow control files to manually switch on/off SQLite CondDB publishing/updating process

>
>

Creating flow control files to manually switch on/off SQLite CondDB publishing/updating process

 
Changed:
<
<
From time to time, acron jobs check a source directory (default is $LHCBDEV/DBASE/Det/SQLDDDB/) for updates, and publish the changed files under that directory. If you want have some control over the acron jobs, just do as follows:
>
>
CondDB acron pilots check regularly the CondDB SQLite Gateway directory (currently $LHCBHOME/software/SQLiteMaster/SQLDDDB[_Upgrade] ) for modified files, and publish the latter to the CondDB SQLite distribution server. There are following controls over the CondDB acron pilots:
 
Changed:
<
<
  • In order to switch off the publishing process, you need to create a flow control file named ".stopPublishing" using touch command under the source directory.
  • Likewise, in order to enable any changes under the source path to be published, you need to remove the stop control file ".stopPublishing" first if it exists, and create another flow control file named ".startPublishing" (NB: the file ".stopPublishing" will override the file ".startPublishing"). If all updates are successfully published, this control file ".startPublishing" will be deleted automatically.
  • In order to disable the updating process for the SQLite snapshots, a flow control file named ".stopUpdatingSnapshots" need to be placed under the destination directory (Path to SQLDDDB package ONLINE, default is $SQLDDDBROOT).
>
>
  • In order to switch off the publishing process, you need to create a flow control file named ".stopPublishing" using touch command under the gateway directory.
  • Likewise, in order to enable any changes under the gateway path to be published, you need to remove the stop control file ".stopPublishing" first if it exists, and create another flow control file named ".startPublishing" (NB: the file ".stopPublishing" will override the file ".startPublishing"). If all updates are successfully published, this control file ".startPublishing" will be deleted automatically.
  • In order to disable the updating process for the SQLite snapshots, a flow control file named ".stopUpdatingSnapshots" need to be placed under the gateway directory.
 

Releasing Oracle CondDB: synchronizing Oracle CondDB from SQLite CondDB

Revision 142013-03-21 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012
Added:
>
>
 

Welcome to the LHCb Conditions Database management page

Line: 19 to 20
  CTS - CondDB compatibility tracking system.
Deleted:
<
<

CondDB documentation

Here is the list of notes and publications with detailed description of the CondDB structure and operation assistance systems:

Title Author(s) Reference Paper Volume, pp Date
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012 LHCb-PROC-2012-058 ; CERN-LHCb-PROC-2012-058 7 October 2012
Proposal on LHCb Software and Conditions Database Cross-Compatibility Tracking System: a Graph-Theory Approach I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-008 10 June 2012
LHCb Conditions Database Operation Assistance Systems I.Shapoval et al. Proceedings of CHEP2012 LHCb draft 22 May 2012
New deployment model for SQLite databases I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-007 15 April 2012
LHCb Distributed Conditions Database M.Clemencic Proceedings of CHEP2007 LHCb-TALK-2007-054, CERN-LHCb-PROC-2007-023 6 Sep 2007
LHCb Conditions Database M.Clemencic et al. Proceedings of CHEP2006 CERN-LHCb-2006-017 4 May 2006
 

CondDB release preparation procedures

Committing CondDB patches

Revision 132013-03-20 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Revision 122013-03-17 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 23 to 23
  Here is the list of notes and publications with detailed description of the CondDB structure and operation assistance systems:
Title Author(s) Reference Paper Volume, pp Date
Changed:
<
<
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012 LHCb-PROC-2012-058 ; CERN-LHCb-PROC-2012-058 7 October 2012
>
>
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012 LHCb-PROC-2012-058 ; CERN-LHCb-PROC-2012-058 7 October 2012
 
Proposal on LHCb Software and Conditions Database Cross-Compatibility Tracking System: a Graph-Theory Approach I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-008 10 June 2012
LHCb Conditions Database Operation Assistance Systems I.Shapoval et al. Proceedings of CHEP2012 LHCb draft 22 May 2012
New deployment model for SQLite databases I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-007 15 April 2012

Revision 112013-01-11 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 23 to 23
  Here is the list of notes and publications with detailed description of the CondDB structure and operation assistance systems:
Title Author(s) Reference Paper Volume, pp Date
Changed:
<
<
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012 LHCb-PROC-2012-058 ; CERN-LHCb-PROC-2012-058 7 October 2012
>
>
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012 LHCb-PROC-2012-058 ; CERN-LHCb-PROC-2012-058 7 October 2012
 
Proposal on LHCb Software and Conditions Database Cross-Compatibility Tracking System: a Graph-Theory Approach I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-008 10 June 2012
LHCb Conditions Database Operation Assistance Systems I.Shapoval et al. Proceedings of CHEP2012 LHCb draft 22 May 2012
New deployment model for SQLite databases I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-007 15 April 2012

Revision 102012-12-17 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 23 to 23
  Here is the list of notes and publications with detailed description of the CondDB structure and operation assistance systems:
Title Author(s) Reference Paper Volume, pp Date
Changed:
<
<
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012     October 2012
>
>
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012 LHCb-PROC-2012-058 ; CERN-LHCb-PROC-2012-058 7 October 2012
 
Proposal on LHCb Software and Conditions Database Cross-Compatibility Tracking System: a Graph-Theory Approach I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-008 10 June 2012
Changed:
<
<
LHCb Conditions Database Operation Assistance Systems I.Shapoval et al. Proceedings of CHEP2012 LHCb draft 22 May 2012
>
>
LHCb Conditions Database Operation Assistance Systems I.Shapoval et al. Proceedings of CHEP2012 LHCb draft 22 May 2012
 
New deployment model for SQLite databases I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-007 15 April 2012
LHCb Distributed Conditions Database M.Clemencic Proceedings of CHEP2007 LHCb-TALK-2007-054, CERN-LHCb-PROC-2007-023 6 Sep 2007
LHCb Conditions Database M.Clemencic et al. Proceedings of CHEP2006 CERN-LHCb-2006-017 4 May 2006

Revision 92012-12-11 - LiangSun

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 114 to 114
 
  • Use rsync with caution to copy files to the gateway. There were a couple of silent problems with it for unknown reasons (most probably due to AFS glitches) for large files when destination was not identical to source in the end.
  • Once the new files are ready to be released try to transmit all of them to the gateway in-between the publishing cron sessions such that when cron detects that the gateway state has changed it pushes all of new files in one go. Otherwise, it may happen that, e.g., the release notes file is published alone with the new tag entries but the SQLite file does not contain those tags yet.
Changed:
<
<

Creating flow control files to switch on/off SQLite CondDB publishing process

>
>

Creating flow control files to manually switch on/off SQLite CondDB publishing/updating process

 
Changed:
<
<
From time to time, acron jobs check a source directory (default is $LHCBDEV/DBASE/Det/SQLDDDB/) for updates, and publish the changed files under that directory. To switch off this publishing process, you can create a flow control file named ".stopPublishing" using touch command under the source directory. Likewise, in order to enable any changes under the source path to be published, you need to remove the stop control file ".stopPublishing" first if it exists, and create another flow control file named ".startPublishing" (NB: the file ".stopPublishing" will override the file ".startPublishing"). If all updates are successfully published, this control file ".startPublishing" will be deleted automatically.

Creating flow control files to switch off SQLite CondDB updating process

>
>
From time to time, acron jobs check a source directory (default is $LHCBDEV/DBASE/Det/SQLDDDB/) for updates, and publish the changed files under that directory. If you want have some control over the acron jobs, just do as follows:

  • In order to switch off the publishing process, you need to create a flow control file named ".stopPublishing" using touch command under the source directory.
  • Likewise, in order to enable any changes under the source path to be published, you need to remove the stop control file ".stopPublishing" first if it exists, and create another flow control file named ".startPublishing" (NB: the file ".stopPublishing" will override the file ".startPublishing"). If all updates are successfully published, this control file ".startPublishing" will be deleted automatically.
  • In order to disable the updating process for the SQLite snapshots, a flow control file named ".stopUpdatingSnapshots" need to be placed under the destination directory (Path to SQLDDDB package ONLINE, default is $SQLDDDBROOT).
 
Deleted:
<
<
In order to disable the updating process for the SQLite snapshots, a flow control file named ".stopUpdatingSnapshots" need to be placed under the destination directory (Path to SQLDDDB package ONLINE, default is $SQLDDDBROOT).
 

Releasing Oracle CondDB: synchronizing Oracle CondDB from SQLite CondDB

From lxplus5 (still to be validated on lxplus6), prepare the environment with

Revision 82012-12-05 - LiangSun

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 29 to 32
 

CondDB release preparation procedures

Committing CondDB patches

Added:
>
>
 In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment:
Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
CondDBAdmin_Commit.py -h
<!-- end SyntaxHighlightingPlugin -->
The script can take an input to be committed in two interchangeable forms:
>
>
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb CondDBAdmin _Commit.py -h 
<!-- end SyntaxHighlightingPlugin -->
The script can take an input to be committed in two interchangeable forms:
 
  • XML files
  • SQLite database slice
If there are no completely new files in a patch then either the XML input file tree, or the SQLite DB slice internal nodes tree, has to reproduce the internal tree of the destination CondDB SQLite database. Otherwise, the CondDB node which has a wrong/incomplete path will be added as a new node.
Line: 47 to 49
 The first one is the standard SQLite CondDB location which you get when requesting the development environment (SetupProject LHCb --dev). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).

An example of a patch commitment looks like:

Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb --dev
echo $SQLITEDBPATH
CondDBAdmin_Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME
<!-- end SyntaxHighlightingPlugin -->
where
>
>
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb --dev echo $SQLITEDBPATH CondDBAdmin _Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME 
<!-- end SyntaxHighlightingPlugin -->
where
 
  • NAME OF REQUESTOR - full name of the person submitted the patch
  • PATCH_NUMBER - Savannah patch number (e.g. 5493)
  • DATATYPES_LIST - list of data types the patch is meant for (if many then one has to use comma separated list without spaces. E.g. 2012,2011,test)
Line: 65 to 62
 After an execution of this command line the script will diff the SOURCE with the cumulative database content (at the HEAD of the database in this example), and will schedule the real changes for the commitment (a difference in only one space in a file is enough for the script to think it is really different). The commitment is not done immediately upon the command line execution. The script will provide a commitment summary to a user. At this point one has to cross-check all the settings (especially, the SOURCE and DESTINATION values), and confirm or reject the commitment.

After committing a patch check the result with CondDBBrowser:

Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser PARTITION
<!-- end SyntaxHighlightingPlugin -->
If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:
<!-- SyntaxHighlightingPlugin -->
pbzip2 -vkc $SQLITEDBPATH/PARTITION.db > SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2
<!-- end SyntaxHighlightingPlugin -->
This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.
>
>
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser PARTITION 
<!-- end SyntaxHighlightingPlugin -->
If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:
<!-- SyntaxHighlightingPlugin -->
pbzip2 -vkc $SQLITEDBPATH/PARTITION.db &gt; SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2 
<!-- end SyntaxHighlightingPlugin -->
This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.
 

Committing CondDB Upgrade patches

Added:
>
>
 The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh):
Changed:
<
<
%CODE{ "tcsh" }% SetupProject LHCb setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db
>
>
%CODE{ "tcsh" }% SetupProject LHCb setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db
 %ENDCODE%

From now on the procedure is the same as in previous section.

Line: 84 to 76
 From now on the procedure is the same as in previous section.

Committing CondDB DQFlags patches

Added:
>
>
 Prepare the LHCb project environment and check the script
Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
CondDBAdmin_DQ_Commit.py -h
<!-- end SyntaxHighlightingPlugin -->
An example of how to commit a DQ patch is as follows:
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb --dev
CondDBAdmin_DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907
<!-- end SyntaxHighlightingPlugin -->
Check the result with CondDBBrowser: %CODE{"tcsh"}% CondDBBrowser DQFLAGS
>
>
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb CondDBAdmin _DQ_Commit.py -h 
<!-- end SyntaxHighlightingPlugin -->
An example of how to commit a DQ patch is as follows:
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb --dev CondDBAdmin _DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907 
<!-- end SyntaxHighlightingPlugin -->
Check the result with CondDBBrowser: %CODE{"tcsh"}% CondDBBrowser DQFLAGS
 %ENDCODE%

Creating a global tag in CondDB

Added:
>
>
 Prepare the LHCb project environment and check the script
Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
CondDBAdmin_GlobalTag.py -h
<!-- end SyntaxHighlightingPlugin -->
An example of global tagging is as follows: %CODE{"tcsh"}% SetupProject LHCb --dev CondDBAdmin_GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831
>
>
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb CondDBAdmin _GlobalTag.py -h 
<!-- end SyntaxHighlightingPlugin -->
An example of global tagging is as follows: %CODE{"tcsh"}% SetupProject LHCb --dev CondDBAdmin _GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831
 %ENDCODE%

Releasing CondDB

Line: 117 to 101
 The release procedure is simple and involves copying the new SQLite CondDB files to the CondDB (or CondDB Upgrade) gateway which is regularly analyzed by dedicated acron jobs. The publishing frequency is "every time the LHCb Online run is finished" but not more frequently than once per 10 minutes. If that acron job finds the CondDB gateway state changed it publishes the changed files to the web server based public repository. From that moment another machinery comes into action deploying the new file to all release locations: AFS, CVMFS, LHCb Pit.

So, to make the CondDB release set the correct gateway path

Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB
<!-- end SyntaxHighlightingPlugin -->
OR, for the case of CondDB Upgrade
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade
<!-- end SyntaxHighlightingPlugin -->
and just copy the files in a safe manner to the gateway: %CODE{ "tcsh" }% cp -pfv PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ diff PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ cp -pfv PATH2NEWSQLITEFILE/../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~ mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml
>
>
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB 
<!-- end SyntaxHighlightingPlugin -->
OR, for the case of CondDB Upgrade
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade 
<!-- end SyntaxHighlightingPlugin -->
and just copy the files in a safe manner to the gateway: %CODE{ "tcsh" }% cp -pfv PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ diff PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ cp -pfv PATH2NEWSQLITEFILE/../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~ mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml
 %ENDCODE%

As already mentioned above, the publishing acron job is executed every 10 minutes but the actual check whether the gateway state has changed is started only if there is a new LHCb run.

Line: 139 to 114
 
  • Use rsync with caution to copy files to the gateway. There were a couple of silent problems with it for unknown reasons (most probably due to AFS glitches) for large files when destination was not identical to source in the end.
  • Once the new files are ready to be released try to transmit all of them to the gateway in-between the publishing cron sessions such that when cron detects that the gateway state has changed it pushes all of new files in one go. Otherwise, it may happen that, e.g., the release notes file is published alone with the new tag entries but the SQLite file does not contain those tags yet.
Added:
>
>

Creating flow control files to switch on/off SQLite CondDB publishing process

From time to time, acron jobs check a source directory (default is $LHCBDEV/DBASE/Det/SQLDDDB/) for updates, and publish the changed files under that directory. To switch off this publishing process, you can create a flow control file named ".stopPublishing" using touch command under the source directory. Likewise, in order to enable any changes under the source path to be published, you need to remove the stop control file ".stopPublishing" first if it exists, and create another flow control file named ".startPublishing" (NB: the file ".stopPublishing" will override the file ".startPublishing"). If all updates are successfully published, this control file ".startPublishing" will be deleted automatically.

Creating flow control files to switch off SQLite CondDB updating process

In order to disable the updating process for the SQLite snapshots, a flow control file named ".stopUpdatingSnapshots" need to be placed under the destination directory (Path to SQLDDDB package ONLINE, default is $SQLDDDBROOT).

 

Releasing Oracle CondDB: synchronizing Oracle CondDB from SQLite CondDB

Added:
>
>
 From lxplus5 (still to be validated on lxplus6), prepare the environment with
Changed:
<
<
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
<!-- end SyntaxHighlightingPlugin -->
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the those partitions (DDDB, LHCBCOND or SIMCOND) which have been modified:
<!-- SyntaxHighlightingPlugin -->
coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB"
coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND"
coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND"
<!-- end SyntaxHighlightingPlugin -->
If there are new files (nodes) in the replicated changes, then also access permissions have to be updated: %CODE{ "tcsh" }% coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader
>
>
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb 
<!-- end SyntaxHighlightingPlugin -->
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the those partitions ( DDDB, LHCBCOND or SIMCOND) which have been modified:
<!-- SyntaxHighlightingPlugin -->
coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB" coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND" coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND" 
<!-- end SyntaxHighlightingPlugin -->
If there are new files (nodes) in the replicated changes, then also access permissions have to be updated: %CODE{ "tcsh" }% coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader
 %ENDCODE%

NO-GOs:

Line: 166 to 140
 New tags have to be passed to the integrity tracking system in order to let it track their deployment and integrity status.

Activating new global tags for the LHCb HLT

Changed:
<
<
This has to be done only for those CondDB global tags, which introduce changes important for HLT. The latter knowledge, typically, is provided by the patch requester. In any case, this action has to be discussed with the LHCb Computing project leader and the LHCb HLT coordinator. It may happen that, even though a new tag has changes relevant for HLT, putting this new tag in action for HLT may be postponed for whatever reasons.
>
>
This has to be done only for those CondDB global tags, which introduce changes important for HLT. The latter knowledge, typically, is provided by the patch requester. In any case, this action has to be discussed with the LHCb Computing project leader and the LHCb HLT coordinator. It may happen that, even though a new tag has changes relevant for HLT, putting this new tag in action for HLT may be postponed for whatever reasons.
 

Adding compatibility relations for the new CondDB tags to the CTS

Revision 72012-10-08 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 144 to 144
 
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
<!-- end SyntaxHighlightingPlugin -->
Changed:
<
<
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the three partitions DDDB, LHCBCOND and SIMCOND
>
>
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the those partitions (DDDB, LHCBCOND or SIMCOND) which have been modified:
 %CODE{ "tcsh" }% coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB" coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND"

Revision 62012-09-10 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 78 to 78
 The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh): %CODE{ "tcsh" }% SetupProject LHCb
Changed:
<
<
setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db/
>
>
setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db
 %ENDCODE%

From now on the procedure is the same as in previous section.

Revision 52012-09-09 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 10 to 10
  Limited patches management responsibility for the CondDB Upgrade project is conferred on Sajan Easo.
Added:
>
>

CondDB operation assistance systems

DBS - CondDB deployment and backup system;

ITS - CondDB integrity tracking system;

CTS - CondDB compatibility tracking system.

 

CondDB documentation

Here is the list of notes and publications with detailed description of the CondDB structure and operation assistance systems:
Title Author(s) Reference Paper Volume, pp Date
Line: 17 to 24
 
Proposal on LHCb Software and Conditions Database Cross-Compatibility Tracking System: a Graph-Theory Approach I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-008 10 June 2012
LHCb Conditions Database Operation Assistance Systems I.Shapoval et al. Proceedings of CHEP2012 LHCb draft 22 May 2012
New deployment model for SQLite databases I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-007 15 April 2012
Changed:
<
<
LHCb Distributed Conditions Database M.Clemencic Proceeding of CHEP2007 LHCb-TALK-2007-054, CERN-LHCb-PROC-2007-023 6 Sep 2007
LHCb Conditions Database M.Clemencic et al. Proceeding of CHEP2006 CERN-LHCb-2006-017 4 May 2006
>
>
LHCb Distributed Conditions Database M.Clemencic Proceedings of CHEP2007 LHCb-TALK-2007-054, CERN-LHCb-PROC-2007-023 6 Sep 2007
LHCb Conditions Database M.Clemencic et al. Proceedings of CHEP2006 CERN-LHCb-2006-017 4 May 2006
 

CondDB release preparation procedures

Committing CondDB patches

Line: 49 to 56
 
  • NAME OF REQUESTOR - full name of the person submitted the patch
  • PATCH_NUMBER - Savannah patch number (e.g. 5493)
  • DATATYPES_LIST - list of data types the patch is meant for (if many then one has to use comma separated list without spaces. E.g. 2012,2011,test)
Changed:
<
<
  • PATCH DESCRIPTION - short, but meaningful, patch description which will appear on the web CondDB Release Notes portal
>
>
  • PATCH DESCRIPTION - short, but meaningful, patch description which will appear on the web CondDB Release Notes portal
 
  • SOURCE - path to the source of changes containing either XML files, or SQLite DB slice.
  • DESTINATION_PARTITION - a CondDB destination partition. Currently the following ones are available: DDDB, LHCBCOND, SIMCOND (there is also fourth one - DQFLAGS, but it is STRONGLY FORBIDDEN to commit anything to it with the CondDBAdmin_Commit.py script, there is another one to do that)
  • HEAD - has to stay the same, which means that the script will compare the new changes provided in SOURCE to the latest previously committed to CondDB content.
Line: 104 to 111
 %ENDCODE%

Releasing CondDB

Changed:
<
<
To make a CondDB release all procedures, mentioned below, have to be completed in the order stated below.
>
>
To make a CondDB release a set of procedures has to be completed in the order stated below.
 

Releasing SQLite CondDB

The release procedure is simple and involves copying the new SQLite CondDB files to the CondDB (or CondDB Upgrade) gateway which is regularly analyzed by dedicated acron jobs. The publishing frequency is "every time the LHCb Online run is finished" but not more frequently than once per 10 minutes. If that acron job finds the CondDB gateway state changed it publishes the changed files to the web server based public repository. From that moment another machinery comes into action deploying the new file to all release locations: AFS, CVMFS, LHCb Pit.
Line: 137 to 144
 
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
<!-- end SyntaxHighlightingPlugin -->
Changed:
<
<
Change directory to location where authentication.xml and dblookup.xml files are located (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the three partitions DDDB, LHCBCOND and SIMCOND
>
>
Change directory to location with authentication.xml and dblookup.xml files present (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the three partitions DDDB, LHCBCOND and SIMCOND
 
<!-- SyntaxHighlightingPlugin -->
coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB"
coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND"
coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND"
<!-- end SyntaxHighlightingPlugin -->
Changed:
<
<
If there are new files in the propagated changes, then also the permissions have to be updated:
>
>
If there are new files (nodes) in the replicated changes, then also access permissions have to be updated:
 %CODE{ "tcsh" }% coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader
Line: 154 to 161
 
  • Never, ever replicate to Oracle the same SQLite database more than once: this may be done by mistake if, e.g., the $SQLITEDBPATH points to previously replicated SQLite files, and will result in corrupted Oracle state.
  • Never, ever modify/re-apply already replicated to Oracle tags (either local, or global ones). Apart from that this is never allowed from the point of view of LHCb Productions this attempt will result in failed replication and corrupted Oracle state.
Added:
>
>

Adding new CondDB tags to ITS

New tags have to be passed to the integrity tracking system in order to let it track their deployment and integrity status.

Activating new global tags for the LHCb HLT

This has to be done only for those CondDB global tags, which introduce changes important for HLT. The latter knowledge, typically, is provided by the patch requester. In any case, this action has to be discussed with the LHCb Computing project leader and the LHCb HLT coordinator. It may happen that, even though a new tag has changes relevant for HLT, putting this new tag in action for HLT may be postponed for whatever reasons.

Adding compatibility relations for the new CondDB tags to the CTS

Adding new CondDB tags to the BookKeeping database

Revision 42012-09-07 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 10 to 10
  Limited patches management responsibility for the CondDB Upgrade project is conferred on Sajan Easo.
Changed:
<
<

CondDB Publications

>
>

CondDB documentation

Here is the list of notes and publications with detailed description of the CondDB structure and operation assistance systems:
 
Title Author(s) Reference Paper Volume, pp Date
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012     October 2012
Proposal on LHCb Software and Conditions Database Cross-Compatibility Tracking System: a Graph-Theory Approach I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-008 10 June 2012
Line: 22 to 23
 

CondDB release preparation procedures

Committing CondDB patches

In order to commit the CondDB (or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment:
Changed:
<
<
>
>
%CODE{ "tcsh" }%
 SetupProject LHCb CondDBAdmin_Commit.py -h
Changed:
<
<
>
>
%ENDCODE%
 The script can take an input to be committed in two interchangeable forms:
  • XML files
  • SQLite database slice
Line: 39 to 40
 The first one is the standard SQLite CondDB location which you get when requesting the development environment (SetupProject LHCb --dev). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).

An example of a patch commitment looks like:

Changed:
<
<
>
>
%CODE{ "tcsh" }%
 SetupProject LHCb --dev echo $SQLITEDBPATH CondDBAdmin_Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME
Changed:
<
<
>
>
%ENDCODE%
 where
  • NAME OF REQUESTOR - full name of the person submitted the patch
  • PATCH_NUMBER - Savannah patch number (e.g. 5493)
Line: 56 to 57
  After an execution of this command line the script will diff the SOURCE with the cumulative database content (at the HEAD of the database in this example), and will schedule the real changes for the commitment (a difference in only one space in a file is enough for the script to think it is really different). The commitment is not done immediately upon the command line execution. The script will provide a commitment summary to a user. At this point one has to cross-check all the settings (especially, the SOURCE and DESTINATION values), and confirm or reject the commitment.
Added:
>
>
After committing a patch check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser PARTITION
<!-- end SyntaxHighlightingPlugin -->
 If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:
Changed:
<
<
>
>
%CODE{ "tcsh" }%
 pbzip2 -vkc $SQLITEDBPATH/PARTITION.db > SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2
Changed:
<
<
>
>
%ENDCODE%
 This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.

Committing CondDB Upgrade patches

The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh):
Changed:
<
<
>
>
%CODE{ "tcsh" }%
 SetupProject LHCb setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db/
Changed:
<
<
>
>
%ENDCODE%
  From now on the procedure is the same as in previous section.
Added:
>
>

Committing CondDB DQFlags patches

Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
CondDBAdmin_DQ_Commit.py -h
<!-- end SyntaxHighlightingPlugin -->
An example of how to commit a DQ patch is as follows:
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb --dev
CondDBAdmin_DQ_Commit.py -c "Marco Adinolfi" -P 9999 -m "VELO flag set as 'BAD' during [2012-08-13_01:19:00, 2012-08-13_06:58:42)." VELO 1 -s 2012-08-13_01:19:00 -u 2012-08-13_06:58:42 velo-20120907
<!-- end SyntaxHighlightingPlugin -->
Check the result with CondDBBrowser:
<!-- SyntaxHighlightingPlugin -->
CondDBBrowser DQFLAGS
<!-- end SyntaxHighlightingPlugin -->

Creating a global tag in CondDB

Prepare the LHCb project environment and check the script
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
CondDBAdmin_GlobalTag.py -h
<!-- end SyntaxHighlightingPlugin -->
An example of global tagging is as follows:
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb --dev
CondDBAdmin_GlobalTag.py -c "Illya Shapoval" -d 2012,HLT LHCBCOND cond-20120831 cond-20120829 rich-20120831-AerogelCalib rich-20120831-MirrorAlign it-20120831
<!-- end SyntaxHighlightingPlugin -->
 

Releasing CondDB

Added:
>
>
To make a CondDB release all procedures, mentioned below, have to be completed in the order stated below.

Releasing SQLite CondDB

 The release procedure is simple and involves copying the new SQLite CondDB files to the CondDB (or CondDB Upgrade) gateway which is regularly analyzed by dedicated acron jobs. The publishing frequency is "every time the LHCb Online run is finished" but not more frequently than once per 10 minutes. If that acron job finds the CondDB gateway state changed it publishes the changed files to the web server based public repository. From that moment another machinery comes into action deploying the new file to all release locations: AFS, CVMFS, LHCb Pit.

So, to make the CondDB release set the correct gateway path

Changed:
<
<
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB/
or
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade/
>
>
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB
<!-- end SyntaxHighlightingPlugin -->
OR, for the case of CondDB Upgrade
<!-- SyntaxHighlightingPlugin -->
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade
<!-- end SyntaxHighlightingPlugin -->
 and just copy the files in a safe manner to the gateway:
Changed:
<
<
cp -afv PATH2NEWSQLITEFILE/PARTITION.db  $gateway/db/PARTITON.db-tmp~
cp -afv PATH2NEWSQLITEFILE/../doc/release_notes.xml  $gateway/doc/release_notes.xml-tmp~
>
>
%CODE{ "tcsh" }% cp -pfv PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ diff PATH2NEWSQLITEFILE/PARTITION.db $gateway/db/PARTITON.db-tmp~ cp -pfv PATH2NEWSQLITEFILE/../doc/release_notes.xml $gateway/doc/release_notes.xml-tmp~
 mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml
Changed:
<
<
>
>
%ENDCODE%

As already mentioned above, the publishing acron job is executed every 10 minutes but the actual check whether the gateway state has changed is started only if there is a new LHCb run.

CAUTIONs:

  • Use rsync with caution to copy files to the gateway. There were a couple of silent problems with it for unknown reasons (most probably due to AFS glitches) for large files when destination was not identical to source in the end.
  • Once the new files are ready to be released try to transmit all of them to the gateway in-between the publishing cron sessions such that when cron detects that the gateway state has changed it pushes all of new files in one go. Otherwise, it may happen that, e.g., the release notes file is published alone with the new tag entries but the SQLite file does not contain those tags yet.

Releasing Oracle CondDB: synchronizing Oracle CondDB from SQLite CondDB

From lxplus5 (still to be validated on lxplus6), prepare the environment with
<!-- SyntaxHighlightingPlugin -->
SetupProject LHCb
<!-- end SyntaxHighlightingPlugin -->
Change directory to location where authentication.xml and dblookup.xml files are located (the ones needed to connect to CERN Oracle CondDB with read/write privileges). Then ensure you are going to use correct SQLite source (echo $SQLITEDBPATH) and call the coolReplicateDB tool on the three partitions DDDB, LHCBCOND and SIMCOND
<!-- SyntaxHighlightingPlugin -->
coolReplicateDB sqlite_file:$SQLITEDBPATH/DDDB.db/DDDB "CondDB(owner)/DDDB"
coolReplicateDB sqlite_file:$SQLITEDBPATH/LHCBCOND.db/LHCBCOND "CondDB(owner)/LHCBCOND"
coolReplicateDB sqlite_file:$SQLITEDBPATH/SIMCOND.db/SIMCOND "CondDB(owner)/SIMCOND"
<!-- end SyntaxHighlightingPlugin -->
If there are new files in the propagated changes, then also the permissions have to be updated:
<!-- SyntaxHighlightingPlugin -->
coolPrivileges "CondDB(owner)/DDDB" GRANT READER lhcb_conddb_reader
coolPrivileges "CondDB(owner)/LHCBCOND" GRANT READER lhcb_conddb_reader
coolPrivileges "CondDB(owner)/SIMCOND" GRANT READER lhcb_conddb_reader
<!-- end SyntaxHighlightingPlugin -->

NO-GOs:

  • Never, ever replicate to Oracle the same SQLite database more than once: this may be done by mistake if, e.g., the $SQLITEDBPATH points to previously replicated SQLite files, and will result in corrupted Oracle state.
  • Never, ever modify/re-apply already replicated to Oracle tags (either local, or global ones). Apart from that this is never allowed from the point of view of LHCb Productions this attempt will result in failed replication and corrupted Oracle state.

Revision 32012-08-22 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

Line: 24 to 24
 In order to commit the CondDB ( or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment:
SetupProject LHCb
Changed:
<
<
CondDBAdmin_commit.py -h
>
>
CondDBAdmin_Commit.py -h
 
Changed:
<
<
The script can take the input changes to be committed in two equivalent forms:
  • The XML files
  • The SQLite database slice
If there are no new files in a patch either the input file tree, or the SQLite DB slice nodes tree, has to reproduce the destination tree of the master CondDB database. Otherwise, the CondDB node which has a wrong path will be treated as a new node.
>
>
The script can take an input to be committed in two interchangeable forms:
  • XML files
  • SQLite database slice
If there are no completely new files in a patch then either the XML input file tree, or the SQLite DB slice internal nodes tree, has to reproduce the internal tree of the destination CondDB SQLite database. Otherwise, the CondDB node which has a wrong/incomplete path will be added as a new node.
 
Added:
>
>
The lookup of the destination SQLite database is performed by the script at the $SQLITEDBPATH location. So it is usually a good idea to do echo $SQLITEDBPATH before committing to cross-check the destination. Currently, there are two locations commonly used to commit and test CondDB patches at:
$LHCBDEV/DBASE/Det/SQLDDDB/db/
$LHCBDEV/DBASE/Det/SQLDDDB_Upgrade/db/
The first one is the standard SQLite CondDB location which you get when requesting the development environment (SetupProject LHCb --dev). While the second one, as it is easy to see, is for the SQLite CondDB Upgrade files (see next section).

An example of a patch commitment looks like:

SetupProject LHCb --dev
echo $SQLITEDBPATH
CondDBAdmin_Commit.py -c "NAME OF REQUESTOR" -P PATCH_NUMBER -t DATATYPES_LIST -m "PATCH DESCRIPTION" SOURCE DESTINATION_PARTITION HEAD LOCAL_TAG_NAME
where
  • NAME OF REQUESTOR - full name of the person submitted the patch
  • PATCH_NUMBER - Savannah patch number (e.g. 5493)
  • DATATYPES_LIST - list of data types the patch is meant for (if many then one has to use comma separated list without spaces. E.g. 2012,2011,test)
  • PATCH DESCRIPTION - short, but meaningful, patch description which will appear on the web CondDB Release Notes portal
  • SOURCE - path to the source of changes containing either XML files, or SQLite DB slice.
  • DESTINATION_PARTITION - a CondDB destination partition. Currently the following ones are available: DDDB, LHCBCOND, SIMCOND (there is also fourth one - DQFLAGS, but it is STRONGLY FORBIDDEN to commit anything to it with the CondDBAdmin_Commit.py script, there is another one to do that)
  • HEAD - has to stay the same, which means that the script will compare the new changes provided in SOURCE to the latest previously committed to CondDB content.
  • LOCAL_TAG_NAME - a name of a local tag (e.g. it-20120101)

After an execution of this command line the script will diff the SOURCE with the cumulative database content (at the HEAD of the database in this example), and will schedule the real changes for the commitment (a difference in only one space in a file is enough for the script to think it is really different). The commitment is not done immediately upon the command line execution. The script will provide a commitment summary to a user. At this point one has to cross-check all the settings (especially, the SOURCE and DESTINATION values), and confirm or reject the commitment.

If changes are committed and you are sure everything is correct it is advisable to create a compressed backup of the destination SQLite file, e.g.:

pbzip2 -vkc $SQLITEDBPATH/PARTITION.db > SOME_BACKUP_LOCATION/PARTITION+LOCAL_TAG_NAME.db.bz2
This may be useful in cases when the subsequent commits fail: you will be able then to rollback to whatever backed up CondDB state.

Committing CondDB Upgrade patches

The procedure is the same as for the regular CondDB patches but with one difference. You have to change the cumulative SQLite CondDB files destination location by hand. So the environment preparation will be (in tcsh):
SetupProject LHCb
setenv SQLITEDBPATH /afs/cern.ch/lhcb/software/DEV/DBASE/Det/SQLDDDB_Upgrade/db/

From now on the procedure is the same as in previous section.

 

Releasing CondDB

Added:
>
>
The release procedure is simple and involves copying the new SQLite CondDB files to the CondDB (or CondDB Upgrade) gateway which is regularly analyzed by dedicated acron jobs. The publishing frequency is "every time the LHCb Online run is finished" but not more frequently than once per 10 minutes. If that acron job finds the CondDB gateway state changed it publishes the changed files to the web server based public repository. From that moment another machinery comes into action deploying the new file to all release locations: AFS, CVMFS, LHCb Pit.

So, to make the CondDB release set the correct gateway path

set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB/
or
set gateway=/afs/cern.ch/lhcb/software/SQLiteMaster/SQLDDDB_Upgrade/
and just copy the files in a safe manner to the gateway:
cp -afv PATH2NEWSQLITEFILE/PARTITION.db  $gateway/db/PARTITON.db-tmp~
cp -afv PATH2NEWSQLITEFILE/../doc/release_notes.xml  $gateway/doc/release_notes.xml-tmp~
mv -f $gateway/db/PARTITON.db-tmp~ $gateway/db/PARTITON.db
mv -f $gateway/doc/release_notes.xml-tmp~ $gateway/doc/release_notes.xml

Revision 22012-08-21 - IllyaShapoval

Line: 1 to 1
 
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

General information

Changed:
<
<

CondDB Project Management Team

CondDB Publications

>
>

CondDB Project Management Team

Current CondDB and CondDB Upgrade management team consists of the following people: Marco Clemencic, Illya Shapoval.
 
Added:
>
>
Limited patches management responsibility for the CondDB Upgrade project is conferred on Sajan Easo.

CondDB Publications

 
Title Author(s) Reference Paper Volume, pp Date
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012     October 2012
Proposal on LHCb Software and Conditions Database Cross-Compatibility Tracking System: a Graph-Theory Approach I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-008 10 June 2012
Line: 16 to 19
 
LHCb Distributed Conditions Database M.Clemencic Proceeding of CHEP2007 LHCb-TALK-2007-054, CERN-LHCb-PROC-2007-023 6 Sep 2007
LHCb Conditions Database M.Clemencic et al. Proceeding of CHEP2006 CERN-LHCb-2006-017 4 May 2006
Added:
>
>

CondDB release preparation procedures

Committing CondDB patches

In order to commit the CondDB ( or CondDB Upgrade) patches to the master CondDB (or CondDB Upgrade) databases one has to use the CondDBAdmin_Commit.py script, available under the LHCb Project environment:
SetupProject LHCb
CondDBAdmin_commit.py -h
The script can take the input changes to be committed in two equivalent forms:
  • The XML files
  • The SQLite database slice
If there are no new files in a patch either the input file tree, or the SQLite DB slice nodes tree, has to reproduce the destination tree of the master CondDB database. Otherwise, the CondDB node which has a wrong path will be treated as a new node.
 
Changed:
<
<

CondDB release preparation procedures

Committing CondDB patches

Releasing CondDB

>
>

Releasing CondDB

Revision 12012-08-21 - IllyaShapoval

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="LHCbComputing"
-- IllyaShapoval - 21-Aug-2012

Welcome to the LHCb Conditions Database management page

General information

CondDB Project Management Team

CondDB Publications

Title Author(s) Reference Paper Volume, pp Date
LHCb Software and Conditions Database Cross-Compatibility Tracking: a Graph-Theory Approach I.Shapoval et al. Proceedings of IEEE NSS 2012     October 2012
Proposal on LHCb Software and Conditions Database Cross-Compatibility Tracking System: a Graph-Theory Approach I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-008 10 June 2012
LHCb Conditions Database Operation Assistance Systems I.Shapoval et al. Proceedings of CHEP2012 LHCb draft 22 May 2012
New deployment model for SQLite databases I.Shapoval et al. LHCb Note CERN-LHCb-PUB-2012-007 15 April 2012
LHCb Distributed Conditions Database M.Clemencic Proceeding of CHEP2007 LHCb-TALK-2007-054, CERN-LHCb-PROC-2007-023 6 Sep 2007
LHCb Conditions Database M.Clemencic et al. Proceeding of CHEP2006 CERN-LHCb-2006-017 4 May 2006

CondDB release preparation procedures

Committing CondDB patches

Releasing CondDB

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback