1 Persistency software release validation

This page describes how to run the CORAL and COOL tests to validate a production release installed on AFS or cvmfs. It was created to better document the latest release process since the move from CMT to cmake (see CORALCOOL-2896).

With respect to the previous release validation process using CMT (and dating back to 2013 approximately) described here, a few important differences apply.

  • The validation procedure is performed (not on a systematic basis) by the CORAL/COOL team. For a brief period in the past it had been the responsability of the SPI team, who now only take car of release builds and installations.
  • The validation procedure does not require any write access to AFS (or cvmfs). The CORAL/COOL functional tests are executed in /tmp using the software installed on AFS or cvmfs.
  • The validation procedure no longer includes the dedicated CoralServer test suite (see CORALCOOL-2905 and CORALCOOL-2906). It only includes the CORAL/COOL functional tests.

Note that the validation procedure can be used for any complete installation of the CORAL or COOL software. This includes nightly builds installed on AFS (eventually on cvmfs) and also private development builds, not only production releases. It is worth noting the following subtle differences.

  • As mentioned below, some changes were applied to trunk after LCG85, which is currently the latest installed production release. For this reason, the procedure describe below is only fully valid on nightly builds and private builds, as well as on any future production releases LCG86 and above.
  • The validation procedure involves a full checkout of test logs in an area where new logs can be written. Production releases and nightly builds are installed on read-only areas of AFS and cvmfs, while development builds are installed on directories where developers have write access. For this reason the test suite scripts check out test logs under /tmp for production releases and nightly builds, and in the local project area for development builds.
  • Production releases and nightly builds are built from code that cannot be modified until the next release or nightly build. The procedure described in the "Fix the code and rerun tests" section below only applies to privae development builds.

1.1 Prerequisite: database account configuration

As described in PersistencyTests, access to Oracle and MySQL accounts and to properly configured Frontier and CORAL servers is needed to run the CORAL and COOL tests. On the client side, two files authentication.xml and dblookup.xml containing the relevant parameters must be available, in directories indicated by the CORAL_AUTH_PATH and CORAL_DBLOOKUP_PATH, respectively.

The database servers and accounts used for the nightly tests executed as Linux user sftnight are described in PersistencyTestServers. To avoid clashes between manually executed release validation tests and automatically executed nightly tests, it is better to use two separate sets of accounts for these tests.

In the following, CORAL_AUTH_PATH and CORAL_DBLOOKUP_PATH are set to point to /afs/cern.ch/sw/lcg/app/releases/CORAL/internal/private, where two minimal XML files used by members of the development team have been specially prepared. This directory is only readable by selected members of the development team. For simplicity, all database aliases are named after the Linux user sftnight, and for this reason it is necessary to set COOL_QMTEST_USER=sftnight before executing the COOL tests (see CORALCOOL-2888), but these are not the same accounts as those used by sftnight in the nightlies.

It should be noted that read access to /afs/cern.ch/sw/lcg/app/pool/db is no longer needed to execute the CORAL network glitch test (CORALCOOL-2929). Previously this path was hardcoded in the test, but it is now possible to run all tests with appropriate authentication.xml and dblookup.xml from any user-defined path. The change was implemented after LCG85, so read access to this directory is necessary to perform the validation of the LCG85 release.

1.2 Execute the CORAL and COOL tests for a given platform

The following is an example showing how to execute the CORAL and COOL tests on an SLC6 machine. Note that the CORAL and COOL tests for the same platform can be executed in parallel as they use different sets of tables, even when they use the same database accounts.

This procedure is only available and fully functional as of LCG_85, while older releases may be affected by some runtime configuration issues due to lcgcmake.


export CORAL_AUTH_PATH=/afs/cern.ch/sw/lcg/app/releases/CORAL/internal/private
export CORAL_DBLOOKUP_PATH=/afs/cern.ch/sw/lcg/app/releases/CORAL/internal/private


export CORAL_AUTH_PATH=/afs/cern.ch/sw/lcg/app/releases/CORAL/internal/private
export CORAL_DBLOOKUP_PATH=/afs/cern.ch/sw/lcg/app/releases/CORAL/internal/private
export COOL_QMTEST_USER=sftnight

Before launching the tests, the scripts above check out from SVN the most recent test logs into /tmp/${USER}/LCG_85/CORAL/3_1_5/logs and /tmp/${USER}/LCG_85/COOL/3_1_5/logs, respectively. The results of the tests will then be available in those directories, where they can be analyzed as described in the following subsections.

1.3 Analyse test results and commit them to SVN for reference

A short summary of test results is printed out by the scripts above when they terminate. The test suite should be considered successful when all individual tests either PASS or are UNTESTED, but there should be no tests that FAIL. As an example, the results of the CORAL 3.1.5 test suite in LCG85 are the following:

--- STATISTICS ---------------------------------------------------------------
     282        tests total
      10 (  4%) tests FAIL
     146 ( 52%) tests UNTESTED
     126 ( 45%) tests PASS

After executing the test suites, the following script should be launched to produce dumps of the full test output in xml format (x86_64-slc6-gcc49-opt.xml) and detailed test summaries (x86_64-slc6-gcc49-opt.summary) from the raw qmtest dumps (x86_64-slc6-gcc49-opt.qmr). The xml and summary files can then be committed to SVN for reference, while the qmr file can be discarded.

cd /tmp/${USER}/LCG_85/CORAL/3_1_5/logs/qmtestCoral/
./summarizeAll.sh x86_64-slc6-gcc49-opt
svn commit x86_64-slc6-gcc49-opt.xml x86_64-slc6-gcc49-opt.summary
cd /tmp/${USER}/LCG_85/COOL/3_1_5/logs/qmtestCool/
./summarizeAll.sh x86_64-slc6-gcc49-opt
svn commit x86_64-slc6-gcc49-opt.xml x86_64-slc6-gcc49-opt.summary

If some of the tests failed, more details about the reasons for the failures can be analysed in the summary files.

-- AndreaValassi - 2016-07-28

Edit | Attach | Watch | Print version | History: r11 | r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r6 - 2016-08-01 - AndreaValassi
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Persistency All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback