CORAL and COOL functional tests

This twiki page describes the functional tests of CORAL and COOL executed against all supported backends (Oracle, MySQL, SQLite, Frontier and CoralServer). More information about the servers aganst which these tests are executed can be found in the PersistencyTestServers page.

1 Two functional test suites for long (manual) and quick (nightly) tests

There are two sets of test suites:

  • The full test suites include all functional tests developed over the years for CORAL and COOL. These tests take a long time (~15 minutes for CORAL, ~2 hours for COOL). They are meant to be executed by the development team to validate important code changes and new releases. They are launched using the qmtestRun.sh script, created by cmake from the templates defined for CORAL and COOL.
  • The quick test suites for the nightlies are subsets of the full test suites. They contain few selected tests to fit within less than 5 minutes of execution time, while still providing very basic coverage of all supported backend for both CORAL and COOL. This was agreed with Pere Mato from EP-SFT in January 2015. They are launched using the run_nightly_tests_cmake.sh script, created by cmake from the templates defined for CORAL and COOL.

The test suites are maintained using qmtest:

Note that functional tests may be executed also individually, not only within qmtest test suites.

  • For the tests executed within qmtest test suites, some environment variables are set within the qmtest drivers, which are different for CORAL and COOL: CoralTest/qmtest/LCG_QMTestExtensions.py for CORAL and CoolTest/qmtest/COOLTests.py for COOL.
  • For the tests executed individually, outside qmtest test suites, these environment variables are not set automatically and it is necessary to configure them manually if necessary/appropriate. This is the case for instance for the CORAL_AUTH_PATH, CORAL_DBLOOKUP_PATH and TNS_ADMIN environment variables described in the section below.

1.1 Connection strings and table names for the tests (hardcoded values and how to override them)

The configuration of connection strings and table names for individual tests relies on hardcoded defaults using database aliases, which can be overridden using runtime environment variables. The detailed strategy for configuring connection strings is different in CORAL and COOL. In both cases, the functional tests include both C++ and Python tests, which are configured using similar mechanisms, but in different parts of the code.

For all tests, table names are built by adding a test-specific suffix to a table name prefix that generally depends from the platform and nightly slot where tests are being executed. For simplicity the table name prefix will be called a "hash" in the following (for COOL this is indeed built as a hash, but for CORAL it is not). The motivation for using different table names for different platforms and slots is to avoid interference between the different simultaneous nightly builds that are executed every night. For instance, this prevents a dev3 SLC6 test from deleting a table wthat has just been created by a dev4 CC7 test.

1.1.1 CORAL C++ tests

For CORAL C++ tests, table name prefixes are defined in CoralCppUnitTest.h. A unique table name prefix ("hash" in the following) is assigned to any combination of platform (based on the platform-dependent CORAL_BINARY_TAG_HASH environment variable) and nightly slot (based on the slot-dependent LCG_NGT_SLT_NAME environment variable).

Connection strings are defined in CoralCppUnitDBTest.h. This header includes CoralCppUnitTest.h and is included by all relevant unit and integration tests. Integration tests, in particular, require the choice of database backend as their first command-line argument. The following mapping is defined:

  • <test> oracle[:oracle] writes to alias CORAL-Oracle-sftnight/admin and reads from alias CORAL-Oracle-sftnight/reader
  • <test> oracle:frontier writes to alias CORAL-Oracle-sftnight/admin and reads from alias CORAL-Frontier-sftnight/reader
  • <test> oracle:coral writes to alias CORAL-Oracle-sftnight/admin and reads from alias CORAL-CoralServer-Oracle-sftnight/reader
  • <test> mysql[:mysql] writes to alias CORAL-MySQL-sftnight/admin and reads from alias CORAL-MySQL-sftnight/reader
  • <test> mysql:coral writes to alias CORAL-MySQL-sftnight/admin and reads from alias CORAL-CoralServer-MySQL-sftnight/reader
  • <test> sqlite[:sqlite] writes to =sqlite_file:/tmp/<user>_<hash>.db

Note that (with the exception of SQLite tests, which use file names depending on the USER environment variable), sftnight is hardcoded in all aliases. This cannot (yet?) be overridden (as discussed in CORALCOOL-2888).

... to be continued ...

1.1.2 CORAL python tests

1.1.3 COOL C++ tests

1.1.4 COOL python tests

#Runtime environment

1.2 Setting the runtime environment for the tests

CORAL and COOL builds are performed with cmake. This is described in the PersistencyCMake page, which also contains a section about the runtime environment setup for these builds. To use the libraries and executables from a specific local (e.g. trunk) or release (e.g. LCG84) build of CORAL or COOL, environment variables such as PATH and LD_LIBRARY_PATH must be set appropriately. In summary, this can be done using one of these two commands:

  • run cc-run <command> [<arguments>] to execute a command with the appropriate CORAL/COOL environment
  • run cc-sh to enter a temporary bash shell with the appropriate CORAL/COOL environment

The cc-run and cc-sh scripts are deployed in the install directory of each build and, in principle, can be directly used from there.

  • For local builds, this is always true. For instance, run /build/CORAL/trunk/x86_64-slc6-gcc49-dbg/cc-sh to enter a temporary shell and execute a CORAL test from there.
  • For release builds, which are prepared on build nodes and then copied/relocated to AFS or cvmfs, some issues affect all release up to LCG84 included (see CORALCOOL-2873 and SPI-855), but everything should be fixed as of the LCG85swan1 release. For instance, run /cvmfs/sft.cern.ch/lcg/releases/LCG_85swan1/COOL/3_1_4/x86_64-slc6-gcc49-dbg/cc-sh to enter a temporary shell and execute a COOL test from there.

As previously mentioned, CORAL/COOL tests can be executed either individually or within qmtest test suites. In the latter case, the qmtestRun.sh and run_nightly_tests_cmake.sh scripts, which can also be found in the install directory of each build, should be used. It should be noted that these scripts already take care of setting up the runtime environment by internally calling the appropriate cc-run script, hence it is not necessary to explicitly call cc-run to call them.

In summary, several options are available. For instance, after changing directory to the install directory of the relevant build with cd /home/avalassi/COOL/trunk/x86_64-slc6-gcc49-dbg, one can execute individual or subsets of COOL tests as follows:

  • run ./cc-run test_CoolKernel_Record to execute the Record test standalone from the original shell
  • run ./cc-sh and then test_CoolKernel_Record to execute the Record test standalone from a COOL shell
  • run ./qmtestRun.sh coolkernel.record to execute the Record test within the general qmtest wrapper (run ./cc-run qmtest -D CoolTest/qmtest/ ls -R to list all tests)
  • run ./qmtestRun.sh coolkernel.record coolkernel.recordadapter to execute the Record and RecordAdapter tests within the general qmtest wrapper
  • run ./qmtestRun.sh to execute the full test suite within the general qmtest wrapper (this defaults to executing the ALL test suite)
  • run ./qmtestRun.sh ALL to execute the full test suite within the general qmtest wrapper
  • run ./qmtestRun.sh QUICK to execute the quick (nightly) test suite within the general qmtest wrapper
  • run ./run_nightly_tests_cmake.sh to execute the quick (nightly) test suite within the nightly qmtest wrapper

export CORAL_TESTSUITE_SLEEPFOR01466=1

export CORAL_SQLITE_SYNCHRONOUS_OFF=1

1.3 The authentication.xml, dblookup.xml and tnsnames.ora files (CORAL_AUTH_PATH, CORAL_DBLOOKUP_PATH and TNS_ADMIN)

The functional tests of CORAL and COOL are executed against all supported backends (Oracle, MySQL, SQLite, Frontier and CoralServer). More information about the servers against which these tests are executed can be found in the PersistencyTestServers page. The configuration of connection strings for individual tests relies on hardcoded defaults using database aliases, which can be overridden using runtime environment variables as described in another section. In any case, three files are extremely important to all tests and must be located when the tests start:

  • The authentication.xml file contains usernames and passwords for the Oracle and MySQL connection strings used by the tests. It is looked up via the CORAL_AUTH_PATH environment variable.
  • The dblookup.xml file contains the mapping between the aliases used by the tests and the connection strings for the corresponding replicas. It is looked up via the CORAL_DBLOOKUP_PATH environment variable.
  • The tnsnames.ora file contains the mapping between the Oracle instance names used by the connection strings and the corresponding host names and ports. It is looked up via the TNS_ADMIN environment variable.

In most cases, CORAL_AUTH_PATH and CORAL_DBLOOKUP_PATH are set to the same value, indicating a directory that contains both authentication.xml and dblookup.xml. This directory must be access-restricted as authentication.xml contains database credentials.

Whenever available, AFS is used to simplify the distribution and access control for all of these files. By default, nightly tests on Linux look up these three files from the following AFS directories:

  • authentication.xml and dblookup.xml from /afs/cern.ch/sw/lcg/app/pool/db/ (access-protected and readable by sftnight and by CORAL/COOL team members with their AFS token)
  • tnsnames.ora from /afs/cern.ch/sw/lcg/app/releases/CORAL/internal/oracle/admin (where it is available as a symbolic link to the master copy on /afs/cern.ch/project/oracle/admin, together with a custom sqlnet.ora file for CORAL/COOL tests which is also needed to control Oracle client-server communication)

On platforms where AFS is not available (or where the AFS filesystem is available but AFS tokens are not available to the users running the jobs), various options are possible. In the nightlies (e.g. for Mac), Oracle and MySQL tests are skipped and only SQLite tests are executed. For manual tests, developers can work around the issue by copying the relevant files to a directory of their choice and setting the CORAL_AUTH_PATH, CORAL_DBLOOKUP_PATH and/or TNS_ADMIN environment variables accordingly. By default, if CORAL_AUTH_PATH and CORAL_DBLOOKUP_PATH are not set, their values often default to $HOME/private as this is where they typically reside. Using custom authentication.xml and dblookup.xml files (typically in $HOME/private) is also needed for developers who want to use their own Oracle or MySQL test accounts, for instance.

For the tests executed within qmtest test suites, the default configuration of these three environment variables is defined within the qmtest drivers, CoralTest/qmtest/LCG_QMTestExtensions.py for CORAL and CoolTest/qmtest/COOLTests.py for COOL. It is these two files that contain special settings for user sftnight (depending on the platform) and also set CORAL_AUTH_PATH and CORAL_DBLOOKUP_PATH to $HOME/private if not yet defined.

1.4 Running the tests

2 Nightly builds and tests (jenkins, cdash, lcgcmake)

The CORAL and COOL software is built and tested automatically every night, thanks to the infrastructure provided by the SPI team in EP-SFT.

The nightlies test the software "in three dimensions":

  • Platform (e.g. x86_64-slc6-gcc49-dbg). The tests are built on all production platforms, as well as a few development platforms. Linux SLC6/CC7 on Intel processors with 64-bit builds using gcc49 is currently the only supported platform in June 2016, but support for other platforms is being added, including MacOSX 10.11 El Capitan (e.g. CORALCOOL-2884), gcc6.1 (CORALCOOL-2895), clang 3.8 (CORALCOOL-2901), ARM64 (CORALCOOL-2857). Windows builds are no longer supported. Some of the nightlies are executed using virtual machines instead of the native O/S they are testing.
  • Day of the week (e.g. Mon, Tue... Sun). All projects are compiled every night. Usually the building and testing starts around midnight (using tarballs created from the appropriate software tags as collected at 23.15, i.e. 45 minutes in advance) and results are provided later in the morning. The produced binaries and results are kept for one week (but a summary of the results of earlier builds and tests can, in principle, be recovered from AFS backup even later - if they were installed on AFS in the first place, which is not the case for all platforms).
  • Slot (e.g. dev2, dev3, dev4, experimental...). Projects are built in different configurations, i.e. a set of versions of AA packages that are supposed to work together. For instance, one configuration slot may test a preview of the "HEAD" of all projects, while other configuration slots may test branches with patches over the current production release. The "dev" slots are generally more stable than the "experimental" slot, where new development platforms and new disruptive extenal package versions tend to appear first. Presently (June 2016), only results from the "dev" slots are copied to AFS, while those from the "experimental" slot stay on the build nodes and are published on the web. The top of the PersistencyReleaseNotes page normally contains a brief description of the differences between the currently active dev slots.

The tests are executed using jenkins and can be controlled on the "LCG Externals" tab of the phsft-jenkins.cern.ch web interface. This makes it possible to start new unscheduled tests (please check with the SPI team in EP-SFT before!), by clicking on "Build with Parameters" in the appropriate slot, e.g. lcg_experimental. If you made important changes to the code between 23.15 and midnight, and you want these to be included in the next nightly build, this inteface also allows you to recreate the tarballs from your software tags, by clicking on "Build with Parameters" in the lcg-prepare-builds section.

The results of the nightlies are available on the "LCGSoft" tab of cdash.cern.ch. For those platforms and slots that get installed on AFS, the binaries and results are also available under /afs/cern.ch/sw/lcg/app/nightlies, for instance on /afs/cern.ch/sw/lcg/app/nightlies/dev2/Fri/CORAL for the dev2 build on CORAL on the most recent Friday.

The nightly builds and tests of CORAL and COOL are started by jenkins within a job that performs a full build of the LCG stack using lcgcmake. The relevant configuration for CORAL and COOL builds and tests is all contained within the externals/CMakeLists.txt file. In particular, this is the file that sets up the nightly tests to use the run_nightly_tests_cmake.sh script.

The nightlies are all executed as user sftnight. On the main Linux platforms, a valid AFS token for sftnight is available for the duration of the build job.

-- AndreaValassi - 2016-06-24

Edit | Attach | Watch | Print version | History: r15 | r13 < r12 < r11 < r10 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r11 - 2016-08-07 - AndreaValassi
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Persistency All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback