Guidelines to write new Tests

In order to implement a common testing strategy for the different Data Management products, the development and execution of tests must follow commons guidelines. This common approach will help the maintenance of the tests and allow a faster integration of new tests. This page describes the identified guidelines.

General Structure

All tests are located under a folder called /testing directly under the root of the project's subversion tree:

• https://svn.cern.ch/reps/lcgdm/testing/
• https://svn.cern.ch/reps/lcgutil/testing/
• https://svn.cern.ch/reps/glitefts/testing/

Each folder contains the three usual subversion folders: tags, branches and trunk.

• Trunk is used to host the current set of tests;
• Tags is used to mark a set of existing tests executed in the context of a particular certification;
• Branches is not used right now. It might be used for major rewriting of existing tests.

In each of these three folders (e.g. trunk) there are one or more directories, containing different types of tests (e.g, functional, regression and performance).

So the overall structure, replicated in the three subversion repositories, following this organization:

• /testing/
  • branches/
  • tags/
    • certification_1_8_0/
      • Whatever-config
      • functional/
      • regression/
  • trunk/
    • Whatever-config
    • functional/
    • regression/
    • unstable/

Naming conventions

The term "certification tests" is not longer used, as it is usually confusing: it is mixing what a test does and when it is used. Instead, the following terminology is used:

• Functional tests - test concrete use cases and/or functionalities;
• Regression tests - test specific bugs detected in the past to prevent reappearances;
• Performance tests - test the behaviour under high load
• Unstable tests - tests still in development or may fail because of dependencies and resources: testbed does not support the required functionality, etc.

Usually there is one or more components per product (e.g. dpns, dpm, rfio for DPM_mysql), so their tests must be grouped in different subfolders, using a common prefix for them (e.g. dpns/dpns-ls). The best way of seeing this is having a look into the repositories you have at the beginning of this page.

Finality, the files following the pattern *-config are used to set up the environment properly.

In summary:

1. The test must be under trunk
2. Then, it must be under the appropriate subdirectory, grouping by type of test: functional, regression, unstable (usually it should start here)...
3. In must be under a directory grouping by component and language if relevant (dpns, dpm, lfc, lfc-api-python, gfal, ...)
4. It must use as a prefix the name of the tested component
5. The rest of the name is up to the developer, but it should be descriptive enough

Tests Development

The develop a new test, there are four main steps:

1. Create and executable script inside unstable/component/ (e.g. unstable/dpns)
2. Add the common header
3. Start writing!
4. When ready inform it-dep-gt-dms@cernNOSPAMPLEASE.ch about the test availability to allow its verification.

New tests shouldn't go into "functional" or "regression" folders until they are properly working. They should be added to the "unstable" folder.

#!/bin/bash
##############################################################################
# Copyright (c) Members of the EGEE Collaboration. 2010.
# See http://www.eu-egee.org/partners/ for details on the copyright
# holders.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##############################################################################
#
# AUTHORS: Your name <Your email>
#
# DESCRIPTION:
#
##############################################################################
# meta: proxy=True
# meta: preconfig=../../DPM-config

The first "meta" specify if the test need the user proxy, and the second which script sets up the environment. If more than one script have the same dependency, it will be executed only once anyway.

It is also important to know that the tests must get their parameters ONLY from environment variables (e.g. LFC_HOST, VO, ...), which are set by the configuration script. This is done like this to have a uniform way of invoking tests, so all of them can be used by the same wrapper.

You should do this step at least once, as it wouldn't be the first time a test leaves an inconsistent state for the next ones.

Tests with multiple files

In that case, you should create a folder with the name of the test, and locate under it all the files you need, and a script called "execute_test.sh". The wrapper will realise this, and execute the test through this director script.

Tests with subtests

It may happen that you have a single file - a binary - that actually performs a bunch of different subtests. In that case, you don't want to treat it as a single test entry. For those cases, the wrapper support "plugins": you can write your own parser that will generate the subtest list from the test output.

Those plugins are just a function called "parse" inside a Python module in the folder "plugins". That function should return None if it doesn't recognise the format, or a TestSet if it does.

You can see an example for the SRM tests from Todor.

Tests Execution

Usually it is a good idea to deploy a machine with the service you want to test. In that case, the best and easier way is to deploy a VNode machine, and then use Yaimgen to install and configure the service.

For instance:

[root@vtb-generic-79 ~]# svn co http://svnweb.cern.ch/guest/yaimgen/yaimgen3/tags/stable/ yaimgen
[root@vtb-generic-79 ~]# cd yaimgen
[root@vtb-generic-79 yaimgen]# ./yaimgen.sh -n glite-SE_dpm_mysql

And that's it. Of course yaimgen accepts more parameters (user defined repo files, oracle user and password, etc). Just type ./yaimgen.sh -h to get more help.

Note: Yaimgen will always execute the tests at the end unless the -notest flag is passed. If you want the tests to be executed, you will have to specify the test user password.

Batch execution

*IMPORTANT*: This section is outdated. You should go to the SAKET Trac instead.

Once you have deployed the service, you can executed all the tests at once, and get a XML report. If you disabled the test execution, then you will have to checkout manually the wrapper from subversion:

[root@vtb-generic-79 ~]# svn co http://svnweb.cern.ch/guest/lcgutil/tests-tools/tags/stable/ tests-tools

And the tests:

[root@vtb-generic-79 ~]# svn co https://svn.cern.ch/reps/lcgdm/testing/trunk/ lcgdm-tests

And then execute the wrapper specifying the test location, the test user password, and the groups (remember the prefixes) you want to execute. For instance:

[root@vtb-generic-79 ~]# cd tests-tools
[root@vtb-generic-79 tests-tools]# ./tests-wrapper.py -v -d ~/lcgdm-tests/functional -p <test user password> -o log.xml dpns dpm rfio

Again, just do ./tests-wrapper.py --help to get a description of the options.

NOTE: The wrapper will install the VOMS client if needed, and if you are root. NOTE: The report is a XML file. There is a XSLT and a CSS file in the Yaimgen folder that can be used to see it in a browser.

What about using a different configuration file?

The wrapper admits another parameter that allows to specify a different configuration script. It would be like:

[root@vtb-generic-79 tests-tools]# ./tests-wrapper.py -v -d ~/lcgdm-tests/functional -p <test user password> -o log.xml -c ~/lcgdm-tests/myconfig dpns dpm rfio

That script MUST be a bash script, as it will be sourced. Remember that any parameter you want to pass to your test, should be passed through environment variables. E.g:

export LFC_HOST=mylfc.cern.ch
export VO=dteam
export WHATEVER_VARIABLE=whatever_value

Manual execution

The previous wrapper is good if you want to execute the whole bunch of tests but, what if you want to execute only one manually? Well, as mentioned before there is a configuration file that will set up the environment (except the VOMS client!!), so the steps would be:

[root@vtb-generic-79 ~]# cd lcgdm-tests
[root@vtb-generic-79 lcgdm-tests]# source DPM-config
[root@vtb-generic-79 lcgdm-tests]# cd functional/dpns
[root@vtb-generic-79 dpns]# ./dpns-ls

That's is! Take into account that most times some extra RPMs will be needed, so you will need to be root when executing DPM-config (at least the first time).

Proxy

Remember to get your user proxy before executing the tests:

[root@vtb-generic-79 ~]# export X509_USER_CERT=/root/user_certificates/test_user_501_cert.pem
[root@vtb-generic-79 ~]# export X509_USER_KEY=/root/user_certificates/test_user_501_key.pem
[root@vtb-generic-79 ~]# voms-proxy-init --voms dteam:/dteam

Remember to install glite-security-voms-clients if you haven't already done it.

If you have any doubts or comments, just contact with Alejandro Álvarez