Introduction

This is a CREAM ES test-suite developed using Robot Framework and python. It provides functional testing of the CREAM Execution Service. It is a work in progress and more test cases will be included in the future. The test suite supports Scientific Linux 6. If an SL5 compliant version is required, please feel free to contact me. Also, it can be easily ported to any Linux platform.

(anyone wanting to use the test suite in a non supported environment, please feel free to contact me)

Scientific Linux 6 Installation

The test suite has to be run on a host which has the capabillities to submit jobs to a CREAM endpoint. Also, the CREAM endpoint must have the ES interface enabled (you can find out how here: https://wiki.italiangrid.it/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI2#1_2_2_EMI_Execution_Service). Namely, an EMI-UI is ideal for the job.

In order to install the test-suite, execute as root:

wget http://yum.gridctb.uoa.gr/repository/robot_testing_sl6.repo -O /etc/yum.repos.d/robot_testing_sl6.repo
yum install cream_es_test

(note that the dependencies are automatically taken care of)

You can also install the RIDE editor. RIDE is a wysiwyg editor for Robot Framework's test case files. The available packages are for Scientific Linux only, but it can be easily installed in any Linux environment. In the case of SL, in order to install it, you first need to install wxPython with unicode support:

yum install wxPython
yum install robotframework-ride

If you want to use it by X11 forwarding, you must also have the xorg-x11-xauth package installed:

yum install xorg-x11-xauth

An alternative would be, to install it on your local machine and edit the remote test case files by mounting the remote host's file system through ssh for example.
If you only want to install robotframework,:

yum install robotframework

Scientific Linux 6 Dependencies

Package	Dependency
cream_es_test	glite-ce-cream-cli
	uberftp
	pexpect
	robotframework
robotframework	python >= 2.6 & < 3
Paramiko:	python-crypto
RIDE	python >= 2.6 & <3
	wxPython >= 2.8.6 with Unicode support

The packages needed by the test suite and their dependencies are provided in the aforementioned repository. Since the default Python for SL6 is version 2.6, there is no need for any external packages. All dependencies are either installed by default, or available in the standard repos (SL, EPEL etc)

Installed files

The CREAM ES testsuite installs the following files:

File	Description
/opt/cream_es_test/testsuite/cream_es_test_suite.html	The html testing module for Robot Framework
/opt/cream_es_test/lib/cream_es_testing.py	The python module with the methods/keywords needed by the test
/opt/cream_es_test/lib/vars_es.py	Python module, containing the variables needing to be set for the test
/opt/cream_es_test/docs/cream_es_testing_keywords.html	Html documentation of the keywords provided by the test library
/opt/cream_es_test/docs/cream_es_testing_libdoc.html	Html Python module documentation for the test library
/opt/cream_es_test/docs/Cream_Es_Test_Suite-doc.html	Html test suite documentation
/opt/cream_es_test/docs/COPYING	Copyright document
/opt/cream_es_test/docs/cream_es_test.7.gz and /usr/share/man/man7/cream_es_test.7.gz	The test suite's man page
/opt/cream_es_test/docs/CHANGELOG	The test suite's changelog

Also, once installed, the man page "cream_es_test" is available.

Deployment

Once the cream test package group is installed, you have to deploy the test to your environment. First of all,you can do a:

man cream_es_test

Standard Deployment Method

In order to do so, edit the file

 /opt/cream_es_test/lib/vars_es.py 

with the desired values. Examples are provided in the file itself, in the man page and in later parts of this document. Then you can execute the test suite.

Deploy with RIDE

If you have a graphical environment available or the ability to X-forward GUI applications from the testing host to a graphical environment, you can install the RIDE editor. RIDE is a WYSIWYG editor for Robot Framework test cases. This might be the most intuitive way to create a new test, since it exposes the test-suite logic to the user.

1. Create a copy of the file /opt/cream_es_test/lib/vars_es.py
2. Edit it, assigning the desired values
3. Create a copy of the file /opt/cream_es_test/testsuite/cream_es_test_suite.html
4. Edit it with the RIDE editor, changing the path pointing to the variables file (by selecting the test suite's name in the list at the left part of the screen)
5. Save the produced file

Test Execution

In the directory in which the test-suite HTML file resides,run:

pybot -W $COLUMNS /relative/or/absolute/path/to/test/html/file

The -W flag is used to tell pybot how many columns of text to print. The variable COLUMNS contains the number of columns in the current shell by default, so it is used for convenience. Unless changed from the original deployment scenario, the test suite file is /opt/cream_es_test/testsuite/cream_es_test_suite.html.

Advanced Test Execution

1. If you want to execute a certain test case in the test suite,run:

pybot -i test_case_tag testsuite.html

Multiple test cases can be included in the same way:

pybot -i tag1 -i tag2 testsuite.html

2. If you want to exclude a test case from execution,run:

pybot -e test_case_tag testsuite.html

Multiple test cases can be excluded in the same way:

pybot -e tag1 -e tag2 testsuite.html

3. You can set the log level with the -L flag. Possible values are (from lowest to highest): WARN, INFO, DEBUG, TRACE. Example:

pybot -L TRACE testsuite.html

4. You can set whether a test case is critical or not. This means that if a test case isn't critical (by default all are), if it fails, the test-suite will not fail. If a critical test case fails, the test-suite fails. This doesn't affect whether the rest of the test cases will be executed, it affects only the final test report. Examples:

pybot --critical tag testsuite.html
pybot --noncritical tag testsuite.html

Variables

The test's variables have explanations and examples in both the test-suite's HTML file and the variables.txt file. Here follows the same info:

Variable Name	Description
ce_endpoint	The cream endpoint to be used (e.g.: ctb04.gridctb.uoa.gr:8443 )
queue	The cream queue to be used (e.g.: cream-pbs-see )
vo	The user’s submitting the jobs virtual organisation (e.g.: see )
proxy_pass	The user’s submitting the jobs proxy password (e.g.: p4sSw0rD )
log_level	The log level used during the test.Default is INFO.For extra output,set to DEBUG or TRACE. (possible values: NONE FAIL WARN INFO DEBUG TRACE)
tmp_dir	The path in which temporary files will reside. They will be automatically cleaned up unless you set the variable delete_files to "False" or explicitely don’t run the cleanup test case. The path will be created -with its parents-, it doesn’t have to exist. You can leave it empty and a temporary directory will be created for you. In order to know which temp random directory is used, it is printed in standard output and in the final test suite report. Warning: any parent directories created, are not removed! All in all, unless needed for specific reasons, you should leave this variable empty.
delete_files	Delete temporary files (jdl and script files created during the test) or not. Possible values: True False. Defaults to "True"

Test Cases

Expand All + - TEST SUITE: Cream Es Test Suite

Number of Tests: 13 - + TEST CASE: Set Log Level Full Name: Cream Es Test Suite.Set Log Level Documentation: Set the log level used for the test suite. This changes the verbosity of the final test suite log files. Tags: loglevel, setup - + TEST CASE: Check Executables Full Name: Cream Es Test Suite.Check Executables Documentation: Check whether the needed executables for the test are available in the underlying system. Tags: check_exe, setup - + TEST CASE: Delegation Full Name: Cream Es Test Suite.Delegation Documentation: Create a delegation and check that its lifetime is the same as the proxy it originated from. A tolerance of 5 minutes (300 seconds) is used, just in case the clocks of the hosts aren't synchronized. Nodes must be syncronized for this test to work. Tags: deleg - + TEST CASE: Proxy Renewal Full Name: Cream Es Test Suite.Proxy Renewal Documentation: Create a one hour proxy and delegate it. Then make a two hour proxy and renew the delegation. In each case, the delegation's lifetime must be similar to the proxy's lifetime. Tags: proxy_renew - + TEST CASE: Activity without Data Staging Full Name: Cream Es Test Suite.Activity without Data Staging Documentation: Submit and collect a simple activity, executing 'sleep'. Tags: simple_activity - + TEST CASE: Activity with FailIfExitCodeNotEqualTo Full Name: Cream Es Test Suite.Activity with FailIfExitCodeNotEqualTo Documentation: Submit and collect a simple activity, executing 'sleep', using the "FailIfExitCodeNotEqualTo" attribute of the ad language. Tags: failif - + TEST CASE: Multiple Activities Full Name: Cream Es Test Suite.Multiple Activities Documentation: Submit and collect two activities, described in the same adl file. Tags: multi - + TEST CASE: Unsupported Capabillity Full Name: Cream Es Test Suite.Unsupported Capabillity Documentation: Submit a job describing an invalid capability. Try it by using the option "optional" as true, false or not at all. Verify the expected behaviour (fail for "false" and no option, success for "true"). Tags: unsupported - + TEST CASE: List Full Name: Cream Es Test Suite.List Documentation: Verify the proper usage of the listing facilities of the ES. Tests: 1) List activities before and after a submission, checking the expected results. 2) List a certain number of activities and verify that this number was returned. 3) Test the time filter, by listing jobs coming from the future (should return no jobs). Tags: list - + TEST CASE: Activity Info Full Name: Cream Es Test Suite.Activity Info Documentation: Same as the "Activity without Data Staging" test. Used to verify that the es-activity-info works as expected. Tags: info - + TEST CASE: Pause and Resume Full Name: Cream Es Test Suite.Pause and Resume Documentation: Submit an activity, wait, pause it, wait, resume it. In all intermediate steps, the activity's status and attributes must correspond to the specification (running, paused, etc). Tags: pause_resume - + TEST CASE: Cancel Full Name: Cream Es Test Suite.Cancel Documentation: Submit and cancel an activity. Verify that the end status and attributes are the right ones (terminal and xxx_cancel respectively). Tags: cancel - + TEST CASE: Wipe Full Name: Cream Es Test Suite.Wipe Documentation: Perform an activity and then wipe it. Then check that it is untraceable through es-activity-status and its sandbox has been deleted. Tags: wipe

Library Reference

Shortcuts

Keywords

Keyword	Arguments	Documentation
Check Proxy	time_left=None	Description: Check whether the proxy exists and if it has any time left. Arguments: Without any arguments,it checks if the proxy exists and has any time left With one argument,it checks if the proxy exists and has greater than or equal to the requested time left. Returns: nothing
Create Proxy	password, vo, cert=None, key=None, time=None	Description: Create a user proxy. Arguments: password the user's proxy password vo for the voms extention. cert non standard certificate path key non standard key path time the validity period of the proxy. Form: HH:MM Returns: nothing.
Destroy Proxy		Description: Delete a user's proxy. Arguments: none. Returns: nothing.
Diff Seconds	t1, t2, tolerance=0	Description: calculate difference of t1 from t2, in seconds Arguments: t1 a timestamp in seconds t2 a timestamp in seconds tolerance the maximum difference between the two timestamps Returns: diff in seconds
Es Activity Cancel	ep, aid	Description: Execute the glite-es-activity-cancel command on the given endpoint for this aid Arguments: ep the endpoint on which to run the command  (format: host:port ) aid the activity id Returns: nothing or error
Es Activity Create	ep, adl, retOut=False	Description: Submit the activity described in the file adl to the endpoint ep Arguments: ep the endpoint on which to run the command  (format: host:port ) adl path to adl file retOut if set to True, the output will also be returned Returns: the submitted aid  and possibly the command's output
Es Activity Current Status	ep, aid	Description: Execute the glite-es-activity-status command on the given endpoint Arguments: ep the endpoint on which to run the command  (format: host:port ) aid the activity id Returns: the activity's status and attributes
Es Activity Final Status	ep, aid	Description: Execute the glite-es-activity-info command on the given endpoint and wait until the job has reached TERMINAL status Arguments: ep the endpoint on which to run the command  (format: host:port ) aid the activity id Returns: the activity's status (normaly TERMINAL) and attributes
Es Activity Info	ep, aid	Description: Execute the glite-es-activity-info command on the given endpoint for this aid Arguments: ep the endpoint on which to run the command  (format: host:port ) aid the activity id Returns: the activity's info
Es Activity List	ep, fMax=None, fFrom=None, fTo=None, fState=None	Description: Execute the glite-es-activity-list command on the given endpoint, using the specified filters. Arguments: ep the endpoint on which to run the command  (format: host:port ) fMax the maximum returned entries filter fFrom FROM YYYY-MM-DD HH:mm:ss date filter fTo TO YYYY-MM-DD HH:mm:ss date filter fState Sets the status/status-attrs filter: each status/status-attr is a string in this format <STATUS>[:<STATUS_ATTRIBUTE>]; multiple status/status-attr items can be separated by ',' and are OR'ed by the service when it tries to selects activities to return. Example: TERMINAL,PREPROCESSING:VALIDATING,TERMINAL:APP_FAILURE Check corresponding man page for relevant statuses and attributes. Returns: list of aid(s)
Es Activity Pause	ep, aid	Description: Execute the glite-es-activity-pause command on the given endpoint for this aid Arguments: ep the endpoint on which to run the command  (format: host:port ) aid the activity id Returns: nothing or error
Es Activity Resume	ep, aid	Description: Execute the glite-es-activity-resume command on the given endpoint for this aid Arguments: ep the endpoint on which to run the command  (format: host:port ) aid the activity id Returns: nothing or error
Es Activity Wipe	ep, aid	Description: Execute the glite-es-activity-wipe command on the given endpoint for this aid Arguments: ep the endpoint on which to run the command  (format: host:port ) aid the activity id Returns: nothing or error
Es Delegate Proxy	ep	Description: Execute the glite-es-delegate-proxy command on the given endpoint Arguments: ep the endpoint on which to run the command  (format: host:port ) Returns: the delegation id or raises error
Es Delegation Info	ep, did, time=False	Description: Execute the glite-es-delegation-info command on the given endpoint Arguments: ep the endpoint on which to run the command  (format: host:port ) did the delegation id time if set to True, the time remaining is instead returned Returns: the command's output or the time remaining (in seconds)
Es Delegation Renew	ep, did	Description: Renew a delegation with glite-es-delegation-renew Arguments: ep the endpoint on which to run the command  (format: host:port ) did the delegation id Returns: the command's output
Es Multi Activity Create	ep, adl	Description: Submit a multiple activity described in the file adl to the endpoint ep Arguments: ep the endpoint on which to run the command  (format: host:port ) adl path to adl file Returns: list of aids
Es Multi Activity Current Status	ep, aids	Description: Execute the glite-es-activity-status command on the given endpoint for multiple aids Arguments: ep the endpoint on which to run the command  (format: host:port ) aids a list of aids Returns: two lists with statuses and attributes, in correct order
Execute Command	com	Description: Runs the given command through the subprocess module Arguments: com a string with the command and its arguments Returns: (int,str) tuple, containing the returnValue and the output of the command
Execute Uberftp Command	uberftp_command, gridftp_server, gridftp_path	Description: Execute an uberftp command on a gridftp url Arguments: uberftp_command one of cat,chgrp,chmod,dir,ls,mkdir, rename,rm,rmdir,size,stage gridftp_server the gridftp server hostname gridftp_path the path in the gridftp server Returns: The output of the command
Extract Error From Activity Submit	s	Description: calculate difference of t1 from t2, in seconds Arguments: s activity submit output Returns: error message (may be empty)
Get Activity Sb	ep, aid	Description: Find the gridftp url of the ISB of the given job Arguments: aid activity id returned by submit operation ep enpoint Returns: (gridftp server, gridftp path)
Multi Adl	queue, secs, output_dir	Description: Multi adl file.Executes /bin/sleep for the defined number of seconds. Arguments: queue the queue to submit the activity secs seconds to sleep output_dir the directory to put the file in Returns: Temporary file name.
Sleep Adl	queue, secs, output_dir	Description: Simple adl file.Executes /bin/sleep for the defined number of seconds. Arguments: queue the queue to submit the activity secs seconds to sleep output_dir the directory to put the file in Returns: Temporary file name.
Sleep Failif Ne Adl	queue, secs, output_dir, failif_exitcode_ne=None	Description: Simple adl file, using the FailIfExitCodeNotEqualTo tag. Executes /bin/sleep for the defined number of seconds. Arguments: queue the queue to submit the activity secs seconds to sleep output_dir the directory to put the file in failif_exitcode_ne the FailIfExitCodeNotEqualTo argument (ignore if None) Returns: Temporary file name.
Unsupported Capabillity Adl	queue, output_dir, optional=None	Description: Simple adl file, specifying an unsupported attribute. Arguments: queue the queue to submit the activity output_dir the directory to put the file in optional the optional value for the capabillity being tested (either true, false or none(not present) Returns: Temporary file name.

Changelog

First version, no changelog available.

Check Also

Robot Framework Quick Start Guide https://twiki.cern.ch/twiki/bin/view/EMI/RobotFrameworkQuickstartGuide

Robot Framework Advanced Guide https://twiki.cern.ch/twiki/bin/view/EMI/RobotFrameworkAdvancedGuide

Robot Framework http://code.google.com/p/robotframework/

RIDE Editor for Robot Framework test suites http://code.google.com/p/robotframework-ride/

pybot --help

man cream_es_test