Introduction



This is a CREAM test-suite developed using Robot Framework and python. It provides functional testing of CREAM services and related utilities. It is a work in progress and more test cases will be included in the future. The test suite supports Scientific Linux 5 and 6, as well as EMI 1 and EMI 2 releases. Also, it can easily be ported to any Linux platform. (anyone wanting to use the test suite in a non supported environment, please feel free to contact me)

For news and more info, you can follow us on Twitter and Posterous!

Scientific Linux 6 Installation



The test-suite has to be run in a host which has the capabilities to submit jobs to a CREAM endpoint. Namely, an EMI-UI is ideal for the job.

In order to install the test-suite, execute as root:

wget http://yum.gridctb.uoa.gr/repository/robot_testing_sl6.repo -O /etc/yum.repos.d/robot_testing_sl6.repo
yum install cream_test
(please note that the dependencies are automatically taken care of. Also,in previous versions, the robot_testing repository had to take precedence over the standard SL repos. This is no longer the case.)

If you have previously installed either the cream_test package, remove cream_test and then reinstall it:

yum remove cream_test
yum clean all
yum install cream_test



You can also install the RIDE editor. RIDE is a wysiwyg editor for Robot Framework's test case files. The available packages are for Scientific Linux only, but it can be easily installed in any Linux environment. In the case of SL, in order to install it, you first need to install wxPython with unicode support:

yum install wxPython
yum install robotframework-ride

If you want to use it by X11 forwarding, you must also have the xorg-x11-xauth package installed:

yum install xorg-x11-xauth

An alternative would be, to install it on your local machine and edit the remote test case files by mounting the remote host's file system through ssh for example.



If you only want to install robotframework,:

yum install robotframework

Scientific Linux 6 Dependencies



The available packages and their dependencies are:

Package Dependency
cream_test glite-ce-cream-cli
  lcg-util
  lfc
  uberftp
  subversion
  newt
  python-paramiko
  pexpect
  robotframework
robotframework python >= 2.6 & < 3
Paramiko: python-crypto
RIDE python >= 2.6 & <3
  wxPython >= 2.8.6 with Unicode support

The packages needed by the test suite and their dependencies are provided in the aforementioned repository. Since the default Python for SL6 is version 2.6, there is no need for any external packages. All dependencies are either installed by default, or available in the standard repos (SL, EPEL etc)

Scientific Linux 5 Installation



The test-suite has to be run in a host which has the capabilities to submit jobs to a CREAM endpoint. Namely, an EMI-UI is ideal for the job.

In order to install the test-suite, execute as root:

wget http://yum.gridctb.uoa.gr/repository/robot_testing_sl5.repo -O /etc/yum.repos.d/robot_testing_sl5.repo
yum install cream_test
(please note that the dependencies are automatically taken care of. Also,in previous versions, the robot_testing repository had to take precedence over the standard SL repos. This is no longer the case.)

If you have previously installed either the cream_test package, remove cream_test and then reinstall it:

yum remove cream_test
yum clean all
yum install cream_test



You can also install the RIDE editor. RIDE is a wysiwyg editor for Robot Framework's test case files. The available packages are for Scientific Linux only, but it can be easily installed in any Linux environment. In the case of SL, in order to install it, you first need to install wxPython for python26 with unicode support:

yum --disablerepo "*" --enablerepo "robot_testing_sl5" install wxPython
yum install robotframework-ride

If you want to use it by X11 forwarding, you must also have the xorg-x11-xauth package installed:

yum install xorg-x11-xauth

An alternative would be, to install it on your local machine and edit the remote test case files by mounting the remote host's file system through ssh for example.



If you only want to install robotframework, you must first install python26:

yum install python26
yum install robotframework

Scientific Linux 5 Dependencies



The available packages and their dependencies are:

Package Dependency
cream_test glite-ce-cream-cli
  lcg-util
  lfc
  uberftp
  subversion
  newt
  python26-paramiko
  python26-pexpect
  robotframework
robotframework python >= 2.6 & < 3
Paramiko: python26-crypto
RIDE python >= 2.6 & <3
  wxPython >= 2.8.6 with Unicode support

The packages needed by the test suite are provided in the aforementioned repository. Also, python26-pexpect and wxPython with unicode support for python26 are provided by our repository. The test suite needs python26 installed, since the default python for Scientific Linux 5 is 2.4. All other dependencies are either installed by default, or available in the standard repos (SL, EPEL etc)

Installed files

The CREAM testsuite installs the following files:
File Description
/opt/cream_test/testsuite/cream_test_suite.html The html testing module for Robot Framework
/opt/cream_test/lib/cream_testing.py The python module with the methods/keywords needed by the test
/opt/cream_test/lib/vars.py Python module, containing the variables needing to be set for the test
/opt/cream_test/bin/setup_keys.sh Bash script, utilizing newt, to automatically set up ssh key authentication with needed hosts
/opt/cream_test/docs/cream_testing_keywords.html Html documentation of the keywords provided by the test library
/opt/cream_test/docs/cream_testing_libdoc.html Html Python module documentation for the test library
/opt/cream_test/docs/Cream_Test_Suite-doc.html Html test suite documentation
/opt/cream_test/docs/COPYING Copyright document
/opt/cream_test/docs/cream_test.7.gz and /usr/share/man/man7/cream_test.7.gz The test suite's man page
/opt/cream_test/docs/CHANGELOG The test suite's changelog
Also, once installed, the man page "cream_test" is available.

Deployment



Once the cream test package group is installed, you have to deploy the test to your environment. First of all,you can do a:

man cream_test
to view information about the testsuite and how to deploy it. There are also available these tutorial videos, but please note that they are obsolete. New ones will be uploaded as soon as possible:

In order to deploy the test a set of variables that depend on the testing environment must be configured accordingly.

SSH Key Authentication



In order to use the test suite, the host executing it must have automatic ssh access through keys, to some remote service hosts, namely CREAM, LRMS and ARGUS. A special, interactive dialog script is provided in order to help with creating and deploying those keys quickly and effortlessly:

/opt/cream_test/bin/setup_keys.sh

Standard Deployment Method



In order to do so, edit the file

 /opt/cream_test/lib/vars.py 
with the desired values. Examples are provided in the file itself, in the man page and in later parts of this document. Then you can execute the test suite.

Deploy with RIDE



If you have a graphical environment available or the ability to X-forward GUI applications from the testing host to a graphical environment, you can install the RIDE editor. RIDE is a WYSIWYG editor for Robot Framework test cases. This might be the most intuitive way to create a new test, since it exposes the test-suite logic to the user.

  1. Create a copy of the file /opt/cream_test/lib/vars.py
  2. Edit it, assigning the desired values
  3. Create a copy of the file /opt/cream_test/testsuite/cream_test_suite.html
  4. Edit it with the RIDE editor, changing the path pointing to the variables file (by selecting the test suite's name in the list at the left part of the screen)
  5. Save the produced file

Test Execution



In the directory in which the test-suite HTML file resides,run:

pybot -W $COLUMNS /relative/or/absolute/path/to/test/html/file
The -W flag is used to tell pybot how many columns of text to print. The variable COLUMNS contains the number of columns in the current shell by default, so it is used for convenience. Unless changed from the original deployment scenario, the test suite file is /opt/cream_test/testsuite/cream_test_suite.html.

Advanced Test Execution



1. If you want to execute a certain test case in the test suite,run:

pybot -i test_case_tag testsuite.html
Multiple test cases can be included in the same way:
pybot -i tag1 -i tag2 testsuite.html

2. If you want to exclude a test case from execution,run:

pybot -e test_case_tag testsuite.html
Multiple test cases can be excluded in the same way:
pybot -e tag1 -e tag2 testsuite.html

3. You can set the log level with the -L flag. Possible values are (from lowest to highest): WARN, INFO, DEBUG, TRACE. Example:

pybot -L TRACE testsuite.html

4. You can set whether a test case is critical or not. This means that if a test case isn't critical (by default all are), if it fails, the test-suite will not fail. If a critical test case fails, the test-suite fails. This doesn't affect whether the rest of the test cases will be executed, it affects only the final test report. Examples:

pybot --critical tag testsuite.html
pybot --noncritical tag testsuite.html

Test Suite Description



Variables



The test's variables have explanations and examples in both the test-suite's HTML file and the variables.txt file. Here follows the same info:

Variable Name Description
ce_endpoint The cream endpoint to be used (e.g.: ctb04.gridctb.uoa.gr:8443 )
cream_queue The cream queue to be used (e.g.: cream-pbs-see )
deleg_endpoint The CREAM endpoint in use for delegation (e.g.: https://ctb04.gridctb.uoa.gr:8443//ce-cream/services/gridsite-delegation)
vo The userís submitting the jobs virtual organisation (e.g.: see )
proxy_pass The userís submitting the jobs proxy password (e.g.: p4sSw0rD )
gridftp_server Gridftp server used for data transfers (e.g.: se01.isabella.grnet.gr )
lfn_dir LFC directory for data transfers. This directory will be created, it doesnít have to allready exist. Example: /grid/dteam/cream_testing
deleg_id A delegation id string (e.g.: me_deleg_id )
log_level The log level used during the test.Default is INFO.For extra output,set to DEBUG or TRACE. (possible values: NONE FAIL WARN INFO DEBUG TRACE)
batch_system The underlying batch system of the CREAM endpoint.Either pbs or lsf.
gridftp_dir A directory in the gridftp server.This directory has to allready exist and your vo have write access to it. Used for OSB file storage. Example: /tmp
lrms_host The hostname where TORQUE is running. Example: ctb07.gridctb.uoa.gr
tmp_dir The path in which temporary files will reside. They will be automatically cleaned up unless you set the variable delete_files to "False" or explicitely donít run the cleanup test case. The path will be created -with its parents-, it doesnít have to exist. You can leave it empty and a temporary directory will be created for you. In order to know which temp random directory is used, it is printed in standard output and in the final test suite report. Warning: any parent directories created, are not removed! All in all, unless needed for specific reasons, you should leave this variable empty.
delete_files Delete temporary files (jdl and script files created during the test) or not. Possible values: True False. Defaults to "True"
sec_cert Path to a second user certificate.
sec_key Path to a second user key.
sec_proxy_pass Password for the aforementioned second certificate.
creamdb_host The host of the cream database.
creamdb_port The port in which the cream database is listening for connections.
creamdb_user The user of the cream data base
creamdb_pass The database password for the aforementioned user
authz_model The authorization model in use by CREAM. Either gjaf or argus.
argus_host The host of the argus service, if applicable.
middleware_version The version of the middleware being tested. Either EMI1 or EMI2.
cream_ldap_port The ldap port for the CREAM resource bdii. Defaults to 2170.
lfc_host The LFC_HOST environmental variable to be set on the WN for output data operations.
lcg_gfal_infosys The LCG_GFAL_INFOSYS environmental variable to be set on the WN for outputdata operations
blparser_host The (old) blparser host. Defaults to the ce_endpoint. Could also be lrms_host, or a whole different host

Test Cases



Documentation for Cream Test Suite

Documentation for Cream Test Suite

 
Expand All TEST SUITE: Cream Test Suite
TEST CASE: Set Log Level
TEST CASE: Check Executables
TEST CASE: Simple Submit
TEST CASE: ISB Base URI
TEST CASE: ISB Client to CE
TEST CASE: ISB gsiftp to CE
TEST CASE: OSB localhost
TEST CASE: OSB gsiftp BaseDestURI
TEST CASE: OSB gsiftp DestURI
TEST CASE: Environment
TEST CASE: Epilogue
TEST CASE: Prologue
TEST CASE: Explicit Delegation
TEST CASE: Job Cancel
TEST CASE: Job List
TEST CASE: Job Suspend - Resume
TEST CASE: Manually Cancel Job
TEST CASE: Verify CE service info
TEST CASE: Enable - Disable Submission
TEST CASE: Job Purge
TEST CASE: Admin Manage Job
TEST CASE: Limiter
TEST CASE: Ban User
TEST CASE: Sandbox Staging
TEST CASE: Validate Glue Schema
TEST CASE: Automatic Proxy Purging
TEST CASE: Verify Job Status Output
TEST CASE: OutputData
TEST CASE: Automatic Job Purge
TEST CASE: Proxy Renewal
TEST CASE: CPU Allocation 1
TEST CASE: CPU Allocation 2
TEST CASE: CPU Allocation 3
TEST CASE: CPU Allocation 4
TEST CASE: CPU Allocation 5
TEST CASE: CPU Allocation 6
TEST CASE: Resource BDII
TEST CASE: Logging and Bookkeeping
TEST CASE: Old BLParser Restart

Library Reference



cream_testing Shortcuts

Allowed Submission  ·  Ban User Argus  ·  Ban User Gjaf  ·  Bkill Job  ·  Cancel All Jobs  ·  Cancel Job  ·  Ce Service Info  ·  Change Sandbox Transfer Method  ·  Check Delegation Id In Filesystem  ·  Check Osb Basedesturi Files  ·  Check Osb Desturi Files  ·  Check Proxy  ·  Check Resource Bdii Published Values  ·  Check Wms Logging Info  ·  Combo Jdl  ·  Cpu Allocation Jdl  ·  Cpu Allocation Test  ·  Cpunumber Jdl  ·  Create Delegation  ·  Create Dummy Script  ·  Create Proxy  ·  Delegation Info  ·  Delete Osb Basedesturi Files  ·  Delete Osb Desturi Files  ·  Destroy Delegation  ·  Destroy Proxy  ·  Disable Cream Admin  ·  Disable Submission  ·  Enable Cream Admin  ·  Enable Submission  ·  Environment Jdl  ·  Epilogue Jdl  ·  Execute Noninteractive Ssh Com  ·  Execute Uberftp Command  ·  Fetch Output Files  ·  File Should Contain  ·  Files Should Not Be Empty  ·  Get Arch Smp Size  ·  Get Current Status  ·  Get Current Status Verbose  ·  Get Deleg Id From Status  ·  Get Delegation Id  ·  Get Final Status  ·  Get Guid  ·  Get Job Sb  ·  Get Osb  ·  Get Proxy Dn  ·  Get Status With Filter  ·  Get Time In Status Format  ·  Hostnumber Jdl  ·  Isb Baseuri Jdl  ·  Isb Client To Ce Jdl  ·  Isb Gsiftp To Ce Jdl  ·  Job Status Should Be  ·  Jobdbadminpurger  ·  Lcg Cp  ·  Lcg Cr  ·  Lcg Del  ·  Ldap Search  ·  Lfc Ls  ·  Lfc Mkdir  ·  Lfc Rmdir  ·  List Jobs  ·  List Proxy Sandbox  ·  Localhost Output Jdl  ·  Modify Cream Config Xml  ·  Multiple Lcg Cp  ·  Osb Basedesturi Jdl  ·  Osb Desturi Jdl  ·  Output Data Jdl  ·  Prologue Jdl  ·  Proxy Info  ·  Purge All Jobs  ·  Purge Job  ·  Qdel Job  ·  Renew Proxy  ·  Reset Cream Config Xml  ·  Reset Limiter Threshold  ·  Restart Cream  ·  Resume All Jobs  ·  Resume Job  ·  Save Batch Job Submission Script Off  ·  Save Batch Job Submission Script On  ·  Send Signal To Process  ·  Set Limiter Threshold  ·  Simple Jdl  ·  Sleep Jdl  ·  Smpgranularity Jdl  ·  Start Old Blparser  ·  Stop Old Blparser  ·  String Should Contain  ·  Submit And Wait  ·  Submit Job  ·  Suspend All Jobs  ·  Suspend Job  ·  Uberftp Upload  ·  Unban User Argus  ·  Unban User Gjaf  ·  Validate Ce Service Info  ·  Validate Glue  ·  Validate Job Status  ·  Wait For Status  ·  Wholenodes Jdl  ·  Wms Get Final Job Status  ·  Wms Get Job Status  ·  Wms Job Logging Info  ·  Wms Job Submit
Keywords
Keyword Arguments Documentation
Allowed Submission ce_endpoint
Description: Return the output of the glite-ce-allowed-submission command

Arguments: ce_endpoint the cream endpoint

Returns: the command's output.
Ban User Argus dn, argus_host
Description: Ban the specified DN using pap-admin cli on the argus host

Arguments: dn Distinguished Name to be added
argus_host the argus host

Returns: Nothing (raises exception upon error)
Ban User Gjaf dn, ce_endpoint
Description: Add the specified DN as a CREAM banned user (add it to /etc/lcas/ban_users.db)

Arguments: dn Distinguished Name to be added
ce_endpoint the cream endpoint

Returns: Nothing (raises exception upon error)
Bkill Job jid, lsf_host
Description: Manually bkill a job.

Arguments: jid job identifier
lsf_host the server hosting lsf, either ip or name

Returns: Nothing
Cancel All Jobs ce_endpoint
Description: Cancel all the user's jobs in the given cream endpoint.

Arguments: ce_endpoint cream endpoint.

Returns: nothing.
Cancel Job job_id
Description: Cancel the given job.

Arguments: job_id as returned by the submit operation.

Returns: nothing.
Ce Service Info ce_endpoint, verbosity
Description: Get the service info of a CREAM ce.
Arguments: ce_endpoint the CREAM endpoint
verbosity how much information will be shown, either 1 or 2
Returns: The output of the command
Change Sandbox Transfer Method ce_endpoint
Description: Change the SANDBOX_TRANSFER_METHOD between GSIFTP and LRMS (@ /etc/lcas/ban_users.db)

Arguments: ce_endpoint the cream endpoint

Returns: Nothing (raises exception upon error)
Check Delegation Id In Filesystem contents_before, contents_after, deleg_id
Description: Check the presense (or not) of a delegation id in the respective cream sandbox dir

Arguments: contents_before fs contents before running the purger
contents_after fs contents after running the purger
deleg_id the delegation id searched for

Returns: nothing (raises exception if the desired outcome isn't met)
Check Osb Basedesturi Files jdl_path, gridftp_server, gridftp_path
Description: Read from a jdl file the list of files contained in the OutputSandBox and OutputSandboxBaseDestURI attribute.
Then check if these files exist or not.

Arguments: jdl_path path to the jdl file
gridftp_server the gridftp server in use
gridftp_path the path within the gridftp server

Returns: True or False.
Check Osb Desturi Files jdl_path, gridftp_server, gridftp_path
Description: Read from a jdl file the list of files contained in the OutputSandboxDestURI attribute.
Then check if these files exist or not.

Arguments: jdl_path path to the jdl file
gridftp_server the gridftp server in use
gridftp_path the path within the gridftp server

Returns: True or False.
Check Proxy time_left=None
Description: Check whether the proxy exists and if it has any time left.

Arguments: Without any arguments,it checks if the proxy exists and has any time left
With one argument,it checks if the proxy exists and has greater than or equal to the requested time left.

Returns: nothing
Check Resource Bdii Published Values results
Description: Validate the CREAM resource BDII published values according to
https://wiki.italiangrid.it/twiki/bin/view/CREAM/CreamTestWorkPlan#Resource_BDII_test

Arguments: results list with ldapsearch results

Returns: Nothing. Raises exception upon non-confirmation.
Check Wms Logging Info output
Description: Validates the output of a glite-wms-job-logging-info command

Arguments: output the output of a wms job status logging ingo command

Returns: Nothing (raises exception uppon non-validation)
Combo Jdl vo, wholenodes, hostnumber, smpgranularity, output_dir
Description: Attribute checking jdl file.Sets the jdl attributes "WholeNodes","HostNumber" and "SMPGranularity"
to the given numbers/values. It is used to test whether these attributes can be combined. In theory,
when WholeNodes is "False",the other two shouldn't be able to be set and an error should be produced
during jdl submission. This should be tested for arbitrary values of hostnumber and smpgranularity,
and for both cases of wholenodes (set to "True" and set to "False").

Arguments: vo virtual organisation
smpgranularity jdl attribute
hostnumber jdl attribute
wholenodes jdl attribute
output_dir the directory to put the file in
Returns: Temporary file name.
Cpu Allocation Jdl output_dir, vo, wholenodes=None, hostnumber=None, smpgranularity=None, cpunumber=None
Description: Attribute checking jdl file.Sets the jdl attributes "WholeNodes","HostNumber" and "SMPGranularity"
to the given numbers/values. If a value isn't given, then it is ignored in the final jdl file.

Arguments: output_dir the directory to put the file in
vo virtual organisation
smpgranularity jdl attribute
hostnumber jdl attribute
wholenodes jdl attribute
cpunumber jdl attribute

Returns: Temporary file name.
Cpu Allocation Test jid, ce_endpoint, test_type=0, wholenodes=0, cpunumber=0, smpgranularity=0, hostnumber=0, smpsize=0, batch_system=0
Description: Validate the contents of the batch job submission script, according to:
https://wiki.italiangrid.it/twiki/bin/view/CREAM/CreamTestWorkPlan
cpu allocation test specifications. Per test information follows:
Wherever it appears,S is the value published as GlueHostArchitectureSMPSize
Test type 1
Jdl Attributes: WholeNodes=true SMPGranularity=G HostNumber=H, H>1
Verification for LSF:  BSUB -n S*H, BSUB -R "span[ptile=S]", BSUB -x
Verification for PBS:  PBS -l nodes=H:ppn=S, PBS -W x=NACCESSPOLICY:SINGLEJOB
Test type 2
JDL Attributes: WholeNodes=true SMPGranularity=G
Verification for LSF: BSUB -n S, BSUB -R "span[jpsts=1]", BSUB -x
Verification for PBS: PBS -l nodes=1:ppn=S, PBS -W x=NACCESSPOLICY:SINGLEJOB
Test type 3
JDL Attributes: WholeNodes=true, HostNumber=H
Verification for LSF: BSUB -n S*H, BSUB -R "span[ptiles=S]", BSUB -x
Verification for PBS: PBS -l nodes=H:ppn=S, PBS -W x=NACCESSPOLICY:SINGLEJOB
Test type 4
JDL Attributes: WholeNodes=false, SMPGranularity=G, CPUNumber=C
Verification for LSF: BSUB -n C, BSUB -R "span[ptile=G]"
Verification for PBS: PBS -l nodes=N:ppn=G { [+1:ppn=R] if r>0 }
Test type 5
JDL Attributes: WholeNodes=false, HostNumber=H, CPUNumber=C, H>1
Verification for LSF: BSUB -n C, BSUB -R "span[ptile={ N if R=0 ; N+1 if R>0 }]"
Verification for PBS: PBS -l nodes=H-R:ppn=N { [+R:ppn=N+1] if R>0 }
Test type 6
JDL Attributes: WholeNodes=false, CPUNumber=C
Verification for LSF: BSUB -n C
Verification for PBS: PBS -L nodes=C
NOTE: Default values are used due to robot framework limitations for named
arguments. These default values serve no other purpose.
Arguments: jid CREAM job identifier
ce_endpoint the CREAM endpoint
test_type one of the aforementioned test types, as an integer
wholenodes jdl attribute in use (either true or false!)
cpunumber jdl attribute in use
smpgranularity jdl attribute in use
hostnumber jdl attribute in use
smpsize the value published as GlueHostArchitectureSMPSize
batch_system the underlying lrms system

Returns: Nothing (raises exception if anything is out of order)
Cpunumber Jdl vo, cpunumber, output_dir
Description: Attribute checking jdl file.Sets the jdl attribute "CPUNumber" to the given number.

Arguments: vo virtual organisation
cpunumber jdl attribute
output_dir the directory to put the file in

Returns: Temporary file name.
Create Delegation cream_endpoint, delegId
Description: Delegate user's proxy credentials,to be used later for job submissions.

Arguments: cream_endpoint the cream endpoint
delegId the delegation id string

Returns: nothing
Create Dummy Script output_dir
Description: Create a dummy bash script executing ls.

Arguments: output_dir the directory to put the file in

Returns: Temporary file name
Create Proxy password, vo, cert=None, key=None, time=None
Description: Create a user proxy.

Arguments: password the user's proxy password
vo for the voms extention.
cert non standard certificate path
key non standard key path
time the validity period of the proxy. Form: HH:MM

Returns: nothing.
Delegation Info deleg_endpoint, deleg_id
Description: Fetch the information regarding a delegation.

Arguments: deleg_endpoint the delegation endpoint of a cream service
deleg_id the delegation id

Returns: The commands output or the string "NOT IN STORAGE" if the specified
delegation id doesn't exist in the specified endpoint
Delete Osb Basedesturi Files jdl_path, gridftp_server, gridftp_path
Description: Read from a jdl file the list of files contained in the OutputSandBox and OutputSandboxBaseDestURI attribute.
Then delete these files with lcg-del.

Arguments: jdl_path path to the jdl file
gridftp_server the gridftp server in use
gridftp_path the path within the gridftp server

Returns: nothing.
Delete Osb Desturi Files jdl_path, gridftp_server
Description: Read from a jdl file the list of files contained in the OutputSandboxDestURI attribute.
Then delete these files with lcg-del.

Arguments: jdl_path path to the jdl file
gridftp_server the gridftp server in use

Returns: nothing.
Destroy Delegation deleg_endpoint, deleg_id
Description: Delete a delegation.
s
Arguments: deleg_endpoint the delegation endpoint of a cream service
deleg_id the delegation id

Returns: nothing.
Destroy Proxy
Description: Delete a user's proxy.

Arguments: none.

Returns: nothing.
Disable Cream Admin dn, ce_endpoint
Description: Remove the specified DN as a CREAM administrator (add it to /etc/grid-security/admin-list)

Arguments: dn Distinguished Name to be added
ce_endpoint the cream endpoint

Returns: Nothing (raises exception upon error)
Disable Submission ce_endpoint
Description: Disable the submission to the designated CREAM endpoint.

Arguments: ce_endpoint the cream endpoint

Returns: nothing (raises exception in case of error)
Enable Cream Admin dn, ce_endpoint
Description: Add the specified DN as a CREAM administrator (add it to /etc/grid-security/admin-list)

Arguments: dn Distinguished Name to be added
ce_endpoint the cream endpoint

Returns: Nothing (raises exception upon error)
Enable Submission ce_endpoint
Description:    Enable the submission to the designated CREAM endpoint.

Arguments:      ce_endpoint the cream endpoint

Returns:        nothing (raises exception in case of error)
Environment Jdl vo, output_dir
Description: Attribute checking jdl file. Set an environmental variable and check its value during runtime.

Arguments: vo virtual organisation
output_dir the directory to put the file in

Returns: Temporary jdl file name,temporary shell script file name.
Epilogue Jdl vo, output_dir
Description: Attribute checking jdl file. Use the epilogue argument and execute an epilogue script.

Arguments: vo virtual organisation
output_dir the directory to put the file in

Returns: Temporary jdl file name,temporary shell script file name.,temporary epilogue script.
Execute Noninteractive Ssh Com command, host, user, port=22
Description: Execute a non interactive command through ssh on another host
NOTE: Shell metacharacters are NOT recognized. Call them through 'bash -c'.
Example: '/bin/bash -c "ls -l grep LOG > log_list.txt"'

Arguments: host ssh host
port ssh port, 22 by default
user the user name to use for the ssh connection
command the command to execute

Returns: the command's output
Execute Uberftp Command uberftp_command, gridftp_server, gridftp_path
Description: Execute an uberftp command on a gridftp url

Arguments: uberftp_command one of cat,chgrp,chmod,dir,ls,mkdir,rename,rm,rmdir,size,stage
gridftp_server the gridftp server hostname
gridftp_path the path in the gridftp server

Returns: The output of the command
Fetch Output Files dest_dir, job_id, timeout=0
Description: Gather the files from a certain job,store them in the specified directory and return the list of files transfered
as they exist on the disk after the transfer operation.
Note: Target directory must be empty!Any existing output files with the same name are overwritten!

Arguments: dest_dir directory to store the files locally
job_id the job's cream job id
timeout time to wait before fetching output (used to wait until files are uploaded to CE's gridftp server)

Returns: a list with the files transfered
File Should Contain file_path, search_string
Description: Check whether the given file contains the given string.
Note that this must be used for small files,since it reads it all in memory.

Arguments: file_path the path pointing to the file
search_string the string searching for

Returns: True or False.
Files Should Not Be Empty input_files
Description: Copy a file from an SE to localhost,using lcg-cp.

Arguments: files file(s) to be checked for being non-empty
this argument can either be a list of strings or a single string

Returns: Nothing - raises error if any of the files given are empty
Get Arch Smp Size ce_endpoint, ldap_port
Description: Get the published value of the GlueHostArchitectureSMPSize attribute

Arguments: ce_endpoint the CREAM endpoint
ldap_port the ldap port to use

Returns: the value of the attribute
Get Current Status job_id
Description: Return the current status of a job,with the use of the glite-ce-job-status command.
This function will NOT wait until the job is in a final state.

Arguments: job_id the job id,as returned by the submit operation.

Returns: job's status as a string.
Get Current Status Verbose job_id, verbosity=2
Description: Return the current status of a job,with the use of the glite-ce-job-status command,with the given verbosity
This function will NOT wait until the job is in a final state.

Arguments: job_id as returned by the submit operation.
verbosity 0,1 or 2
Returns: a string.
Get Deleg Id From Status status
Description: Return the delegation id in use by a job

Arguments: status the job's verbose status

Returns: the delegation id (string)
Get Delegation Id jid
Description: Get the delegation id in use by a job

Arguments: jid CREAM job identifier

Returns: the delegation id string
Get Final Status job_id
Description: Return the final status of a job,with the use of the glite-ce-job-status command.
This command will wait until the job is in a final state.

Arguments: job_id the job id,as returned by the submit operation.

Returns: job's final status as a string.
Get Guid file_path, prefix=None
Description: Find all the guids of the format "guid:36 char string" in a file

Arguments: file_path the path to the file containing the guids (most probanly a DSUpload file)
this argument can either be a list of strings or a single string
prefix common prefix for all the files of the first argument, e.g.: /tmp/
This argument isn't mandatory

Returns: list with corresponding guids
Get Job Sb jid
Description: Find the gridftp url of the ISB of the given job

Arguments: jid job id returned by job submit operation

Returns: (gridftp server, gridftp path)
Get Osb jdl_path
Description: Read from a jdl file the list of files contained in the OutputSandBox attribute.
Arguments: jdl_path        path to the jdl file.
Returns: a list with the files in the output sandbox
Get Proxy Dn
Description:    Get user's proxy DN.

Arguments:      none.

Returns:        The user's proxy DN.
Get Status With Filter ce_endpoint, filter_type1, filter_data1, filter_type2=None, filter_data2=None
Description: Return the current statuses of jobs,with the use of the glite-ce-job-status command and a filter
This function will NOT wait until the job is in a final state.
filter_type2 and filter_data2 are optional

Arguments: ce_endpoint the cream endpoint being questioned
filter_type1/2 the filter being used, one of "from", "to", or "status"
filter_data1/2 the corresponding's filter data, either a date (as accepted from the glite
status command "YYYY-MM-DD HH:mm:ss" or a set of job statuses e.g. "DONE-OK:DONE-FAILED"

Returns: list with corresponding job ids and statuses
Get Time In Status Format
Description: Returns the time in the format accepted by glite-ce-job-status

Arguments: N/A N/A

Returns: string (containing the time-date in format YYYY-MM-DD HH:mm:ss)
Hostnumber Jdl vo, hostnumber, output_dir
Description: Attribute checking jdl file.Sets the jdl attribute "HostNumber" to the given number.

Arguments: vo virtual organisation
hostnumber jdl attribute
output_dir the directory to put the file in

Returns: Temporary file name.
Isb Baseuri Jdl vo, gridftp_server, gridftp_path, uploaded_file_name, output_dir
Description: File transfer checking jdl file. Store input files to a gridftp server,and use the jdl attribute
InputSandboxBaseURI to fetch them.

Arguments: vo virtual organisation
gridftp_server gridftp server to set ISBBaseURI
gridftp_path gridftp path to set ISBBaseURI
uploaded_file_name file name of the uploaded file
output_dir the directory to put the file in

Returns: Temporary file name.
Isb Client To Ce Jdl vo, output_dir
Description: File transfer checking jdl file. Stage input files from client to CE node.

Arguments: vo virtual organisation
output_dir the directory to put the file in

Returns: Temporary jdl file name,temporary shell script file name.
Isb Gsiftp To Ce Jdl vo, gridftp_server, gridftp_path, uploaded_file_name, output_dir
Description: File transfer checking jdl file. Store input files to a gridftp server,and use the jdl attribute
InputSandbox to fetch them.

Arguments: vo virtual organisation
gridftp_server gridftp server to set ISBBaseURI
gridftp_path gridftp path to set ISBBaseURI
uploaded_file_name file name of the uploaded file
output_dir the directory to put the file in

Returns: Temporary file name.
Job Status Should Be jid, status
Description: Check that the given job's status is the one given.

Arguments: jid job identifier
status the status comparing to

Returns: Nothing
Jobdbadminpurger jid, ce_endpoint, db_user, db_pass, middleware_version
Description: Purge a job using the JobDBAdminPurger.sh script, run on cream host

Arguments: jid the job to purge job id as returned by glite-ce-job-submit
ce_endpoint the host of the cream service
db_user cream's db user
db_pass aforementioned user's db pass
middleware_version the middleware version, either "EMI1" or "EMI2"

Returns: nothing
Lcg Cp vo, identifier, local_fname
Description: Copy a file from an SE to localhost,using lcg-cp.
Note: without the LCG_GFAL_INFOSYS variable set, this
keyword will NOT work.
Arguments: vo virtual organisation
identifier lfn or guid to download
Must include lfn: or guid: at the start
local_fname local file name to use

Returns: Nothing.
Lcg Cr vo, local_fname, storage_element, lfn
Description: Upload a file to an SE,using lcg-cr.

Arguments: vo virtual organisation
local_fname local file's name to upload
storage_element storage element to be used
lfn the logical file name to use

Returns: Nothing.
Lcg Del vo, lfn
Description: Delete all replicas of a file and its LFC entry,using lcg-del.

Arguments: vo virtual organisation
lfn logical file name to delete

Returns: Nothing.
Ldap Search host, port, search_base, search_filter
Description: Perform a simple ldapsearch. The option -LLL is always set.

Arguments: host the ldap host
port the ldap listening port
search_base the search base
search_filter the search filter

Returns: The commands output.
Lfc Ls lfn_path
Description: List an lfn,using lfc-ls.

Arguments: lfn_path imaginary lfc path

Returns: List containing all the files.
Lfc Mkdir lfn_path
Description: Create a folder in the LFC,using lfc-mkdir.

Arguments: lfn_path imaginary lfc path

Returns: Nothing.
Lfc Rmdir lfn_path
Description: Delete a folder in the LFC,using lfc-rm -r.

Arguments: lfn_path imaginary lfc path

Returns: Nothing.
List Jobs ce_endpoint
Description: List the jobs not purged by the specified cream endpoint for the user executing the command.

Arguments: ce_endpoint cream endpoint.

Returns: a list containing all the relevant job ids
List Proxy Sandbox ce_endpoint, job_status
Description: Check the presense (or not) of a delegation id in db queries contents

Arguments: ce_endpoint the cream endpoint
job_status a job's verbose status

Returns: a list with the contents of the directory
Localhost Output Jdl vo, output_dir
Description: File transfer checking jdl file. Stage files to the CE node (gsiftp://localhost).

Arguments: vo virtual organisation
output_dir the directory to put the file in

Returns: Temporary file name.
Modify Cream Config Xml ce_endpoint, attr, val
Description: Modify the given attribute of the cream config xml file

Arguments: ce_endpoint the cream endpoint
attr the option to change it's value
val the value to set it

Returns: Nothing (raises exception upon error or if attr isn't found)
Multiple Lcg Cp vo, identifiers, id_type, output_dir
Description: Copy a file from an SE to localhost,using lcg-cp.

Arguments: vo virtual organisation
identifiers list of lfns or guids to download
id_type either lfn or guid
output_dir the directory where the files will be placed

Returns: List of files downloaded
Osb Basedesturi Jdl vo, gridftp_server, gridftp_path, output_dir
Description: File transfer checking jdl file. Stage output files to a gridftp server,with the use of the
OutputSandboxBaseDestURI jdl attribute.

Arguments: vo virtual organisation
gridftp_server gridftp server for BaseDestURI
gridftp_path gridftp path for BaseDestURI
output_dir the directory to put the file in

Returns: Temporary file name.
Osb Desturi Jdl vo, gridftp_server, gridftp_path, output_dir
Description: File transfer checking jdl file. Stage output files to a gridftp server,with the use of the
OutputSandboxDestURI jdl attribute.

Arguments: vo virtual organisation
gridftp_server gridftp server to set DestURI
gridftp_path gridftp path to set DestURI
output_dir the directory to put the file in

Returns: Temporary file name.
Output Data Jdl vo, output_dir, gridftp_server=None, lfn_dir=None, lfc=None, infosys=None
Description: Attribute checking jdl file. Uses a combination of the output data attributes.

Arguments: vo virtual organisation
output_dir the directory to put the file in
lfc the value to set the environmental variable LFC_HOST on the WN
infosys the value to set the environmental variable LCG_GFAL_INFOSYS on the WN

Returns: Temporary jdl file name,temporary shell script file name.
Prologue Jdl vo, output_dir
Description: Attribute checking jdl file. Use the prologue argument and execute a prologue script.

Arguments: vo virtual organisation
output_dir the directory to put the file in

Returns: Temporary jdl file name,temporary shell script file name.,temporary prologue script.
Proxy Info
Description:    Read user's proxy info.

Arguments:      None.

Returns:        Command's output.
Purge All Jobs ce_endpoint
Description: Purge all the user's jobs in the given cream endpoint.

Arguments: ce_endpoint cream endpoint.

Returns: nothing.
Purge Job jid, ce_endpoint
Description: Purge the job with the corresponding jid at the given CREAM endpoint

Arguments: jid job id returned by job submit operation
ce_endpoint the CREAM endpoint

Returns: nothing.
Qdel Job jid, torque_host
Description: Manually qdel a job.

Arguments: jid job identifier
cream_host the server hosting torque, either ip or name
cream_admin_user a user exiting on the torque host, having admin priviledges

Returns: Nothing
Renew Proxy ce_endpoint, deleg_id
Description: Renew a delegated proxy in a CREAM endpoint

Arguments: ce_endpoint the CREAM endpoint
deleg_id delegation id to renew

Returns: nothing
Reset Cream Config Xml ce_endpoint
Description: Reset the cream config xml file to the original

Arguments: ce_endpoint the cream endpoint

Returns: Nothing
Reset Limiter Threshold ce_endpoint, middleware_version
Description: Reset the cream limiter perl script to the original unmodified one

Arguments: ce_endpoint The cream host
middleware_version The middleware version, either "EMI1" or "EMI2"

Returns: Nothing
Restart Cream ce_endpoint
Description: Restart the tomcat service, thus restarting cream

Arguments: ce_endpoint the cream endpoint

Returns: Nothing (raises exception upon error or if attr isn't found)
Resume All Jobs ce_endpoint
Description: Suspend all the user's jobs in the given cream endpoint.

Arguments: ce_endpoint cream endpoint.

Returns: nothing.
Resume Job job_id
Description: Resumes the given (previously suspended) job.

Arguments: job id as returned by the submit operation.

Returns: nothing.
Save Batch Job Submission Script Off ce_endpoint, version
Description: Modify blah_common_submit_functions.sh in order to NOT save the job submission script

Arguments: ce_endpoint the cream endpoint
version EMI1 or EMI2 (to determine the path of the script file)

Returns: Nothing (raises exception upon error)
Save Batch Job Submission Script On ce_endpoint, version
Description: Modify blah_common_submit_functions.sh in order to save the job submission script

Arguments: ce_endpoint the cream endpoint
version EMI1 or EMI2 (to determine the path of the script file)

Returns: Nothing (raises exception upon error)
Send Signal To Process pid, sigNo
Description: Send a process a certain signal.

Arguments: pid the proc's pid
sigNo signal number to send
Returns: nothing
Set Limiter Threshold thresh_name, thresh_value, ce_endpoint, middleware_version
Description: Set one threshold of the cream limiter perl script

Arguments: thresh_name The threshold's name
thresh_value The threshold's value
ce_endpoint The cream host
middleware_version The middleware version, either "EMI1" or "EMI2"

Returns: Nothing
Simple Jdl vo, output_dir
Description: Simple jdl file.Executes /bin/uname -a.

Arguments: vo virtual organisation
output_dir the directory to put the file in

Returns: Temporary file name.
Sleep Jdl vo, secs, output_dir
Description: Simple jdl file.Executes /bin/sleep for the defined number of seconds.

Arguments: vo virtual organisation
secs seconds to sleep
output_dir the directory to put the file in

Returns: Temporary file name.
Smpgranularity Jdl vo, smpgranularity, output_dir
Description: Attribute checking jdl file.Sets the jdl attribute "SMPGranularity" to the given number.

Arguments: vo virtual organisation
smpgranularity jdl attribute
output_dir the directory to put the file in

Returns: Temporary file name.
Start Old Blparser host
Description: Start the old blparser service

Arguments: host the service's host

Returns: Nothing (raises exception upon error)
Stop Old Blparser host
Description: Stop the old blparser service

Arguments: host the service's host

Returns: Nothing (raises exception upon error)
String Should Contain string, search_string
Description: Check whether the given string contains the given substring.

Arguments: string string haystack
search_string string needle

Returns: True or raise error
Submit And Wait jdl_path=None, ce_endpoint=None, delegId=None
Description: Submit a job and wait until it finishes.

Arguments: Same as submit_job.

Returns: (int,string) as jid and final job state.
Submit Job jdl_path, ce_endpoint, delegId=None
Description: Submit a job with automatic or explicit delegation and return it's job id.

Arguments: jdl_path path to the jdl file
ce_endpoint the cream endpoint,containing the queue
delegId if specified,uses the given delegation id, else it uses automatic delegation

Returns: the resulting cream job id as a string
Suspend All Jobs ce_endpoint
Description: Suspend all the user's jobs in the given cream endpoint.

Arguments: ce_endpoint cream endpoint.

Returns: nothing.
Suspend Job job_id
Description: Suspends the given job.

Arguments: job id as returned by the submit operation.

Returns: nothing.
Uberftp Upload gridftp_server, gridftp_path, local_file_path
Description: Upload a file to a gridftp server with uberftp

Arguments: gridftp_server the gridftp server hostname
gridftp_path the path in the gridftp server
local_file_path the full path of the local path

Returns: The output of the command
Unban User Argus dn, argus_host
Description: Un-Ban the specified DN using pap-admin cli on the argus host

Arguments: dn Distinguished Name to be added
argus_host the argus host

Returns: Nothing (raises exception upon error)
Unban User Gjaf dn, ce_endpoint
Description: Remove the specified DN as a CREAM banned user (remove it from /etc/lcas/ban_users.db)

Arguments: dn Distinguished Name to be added
ce_endpoint the cream endpoint

Returns: Nothing (raises exception upon error)
Validate Ce Service Info output
Description: Validates the output of a glite-ce-service-info command

Arguments: output  the output of the method ce_service_info()

Returns: Nothing (raises exception uppon non-validation)
Validate Glue ce_endpoint, port, bind, glue_version
Description: Run the glue validator for the designated CREAM endpoint

Arguments: ce_endpoint the cream endpoint
port the ldap listening port
bind the point where glue validator should bind
glue_version the glue version to be tested by glue validator

Returns: Nothing (raises exception upon error)
Validate Job Status output, verbosity
Description: Validates the output of a glite-ce-job-status command

Arguments: output the output of a job status command
verbosity the verbosity of the status command

Returns: Nothing (raises exception uppon non-validation)
Wait For Status jid, status, timeout=3600
Description: Wait for the given job to reach the given status.

Arguments: jid jod identifier
status the status waiting for
timeout timeout after which the operation fails (in seconds)
Defaults in one hour timeout.

Returns: Nothing
Wholenodes Jdl vo, wholenodes, output_dir
Description: Attribute checking jdl file.Sets the jdl attribute "WholeNodes" to the given value.

Arguments: vo              virtual organisation
wholenodes      jdl attribute
output_dir      the directory to put the file in

Returns: Temporary file name.
Wms Get Final Job Status job_id, verbosity=1
Description: Return the final status of a wms job,with the use of the glite-wms-job-status command.
This function WILL wait until the job is in a final state.

Arguments: job_id the job id,as returned by the submit operation.
verbosity the status operation's verbosity. One of 0,1,2,3

Returns: job's status as a string.
Wms Get Job Status job_id, verbosity=1
Description: Return the current status of a wms job,with the use of the glite-wms-job-status command.
This function will NOT wait until the job is in a final state.

Arguments: job_id the job id,as returned by the submit operation.
verbosity the status operation's verbosity. One of 0,1,2,3

Returns: job's status as a string.
Wms Job Logging Info job_id, verbosity=1
Description: Return the current status of a wms job,with the use of the glite-wms-job-status command.
This function will NOT wait until the job is in a final state.

Arguments: job_id the job id,as returned by the submit operation.
verbosity the status operation's verbosity. One of 0,1,2,3

Returns: job's status as a string.
Wms Job Submit jdl_path, wmproxy_endpoint=None, ce_endpoint=None, delegId=None
Description: Submit a job to the WMS, with automatic or explicit delegation and return it's job id.
NOTE: this method only supports some basic options of the Workload Management System!

Arguments: jdl_path path to the jdl file
ce_endpoint the cream endpoint,containing the queue
wmproxy_endpoint the wmproxy endpoint
delegId if specified,uses the given delegation id, else it uses automatic delegation

Returns: the resulting cream job id as a string

Changelog



!!!!!!!!!!!!!!!
Version 1.8 (1st EMI2 version)
!!!!!!!!!!!!!!!



====
LIBRARY
====
> Added "LCG_GFAL_VO..." to the outputdata jdl. Apparently, doesn't work without it
=> get_guid() returns an empty list if no guids are found, instead of raising an error.
=> restart_cream() auto-detects tomcat version (through SL version)
=> Added ldap_search() keyword
=> Added check_resource_bdii_published_values() keyword
=> Added wms_job_submit() keyword
=> Added wms_get_job_status() keyword
=> Added wms_get_final_job_status() keyword
=> Added wms_job_logging_info() keyword
=> Added check_wms_logging_info() keyword
=> Added stop_old_blparser() keyword
=> Added start_old_blparser() keyword
=> Added _get_ssh_connection() method
=> _enisc doesn't utilize password's anymore. It expects to be authenticated automagically, through ssh keys.
=> Removed execute_remote_mysql_query()


===========
TEST SUITE
===========
=> OutputData test case: check guids for existence and log them explicitely (was only implicit till now)
=> Automatic Proxy Purging Teardown test case: fixed delegation name
=> Added Explicit Delegation Teardown (used in the test case with the same name, to handle delegation destruction in case of error)
=> Added Resource BDII test case
=> Added Logging and Bookkeeping test case
=> Added Old BLParser Restart test case


==========
VARIABLES
==========
=> Added blparser_host
=> Added blparser_root_pass
=> Removed lrms_admin_pass and eliminated its usage
=> Removed lrms_admin_user and replaced it with root
=> Removed cream_root_pass and eliminated its usage
=> Removec blparser_root_pass and eliminated its usage


===================
DEPENDENCIES ADDED
===================
=> openldap-clients
=> whiptail


Check Also



Robot Framework Quick Start Guide https://twiki.cern.ch/twiki/bin/view/EMI/RobotFrameworkQuickstartGuide

Robot Framework Advanced Guide https://twiki.cern.ch/twiki/bin/view/EMI/RobotFrameworkAdvancedGuide

Robot Framework http://code.google.com/p/robotframework/

RIDE Editor for Robot Framework test suites http://code.google.com/p/robotframework-ride/

pybot --help

man cream_test

-- DimosthenesFioretos - 07-Oct-2011

Edit | Attach | Watch | Print version | History: r16 < r15 < r14 < r13 < r12 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r16 - 2012-08-13 - DimosthenesFioretosExCern
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback