Purpose of this document

This is a page to be used as reference and guide to the work done at the National and Kapodestrian University of Athens,concerning the development of a CREAM testsuite with the use of Robot Framework.

What these tests consist of

The tesuite will be comprising a python module/library to be used with Robot Framework.The library will contain a set of keywords or "actions" which are the directives given to the Robot Framework process for execution.These actions will be in a range from the most basic tasks available from the tools at hand -e.g. the CREAM client api and cli utilities- to the most complex -and still usefull- cases,such as submitting a set of jobs with certain parameters and verifying the data- and work-flow they produce.

Data- and Work-flow Verification

The testsuite's needs give rise to the problem of how to actually test a scenario -which steps should be taken so as to verify the correct behaviour- and how to exactly verify that the data- and work-flow reported is actually what happens underneath,inside the CREAM backend.In order to satisfy these needs,the following ideas have been proposed:
1) Create a set of keywords/actions with maximum granularity.Thus,implement actions representing the most basic and fundamental CREAM functions provided by the client tools.In this way,by having many and solid building blocks,the "construction" of complex testing scenarios will be easier not only to create but also to escalate -and also,of course,debug- .
2) Support a non-basic variety of "error-success handling" or "level of testing".The concept behing this idea is that when a client tool reports something -e.g. a job status change- ,three steps should be taken: firstly,to check whether the actuall client command succeded (return value),secondly,to check whether the given output belongs to a set of predefined outputs which the command should create and finaly,thirdly,to check that what is reported from the succesful command's (1) output (2),is indeed present in the CREAM database.This is only an example of the "level of testing" concept,which will be utilized during the development of tests.

JDL Files Needed

1) Simple
2) ISB from UI to CE
3) ISB from gridftp server to CE
4) OSB from CE to gridftp server
5) OSB stored on CE (gsiftp://localhost - check glite-ce-job-output man page top)
6) ISB with InputSandboxBaseURI
7) OSB with OutputSandboxDestURI
8) OSB with OutputSandboxBaseDestURI
9) Job with OutputData JDL attributes considering all the possible combinations.


10) Job with prologue
11) Job with epilogue

12) Job with env. vars set through JDL (check if these vars are set in the job being run)
13) Job with CPUNumber and/or SMPGranularity and/or WholeNodes and/or HostNumber (TBD???)

Test Cases

1) Verify the proper functionality of the glite-ce-job-submit with different JDLs
2) E Verify job submissions with automatic and explicit delegations (-a,-D respectively)
3) Verify admin/non-admin has/doesn't have control over other people's jobs
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_9_4_How_to_define_a_CREAM_admi
4) Verify the proper functionality of the glite-ce-status command for all verbosity levels and all filters (--from,--to,--status)
5) Verify the proper functionality of the glite-ce-job-output command
6) Verify the proper functionality of the glite-ce-job-list command
7) Verify the proper functionality of the glite-ce-job-purge command (verify that the job has been removed from the CREAM DB and the job sandbox has been scratched)
8) Verify the proper functionality of the automatic job purger(verify that the job has been removed from the CREAM DB and the job sandbox has been scratched -
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_15_1_Automatic_job_purging
9) Verify the proper functionality of the gilte-ce-job-suspend/glite-ce-job-resume commands (verify that glite-ce-job-status that the job went through the HELD status)
10) Verify the proper functionality of the glite-ce-job-cancel command (verify that the job status is eventually cancelled (by user) )
11) Try to manually (bkill/qdel) a job (verify that eventually job status is DONE-FAILED or CANCELLED (by admin) )
12) Verify the limiter works and job submissions are indeed disabled when it detects that at least a threshold has been overtaken
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_12_Self_limiting_CREAM_behavio
13) Verify proper functionality of glite-ce-disable-submission/enable-submission.Verify they can be issued only by an admin.Verify that job submission is really disabled/enabled.
14) Verify the proper functionality of the glite-ce-allowed-subission command.
15) Verify the functionality of the glite-ce-service-info command (verify that all the relevant information is reported in the output)
(what is included in "relevant" information exactly?)
16) Verify the proper functionality of the glite-ce-proxy-renew command (verify that the proxy is indeed renewed on the WN (note that the proxy on the WN is supposed to be renewed when needed, i.e. before proxy expiration, not when the glite-ce-proxy-renew is issued) )
17) Try the functionality of the BLparser (it can be verified with the cancel operations) with both the new and old blparser
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#1_2_3_Choose_the_BLAH_BLparser_d
18) Try the functionality of CREAM with both Argus and gJAF authorizations

http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#1_2_2_Choose_the_authorization_m
19) Check if the resource bdii publishes all the relevant objectclasses for both glue1 and glue2.
Check with glue validator the corectness of the published information.
https://tomtools.cern.ch/confluence/display/IS/GLUEValidator
20) Check if the banning of a user works
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_9_2_How_to_block_ban_a_user
21) Try the file transfers between CE and WN with both available options
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_10_Input_and_Output_Sandbox_fi
22) Check if the automatic purging of expired proxies works
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_16_Proxy_purging

Test Cases Difficulty Classification

Easy


1) Verify the proper functionality of the glite-ce-job-submit with different JDLs (difficulty actually depends on jdl complexity)
2) Verify job submissions with automatic and explicit delegations (-a,-D respectively)
5) Verify the proper functionality of the glite-ce-job-output command
6) Verify the proper functionality of the glite-ce-job-list command
9) Verify the proper functionality of the gilte-ce-job-suspend/glite-ce-job-resume commands (verify that glite-ce-job-status that the job went through the HELD status)
10) Verify the proper functionality of the glite-ce-job-cancel command (verify that the job status is eventually cancelled (by user) )

Intermediate


11) Try to manually (bkill/qdel) a job (verify that eventually job status is DONE-FAILED or CANCELLED (by admin) )
15) Verify the functionality of the glite-ce-service-info command (verify that all the relevant information is reported in the output)
(what is included in "relevant" information exactly?)
14) Verify the proper functionality of the glite-ce-allowed-subission command.

Hard


16) Verify the proper functionality of the glite-ce-proxy-renew command (verify that the proxy is indeed renewed on the WN (note that the proxy on the WN is supposed to be renewed when needed, i.e. before proxy expiration, not when the glite-ce-proxy-renew is issued) )
19) Check if the resource bdii publishes all the relevant objectclasses for both glue1 and glue2.
Check with glue validator the corectness of the published information.
https://tomtools.cern.ch/confluence/display/IS/GLUEValidator

Unknown


3) Verify admin/non-admin has/doesn't have control over other people's jobs
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_9_4_How_to_define_a_CREAM_admi
4) Verify the proper functionality of the glite-ce-status command for all verbosity levels and all filters (--from,--to,--status)
7) Verify the proper functionality of the glite-ce-job-purge command (verify that the job has been removed from the CREAM DB and the job sandbox has been scratched)
8) Verify the proper functionality of the automatic job purger(verify that the job has been removed from the CREAM DB and the job sandbox has been scratched -
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_15_1_Automatic_job_purging
12) Verify the limiter works and job submissions are indeed disabled when it detects that at least a threshold has been overtaken
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_12_Self_limiting_CREAM_behavio
17) Try the functionality of the BLparser (it can be verified with the cancel operations) with both the new and old blparser)
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#1_2_3_Choose_the_BLAH_BLparser_d
18) Try the functionality of CREAM with both Argus and gJAF authorizations
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#1_2_2_Choose_the_authorization_m
21) Try the file transfers between CE and WN with both available options
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_10_Input_and_Output_Sandbox_fi
22) Check if the automatic purging of expired proxies works
http://wiki.italiangrid.org/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI1#2_16_Proxy_purging

JDL Files Descriptions and Testing Proposals

1) simple.jdl
Description: Execute /bin/uname -a.
Test: 1) Submit the jdl.
2) Final job status should be done-ok.

2) sleep100.jdl
Description: sleep 100 secs
Test: N/A

3) isb_baseuri.jdl
Description: Execute a bash shell script.The script is stored in a gridftp server.The jdl attribute InputSandboxBaseURI is used.
Arguments: 1) gridftp server 2) gridftp server path 3) local path to shell script
Test: 1) Upload executable script.
e.g.: lcg-cr -d srm://se01.isabella.grnet.gr/dpm/isabella.grnet.gr/home/see/dirgoediv/r_ls.sh /home/dfiore/cream/jdls/r_ls.sh
2) Submit jdl.
3) Final job status should be done-ok.

4) isb_client_to_ce.jdl
Description: Execute a bash shell script.The script is stored on the UI.The jdl attribute InputSandbox is used.
Arguments: 1) local path to shell script
Test: 1) Submit jdl.
2) Final job status should be done-ok.

5) isb_gsiftp_to_ce.jdl
Description: Execute a bash shell script.The script is stored in a gridftp server.The jdl attribute InputSandbox is used.
Arguments: 1) gridftp server 2) gridftp server path 3) local path to shell script
Test: 1) Upload executable script.
e.g.: lcg-cr -d srm://se01.isabella.grnet.gr/dpm/isabella.grnet.gr/home/see/dirgoediv/r_ls.sh /home/dfiore/cream/jdls/r_ls.sh
2) Submit jdl.
3) Final job status should be done-ok.

6) localhost_output.jdl
Description: Execute /bin/uname -a.Store std out and error streams on "gsiftp://localhost".
Arguments: None.
Test: 1) Submit jdl.
2) Download files produced from job.
3) Get output file list from jdl.
4) (2) & (3) should be equal.

7) osb_gsiftp_basedesturi.jdl
Descriptiion: Execute /bin/uname -a.Store std out and error streams on a gridftp server.The jdl attribute OutputSandboxBaseDestURI is used.
Arguments: 1) gridftp server 2) gridftp server path
Test: 1) Get output file list from jdl,base dest uri.
2) lcg-ls base dest uri and files shouldn't exist
3) Execute jdl
4) lcg-ls base dest uri and files should exist

8) osb_gsiftp_desturi.jdl
Description: Execute /bin/uname -a.Store std out and error streams on a gridftp server.The jdl attribute OutputSandboxDestURI is used.
Arguments: 1) gridftp server 2) gridftp server path 3) output sandbox file list
Test: 1) Get dest uri(s) from jdl file
2) lcg-ls dest uri(s) and they shouldn't exist
3) Execute jdl
4) lcg-ls dest uri(s) and they should exist

9) environment.jdl/environment.sh
Description: Execute a bash shell script.The script is stored localy.The jdl sets an environmental variable.The script prints the value of that variable.
Arguments: 1) local path to shell script
Test: 1) Submit jdl
2) Fetch output files
3) Check that in the job's std.out exists the env var as set in the jdl.

10) epilogue.jdl/epilogue.sh
Description: Execute two bash shell scripts,one for the job and the other as an epilogue.The epilogue script doesn't do anything.
Arguments: 1) path to job script 2) path to epilogue script
Test: 1) Submit jdl
2) Final job status should be done-ok (right?)

11) prologue.jdl/prologue.sh
Description: Execute two bash shell scripts,one for the job and the other as prologue.The prologue script creates a file.The job script confirms it is there (ls .).
Arguments: 1) path to job script 2) path to prologue script
Test: 1) Submit jdl
2) Fetch output files
3) Job's std.out file should contain the file created in the prologue script.

12) cpunumber.jdl
Description: Execute /bin/uname -a.Set the jdl attribute CPUNumber to a certain number.
Arguments: 1) the value CPUNumber will be set to.
Test: 1) Submit jdl and:
2) fail for 0
3) succeed for 1
4) greater values must be supported by the tested site,or they will fail.

13) hostnumber.jdl
Description: Execute /bin/uname -a.Set the jdl attribute HostNumber to a certain number.
Arguments: 1) the value HostNumber will be set to
Test: 1) Submit jdl and:
2) fail for 0
3) succeed for 1/2/3(/number of wn hosts in site???)

14) smpgranularity.jdl
Description: Execute /bin/uname -a.Set the jdl attribute SMPGranularity to a certain number.
Arguments: 1) the value SMPGranularity will be set to
Test: 1) Submit jdl and:
2) fail for 0
3) succeed for 1
4) greater values must be supported by the tested site,or they will fail

15) wholenodes.jdl
Description: Execute /bin/uname -a.Set the jdl attribute WholeNodes to true or false.
Arguments: 1) the boolean value WholeNodes will be set to
Test: 1) Submit jdl and:
2) succeed for True
3) succeed for False

16) hostnumber_smpgranularity_falseWholeNodes.jdl
Description: Execute /bin/uname -a.Set HostNumber and SMPGranularity to a random -correct- value,and WholeNodes to False.
Arguments: None.
Test: 1) Jdl submission should be aborted.

17) hostnumber_smpgranularity_trueWholeNodes.jdl
Description: Execute /bin/uname -a.Set HostNumber and SMPGranularity to a random -correct- value,and WholeNodes to True.
Arguments: None.
Test: 1) Submit jdl.
2) Final job status should be done-ok.

References

User's Guide
JDL Guide
Sysadmin Guide

-- DimosthenesFioretos - 07-Jun-2011

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2011-06-17 - DimosthenesFioretosExCern
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback