L&B Test Plan

Service Description

Logging and Bookkeeping (LB for short) is a Grid service that keeps a short-term trace of Grid jobs as they are processed by individual Grid components.

The LB was initially developed in the EU DataGrid project as a part of the Workload Management System (WMS). The development continued in the EGEE project (EGEE-II, EGEE-III respectively), where LB became an independent part of the gLite middleware.

L&B-related documentation can be found in the EGEE Pproject's CVS. For convenience, documentation is also available for download throught the following links:

TestPlan test suite documentation:

Features/Scenarios to be tested

Service Ping Test (implemented)
This test checks that all services required to provide logging and querying capability are configured and running in the environment.

Normal workflow – correct input
Check if all services are accessible and listening where expected.

Pass/Fail Criteria
Pass: All services are listening as expected
Fail: One or more services are not available

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Job Registration (implemented)
Test if jobs can be correctly registered.

Normal workflow – correct input
Register a job

Pass/Fail Criteria
Pass: Job has been registered and assigned a Job ID
Fail: Registration failed

Error workflow – erroneous input
Try registering a job with the same Job ID and the EDG_WLL_LOGLFLAG_EXCL flag set.

Pass/Fail Criteria
Pass: Registration of a recycled Job ID refused
Fail: Registration of a recycled Job ID allowed

Event Delivery (implemented)
Test if events are being correctly delivered.

Normal workflow – correct input
Register a job and generate events. Test if events are delivered correctly and job state changes accordingly.

Pass/Fail Criteria
Pass: Events get delivered and job state changes as expected.
Fail: Events do not get delivered, job state does not change as per the state diagram.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

HTTPs Interface (implemented)
Getting lists and status information for jobs via the HTTPs interface.
_Note: While this test is not implemented yet, it is extremely easy to carry out manually._

Normal workflow – correct input
Register a job. Get a list of known jobs from the HTTPs interface (possibly check whether the listing contains the test job or, inded, any job). Get the status of the job using the HTTPs interface (using the jobID as an URL).

Pass/Fail Criteria
Pass: Job IDs were present in the listing and it was possible to download status information for the test job registered previously.
Fail: HTTPs interface was not accessible, test failed to download the list of jobs or

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Local Logger Detached (Standalone) Operation (implemented)
Test if local logger running in a detached (standalone) mode accepts events and stores them correctly for future delivery

Normal workflow – correct input
Run a Logger detached, register events, and test if they get accepted and stored properly.

Pass/Fail Criteria
Pass: Event logging works without blocking program execution, local logger caches events as expected.
Fail: Unable to log events or events get accepted but are not stored for future delivery

Error workflow – erroneous input
N/A

Pass/Fail Criteria
NA/

Interlogger Recovery (implemented)
Test if the Interlogger processes cached events correctly once it recovers from crash/loss of connectivity

Normal workflow – correct input
At startup, point the Interlogger at events cached by the local logging service. Test if the events get processed and delivered correctly.

Pass/Fail Criteria
Pass: Cached events get delivered correctly, job state changes accordingly.
Fail: Cached events have not been delivered.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Job State Computation (implemented)
Test if job states are being computed es expected.

Normal workflow – correct input
Register a job, then log a series of events. Watch the job state change accordingly as the events are being logged.

Pass/Fail Criteria
Pass: Job states change correctly, each event results in the appropriate state change.
Fail: Observed behavior contradicts the state diagram, events do not get delivered or fail to trigger state changes as expected.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Proxy-based Event Delivery (implemented)
Test if events are being correctly delivered via L&B proxy.

Normal workflow – correct input
Register a job and generate events. Test if events are delivered correctly through the L&B proxy, and that job state changes accordingly.

Pass/Fail Criteria
Pass: Events get delivered and job state changes as expected.
Fail: Events do not get delivered, job state does not change as per the state diagram.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Web Service Interface (implemented)
Query the server through the WS interface.

Normal workflow – correct input
Register a job and query the job status and job log through the WS interface.

Pass/Fail Criteria
Pass: The server returned information as expected (job in a submitted state, registration event present in the job log).
Fail: The server failed to provide the information or the information was incorrect.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Notification Delivery (implemented)
Test if L&B notifications are delivered correctly.

Normal workflow – correct input
Register a job, register a notification and start receiving notifications. Log events concerning that job and test if notifications are being delivered as expected.

Pass/Fail Criteria
Pass: Notifications have been delivered as expected.
Fail: Notifications have not been delivered.

Error workflow – erroneous input
Generate events not matching the notification's criteria.

Pass/Fail Criteria
Pass: No notifications have been generated for events not matching the criteria.
Fail: Notifications were generated even for events not matching the criteria

Stream Notification Delivery (implemented)
Test if L&B stream (bootstrap) notifications are delivered correctly.

Normal workflow – correct input
Register a job and log events concerning that job. Create the stream notification and start receiving notifications, test if notifications were being delivered as expected.

Pass/Fail Criteria
Pass: Notifications have been delivered as expected.
Fail: Notifications have not been delivered.

Error workflow – erroneous input
Generate events not matching the notification's criteria.

Pass/Fail Criteria
Pass: No notifications have been generated for events not matching the criteria.
Fail: Notifications were generated even for events not matching the criteria

Changing Notification Criteria (implemented)
Test if L&B notifications are delivered correctly after notification criteria changed.

Normal workflow – correct input
Register a job, register a notification, (check that the notification works,) change notification criteria (e.g. job ID), and start receiving notifications. Log events matching the criteria and test if notifications are being delivered as expected.

Pass/Fail Criteria
Pass: Notifications have been delivered as expected.
Fail: Notifications have not been delivered.

Error workflow – erroneous input
Generate events not matching the updated criteria. Check that notifications have not been sent out for such events.

Pass/Fail Criteria
Pass: No notifications have been generated for events not matching the criteria.
Fail: Notifications were generated even for events not matching the criteria

Delayed Notification Delivery (implemented)
Test if L&B notifications are cached correctly and delivered appropriately once the client starts.

Normal workflow – correct input
Register a job, register a notification. Log events concerning that job and test if notifications are delivered as expected once the client starts listening.

Pass/Fail Criteria
Pass: Notifications have been delivered as expected.
Fail: Notifications have not been delivered.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

L&B Server Purge (implemented)
Test if L&B Server purge works correctly. Also test if purged jobs are remembered (recognized as having once existed).

Normal workflow – correct input
Purge jobs from the server. Try a full purge as well as conditioned purges. Check the state of a purged job.

Pass/Fail Criteria
Pass: Server database has been purged properly on all occasions, querying a purged job returns a 'purged' state.
Fail: Failed to purge jobs that should have been purged, purged jobs that should not have been purged, or did not recognize the previous existence of a job.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

L&B vs. real-world WMS (implemented)

Normal workflow – correct input
  1. Submit a simple hello-world type job.
  2. Submit a simple job and cancel it.
  3. Submit a collection of simple jobs.
  4. Submit a collection and cancel it.
In all above cases: Watch the life cycle. Check the resulting state (Cleared or Cancelled). Check events received in the course of the job's execution; events from all relevant components must be present (NS, WM, JC, LM, and LRMS).

Pass/Fail Criteria
Pass: Jobs were submitted. Cancel operation worked where applicable. Resulting state was as expected (Cleared or Cancelled). Events were received from all components as expected.
Fail: Failure to achieve results outlined above (unable to submit a job, wrong state reached, events were not received from all components as expected). Note: Events do not necessarily need to arrive in the correct order. Events recieved out of order do not constitute a failed test!

Error workflow – erroneous input
  1. Submit a simple job that is sure to fail.
  2. Submit a collection of jobs, one of which is sure to fail.

Pass/Fail Criteria
Pass: Jobs were submitted. Resulting state was as expected (Aborted).
Fail: Correct state was not reached.

Testing BDII Response (implemented)
Check if an query to the BDII returns correct answers.

Normal workflow – correct input
Query the ldap service on an L&B machine. Parse results to make sure the expected information is returned. In particular, make sure that service version is reported correctly.

Pass/Fail Criteria
Pass: Information returned, matching expectations.
Fail: No response to query, incomplete, or erroneous.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Testing ACL settings (implemented)
Test ACL modification.

Normal workflow – correct input
Register a job, register a ChangeACL event to modify the job's ACL. Check job status information to see that the event was interpreted correctly.
Note: This test does not attempt to actually test access control since using two identities within a single automated tool is rather challenging. Only the correct interpretation of the ChangeACL event is being tested.

Pass/Fail Criteria
Pass: The requested change has been applied to the ACL appropriately.
Fail: The event has not been delivered, or was not interpreted correctly.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Testing the SandBox Transfer State Machine (implemented)
This test makes sure that logging sandbox transfers works properly.

Normal workflow – correct input
  1. Register a compute job.
  2. Register input sandbox trasfer.
  3. Register output sandbox transfer.
  4. Generate events to trigger job state changes in one of the sandbox transfer jobs.
    1. Start the transfer and check that state has changed appropriately.
    2. Finish the transfer and check that state has changed appropriately.
  5. Check that the compute job and its sandbox transfer jobs link up correctly.

Pass/Fail Criteria
Pass: Sandbox Transfer jobs properly registered, states following the transfer procedure correctly, job IDs set correctly for all jobs.
Fail: Any failure (no registration, bad job type, failed state changes, job IDs of related jobs not known or reported)

Error workflow – erroneous input
Use another sandbox transfer job registered above to start, then fail the transfer and check that this is reflected by the resulting transfer job status.

Pass/Fail Criteria
Pass: Sandbox Transfer jobs properly registered, states following the transfer procedure correctly.
Fail: Any failure (no registration, bad job type, failed state changes)

Testing Statistic Functions (implemented)
Test if statistics provided by L&B to WMS work properly

Normal workflow – correct input
  1. Register a series of jobs.
  2. Generate events to push jobs to a given state.
  3. Run statistics function to calculate rate of jobs reaching that state.
  4. Query for average time needed by testo jobs to go from one state to another.
  5. Check if the statistics returned reasonable results.

Pass/Fail Criteria
Pass: Reasonable values were returned.
Fail: The test returned errors, negative values or other non-realistic output.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Testing Multi-Threaded Operation (implemented)
Test if L&B client works in multi-threaded environment, at least in essence.

Normal workflow – correct input
  1. Register a series of jobs.
  2. Run a client using multiple threads to query the server simultaneously.
  3. Check if all threads finished OK.

Pass/Fail Criteria
Pass: Test did not hang, all threads finished correctly.
Fail: The test hung, ended with a segmentation fault or a similar error. Correct L&B error messages, such as connection refused due to server overload, are acceptable if produced occasionally.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Test L&B Harvester (implemented)
Test if L&B harvester receives and stores notifications correctly.

Normal workflow – correct input
  1. Run the L&B harvester, configured to receive relevant information.
  2. Register jobs and generate events.
  3. Check that resulting notifications are properly received and information stored by the Harvester.

Pass/Fail Criteria
Pass: All expected information was received.
Fail: Any problem occurred.

Test L&B Nagios probe (implemented)
Test if the L&B nagios probe runs correctly

Normal workflow – correct input
  1. Run the probe from command line.
  2. Check if the probe returns correct text and exit value.

Pass/Fail Criteria
Pass: The probe performed as expected (returned OK for a running server or, if justified, another consistent information for a server in a different state).
Fail: Any problem occurred. Interpretting and reporting a malfunction correctly is not considered a problem on the nagios probe side.

Error workflow – erroneous input
N/A

Pass/Fail Criteria
N/A

Features not to be tested

Querying the L&B API

There is no test focusing specifically on querying the L&B server API; however, such tests are inherently included in scenarios defined above as all of them need to test their outcome by generating input first and querying for results thereafter.

Tests

There is no scenario to test if the above tests test the behavior of the service correctly. This is left completely to the authors and users of the tests.

L&B Stress Testing

L&B components support stress testing options. There is a separate document dealing with L&B Performace/Stress Testing.

Regression testing

bug #55482: LB server version not published correctly (Implemented)

Bug link https://savannah.cern.ch/bugs/?55482
Test Run a query over LDAP, check if a correct version is returned

-- ZdenekSustr – 24 Feb 2009

Edit | Attach | Watch | Print version | History: r23 < r22 < r21 < r20 < r19 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r23 - 2011-10-06 - ZdenekSustr
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright & by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback