APEL Parsers Test Plan (in progress)

EMI Component Description and Version

The APEL parser service is described in the APEL Parser Reference Card

Unit testing, deployment tests and functionality tests are common for the different APEL parsers. Regression tests can be common or specific for a parser flavour.

Unit Tests

TO BE COMPLETED

Deployment Tests

Installation

  • The YUM repository needed to install the component is:
  • The YUM command needed to test a clean installation is: yum install glite-apel-batchsystem
  • This is the first release of the APEL Parsers under EMI, so no upgrade installation instructions are available.

Configuration

The APEL parsers are configured with YAIM as part of the CE, CREAM-CE or batchsystem_utils service. For that reason, configuration of the APEL parser in this test plan with YAIM is out of scope.

System Tests

Basic Functionality Tests

F1. Running the parser with DBDeleteProcessor set to cleanAll="yes": implemented
The parser deletes all the records from all the accounting tables (GkRecords, EventRecords, MessageRecords, LcgRecords, SpecRecords, LcgProcessedFiles, RepublishInfo and BlahdRecords).

F2. Running the parser with DBDeleteProcessor set to cleanAll="no": implemented
The parser deletes any processed records (with Processed=1) from GkRecords, EventRecords, MessageRecords and BlahdREcords only. The rest of the tables remain unchanged.

F3. Running the parser with DBProcessor set to inspectTables="yes": implemented
The parser checks all the accounting tables for differences in schema. It then alters the schema if needed.

F4. Running the parser with DBProcessor set to inspectTables="no": implemented
The parser does not check the schema of any table.

F5. Running the parser with CPUProcessor reading from standard port 2170

F6. Running the parser with CPUProcessor reading from standard port 2135

F7. Running the parser with EventLogProcessor set to subdirectories="yes": implemented
The parser reads any batch system log files in the directory specified and in all subdirectories.

F8. Running the parser with EventLogProcessor set to subdirectories="no": implemented
The parser reads any batch system log files in the directory specified but doesn't read any subdirectories.

F9. Running the parser with EventLogProcessor set to reprocess="yes": implemented

F10. Running the parser with EventLogProcessor set to reprocess="no": implemented

F11. Running the parser with EventLogProcessor using extra files: implemented

F12. Running the parser with BlahdLogProcessor set to subdirectories="yes": implemented

F13. Running the parser with BlahdLogProcessor set to subdirectories="no": implemented

F14. Running the parser with BlahdLogProcessor set to reprocess="yes": implemented

F15. Running the parser with BlahdLogProcessor set to reprocess="no": implemented

F16. Running the parser with EventLogProcessor using grid_jobmap prefix: implemented

F17. Running the parser with EventLogProcessor using a different prefix: implemented

Regression Tests

R1. #23796: Too many open files (common)
1. Reproduce the bug in previous version of glite-apel-core and glite-apel-batchsystem.
  • Download the previous versions of glite-apel-core and glite-apel-batchsystem.
wget http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.updates/glite-apel-core-2.0.11-0.noarch.rpm
wget http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.release/glite-apel-lsf-2.0.10-1.noarch.rpm
wget http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.release/glite-apel-sge-2.0.8-1.noarch.rpm
wget http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.release/glite-apel-pbs-2.0.5-2.noarch.rpm
Note: There is not glite-CONDOR_utils in gLite 3.2.

  • Reduce ulimit to a smaller number (like 500).
ulimit -n 500

  • Create 1000 log files for testing (see tar attached to page)
./create_files.sh lsb.acct. lsb.acct
./create_files.sh history. history
../create_files.sh "" 20100816
./create_files.sh accounting. accounting

  • Run the parser with the following configuration

<?xml version="1.0" encoding="UTF-8"?>

<ApelConfiguration enableDebugLogging="yes">

    <SiteName>TEST_SITE</SiteName>
    <DBURL>jdbc:mysql://cano.esc.rl.ac.uk:3306/accounting</DBURL>
    <DBUsername>accounting</DBUsername>
    <DBPassword>apel_password</DBPassword>

    <EventLogProcessor>

        <SubmitHost>testce</SubmitHost>
        <Logs searchSubDirs="no" reprocess="no">
            <Dir>location_of_the_testing_log_files</Dir>
        </Logs>
        <Timezone>UTC</Timezone>
    </EventLogProcessor>
</ApelConfiguration>

2. Update to emi-batchsystem-utils

  • Uninstall glite-apel-core and glite-apel-batchsystem
  • Yum install emi-batchsystem-utils
  • Run the parser with the same configuration

R2. #30041: Problem in parsing lsf accounting logs (glite-apel-lsf)

R3. #64169: APEL LSF parser fails on Java 64bit (glite-apel-lsf)

R4. #65723: APEL assumes cluster and subcluster ID are the same as the CE hostname (common)

R5. #72270: APEL parser automatically adds a check table for the BlahdRecords table (common)

R6. #73603: Logging of BlahLogProcessor too verbose (common)

R7. #76204: APEL LSF parser to normalise CPU/WCT data with hostFactor value per job (glite-apel-lsf)

Performance and Scalability Tests

NA

Standard Compliance and Conformance Tests

NA

Inter-component Tests

NA
Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatgz create_files.tar.gz r1 manage 133.2 K 2011-03-30 - 17:13 CristinaDelCanoNovalesExternal Script to create log files for testing bug #23796
Edit | Attach | Watch | Print version | History: r13 | r7 < r6 < r5 < r4 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r5 - 2011-03-30 - CristinaDelCanoNovalesExternal
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback