APEL Parsers Test Plan

EMI Component Description and Version

The APEL parser service is described in the APEL Parser Reference Card

Unit testing, deployment tests and functionality tests are common for the different APEL parsers. Regression tests can be common or specific for a parser flavour.

Unit Tests

Unit testing for the APEL components is done as part of the build.

Deployment Tests

Installation

• The YUM repository needed to install the component is:
• The YUM command needed to test a clean installation is: yum install glite-apel-batchsystem
• This is the first release of the APEL Parsers under EMI, so no upgrade installation instructions are available.

Configuration

The APEL parsers are configured with YAIM as part of the CE, CREAM-CE or batchsystem_utils service. For that reason, configuration of the APEL parser in this test plan with YAIM is out of scope.

System Tests

Basic Functionality Tests

F1. Running the parser with DBDeleteProcessor set to cleanAll="yes": implemented

The parser deletes all the records from all the accounting tables (GkRecords, EventRecords, MessageRecords, LcgRecords, SpecRecords, LcgProcessedFiles, RepublishInfo and BlahdRecords).

F2. Running the parser with DBDeleteProcessor set to cleanAll="no": implemented

The parser deletes any processed records (with Processed=1) from GkRecords, EventRecords, MessageRecords and BlahdREcords only. The rest of the tables remain unchanged.

F3. Running the parser with DBProcessor set to inspectTables="yes": implemented

The parser checks all the accounting tables for differences in schema. It then alters the schema if needed.

F4. Running the parser with DBProcessor set to inspectTables="no": implemented

The parser does not check the schema of any table.

F5. Running the parser with CPUProcessor reading from standard port 2170

F6. Running the parser with CPUProcessor reading from standard port 2135

F7. Running the parser with EventLogProcessor set to subdirectories="yes": implemented

The parser reads any batch system log files in the directory specified and in all subdirectories.

F8. Running the parser with EventLogProcessor set to subdirectories="no": implemented

The parser reads any batch system log files in the directory specified but doesn't read any subdirectories.

F9. Running the parser with EventLogProcessor set to reprocess="yes": implemented

F10. Running the parser with EventLogProcessor set to reprocess="no": implemented

F11. Running the parser with EventLogProcessor using extra files: implemented

F12. Running the parser with BlahdLogProcessor set to subdirectories="yes": implemented

F13. Running the parser with BlahdLogProcessor set to subdirectories="no": implemented

F14. Running the parser with BlahdLogProcessor set to reprocess="yes": implemented

F15. Running the parser with BlahdLogProcessor set to reprocess="no": implemented

F16. Running the parser with EventLogProcessor using grid_jobmap prefix: implemented

F17. Running the parser with EventLogProcessor using a different prefix: implemented

F18. Running the LSF parser with EventLogProcessor using useScalingFactor="yes": implemented

F19. Running the LSF parser with EventLogProcessor using useScalingFactor="no": implemented

Regression Tests

R1. #23796: Too many open files (common)

1. Reproduce the bug in previous version of glite-apel-core and glite-apel-batchsystem.

• Download the previous versions of glite-apel-core and glite-apel-batchsystem.

wget http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.updates/glite-apel-core-2.0.11-0.noarch.rpm
wget http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.release/glite-apel-lsf-2.0.10-1.noarch.rpm
wget http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.release/glite-apel-sge-2.0.8-1.noarch.rpm
wget http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.release/glite-apel-pbs-2.0.5-2.noarch.rpm

Note: There is not glite-CONDOR_utils in gLite 3.2.

• Reduce ulimit to a smaller number (like 500).

ulimit -n 500

• Create 1000 log files for testing (see tar attached to page)

./create_files.sh lsb.acct. lsb.acct
./create_files.sh history. history
../create_files.sh "" 20100816
./create_files.sh accounting. accounting

• Run the parser with the following configuration

<?xml version="1.0" encoding="UTF-8"?>

<ApelConfiguration enableDebugLogging="yes">

    <SiteName>TEST_SITE</SiteName>
    <DBURL>jdbc:mysql://cano.esc.rl.ac.uk:3306/accounting</DBURL>
    <DBUsername>accounting</DBUsername>
    <DBPassword>apel_password</DBPassword>

    <EventLogProcessor>

        <SubmitHost>testce</SubmitHost>
        <Logs searchSubDirs="no" reprocess="no">
            <Dir>location_of_the_testing_log_files</Dir>
        </Logs>
        <Timezone>UTC</Timezone>
    </EventLogProcessor>
</ApelConfiguration>

2. Update to emi-batchsystem-utils

• Uninstall glite-apel-core and glite-apel-batchsystem
• Yum install emi-batchsystem-utils
• Run the parser with the same configuration

R2. #30041: Problem in parsing lsf accounting logs (glite-apel-lsf): implemented

R3. #65723: APEL assumes cluster and subcluster ID are the same as the CE hostname (common)

R4. #72270: APEL parser automatically adds a check table for the BlahdRecords table (common)

• Enable the query logging in the host where the testing database is installed (Adding the line log=/tmp/queries.log to /etc/my.cnf and restarting mysqld)
• Run test F3
• Check that there is no CHECK TABLE BlahdRecords in /tmp/queries.log

R5. #73603: Logging of BlahLogProcessor too verbose (common): implemented

R6. #76204: APEL LSF parser to normalise CPU/WCT data with hostFactor value per job (glite-apel-lsf): replaced by F18

R7. #79175: EMI APEL path for configuration file wrong (common): implemented

R8. #79417: glite-apel-lsf is not FHS compliant (glite-apel-lsf)

• List the files in the rpm

rpm -ql glite-apel-lsf

It should return:

/etc/glite-apel-lsf
/etc/glite-apel-lsf/log4j.properties
/etc/glite-apel-lsf/parser-config.xml
/usr/bin
/usr/bin/apel-lsf-log-parser
/usr/share/doc
/usr/share/doc/glite-apel-lsf
/usr/share/doc/glite-apel-lsf/LICENSE
/usr/share/doc/glite-apel-lsf/html
/usr/share/doc/glite-apel-lsf/html/allclasses-frame.html
/usr/share/doc/glite-apel-lsf/html/allclasses-noframe.html
/usr/share/doc/glite-apel-lsf/html/constant-values.html
/usr/share/doc/glite-apel-lsf/html/deprecated-list.html
/usr/share/doc/glite-apel-lsf/html/help-doc.html
/usr/share/doc/glite-apel-lsf/html/index-all.html
/usr/share/doc/glite-apel-lsf/html/index.html
/usr/share/doc/glite-apel-lsf/html/org
/usr/share/doc/glite-apel-lsf/html/org/glite
/usr/share/doc/glite-apel-lsf/html/org/glite/apel
/usr/share/doc/glite-apel-lsf/html/org/glite/apel/lsf
/usr/share/doc/glite-apel-lsf/html/org/glite/apel/lsf/ApelLogParser.html
/usr/share/doc/glite-apel-lsf/html/org/glite/apel/lsf/class-use
/usr/share/doc/glite-apel-lsf/html/org/glite/apel/lsf/class-use/ApelLogParser.html
/usr/share/doc/glite-apel-lsf/html/org/glite/apel/lsf/package-frame.html
/usr/share/doc/glite-apel-lsf/html/org/glite/apel/lsf/package-summary.html
/usr/share/doc/glite-apel-lsf/html/org/glite/apel/lsf/package-tree.html
/usr/share/doc/glite-apel-lsf/html/org/glite/apel/lsf/package-use.html
/usr/share/doc/glite-apel-lsf/html/overview-tree.html
/usr/share/doc/glite-apel-lsf/html/package-list
/usr/share/doc/glite-apel-lsf/html/resources
/usr/share/doc/glite-apel-lsf/html/resources/inherit.gif
/usr/share/doc/glite-apel-lsf/html/stylesheet.css
/usr/share/java
/usr/share/java/glite-apel-lsf.jar

R9. #79517: SGE parser reprocesses every log file (glite-apel-sge): implemented

R10. #79898: GkRecords and MessageRecords not being inspected due to typo (common) => Tested in F3.

R11. #79917: Reprocessing a batch system log file doesn't overwrite the entries in the db (common) => Tested in F9.

R12. #84078: SGE parser reading wrong fields (glite-apel-sge): implemented

Performance and Scalability Tests

NA

Standard Compliance and Conformance Tests

NA

Inter-component Tests

NA

Test Scripts

* apel-parsers_tests.zip: Test scripts for APEL Parsers

Test Results

• test_results_18692_RC4.txt: Test results for patch #18692 (EMI1-RC4)
• test_results_21843.txt: Test results for patch #21843 (EMI1 update)