Standard Compliance Testing

This page describes the available and planned test suites for standard compliance testing in EMI.

Available test suites

Testsuite Standard tested Short description
litmus HTTP and WebDAV litmus is a WebDAV server test suite, which aims to test whether a server is compliant with the WebDAV protocol as specified in RFC2518
nfs connectathon NFS4.1/pNFS Programs that can be used to test an implementation of the NFS Protocol. The tests run on a UNIX client and test server and client functions
pynfs NFS4.1/pNFS CITI maintains pynfs, a suite of several Python tools for NFS4, useful for testing purposes. Its primary components are a server tester, useful for testing server response to unusual error conditions and/or a badly behaving client
GlueValidator GLUE 2.0 The GLUE Validator is a simple python-based unit test suite that validates information against test cases. It works against both LDIF files and LDAP servers. In addition, a GLUE 2.0 LDAP schema validator is available
EMI CAR validator EMI CAR Test suite for validating EMI Compute Accounting Records
!StARval EMI StAR Test suite for validating EMI Storage Accounting Records
EMI ES Creation and Management Ports Testsuite EMI ES Management and Creation port types EMI ES Management and Creation testsuite based on the UNICORE Commandline Client (UCC)
EMI ES Delegation Port Testsuite EMI ES Delegation port type EMI ES Delegation testsuite
arcemiestest EMI ES Information port type EMI ES Information testsuite based on ARC
CANL OCSP, SHA2, others Auth validators for individual CANL packages

EMI 3 standard compliance testing

The table below list the MANDATORY standard compliance tests to be preformed by the product teams.

Description

  • Test id: ID
  • Product: The product to be tested.
  • Test suit: The test utility to be used.
  • Fields to be filled by PTs:
    • Test page: Link to the detailed test plan or a dedicated test area, if available. If special configurations of the testbed are needed, these should be stated here.
    • Comment: Any relevant information to share. E.g. note here if the test is run automatically e.g. via a unit testing or similar framework.
    • Test outcome: For executed tests this field should be either FAILED or SUCCESS. If available, please provide URLs of problem tickets (FAILED) or URL to the test report (SUCCESS).

Test id Product Test suite Test page Comment Test outcome
S1 UNICORE CE EMI-ES delegation  

Test written by lisa.zangrando@pdNOSPAMPLEASE.infn.it

There were 4 failures because UNICORE does NOT implement the following operations:
getInterfaceVersion(), getTerminationTime(), getVersion(), renewProxyReq()

FAILED

Test report: https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/standards_test_emi_delegation_UNICORE_report.txt

S2 CREAM EMI-ES delegation   Test written by lisa.zangrando@pdNOSPAMPLEASE.infn.it

SUCCESS

Test report: https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/DelegationService_report_CREAM.txt

S3 ARC CE EMI-ES delegation   Test written by lisa.zangrando@pdNOSPAMPLEASE.infn.it

SUCCESS

Test report:https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/standards_test_emi_delegation_ARC_report.txt

S4 UNICORE CE EMI-ES Info http://svn.nordugrid.org/trac/nordugrid/browser/arc1/trunk/src/hed/acc/EMIES Test written by aleksandr.konstantinov@fysNOSPAMPLEASE.uio.no SUCCESS Test report: https://savannah.cern.ch/file/EMI-ES%20ResourceInfo%20validation.txt?file_id=31610
S5 CREAM EMI-ES Info http://svn.nordugrid.org/trac/nordugrid/browser/arc1/trunk/src/hed/acc/EMIES Test written by aleksandr.konstantinov@fysNOSPAMPLEASE.uio.no SUCCESS Test report: https://savannah.cern.ch/file/EMI-ES%20ResourceInfo%20validation.txt?file_id=31610
S6 ARC CE EMI-ES Info http://svn.nordugrid.org/trac/nordugrid/browser/arc1/trunk/src/hed/acc/EMIES Test written by aleksandr.konstantinov@fysNOSPAMPLEASE.uio.no SUCCESS Test report: https://savannah.cern.ch/file/EMI-ES%20ResourceInfo%20validation.txt?file_id=31610
S7 UNICORE CE EMI-ES Crt&Mgmt   Test written by b.schuller@fz-juelichNOSPAMPLEASE.de

SUCCESS

https://savannah.cern.ch/task/download.php?file_id=31620 https://savannah.cern.ch/task/download.php?file_id=32018
S8 CREAM EMI-ES Crt&Mgmt   Test written by b.schuller@fz-juelichNOSPAMPLEASE.de

SUCCESS

https://savannah.cern.ch/task/download.php?file_id=32015 https://savannah.cern.ch/task/download.php?file_id=32017
S9 ARC CE EMI-ES Crt&Mgmt   Test written by b.schuller@fz-juelichNOSPAMPLEASE.de

SUCCESS

https://savannah.cern.ch/task/download.php?file_id=31621 https://savannah.cern.ch/task/download.php?file_id=32016
S10 dCache litmus-webdav     RUN - basic tests pass, some advanced tests fail. See report for details: http://svn.dcache.org/build/view/emi_certification/job/litmus%20-%20WebDAV%20tests/11/console
S11 DPM litmus-webdav     SUCCESS: https://savannah.cern.ch/task/index.php?34046
S12 StoRM litmus-webdav   The test is considered successfull since PROPPATCH and LOCK operations have not been implemented because not of interest to StoRM users SUCCESS: https://savannah.cern.ch/task/index.php?34065
S13 dCache NFS4.1    

SUCCESS

http://svn.dcache.org/build/view/emi_certification/job/nfs4.1_Tests/14/
S14 DPM NFS4.1     SUCCESS: https://savannah.cern.ch/task/index.php?34046
S15 StoRM NFS4.1   Test suite not suitable, StoRM uses GPFS NOT RUN
S16 Dcache STARval     SUCCESS
S17 DPM STARval     SUCCESS: https://savannah.cern.ch/task/index.php?34046
S18 StoRM STARval   Feature is not implemented yet NOT RUN
S19 UNICORE CE CAR validator https://github.com/guarise/emicar Test run manually. CAR records are not stored on disk as text files normally, so the input was taken from the log dumps which can be turned on on the rus-service. SUCCESS Test report: https://savannah.cern.ch/task/download.php?file_id=31548
S20 CREAM CAR validator https://github.com/guarise/emicar A 'minor' fail. If Processors or NodeCount information is not present, APEL publishes '0' instead of omitting the tag. FAIL
S21 ARC CE CAR validator https://github.com/guarise/emicar Generate CAR records with the JURA from the ARC CE reports and run the emicar validator. SUCCESS Test report: ARC_CE_CAR_validation
S22 CANL JAVA auth validator http://csrc.nist.gov/groups/ST/crypto_apps_infra/documents/PKITS.pdf All tests are run by JUnit, so are guaranteed to be passed. There is one real std. testsuite implemented: NIST http://csrc.nist.gov/groups/ST/crypto_apps_infra/pki/pkitesting.html (Path validation). Some optional parts which are unsupported by canl are not implemented. There are some functional tests for OCSP and SHA2 - linked too, though are not standards compliance tests. SUCCESS NIST report: https://savannah.cern.ch/task/download.php?file_id=31557 OCSP: https://savannah.cern.ch/task/download.php?file_id=31559 SHA2: https://savannah.cern.ch/task/download.php?file_id=31558
S23 CANL C auth validator https://twiki.cern.ch/twiki/bin/view/EMI/CaNlC-partEMI3FinalRC#C_Part OCSP and SHA2 use case tests run as a part of the normal caNl test Suite. The do not constitute a full standards compliance test, though. DONE
S24 CANL C++ auth validator http://svn.nordugrid.org/trac/workarea/browser/caNl%2B%2B/trunk/src/canl-c%2B%2B/test/ Test written by weizhong.qiang@fysNOSPAMPLEASE.uio.no. SHA2 test is included in test suite CredentialsTest.cpp; OCSP and OCSP stapling are included in test suite CredentialsTest.cpp and OCSPStaplingTest.cpp. NIST certificates suite has not been completely supported by canl c++, so the test suite ( NIST_PKITSTest.cpp) is not supposed to succeed for all of the NIST certificates. SUCCESS (for SHA2 and OCSP/OCSP stapling), test report: https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/sha2_and_ocsp_canlxx https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/ocsp_stapling_canlxx
S25 UNICORE glue2 validator   4 failing tests: three because of 'open enumeration' values that are wrongly checked by the validator FAIL, report https://savannah.cern.ch/task/download.php?file_id=32014
S26 DPM glue2 validator     FAIL, report and ticket at https://its.cern.ch/jira/browse/LCGDM-934
S27 dCache glue2 validator     RUN for glue1 and glue2 see reports: http://svn.dcache.org/build/view/emi_certification/job/glue-validator%20tests/7/console
S28 StoRM glue2 validator     RUN, report: glue_validator_storm_new.txt
S29 FTS3 glue2 validator   FTS3 is still prototype NOT RUN
S30 CREAM glue2 validator   Modified the types.py for extending the default values of Capability_t, InterfaceName _t and Platform_t GLUE2 types. This is needed for supporting the GLUE2 extensions introduced by the EMI-ES specification (e.g.'org.ogf.glue.emies.activityinfo', 'org.ogf.glue.emies.activitycreation' ... interface names, etc). SUCCESS, report: glue2_validator_report_CREAM.txt
S31 WMS glue2 validator     SUCCESS, report: glue_validator_wms.txt
S32 ARC CE glue2 validator https://wiki.nordugrid.org/index.php/Testing/ARC-CE#CE-F15:_infopublishing:_glue2_LDAP_schema Test fails because of missing hardcoded open enumerations; I think this is wrong, it should just give a warning. Some open enumerations are new and are not part of the validator code FAILED but false positives - see log: https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/S32_glue_validator_ARC_CE.log
S33 VOMS glue2 validator     SUCCESS,on SL6 report: glue_validator_voms.txt
S34 ARGUS glue2 validator   The glue-validator tests failed, but the errors are in the test themself, not in Argus. Reports: argus-EMI3_RC-glue-validator-SL6.txt argus-EMI3_RC-glue-validator-SL5.txt RUN
S35 L&B glue2 validator https://twiki.cern.ch/twiki/bin/view/EGEE/TestReportForLBEMI3FinalRC#Glue_Validator Fixes were required in L&B (fixed immediately) and Info Providers (https://savannah.cern.ch/bugs/?100126). All OK now. SUCCESS

-- JonKerrNilsen - 11-Nov-2011

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatext ARC_CE_CAR_validation r1 manage 13.0 K 2013-02-19 - 10:36 UnknownUser ARC CE - generated CAR record validation results
Texttxt DelegationService_report_CREAM.txt r1 manage 1.0 K 2013-02-07 - 16:59 LisaZangrando  
Unknown file formatlog S32_glue_validator_ARC_CE.log r1 manage 22.2 K 2013-02-19 - 12:24 FloridoPaganelli Test S32 - ARC CE GLUE2 Ldap validation results
Texttxt glue2_validator_report_CREAM.txt r1 manage 8.5 K 2013-02-07 - 16:20 LisaZangrando  
Texttxt glue_validator_argus.txt r1 manage 5.5 K 2013-02-15 - 17:23 DoinaCristinaAiftimiei VOMS & ARGUS glue_validator resutls
Texttxt glue_validator_storm.txt r1 manage 9.0 K 2013-02-15 - 17:32 DoinaCristinaAiftimiei StoRM glue_validator resutls
Texttxt glue_validator_storm_new.txt r1 manage 16.8 K 2013-02-19 - 23:59 DoinaCristinaAiftimiei StoRM glue_validator resutls
Texttxt glue_validator_voms.txt r1 manage 2.1 K 2013-02-15 - 17:23 DoinaCristinaAiftimiei VOMS & ARGUS glue_validator resutls
Texttxt glue_validator_wms.txt r1 manage 3.7 K 2013-02-15 - 15:14 DoinaCristinaAiftimiei WMS glue_validator resutls
Unknown file formatext ocsp_stapling_canlxx r1 manage 7.7 K 2013-02-16 - 17:24 UnknownUser Test report for canlxx (SHA2 and OCSP support)
Unknown file formatext sha2_and_ocsp_canlxx r1 manage 5.3 K 2013-02-16 - 17:18 UnknownUser Test report for canlxx (SHA2 and OCSP support)
Texttxt standards_test_emi_delegation_ARC_report.txt r1 manage 0.7 K 2013-02-20 - 12:01 LisaZangrando  
Texttxt standards_test_emi_delegation_UNICORE_report.txt r1 manage 1.3 K 2013-02-26 - 17:31 LisaZangrando  
Edit | Attach | Watch | Print version | History: r37 < r36 < r35 < r34 < r33 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r37 - 2013-06-06 - JonKerrNilsen
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback