NEWS / WARNINGS
 
DOWNTIMES ANNOUNCEMENT
 

EMI Large Scale Acceptance Testbed

EMI large-scale testbeds are necessary to perform acceptance, interoperability and scalability tests required to validate release candidates in conditions as close as possible to real production environments.

1. Testbed News / Warnings

Check this section before using testbed or asking for support. Subscribe to the twiki notification system to get news on these issues.

2. Testbed Overview

Large Scale Testbeds are necessary to perform acceptance, interoperability and scalability tests.
These tests require a different implementation with respect to that of EMI Integration testing testbed, since the main objective is to stress the service under test reproducing conditions as similar as possible to real production environment. Approaching the goal above implementing a stable and monitored testbed infrastructure permanently replicating a production environment is both very difficult to achieve in terms of contributors availability and also very inefficient, being at the same time, expensive to maintain and never fit for any test in particular. Hence the proposal of two larger scale testing scenarios:

  1. Pre-Deployment across voluntary sites of software product at Release Candidate stage, with standard deployment modality and service usage in production environment.
    • Pre-Requirements:
      1. Standard deployment material: Documentation, Release Notes, Repository for installation
      2. Generic benchmarks, feedback forms
    • Expected Results:
      1. The aim is to get early feedback on RCx to improve components and fine-tune EMI procedures. Feedback is expected on:
        • Release Notes and Documentation
        • Installation / deployment
        • Production environment usage product functionality
  2. Large Scale specific product testing, according to software component test plans or for multiple components scalability/performance testing at a larger scale. The testbed definition and implementation is achieved through a "Demand and Supply" approach, meaning:
    • DEMAND: A request of test describing the testing scenario (resources needed, services involved, number of instances, number of users involved etc.etc.).
    • SUPPLY: A community of partners (NGI, EGI, PRACE, EMI internal partners, EMI user community partner or whoever interested) available to participate to specific tests campaigns with X effort, Y resources for Z time to test P1, P2,.., PN product. Declaring availability just for specific products testing will enhance the chance of having motivated contributors, which is a key aspect for the success of the model.

The role of EMI quality assurance WP to implement this model is to provide operational resources (coordination, communication channels, expertise in testbed setup, general purpose utilities (fake certificates or fake CA generation utility)) to match DEMAND and SUPPLY.

Agreement with EMI partners:

  • EGI and NGIs The informal and operational agreement with EGI project is summarized by the following EGI statement of intent

Site/Institute Type of Contribution Details Contact
INFN-Catania HW/sys admin effort 2 machines
User Interface
emidio.giorgio@ctNOSPAMPLEASE.infn.it
INFN-CNAF HW/sys admin effort 2 machines
WMS /LB
andrea.cristofori@cnafNOSPAMPLEASE.infn.it
D4Science-II HW/sys admin effort Will install some services; can provide around 4-5 machine for the tests
DPM, CREAM, Site BDII, WN, UI
andrea.manzi@cernNOSPAMPLEASE.ch
EGEE-SEE-CERT HW/sys admin effort EMI Products: CREAM, site-bdii,dpm head,dpm pool,glite wn,glite ui,stand alone torque node and a stand alone mysql node.
Number of Boxes: 13 boxes.
Dimosthenes Fioretos dfiore@nocNOSPAMPLEASE.edunet.gr, Institution: National and Kapodestrian University of Athens / GRNET
e-NMR Project Testing Effort Requires e-NMR VO to be configured on resources marco.verlato@pdNOSPAMPLEASE.infn.it

Key Performance Indicators for EMI Large Scale Acceptance Testbed is:

CODE KPI Description Method to Measure Estimated Targets
KSA2.3 SA2 Distributed Testbed Size Number of CPUs available for distributed testing through collaborations with external providers (NGIs sites, commercial providers, other projects, etc) Participating sites monitoring tools Year 1: 50 CPUs Year 2: 200 CPUs Year 3: 500 CPUs

To match the task KPI of 50 CPU for the first year, we keep track of the testbeds provided per campaign (x instances for y time).

3. Large Scale Testbed HOWTO for Volunteers Sites

Large Scale testbed success depends on the availability of volunteer contributors. Contributors can be either EMI partners or EMI product users ( EGI, NGIs, potential customers, Virtual Organizations etc.) wishing to provide some HW resources and/or collaborative effort to test EMI products they are interested in. Being aware of the fact that interest or effort/resources contribution is always limited in quantity and time, as a contributor you can declare your availability just for a subset of products for a limited number of campaigns per year. Even 2 weeks and few CPUs per year can be very helpful for EMI Testbed.
To become a EMI Large Scale testbed contributor contact EMI Testbed Staff providing the following information:

  1. Your Name, Role and Institution
  2. Which EMI products testing activity you may be interested in and why.
  3. Number of boxes ready to deploy EMI components you may provide and for how long
  4. Number of testing campaign (installation kept for y days during campaign) you may contribute to.
  5. System administration effort you may provide during the campaigns or availability to grant access to external administrators + minimal support from local system administrators.

  1. Default Pre-Deployment scenario:
    1. Deployment modality: please deploy the service according to your site typical deployment, this will provide us with use cases representative of real production environment. You are also requested to enable emi-testers.emi-eu.eu VO on your instance ( configuration details-> section 3.5 )
    2. Usage / Testing: this scenario will take advantage of service exposition to normal production usage.
    3. Feedback Provision: please fill the following Large Scale Testbed Pre-Deployment Template and mail it back to contact EMI Testbed Staff
  2. Demand/Supply scenario*:
    1. Deployment modality: Depending on the type of deployment request, you will be given specific testing case instruction which may sometime involve particular deployment scenarios, even though the fous is always on keeping close to real production needs. You are also requested to enable emi-testers.emi-eu.eu VO on your instance ( configuration details-> section 3.5 )
    2. Usage / Testing: Usage and Testing instruction will be provided in the testbed use case setup definition.
    3. Feedback Provision: Report about the test can be sent to contact EMI Testbed Staff.

WORK IN PROGRESS:

Component Name Requests for Large Scale Deployment (Y/N) Flavour Deployed at Site N Instances Notes
AMGA v.2.1.2          
APEL parsers v.0.0.0          
APEL publisher v.3.2.7          
ARC CE v.1.0.0 Y with gridftp interface and ARIS - -  
ARC Clients v.1.0.0          
ARC Core v.1.0.0          
ARC gridftp server v.1.0.0          
ARC InfoSys v.1.0.0          
ARGUS v.1.3.0 Y Site + Top + VO level - 1 Site level Argus per Site, Target >= 2 Sites  
ARGUS-EES v.0.1.0          
BLAH v.1.16          
CEMon v.1.13          
core BDII v.0.0.0          
CREAM v.1.13 Y LFS + Torque - Target >= 2 Sites  
dCache v.1.9.12 Y - - -  
Delegation Java v.2.0.0          
DGAS-sensors v.4.0.1          
DPM v.1.8.1 Y - - -  
emi-sherpa: an example component entry          
emi-ui v.0.0.0          
emi-wn v.0.0.0 Y glexec enabled - Target >= 2 Sites  
FTS v.2.2.6 Y - - -  
GFAL/lcg_util v.1.11.18 Y        
gLExec v.0.8.3 Y - - Target >= 2 Sites  
glite-gsoap/gss v.3.0.1          
gLite-MPI v.1.0.0          
glite-proxyrenewal v.1.3.17          
glite-yaim-core v.5.0.0          
gridsite v.1.7.12          
Hydra v.1.0.1          
L&B v.3.0.5 Y - - -  
LCAS v.1.4.0          
lcg-info-clients v1.0.0          
LCMAPS v.1.5.0          
LCMAPS-plugins-c-pep v.1.1.0          
LFC v.1.8.1          
RAL-SAGA-SD v.1.0.0          
site BDII v.1.0.0          
SLCS v.0.0.0          
StoRM SE v.1.7.0          
top BDII v.1.0.0          
Trustmanager v.3.0.3          
UNICORE AIP v.2.0.0          
UNICORE Client v.6.4.0          
UNICORE Gateway v.6.4.0          
UNICORE HILA v.2.2          
UNICORE Registry v.6.4.0          
UNICORE sec libs v.2.0.0          
UNICORE Services Environment v.6.4.0          
UNICORE TSI v.6.4.0          
UNICORE WS v.6.4.0          
UNICORE XACML PDP v.2.0.0          
UNICORE XNJS v.1.4.0          
UNICORE XUUDB v.1.3.2          
UVOS v.1.4.1          
VOMS v.2.0.0 Y - INFN_CNAF 1 + Replica Just the CNAF instance will be used,
but we include it here, since we want
Large Scale users to test the clients.
VOMS-Admin 2.6.0 N - - -  
WMS v.3.3.0 Y - - -  

To deploy testers.emi-eu.eu on your service you need:

  1. To enable the VO add it in the VOS variable of the yaim site-info.def file of your service (ex. VOS="alice atlas cms dteam infngrid lhcb ops testers.eu-emi.eu")
  2. Create in the directory with site-info.def a directory vo.d (if not existing) and save there the following file (including replica): testers.eu-emi.eu
  3. Check you have INFN CA installed with command $> rpm -qa |grep INFN .... output should be somethig like "ca_INFN-CA-2006-1.37-1"
  4. For pool accounts add in users.conf the following sample: users_testers.conf
  5. For groups enabling add in groups.conf the lines in the attached file: groups.conf
  6. Add new certificates (master and replica) to /etc/grid-security/vomsdir/ directory:

For the manual configuration of vomses parameters ( ex. .voms/vomses , .glite/vomses on the UI) use the following parameters:

"testers.eu-emi.eu" "emitestbed07.cnaf.infn.it" "15002" "/C=IT/O=INFN/OU=Host/L=CNAF/CN=emitestbed07.cnaf.infn.it" "testers.eu-emi.eu"
"testers.eu-emi.eu" "emitestbed01.cnaf.infn.it" "15002" "/C=IT/O=INFN/OU=Host/L=CNAF/CN=emitestbed01.cnaf.infn.it" "testers.eu-emi.eu"
The second line is for the replica In the near future, the VO will be deployed with EMI releases, and above info and files integrated in yaim profiles/examples and the certificate deployed through an rpm file

For ARC users the VO configuration is described at http://www.nordugrid.org/documents/voms-notes.html . For vomses file use the input as above.

For Unicore users: UNICORE does not support VOMS, so no configuration can be provided. UVOS, UNICORE VO solution, is to be phased out during the course of EMI, so we didn't deploy it in the testbed. It is planned to support the SAML based VOMS later on, but that is not available yet. If you require access to the testbed, you'll have to open a GGUS tkt send me your certificate and I will register it.

  • Testers VO subscription: The default VO for testing in EMI is testers.eu-emi.eu. So, for authentication on services supporting voms certificates, users need to subscribe to the VO for accessing the testbed. To enable the VO on Product Teams instances or User Interface follow instructions in section 3.3 in this page.

  1. General Configuration Issues:
    1. EMI Repository Link
    2. General EMI-1 Release and per product Release Notes documentation.
    3. To let EMI Testbed team test your correct installation/configuration and volunteer users test service functionality, some services need to be configured to interact with each other. In particular:
      1. To publish your services you can use at first the EMI Infrastructure information system bdii for configuration , and then the volunteer site deploying site and core bdii will be configured to see your resource. A gLite information system (BDII) was setup ( certtbpre-bdii.cern.ch working as site and top bdii) for gLite and dCache resources.
        1. Top-level BDII . Contact it from UI by command: ldapsearch -x -H ldap://certtbpre-bdii.cern.ch:2170 -b mds-vo-name=local,o=grid (see also details )
        2. Site-level BDII . Contact it from UI by command: ldapsearch -x -H ldap://certtbpre-bdii.cern.ch:2170 -b mds-vo-name=cert-tbpre-cern,o=grid (see also details )
      2. You are also requested to enable emi-testers.emi-eu.eu VO on your instance ( -> section 3.3 )
      3. Depending on the service you are configuring you would to set your configuration values to point to other sites deployed services. For configuration values please contact EMI Testbed Team.

4. Large Scale Testbed HOWTO for Developers

EMI testbed unit provides coordination effort to match testbed requirements for specific testing needs with infrastructural resources provided on a voluntary basis from a group of providers internal or external to EMI project. The setup of a new large scale testbed infrastructure starts from the accurate definition of testing case, needed resources, testing-period etc. First step in requiring a testbed is then the submission of a testbed request according to the following procedure:

  • Testbed Requests: WHO
    • Product Team members needing to test their product under specific conditions
    • EMI Release Manager, Quality Assurance leaders, Technical Area leaders
    • Users Community interested in specific testing campaign of Release Candidate version products
  • Testbed Requests: HOW
  • Testbed Requests: WHEN
    • Request tickets will be evaluated and handled eventually leading to the setup of a specific description paragraph in the "Active Testbed Description" section below.
    • Notice that the testbed setup process can take a few weeks depending on the complexity of testbed request and availability of resource providers. Therefore apply for a testbed as soon as you realize you may need it.

5. Available Resources Inventory

0 EMI Product EndPoint Site/Institute HW Details Configuration Details
1 emiUI emi-tutor.ct.infn.it INFN-Catania Quad Core AMD (2x2), 1.8 GHz, RAM 4G, HD 70 G porta ssh 2222
2 emiWMS wms013.cnaf.infn.it INFN-CNAF nehalem quad core E5630 2x300GB sata HD 2.2 GHz VO: cms
BDII: large scale top (including WLCG resources
3 emiCREAM cream.emi.d4science.research-infrastructures.eu:8443/cream-pbs-emi-demo D4Science-II Quad-Core AMD Opteron(tm) Processor 2356 OS: SL5

SW VERSION: EMI-1 Prod, Update 9

VO : testers.eu-emi.eu
SITE-BDII: bdii.emi.d4science.research-infrastructures.eu
WN:
4 emi sitebdii bdii.emi.d4science.research-infrastructures.eu D4Science-II SL5 Quad-Core AMD Opteron(tm)Processor 2356 EMI-1 Version
5 emi DPM dpm.emi.d4science.research-infrastructures.eu D4Science-II SL5 Quad-Core AMD Opteron(tm) Processor 2356 EMI-1 Version
VO : testers.eu-emi.eu
6 emi WN wn01.emi.d4science.research-infrastructures.eu D4Science-II SL5 Quad-Core AMD Opteron(tm) Processor 2356 EMI-1 Version
VO : testers.eu-emi.eu CREAM: cream.emi.d4science.research-infrastructures.eu:8443/cream-pbs-emi-demo
7 emiCREAM ctb04.gridctb.uoa.gr EGEE-SEE-CERT CPU 3.0 GHz Intel Xeon EMT64 RAM 512MB - 2GB DDR2 HDD 80GB SATA EMI-1 Version
OS: SL release 5.6 (Boron)
SW VERSION: EMI-1 Prod, Update 9

VO : testers.eu-emi.eu
SITE-BDII: bdii.emi.d4science.research-infrastructures.eu
WN:
8 Torque Server ctb07.gridctb.uoa.gr EGEE-SEE-CERT Scientific Linux SL release 5.6 (Boron)
CPU 3.0 GHz Intel Xeon EMT64
RAM 512MB - 2GB DDR2
HDD 80GB SATA
torque server for ctb04.gridctb.uoa.gr
9 siteBDII ctb01.gridctb.uoa.gr EGEE-SEE-CERT Scientific Linux SL release 5.6 (Boron)
CPU 3.0 GHz Intel Xeon EMT64
RAM 512MB - 2GB DDR2
HDD 80GB SATA
EMI-1 Version
10 SE ctb06.gridctb.uoa.gr EGEE-SEE-CERT Scientific Linux SL release 5.6 (Boron)
CPU 3.0 GHz Intel Xeon EMT64
RAM 512MB - 2GB DDR2
HDD 80GB SATA
EMI-1 Version
VO : testers.eu-emi.eu
11 WN ctb05.gridctb.uoa.gr EGEE-SEE-CERT Scientific Linux SL release 5.6 (Boron)
CPU 3.0 GHz Intel Xeon EMT64
RAM 512MB - 2GB DDR2
HDD 80GB SATA
EMI-1 Version
VO : testers.eu-emi.eu
Torque Server: ctb07.gridctb.uoa.gr
12 WN ctb10.gridctb.uoa.gr EGEE-SEE-CERT Scientific Linux SL release 5.6 (Boron)
CPU 3.0 GHz Intel Xeon EMT64
RAM 512MB - 2GB DDR2
HDD 80GB SATA
EMI-1 Version
VO : testers.eu-emi.eu
Torque Server: ctb07.gridctb.uoa.gr
13 WN ctb11.gridctb.uoa.gr EGEE-SEE-CERT Scientific Linux SL release 5.6 (Boron)
CPU 3.0 GHz Intel Xeon EMT64
RAM 512MB - 2GB DDR2
HDD 80GB SATA
EMI-1 Version
VO : testers.eu-emi.eu
Torque Server: ctb07.gridctb.uoa.gr
14 WN ctb12.gridctb.uoa.gr EGEE-SEE-CERT Scientific Linux SL release 5.6 (Boron)
CPU 3.0 GHz Intel Xeon EMT64
RAM 512MB - 2GB DDR2
HDD 80GB SATA
EMI-1 Version
VO : testers.eu-emi.eu
Torque Server: ctb07.gridctb.uoa.gr
15 ARGUS ctb02.gridctb.uoa.gr EGEE-SEE-CERT   EMI-1 Version

6. Active Testbed Description

Testbed Applicant: Andrea Ceccanti (Argus PT)
Tracking: GGUS tkt
Status: Waiting for Argus Team to define policies to be tested
Logbook: ArgusTestBed0012011

Testbed Applicant: Shiraz Memon (Emir PT)
Tracking: GGUS tkt
Status: Waiting for Services to be Ready for Deployment
Logbook: EmirTestBed0012012

Testbed Applicant: EGI-EMI
Tracking: SAVANNAH EMI-SA2 TASK
Status: Waiting for Services to be Ready for Deployment
Logbook: NagiosServerEMITestbed0022012

Testbed applicant: Marco Verlato (WeNMR grid manager)
Tracking: https://ggus.eu/ws/ticket_info.php?ticket=89884
Status: Waiting for reply. The application segfaults.

Testbed x, request description, implementation, contributors, Notes, period....etc.etc.

7. Closed Testbed Description

Testbed x, request description, implementation, contributors, Notes, period....etc.etc.

Edit | Attach | Watch | Print version | History: r36 < r35 < r34 < r33 < r32 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r36 - 2013-05-02 - FabioCapanniniExternal
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback