EC2 Submitter Design

Description

The EC2 Submitter runs ETICS build and test jobs on EC2.

ETICS Platforms and AMI images

An ETICS platform corresponds to an EC2 AMI. The EC2 Submitter maintainer will need to maintain a set of AMI that corresponds to the ETICS platform he want to support, just as WMWare images are maintained for the current ETICS project services. Most popular distributions can be found already among the community available AMI. AMIs for SL* machines will likely need be created. Recipes for converting WMWare images to AMI are available online.

Components

ec2_submitter_components.jpg

Web Service

This is the web service layer implementing the submitter interface.

The current WSDL is available on the ETICS repository http://etics-repository.cern.ch:8080/repository/download/registered/org.etics/org.etics.submission.webservice-interface/0.9.0/noarch/org.etics.submission.webservice-interface-0.9.0-1.tar.gz

The implementation will be done using Apache CXF, and the service will run in an embedded Jetty container.

Persistence

The persistence layer maintains the state of the service in a reliable storage, typically a RDBMS.

The interface will follow the DAO pattern. Implementation will be done using JPA and Hibernate, and an embedded RDBMS like hsqldb or Derby.

Scheduler

The scheduler is responsible for scheduling builds and tests for execution on an EC2 instance. It runs at fixed intervals, get unscheduled submissions from the persistence layer, get an instance where to run it, and set the state to scheduled.

The scheduler uses the instance pool to get instances where to run builds.

PlatformAMIMapper

This component is responsible for maintaining a list of the supported platform. For each of the platform in the list, it will have a correspondent AMI id.

The mapping must be easy to edit for the maintainer. An XML configuration file will be used. Ideally no restart of the service should be needed.

Runner

The runner is responsible for running jobs on the given EC2 instance. It runs at fixed intervals, get scheduled jobs from the persistence layer, run it on the instance they were scheduled to run on.

The runner releases instances back to the instance pool when done,

InstancePool

The instance pool is responsible for managing the EC2 instances on which the builds will run. It will call the EC2 web service to start instances, and tell the scheduler which instance it's available for a build. The pool must minimize the cost of running instances (will return instances which running time is farer to the hour).

It exposes an interface following the object pool pattern, with operation for acquiring and releasing instances, and setters for configuring the maximum pool size. The quantity relevant for a pool of EC2 instances are the maximum budget per month (or other time period), and the maximum parallel instance allowed.

This is the component that uses the EC2 stubs. It needs to be configured with credentials for accessing EC2 (certificate and private key preferable over access keys). Typically it will configure the instances to be accessed via ssh from the submitter host.

CommandExecutor

The command executor is responsible for executing shell commands on a EC2 instance.

This is the component that accesses the EC2 instance. It needs to be configured with the rsa key to access the instances.

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2009-05-08 - ValerioVenturi
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ETICS All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback