Integrating Cloud Computing resources into the CMS framework

This page aims to collect the activities driven by the IT-ES group in order to integrate Cloud resources of different providers with the CMS experiment framework.

Regular report summary are collected in CloudTesting.

Why testing CMS workflows in the Cloud? Future main goals:

  • Ability to transparently interface with different cloud providers
  • Understand performance and reliability compared to a Grid site
  • Evaluate to transparently offload work to a cloud resource
  • Possible future ambition:
    • use commercial services as a significant fraction of overall resources available?
    • Gird sites offering Cloud resources/infrastructure?

Contextualization of VM

The contextualization of the virtual machines should follow the guide lines provided by the HEPiX forum.


  • Participant: Mattia, Cristovao Jose Domingues Cordeiro
  • Evaluating cloud-init as a contextualization method


Using HammerCloud to perform the tests allows to:
  • automatize the jobs submission
  • retrieve a set of standard metrics already defined for CMS Grid sites
  • compare results among different cloud providers and different Grid sites
Work done with the support of Ramon.

Interfacing GlideinWMS to Cloud providers

The possibility to interface GlideinWMS to a cloud provider would give the possibility to transparently offload work to a cloud resource. This would make possible to:
  • automatize the submission to the Cloud provider resources
  • include the interaction with the cloud provider by also handling the virtual machine life cycle
  • in addition it will allow to re-use the very same workflow being used for the CMS jobs execution, for both the resource allocation and the job management.

More information:

Cloud providers

Provider Comments Website Test documents Infrastructure EC2 interface
StratusLab Computing meeting report OpenNebula no by default, it could be enabled
Lxcloud   IT web site LxCloudTestingCMS OpenNebula no by default
AI@CERN   AgileInfrastructure CMS Offline Computing week of March 2013 report OpenStack yes
RackSpace     CMS Offline Computing week of March 2013 report OpenStack yes


  • Participant: Mattia, Swati Katta, Hasier Rodriguez Valencia
  • Work done
    • Running MC jobs
    • Running analysis jobs
    • Development of VM manager


  • Participant: Mattia, Marek Denis
  • Work done is reported here: LxCloudTestingCMS
    • Contextualization of WN
    • Contextualization of Squid proxy server


  • Participant: Mattia, Marek Denis


  • Participant: Mattia, Marek Denis

Todo list

Done? Name Comments Documentation Results
No Squid Evaluate the job performance when deploying a squid server in the Cloud SquidForCMS  
No xrootd Evaluate the network latency and general performance impact when remotely reading analysis data through xrootd    
No Grid site comparison Compare the performances of a deployment in the cloud to average performance in a Grid sites    
No Output data management Understand and evaluate the possible solutions to manage the job's output (AsyncStageOut)    

18 July 2012

-- MattiaCinquilli - 25-Jul-2012

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2013-03-21 - MattiaCinquilli
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback