pre-GDB about Cloud Issues, NIKHEF, March 10, 2015


Introduction - M. Jouvin

Working group goals: explore possibility to use private and community clouds as a replacement for Grid CEs

Meeting today: review work in progress since last meeting in September on several topics

  • Target shares: achieve the equivalent of a fairshare in a batch system
  • Accounting also very important
    • status of APEL to be presented.
    • Task force created on cloud accounting and reporting
  • Security: paradigm change with respect to grid shifting responsibilities to the VOs
    • task force on traceability launched

Also take the opportunity of being in Amsterdam to establish closer contact with EGI FedCloud

Resource Provisionning - A. McNab

Vac and Vcycle - A. McNab

Both VM lifecycle manager: vcycle manages VMs in a cloud where Vac is a standalone daemon that can create the VM

  • Same assumptions about VM behaviour and what should be done
  • vcycle can be run by a site, an experiment or a regional group
  • See Andrew's presentation at CERNVM workshop last week

New since 6 months

  • user_data (contextualization data) and images can be specified as URL to experiment web site: cached at vcycle/vac server and rechecked each time
  • Allows experiments to update user_data and images (CERNVM, small ~20 MB) without interaction with the site
  • Easier support of new communities
  • See example in presentation

vcycle refactored

  • Support backend plugins for OCCI, EC2 and potentially others
  • Default OpenStack plugin now using a REST API rather than the Nova library
  • More code common between Vac and Vcycle
  • Support for RFC proxies
  • ssh keypairs managed by vcycle
  • Code on GitHub but may require some assistance to start

Target shares are instantaneous: no history stored, may require some target shares modification to get the appropriate sharing on a long period

  • Now implemented the same way in Vcyle as in Vac
  • Units is HS06/VM
  • Vcycle instances can support multiple tenancies at multiple remote sites
    • Different target share pools in different tenancies
  • Multiprocessors taken into account

Accounting: very close to a grid accounting record

  • APEL SSM record is written in a directory at the end of each VM
  • Can be sent to APEL through ssmsend as ARC does
  • No need to install OpenStack accounting reporting tools
  • Submit host build from the vac/vcycle instance name and the VM type: close to a grid submit host
  • VM UUID is used as the LocalJobId


  • syslog prototype: configured with /etc/machinefeatures/syslog.
    • Current Vac implementation allows one rsyslogd per hypervisor for greater flexibility
  • UDP syslog probably a good choice: universally supported, gateway to other protocols
  • Another syslog prototype started based on contextualization

Next steps

  • New VacQuery UDP protocol, more resilient to high packet loss
  • Container support in Vac (probably with libvirt)
  • Overnight use of Vac (on interactive machines over night)
  • CentOS 7 and Python 2.7 support
  • vcycle: OCCI and EC2 plugins

Current usage: Vac and Vcycle installed at a number of sites, mainly in UK, the total representing a good T2

FairShare Scheduler - L. Zangrando

FairShare Scheduler for OpenStack: queuing of requests + scheduling based on SLURM scheduler

Initial version as a replacement for NOVA scheduler: not sustainable with new versions of OpenStack

  • Need to adapt FS Scheduler for each version of OpenStack
  • Suggestion from OpenSTack developers to redesign/reimplement it as a NOVA external manager: no tight coupling in term of code
    • Interaction through API

Implementation just started: 2 months planned

  • Based on Icehouse and Juno
  • Part of DATACLOUD project workplan
  • Will be available as a StackForge project before incubation and possibly integration into OpenStack

DataBridge: Accessing Cloud Storage from Grid - F. Furano

Apache + Dynamic Federation technology with an S3 backend

  • S3 provides an efficient delegation mechanism
  • Apache can host any number of standard authentication methods

BOINC workflow

  • Volunteer PC gets a job agent
  • Jobs get some work and write output in a known place
  • After validation, FTS transfer the output to the production storage
  • Need to bridge authentication: X509 on grid side, mysql-based user/pwd on BOINC side

DataBridge: mod_auth_mysql + gridsite

  • S3 doing redirection using S3 signatures
  • Dynafed for scalability, performance and uniform/federated namespace
  • No mapping file to maintain
  • Simple message queue implemented as a web app made of 2 CGIs and using the Dav backend

Already adopted by several BOINC-based projects at CERN: Beauty@Home, Test4Theory, CMS@Home

  • Use not restricted to volunteer computing
  • Plan to explore integration with CERN SSO and Identity Federations

Security & Traceability - I. Collier

Security incidents are a reality: need to contain them and limit their impact

  • Reputation and resource usage
  • 70+ incidents in WLCG since 2006
  • Traceability critical: who did what, when and where

Traceability TF created last Fall: first F2F meeting at CERN in February

  • See summary


  • Should primarily depend on externally observable behaviour: hypervisor, cloud MW, network
  • Ability to correlate and search logs
  • Comparison between machine features and site contextualization for configuring syslog started
  • Sites with experience started to discuss the use of netflow for network traffic logging

Quarantining VMs

  • Integrated into StratusLab only
  • No off-the-shelf solutions for others
  • RAL started to implement it in HTCondor
  • Usage/usability may be limited by the impact on storage (storage cost)

VO logging: to assess potential gap, a security challenge is planned

More participants welcome!


  • Agreement that WLCG use case is a simple use case not representative of all the possible cloud usages
    • In particular EGI FedCloud use cases are more complex
    • Pragmatic approach: try to solve "properly" or improve the situation for the simple WLCG use case and learn from experience to address more complex use cases
  • Jeff: said that WLCG use case is simpler because we have a high degree of trust with experiments. But this is the wrong way: when we have a problem, it always comes as a surprise for experiments...
    • Ian Collier (IC): this is why we plan to have security challenges. Also in the grid, we have the experience that when there is a problem (reported by a site to EGI CSIRT), there is always a need to contact the VO to identify the user and it always worked. Should not be too negative.
    • Sven/David: but challenges can only point that there is a problem but asbence of problem with a challenge doesn't prove anything. Cannot test every possible use case.
    • Michel: trust is not only an abstract thing, it is the result of agreed practices and resulting from the fact that almost everybody is using the same VM (CERNVM), that we know who builds it and who update CVMFS contents and that VMs are instantiated as part of the pilot frameworks: users don't request VMs, they submit jobs to experiment frameworks.
    • Jeff: but sooner or later, we'll have a VM run which was not the standard VM... It has always been like that!
  • We need to raise awareness in VOs about the specific responsibilities of VOs in the cloud world
    • Michel: this is why discussions between sites, experiments and experts are important, like in the TF. Experiments have a good will and this working together that each party understands better its responsibilites.
  • Michel: good news from TF work so far is that we can make a significant step forward without inventing/implementing complex things. Mainly a matter of putting together existing tools in the appropriate way.


Accounting TF Report - L. Field

TF work done through an email list

  • Meeting archived in a specific Indico category (see slides)

Use case: WLCG supported by MoUs and pledges are a renewal of that commitment

  • Ensure enough resources are in place (REBUS)
  • Report on actual usage: mostly from accounting portal
  • Need to ensure the same happens with the same information for cloud resources

Proposal: build upon existing tooling

  • APEL: including HS06 benmarking metric
  • EGI Accounting portal
  • REBUS: incorporate cloud resources both in single and combined views


  • Cloud accounting record: specification exists
  • Publishers with supporting doc
  • EGI account portal with new views
    • Validation of results
  • REBUS: identify the gaps

Finalizing the work plan document for identified gaps

  • Should be ready in the next weeks
  • Will be circulated to GDB and MB


  • Possibility of double counting for sites that received their workload through the grid and run it in VMs (e.g. Vac)
    • Both accounting are complementary: need to find a way to define how to merge them when both exist for the same resource
    • Issue to discuss in the accounting TF

APEL Update - J. Gordon

Most of the work in the FedCloud: 20 OS and ONE sites reporting to APEL

  • Last version of the publishing scripts fixed the known issues publishing VOs and wallclock
  • per site and per VO reports

UR v0.4 agreed in January by FedCloud

  • Added benchmarking value
  • Structure within a site (multiple clouds)
  • Report image from the marketplace
  • Implementation planned in two phases
    • Loader and database
    • publishing scripts
  • Still some uncertainties, in particular for defining storage associated with the VM
    • Mainly storage internal to the VM
  • Requirements not yet addressed
    • Assigning usage across time: partial report for a long running VM. Required to assign to each month the usage rather than the accumulated time on the last month

WLCG issues: APEL is just a flat database of site data, the site structure is applied by accounting portal based on external data

  • Do we define a WLCG cloud or just keep the current T1/T2 structure
  • Use REBUS as the source of T1/T2 structure as for grid

Several issues on the portal being worked on

  • T2 and T1 structure view, drill down from tree to sites
  • Combined view grid+cloud
  • CPU/wallclock usage still reported in seconds rather than HS06

Vac/Vcycle cut their own grid record: nothing says that they are coming from clouds

  • Proposal to use Infrastructure_Type to flag this

VM Benchmarking - U. Schwickerath

Classification of WNs coming from LSF

  • Classify VM cores from standard info obtained from the OS (OS, cpuinifo, dmidecode)
    • Mem speed not available inside VMs: hope this will change in SL7
  • Benchmark a large number of VMs per class
  • Build a table of performance per machine type: be pessimistic about the performance

Fully deployed at CERN batch farm

Local cloud accounting based on Ceilometer

  • Not enough information to determine HW type from the Ceilometer accounting
  • Use the hypervisor perf to estimate HS06 of a VM

APEL-based cloud accouting @CERN

  • Based on Ceilometer data only
  • Currently unable to report performance because there is no way to match a VM with an HW type or hypervisor with Ceilometer data
  • Still to understand how to handle long running VMs


  • Andrew: would be nice to get information about hypervisors and HW types for doing performance studies in experiments
    • Jan: can understand but this is not in the cloud culture. No real chance to see it happening: cloud providers are eager to prevent users looking under the hood...

EGI FedCloud - P. Solagna

Several use cases covered: service deployment, HTC over cloud, heavy memory apps

Different access levels: IaaS, PaaS, SaaS

Integrated with EGI core infrastructure and operational tools

  • Monitoring: an EGI-wide monitoring infrastructure for cloud resources, as it it the case for grid resources
  • Accounting: heavily involved in discussions with APEL
  • Security: authentication, incident handling

VM Management through OCCI

  • Contextualization through ClouldInit: all VMs delivered by EGI have the CloudInit hooks
  • EGI wants to extend the support to native interfaces when required by user communities but this poses a problem for monitoring
    • Will require using the probes already available for native interfaces
    • Tricky: managing credentials for each native interface (currently only X509 in EGI)

VM image management based on AppDB

  • Allow to implement the endorsement policy
  • Integrated with EGI Information System

Working on integrating authentication with Federated Identities


  • Improved user tracking through a User UID info in robot generated proxies


  • OpenStack Keystone X509 support: 2 solutions available
    • Native support, contributed by CERN, available since the last release: only plain X509, no support for VOMS proxies
    • Keystone patch from EGI FedCloud: support for VOMS proxies, not contributed upstream (yet?)
  • Michel: monitoring infrastructure setup by EGI is of interest for WLCG but OCCI requirement may be a barrier if this is the only reason to set it up
    • Peter: aware of this, no solution yet, we need to be able to speak with cloud infrastructures
    • Michel: a possibility to explore could be to move the interface abstraction on the client side (monitoring infrastructure), using a method similar to what has been used by LHC experiment frameworks (libCloud, DeltaCloud, JCloud...)
  • Peter/Michel: important to foster EGI/WLCG collaboration on the topics relevant to both. Main candidates: accounting (already happening), monitoring, authentication through identity federations
    • Michel: VM image management is probably not very relevant for WLCG. Only one VM image: CERNVM. Evolved quite a lot since the original HEPiX work.

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2015-03-17 - MichelJouvin
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback