vNode / Xen installation and configuration

Before we start [ISSUES]

  • vNode security is very basic and should only be used on a trusted LAN, protect the machines with firewall and some other security mechanism. Hopefully this issue will be improved in the next release.
  • Getting access to use the SA3 machines is described in VNode Quick Start


  • A couple of machines with good hardware

  • vNode "server" and "node":
    • python >= 2.4 with (simple)json, ssl, httplib
    • httpd with mod_ssl and mod_python
  • vNode "node":
    • libvirt and libvirt-python
    • Volume Group with at least 20G of free space
    • Xen installed and booting nicely

vNode installation

server and node

  1. Create directory where you want to install vnode service files (choosing location remember that they must be available for httpd).
          $ mkdir /usr/local/vnode ; cd /usr/local/vnode/
  2. Get vNode and put to the directory. Currently vNode is available only directly from SVN, you can do SVN checkout or export (export is better for installation).
    • server:
              $ svn export
              $ svn export
    • node:
              $ svn export
      Note that you can install server and node part on one machine (even in the same location!).

  1. You can also get example configuration files:
            $ svn export


Create a directory (usually somewhere on your account) and:
        $ svn export
See also VNodeCLI.

vNode configuration

You can find example configuration files here, have a look also at example config files for server and node.

Configure OS

Httpd configuration

############ START ############ 
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.

User apache
Group apache

DocumentRoot "/usr/local/vnode"

<Directory "/usr/local/vnode">

# Possible values for the Options directive are "None", "All",
# or any combination of:
#   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
# The Options directive is both complicated and important.  Please see
# for more information.
    Options Indexes FollowSymLinks

# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
#   Options FileInfo AuthConfig Limit
    AllowOverride None

# Controls who can get stuff from this server.
    Order allow,deny
    Allow from all


Alias /vnode-2.0.2/ "/usr/local/vnode/server/template/html/"
Alias /server/ "/usr/local/vnode/server/"

<Directory "/usr/local/vnode/server/template/html/">
    AllowOverride AuthConfig Options
    Options None ExecCGI
    Order allow,deny
    Allow from all

ScriptAlias /cgi/ "/usr/local/vnode/"

# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
<Directory "/usr/local/vnode">
    AllowOverride AuthConfig Options
    Options None ExecCGI
    AddHandler cgi-script .py
    Order allow,deny
    Allow from all

############ END ############ 

Init scripts

(example for RedHat-based systems)
      # /etc/init.d/xend start
      # /etc/init.d/xendomains start
      # /etc/init.d/httpd start  
To avoid restart of the services when reboot occurs, do the following:
      # chkconfig --add xend 
      # chkconfig --level 345 xend on
      # chkconfig --add xendomains
      # chkconfig --level 345 xendomains on
      # chkconfig --add httpd 
      # chkconfig --level 345 httpd on
Disable libvirtd - it might cause networking problems (creating a bridge with STP protocol switched on!)
      # /etc/init.d/libvirtd stop
      # chkconfig --del libvirtd


Change the /etc/sudoers file to allow httpd user run the commands it needs
root  ALL=(ALL) ALL
apache  ALL = NOPASSWD: /usr/sbin/xm, /bin/tar, /usr/sbin/lvcreate, /sbin/mkswap, /sbin/mkfs.ext3, /usr/sbin/lvremove, /bin/mount, /bin/umount, /bin/mv, /bin/rm, /usr/sbin/vgs, /usr/sbin/lvs, /bin/cp
If there is
Defaults    requiretty
in sudoers you have to remove it.

Permissions for httpd/apache user

Change the ownership of the vnode directory to httpd (or apache / ...) user
# chown -R apache:apache <directory where vnode is>

Configure vNode

The examples below configure vNode to use PC01 PC02 PC03 PC04 as nodes and PC05 as server. All the node machines have a Logical Volume Group called vg1 and the images will be in /images/ and are tar.gz files.


(management host, providing also web browser layer) Open common.cfg and server.cfg and edit to your needs (check below for examples)

Example for node PC05 (same logic for the others)
#add the nodes and the number of virtual machines they can run. 
[PhysicalHostsPool] 3 3 3 3

#PC05 can deploy the following hostnames as virtual machines in the nodes
[VirtualHostsPool] 0 0 0 0 0 0

# List of users who have administrative role on the portal
# Add admin username
1 = admin

#General server options
# blocked: Controls access to the portal
# 0 - Allow access to the portal to all users
# 1 - Block access to the portal to all users except admin
blocked = 0
# protocol: Specifies the communication protocol to be used between server and nodes
protocol = https://
# warningMails: If enabled then each time the deployment of a virtual machine fails a mail is sent to the specified 'adminMail'
# NOTE! For this to work linux 'mail' command has to be configured and working
warningMails = 0
adminMail =

User and machine groups (pools)
Currently VNode has additional configuration file server/configuration/pool.cfg allowing to define groups of users (section [UserGroup]), physical ([PhysicalGroup]) and virtual machines ([VMGroup]). Grouping allows to simplify configuration of access control, physical-virtual machine dependencies and limits. Idea is simple: with a unique(!) name you can associate a number of hostnames / users or - other group(!) (of the same type of course). With the second you should be careful with creating loops (making group A a member of group B, which is member of A...), although the code should deal nicely with that too wink

Access Control and physical/virtual machine dependencies
Access control config file server/configuration/acl.cfg allows to define following dependencies:
  • user permissions to VM and Physical machines
  • where each vm can be deployed - mostly because of networking issues, VM cannot be deployed in different subnet that specified etc.
  • action allowed to perform on physical / virtual hosts and what each user can do

Structure of the file is simple: eg. in section [VMacl] one line should contain:

  • hostname of VM, group of virtual machines(!) or keyword 'default' (interpreted if hostname/group for host not found) followed by ":"
  • then either:
    • comma-separated list of users or groups of users allowed to use this virtual host / group
    • or "Any" which means anything/anyone is OK.
See also example file.

Important part, having a bit different meaning, is [PhysVirt] which allows to define which virtual host can be deployed on which virtual machine (see example file).

Note that it is possible to add quite simply any other similar dependencies (implementing corresponding methods in modules/

The limits for max. no. of deployed VMs, max. amount of memory, size of filesystem or number of partitions can be defined in the file server/configuration/limits.cfg.

Secure access to the web interface with certificates

Add the host certificate, host key and CA for PC05 in ssl.conf

#   Server Certificate:
# Point SSLCertificateFile at a PEM encoded certificate.  If
# the certificate is encrypted, then you will be prompted for a
# pass phrase.  Note that a kill -HUP will prompt again.  A new
# certificate can be generated using the genkey(1) command.
SSLCertificateFile /etc/PC05/hostcert.pem

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
SSLCertificateKeyFile /etc/PC05/hostkey.pem

#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
SSLCACertificatePath /etc/grid-security/certificates/

SSLCARevocationPath /etc/grid-security/certificates/
#   Client Authentication (Type):
#   Client certificate verification type and depth.  Types are
#   none, optional, require and optional_no_ca.  Depth is a
#   number which specifies how deeply to verify the certificate
#   issuer chain before deciding the certificate is not valid.

Check the server/cgi-bin/.htaccess file in server directory and change whatever you need. If you want to use authentication with user certificate add the DN of vNode users that you would like to give access to (/etc/httpd/conf/httpd.passwd)

SSLVerifyDepth 5
SSLVerifyClient require
SSLOptions +StdEnvVars +StrictRequire +OptRenegotiate
SSLOptions +FakeBasicAuth

AuthType Basic
AuthName "vNode"
AuthUserFile /etc/httpd/conf/httpd.passwd
Require valid-user


- machines with hypervisors, where virtual ones will be deployed

Open node/configuration/common.cfg and node/configuration/node.cfg and edit to your needs (check below for example)

Example for node PC01 (same logic for the others)
#PC01 can deploy 3 virtual machines
[PhysicalHostsPool] 3

#PC01 can deploy the following hostnames as virtual machines
[VirtualHostsPool] 0 0 0 0 0 0

imageDirectory: /images/
volumeGroupName: vg1
protocol: https://

#Remove the images you don't need
SLC-4-32: SLC-4-x86.tar.gz
SLC-4-64: SLC-4-x86_64.tar.gz
SL-4-32: SL-4-x86.tar.gz
SL-4-64: SL-4-x86_64.tar.gz
DEBIAN-4-32: debian-4-x86.tar.gz
DEBIAN-4-64: debian-4-x86_64.tar.gz
SLC-3-32: SLC-3-x86.tar.gz
SL-3-32: SL-3-x86.tar.gz
SL-5-32: SL-5-x86.tar.gz
SL-5-64: SL-5-x86_64.tar.gz

default: 512

default: xen

SL-3-32: mkfs.ext3 -O none,has_journal,filetype,sparse_super
default: mkfs.ext3

Uff, finally its done smile

Where to get Images

Try the script libfsimage

$ cvs -d co libfsimage

I can provide already working SLC/SL5, SLC/SL4 and debian 4 images, contact me for more information.

For people at CERN, more details are on VNodeImagesCERN


  • Check if python doesn't complain when importing required modules
Edit | Attach | Watch | Print version | History: r21 < r20 < r19 < r18 < r17 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r21 - 2010-08-12 - TomaszWolak
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EGEE All webs login

This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright & by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback