vNode / Xen installation and configuration

Before we start [ISSUES]

• vNode security is very basic and should only be used on a trusted LAN, protect the machines with firewall and some other security mechanism. Hopefully this issue will be improved in the next release.
• Getting access to use the SA3 machines is described in VNode Quick Start

Requirements

• A couple of machines with good hardware

• vNode "server" and "node":
  • python >= 2.4 with (simple)json, ssl, httplib
  • httpd with mod_ssl and mod_python
• vNode "node":
  • libvirt and libvirt-python
  • Volume Group with at least 20G of free space
  • Xen installed and booting nicely

vNode installation

server and node

1. Create directory where you want to install vnode service files (choosing location remember that they must be available for httpd).

         $ mkdir /usr/local/vnode ; cd /usr/local/vnode/

2. Get vNode and put to the directory. Currently vNode is available only directly from SVN, you can do SVN checkout or export (export is better for installation).
   • server:

             $ svn export http://svnweb.cern.ch/guest/vNode/trunk/server
             $ svn export http://svnweb.cern.ch/guest/vNode/trunk/www

   • node:

             $ svn export http://svnweb.cern.ch/guest/vNode/trunk/node

     Note that you can install server and node part on one machine (even in the same location!).

1. You can also get example configuration files:

           $ svn export http://svnweb.cern.ch/guest/vNode/trunk/cfg

vnodecli

Create a directory (usually somewhere on your account) and:

        $ svn export http://svnweb.cern.ch/guest/vNode/trunk/cli

See also VNodeCLI.

vNode configuration

You can find example configuration files here, have a look also at example config files for server and node.

Configure OS

Httpd configuration

• Create a vnode configuration file for httpd and copy it to httpd conf.d directory. Template (do any changes to suit your needs):

############ START ############ 
#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#

User apache
Group apache

DocumentRoot "/usr/local/vnode"

<Directory "/usr/local/vnode">

#
# Possible values for the Options directive are "None", "All",
# or any combination of:
#   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
#
# The Options directive is both complicated and important.  Please see
# http://httpd.apache.org/docs/2.2/mod/core.html#options
# for more information.
#
    Options Indexes FollowSymLinks

#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
#   Options FileInfo AuthConfig Limit
#
    AllowOverride None

#
# Controls who can get stuff from this server.
#
    Order allow,deny
    Allow from all

</Directory>

Alias /vnode-2.0.2/ "/usr/local/vnode/server/template/html/"
Alias /server/ "/usr/local/vnode/server/"

<Directory "/usr/local/vnode/server/template/html/">
    AllowOverride AuthConfig Options
    Options None ExecCGI
    Order allow,deny
    Allow from all
</Directory>

ScriptAlias /cgi/ "/usr/local/vnode/"

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/usr/local/vnode">
    AllowOverride AuthConfig Options
    Options None ExecCGI
    AddHandler cgi-script .py
    Order allow,deny
    Allow from all
</Directory>

############ END ############ 

Init scripts

(example for RedHat-based systems)

      # /etc/init.d/xend start
      # /etc/init.d/xendomains start
      # /etc/init.d/httpd start  

To avoid restart of the services when reboot occurs, do the following:

      # chkconfig --add xend 
      # chkconfig --level 345 xend on
      # chkconfig --add xendomains
      # chkconfig --level 345 xendomains on
      # chkconfig --add httpd 
      # chkconfig --level 345 httpd on

Disable libvirtd - it might cause networking problems (creating a bridge with STP protocol switched on!)

      # /etc/init.d/libvirtd stop
      # chkconfig --del libvirtd

/etc/sudoers

Change the /etc/sudoers file to allow httpd user run the commands it needs

root  ALL=(ALL) ALL
apache  ALL = NOPASSWD: /usr/sbin/xm, /bin/tar, /usr/sbin/lvcreate, /sbin/mkswap, /sbin/mkfs.ext3, /usr/sbin/lvremove, /bin/mount, /bin/umount, /bin/mv, /bin/rm, /usr/sbin/vgs, /usr/sbin/lvs, /bin/cp

If there is

Defaults    requiretty

in sudoers you have to remove it.

Permissions for httpd/apache user

Change the ownership of the vnode directory to httpd (or apache / ...) user

# chown -R apache:apache <directory where vnode is>

Configure vNode

The examples below configure vNode to use PC01 PC02 PC03 PC04 as nodes and PC05 as server. All the node machines have a Logical Volume Group called vg1 and the images will be in /images/ and are tar.gz files.

Server

(management host, providing also web browser layer) Open common.cfg and server.cfg and edit to your needs (check below for examples)

server/configuration/common.cfg

Example for node PC05 (same logic for the others)
#add the nodes and the number of virtual machines they can run. 
[PhysicalHostsPool]
PC01.cern.ch: 3
PC02.cern.ch: 3
PC03.cern.ch: 3
PC04.cern.ch: 3

#PC05 can deploy the following hostnames as virtual machines in the nodes
[VirtualHostsPool]
virtualHostname1.cern.ch: 0
virtualHostname2.cern.ch: 0
virtualHostname3.cern.ch: 0
virtualHostname4.cern.ch: 0
virtualHostname5.cern.ch: 0
virtualHostname6.cern.ch: 0

server/configuration/server.cfg

# List of users who have administrative role on the portal
# Add admin username
[Administrators]
1 = admin

#General server options
[General]
# blocked: Controls access to the portal
# 0 - Allow access to the portal to all users
# 1 - Block access to the portal to all users except admin
blocked = 0
# protocol: Specifies the communication protocol to be used between server and nodes
protocol = https://
# warningMails: If enabled then each time the deployment of a virtual machine fails a mail is sent to the specified 'adminMail'
# NOTE! For this to work linux 'mail' command has to be configured and working
warningMails = 0
adminMail = someMail@cern.ch

User and machine groups (pools)

Currently VNode has additional configuration file server/configuration/pool.cfg allowing to define groups of users (section [UserGroup]), physical ([PhysicalGroup]) and virtual machines ([VMGroup]). Grouping allows to simplify configuration of access control, physical-virtual machine dependencies and limits. Idea is simple: with a unique(!) name you can associate a number of hostnames / users or - other group(!) (of the same type of course). With the second you should be careful with creating loops (making group A a member of group B, which is member of A...), although the code should deal nicely with that too

Access Control and physical/virtual machine dependencies

Access control config file server/configuration/acl.cfg allows to define following dependencies:

• user permissions to VM and Physical machines
• where each vm can be deployed - mostly because of networking issues, VM cannot be deployed in different subnet that specified etc.
• action allowed to perform on physical / virtual hosts and what each user can do

Structure of the file is simple: eg. in section [VMacl] one line should contain:

• hostname of VM, group of virtual machines(!) or keyword 'default' (interpreted if hostname/group for host not found) followed by ":"
• then either:
  • comma-separated list of users or groups of users allowed to use this virtual host / group
  • or "Any" which means anything/anyone is OK.

See also example file.

Important part, having a bit different meaning, is [PhysVirt] which allows to define which virtual host can be deployed on which virtual machine (see example file).

Note that it is possible to add quite simply any other similar dependencies (implementing corresponding methods in modules/acl.py).

Limits

The limits for max. no. of deployed VMs, max. amount of memory, size of filesystem or number of partitions can be defined in the file server/configuration/limits.cfg.

Secure access to the web interface with certificates

Add the host certificate, host key and CA for PC05 in ssl.conf

#   Server Certificate:
# Point SSLCertificateFile at a PEM encoded certificate.  If
# the certificate is encrypted, then you will be prompted for a
# pass phrase.  Note that a kill -HUP will prompt again.  A new
# certificate can be generated using the genkey(1) command.
SSLCertificateFile /etc/PC05/hostcert.pem

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
SSLCertificateKeyFile /etc/PC05/hostkey.pem

#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
SSLCACertificatePath /etc/grid-security/certificates/

SSLCARevocationPath /etc/grid-security/certificates/
#   Client Authentication (Type):
#   Client certificate verification type and depth.  Types are
#   none, optional, require and optional_no_ca.  Depth is a
#   number which specifies how deeply to verify the certificate
#   issuer chain before deciding the certificate is not valid.

Check the server/cgi-bin/.htaccess file in server directory and change whatever you need. If you want to use authentication with user certificate add the DN of vNode users that you would like to give access to (/etc/httpd/conf/httpd.passwd)

<Files main.py>
SSLRequireSSL
SSLVerifyDepth 5
SSLVerifyClient require
SSLOptions +StdEnvVars +StrictRequire +OptRenegotiate
SSLOptions +FakeBasicAuth

AuthType Basic
AuthName "vNode"
AuthUserFile /etc/httpd/conf/httpd.passwd
Require valid-user
</Files>

Nodes

- machines with hypervisors, where virtual ones will be deployed

Open node/configuration/common.cfg and node/configuration/node.cfg and edit to your needs (check below for example)

common.cfg

Example for node PC01 (same logic for the others)
#PC01 can deploy 3 virtual machines
[PhysicalHostsPool]
PC01.cern.ch: 3

#PC01 can deploy the following hostnames as virtual machines
[VirtualHostsPool]
virtualHostname1.cern.ch: 0
virtualHostname2.cern.ch: 0
virtualHostname3.cern.ch: 0
virtualHostname4.cern.ch: 0
virtualHostname5.cern.ch: 0
virtualHostname6.cern.ch: 0

node.cfg

[General]
serverHostname: PC05.cern.ch
imageDirectory: /images/
volumeGroupName: vg1
protocol: https://

#Remove the images you don't need
[VMImageFiles]
SLC-4-32: SLC-4-x86.tar.gz
SLC-4-64: SLC-4-x86_64.tar.gz
SL-4-32: SL-4-x86.tar.gz
SL-4-64: SL-4-x86_64.tar.gz
DEBIAN-4-32: debian-4-x86.tar.gz
DEBIAN-4-64: debian-4-x86_64.tar.gz
SLC-3-32: SLC-3-x86.tar.gz
SL-3-32: SL-3-x86.tar.gz
SL-5-32: SL-5-x86.tar.gz
SL-5-64: SL-5-x86_64.tar.gz

[VMSwapSize]
default: 512

[VMTypes]
default: xen

[XenDeployerFSCreateCmd]
SL-3-32: mkfs.ext3 -O none,has_journal,filetype,sparse_super
default: mkfs.ext3

Uff, finally its done

Where to get Images

Try the script libfsimage

$ cvs -d :pserver:anonymous@isscvs.cern.ch:/local/reps/xenvirt/ co libfsimage

I can provide already working SLC/SL5, SLC/SL4 and debian 4 images, contact me for more information.

For people at CERN, more details are on VNodeImagesCERN

Tests

• Make sure that you can access https://nodehostname.

• Check if python doesn't complain when importing required modules