Virtualization @ IT-FIO

TIP Complex document layout ahead: do not edit in WYSIWYG mode, please! TIP

1. Introduction / Abstract

One may say that virtualization techniques were present since some years. But this year it seems to be that virtualization surged ahead of "Web 2.0" as the buzziest of buzzwords in the sector of information technology.

In the first half of March 2007 the FIO group launched its virtualization project. The main object is to use the availability of cheaper and more powerful machines to consolidate low utilisation services. Besides this, there is an increasing demand for test and development machines, which are usually not heavily loaded and thus could be easily virtualised.

This Twiki page likes to share experiences the project team made during the integration of a Xen based virtual machine environment into the existing infrastructures of CERNs Computing Centre. This covers issues regarding configuration, installation, monitoring, managing and modelling of the virtual machine environment. Additionally it will give detailed instructions for operators and system administrators.

2. Prerequisites

The page will not give any introduction into the basics of virtualization, nor of the CERN system environment in general. Before you start reading, please be sure you are familiar with all the terms which are specified subsequently.

2.1. Resources

2.1.1. About virtualization in general

  • A short introduction is given by Amit Singh at kernelthread.com - link External site
  • Wikipedia has a rather small article but some references which might be of help - link External site
  • If you want to spent some money on books or if you are able to find some in the library, you may want to read these:

    • Virtualization: From the Desktop to the Enterprise (Books for Professionals by Professionals) - link External site
    • Virtualization with Xen(tm): Including Xenenterprise, Xenserver, and Xenexpress - link External site
    • Professional Xen Virtualization - link External site
The latter is already preorderd by the CERN Library and should be available at Véronique Lefèbures office as soon as Amazon.com starts to ship it (by January 8th 2007 the estimated shipping date was January 29th 2008).

The literature mentioned before covers virtualization itself and some of todays widely available virtualization technologies like VMWare, Xen, Microsoft HyperV, QEMU and so on.

Since the CERN FIO group decided to use Xen as their virtualization environment it might be a good idea to have a closer look at this book:

    • The Definitive Guide to the Xen Hypervisor - link External site

I would like to mention the Xen Wiki External site which contains documentation, FAQs, HowTos and more detailed information. Furthermore, it is always helpful to scan the Xen mailing lists External site, especially xen-users External site.

2.1.2. About system environment at CERN

To run the Computing Centre at CERN it needs a lot of manpower and even more pieces of software. The key to survive at the CERN is, like everywhere else in an IT department, to know your tools External site or at least to know somebody who knows them. Here is a short list of tools and systems one should know when working in the FIO Group and especially in the FS Section:

  • ELFms (Extreme Large Fabric management system) - link, link - with its subsystems:
    • Lemon as monitoring framework - link
    • Quattor as system installation and configuration toolsuite - link, link External site (accessible only from within CERN with SpringerLink account), Section 2.2 in Massimiliano Masi's master thesis External site.
      • Configuration Database (CDB)
    • SLS as service level status (virtual machines not yet integrated) - link
    • LEAF Tools for hardware and state management - link
  • Access on lxadm cluster
  • Know how to setup a cluster in CDB - link, link External site
  • Know how to use PrepareInstall to install machines
  • Know standard tools like Kickstart and Netboot (AIMS)
  • Access to LanDB
  • Access to SINDES

2.1.3. About virtualization at CERN

All existing documentation about the FIO virtualization project can be found in the CERN TWiki. I will give you a shortcut and subsequently list all important pages:

Besides this, it is interesting to read the minutes of some meetings which deal with the virtualization project:

2.2. Virtualization Environment at CERN (Facts and Figures)

This shows the current Xen-based configuration environment at the CERN.

2.2.1. Hosting machines (Dom0):

Operating System (Base Release): Scientific Linux CERN 4.6 (only when all updates are installed)
Kernel Version (32-bit and 64-bit): 2.6.18-8.1.14.slc4xen (updated on 10/09/2007)

All packages are available from the software repository (swrep):

xen_major: 3
xen_minor: 0
xen_extra: .3-rc5-8.1.14.s
xen_caps: xen-3.0-x86_32p
xen_changeset: unavailable
cc_compiler: gcc version 3.4.6 20060404 (Red Hat 3.4.6-8)
cc_compile_date: Fri Oct 5 15:08:54 CEST 2007
xend_config_format: 2

Linux Support always tries to keep the modified Xen kernels, Xen and related packages (glibc, xen-tools, openafs) the same for both architectures, i386 and x86_64. The host machines are organized in the lxxen cluster. See the LemonWeb

This query results in an overview of all machines in the CERN computing center which are running Xen as a Dom0 (link only works from inside the CERN).

2.2.2. Virtual machines (DomU):

Operating System (Base Release): Scientific Linux CERN 4.6 (depends on subcluster) - other operating systems (SLC3, other linux distributions are not supported by FIO)
Kernel Version (32-bit and 64-bit): 2.6.9-55.0.2.EL.CERNxenU (updated on 10/09/2007)

This query helps to get an overview of all virtual machines configured in CDB.

2.2.3. Test Environment

As previously mentioned, there are a lot of machines installed already with a running Xen based virtualization environment. These machines are used in production as for the grid testing and certifying process or retirement of old machines.

During the early days of the virtualization project in the FIO, lxdev19, a development machine, was drained to use it as a test machine for Xen. This machine has installed an 32-bit version of SLC 4 and Xen. It has not been updated for a while and might not be up to date.
Later on, a second machine, lxbuild048, was acquired and installed with a 64-bit version of SLC4 and Xen. On both of the machines different test scenarios were carried out which will be described in the following.

lxdev19:
  deployed virtual machines:
   
  • compnode
  •  
  • worknode
  •  
  • stornode
  •  
  • vm010
  •   test cases:
       
  • development, testing and building for lemon-sensor-vm
  •  
  • impact testing from ipmi on Xen networking
  •  
  • virtbench test and build
  •  
  • libvirt test and build
  •  
  • enomalism test
  •  
  • cpuburn tests
  •  
  • pypxeboot testing and development
  •   special installations:
       
  • ipmi toolsuite
  •  
  • pypxeboot testing and development
  •  
  • freeipmi
  •  
  • virt-top
  •  
  • cpuburn
  •  
  • xen development libraries and tools
  •  
  • lemon sensor development rpms
  •  
  • pypxeboot test setup
  •  
  • Xen sources
  • lxbuild048:
      deployed virtual machines:
       
  • vm0[06 - 10]
  •   test cases:
       
  • ncm-filesystems development and testing
  •  
  • ncm-xen testing
  •  
  • 64-bit Xen kernel testing
  •  
  • build and test host for lemon-sensor-vm
  •  
  • test host when modifications where made on PrepareInstall in regard to automated installation with Xen
  •  
  • virtual machines were first test candidates for lxvirt cluster
  •  
  • testing of lxxen cluster
  •   special installations:
       
  • none
  • The Linux Support Team which builds the Xen kernels has its own test machines. The ones mentioned above are for FIO/FS only.

    It is recommended to keep at least these two machines for testing purposes. It guarantees stress-free parallel testing for both architectures. If possible, one of the two machines should be replaced by one with hardware supported virtualization. Neither of the existing ones has this feature.

    2.3. Four simple definitions

    To keep the documentation and the understanding of virtualization a bit more interesting, varied and less boring for the people who are working in this area, many synonyms had been introduced. In this document some of them are used. To make sure that the reader really knows what is written about, the terms are defined in the following.

    Figure: Xen architecture

    a) Dom0 / Domain0 / privileged domain / xen-server / host

    Dom0, or domain zero, to expand the abbreviation, is the first domain started by the Xen hypervisor on boot. It has special privileges like being able to cause new domains to start or being able to access the hardware directly. It is responsible for running all of the device drivers for the hardware. For hardware that is made available to other domains, like network interfaces and disks, it will run the BackendDriver, which multiplexes and forwards to the hardware requests from the FrontendDriver in each DomU.
    Although any operating system can be ported to run on Xen as a DomU, only Linux has been given the tools and kernel patches necessary to run in Dom0.

    b) DomU / DomainU / unprivileged domain / virtual machine / vm / guest / partition

    A DomU is the counterpart to Dom0; it is an unprivileged domain with (by default) no access to the hardware. It must run a FrontendDriver for multiplexed hardware it wishes to share with other domains. A DomU is started by xend in Dom0, which the user accesses with the xm command-line tool.

    c) hypervisor / virtual machine monitor

    The hypervisor is Xen itself. It is between the hardware and the operating system of the various domains. The hypervisor is responsible for checking page tables, allocating resources for new domains, and scheduling domains. It presents the domains with a virtual machine that looks similar but not identical to the native architecture. It is also responsible for booting the machine to start the Dom0.
    Just as applications interact with an OS by giving it syscalls, domains interact with the hypervisor by giving it hypercalls. The hypervisor responds by sending an event to the domain. The event fulfills the same function as an IRQ on real hardware.

    d) virtual machine environment / virtualization environment

    The virtual machine environment puts it all together. The Dom0, the DomU and the hypervisor make up the virtualization environment. The whole system is able to run multiple operating systems simultaneously.

    All definitions are taken from the XenWiki.

    Once you have done this, your head may feel like it could explode every second, because of all the information you gathered. But now you have got an idea of what I am writing about in this article.

    3. Sysadmins, Operators and Procedures

    Production machines in the CERN Computing Centre are run by two groups of people - operators and system administrators.
    Operators cover daily management of the CC (including 7/24 operator coverage), logistical planning for the CC, hardware movements in the CC and warranty/escalation follow-ups for the systems in the CC.

    The system administration team is in charge of the administration of machines located in the Computing Centre, including large batch farms, data servers, interactive services, mail and Web facilities. The team, which started its activities in 2003, includes 9 system administrators on 3-year contracts and is partly renewed each year.

    Everything operators and system administrators need to know about a virtual machine environment, its configuration, installation and management, will be covered in the following chapters of this TWiki article. The aspects of configuration and installation are described in detail in separate sections. Questions of managing a virtual machine environment, additional hints, and useful tricks can be found in chapter XEN How To / FAQ and Tools.

    ALERT! Warning: Before you proceed to the next chapter and start configuring and installing the Xen-based virtualization environment, please make sure that the machines you will touch are in maintenance state!

    You can do this by using sms on an lxadm machine:

    [lxadm03] /afs/cern.ch/project/linux/redhat/kickstart/bin > sms get lxxen003  
    
    lxxen003: production
    Reason: other
    Comment: runs production Virtual Machines
    User: lefebure
    Request Date: 18/01/2008 18:09:52

    This machine is in production right now. To put it in maintenance mode do the following:

    [lxadm03] /afs/cern.ch/project/linux/redhat/kickstart/bin > sms set maintenance other 'xen installation' lxxen003
    Enter username (jmichael): 
    Enter password: 
    waiting for response...
    lxxen003: OK
    ...
    [lxadm03] /afs/cern.ch/project/linux/redhat/kickstart/bin > sms get lxxen003
    
    lxxen003: maintenance
    Reason: other
    Comment: Xen installation
    User: jmichael
    Request Date: 18/01/2008 18:26:52

    4. Configuration

    A virtualization environment based on Xen technology usually consists of one real physical machine, called the Dom0, and one or more virtual machines, called the DomU. The latter are depending on the physical machine configuration. Simply said: The better you equip the machine (with memory, processing power and disk space) the more virtual machines (DomUs) can be deployed on it.

    4.1. Select a physical machine

    Before you can start to set up a virtual machine environment, you need to have a used physical machine or bare metal. This is a recommended configuration for a physical machine:

    Manufacturer (Model): Elonex "ex_05_1"
    CPU: 2 * 2,8GHz Xeon
    RAM: 4096 MB
    Hard Disk: Hitatchi 250 GB SATA
    Network: Intel Gigabit Ethernet

    Assuming that you only have low utilized services, you can easily put five virtual machines (with 512 MB of RAM each) on this machine.
    Please keep in mind that services with heavy disk access are not suited for virtual machines. The disk is the weak link External site.

    One thing to note when choosing a hardware platform is, that you should take a system with hardware support for virtualization. This wikipedia article External site describes what hardware support means and what you have to look for.

    4.1.1. Hardware virtualization support

    To run full virtualized guests on systems with Hardware-assisted Virtual Machine (HVM) support, mostly Intel or AMD platforms, you must check your hardware to ensure that your CPUs have the capabilities needed to do so.

    To check if your machines' CPU has flags for Intel support, enter the following:

    grep vmx /proc/cpuinfo

    The output displays something like this:

    flags	:  fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm 
    syscall	nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm

    If a vmx flag appears then you have Intel support. To check if your machines CPU has flags for AMD support, enter the following:

    cat /proc/cpuinfo | grep svm

    The output displays:

    flags	:  fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dt acpi mmx fxsr sse sse2 ss ht tm 
    syscall nx mmtext fxsr_opt rdtscp lm 3dnowext pni cx16 lahf_lm cmp_legacy svm cr8_legacy

    If an svm flag appears you have AMD support.

    TIP Hint: In addition to checking the CPU flags, you should enable full virtualization in your systems' BIOS.

    One should also make sure to use native 64-bit machines since with all newer versions of Xen (and other virtualization technologies) you can run 32-bit and 64-bit guests (DomUs) on a 64-bit host.

    4.2. Register hosts and guests in LanDB

    Given the fact that you have found a physical machine, you have to register it in LanDB, if not already done. The hosting machine can be registered like a normal machine. Please regard the following naming convention: Xen hosts are named

    lxxen[012]

    The prefix should always be lxxen, following a sequential number with three digits. Please consider to number your machines consecutively. Use this link to find out which one is the machine with the highest number.

    An example can be found here: https://network.cern.ch/sc/fcgi/sc.fcgi?Action=SearchForDisplay&DeviceName=LXXEN001

    The registration of a virtual machine (DomU) is a bit more tricky, since it is not possible to specify information like manufacturer, model/type, serial number and mac address. The best way would be to take an existing virtual machine like lxvm0255, write an email to the Nework Operation Team, and ask them to create a virtual machine in LanDB for you. They know what to do and they have done this many times before. Do not hesitate to ask them for support. Registering virtual machines by contacting the Nework Operation Team also avoids the collision of two identical mac addresses in the network. Just tell them to take an existing machine as sample and in which subnet the new machine will reside.

    4.2.1. Separate Subnets for Dom0s and DomUs

    In the CERN the network infrastructure is designed in that way, that in one subnet only 240 IP addresses are free to use for machines. This limitation restricts the number of Dom0s and DomUs in one subnet. It was decided to put hosts and guests in different subnets.

    See service (its how the network operation team calls a subnet) S513-C-SI102 in LanDB for virtual machines (DomUs) and service S513-C-IP102 for host machines. It was a special configuration to the routers and other services like DHCP necessary to get this working. By the time I left CERN (end of September 2007) the DHCP changes had not been completed, so that we put some virtual machines (DomUs) in the same service with the hosts (Dom0s). For more information on this topic please contact:

    4.3. Configuration Database

    With having the the machines registered in LanDB we have fulfilled all premises to proceed with the configuration in CDB.

    4.3.1. Generating templates with network information

    All the network information of machines is stored in LanDB, but the configuration information, which is the basis for an installation of a machine, is stored in the configuration database (CDB). Every object profile (a machine profile), includes a template which contains all the networking information from LanDB. These templates are named:

    netinfo_<machine name>

    A little helper tool on lxadm cluster helps to generate the netinfo template:

    [lxadm03] /afs/cern.ch/user/j/jmichael > LEAFGenerateNetinfo lxvm0254
    User name: jmichael
    Password: 
    
    ############################################################
    #
    # template netinfo_lxvm0253
    #
    # Generated automatically with /usr/bin/LEAFGenerateNetinfo script
    #
    # Date: Fri Jan 18 13:02:37 2008 by jmichael
    #
    # Do NOT edit as your modifications might be overwritten
    # by the same script.
    #
    #
    ############################################################
    
    template netinfo_lxvm0253;
    
    
      "/system/network/hostname"				= "lxvm0253";
      "/system/network/domainname"				= "cern.ch";
      "/system/responsible/forename"			= "FS";
      "/system/responsible/familyname"			= "ADMINISTRATOR";
      "/hardware/location"						= "0513 R-0050";
    
      "/system/network/interfaces/eth0/ip"		= "128.142.129.205";
      "/system/network/interfaces/eth0/gateway" = "128.142.1.1";
      "/system/network/interfaces/eth0/netmask" = "255.255.0.0";
      "/system/network/interfaces/eth0/switchmedium" = 99;
    

    This terminal snippet shows the generation of the netinfo template for the virtual machine lxvm0253. Now cdbop (CDB command line interface) on lxadm cluster can be used to create a new template. Just copy the information printed out by LEAFGenerateNetinfo and paste it into a new file. Then upload and commit it via cdbop.

    More help on cdbop, especially on how to use it, can be found in the man page. CDBOP is only available on lxadm cluster.

    Having the template successfully committed to CDB we have the necessary network information for our machines (DomU and Dom0) in place for further configuration.

    4.3.2. Cluster lxvirt for virtual machines (DomUs)

    For all virtual machines a new cluster called lxvirt was defined, which holds the special configuration for virtual machines (kernel version, openafs and so on). It exist a separate profile for each architecture, the 32-bit (pro_type_lxvirt_slc4) and the 64-bit (pro_type_lxvirt_x86_64_slc4) one. Usually this cluster contains development or test machines, which are usually in maintenance state, but also machines on which are not yet decided in which cluster they are put in next.

    Therefore this cluster configuration provides an ideal template for designing new clusters.

    ALERT! Warning: It is not advisable to put a virtual machine in an conventional (non-virtual machine) cluster, because this causes definitively problems during installation and running!

    Now the cluster configuration will be described in detail, mainly those parts which differ from "normal" hosts, on template pro_type_lxvirt_x86_64_slc4:

    template pro_type_lxvirt_x86_64_slc4; 
    
      include pro_software_components_slc4; 
      include pro_system_lxvirt; 
    
    # select software packages 
      delete "/software/packages"; 
      include pro_software_lxvirt_x86_64_slc4; 
    
      "/system/cluster/tplname" = "pro_type_lxvirt_x86_64_slc4";
    

    Nothing special so far. The template contains the usual definitions and the clusters template name is set (last line).

    # Lemon monitoring 
      include pro_monitoring; 
      include pro_monitoring_metrics_default; 
      include pro_monitoring_metrics_afs; 
      include pro_monitoring_metrics_linux; 
      include pro_monitoring_metrics_fmonagent;
    

    In comparison to the lxdev cluster profile some unnecessary monitoring was removed.

    # DomU configuration 
      include pro_service_xenu;			
    

    This template refines the configuration for virtual machine. This template is covered in detail subsequently.

    # software component configuration 
      "/software/components/chkconfig/service/apmd/del" = true; 
      "/software/components/chkconfig/service/gpm/del" = true; 
      "/software/components/chkconfig/service/lpd/del" = true; 
      "/software/components/chkconfig/service/xfs/del" = true; 
      "/software/components/chkconfig/service/psacct/on" = ""; 
    
    # disable services which are not working on a DomU 
      "/software/components/chkconfig/service/ntpd/del" = true; 
      "/software/components/chkconfig/service/smartd/del" = true; 
      "/software/components/chkconfig/service/microcode_ctl/del" = true; 
      "/software/components/chkconfig/service/cc-ipmi/del" = true; 
    
      "/software/components/regisclient/localusers" = true; 
    
      include pro_declaration_component_logrotate; 
      "/software/components/logrotate/active" = true; 
    
      include pro_declaration_component_sysacct; 
      "/software/components/sysacct/active" = true; 
      "/software/components/sysacct/sendstatistics" = "no"; 
      "/software/components/sysacct/archive" = "no"; 
    
      include pro_declaration_component_ssh; 
      "/software/components/ssh/daemon/options/AFSTokenPassing" = "yes"; 
      "/software/components/ssh/daemon/options/ChallengeResponseAuthentication" = "no"; 
      "/software/components/ssh/daemon/options/GSSAPIAuthentication" = "yes"; 
      "/software/components/ssh/daemon/options/KerberosAuthentication" = "yes"; 
      "/software/components/ssh/daemon/options/KerberosGetAFSToken" = "yes"; 
      "/software/components/ssh/daemon/options/KerberosTgtPassing" = "yes"; 
      "/software/components/ssh/daemon/options/PasswordAuthentication" = "yes"; 
      "/software/components/ssh/daemon/options/PermitEmptyPasswords" = "no"; 
      "/software/components/ssh/daemon/options/PermitRootLogin" = "yes"; 
      "/software/components/ssh/daemon/options/UsePAM" = "yes"; 
      "/software/components/ssh/daemon/options/UsePrivilegeSeparation" = "yes"; 
      "/software/components/ssh/daemon/options/X11UseLocalhost" = "yes"; 
      "/software/components/ssh/daemon/options/MaxStartups" = 50; 
      "/software/components/ssh/active" = true; 
      "/software/components/ssh/daemon/options/Protocol" = "2"; 
    
    # create access lists 
      "/software/components/access_control" = generate_acl_list(value("/software/components/access_control")); 
    
    # End-of-File
    

    Please note that some hardware related services where removed because a virtual machine operates on virtual hardware and therefore a service like smartd or the "microcode_ctl" simply will not work. Additionally the SSH service is prepared for authentication with Kerberos and protocol version 2 only.

    4.3.2.1. Service Template for virtual machines

    Now we get to the heart of it: pro_service_xenu. This template is the same for all architectures.

    ############################################################################## 
    # # 
    # Pan template name : pro_service_xenu # 
    # Responsible : Jan Michael # 
    # Creation Date : Mon Aug 13 13:40:37 CEST 2007 # 
    # Url : # 
    # Description : Service template for Xen based domUs # 
    # Modifications : # 
    # * 2007-09-27 Jan Michael # 
    # - updated kernel version for x86_64 architecture. # 
    # * 2007-09-17 Jan Michael # 
    # - changed kernel and modules for i386. # 
    # * 2007-08-13 Jan Michael # 
    # - template created. # 
    # # 
    ############################################################################## 
    
    template pro_service_xenu; 
    
    # variable that holds architecture information 
      define variable architecture = value("/system/architecture"); 
    

    As said afore, the template architecture is independent. The variable architecture helps later to configure the correct packages.

    # put machine status in maintenance since there are no operational procedures defined yet 
      "/system/defaultstate" = "maintenance"; 
    

    All machines which have this template included are put in maintenance state by default. In the beginning of the introduction of virtual machine this was necessary to avoid alarms, caused by misconfiguration. Nowadays this could be moved to some hierarchically higher profile.

    # openafs packages 
      "/software/packages" = pkg_repl("openafs","1.4.4-3.CERN",architecture); 
      "/software/packages" = pkg_repl("openafs-krb5","1.4.4-3.CERN",architecture); 
      "/software/packages" = pkg_repl("openafs-compat","1.4.4-3.CERN",architecture); 
      "/software/packages" = pkg_repl("openafs-kpasswd","1.4.4-3.CERN",architecture); 
      "/software/packages" = pkg_repl("openafs-client","1.4.4-3.CERN",architecture); 
    

    ALERT! Warning: These are special openafs packages, which directly correspond to the kernel module and hence to a specific kernel version. If the kernel have to be updated due to some security or any other reason, please make sure that you do not break AFS!

    # kernel packages 
      "/software/packages"={ 
    	if ( architecture == "i386"){ 
    	  pkg_add("kernel-module-openafs-2.6.9-55.0.2.EL.CERNxenU","1.4.4-3.CERN","i686"); 
    	  pkg_add("kernel-xenU","2.6.9-55.0.2.EL.CERN","i686"); 
    	  pkg_add("kernel-xenU-devel","2.6.9-55.0.2.EL.CERN","i686"); 
    	} else { 
    	  pkg_add("kernel-module-openafs-2.6.9-55.0.6.EL.CERN1xenU","1.4.4-3.CERN",architecture); 
    	  pkg_add("kernel-xenU","2.6.9-55.0.6.EL.CERN1",architecture); 
    	  pkg_add("kernel-xenU-devel","2.6.9-55.0.6.EL.CERN1",architecture); 
    	}; 
      }; 
    
    # kernel version 
      "/system/kernel/version" = { 
    	if (architecture == "i386"){ 
    		"2.6.9-55.0.2.EL.CERNxenU"; 
    	} else { 
    		"2.6.9-55.0.6.EL.CERN1xenU"; 
    	}; 
      }; 
    
    # disable some certain monitoring stuff 
    # switch off smartd exception 
      "/system/monitoring/exception/_30056/active" = false; 
      "/software/components/smartd/active" = false; 
    
    # only 1 CPU will be monitored 
      "/system/monitoring/exception/_30501/correlation" = "4115:6 != 1"; 
    
    # End-of-File 
    

    This last section covers the kernel configuration and adjusts some lemon monitoring details.

    The pro_type_lxvirt_<x86_64>_slc4 should be the only template which contains configuration options for DomUs. If the kernel will be updated or some additional options needed to be specified it should be done in this template globally.

    4.3.3. Cluster lxxen for hosts (Dom0)

    Nearly all Xen host environments reside in the lxxen cluster or one of its sub clusters. There are some exceptions with old machines which already existed before lxxen was created.
    In general these machines have no service specific configuration, which would make it necessary to put them in another cluster. Their only purpose is to host virtual machines in different configurations, but they have all the same software configuration. Only service managers, system administrators and operators should have (root) access to these machines.

    Just like the lxvirt cluster, the lxxen cluster is defined by a set of templates based on the pro_type_lxxen_x86_64_slc4 for the 64-bit machines and on pro_type_lxxen_slc4 for the 32-bit ones.

    The pro_type_lxxen_x86_64_slc4 from the production stage looks like this:

    template pro_type_lxxen_x86_64_slc4; 
    
    # select system 
      include pro_system_lxxen; 
    
    # select software packages 
      include pro_software_lxxen_x86_64_slc4; 
    

    The software template is roughly equivalent to a default SLC4 one with one exception. It contains this additional part

    # Software for quattor components 
      # ncm-serial_console 
      include pro_software_console;
    

    which configures the installation of the software for the console server and client.

    # components 
      include pro_software_components_slc4; 
    
    # Lemon monitoring 
      include pro_monitoring_slc4; 
      include pro_monitoring_vm; 
      # Dom0 kernels do not support AFS, but it is included in pro_monitoring_slc4 --> so do not monitor it! 
      "/system/monitoring/metric/_46/active" = false; 
      "/system/monitoring/exception/_30020/active" = false; 
      "/system/monitoring/metric/_910/active" = false; 
      "/system/monitoring/exception/_30086/active" = false; 
      "/system/monitoring/metric/_911/active" = false; 
      "/system/monitoring/exception/_30064/active" = false; 
    
      # disabling the HWSCAN_WRONG exception which occurs due to changing memory size in Dom0 
      "/system/monitoring/exception/_30535/correlation" = "5102:1>19 && 5102:1!=41 && 5102:2 ne 'OK' && 5102:1!=90'"; 
      "/system/monitoring/metric/_4008/param" = list("name","atd","uid","0","ppid","1"); # atd runs as root on SLC4/RHES4... 
    

    Some specialties for lemon monitoring including a new sensor configured by pro_monitoring_vm which takes care of measuring the cpu utilization of host and virtual machines together.

    # Xen stuff (inlcudes software packages, monitoring, disk layout, preconfigured virtual machines) 
      include pro_service_lxxen; 
    

    This template is mainly responsible for special kernel versions and preparation of the potential guest. It will be described in detail later.

      "/system/cluster/tplname" = "pro_type_lxxen_x86_64_slc4"; 
    
    # software component configuration 
      "/software/components/chkconfig/service/apmd/del" = true; 
      "/software/components/chkconfig/service/gpm/del" = true; 
      "/software/components/chkconfig/service/lpd/del" = true; 
      "/software/components/chkconfig/service/xfs/del" = true; 
      "/software/components/chkconfig/service/psacct/on" = ""; 
    
      "/software/components/regisclient/localusers" = true; 
    
      include pro_declaration_component_logrotate; 
      "/software/components/logrotate/active" = true; 
    
      include pro_declaration_component_ssh; 
      "/software/components/ssh/daemon/options/AFSTokenPassing" = "yes"; 
      "/software/components/ssh/daemon/options/ChallengeResponseAuthentication" = "no"; 
      "/software/components/ssh/daemon/options/GSSAPIAuthentication" = "yes"; 
      "/software/components/ssh/daemon/options/KerberosAuthentication" = "yes"; 
      "/software/components/ssh/daemon/options/KerberosGetAFSToken" = "yes"; 
      "/software/components/ssh/daemon/options/KerberosTgtPassing" = "yes"; 
      "/software/components/ssh/daemon/options/PasswordAuthentication" = "yes"; 
      "/software/components/ssh/daemon/options/PermitEmptyPasswords" = "no"; 
      "/software/components/ssh/daemon/options/PermitRootLogin" = "yes"; 
      "/software/components/ssh/daemon/options/UsePAM" = "yes"; 
      "/software/components/ssh/daemon/options/UsePrivilegeSeparation" = "yes"; 
      "/software/components/ssh/daemon/options/X11UseLocalhost" = "yes"; 
      "/software/components/ssh/daemon/options/MaxStartups" = 50; 
      "/software/components/ssh/active" = true; 
      "/software/components/ssh/daemon/options/Protocol" = "2"; 
    
      include pro_declaration_component_serial_console; 
      "/software/components/serial_console/active" = true; 
    
    # create access lists 
      "/software/components/access_control" = generate_acl_list(value("/software/components/access_control")); 
    
    # End-Of-File
    

    Also nothing uncommon in here. The comprehension of the serial console component, which gives access to Dom0 as well as DomU via headnode.

    4.3.3.1. Service template for hosting machines

    Now we take a closer look to the pro_service_lxxen template. This template can be used for either architectures, 32- and 64-bit.

    ################################################################################ 
    # # 
    # Pan template name : pro_service_lxxen # 
    # Responsible : Jan Michael # 
    # Creation Date : Tue Sep 4 13:59:19 CEST 2007 # 
    # Url : https://twiki.cern.ch/twiki/bin/view/LinuxSupport/ # 
    # InstallingXenHypervisor # 
    # Description : Type template for virtualization cluster "lxxen" # 
    # Modifications : # 
    # * 2007-09-17 Jan Michael # 
    # - commented out netinfo_xen_guestmap. # 
    # * 2007-09-17 Jan Michael # 
    # - reorganized monitoring stuff to pro_type_lxxen. # 
    # * 2007-09-04 Jan Michael # 
    # - template created. # 
    # # 
    ################################################################################ 
    template pro_service_lxxen; 
    
    define variable myarch = value("/system/architecture"); 
    
    # Software Packages 
      "/software/packages" = pkg_repl("bridge-utils","1.0.4-4",myarch); 
      "/software/packages" = pkg_repl("device-mapper-multipath","0.4.5-16.1.RHEL4",myarch); 
      "/software/packages" = pkg_repl("e2fsprogs-libs","1.39-7.slc4",myarch); 
    
      "/software/packages" = pkg_del("glibc","2.3.4-2.25","i686"); 
      "/software/packages" = pkg_del("glibc-common","2.3.4-2.25","i686"); 
      "/software/packages" = pkg_del("glibc-devel","2.3.4-2.25","i686"); 
      "/software/packages" = pkg_del("glibc-headers","2.3.4-2.25","i686"); 
    
      "/software/packages" = pkg_del("glibc","2.3.4-2.36","i686"); 
      "/software/packages" = pkg_del("glibc-common","2.3.4-2.36","i686"); 
      "/software/packages" = pkg_del("glibc-devel","2.3.4-2.36","i686"); 
      "/software/packages" = pkg_del("glibc-headers","2.3.4-2.36","i686"); 
    
      "/software/packages" = pkg_repl("e2fsprogs","1.39-7.slc4",myarch); 
      "/software/packages" = pkg_repl("e2fsprogs-devel","1.39-7.slc4",myarch); 
      "/software/packages" = pkg_repl("mkinitrd","5.0.26-3.slc4",myarch); 
      "/software/packages" = pkg_repl("xfsprogs"); 
      "/software/packages" = pkg_repl("xen-libs","3.0.3.0-12.slc4",myarch); 
      "/software/packages" = pkg_repl("sysfsutils","1.2.0-1",myarch); 
      "/software/packages" = pkg_repl("xen","3.0.3.0-12.slc4",myarch); 
      "/software/packages" = pkg_repl("glibc-common","2.3.4-2.25.xen",myarch,"multi"); 
      "/software/packages" = pkg_repl("glibc-devel","2.3.4-2.25.xen",myarch,"multi"); 
      "/software/packages" = pkg_repl("glibc-headers","2.3.4-2.25.xen",myarch,"multi"); 
      "/software/packages" = pkg_add("glibc","2.3.4-2.25.xen","i686","multi"); 
    

    This section defines all Xen based packages as well as necessary dependent packages.
    The Linux Support Team releases regularly updates for Xen and the related packages. Once the responsible service manager approves new package(s) they should be updated here by replacing the old ones.

    Because Scientific Linux CERN (SLC4) is based on the Red Hat Enterprise Linux (RHEL) External site in version 4, the updates by the Linux Support Team rely usually on the updates of RHEL4 External site. Exceptions are special CERN related packages or security related updates. These procedures apply also to all upcoming versions of SLC, like version 5 in 2008.

      "/software/packages" = pkg_add("ncm-xen"); 
    

    The ncm-xen package is essential for the automated creation of DomU configuration files on the Dom0. It is primarily responsible for translating options specified in the CDB profile into Xens domain configuration files (usually in /etc/xen). Some of the package details are explained below.

      "/software/packages"={ 
      # X86_64 
    	  if (myarch == "x86_64"){ 
    
      # Additional packages: 
    		  pkg_repl("kernel-xen","2.6.18-1.2835.slc4","x86_64"); 
    		  pkg_add("glibc","2.3.4-2.25.xen","x86_64","multi"); 
    	 } else { 
    
      # i386 
    		  pkg_repl("kernel-xen","2.6.18-1.2835.slc4","i686"); 
    	 }; 
      }; 
    
    # Grub 
      "/system/kernel/version" = "2.6.18-1.2835.slc4xen"; 
    

    This stuff was already described within the pro_service_xenU template section. It simply configures the appropriate kernel version for the DomU and arranges GNU Grub External site to set the chosen kernel to default. This part of the template has to be modified when updating the kernel or the original Xen packages.

    # Xen configuration: 
    # The hypervisor will be prepared (disk and xen) for 5 virtual machines 
      include pro_software_component_xen; # Xen component 
      #include netinfo_xen_guest_map; # relation Dom0 - DomU and mac addresses 
      include pro_software_component_filesystems; # filesystems component for partition layout for DomU 
    

    This part includes the default configuration for the ncm-xen component which was included beforehand and also includes the default configuration and the software package for ncm-filesystems component.
    The ncm-filesystems component is developed by Luis Fernando Muñoz Mejías based on the specifications of the TWiki page New Quattor blockdevice structure proposal.

    Virtual machines (DomUs) make use of a lvm-backed External site virtual block device. The component is responsible to prepare a logical volume for each virtual machine on the free disk space remaining on the host.

    ALERT! Warning: The ncm-filesystems component is not yet ready for production use!

    It was tested by me just to fulfill the purpose of creating partitions and lvm stuff for the virtual guests (DomUs). It may be used for other purposes too, but not without consultation of Luis Fernando Muñoz Mejías. In addition to that there are also further plans to move the definition paths to other locations.

      # xend daemon 
        "/software/components/chkconfig/service/xend/add" = true; 
        "/software/components/chkconfig/service/xend/on" = ""; 
    

    The xend (xen daemon) External site is added to the machines services and is switched on by default.

      # component dependencies 
        "/software/components/xen/dependencies/pre" = list("filesystems"); # This should be moved to pro_software_component for production 
    

    The Xen component can only configure virtual machines if there is an appropriate disk back-end. To ensure this the ncm-filesystems component is set to be run before the ncm-xen component can.

      # variables, needed by component configuration 
        variable XEN_GUESTS_NAME = list ("xxxxxx1", "xxxxx2", "xxxxx3", "xxxxx4", "xxxxx5"); 
        variable XEN_LVOL_NAME = list ("vol1", "vol2", "vol3", "vol4", "vol5"); 
        variable XEN_GUESTS_MAC = list ("xx:xx:xx:xx:xx:xx", "xx:xx:xx:xx:xx:xx", "xx:xx:xx:xx:xx:xx", "xx:xx:xx:xx:xx:xx", "xx:xx:xx:xx:xx:xx"); 
    

    These are some variables to avoid redundancy for later reuse in the template. XEN_GUESTS_NAME and XEN_GUESTS_MAC variables will be overwritten in a concrete Dom0 profile.

      # ncm-filesystems configuration 
        "/software/components/filesystems/blockdevices" = nlist ( 
      	  "physical_devs", nlist ( 
      							  "sda", nlist ("label", "msdos") 
      							 ), 
      	  "partitions", nlist ( 
      							  "sda4", nlist ( 
      											 "holding_dev", "sda", 
      											 "type", "extended", 
      											), 
      							  "sda5", nlist ( 
      											 "holding_dev", "sda", 
      											 "type", "logical", 
      											) 
      						  ), 
      	  "volume_groups", nlist ( 
      							  "xen_vg", nlist ( 
      											   "device_list", list ("partitions/sda5"), 
      											  ) 
      							 ), 
      	  "logical_volumes", nlist ( 
      								XEN_LVOL_NAME[0], nlist ( 
      														   "size", 20480, 
      														   "volume_group", "xen_vg" 
      														  ), 
      								XEN_LVOL_NAME[1], nlist ( 
      														   "size", 20480, 
      														   "volume_group", "xen_vg" 
      														  ), 
      								XEN_LVOL_NAME[2], nlist ( 
      														   "size", 20480, 
      														   "volume_group", "xen_vg" 
      														  ), 
      								XEN_LVOL_NAME[3], nlist ( 
      														   "size", 20480, 
      														   "volume_group", "xen_vg" 
      														  ), 
      								XEN_LVOL_NAME[4], nlist ( 
      														   "size", 20480, 
      														   "volume_group", "xen_vg" 
      														  ) 
      							   ) 
        ); 
    
        "/software/components/filesystems/filesystemdefs" = { 
    
        result = list(); 
        i = 0; 
    
        while (i < length(XEN_GUESTS_NAME)) { 
    
      	result[i]["mount"] = false; 
      	result[i]["mountpoint"] = "/var/xen/mnt/" + XEN_LVOL_NAME[i]; 
      	result[i]["preserve"] = false; 
      	result[i]["format"] = false; 
      	result[i]["mountopts"] = "noauto"; 
      	result[i]["block_device"] = "logical_volumes/" + XEN_LVOL_NAME[i]; 
      	result[i]["type"] = "none"; 
      	result[i]["freq"] = 0; 
      	result[i]["pass"] = 1; 
    
      	i = i + 1; 
        }; 
    
        result; 
      }; 
    

    This template snippet represents the default configuration for ncm-filesystems.
    At first a physical device (physical_devs) called sda is created to hold all further configuration. On top of that, two partitions, sda4 as an extended one, and sda5 as the a big logical one, which claims all the remaining disk space. The sda5 partition is further used to create a volume group called xen_vg. This group serves as the base for five logical volumes XEN_LVOL_NAME[0-4], each with 20 GB in size.

    Additionally the components needs a filesystem definition for each logical volume. This is done within a while loop in the last lines of the code snippet above.
    Actually there is no filesystem needed to be created in the logical volume now, because this is done later at installation time of the virtual machine. However it is mandatory. A minimal set of options is given to preserve the functionality of the component.

    An explanation of the options used here can be found in the component docuementation.

    ALERT! Warning: This is a very static configuration and not suitable for all types of hardware, especially not for machines with sata disk interfaces or newer and more powerful disk array layouts. In the latter cases the device mapping will be different and will definitively break the installation

    # ncm-xen configuration 
      "/software/components/xen/create_filesystems" = false; # Filesystems must not be created. This will be done by ncm-filesystems. 
      "/software/components/xen/create_domains" = true; 
    
      "/software/components/xen/domains" = { 
    
       result = list(); 
       i = 0; 
    
       while (i < length(XEN_GUESTS_NAME)) { 
    
    	options = nlist(); # configuration for normal boot 
    	install_options = nlist(); 
    
    	options["memory"] = 512; 
    	options["name"] = XEN_GUESTS_NAME[i]; 
    	options["vif"] = list("mac=" + XEN_GUESTS_MAC[i]); 
    	options["ip"] = "dhcp"; 
    	options["disk"] = list( nlist( "type", "lvm", 
    								   "hostdevice", "xen_vg", 
    								   "path", "", 
    								   "size", 0, 
    								   "guestdevice", "xvda", 
    								   "rw", "w", 
    								   "hostvol", XEN_LVOL_NAME[i] 
    								 ) 
    						  ); 
    	options["on_reboot"] = "restart"; 
    	options["on_crash"] = "destroy"; 
    	options["on_poweroff"] = "destroy"; 
    
    	result[i]["options"] = options; 
    	result[i]["options"]["bootloader"] = "/usr/bin/pygrub"; 
    
    	result[i]["install_options"] = options; 
    	result[i]["install_options"]["kernel"] = "/etc/xen/vmlinuz"; 
    	result[i]["install_options"]["on_reboot"] = "destroy"; 
    	result[i]["install_options"]["ramdisk"] = "/etc/xen/initrd.img"; 
    	result[i]["install_options"]["download"] = list("/usr/bin/wget -N -P /etc/xen http://linuxsoft.cern.ch/CERN/slc45/" + myarch + "/images/SL/xen/initrd.img", 
    													"/usr/bin/wget -N -P /etc/xen http://linuxsoft.cern.ch/CERN/slc45/" + myarch + "/images/SL/xen/vmlinuz"); 
    	result[i]["install_options"]["extra"] = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + XEN_GUESTS_NAME[i] + ".ks"; 
    
    	result[i]["auto"] = false; 
    
    	i = i + 1; 
      }; 
    
      result; 
      }; 
    
    # End-Of-File 
    

    The last paragraph of the template defines the Xen configuration of five virtual machines, which is hold in /software/components/xen/.

    For every DomU two configuration files will be created. One contains the configuration for the installation. These options are located in the install_option sub-paths. The installation configuration is different from the normal configuration for running the machine because it contains additional parameter like an installation kernel (result[i]["install_options"]["kernel"]), an image with installation files (result[i]["install_options"]["ramdisk"]), the location of the kickstart file, other kernel options (result[i]["install_options"]["extra"]) and instructions for the ncm-xen component to download the kernel and the initrd image (result[i]["install_options"]["download"]).

    These two different configurations are the reason for the semi-automated installation procedure and the necessary manual intervention to be made by sysadmins or operators. A possible solution to fix this is described in the open challenges chapter.

    The default configuration assigns 512 MB memory (can be overwritten if required), the network address to be acquired by DHCP, the disk to be used (one of the logical volumes created earlier) and pygrub to be used as boot loader to each of the default virtual machines.

    There are also instructions given to the Xen daemon what to do if the virtual machine reboots (options["on_reboot"] = "restart"), crashes (options["on_crash"] = "destroy") or is shut down (options["on_poweroff" = "destroy").

    All these configuration settings give a rudimental setup for five virtual machines and reduce lots of paperwork later. Every option can be redefined and thus be overwritten in the Dom0 or DomU profile.

    Both clusters, lxxen for the hosts (Dom0) and lxvirt for virtual machines (DomU) provides a solid fundament for either putting machines in it or derive new cluster configuration from them.

    4.3.4. Modelling the relationship between Dom0 and DomU

    With the introduction of virtual machines and other systems like blades or twin system to the CERN Computing Centre an enclosure contains now more than one computing element. This complicates the way machines were managed so far. If, for example, the host system of a virtual machine needs to get moved to another location or the hard disks have to get a firmware update than one needs also to take care of the virtual machines deployed on this host.

    There is Twiki page set up by Alejandro Iribarren half a year ago which addresses this problem and give a more detailed description. It includes also a list of all tools that which have been or will be updated in this context.

    In short:

    From two possible solutions it was agreed to let the parent reference the child in CDB. This means that in case of a virtual machine environment the host (Dom0) profile references its virtual machine (DomU) profiles.

    The attentive reader has already noticed that this would never help to get to know on which Dom0 a DomU is deployed.
    To provide the inverse relationship between the child (DomU) and the parent (Dom0) a special table view enclosures has been set up in CDBSQL. To make use of this one can either connect to CDBSQL directly using the appropriate interface in the programming or scripting language of choice or execute the CDBDumptable.pl on one of the lxadm machines:

    [lxadm02] /afs/cern.ch/user/j/jmichael > CDBDumptable.pl ENCLOSURES
    
    ID=639	HOSTNAME=lxbladedell    CHILDNAME=lxbladedell02     ENCLOSURETYPE=blade         MAXCHILDREN=10
    ID=640	HOSTNAME=lxbladedell    CHILDNAME=lxbladedell03     ENCLOSURETYPE=blade         MAXCHILDREN=10
    ID=641	HOSTNAME=lxbuild048     CHILDNAME=vm006             ENCLOSURETYPE=hypervisor    MAXCHILDREN=5
    ID=642	HOSTNAME=lxbuild048     CHILDNAME=vm007             ENCLOSURETYPE=hypervisor    MAXCHILDREN=5
    ID=643	HOSTNAME=lxbuild048     CHILDNAME=vm008             ENCLOSURETYPE=hypervisor    MAXCHILDREN=5
    ID=644	HOSTNAME=lxbuild048     CHILDNAME=vm009             ENCLOSURETYPE=hypervisor    MAXCHILDREN=5
    ID=645	HOSTNAME=lxbuild048     CHILDNAME=vm010             ENCLOSURETYPE=hypervisor    MAXCHILDREN=5
    ID=646	HOSTNAME=lxxen001       CHILDNAME=lxvm0240          ENCLOSURETYPE=hypervisor    MAXCHILDREN=5
    ID=647	HOSTNAME=lxxen001       CHILDNAME=lxvm0242          ENCLOSURETYPE=hypervisor    MAXCHILDREN=5
    

    4.3.5. Creating a virtual machine profile

    The issue mentioned before and the fact that values from the DomU can be reused in the Dom0, forces us to first create a DomU CDB object profile. This will be included later in the Dom0 profile.

    There are several ways of creating a new profile for a virtual machine. The easiest and at least complex method would be to reuse an existing template and just copy and modify it. Another approach would be using the LEAFAddHost script from the LEAFTools.

    [lxadm02] /afs/cern.ch/user/j/jmichael > LEAFAddHost 
    Usage: 
    	/usr/bin/LEAFAddHost 
    		 --new_host= 
    		 --cluster= 
    		 --serial_number 
    		 --rack= 
    		 [--contractid=] 
    
    Quattor-Managed:
    
    		 --hardware=	
    
    Non-Quattor-Managed:
    
    		 --nq	
    		 --service_code= 
    
    Options:
    
    		 --files  
    		 --mac1=
    		 --mac2=
    		 --mac3=
    		 --mac4=
    		 --mac5=
    		 --mac6=
    		 --serialinfo=";;;;;"
    		 --enclosureinfo=";;;;"
      or
    		 --enclosureinfo=";;;;" if it is known that the enclosure is already in CDB	
    		 --extra=
    		 --cdbcfgfile=
    
    Authorization:
    	May use CDB_USER and CDB_PASSWORD environment variables;
    	or create a CDB config file containing:
    
    		CDB_USER=fred
    		CDB_PASSWORD=xxx
    
    	and give its location with the --cdbcfgfile option;
    	or simply enter them from STDIN when asked for.
    
    Wrong Arguments
     Please define the cluster type with --cluster option				
    

    The LEAFAddHost script creates a basic profile which not necessarily works out of the box and therefore it can not be recommended creating a profile this way.

    Below you will find a typical DomU profile. It is a snapshot of a 32-bit virtual machine from Mon, 14 January 2008 18:21:27, named lxvm0254:

    ################################################################################ 
    # # 
    # Pan template name : profile_lxvm0254 # 
    # Responsible : Jan Michael # 
    # Creation Date : Fri Sep 14 10:40:40 2007 # 
    # Url : # 
    # Description : object template for virtual machine lxvm0254 (32-bit) # 
    # Modifications : # 
    # * 2007-09-27 Jan Michael # 
    # - added root access for Daniel Dengate (dengate). # 
    # - added yum (pro_service_yum). # 
    # * 2007-09-14 Jan Michael # 
    # - template created. # 
    # # 
    ################################################################################ 
    
    object template profile_lxvm0254; 
    include stages/prod; 
    
    # include profile_base for use of typed properties 
      include pro_declaration_profile_base; 
    
    # used resources 
      include pro_hardware_xenu; 
    

    The profile includes a template with virtual hardware defined for Xen based virtual machines.
    In correspondence to the service template pro_service_lxxen this template contains a 20GB hard drive, a virtual network card, one virtual CPU and RAM of 1MB. The values for disk, cpus and ram recur in the Dom0 profile and in the Xen configuration file. Currently they are defined two times. One time in the host profile and one time in the DomU profile.

    In the future it is planned to have only a single point where the data should kept. And this is in the DomU profile. Every other place where those values are needed should reference to the DomU profile. However this requires a lot of changes in all profiles and can therefore be found in the open challenges chapter.

      include netinfo_lxvm0254; 
      include pro_type_lxvirt_slc4; 
    

    The netinfo_lxvm0254 template was generated in section Generating templates with network information and contains the networking information for the machine. With having the pro_type_lxvirt_slc4 template included, this machine becomes part of the lxvirt cluster.

    # redefine hardware parameter like CPU, DISK and RAM 
      "/hardware/ram" = list(nlist("size", 512)); 
      "/hardware/harddisks/xvda/capacity/" = 20*GB; 
      "/hardware/cpus" = list (create("pro_hardware_cpu_virtual"), 
    											   create("pro_hardware_cpu_virtual") 
    											  ); 
    

    In this section the hardware specification gets redefined. In some cases the default configuration, which has been set up in the hardware template pro_hardware_xenu, is not sufficient. For instance the client needs more disk space, more RAM or requests more CPU power. Than this can be done in this section accordingly.

    # virtual machines have no serial number or rack name (see hypervisor for those information) 
      "/hardware/serialnumber" = ""; 
      "/hardware/rack/name" = ""; 
    

    Usually the serial number of a machine is used for physical machines as a unique id to track them in the CC (Computing Centre) or settle vendor specific affairs. The rack name helps to locate the machine in the CC via pc finder (integrated in lemonweb) or CC Tracker, a special tool, written in Java and only dedicated to this purpose.

    Both data fields do not fit for a virtual machine. A serial number is not needed because the hardware is virtual and can not be directly referenced. A similar problem occurs with the rack name. Virtual machines are not bound to specific machines. The currently used version of Xen and more than ever newer version provide the possibility of live migration, which allows one to move a virtual machine everywhere around in the CC. This would mean that the rack name had to be updated every time a DomU moves.

    Right now it is not intended to use live migration at CERN, but maybe, as time goes by, it becomes relevant in the future.

    Tracking of virtual machines is ensured with proper modeling of enclosures as described in section Modelling the relationship between Dom0 and DomU.

    # MAC address should be in colons. Otherwise configuration file for hypervisor get broken! 
      "/hardware/cards/nic/0/hwid" = "[obfuscated for security reason]"; 
    
    # further customization 
    
      # machine function 
      "/system/function" = "Cobbler/Koan tests by Daniel Dengate (daniel.robert.dengate@cern.ch)"; 
    
      # allow user installed rpms 
      "/software/components/spma/userpkgs" = "yes"; 
      "/software/components/spma/userprio" = "yes"; 
    
      include pro_service_yum; 
    
      # access controls 
      "/software/components/access_control/roles/dengate/0"="dengate"; 
      "/software/components/access_control/privileges/acl_root/role/dengate/0/targets"=list("+node::lxvm0254"); 
      "/software/components/access_control/privileges/acl_interactive/role/dengate/0/targets" = list("+node::lxvm0254"); 
      "/software/components/access_control" = generate_acl_list(value("/software/components/access_control")); 
    
    # End-of-File
    

    The last section of the object profile makes some user specific customizations to the machine which are not related to its status as virtual machine.

    The profile needs now to be added or modified and then committed to CDB using cdbop. Once all the DomU profiles which should be deployed on a host (Dom0) are in CDB, one can continue with creating the profile for the Dom0.

    4.3.6. Creating a host (Dom0) profile

    To create the host (Dom0) profile and be able to successfully commit it to CDB, all child profiles, which means all profiles of all virtual machines which should be deployed on this machine, must be already committed to CDB.
    To find out how to do that, read section Creating a virtual machine profile.

    In most cases a host profile will not be created from scratch. Instead of, one takes an existing machine with an existing profile and modifies it. Two ways to create a completely new profile can be found in section Creating a virtual machine profile.

    Whatever way one prefers to create the host profile, it is strongly recommended to read the following notes to the template of profile_lxxen003:

    ################################################################################ 
    # # 
    # Pan template name : profile_lxxen003 # 
    # Responsible : Jan Michael # 
    # Creation Date : Wed Sep 5 09:56:13 CEST 2007 # 
    # # 
    ################################################################################ 
    
    object template profile_lxxen003; 
    include stages/usertest/vero/lxxen; 
    

    Typically the profile should include stages/prod, which indicates that the profile is in production state. Right now it includes the testing stage for lxxen machines of Véronique Lefèbure. With the utmost probability there is some testing in progress.

    # include profile_base for use of typed properties 
      include pro_declaration_profile_base; 
    
    # used resources 
      include pro_hardware_ex_05_1; 
      include netinfo_lxxen003; 
      include pro_type_lxxen_slc4; 
    

    As recommended in section Cluster "lxxen" for hosts (Dom0) the template pro_type_lxxen_slc4 for 32-bit Dom0s is included. In consequence to that the machine will belong to the lxxen cluster.
    Please note also that a hardware profile (pro_hardware_ex_05_1) of a real machine is included. Which hardware template one has to use, depends on the physical machines hardware model. Please ask the procurement team for further details on your specific case.

    # lemon monitoring 
      include pro_monitoring_hardware_ex_05_1; 
      # disabling the HWSCAN_WRONG exception which occurs due to changing memory size in Dom0 
      "/system/monitoring/exception/_30535/correlation" = "5102:1>19 && 5102:1!=41 && 5102:2 ne 'OK' && 5102:1!=90"; 
    

    The monitoring of the host (Dom0) needs some specific modification. Once a virtual machine gets deployed on a host it requests exactly that amount of memory from the Dom0 that is needed for the DomU itself. But this reduces the memory of the Dom0 and raises an exception, which is in that situation a false alarm and therefore needs to be disabled.

    # modelling the enclosure (hypervisor) 
      "/system/enclosure/children" = push("lxvm0250"); 
      "/system/enclosure/children" = push("lxvm0251"); 
      "/system/enclosure/children" = push("lxvm0252"); 
      "/system/enclosure/children" = push("lxvm0253"); 
    

    This is the point where the relationship between Dom0 and DomUs is defined. This machine has four virtual machines. All the virtual machines must have a working profile in CDB.

    # hardware information 
      "/hardware/serialnumber" = "ch516-548-175"; 
      "/hardware/rack/name" = "rb38"; 
      "/hardware/cards/nic/0/hwid" = "[obfuscated for security reason]"; 
      include serial_map_lxc1rb46; 
      include diskinfo_lxxen003; 
    

    Standard configuration. Nothing special.

    # Xen guests 
      define variable guest0 = value("/system/enclosure/children/0"); 
      "/software/components/xen/domains/0/options/name" = guest0; 
      "/software/components/xen/domains/0/options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest0 + ".ks"; 
      "/software/components/xen/domains/0/options/vif/0" = "mac=" + value("//profile_"+ guest0 + "/hardware/cards/nic/0/hwid"); 
      "/software/components/xen/domains/0/options/memory" = value("//profile_"+ guest0 + "/hardware/ram/0/size"); 
      "/software/components/xen/domains/0/install_options/name" = guest0; 
      "/software/components/xen/domains/0/install_options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest0 + ".ks"; 
      "/software/components/xen/domains/0/install_options/vif/0" = "mac=" + value("//profile_"+ guest0 + "/hardware/cards/nic/0/hwid"); 
      "/software/components/xen/domains/0/install_options/memory" = value("//profile_"+ guest0 + "/hardware/ram/0/size"); 
    
      define variable guest1 = value("/system/enclosure/children/1"); 
      "/software/components/xen/domains/1/options/name" = guest1; 
      "/software/components/xen/domains/1/options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest1 + ".ks"; 
      "/software/components/xen/domains/1/options/vif/0" = "mac=" + value("//profile_"+ guest1 + "/hardware/cards/nic/0/hwid"); 
      "/software/components/xen/domains/1/options/memory" = value("//profile_"+ guest1 + "/hardware/ram/0/size"); 
      "/software/components/xen/domains/1/install_options/name" = guest1; 
      "/software/components/xen/domains/1/install_options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest1 + ".ks"; 
      "/software/components/xen/domains/1/install_options/vif/0" = "mac=" + value("//profile_"+ guest1 + "/hardware/cards/nic/0/hwid"); 
      "/software/components/xen/domains/1/install_options/memory" = value("//profile_"+ guest1 + "/hardware/ram/0/size"); 
    
      define variable guest2 = value("/system/enclosure/children/2"); 
      "/software/components/xen/domains/2/options/name" = guest2; 
      "/software/components/xen/domains/2/options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest2 + ".ks"; 
      "/software/components/xen/domains/2/options/vif/0" = "mac=" + value("//profile_"+ guest2 + "/hardware/cards/nic/0/hwid"); 
      "/software/components/xen/domains/2/options/memory" = value("//profile_"+ guest2 + "/hardware/ram/0/size"); 
      "/software/components/xen/domains/2/install_options/name" = guest2; 
      "/software/components/xen/domains/2/install_options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest2 + ".ks"; 
      "/software/components/xen/domains/2/install_options/vif/0" = "mac=" + value("//profile_"+ guest2 + "/hardware/cards/nic/0/hwid"); 
      "/software/components/xen/domains/2/install_options/memory" = value("//profile_"+ guest2 + "/hardware/ram/0/size"); 
    
      define variable guest3 = value("/system/enclosure/children/3"); 
      "/software/components/xen/domains/3/options/name" = guest3; 
      "/software/components/xen/domains/3/options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest3 + ".ks"; 
      "/software/components/xen/domains/3/options/vif/0" = "mac=" + value("//profile_"+ guest3 + "/hardware/cards/nic/0/hwid"); 
      "/software/components/xen/domains/3/options/memory" = value("//profile_"+ guest3 + "/hardware/ram/0/size"); 
      "/software/components/xen/domains/3/install_options/name" = guest3; 
      "/software/components/xen/domains/3/install_options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest3 + ".ks"; 
      "/software/components/xen/domains/3/install_options/vif/0" = "mac=" + value("//profile_"+ guest3 + "/hardware/cards/nic/0/hwid"); 
      "/software/components/xen/domains/3/install_options/memory" = value("//profile_"+ guest3 + "/hardware/ram/0/size"); 
    
      #define variable guest4 = value("/system/enclosure/children/4"); 
      #"/software/components/xen/domains/4/options/name" = guest4; 
      #"/software/components/xen/domains/4/options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest4 + ".ks"; 
      #"/software/components/xen/domains/4/options/vif/0" = "mac=" + value("//profile_"+ guest4 + "/hardware/cards/nic/0/hwid"); 
      #"/software/components/xen/domains/4/options/memory" = value("//profile_"+ guest4 + "/hardware/ram/0/size"); 
      #"/software/components/xen/domains/4/install_options/name" = guest4; 
      #"/software/components/xen/domains/4/install_options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest4 + ".ks"; 
      #"/software/components/xen/domains/4/install_options/vif/0" = "mac=" + value("//profile_"+ guest4 + "/hardware/cards/nic/0/hwid"); 
      #"/software/components/xen/domains/4/install_options/memory" = value("//profile_"+ guest4 + "/hardware/ram/0/size"); 
    

    Each block in the previous template snippet customizes certain data fields for a single virtual machine (DomU). Most of the values have already been pre-defined in the template pro_service_lxxen, which belongs to the cluster template, and need now to be configured independently.

    A variable for the first guest (DomU) is created. The name of the first child is assigned to it. In this particular case the first children was assigned with this instruction earlier in the template to the current machine:

    "/system/enclosure/children" = push("lxvm0250");
    

    The variable guest0 now contains the machine name lxvm0250, which is apparently a DomU.

    define variable guest0 = value("/system/enclosure/children/0");
    "/software/components/xen/domains/0/options/name" = guest0; 
    

    Some kernel parameters are defined as well as the location of the kickstart file for the virtual machine. The kickstart file gets created during the installation process and uploaded to the denoted location.

    "/software/components/xen/domains/0/options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest0 + ".ks"; 
    

    The values for memory and mac address of the virtual machine are referenced from the virtual machines profile. The advantage in doing this, is having information about the virtual machine only stored in its own profile. No virtual machine related data should be stored in a profile which is not the virtual machine ones.

    "/software/components/xen/domains/0/options/vif/0" = "mac=" + value("//profile_"+ guest0 + "/hardware/cards/nic/0/hwid"); 
    "/software/components/xen/domains/0/options/memory" = value("//profile_"+ guest0 + "/hardware/ram/0/size"); 
    

    The following instructions just repeat the previous assignments for the installation configuration of the virtual machine.

    "/software/components/xen/domains/0/install_options/name" = guest0; 
    "/software/components/xen/domains/0/install_options/extra" = "serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/" + guest0 + ".ks"; 
    "/software/components/xen/domains/0/install_options/vif/0" = "mac=" + value("//profile_"+ guest0 + "/hardware/cards/nic/0/hwid"); 
    "/software/components/xen/domains/0/install_options/memory" = value("//profile_"+ guest0 + "/hardware/ram/0/size");
    

    The previous definitions together with the default ones made in the pro_service_lxxen template result in a complete definition of the Xen configuration for the ncm-xen component, which can now create a full Xen configuration file at installation time.

    # history 
      "/system/oldnames/0/" = nlist("name", "lxb6603", "date", "11.09.07" ); 
    # End-of-File 
    

    In the end some information to keep track of the history.

    This completes the configuration step for the Xen-based virtual machine environment. It can now be proceeded with the installation.

    5. Installation

    To start the installation of a Xen-based virtual machine environment it is indispensable that the configuration has been completed. If not, start over with chapter configuration.

    In addition to that the the person which installs the system should have proper access rights:

    • to initiate the installation by running !PrepareInstall script from lxadm
    • to SINDES to store machine certificates
    • to the AFS directory where the kickstart file gets stored
    • to login as root on the future Dom0 and DomUs
    • to login as root on the current Dom0 (and DomUs) to reboot them for installation
    • to use the connect2console.sh script to connect to console sessions of Dom0 and DomUs
    • to enter the Computing Center building 513 (both levels) and check the installation process directly at the machine
    • to use sms to change state of a quattor managed node before and after installation

    TIP Reminder: Machines are already in maintenance state?

    Ready, set, go!

    5.1. PrepareInstall

    PrepareInstall script performs all necessary steps to install an Elfms-managed machine in the CERN Computing Center:

    • generate KickStart files
    • use SINDES to generate GPG keys, and open the necessary time windows
    • register the KickStart files in AIMS, and enable PXE

    These steps require certain privileges:

    a) Ask Linux 3rd Level Support to be allowed to upload Kickstart files in AIMS.
    b) Ask SINDES Support to add your AFS account to sindes-server:/home/sindes/.klogin to be able to run the SinDes tools.

    Please contact project-elfms@cernNOSPAMPLEASE.ch to obtain them.

    Now run PrepareInstall on any lxadm machine for the Dom0 machine (here lxxen003) and its children, the virtual machines (here lxvm025[0,1,2,3]):

    [lxadm03] /afs/cern.ch/user/j/jmichael > PrepareInstall --verbose lxxen003 lxvm0250 lxvm0251 lxvm0252 lxvm0253
    	...
    

    This time PrepareInstall is used with the "--verbose" options which gives you a lots of useful information during the processing of the script and also formats your kickstart file nicely for easier reading. It also write a time-stamp in the kickstart file which lets you crosscheck wether this file was generated from your PrepareInstall run or if it is an old one.

    And by the way. Now you have 24 hours to install your machines. This is the time slot given by the SINDES server to machines to request their generated certificates. If time runs out and you try to install or maybe reinstall the machine beyond this time window, the installation will fail. The installation will also abort if you try to install the machine within the time slot a second time, because SINDES server denies the machine to download its certificate again.

    It is always a good idea to run PrepareInstall once again.

    5.1.1. Crosschecks for successful run of PrepareInstall

    a) If there is no error you will end up with the kickstart files created here:

    LinuxSoft: http://linuxsoft.cern.ch/kickstart/fio-is/<machine name>.ks
    AFS: /afs/cern.ch/project/linux/redhat/kickstart/cfg/fio-is/

    Rename "fio-is" in both paths according to your aims group area.

    b) You can also check for SINDES if the the machine to be installed can request their certificate.

    [lxadm03] /afs/cern.ch/user/j/jmichael > ssh sindes@sindes-server
    SINDESsh  > acl -print
    ------------------------------------------------
    |       hostname        TTL       Request Right|
    ------------------------------------------------
    ...
    |       lxxen003        23:59               YES|
    ... 
    |       lxvm0250        23:59               YES|
    |       lxvm0251        23:59               YES|
    |       lxvm0252        23:59               YES|
    |       lxvm0253        23:59               YES|
    ...
    

    c) You can check if PXE is enabled or not with:

    [lxadm03] /afs/cern.ch/user/j/jmichael > /afs/cern.ch/project/linux/redhat/kickstart/bin/aims show 
    

    It is possible to cancel reinstallation after doing PrepareInstall, but before rebooting the node with:

    [lxadm03] /afs/cern.ch/user/j/jmichael > /afs/cern.ch/project/linux/redhat/kickstart/bin/aims pxeoff 
    

    5.2. Booting/Rebooting Dom0

    With booting/rebooting the Dom0 machine the installation of the Dom0, not the DomUs, starts automatically.
    For booting a machine which has not yet powered on, one most go to the CC, find the machine and press the power button smile

    The same procedure applies for machines you can not log in as root or which are not reachable. frown

    If none of previous cases apply to your situation, you are lucky and can reboot the machine by issuing:

    ssh root@ "shutdown -r now"
    

    Depending on the machines performance the basic installation starts when the machine boots and can take anything from 30 minutes to one hour. If something goes wrong and the installation of the machine fails, an email will be send to the user who has executed PrepareInstall unless otherwise noted.

    5.3. Check for successful installation of Dom0

    a) Login to the newly installed machine as root:

    You should be able to login as root with your kerberos ticket. If not, destroy your kerberos ticket, recreate it, and try again. In case you are still unable to login, either you are not in the authorized user list in CDB or the installation is not finished yet.

    b) Check that the correct kernel is running:

    If you are logged in as root to the Dom0 machine, you first check if the appropriate kernel is running:

    [root@lxbuild048 ~]# uname -ar
    Linux lxbuild048.cern.ch 2.6.18-1.2835.slc4xen #1 SMP Thu Nov 30 11:17:11 CET 2006 x86_64 x86_64 x86_64 GNU/Linux
    

    The kernel version can be differ but must include the token slc4xen.

    c) Check the ks-post-anaconda.log for errors:

    The home directory of the root user contains a log file called ks-post-anaconda.log. This file should not contain any message indicating that the installation was aborted due to some error. Most likely you will find those candidates by looking for the string "error" or "fail". Warnings can be ignored without hesitations.

    d) Check for creation of Xen configuration files:

    The directory /etc/xen contains the Xen configuration of the Dom0 and the DomUs. This directory should look similar to this terminal snippet:

    [root@lxbuild048 ~]# cd /etc/xen/
    [root@lxbuild048 xen]# ll
    total 2852
    drwxr-xr-x	2 root root	   4096 Nov 30	2006 auto
    -rw-r--r--	1 root root 1416549 May 16	2007 initrd.img
    -rwxr-xr-x	1 root root		164 Nov 30	2006 qemu-ifup
    drwxr-xr-x	2 root root	   4096 Sep 27 18:19 scripts
    -rw-r--r--	1 root root		788 Sep 27 18:26 vm006
    -rw-r--r--	1 root root		756 Sep 27 18:26 vm006.old
    -rw-r--r--	1 root root		817 Sep 27 18:26 vm006.start
    -rw-r--r--	1 root root		788 Sep 27 18:26 vm007
    -rw-r--r--	1 root root		756 Sep 27 18:26 vm007.old
    -rw-r--r--	1 root root		817 Sep 27 18:26 vm007.start
    -rw-r--r--	1 root root		788 Sep 27 18:26 vm008
    -rw-r--r--	1 root root		756 Sep 27 18:26 vm008.old
    -rw-r--r--	1 root root		817 Sep 27 18:26 vm008.start
    -rw-r--r--	1 root root		788 Sep 27 18:26 vm009
    -rw-r--r--	1 root root		756 Sep 27 18:26 vm009.old
    -rw-r--r--	1 root root		817 Sep 27 18:26 vm009.start
    -rw-r--r--	1 root root		788 Sep 27 18:26 vm010
    -rw-r--r--	1 root root		756 Sep 27 18:26 vm010.old
    -rw-r--r--	1 root root		817 Sep 27 18:26 vm010.start
    -rw-r--r--	1 root root 1212985 May 16	2007 vmlinuz
    -rw-r--r--	1 root root	   4880 Nov 30	2006 xend-config.sxp
    -rw-r--r--	1 root root	   1256 Nov 30	2006 xend-pci-permissive.sxp
    -rw-r--r--	1 root root	   4129 Nov 30	2006 xend-pci-quirks.sxp
    ...
    

    This Dom0 has five configured virtual machines named from vm006 to vm010. The configuration files without a suffix are to boot a installed virtual machine. Such a file looks like this:

    #  -*- mode: python; -*-
    #============================================================================
    # Python configuration setup for 'xm create'.
    # This script sets the parameters used when a domain is created using 'xm create'.
    # You use a separate script for each domain you want to create, or
    # you can set the parameters for the domain on the xm command line.
    #
    # Generated by ncm-xen at Thu Sep 27 18:26:30 2007
    #============================================================================
    bootloader = '/usr/bin/pygrub'
    disk = ['phy:/dev/xen_vg/vol5,xvda,w']
    extra = 'serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/vm010.ks'
    ip = 'dhcp'
    memory = '512'
    name = 'vm010'
    on_crash = 'destroy'
    on_poweroff = 'destroy'
    on_reboot = 'restart'
    vif = ['mac=00:16:3E:01:00:10']
    

    The configuration files with the suffix ".start" are intended to install the virtual machine and look usually similar to this one:

    #  -*- mode: python; -*-
    #============================================================================
    # Python configuration setup for 'xm create'.
    # This script sets the parameters used when a domain is created using 'xm create'.
    # You use a separate script for each domain you want to create, or
    # you can set the parameters for the domain on the xm command line.
    #
    # Generated by ncm-xen at Thu Sep 27 18:26:35 2007
    #============================================================================
    disk = ['phy:/dev/xen_vg/vol5,xvda,w']
    extra = 'serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/vm010.ks'
    ip = 'dhcp'
    kernel = '/etc/xen/vmlinuz'
    memory = '512'
    name = 'vm010'
    on_crash = 'destroy'
    on_poweroff = 'destroy'
    on_reboot = 'destroy'
    ramdisk = '/etc/xen/initrd.img'
    vif = ['mac=00:16:3E:01:00:10']
    

    Please note that in comparison to the configuration file without the ".start" suffix the one with it contains a ramdisk and kernel but no bootloader option.
    These file were created by the ncm-xen component.

    e) Check for creation of logical volumes (lv) for DomUs:

    During the installation process, right before ncm-xen got executed, another component ncm-filesystems prepared the logical volumes for the guests on the host system:

    [root@lxbuild048 xen]# lvdisplay  
      --- Logical volume ---
      LV Name				 /dev/xen_vg/vol1
      VG Name				 xen_vg
      LV UUID				 syPW41-4k76-y7BL-B8km-462Q-TZXF-LrcYZ3
      LV Write Access		 read/write
      LV Status				 available
      # open				 1
      LV Size				 20.00 GB
      Current LE			 5120
      Segments				 1
      Allocation			 inherit
      Read ahead sectors	 0
      Block device			 253:0
    ... [repeats five times for the other four virtual machines]
    

    f) Check if xend, the Xen daemon, is running:

    Xend is the Xen controller daemon, meaning it handles creating new domains, destroying extant domains, migration and many other domain management tasks. A large part of its activity is based on running an HTTP server. The default port of the HTTP socket is 8000, which can be configured. Various requests for controlling the domains are handled by sending HTTP requests for domain creation, domain shutdown, domain save and restore, live migration and more.

    [root@lxbuild048 xen]# /etc/init.d/xend
    xend		xendomains	
    [root@lxbuild048 xen]# /etc/init.d/xend status
    xend is running
    

    Now that we know that xend is running we can try to look if the Dom0 is running by accessing the xend with xm. The xm program is the main interface for managing Xen guest domains. The program can be used to create, pause, and shutdown domains. It can also be used to list current domains, enable or pin VCPUs, and attach or detach virtual block devices.

    [root@lxbuild048 xen]# xm list
    Name                        ID Mem(MiB) VCPUs State     Time(s)
    Domain-0                    0       472     2 r-----    764550.2
    

    After beeing walked through the six steps everything looks ok and very promising so far. It can be assumed that the Dom0 has been installed properly and the machine is ready to deploy DomUs on it.

    If there were any problems during the installation or the post-installation checks, please consult the XEN How To / FAQ and Tools chapter.

    5.4. Installing virtual machines (DomU)

    The Dom0 is running, there are no problems with the xend daemon, access to Dom0 is granted and you are still in within 24 hours since you executed PrepareInstall for your machines? Well done, this gives a good basis to install the DomUs.

    Almost everything for the installation of the DomUs was already prepared during the installation of the Dom0:

    • logical volumes as disk back-end for each DomU were created by ncm-filesystems
    • Xen configuration files for installatioon and normal boot were created by ncm-xen
    • an appropriate kernel and ramdisk image for installation were automatically downloaded by ncm-xen
    • xend was started during last boot of Dom0

    To start the installation log in to the Dom0 machine and change your directory to /etc/xen:

    [lxadm02] /afs/cern.ch/user/j/jmichael > ssh root@lxbuild048
    ... [welcome message] ...
    [root@lxbuild048 ~]# cd /etc/xen
    [root@lxbuild048 xen]# xm create vm006.start
    Using config file "vm006.start".
    Started domain vm006
    

    In this case virtual machine vm006 with configuration file "vm006.start" was created. Some informative messages were provided and the prompt was handed back to the user.

    To check if the installation is running, xm with subcommand "list" can be executed:

    [root@lxbuild048 xen]# xm list
    Name                        ID Mem(MiB) VCPUs State	    Time(s)
    Domain-0                    0	    472	    2 r-----    772686.5
    vm006                       8        512	1 r-----       171.9
    

    This terminal snippet shows us that two machines are running. The topmost machine is Dom0 with 2 virtual CPUs and 472 MiB assigned. At the time the xm list command was issued, Domain-0 has consumed 772686.5 CPU seconds and was in state "r" (running).
    Nearly the same picture shows vm006. This time it was in state "r" but it can also happen that you will find it in state b - blocked. The domain is blocked, and not running or runnable. This can be caused because the domain is waiting on IO (a traditional wait state) or has gone to sleep because there was nothing else for it to do. If the virtual machine is in state "r" or "b" everything is alright. More about the different states a virtual machine or the Dom0 can be in, can be found in the man page of xm.

    A way to see what is going on during the installation of a DomU is to attach a console. From the manpage of xm:

    "This [the console subcommand] uses the back end xenconsole service which currently only works for para-virtual domains. The attached console will perform much like a standard serial console, so running curses based interfaces over the console is not advised. Vi tends to get very odd when using it over this interface."

    This can be done in two ways.

    a) Attaching the console when creating the virtual machine

    [root@lxbuild048 xen]# xm create -c vm006.start
    

    To the initial command for creating a virtual machine the parameter -c (console) is added.

    b) Attaching the console while the virtual machine is running

    [root@lxbuild048 log]# xm console vm006
    

    or

    [root@lxbuild048 log]# xm console 8
    

    The first example uses the subcommand console and the DomU name. The second one is using the DomU id, which can be picked up from the "xm list" command.

    Once the console gets attached to the specified domain the terminal should show something similar to that in the beginning:

    [root@lxbuild048 log]# xm create -c vm006.start
    Using config file "/etc/xen/vm006.start".
    Started domain vm006
    xen_start_info @ffffffff80d4a000
    shared @m00001b3000 @ffffffff80107000 = @ffffffffff5fd000 [0x802]
    xen_start_info: @ffffffff80d4a000
     cr3 0000000000101000 pml4p ffffffff80101ff8
     pml4e 0000000000103067 (real 000000005d8f1067) pgdp ffffff8000103ff0
     pgde 0000000000105067 (real 000000005db70067) pmdp ffffff8000105030
     pmde 0000000000d56067 (real 0000000027c37067) ptep ffffff8000d56a50
     pte 0010000000d4a027 (real 001000001f22d027)
    xen_shared_info: @ffffffffff5fd000
     cr3 0000000000101000 pml4p ffffffff80101ff8
     pml4e 0000000000103067 (real 000000005d8f1067) pgdp ffffff8000103ff8
     pgde 0000000000000000 is none
    PAGE_OFFSET+1.2: @ffffff8000001000
     cr3 0000000000101000 pml4p ffffffff80101ff8
     pml4e 0000000000103067 (real 000000005d8f1067) pgdp ffffff8000103000
     pgde 0000000000d58067 (real 000000005764b067) pmdp ffffff8000d58000
     pmde 0000000000d59067 (real 000000001cbd1067) ptep ffffff8000d59008
     pte 0000000000001167 (real 000000003c861167)
    Bootdata ok (command line is  ip	  = dhcp:1.2.3.4::::eth0:off serial cmdline ks=http://linuxsoft.cern.ch/kickstart/fio-is/vm006.ks)
    Linux version 2.6.9-55.EL.CERNxenU (root@lxcert-amd64) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-8)) #1 SMP Wed May 16 12:06:10 CEST 2007
    kernel direct mapping tables up to 20800000 @ d58000-e5d000
    

    These lines mark the beginning of the installation procedure. Later you would see the console based installer of Scientific Linux CERN - Anaconda.

    The only way to leave the console view and get back to the terminal prompt is pressing the key combination:

    "CTRL + ]"
    

    To install all the virtual machines on the host, create the DomUs as described. But be aware that the installation task is very disk intensive, which significantly slows down the whole machine. It is recommended to start the installation of the virtual machines one after another. Do not launch installation for multiple machines at the same time.

    5.5. Boot the virtual machines (DomU)

    When the installation of a virtual machine is finished, it reboots itself to do some post-installation configuration, just like a physical machine or the Dom0 would do. But there is a catch in it for the virtual machine. The installation configuration file (the one with .start suffix) says:

    on_reboot = 'destroy'
    

    This has been done, because if it would allowed to reboot itself it would reboot with the same configuration file and would start the installation procedure again. Instead of that it is wanted that the machine boots with the normal configuration file (without .start suffix). But this can not be somehow automated with Xen. This must be done by hand.

    The only way, to notice if the installation of a virtual machine has been completed, is to check with

    xm list
    

    if the machine is still alive or already shut down. If the machine does not show up as running, 30 minutes after it was created, it has most likely finished the installation. Now the DomU can be created again using the normal boot config file (without the .start suffix):

    [root@lxbuild048 xen]# xm list
    Name                        ID Mem(MiB) VCPUs State	Time(s)
    Domain-0                     0      472     2 r----- 773767.4
    [root@lxbuild048 xen]# xm create vm007
    Using config file "vm007".
    Going to boot Scientific Linux CERN SLC (2.6.9-55.0.6.EL.CERN1xenU)
      kernel: /vmlinuz-2.6.9-55.0.6.EL.CERN1xenU
      initrd: /initrd-2.6.9-55.0.6.EL.CERN1xenU.img
    Started domain vm007
    [root@lxbuild048 xen]# xm list
    Name                        ID Mem(MiB) VCPUs State	Time(s)
    Domain-0                    0       472     2 r----- 773774.1
    vm007                       11      512     1 r-----      9.1
    

    In this case virtual machine vm007 was created using kernel 2.6.9-55.0.6.EL.CERN1xenU (note the xenU token).

    5.6. Connecting to the virtual machines (DomU)

    Again the machine will need some time to become completely up and running. As usually some post-installation tasks have to be completed. After another 30 minutes the machine can be accessed via ssh or directly by attaching a console:

    [root@lxbuild048 xen]# xm console vm007 
    
    Scientific Linux CERN SLC release 4.5 (Beryllium)
    
    vm007.cern.ch login:
    

    Another way of getting access to the virtual machine is using the Console Service of CC.
    The "connect2console.sh" script is installed on lxplus and lxadm and can be used as follows:

    [lxplus241] /afs/cern.ch/user/j/jmichael > connect2console.sh vm007
    Connecting to the headnode: vm007.headnode.cern.ch ...
    *******************************************************************************
    * http://cern.ch/ComputingRules : Govern the use of CERN computing facilities *
    *******************************************************************************
    Could not chdir to home directory /afs/cern.ch/user/j/jmichael: No such file or directory
    /usr/bin/X11/xauth:	 error in locking authority file /afs/cern.ch/user/j/jmichael/.Xauthority
    [vm007: Attached readwrite on localhost]
    
    Scientific Linux CERN SLC release 4.5 (Beryllium)
    
    vm007.cern.ch login:
    

    Usually it happens that you do not get the login prompt. In this case just hit the key combination "CRTL-D".
    To get disconnected from the console screen a special escape sequence is necessary:

    CTRL + E + C + .
    

    Other escape sequences for the console server client program can be found here External site.

    6. XEN How To / FAQ and Tools

    Most of the question regarding Xen are already answered in the Xen FAQ External site on Xensource.com. In the rare case you will not find an answer to your question over there and the following FAQs will not help as well, the last chance is to ask the community on the mailing lists. Nabble is a good point to start searching the Xen mailing lists.

    6.1. How to get/request a virtual machine or a virtual machine environment?

    Answer:

    It is the same as to request a traditional machine. Go to http://oraweb.cern.ch/pls/cdbsql/web.resource_allocation and fill out the form. There is a radio button called "_My service can run on a virtual machine_", which need to be checked.
    The responsible service manager decides, if your request is suited for a virtual machine or not. If you have questions on this you can ask Véronique Lefèbure. She takes care of all those "virtualization requests".

    If you need a whole virtualization environment you can just drop a line in one of the comment fields.

    6.2. What to do if a virtual machine has no contact?

    Answer:

    This case is dealt with, like with one for a conventional machine, in the first place. Some additional ways to check what is going on with the virtual machine are the following:

    a) The monitoring tools presented in section How to reboot / shutdown a Dom0 with running DomU? can help to determine whether the DomU is stuck or just under heavy load.

    b) The console client connect2console.sh, available on any lxplus or lxadm machine, provides access to the serial console of the virtual machine as well as the xm command on the corresponding Dom0.

    c) On the corresponding Dom0 these things can be verified:

    • Checking running background processes (xend)
    • Checking open sockets on port 8801, 8892 (default)
    • Checking the xend.log/messages (Xen log file) - see Where are the log files?
    • Checking the health/availibility of the domain
    • Checking free disk space

    6.3. How to debug Xen?

    Answer:

    There is an official Xen Debugging wiki page External site which helps as a start. In addition to that, this mail External site to the xen-devel mailing list External site might be of help.

    6.4. How to move a virtual machine?

    Answer:

    Although there is a feature called "live migration", which lets you transfer a virtual machine from one host to another, this has never been tested with the CERN installations. Thus it is recommend not to use it.

    6.5. How to reinstall a virtual machine?

    Answer:

    These are the essential steps to reinstall a DomainU:

    a) Put the relevant machine in maintenance state to avoid any false alarms. On lxadm execute:

    [lxadm03] /afs/cern.ch/user/j/jmichael > sms set maintenance other 'reinstallation' 
    

    b) Shutdown the virtual machine:

    [lxadm02] /afs/cern.ch/user/j/jmichael > ssh root@ 'xm shutdown '
    

    c) Run PrepareInstall for the virtual machine from lxadm:

    [lxadm02] /afs/cern.ch/user/j/jmichael > PrepareInstall --verbose 
    

    d) Create the virtual machine with the installation configuration on the corresponding Domain0 and wait until installation is completed:

    [lxadm02] /afs/cern.ch/user/j/jmichael > ssh root@ 'xm create .start'
    

    e) Launch virtual machine on host with regular boot configuration and wait until ncm configures all necessary components:

    [lxadm02] /afs/cern.ch/user/j/jmichael > ssh root@ 'xm create '
    

    f) Put virtual machine back in production if necessary:

    [lxadm03] /afs/cern.ch/user/j/jmichael > sms set production none '' 
    

    6.6. How to reinstall a virtual machine environment?

    Answer:

    A reinstallation of a complete virtual machine environment can be done by following the instructions in chapter Installation.

    ALERT! Warning: Please keep in mind that all responsible owners of the deployed virtual machine have acknowledged this intervention. All the data stored on the hosting machine will get lost!

    6.7. Where are the log files?

    Answer:

    The xend daemon, which is responsible to execute all instructions a user sends by calling the xm command, write his log messages to:

    /var/log/xen/xend.log
    

    An alternative to view those provides xm itself:

    [root@lxbuild048 xen]# xm log
    [2007-09-27 18:35:36 xend 13657] INFO (SrvDaemon:283) Xend Daemon started
    [2007-09-27 18:35:36 xend 13657] INFO (SrvDaemon:287) Xend changeset: unavailable .
    [2007-09-27 18:35:36 xend.XendDomainInfo 13657] DEBUG (XendDomainInfo:212) XendDomainInfo.recreate({\047paused\047: 0, \047cpu_time\047: 356027421674L, \047ssidref\047: 0, \047handle\047: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \047shutdown_reason\047: 0, \047dying\047: 0, \047dom\047: 0, \047mem_kb\047: 1926256, \047maxmem_kb\047: 17179869180, \047max_vcpu_id\047: 1, \047crashed\047: 0, \047running\047: 1, \047shutdown\047: 0, \047online_vcpus\047: 2, \047blocked\047: 0})
    [2007-09-27 18:35:36 xend.XendDomainInfo 13657] INFO (XendDomainInfo:224) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
    ...
    

    6.8. How to reboot / shutdown a virtual machine?

    Answer:

    A virtual machine can be rebooted from within the machine with

    shutdown -r now
    

    or

    reboot
    

    Another way would be to login somehow to the Dom0 and execute

    xm reboot 
    

    Thus the reboot process will be initiated also within the <domain> and all shutdown scripts will be executed.

    Please be aware that parameter on_reboot in the configuration file for the regarding virtual machine has to be set to restart. And changes which were made to the virtual machines configuration file are not taken into account on reboot. The Changes only become effective if the virtual machine have been (re-)created with

    xm create 
    

    If the parameter "--wait" is added to "xm reboot <domain>" the xm command will wait until the reboot has finished. Another parameter, "--all", will reboot all existing DomUs.

    What's being said before also applies to the shutdown of virtual machines. The command "xm reboot" is just replaced by "xm shutdown".

    6.9. How to reboot / shutdown a Dom0 with running DomU?

    Answer:

    A Domain0 can be rebooted the same way as every other machine in the Computing Center, but there are fews things to keep in mind.
    Most likely virtual machines are running on the Dom0. Before the Dom0 can be rebooted, permission of the owners of the virtual machines must be obtained. After reboot it must be checked whether all DomUs which were running before the reboot, are running again after the reboot.

    If the virtual machines got not created by "xendomains" service, they must be created manually using

    xm create 
    
    .

    A manual creation of domains can be avoided by creating a symbolic link the boot configuration in the directory

    /etc/xen/auto
    

    and adding the service "xendomains" to the appropriate run level. Hence at booting time xendomains takes care of creating the virtual machines.

    6.10.How to monitor a virtual machine environment?

    Answer:

    First point of contact are the default monitoring tools which are usually used in day-to-day work, i.e. LemonWeb et cetera.
    In addition to that the Xen toolbox provides a set of tools which help to check the status of the Dom0 or the DomU:

    xentop

    Provides human readable statistics for a bunch of performance values for all domains:

    [root@lxbuild048 ~]# xentop
    

    Figure: Screenshot of Xentop on lxbuild048

    Xentop must be executed as root user on Dom0.

    xm list

    Provides a short listing of all running domains on the current host. It has to be executed as root on Dom0

    [root@lxbuild048 ~]# xm list
    

    Figure: Screenshot of "xm list" on lxbuild048

    By adding the option "--long" to this command the user gets an extensive report for every running domain about any figure which can be extracted from the system.

    xenmon.py

    Launches a curses application which provides in-detail performance data:

    [root@lxbuild048 ~]# xenmon.py 
    

    Figure: Screenshot of "xenmon.py" on lxbuild048 during execution

    Figure: Screenshot of "xenmon.py" on lxbuild048 after execution

    All these tools are installed on the Dom0 by default.

    6.11. How to change the amount of memory of virtual machine?

    Answer:

    The amount of memory a DomU can grab from its Dom0 is defined in its Xen configuration file. Here is short checklist of what needs to be done:

    a) Use cdbop on lxadm to check out and edit the necessary DomU profile.

    b) Change amount of memory to the required value (in this case it is set to 512 megabyte):

    "/hardware/ram" = list(nlist("size", 512)); 
    

    c) Commit the changed profile, which will cause the recompilation of the DomU and its according Dom0 profile.

    d) Run spma_ncm_wrapper.sh on the parent of DomU with configuration option "xen" to rewrite the corresponding DomU configuration file

    [root@lxbuild048 ~]# spma_ncm_wrapper.sh --co xen
    

    e) Reboot the virtual machine as described here How to reboot/shutdown a virtual machine? for the changes to take effect.

    6.12. Which things have to be considered when updating the Xen version?

    Answer:

    Before you start, please think of the following things:

    • Virtual machine users must be informed about the intervention.
    • Virtual machines must be stopped.
    • All machines which were running before the intervention need to be linked into /etc/xen/auto for automated boot after Dom0 reboot.

    a) Tests on a dedicated machines for 32-bit and 64-bit versions should have been done. One can take lxdev19 (32-bit) or lxbuild048 (64-bit) to check for problems with the new version.

    If the test results are satisfying and the version should be released for production, proceed with the following steps:

    b) If not already done by the Linux Support team, upload the related packages to the software repository (swrep).
    Here is a small tutorial on how to do it.

    c) Locate the affected CDB profiles.

    d) Set the new package version for Xen and its dependent packages. The dependencies should be already known through testing.

    e) If a new kernel is rolled out at the same time set also the new kernel version because otherwise it will not get activated and the machine boots the old kernel.

    # kernel version 
      "/system/kernel/version" = { 
        if (architecture == "i386"){ 
            "2.6.9-55.0.2.EL.CERNxenU"; 
        } else { 
            "2.6.9-55.0.6.EL.CERN1xenU"; 
        }; 
      };
    

    f) Update and commit the new templates with cdbop on lxadm.

    g) Run spma_wrapper.sh to update the machine(s). Use wassh on lxadm for convenience.

    e) Reboot the machines and make sure that also the virtual machine come up again.

    6.13. How to mount filesystem which were created by the DomU inside a logical volume?

    Answer:

    If a virtual machine can not be recreated or relaunched and must be reinstalled or is simply messed up due to misconfiguration, these instructions will help to access important data inside the logical volume of the virtual machine.
    Since the logical volumes are images of a whole disk (including the partition table, MBR etc.) and have been created during the installation of the guest, they can not be mounted simply.

    Assuming the following partition layout of the logical volume vol2:

    [root@vm007 ~]# fdisk /dev/xvda
    
    The number of cylinders for this disk is set to 2610.
    There is nothing wrong with that, but this is larger than 1024,
    and could in certain setups cause problems with:
    1) software that runs at boot time (e.g., old versions of LILO)
    2) booting and partitioning software from other OSs
       (e.g., DOS FDISK, OS/2 FDISK)
    
    Command (m for help): p
    
    Disk /dev/xvda: 21.4 GB, 21474836480 bytes
    255 heads, 63 sectors/track, 2610 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    
        Device Boot      Start         End      Blocks   Id  System
    /dev/xvda1   *           1          13      104391   83  Linux
    /dev/xvda2              14        1624    12940357+  83  Linux
    /dev/xvda3            1625        1885     2096482+  83  Linux
    /dev/xvda4            1886        2610     5823562+   5  Extended
    /dev/xvda5            1886        2146     2096451   82  Linux swap
    /dev/xvda6            2147        2348     1622533+  83  Linux
    /dev/xvda7            2349        2479     1052226   83  Linux
    /dev/xvda8            2480        2610     1052226   83  Linux
    

    and partition /dev/xvda3, which holds the /var directory, should be accessed. This can be accomplished by using lomount on the Dom0:

    [root@lxbuild048 ~]# ll /mnt/
    total 0
    [root@lxbuild048 ~]# lomount -verbose -t ext3 -partition 3 -diskimage /dev/mapper/xen_vg-vol2 /mnt
    mount -oloop,offset=13357854720 /dev/mapper/xen_vg-vol2 -t ext3 /mnt 
    [root@lxbuild048 ~]# ll /mnt/
    total 188
    drwxr-xr-x   2 root    root     4096 Jan 25 00:02 account
    drwxr-xr-x   7 root    root     4096 Sep 27 20:47 cache
    drwxr-xr-x   3 netdump netdump  4096 Sep 27 19:27 crash
    drwxr-xr-x   3 root    root     4096 Sep 27 19:27 db
    drwxr-xr-x   3 root    root     4096 Sep 27 19:27 empty
    drwxr-xr-x   2 root    root     4096 Jan 26  2006 heimdal
    drwxr-xr-x  22 root    root     4096 Jan 19 15:18 lib
    drwxr-xr-x   2 root    root     4096 Mar 14  2005 local
    drwxrwxr-x   7 root    lock     4096 Jan 25 21:01 lock
    drwxr-xr-x  12 root    root     4096 Jan 25 04:11 log
    drwx------   2 root    root    16384 Sep 27 19:22 lost+found
    lrwxrwxrwx   1 root    root       10 Sep 27 19:23 mail -> spool/mail
    drwxr-x---   4 root    named    4096 Sep 27 20:03 named
    drwxr-xr-x   3 root    root     4096 Sep 27 19:38 ncm
    drwxr-xr-x   2 root    root     4096 Mar 14  2005 nis
    drwxr-xr-x   2 root    root     4096 Mar 14  2005 opt
    drwxr-xr-x   2 root    root     4096 Mar 14  2005 preserve
    drwxr-xr-x  15 root    root     4096 Jan 25 21:59 run
    drwxr-xr-x   2 root    root     4096 Sep 27 19:52 sindes
    drwxr-xr-x   2 root    root     4096 Jun 11  2007 spma-cache
    drwxr-xr-x  14 root    root     4096 Jan 23 09:53 spool
    drwxrwxrwt   2 root    root     4096 Jan 25 21:01 tmp
    drwxr-xr-x   3 root    root     4096 Sep 27 19:27 yp
    

    This tool parses the partition table and allows to mount a chosen partition from a logical volume or any other disk image. It automatically computes the needed "offset" mount option.

    6.14. Error: Boot loader did not return any data!

    Answer:

    Sometimes it happens that the xend daemon dies for no apparent reason. The only way to resolve the situation is to restart the xend daemon. This can be done on the Dom0 without any impact on the running virtual machines.

    [root@lxxen001 ~]# xm shutdown lxvm0241
    Error: Boot loader did not return any data!
    Usage: xm shutdown  [-waRH]
    
    Shutdown a domain.
    [root@lxxen001 ~]# /etc/init.d/xend restart
    restart xend:                                              [  OK  ]
    [root@lxxen001 ~]# xm list
    Name                                      ID Mem(MiB) VCPUs State   Time(s)
    Domain-0                                   0     2485     2 r----- 234903.8
    lxvm0240                                  12      256     1 -b----  69146.3
    lxvm0242                                  16      256     1 -b----  68453.2
    lxvm0243                                  14      256     1 -b----  68709.6
    lxvm0244                                  17      512     1 -b----  68074.3
    [root@lxxen001 ~]# xm shutdown lxvm0241
    

    6.15. Useful Tools

    6.15.1. Bash completion for Xen commands

    7. Open challenges or things to do

    So far a lot of effort has been put into the integration of the Xen based virtualization environment and especially of virtual machines into tools and systems of the CERN Computing Center. And thanks to the help of many colleagues, many problems have been solved and a not immaterial level of integration have been achieved.

    Despite all this, there are yet some challenges to be faced. Take the following as a to do list with random order:

    7.1. Automated Installation using pypxeboot

    In the current situation some steps to get virtualization environment completely installed from scratch have to be done manually. This is more or less a result of having two Xen configuration files - one for installation and one for a normal boot. Would not it be better to have only one configuration file which can handle both states? Yes, for example with the following approach:

    Figure: Screenshot of using "pypxeboot" to boot virtual machines

    The aim is, that the xen guest behaves like a typical machine in the CC. Once the virtual machine is created, it requests its IP address and additional configuration information.
    The DHCP server points the Xen guest to ask the AIMS + TFTP server, where a file with a specific boot configuration and an installation image are provided. In the case that the boot configuration contains instructions to (re-)install the machine, the guest will download all necessary files and starts the (re-)installation. If the latter is not requested, it simply drops to the default boot loader pygrub and boots like normal.

    To get this process work a lot of things have to be done. Some of them are already prepared:

    AIMS (Jan Iven)

    The Automated Installation Management System (AIMS) is used for automatically installing Linux nodes at CERN. AIMS allows to perform installations minimizing human intervention. The system is based, and extends the Kickstart software from the RedHat distribution. It also provides prepared boot images for the installation process.

    AIMS is already prepared, which means that an installation for a virtual machine can be chosen and the correct actions, i.e. write the configuration file for the machine and prepare the boot image, will be taken automatically.

    PrepareInstall (Jan van Eldik)

    PrepareInstall is a script which prepares the installation process. It tells for example AIMS to configure a machine to boot into an installation the next time. So PrepareInstall needs also to be aware that it is now possible to install virtual machines. The necessary changes were made and tested by Jan van Eldik.

    Test environment has been set up

    On lxdev19 a test environment has been setup. All tools and patches which are needed for pypxeboot have been installed.

    pypxeboot loader External site

    The pypxeboot loader is a boot loader for Xen guests which simulates a PXE client. Actually it is just a small python script. The script is not yet ready for use in CERN environment and has to be modified. An unfinished version of a modified script can be found on lxdev19 on this path:

    /root/development/pypxeboot
    

    In short:

    • Continue with modifications on pypxeboot script to match CERN Computing Center requirements and test it.

    7.2. Testing of different subnets for Dom0 and DomU

    Due to the small subnet size (less than 255 machines each) in CERN it was decided to put Dom0 and DomU in different ones. Unfortunately it turned out that this solution would not work out of the box. Some modification of the DHCP server was necessary. This has been finished and put in production by David Gutierrez Rueda on October, 2nd in 2007 (reported by him in an email on 3rd October to Véronique Lefèbure). It needs to be tested if it works now.

    In short:

    • Test network structure while having Dom0s and DomUs in two different subnets.

    7.3. Bugs and Errors

    Currently there is something weird with all the Xen virtual machine environment installation. The 32-Bit as well as the 64-Bit machines (Dom0) throw every second nasty kernel messages. This is an example from lxdev19:

    printk: 967 messages suppressed.
    4gb seg fixup, process sendmail (pid 29778), cs:ip 73:0049a6a8
    BUG: sleeping function called from invalid context at kernel/rwsem.c:20
    in_atomic():0, irqs_disabled():1
     [] dump_trace+0x6b/0x1a8
     [] show_trace_log_lvl+0x10/0x20
     [] show_trace+0xa/0xc
     [] dump_stack+0x13/0x15
     [] __might_sleep+0x97/0xa1
     [] down_read+0x12/0x24
     [] pci_get_subsys+0x51/0xcf
     [] pci_get_device+0xb/0xe
     [] do_pci_parity_check+0x34/0x62 [edac_mc]
     [] edac_kernel_thread+0x7/0x3b [edac_mc]
     [] kthread+0x7b/0x9f
     [] kernel_thread_helper+0x5/0xb
    DWARF2 unwinder stuck at kernel_thread_helper+0x5/0xb
    
    Leftover inexact backtrace:
    
     =======================
    

    The issue was initially reported on June, 20th 2007 via email to Linux Support - Tracking ID CT436180, CT436180. And was addressed again on June 27th, July 10th and a last time on September 16th.

    This Bug is not critical because it has no impact on Dom0 and DomU functionality, but is also nothing lovely thus should not be there forever.

    In short:

    • Follow up existing bug with Linux Support and check if it still exists in newer kernel versions.

    7.4. Monitoring Enhancements

    The existing lemon sensor for the virtualization environment reports data about the CPU utilization of the whole machine including the DomUs. This will be visualized later in LemonWeb like that - link.
    Alex Iribarren suggested to have stacked bars instead of graphs for a better understanding of the graph.

    The lemon sensor itself could be extend to deliver data for memory and disk utilization on the whole machine. This would help to decide if it is possible to deploy another DomU on a specific host or if the machine is maybe played out and a virtual machine needs to be moved somewhere else.

    Currently the lemon sensor is on the library libxenstat which is provided by the Xen tools. But it is not sure if it will be supported in the future. That is why it might be a good idea to rewrite the sensor on the basis of another library called libvirt External site.
    This libvirt API is driven by some RedHat developers and therefore support is assured in the future. Another advantage of the libvirt API is, that it currently supports together with Xen OpenVZ, KVM and QEMU as virtualization platforms.

    In short:

    • Enhance lemon-sensor-vm with additional values for memory and disk space (if values can not be acquired from somewhere else).
    • Recode lemon-sensor-vm with libvirt to support more virtualization environments.
    • Check if reformatting graphs with stacks for better visualization is possible without too much effort.

    7.5. Integration with Service Level Status (SLS)

    The Service Level Status (SLS) is a web-based tool that dynamically shows availability, basic information and/or statistics about IT services, as well as dependencies between them.

    The aim is to investigate if virtualization @ FIO as it is described in this TWiki article can be defined as a service and how it can be integrated into SLS. If this is accomplished and it is finally decided to integrate virtualization as a service into SLS it has to be implemented.

    In short:

    • Analyze if virtualization provided by FIO can be defined as a service in terms of SLS.
    • If the outcome of the previous investigation is positive, define the service and implement it into SLS.

    7.6. Quattorized image from OSFarm

    Andreas Unterkircher and his colleagues (see section Who uses the Virtualization Service of FIO don't use QUATTOR based installations for their virtual machines.
    Instead they use generated images. One drawback of this method is, that those machines are not covered by lemon monitoring. To make this possible it is intended to integrate those installation in the existing ELFms world.

    Therefore the images need to be quattorized. Havard Bjerke from OpenLab agreed on making such an image. The image will be available through OSFarm (Link currently dead :-() as soon as it is ready.

    In short:

    • Create quattorized images ready and simple to use for GD people to cover proper monitoring of these machines (detection of unused machines).

    7.7. CDB Configuration

    The way how Dom0s and DomUs are configured in CDB is not optimal. Some values which are needed for the configuration of a DomU can be found in the Dom0 profile too.
    The goal should be, to have all data, that is necessary to define a virtual machine, in the DomUs profile. This is what can be understood by the item "harmonize CDB configuration".

    In short:

    • Harmonize CDB configuration.
    • Validation functions in CDB or a tool for adding DomUs to Dom0s profile.

    7.8. Improvements on CC system and management tools regarding a virtualized environment

    One step to install a virtual machine environment, is to run PrepareInstall from any lxadm machine. Currently the script has to be executed for the Dom0 and all DomUs. For sure all host names can be given as argument to PrepareInstall, but this is not really handy.

    A better solution would be to have some kind of cascading option.
    Running PrepareInstall with the option "--cascade" like that:

    [lxadm03] /afs/cern.ch/user/j/jmichael > PrepareInstall --cascade lxxen003
    

    means that PrepareInstall does a lookup for machine lxxen003 and its related children in CDBSQL. In this case it will find the four virtual machines lxvm0250, lxvm0251, lxvm0252 and lxvm0253 and will perform all necessary tasks to prepare the installation for those machines as well as for lxxen003.

    In short:

    • Extend PrepareInstall with an option called "--cascade".

    7.9. Virtualize machines of the lxdev-cluster

    In July 2007 an investigation was made to determine which machine of the lxdev cluster can be virtualized. The results can be found on this page. The excel sheet attached to the mentioned TWiki article gives a detailed listing about the status of all machines and which ones can be moved to virtual machines.

    All responsible persons of the machines which are ready to move, have already been introduced to these plans.

    In short:

    • Virtualize specified machine of the lxdev cluster in correspondence with their responsible owners and give back unused machines.

    7.10. Xen Test cluster

    At present, both test environments, lxdev19 and lxbuild048, are not part of a dedicated Xen test cluster. For proper test conditions and not to accidently harm any other cluster configuration, it is essential to have at least an own sub cluster for these machines.

    With the introduction of namespace support in CDB different areas for production, pre-production and test can be used. Thus another solution would be to move the profiles of the test machines out of production into test stages.

    In short:

    • Move test machines to a test (sub-)cluster or move their profile in a test or pre-prod stage.

    7.11. Management interface for virtualized environments

    To get an overview about all installed virtual machine environments and its virtual machines from a monitoring and management perspective one may execute many scripts, ask several databases and flicks through a bunch of webpages and applications.

    Would not it be more comfortable to have an integrated solution which can be plugged into the existing infrastructure?
    There are not much solutions in the open source universe which would meet the needs of CERN. Hence this is only a small list of potential candidates:

    a) OpenQRM with XEN plugin External site

    OpenQRM is yet another provisioning and management system for Windows, Linux, Unix and virtual machine environments. Qlusters, the company behind OpenQRM claims itself to automate enterprise data centers and keep them running.

    To get a first impression of the OpenQRM interface and the powerful Xen plugin one can watch this demonstration External site on youtube.com.

    b) Virtual machine manager (virt-manager) External site (very promising developments)

    Virtual Machine Manager is product based on the libvirt API and is developed by RedHat. It is a pure GNOME application and brings all the monitoring features of the Xen tool "xm top" to a nice GUI. Beyond that it covers lots of common management tasks like configuration, creation and adjustment of a domains resource allocation.

    Currently virt-manager supports only management of one virtual machine environment and must be installed on the Dom0, which limits the use in the Computing Center. But the development roadmap External site looks very promising that in the near future virt-manager gets a web-based interface and will manage multiple hosts. Furthermore it seems that RedHat will add virt-manager to their toolbox for virtualization for their next enterprise server version.

    c) Enomalism External site

    Enomalism is a management console for the Xen Hypervisor. It is a web application developed using the python programming language and the TurboGears MVC framework. The application is released under an LGPL license by Enomaly, a consultancy based in Toronto, Canada.

    Half a year ago the project seemed to be nearly dead. But recently they announced a new version to be out in Q1 of 2008. For further reading check out the article in Wikipedia External site (Be aware that the article was written by the founder of the company).

    In short:

    • Evaluate the proposed and further solutions who can help to make management and monitoring of virtual machine environment easier, less complex and more efficient.

    7.12. Upgrade to newer versions of Xen (3.1/3.2)

    Currently Xen version 3.0.3 is used at CERN - see section Virtualization Environment at CERN (Facts and Figures), but the latest official release is 3.1. This brings some major improvements and new features along.

    Besides support for 32-bit DomUs on 64-bit Dom0s and major enhancements for hardware virtualized hosts, it is more important that a new xml-based configurations file structure was introduced, which becomes mandatory when using version 3.1 or above.
    This means that the ncm-xen quattor component, which is used to transform a CDB profile configuration into a Xen configuration, must be adapted to suit the new requirements.

    In short:

    • Upgrade to newer version of Xen if code base from RedHat for SLC gets shipped with newer versions.
    • Modify ncm-xen to meet requirements of new xml configuration files, if Xen 3.1 or above gets introduced.

    7.13. Integration with new automated provisioning infrastructure at CC

    Since September 2007 Dan Dengate (a technical student, working for the LA section) is investigating a solution to replace the current incarnation of AIMS.
    The project documentation can be found here. In the early days it turned out that Cobbler External site, also a RedHat development, could be a good candidate.

    This has certain influence on the way how virtual machine are deployed. Cobbler has already some built-in functions which support the provisioning of virtual machines and thus makes live easier in the future.

    In short:

    • Keep in touch with Dan Dengate and the LA Section to be informed about late-breaking progress on the AIMS rewrite project.

    In certain circumstances it is not necessary to follow up the proposed solution with pypxeboot.

    8. Who uses the Virtualization Service of FIO

    Although the Xen based virtualization environment was put in production by FIO no more than a couple of month ago there is a lot of interest from IT people. The table below shows as of January, 22nd 2008 Users or Projects, the context they are working in and which machines they are using:

    User/Project Context Environment
    James Casey (IT-GD-GIS) WLCG Monitoring, GridMap development and test 2x virtual machines: lxvm0240,lxvm0241
    Andras Horvath (IT-FIO-TSI) build environment for u-boot 2x virtual machines: lxvm0242 (64-bit), lxvm0251 (32-bit)
    Ian Neilson Grid Monitoring, Replacement for lxb0725 1x virtual machine: lxvm0250 (32-bit)
    Ignacio Reguero Test Grid LFC on SLC4/32 1x virtual machine: lxvm0252 (32-bit)
    ETICS, Marian Zurek Test machine for ETICS build system (VMWare only) 1x host system: lxxen002
    EGEE SA3 activities, Andreas Unterkircher (IT-GD-ITR) EGEE SA3-T2 Development of tests for the certification of gLite releases as well as providing a framework for test submission, presentation and archiving 24x host systems (13x 32-bit and 11x 64-bit): lxxen for XenForGD subcluster (18) and lxdev for XenForGD subcluster (6)

    Unfortunately this is a static list. Since the process from the request to the deployment of a virtual machine is not fully automized this was small effort. If the demand for virtual machine environment at CERN grows and fully automized solutions for deployment are implemented this list becomes outdated.

    Topic attachments
    I Attachment History Action Size Date Who Comment
    PNGpng pxe_boot_domU_v2.png r1 manage 296.5 K 2008-02-03 - 16:06 JanMichael  
    PNGpng xen_architecture_rough.png r1 manage 239.5 K 2008-02-03 - 16:06 JanMichael  
    PNGpng xenmon_lxbuild048.png r1 manage 179.8 K 2008-02-03 - 16:05 JanMichael  
    PNGpng xenmon_lxbuild048_1.png r1 manage 106.1 K 2008-02-03 - 16:04 JanMichael  
    PNGpng xentop_lxbuild048.png r1 manage 110.3 K 2008-02-03 - 16:05 JanMichael  
    PNGpng xm_list_lxbuild048.png r1 manage 63.0 K 2008-02-03 - 16:05 JanMichael  
    Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
    Topic revision: r3 - 2008-02-03 - JanMichael
     
      • Cern Search Icon Cern Search
      • TWiki Search Icon TWiki Search
      • Google Search Icon Google Search

      Sandbox All webs login

    This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
    or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback