Available DPM testbeds

DPM trunk testbeds

The DPM trunk testbeds are configured to run the packages built from the trunk code repository of lcgdm/dmlite.

They are configured with DOME + Legacy setup and gridftp redirection

The Public TRUNK testbed is composed by 2 nodes :

  • DPM HEAD node : dpmhead-trunk.cern.ch-> VM Node, 8 CPUs, 16 RAM running CentoOS 7.6 with MariaDB 5.5
  • DPM DISK node : dpmdisk-trunk.cern.ch -> VM Node, 2 CPUs, 4 GB RAM running SL6.9, with a 200GB Volume

The Private TRUNK testbed is composed by 2 nodes:

DPM RC testbeds

The DPM RC testbed are configured to run the packages built from the release candicate code repository of lcgdm/dmlite.

The Public RC testbed is composed by 2 nodes configured with Legacy setup:

  • DPM HEAD node : dpmhead-rc.cern.ch -> VM Node, 8 CPUs, 16 GB RAM running SL6.9 with Mysql 5.1
  • DPM DISK node : dpmdisk-rc.cern.ch -> VM Node, 2 CPUs, 4 GB RAM running SL6.9, with a 200GB Volume

The Private RC testbed is composed by 2 nodes configured with DOME + Legacy setup:

DPM cluster configuration using Puppet

Useful links

Creating and maintaining a VM in the CERN AI (Agile Infrastructure) involves interacting with two complementary systems:

  • the CERN Cloud Infrastructure (Openstack), providing the virtualization layer to instantiate the VMs
  • the CERN Configuration Management System (Puppet/Foreman, GIT), providing the management layer to configure the VMs

CERN AI official documentation

  • CERN Cloud Infrastructure User Guide - cloud docs
  • CERN Configuration Management System User Guide - config docs

CERN AI how-to

  • Module and manifest development cycle (git repositories, branches, shared modules, Foreman environments) - twiki

CERN AI web services

CERN AI SNOW categories

  • Use "Cloud infrastructure" for Openstack related issues and to request VMs and disk space for a service related cluster
  • Use "Configuration service" for Puppet/Foreman related issues

Manage the VMs in the DPM cluster using Puppet

The DPM testbed (host group in Puppet terminology) has been named the '''dpmtb''' host group and is it used for both DPM and LFC services.

Prepare a local clone of the GIT repository

  • Clone the '''it-puppet-hostgroup-dpmtb''' repository and check out the '''qa''' branch:
    • '''git clone https://:@gitlab.cern.ch:8443/ai/it-puppet-hostgroup-dpmtb.git'''
    • '''cd it-puppet-hostgroup-dpmtb; git checkout qa'''
    • Use '''git branch -a''' to list both local and remote branches
  • The repository contains two directories as documented in the twiki:
    • '''code''' stores Puppet manifests, files and templates
    • '''data''' stores Hiera data in yaml format

The simplest workflow is the following:

  • Modify files locally in the '''qa''' branch, then commit them locally and finally push the commits to the server
    • '''git commit -a'''
    • '''git pull --rebase && git push'''
  • Then connect to the node as root and run the puppet agent
    • '''puppet agent -t'''
    • You may also do this after changing Foreman parameters
    • This step is in any case unnecessary as the puppet agent automatically runs every hour

System updates

  • On SLC6/C7, unlike quattor, there is no need to periodically update the O/S date version, as system updates are performed automatically ('''yum distro-sync''' is executed in a daily cron job)

Create Puppet-managed Openstack VMs in the DPM cluster

''Before you start''

  • Read the documentation about getting started
  • Join the '''lxadm-authorized-users''' egroup, to have access to aiadm
  • Request the "Hostgroup manager" role via SNOW, to be able to manage host groups in Foreman
  • Download the openstack RC file and the EC2 credentials from the API access tab in https://openstack.cern.ch/project/api_access/

Check Openstack project (tenant)

Access hostgroup in Puppet/Foreman and in GIT and configure root access

  • Read the documentation about hostgroups and modules and about user access
  • Access to the ''dpmtb'' host group in Puppet/Foreman has to be granted to additional individual users via SNOW ( ticket to SE Configuration Management)
  • Connect to the Puppet/Foreman dashboard on https://judy.cern.ch
    • Create all relevant subgroups in Foreman (Configure -> Host Groups)

Configure root access for the new hostgroup via Puppet/Foreman

  • Read the documentation about user access
  • Connect to the Puppet/Foreman dashboard on https://judy.cern.ch/
    • Add a '''rootusers''' (and/or '''rootegroups''') variable including all relevant users to the Parameters of the '''dpmtb''' host group

Choose a Puppet/Foreman environment (GIT branch)

  • Read the documentation about environments (also on the twiki)
    • Openstack environments correspond to GIT branches
  • Two golden Openstack environments, as well as the corresponding GIT branches, are always defined: master and qa
    • You can start by using the '''qa''' environment

Create your Puppet-managed Openstack VMs using CERN AI tools

  • Read the documentation about creating Puppet-managed VMs
  • Create your VMs on aiadm using the CERN AI tool ''ai-bs'' (after setting up the runtime environment for the Openstack IT_DPM project using . openstack-IT-DPM.sh)
    • Set ''AIBS_HOSTGROUP_NAME'' to specify the Puppet/Foreman host group
    • Set ''AIBS_ENVIRONMENT_NAME'' to specify the Puppet/Foreman environment
    • Set ''AIBS_VMIMAGE_NAME'' to specify the Openstack image
    • Set ''AIBS_VMFLAVOR_NAME'' to specify the Openstack flavor
    • Set ''AIBS_VMAVAILZONE_NAME'' to specify the Openstack availability zone
  • The AI command ''ai-bs'' does two things: it creates the Openstack VM and then registers it in ?Puppet/Foreman
  • When the VMs are ready, you should be ready to ssh as root if your account is included in the ''rootusers'' (and/or ''rootegroups'') variable in Foreman

Resize the swap and root partitions

  • Read the documentation about swap and about storage
  • Resize the swap partition and grow the root partition using command ''growpart'' as described in the documentation about swap

Add storage volumes to the Openstack VMs

  • Read the documentation about adding disk volumes
  • Create storage volumes and attach them to your VMs on aiadm using the Openstack commands ''cinder create'' and ''nova volume-attach'' (after setting up the runtime environment using ''. openstack-IT-DPM.sh'')

Manage the VMs using Puppet

  • You are now ready to manage the VMs using Puppet as described in the Manage section above

-- AndreaManzi - 2018-12-19

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2019-03-19 - AndreaManzi
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    DPM All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback