Jenkins technical documentation (Virtual Machines)

Virtual machine creation

In order to create a virtual machine use CERN openstack infrastructure, a cloud-computing software platform. First of all create a virtual machine for Jenkins server. It will be a master node. It requires less resources, so it is recommended to choose medium virtual machine flavour (details below). Master node’s name should be memorable, because it will be used as Jenkins Web Server domain name (access via browser <master name>.cern.ch:8080) e.g. “totem-ci”. After successful creation of master node create slave a.k.a. worker nodes. They should have more resources (large virtual machine flavour) and it’s recommended that their names reflect their operating systems e.g. “totem-ci-slc6”.

Creation process can be done via console or web GUI. Instruction below describe how to create a virtual machine using graphical interface.

Flavour VCPU RAM [MB] Disk [GB]
tiny 1 512 0
small 1 2048 20
medium 2 4096 40
large 4 8192 80

Flavours details as from 9.07.2015. For more information visit: http://clouddocs.web.cern.ch/clouddocs/using_openstack/vm_flavors.html

    • select boot source from image and appropriate image

Master node Slave node
CC7 Extra 64 bit (newest version) SLCx CERN Server 64 bit (newest version)

    • navigate to Access & Security tab
    • choose key pair or add new one by clicking on “+” button and pasting content of your public key. This key will be injected into authorized_keys file on the virtual machine, so that you can log on to it.
    • click on the Launch button.
  • Wait for the virtual machine to come up. You can check when it’s ready by navigating to its details and choosing console tab. When you will be prompted to log in you are ready to go.

Virtual machine access via SSH

In order to log on to the virtual machine via ssh with default private key (~/.ssh/id_rsa or ~/.ssh/id_dsa) execute in the terminal:

ssh root@<instance name>.cern.ch

or with custom private key:

ssh -i <path to private key> root@<instance name>.cern.ch 

Common virtual machine setup

  • Disable acceptance of host environment variables e.g. locale. Type in the terminal and then reconnect to the virtual machine.

sed -ie s/^AcceptEnv/#AcceptEnv/g /etc/ssh/sshd_config
service sshd restart 

  • Install the newest version of git from source. Navigate to page https://github.com/git/git/releases add check for the latest release. Following commands are valid for git version 2.4.8.

yum install -y wget curl-devel expat-devel gettext-devel openssl-devel zlib-devel perl-devel perl-CPAN
yum groupinstall -y "Development tools"
wget -O v2.4.8.tar.gz https://github.com/git/git/archive/v2.4.8.tar.gz
tar -zxf v2.4.8.tar.gz
cd git-2.4.8/
make configure
./configure --prefix=/usr/local
make
make install
cd ..
rm -rf git-2.4.8/ v2.4.8.tar.gz

Common virtual machine setup (Scientific Linux)

  • In order to enable automatic update system edit /etc/sysconfig/yum-autoupdate file and set:
    • YUMUPDATE=0 to be informed about available updates (by e-mail to root)
    • YUMUPDATE=1 for the automatic updates to be applied (default option)
  • Next configure and start automatic update system:

/sbin/chkconfig --add yum-autoupdate
/sbin/service yum-autoupdate start

Common virtual machine setup (Centos)

  • In order to enable automatic update system execute in the terminal:
yum install -y yum-cron
  • Edit file /etc/yum/yum-cron.conf and set “apply_updates” variable to “yes”
  • Enable and start service:
chkconfig yum-cron on
service yum-cron start

Root partition resize (necessary only for Scientific Linux OS)

This step is necessary only for Scientific Linux OS. In CentOS 7, cloud-init will resize the system disk automatically so no additional work is required. For more details read: http://clouddocs.web.cern.ch/clouddocs/guest_specific_procedures/cern_centos_7.html

  • Log on to the virtual machine via ssh
  • Resize partition
    • For Scientific Linux 5 execute:
      • fdisk /dev/vda
      • type “d” followed by “2” in order to delete partition vda2
      • type “n” followed by “p”, “2” and double enter (accept defaults) in order to create new bigger partition
      • type “p” in order to print and check details of newly created partition
      • type “w” in order to write changes to the system
    • For Scientific Linux 6 execute:
      • growpart /dev/vda 2
  • Reboot the machine by navigating to openstack dashboard https://openstack.cern.ch/dashboard/project/instances/, selecting it and clicking on “Soft Reboot Instances” button
  • When machine comes up log on to it again and type in the terminal:
pvresize /dev/vda2
lvextend -l +100%FREE /dev/mapper/VolGroup00-LogVol00
resize2fs /dev/mapper/VolGroup00-LogVol00 # this process may take significant
# amount of time, depending on the size of partition and the computing power

Master virtual machine setup

This instruction is appropriate for Centos 7. It may not work for different operating systems.

  • Type in the terminal:

yum install -y wget firewalld firewall-config java-1.8.0-openjdk fail2ban

chkconfig firewalld on
service firewalld start
# open 8080 port used by Jenkins
firewall-cmd --zone=public --add-port=8080/tcp --permanent
firewall-cmd --reload

chkconfig fail2ban on
service fail2ban start

wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
yum install -y jenkins
chkconfig jenkins on
service jenkins start
chsh -s /bin/bash jenkins # add shell for jenkins user

  • Reboot the machine by navigating to openstack dashboard https://openstack.cern.ch/dashboard/project/instances/, selecting it and clicking on “Soft Reboot Instances” button.
  • Create master key pair. This is an optional step required only if you want to base SSH communication between master and slaves on public key infrastructure rather than username/password.

su - jenkins
echo “create key pair for Jenkins user and accept the defaults.”
echo “No passphrase will be used.”
echo “It will be used to log on to slaves.”
ssh-keygen -q -f ~/.ssh/id_rsa -N ‘’

echo “You can also secure private key using a passphrase,” 
echo “which have to be later on provided in slave configuration step.”
ssh-keygen -q -f ~/.ssh/id_rsa

echo  “copy masters public key. it will be used in Slave virtual machine  setup step”
cat ~/.ssh/id_rsa.pub

Slave virtual machine setup

  • Install required software and libraries:

yum groupinstall -y "Development tools"

yum install -y java-1.8.0-openjdk useraddcern subversion fail2ban libtool mesa-libGL-devel mesa-libGLU-devel cmake xrootd-devel xrootd-client xrootd-compat-libs xrootd-python xrootd-client-devel castor-devel castor-rfio-client qt qt-devel qt-x11 libxml2 libxml2-devel root root-physics

# if JDK 1.8.0 is not available install java-1.7.0-openjdk - this might be the case of SLC5
yum install -y java-1.7.0-openjdk

  • Enable fail2ban service

chkconfig fail2ban on
service fail2ban start

  • Enable firewalld service (required only for Centos OS)

yum install -y firewalld firewall-config
chkconfig firewalld on
service firewalld start

  • Type in the terminal:

export AGENT_USERNAME=totemjenkins
# execute line below if agent user is available in CERN LDAP and not in your local system
useraddcern --login ${AGENT_USERNAME}
export AGENT_WORKSPACE=/var/jenkins
mkdir -p ${AGENT_WORKSPACE}
AGENT_GROUP_NAME=`id -g ${AGENT_USERNAME}`
chown ${AGENT_USERNAME}:${AGENT_GROUP_NAME} ${AGENT_WORKSPACE}
# if the command below returns a warning it should be ignored 
usermod -d ${AGENT_WORKSPACE} -m ${AGENT_USERNAME}

  • If you have not migrated account from CERN LDAP you may want to change the password for this account:

passwd ${AGENT_USERNAME}

  • Generate Kerberos keytab. This is an optional step if previous keytab is not available for agent user. Copy-paste doesn’t work on the SLC5 machine:

export AGENT_USER_KERBEROS_PASSWORD=<agent password>
echo -e "addent -password -p ${AGENT_USERNAME}@CERN.CH -k 1 -e rc4-hmac\n${AGENT_USER_KERBEROS_PASSWORD}\naddent -password -p ${AGENT_USERNAME}@CERN.CH -k 1 -e aes256-cts\n${AGENT_USER_KERBEROS_PASSWORD}\nwkt ${AGENT_USERNAME}.keytab\nquit" | ktutil

For SLC5 you have to start ktutil program and type manually following commands. Remember to substitute agent username placeholder with agent username (same as ${AGENT_USERNAME} environment variable).

addent -password -p <agent username>@CERN.CH -k 1 -e rc4-hmac
addent -password -p <agent username>@CERN.CH -k 1 -e aes256-cts
wkt <agent username>.keytab
quit

Move keytab to the right directory and set appropriate rights:

export KEYTAB_NAME=${AGENT_USERNAME}.keytab
mv ${KEYTAB_NAME} /etc
chmod 600 /etc/${KEYTAB_NAME}
chown ${AGENT_USERNAME}:${AGENT_GROUP_NAME} /etc/${KEYTAB_NAME}

  • Execute in the terminal:

su  ${AGENT_USERNAME}
mkdir -p ~/.ssh
# if you have created master key pair in the previous section execute the line below
echo <content of master’s public key from “Master virtual machine setup” step> >> ~/.ssh/authorized_keys

  • Append SVN key into known_hosts file:

ssh-keyscan svn.cern.ch >> ~/.ssh/known_hosts

  • Log in as a root user (press Ctrl-D as agent user) and type in the terminal:

# the line below is required for SLC6
semanage fcontext -a -t user_home_t "${AGENT_WORKSPACE}/.ssh(/.*)?"

restorecon -R -v ${AGENT_WORKSPACE}

  • Try to log on to it from master machine using Jenkins user. This process will append slave’s public key to known_hosts file on the master machine. Execute in the terminal of the master machine:

su - jenkins
export AGENT_USERNAME=totemjenkins
ssh ${AGENT_USERNAME}@<slave hostname>
# If you have not created master key pair you will have to provide password to
# AGENT_USERNAME account. If this account comes from CERN LDAP the password
# will be the same as the one used to login to lxplus machine using this account.
# If you have protected master private key with a password you will have to provide it here.
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2015-09-08 - KrzysztofAndrzejTrzepla
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    TOTEM All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback