EGEE Web>YAIM>YaimDevelopersGuide (2010-07-27, MariaALANDESPRADILLO)

Yaim Developers Guide

This guide is intended to help anyone contributing to the YAIM codebase. If you are new in YAIM and you want to write your own YAIM module, please read our Where to start section.

Introduction
Where to start?
Who is Who
CVS
- Makefile
- spec file
ETICS
Creating a new node type
YAIM version
User creation
- edgusers
Environment variables
VO variables
gLite 3.0 and gLite 3.1 configuration
YAIM documentation for your module
Things to remember
Useful BASH
Testing your YAIM module
Including the YAIM module into the gLite release
- Adding batch system support to the gLite release
Getting an AFS account

Introduction

YAIM is a tool which can be used to configure the gLite middleware. It is basically a set of bash scripts, made up of reusable functions, which configure node types. Each node type needs to call a set of specific list of functions to configure itself. Each function list is identified by the target name of the node type and appears under the /opt/glite/yaim/node-info.d directory. In order to configure the node type a file containing all the variables needed for the configuration is also used. This file is called site-info.def and was formerly containing all the variables for all the node types. Nowadays we are moving towards a new approach where the list of service specific configuration variables is distributed under /opt/glite/yaim/services in a file identified by the target name of the node type. site-info.def will contain only common variables to all services.

A node type can be configured by running the yaim script. It will use the specified site-info.def, passed as an argument, and it will call the necessary function list out of the target name, which is also passed as an argument. By default, it will source the files appearing in /opt/glite/yaim/services for the service specific variables. It's still possible to gather all the variables together in site-info.def and leave the /opt/glite/yaim/services empty. This is a system administrator decision.

For more details about YAIM please read the YAIM 4 Guide.

Where to start?

First of all you should send a mail to yaim-contact@cernNOSPAMPLEASE.ch and let us know which module you would like to develop. Maybe you want to help with an existing module or you want to write a yaim module for something new.

After that, in order to start working, you would need:

To provide us with your AFS login to be able to work in our CVS repository. If you don't have an AFS login yet, please read here. .
Then, you should also register in ETICS if you haven't done it yet to be able to create rpms to be included in a release.

But! don't worry if you don't know very well how to use these tools. We can provide you with the CVS module containing the basic things to start working and we can also launch the ETICS build if you don't know how to use ETICS. In any case, we have prepared two sections in this wiki page with all the details you need to know, check how to use CVS and how to use ETICS.

Once you are ready to work with these tools, you just need to implement your functions using BASH. Check the sections about adding new modules and writing YAIM functions.
You should also check the Acceptance criteria to know whether your YAIM module meets the necessary requirements (functions or features that are common to several modules).
Then, you should test your module with the middleware, yaim core and maybe other yaim modules. Check the section about testing your YAIM module to know how to do this.
Finally, your module will be ready and you would like to include it in the gLite release. Follow the steps described here to know how to do this.

If you are new with YAIM, you may want to follow the tutorial on how to develop YAIM modules.

Who is Who

This is the list of people developing YAIM modules:

org.glite.yaim.amga Soonwook Hwang
org.glite.yaim.argus-server Valery Tschopp
org.glite.yaim.bdii Laurence Field
org.glite.yaim.clients Andrew Elwell
org.glite.yaim.cluster David Smith
org.glite.yaim.core Cristina Aifimiei
org.glite.yaim.condor-server Pau Tallada
org.glite.yaim.condor-clients Pau Tallada
org.glite.yaim.condor-utils Pau Tallada
org.glite.yaim.cream-ce Sara Bertocco
org.glite.yaim.dcache Owen Synge
org.glite.yaim.dpm Jean Philippe Baud
org.glite.yaim.fts Jean Philippe Baud
org.glite.yaim.glexec-wn Dennis van Dok, Jan Just Keijser
org.glite.yaim.hydra Kalle Haponnen
org.glite.yaim.lb Zdenek Salvet
org.glite.yaim.lcg-ce David Smith
org.glite.yaim.lfc Jean Philippe Baud
org.glite.yaim.lsf-utils Massimo Sgavarato
org.glite.yaim.mon Steve Fisher, APEL -> Cristina del Cano
org.glite.yaim.mpi John Walsch
org.glite.yaim.myproxy ?
org.glite.yaim.nagios James Casey
org.glite.yaim.scas Dennis van Dok, Jan Just Keijser
org.glite.yaim.sge-server Gonçalo Borges
org.glite.yaim.sge-client Gonçalo Borges
org.glite.yaim.sge-utils Gonçalo Borges
org.glite.yaim.torque-server Dennis van Dok, Jan Just Keijser
org.glite.yaim.torque-clients Dennis van Dok, Jan Just Keijser
org.glite.yaim.torque-utils Dennis van Dok, Jan Just Keijser
org.glite.yaim.voms Dimitar Shiyachki
org.glite.yaim.wms Sara Bertocco

We are interested in involving as many contributors as possible in YAIM. If you are interested in becoming a YAIM developer, please send a mail to yaim-contact@cernNOSPAMPLEASE.ch.

CVS

All code should be checked into the glite CVS. The CVS repository can be browsed from here. The Yaim CVS directories normally have the naming format org.glite.yaim.node_type_name. Each CVS directory should map to one package. The package name has the format glite-yaim-node_type_name. The only exception to this rule is for the core package, where core is assumed to be a node type. For a release, a CVS tag should be used which should be in the format glite-yaim-node-type_R_x_y_z_r, Where x_y_x is the version and r is the release

To check out code from CVS, set up the following environment variables:

   export CVSROOT=:ext:afs_login@glite.cvs.cern.ch:/cvs/glite
   export CVS_RSH=ssh

Please, have a look at org.glite.yaim.bdii to understand the structure of a yaim module.

The YAIM team can prepare the module with its basic content. The module directory should contain:

a Makefile which builds the package
a spec file with the name glite-yaim-node-type.spec
config/ directory containing:
- node-info.d/ contains the function list, with the format suffix-node_type_name
- functions/ contains the functions necessary to configure the node, with the format config_function_name
- services/ contains the service specific variables necessary for the configuration of the node, with the format suffix-node_type_name
- defaults/ contains the services specific variables that have a default value and are used by the configuration scripts, with the format suffix-node_type_name.pre or suffix-node_type_name.post (the last one in case the variables depend on site-info.def or service/ variables).

The difference between the services and the defaults variables is that the variables declared under services should be configured by the sys admin. The ones in the defaults diectory have normally a default value that changes very rarely, only in advanced configurations. Both directories are optional but we are encouraging developers to start putting node type specific variables there.

It should be possible to build the packages directly from CVS on the supported platform without any third party software being installed. The Makefile and spec file have two special features. The first is that the rpm creation can be done as a non-root user and the second is that the version will be taken from the CVS tag.

Whenever you check out a tag, the rpm will contain the version of the tag. No matter what you have in the spec file, this will always be overwitten with the tag number.
Whenever you check out HEAD, you need to modify the spec file manually with the release and version you want for your rpm. After that, just running make rpm will create the rpm with the version and release you've chosen.

Makefile

Template for Makefile. Please do the necessary changes for your yaim module:

prefix=/opt/glite
package= package_name
name=$Name:  $
tag:=$(shell echo $(name) | sed 's/^[^:]*: //' ) 
version:=$(shell echo "$(tag)" | sed 's/^.*R_//' | sed 's/_/\./g') 
release:=$(shell echo "$(version)" | sed 's/.*\.//') 
version:=$(shell echo "$(version)" | sed 's/\(.*\)\.[0-9]*/\1/')

.PHONY: configure install clean rpm

all: configure

install: 
	@echo installing ...
	@mkdir -p $(prefix)/yaim/functions
	@mkdir -p $(prefix)/yaim/node-info.d
	@mkdir -p $(prefix)/yaim/defaults
	@mkdir -p $(prefix)/yaim/examples/siteinfo/services
	@mkdir -p ${prefix}/share/man/man1
	@mkdir -p $(prefix)/yaim/etc/versions
	@echo "$(package) $(version)-$(release)" > $(prefix)/yaim/etc/versions/$(package)

	@install -m 0644 config/functions/config* $(prefix)/yaim/functions
	@install -m 0644 config/node-info.d/suffix* $(prefix)/yaim/node-info.d
	@install -m 0644 config/defaults/suffix* $(prefix)/yaim/defaults
	@install -m 0644 config/services/suffix* $(prefix)/yaim/examples/siteinfo/services/.
	@install -m 0644 config/man/yaim-node_type_name.1 ${prefix}/share/man/man1/

          
clean::
	rm -f *~ test/*~ etc/*~ doc/*~ src/*~  
	rm -rf rpmbuild 

rpm:
	@mkdir -p  RPMS
	@mkdir -p  rpmbuild/RPMS/noarch
	@mkdir -p  rpmbuild/SRPMS/
	@mkdir -p  rpmbuild/SPECS/
	@mkdir -p  rpmbuild/SOURCES/
	@mkdir -p  rpmbuild/BUILD/

ifneq ("$(tag)","ame:")
	@sed -i 's/^Version:.*/Version: $(version)/' $(package).spec
	@sed -i 's/^Release:.*/Release: $(release)/' $(package).spec 
endif
	@tar --gzip --exclude='*CVS*' --exclude='rpmbuild*' -cf rpmbuild/SOURCES/${package}.src.tgz *
	@rpmbuild -ba ${package}.spec
	cp rpmbuild/RPMS/noarch/*.rpm rpmbuild/SRPMS/*.rpm RPMS/.

spec file

Template for glite-yaim-node-type.spec file. Please, do the necessary changes for your yaim module:

%define topdir %(pwd)/rpmbuild
%define _topdir %{topdir}
Summary: yaim-node_type_name module configures the node type NODE_TYPE.
Name: package_name
Version: x
Vendor: EGEE
Release: x
License: EGEE
Group: EGEE
Source: %{name}.src.tgz
BuildArch: noarch
Prefix: /opt/glite
Requires: glite-yaim-core
BuildRoot: %{_tmppath}/%{name}-%{version}-build
Packager: EGEE

%description
This package contains the yaim functions necessary to configure the NODE TYPE.

%prep

%setup -c

%build
make install prefix=%{buildroot}%{prefix}

%files
%defattr(-,root,root)
%{prefix}/yaim/functions/config_*
%config(noreplace) %{prefix}/yaim/node-info.d/suffix*
%{prefix}/yaim/defaults/suffix*
%{prefix}/yaim/examples/siteinfo/services/suffix*
%{prefix}/share/man/man1/yaim-node-type.1
%doc LICENSE
%{prefix}/yaim/etc/versions/%{name}

%clean
rm -rf %{buildroot}

ETICS

It is neccessary to create an ETICS configuration if your yaim package is going to be a release candidate. For that you need to run a remote build using ETICS.

In order to use ETICS, you need to be registered first. Check the ETICS Registration Howto to know more. Once your registration is done, contact the YAIM team so that they give you the necessary rights to modify the ETICS component you want to work with.

Configurations can be managed using the ETICS Browser. Click in Configuration, then select the org.glite project on the Project menu, and then yaim subsystem. The YAIM modules can be found there.

The following description applies for the ETICS browser:

To create a release candidate rpm, you need first to create a configuration in ETICS. You can clone the existing configuration template prepared by the YAIM team in your component. This configuration is normally called glite-yaim-node-type_R_x_y_z_r. The configuration should contain the Default Platform with Build commands, VCS commands and properties. If there's no platform yet, please add the default one by clicking on the "Attach platform" icon on the icon menu above.

The Build commands should contain:

install: make install prefix=${prefix}

The VCS commands should contain:

checkout: cvs -d ${vcsroot} checkout -r ${tag} ${moduleName}

The Properties should contain for the gLite middleware:

package.buildarch: noarch
package.prefix: /opt/glite

You don't need to define any Environment nor Dependencies.

Once you have cloned the template, edit it and configure the following fields:

Name (exactly the same as the CVS tag: change x_y_z_r to match your CVS tag)
CVS Tag (same as before)
version and Age entries (x,y,z and r of your CVS tag)

Now that you have the configuration defined, you can create your rpms in the ETICS repository. This means that you need to build the package using a remote build. In order to run a remote build and create the rpms in the permanent repository, you have to lock the configuration. Click on the Lock icon. A new lock icon appears next to your configuration and a new menu is displayed. First of all, you just need to select in the menu, under Project Configuration, the glite_branch_3_1_0. You can ignore the Subsystem configuration. Now click on the lock icon on the left. Your configuration is now locked.

Now you are ready to launch a remote build. Select the Remote Build icon on the the icon menu. A new menu will appear and there you have to select:

Packaging&Repository
repository: Publish artefacts to Repository

Host
host selection: CERN Scientific Linux 4 (ia32) with gcc 3.4.6

And then you can click on the icon Submit Remote Build.

mkdir /root/etics
   cd /root/etics
   unset ETICS_HOME CVSROOT
   wget "http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py" -O etics-client-setup
   python etics-client-setup
   export ETICS_HOME=/root/etics/etics/
   export PATH=$PATH:$ETICS_HOME/bin
   mkdir myws
   cd myws
   etics-workspace-setup
   export X509_USER_CERT=path_to_the_user_certificate_public_key_registered_in_etics
   export X509_USER_KEY=path_to_the_user_certificate_private_key_registered_in_etics
   etics-submit build --project org.glite --project-config glite_branch_3_1_0 --register --platforms slc4_ia32_gcc346 --continueonerror --versioneddeps -c glite-yaim-<node_type>_R_x_y_z_r org.glite.yaim.<node_type>

-->

For more information on ETICS, please read the ETICS User Guide.

Note : Please, bear in mind that once you lock a configuration you can't change it anymore.

Creating a new node type

To create a new node type you have to

ask for a new CVS and ETICS modules.
develop any new functions relevant to your node type and reuse existing ones if possible.
create the function list file under the node-info.d folder with your node type name like suffix-node_type_name and list inside the functions that will configure it.
create the needed variables to configure the node type in the services/ folder.
create the needed default variables to configure the node type in the defaults/ folder.
create or update the documentation.

Configuration target name syntax

Configuration targets in YAIM shall have one of these suffixes: glite-, lcg-, egee-, followed by an identifier that can't contain -. When using the yaim command, the configuration target can be called with or without suffix. However, all the files needed for the configuration and that will be explained in the following sections, need to include the suffix.

Common header to YAIM files

YAIM files should contain the following header:

Official EGEE gLite Software license. Text can be found here https://twiki.cern.ch/twiki/bin/view/EGEE/EGEEgLiteSoftwareLicense
Name: name of the function/file.
Description: summary of what the function does.
Authors: mails of the authors.
Notes: Relevant information of the latest changes, something important we want to highlight, etc. (optional)
YAIM module: name of the yaim module where the file is distributed.

This is a template for the header:

##############################################################################
# Copyright (c) Members of the EGEE Collaboration. 2004. 
# See http://www.eu-egee.org/partners/ for details on the copyright 
# holders.  
#
# Licensed under the Apache License, Version 2.0 (the "License"); 
# you may not use this file except in compliance with the License. 
# You may obtain a copy of the License at 
#
#    http://www.apache.org/licenses/LICENSE-2.0 
#
# Unless required by applicable law or agreed to in writing, software 
# distributed under the License is distributed on an "AS IS" BASIS, 
# WITHOUT WARRANTIES OR CONDITIONS 
# OF ANY KIND, either express or implied. 
# See the License for the specific language governing permissions and 
# limitations under the License.
##############################################################################
#
# NAME : 	    config_function_name
#
# DESCRIPTION : This function configures the service X.
#
# AUTHORS :     author_name@domain
#
# NOTES : 
#
# YAIM MODULE:  yaim-node-type
#                 
##############################################################################

functions/

General considerations:

Functions are installed under /opt/glite/yaim/functions.
Each function should define the following sub-functions:
- the function itself called config_function_name
- the function to check whether the relevant variables have been defined in the configuration files config_function_name_check
- the function to set the necessary variables to run the services in the environment config_function_name_setenv
Example:
- Create a file with the same name as the function. Define your function there. Example: config_globus_devel file should contain config_globus_devel() function.
- Create a function called funtion_name_check to implement checks for the existence of necessary configuration variables. Use the 'requires' function for this. Example: config_globus_devel_check().
- Create a function called function_name_setenv to implement the definition of environment variables. Example: config_globus_devel_setenv().

config_globus_devel file would contain:

function config_globus_devel_check(){
	requires $1 INSTALL_ROOT
             return $?
}

function config_globus_devel_setenv(){
	yaimgridenv_set GLOBUS_LOCATION ${INSTALL_ROOT}/globus 
	yaimgridenv_set GPT_LOCATION ${INSTALL_ROOT}/gpt
}

function config_globus_devel(){
	${GPT_LOCATION}/sbin/gpt-build -force -nosrc gcc32 gcc32pthr gcc32dbg gcc32dbgpthr
}

In the example above, the $1 is necessary in the check function since sometimes check function has to quit with error (when running configuration) and sometimes it should just complain but keep on running (when doing the verification with -v ).

The _setenv function should use one of the following commands:

yaimgridenv_set: sets an environment variable
yaimgridenv_setind: sets an environment variable if it is not already defined
yaimgridenv_delete: delete a value from an environment variable. If the value becomes null string then it unsets the environment variable.
yaimgridpath_prepend: prepend a value to a path-like environmnet variable. Puts a ':' to the end end the beginning of the variable. Removes the value from the string and removes duplicate '::'. Prepend the value to the beginning of the variable. Removes duplicate '::' and removes ':' from the beginning and the end variable. Export the variable.
yaimgridpath_append: the same as prepend but it appends the value to the end of the variable

All these commands are written into a file called grid-env.(c)sh that it's located in the path defined by GRID_ENV_LOCATION, which is by default /opt/glite/etc/profile.d. This variable is defined in /opt/glite/yaim/defaults/site-info.post These commands are interpreted by grid-env-funcs.sh which is located in the same path.

Some tips...

For a given service, you should probably need to configure:

creation of users and groups: See the User creation section for more details.
creation or modification of configuration files: where you would need to add or modify some variables (sed can be very useful for this).
start/stop daemons: you would probably need to start or stop some services.
- Remember to use chkconfig to make sure the service starts up after a reboot.
- Use /opt/glite/etc/gLiteservices to add the daemons you want to include in the gLite init script.
cron jobs: use the cron_job util to create your cron jobs.
log rotate: remember to configure the log rotate in the relevant log files.

Comments and --explain option

If you want to comment your code use ####@ and in this way, the --explain function of the YAIM command will print all your comments to give an overview of what the function is doing.

yaimlog

Always use the yaimlog command to log information on the YAIM log file. There are two ways to use this utility.

For short logs:

   yaimlog INFO "My string to print out"

For long printouts:

   (cat << EOF
   This is my
        long explanation message,
    I don't have to repeat it every time...
    EOF
    ) | yaimlog INFO

There are different logging levels: ABORT, ERROR, WARNING, INFO, DEBUG. It's up to the developer to choose which information should be present in each log level:
- ABORT - Explains the exit code
- ERROR - Shows which errors occured
- WARNING - Shows warning messages
- INFO - This is the logging level you should use instead of a normal "echo" to print general information.
- DEBUG - Useful information for debugging. Probably the type of messages you print for yourself when you debug your code.

Error codes

In order to provide meaningful error messages, we have implemented a set of error codes and a function that will print the appropiate error message associated with the error code. This is part of the yaim supercommand code. Please, do not hesitate to help us improving this list with more error codes.

#########################################
# These are standard system exit codes, #
#  see /usr/include/sysexits.h          #
#########################################

YEX_OK=0           # successful termination
YEX_NOFUNC=1       # One of the functions returned with error without specifying it's nature !
YEX_USAGE=64       # command line usage error
YEX_DATAERR=65     # data format error
YEX_NOINPUT=66     # cannot open input
YEX_NOUSER=67      # addressee unknown
YEX_NOHOST=68      # host name unknown
YEX_UNAVAILABLE=69 # service unavailable
YEX_SOFTWARE=70    # internal software error
YEX_OSERR=71       # system error (e.g., can't fork)
YEX_OSFILE=72      # critical OS file missing
YEX_CANTCREAT=73   # can't create (user) output file
YEX_IOERR=74       # input/output error
YEX_TEMPFAIL=75    # temp failure; user is invited to retry
YEX_PROTOCOL=76    # remote error in protocol
YEX_NOPERM=77      # permission denied
YEX_CONFIG=78      # configuration error

# These are YAIM own exit codes

YEX_NOTARGET=90    # The function list definition file for the configuration target hasn't been found
YEX_MISSINGVAR=91  # Missing variables in the configuration files
YEX_NOSUCHFILE=92  # No such file or directory
YEX_CONFFILE=93    # Configuration file error
YEX_NOUSER=94      # User doesn't exist
YEX_USERERR=95     # Error when creating user
YEX_GROUPERR=96    # Error when creating group


##############################################
# Here are the error description definitions #
##############################################

YERRSTR_0="YAIM terminated succesfully."
YERRSTR_64="Command line usage error !"
YERRSTR_65="Data format error !"
YERRSTR_66="Cannot open input !"
YERRSTR_67="Address unknown !"
YERRSTR_68="Host name unknown !"
YERRSTR_69="Service unavailable !"
YERRSTR_70="Internal software error !"
YERRSTR_71="System error !"
YERRSTR_72="Critical OS file mising !"
YERRSTR_73="Can't create (user) output file !"
YERRSTR_74="Input/output error !"
YERRSTR_75="Temp failure; user is invited to retry !"
YERRSTR_76="Remote error in protocol !"
YERRSTR_77="Permission denied !"
YERRSTR_78="Configuration error !"


YERRSTR_1="One of the functions returned with error without specifying it's nature !"
YERRSTR_90="The function list definition file for the configuration target hasn't been found. !"
YERRSTR_91="There are missing variables in the configuration files !"
YERRSTR_92="No such file or directory !"
YERRSTR_93="Configuration file error"
YERRSTR_94="User doesn't exist !"
YERRSTR_95="Error when creating user !"
YERRSTR_96="Error when creating group !"

function yestr() {
 YERRORSTR=`eval echo \\${YERRSTR_$1}`
}

An example of how the yestr can be used:

 yestr ${YEX_NOHOST}
 yaimlog ERROR "${YERRORSTR}"
 exit ${YEX_NOHOST}

services/

This is the folder where you should define the list of variables needed to configure the service. These variables have to be defined by the sys admin, their value really depends on the sys admin choice or the site features. You can distinguish mandatory and optional variables by commenting out optional variables. It's useful to put a brief description of the variable as well. Example:

# Hostname of the Lemon host
LEMON_SERVICE_HOST=your_host

# Optional variable. Comment it out if you want to 
# define a maximum number of connections.
# LEMON_MAX_CONNECTIONS=your_number

The services folder is installed under /opt/glite/yaim/examples/siteinfo/services. Remember that only variables specific to your service, not used by other services, should be defined here. Variables affecting several node types are distributed under site-info.def. The name of the file should be: suffix-node_type_name, where node-type is the configuration target name you have chosen, but in lower case. Example: the configuration target glite-SE_dpm_disk would define its variables under services/glite-se_dpm_disk .

Document the variable definition in your service section of the Configuration variables wiki:

Mandatory general variables : In case the service needs to define site-info.def variables, list the variables here.
Mandatory service specific variables : list and describe the service varaibles here.

defaults/

This is the folder where you should define the list of variables with a default value for a standard configuration of your service. These variables do not need to be defined by the sys admin. Only if the sys admin wants a more advanced or customised configuration, he should re-define them under services or site-info.def. It's also useful to put a brief description of the variable. Example:

# Port where the Lemon service listens to
LEMON_PORT=8443

The defaults folder is installed under /opt/glite/yaim/defaults. Remember that only variables specific to your service, not used by other services, should be defined here. Variables affecting several node types are distributed under site-info.pre/post. Check first if yaim core hasn't yet defined the variable in site-info.pre or site-info.post to avoid declaring the same variables twice:

- site-info.pre in CVS
- site-info.post in CVS

The name of the file should be: suffix-node_type_name.pre/post, where node-type is the configuration target name you have chosen, but in lower case. Example: the configuration target glite-SE_dpm_disk would define its variables under defaults/glite-se_dpm_disk.pre/post .The difference between .pre and .post file is that .post includes variables whose value depends on previously defined variables, like INSTALL_ROOT. To better understand where you have to define a variable, always bear in mind the configuration flow.

Remember that variables in suffix-node_type_name.post should always be defined as: VARIABLE_NAME=${VARIABLE_NAME:-${WHATEVER}/whatever, since the user may redefine this variable in site-info.def and its value should be taken.

Document the variable definition in your service section of the Configuration variables wiki.

Default general variables : In case the service uses site-info.pre/post variables, list the variables here.
Default service specific variables : list, describe and define the default value here.

site-info.def, site-info.pre and site-info.post

These are the files where variables affecting several node types are defined. This is normally done by the central team, but you may need to define a variable that affects several services. If the variable needs to be defined by the sys admin, put it in site-info.def. If it has a default value, put in site-info.pre/post.

site-info.def is installed under /opt/glite/yaim/examples/siteinfo.
site-info.pre and post is installed under /opt/glite/yaim/defaults.

site-info.pre/post define variables which have a default value in standard configurations. Remember that the user can always re-define this variable for advanced configurations, but defining variables by default help users to configure services in a standard way and avoids them to worry about variables they might not be interested in. The main difference between .pre and .post is that .post defines variables that depend on previously defined variables, for example, INSTALL_ROOT. To better understand where you have to define a variable, always bear in mind the configuration flow.

These variables should be documented in the YAIM configuration variable wiki:

site-info.def section
- VO related variables subsection
site-info.pre section
site-info.post section

Configuration flow in YAIM

This is the order in which the different configuration files are sourced:

/opt/glite/yaim/defaults/site-info.pre
/opt/glite/yaim/defaults/suffix-node_type_name.pre
/root/siteinfo/site-info.def
/root/siteinfo/services/suffix-node_type_name
/opt/glite/yaim/defaults/site-info.post
/opt/glite/yaim/defaults/suffix-node_type_name.post
/root/siteinfo/nodes/machine.domain (defined by the sys admin)
/root/siteinfo/vo.d/vo_name (defined by the sys admin)
/opt/glite/yaim/node-info.d/suffix-node_type_name

YAIM version

The version of all the installed YAIM modules in a certain host is printed when we run the YAIM command. We used to query the installed yaim rpms to get this information, but this is not a portable solution. What we have implemented now is a directory called versions installed under /opt/glite/yaim/etc that contains one file per yaim module called yaim-glite-module-name. Within the file we print the version and release numbers for each module. For example, for yaim core we would have:

 
cat /opt/glite/yaim/etc/versions/glite-yaim-core
glite-yaim-core 4.0.5-1

This file is created automatically when we package the yaim module. The following lines need to be added into the Makefile and spec files:

Makefile:

@mkdir -p $(prefix)/yaim/etc/versions

@echo "$(package) $(version)-$(release)" > $(prefix)/yaim/etc/versions/$(package)

spec file:

%{prefix}/yaim/etc/versions/%{name}

NOTE : Bear in mind that the file will contain the proper version and release when we checkout a specific tag from CVS, otherwise it has some funny contents. This is always OK when we use ETICS to prepare a new release.

User creation

Users and groups should be created only if CONFIG_USERS=yes. Otherwise, if the needed user/group doesn't exist, an error should be given. An example code to be used for user creation is:

if [ "x${CONFIG_USERS}"="xyes" ]; then 
  if ! (getent passwd | grep ${MY_USER} > /dev/null) 
  then 
    adduser ${MY_USER} 2>/dev/null 
  fi 
else 
  yaimlog WARNING "CONFIG_USERS is set to 'no'. User ${MY_USER} must exist in your system" 
  if ! (getent passwd | grep ${MY_USER} > /dev/null) 
  then 
    yaimlog ERROR "User ${MY_USER} doesn't exist and CONFIG_USERS is set to 'no'. Please, create ${MY_USER} user" 
    yestr ${YEX_NOUSER} 
    yaimlog ERROR "${YERRORSTR}" 
    exit ${YEX_NOUSER} 
  fi 
fi

edgusers

Special users to be created on the service nodes to run the relevant daemons and processes are now contained under /opt/glite/yaim/examples/edgusers.conf. The function that creates these users is config_edgusers. This function takes into account if CONFIG_USERS is set to 'yes' to create the users in YAIM. Otherwise the sys admin is responsible for creating those users. Then, he should use /opt/glite/yaim/examples/edgusers.conf to know which users have to be created. The format is as follows:

UID:LOGIN:GID:GROUP:DESCRIPTION:HOME_DIRECTORY

where:

UID = user ID
LOGIN = login name
GID = group ID
GROUP = group name
DESCIPTION = user description
HOME_DIRECTORY = optional variable to specify a home directory

The existing users are defined through a set of variables under /opt/glite/yaim/defaults/site-info.pre to be able to change their default login name:

DPMMGR_USER=dpmmgr
DPMMGR_GROUP=dpmmgr
LFCMGR_USER=lfcmgr
LFCMGR_GROUP=lfcmgr
EDG_USER=edguser
EDG_GROUP=edguser
EDGINFO_USER=edginfo
EDGINFO_GROUP=edginfo
RGMA_USER=rgma
RGMA_GROUP=rgma
GLITE_USER=glite
GLITE_GROUP=glite
GLITE_HOME_DIR=/home/glite
INFOSYS_GROUP=infosys
BDII_USER=${EDG_USER}
BDII_GROUP=${EDG_GROUP}

Don't hardcode the user and group names but use these variables instead.

Environment variables

The following variables are always exported into the grid-env.(c)sh environment file. You don't need to include them in your yaim module _setenv functions:

  yaimgridenv_set GRID_ENV_LOCATION "${GRID_ENV_LOCATION}"
  yaimgridenv_set EDG_LOCATION ${INSTALL_ROOT}/edg
  yaimgridenv_set LCG_LOCATION ${INSTALL_ROOT}/lcg 
  yaimgridenv_set GLITE_LOCATION ${INSTALL_ROOT}/glite
  yaimgridenv_set LCG_GFAL_INFOSYS ${BDII_HOST}:2170
  yaimgridenv_set GLITE_LOCATION_VAR ${GLITE_LOCATION_VAR}
  yaimgridenv_setind JAVA_HOME ${JAVA_LOCATION}

  # LD_LIBRARY_PATHs
  if [ "x${OS_ARCH}" = "x64BIT" ]; then
   if [ -d ${LCG_LOCATION}/lib64 ]; then
    yaimgridpath_prepend LD_LIBRARY_PATH ${LCG_LOCATION}/lib64
   fi
   if [ -d ${LCG_LOCATION}/lib ]; then
    yaimgridpath_prepend LD_LIBRARY_PATH ${LCG_LOCATION}/lib
   fi
   if [ -d ${GLITE_LOCATION}/lib64 ]; then
    yaimgridpath_prepend LD_LIBRARY_PATH ${GLITE_LOCATION}/lib64
   fi
   if [ -d ${GLITE_LOCATION}/lib ]; then
    yaimgridpath_prepend LD_LIBRARY_PATH ${GLITE_LOCATION}/lib
   fi
  else
   yaimgridpath_prepend LD_LIBRARY_PATH ${LCG_LOCATION}/lib 
   yaimgridpath_prepend LD_LIBRARY_PATH ${GLITE_LOCATION}/lib
  fi
  # PATH 
  yaimgridpath_prepend PATH ${LCG_LOCATION}/bin
  yaimgridpath_prepend PATH ${EDG_LOCATION}/bin
  yaimgridpath_prepend PATH ${GLITE_LOCATION}/bin
  
  # MANPATH
  yaimgridpath_prepend MANPATH ${GLITE_LOCATION}/share/man
  yaimgridpath_prepend MANPATH ${LCG_LOCATION}/man
  yaimgridpath_prepend MANPATH ${LCG_LOCATION}/share/man
  yaimgridpath_prepend MANPATH ${EDG_LOCATION}/share/man
  yaimgridpath_prepend MANPATH ${INSTALL_ROOT}/glite/yaim/man

VO variables

VO variables can be defined under site-info.d or under the vo.d directory. The variable name is different depending on where it's defined.

If it's defined under site-info.def, we have: VO_<vo-name>_VOMSES. It it's defined under vo.d/vo-name, we have VOMSES. It's therefore important to remember that referencing a variable by doing VO_${VO}_VOMSES is incorrent and we should instead use the utilities:

get_vo_param vo-name variable-name
set_vo_param vo-name variable_name

These functions will look for the VO variable in both locations. Like this we can abstract from the location and therefore from the variable name.

For example:

get_vo_param dteam VOMSES

gLite 3.0 and gLite 3.1 configuration

Since YAIM is detecting automatically whether you are installing a 3.1 or a 3.0 node, we can put together both configurations under the same yaim module. In order to do that, the following instructions have to be followed:

When a function with the same name is different in 3.0 and 3.1:
- In functions/ create config_function_30 for 3.0 and config_function for 3.1
- In node-info.d/ create glite-note-type_30 containing the function for 3.0 and suffix-node_type_name containing the function for 3.1

When you need a new function that doesn't exist in 3.0:
- In node-info.d/ create glite-note-type_30 containing the old function list for 3.0 and create suffix-node_type_name containing the function list with the new function for 3.1

YAIM documentation for your module

Release notes in the Savannah patch

When you have a new yaim module rpm ready you should open a patch in Savannah. There you should remember to include a summary of the changes and new features in the release notes.

Man pages

The man pages should be created under org.glite.yaim.module_name/config/man/yaim-node-name.1.

Please, use the following template to create the man pages for your node type:

.TH "YAIM - Yet Another Installation Manager" 1
.SH NAME
YAIM (YAIM Aint an Installation Manager) is a way of configuring grid services.
The aim of YAIM is to provide a simple configuration method that can be used to set up a simple grid site
but that can be easily adapted and extended to meet the needs of larger sites.

.SH DESCRIPTION
The yaim-node-type module allows you to configure the node type.

.SH CONFIGURATION VARIABLES
The list of variables needed to configure the node type can be found in:
.TP
.B https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#node_type

.SH EXAMPLES
How to configure the node type node:
.TP
.B ./yaim -c -s /root/site-info.def -n node-type
.TP
To debug the configuration process:
.TP
.B ./yaim -c -s /root/site-info.def -n node-type -d 6

.SH DOCUMENTATION
You can find useful information on these web pages:
.TP
The YAIM guide:
.TP
.B  https://twiki.cern.ch/twiki/bin/view/LCG/YaimGuide400
.TP
The Generic Installation and Configuration guide:
.TP
.B https://twiki.cern.ch/twiki/bin/view/LCG/GenericInstallGuide310

.SH AUTHORS
YAIM is a collaborative project where different modules are developed and maintened by different
groups. Check the list of authors in: https://twiki.cern.ch/twiki/bin/view/EGEE/YAIMAuthors
and we are happy to receive more contributions !

.SH CONTACT
To contact the YAIM team:
.B yaim-contact@cern.ch

Wiki pages

There is a wiki page where the variables needed to configure each service are described. Please, check that your node type is in the list or otherwise contact the yaim team:

The service configuration variable wiki page: https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables

There are two more documents that might need to be updated as well:

The Generic Installation and Configuration Guide : https://twiki.cern.ch/twiki/bin/view/LCG/GenericInstallGuide310

You have to update this when there is some gLite release specific issue to be mentioned during a general installation process.

The YAIM Guide : https://twiki.cern.ch/twiki/bin/view/LCG/YaimGuide400

Changes concerning YAIM structure, syntax, etc, should go in here.

Please, contact the YAIM team in order to modify these two pages and in this way we will control the modifications.

Things to remember

Don't use perl, python or anything else. Use only bash !

Always comment your code !! YAIM is a good place for documentation. Please use comments in
```
   ####@ This is a comment to be displayed
   
```
form. Lines having this form will be printed out when yaim is invoked with the -e switch.
Each function is implemented in one single file. The file name and the function name should be the same.

A node type function list is defined in /opt/glite/yaim/node-info.d. The function list file has the same name as the node type target name but in lower case. For example, glite-WN has /opt/glite/yaim/node-info.d/glite-wn.

All the functions needed to configure a node type should be listed in the node type functions list.

The format of the function list file is:

      <TARGET_NAME>_FUNCTIONS="config_function_1
                                                      config_function_2
                                                      ..."

Use always standards solutions, do not reinvent the wheel !
- Use cron_job function for installing crons - this enables cron management to be turned off by sites who use fabric management.
- Use the yaimlog utility for printing out logging information.
- Use the requires utility to check for defined variables.

Never hard-code /opt, use ${INSTALL_ROOT} to allow the middleware to be relocatable.

Uses the error codes.

To find out what type of node is being configured within a function, check the NODE_TYPE_LIST variable. Try to avoid checking the output of `hostname -f` because this will fail if aliases or multihoming are being used.

There are now some useful utils you can use in functions/utils which are sourced automatically. Use them if appropriate. Create a new util if you have to, but remember that the functions should remain as human readable as possible.

Remember that YAIM is more often used to upgrade a site from an earlier middleware release than to configure a freshly installed machine. Consider that sometimes an old configuration (perhaps a cron job) may need to be removed.

Put (leave) the creation of users in a separate function so fabric management can remove this if necessary. config_edgusers and config_users already do this.

Remember to turn daemons on and ensure they come back on reboot. Use the full path for the /sbin/chkconfig and /sbin/service commands.

Use always platform independent solutions which are working on Debian, Suse, SLC, etc..

Not all output of the commands invoked by YAIM should be sent to the terminal - much is redirected to /dev/null. Do not do this with output which might be useful for an admin trying to debug a problem. Use the yaimlog utility to handle the output.

Make sure that any function is runable in a standalone mode and it is able to determine the necessary input information by itself.

Useful BASH

Here you can list and contribute with some good ideas to write bash scripts.

Creating files

Random numbers, passwords, temporary files

Use $RANDOM only for example for crontabs to generate random time distribution, but do not use for sensitive things, like passwords.

Always ensure the uniqueness of a temporaly file. Create and delete a temporary files somehow like this:

    PW_FILE=`mktemp /tmp/lfc_upgrade.XXXXXX`
    echo $LFC_DB_PASSWORD > $PW_FILE
    .
    . Here you can do some other stuff
    .
    rm $PW_FILE
    unset PW_FILE

Parsing strings

Always put your variables into curly bracket: ${MY_VARIABLE}
Always put them into double quotes, especially when comparing 2 string: "${MY_VARIABLE}"

Always use an extra character when comparing strings:

   if [ "xYAIM" == "x${MY_VARIABLE}" ]; then
   fi

To replace all occurence of $vartoreplace in the variable $myvar
```
   mynewvar=${myvar//$vartoreplace/$replacement}
   
```

To remove a string ($mystringtoadd) from the beginning/end of a $myvar:

    mynewvar="${myvar#$mystringtoadd}"
    mynewvar="${myvar%$mystringtoadd}"

Exit codes and redirections

Check the exit code of the previous statement by:

   a=`ls -l | grep myfile`
   if [ $? eq 0 ]; 
      yaimlog Success; 
   fi

Daemonize a program by properly redirecting the stderr, stdio, stdin.

    /op/glite/sbin/program 2>/dev/null 1>/dev/null </dev/null &

Create a new file always using the >| redirections. This will overwrite a file even if the noclobber option is set. Don't use simple >
For redirection from/to standard input use -: tar cf - . This will tar the standard input.

Platform (in)dependent solutions to take into account

Always give the -u flag to the id command.
Be aware that the crontab command works differently on Debian and on SLC.
There is no service command on Debian.
Be careful with the paths, for example there is differences in the case of awk package.

Testing your YAIM module

In order to test your module, you need to install the relevant middleware, yaim core and probably other yaim modules. You should visit the YAIM planning page to know the status of YAIM releases so you can decide which yaim module version you are most interested in to use for your tests.

We recommend you to install the middleware from production or pps repositories. Refer to the production or pps documentation to know which repositories you should use. The PPS releases wiki can also be useful to know more details about the latest updates.

If you want to use a YAIM version which is currently in production or pps, when you install the middleware, YAIM will be automatically installed.
If you want to use a YAIM version which is currently in certification:
- For 3.0, use the following APT string:

rpm http://lxb2042.cern.ch/gLite/APT/R3.0-cert rhel30 externals Release3.0 updates updates.certified patchXXXX.uncertified

where patchXXXX correspons to the Savannah patch tracking the new yaim release for gLite 3.0.

- For 3.1, use the following YUM repository:

[patchXXXX]
name=gLite 3.1 patch XXXX
baseurl=http://grid-deployment.web.cern.ch/grid-deployment/glite/integration/cert/3.1/patches/XXXX/sl4/i386/
enabled=1

where XXXX also corresponds to the Savannah patch tracking the new yaim release for gLite 3.1.

It's very important to read the comments and notes in the patch itself to understand how to install the relevant rpms.

If you use a YAIM version which is currenly under development, send a mail to yaim-contact@cernNOSPAMPLEASE.ch to know the status of the yaim modules you are interested in and to know how to obtain them.

Including the YAIM module into the gLite release

Finally, your module will be ready and you would like to include it in the gLite release. Follow the steps described here to know how to do this.

Adding batch system support to the gLite release

If you are developing a yaim module related to batch systems, please, read the wiki for adding batch system support to the gLite release.

Getting an AFS account

In order to work in CVS, you need an AFS account. If you haven't one, please follow the instructions below to obtain one:

Fill in the following registration form
Fax the registration form along with a short letter explaining which institute you are from and what you intend to use your account for, to this number +41 22 766 91 25

Topic revision: r119 - 2010-07-27 - MariaALANDESPRADILLO

EGEE

Main.WebList

Welcome Guest

- Cern Search
- TWiki Search
- Google Search
EGEE All webs

Copyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback