Managing Cern Computer Centre nodes

Before you start...

The first thing that you have to decide is if your machine(s) will be in an independent cluster, or if you want to join any of the existing ones. With the independent cluster, you have total control (and responsibility) for the machines. If you join a cluster, the administrators of that cluster will modify the quattor templates for you.

If you decide to join a cluster, you only need the next section. If you decide to join an existing one, this wiki will give you a good introduction to the things that you have to configure.

Requesting new nodes

If you are going to join an existing cluster, ask the administrators before doing the request. They might combine it with other requests and do all of them in one go.

And here you have another option: if you want a development machine, the easiest way is to ask for a virtual machine using https://vmm.cern.ch. These requests are usually answered within a day. The drawback is that this is supposed to be for temporary machines (up to six months). If you need something for a longer term, you should do a request using the link in the next paragraph.

A request for new nodes shall be done through the hardware request form at snow

Notes:

  • Currently no "IT/ES" cluster exists, this is being discussed and eventually will become operational. At the moment clusters being used are
    Name Nodes Admin Used
    gridops 10, shared with IT/GT Stefan Roiser ATLAS and CMS elog service
    dashboard 36 Pablo Saiz, Edward Karavakis all the dashboard development and production machines
    persistency 2 Andrea Valassi

  • In case you prefer to request a virtual machines, this should be explicitly stated in the "additional comments" section

After you have submitted your request you will be informed that a 'service now' ticket has been created and you can track the progress of your request from there.

Customizing the nodes

Pre-requisites

For using Cdb (see section below) you will need to create a configuration file in your home directory "~/.cdbop.conf" with the content

[lxvoadm04] /afs/cern.ch/user/r/roiser > cat ~/.cdbop.conf 
protocol = https
server = cdbserv.cern.ch

How to change a machine configuration

In order to change the configuration of a quattor managed machine you will need to

  • Change the quattor template of this machine
  • Deploy the changes on the node you intend to modify

How to change a quattor template

The machines operated in the Cern Computer Centre are being defined via "Quattor" templates. Quattor profiles are modified via "cdb".

  • ssh to lxvoadm.cern.ch
  • start cdb using your Cern afs account/password

[lxvoadm04] /afs/cern.ch/user/r/roiser > cdbop 
quattor CDB CLI: Version 2.2.0
Enter user-name (roiser): 
Enter password: 
Connecting to https://cdbserv.cern.ch...
Welcome to CDB Command Line Interface
Opening session...
[INFO] session opened with ID <6bR1o5UPld>
Type 'help' for more info
<cdbop@cdbserv.cern.ch: ~> 

  • Get the template you would like to modify

See below on how to find "your" template. Also note that if the template is already checked out you need to use "get -f" to overwrite the existing local template.

<cdbop@cdbserv.cern.ch: ~> get profiles/profile_cmslogbook
[INFO] 'profiles/profile_cmslogbook.tpl': received
<cdbop@cdbserv.cern.ch: ~> 

  • Modify the template with the external editor of your choice (prepending exclamation mark will call it)

<cdbop@cdbserv.cern.ch: ~> !vi profiles/profile_cmslogbook.tpl
<cdbop@cdbserv.cern.ch: ~> 

  • Update and commit the template

<cdbop@cdbserv.cern.ch: ~> update profiles/profile_cmslogbook.tpl
[INFO] '/profiles/profile_cmslogbook': scheduled to be updated
<cdbop@cdbserv.cern.ch: ~> commit
[INFO] '/profiles/profile_cmslogbook': will be updated
please confirm [yes]: 
Comment: test commit
[INFO] please wait...
[INFO] commit OK
<cdbop@cdbserv.cern.ch: ~> 

  • Exit

<cdbop@cdbserv.cern.ch: ~> exit
[lxvoadm04] /afs/cern.ch/user/r/roiser > 

How to deploy your changes

Now that the configuration of the machine has changed you will need to deploy your changes on the node. You have to become superuser on the destination node and execute either of the two commands

  • ncm_wrapper will allow you to define exactly the sub-systems you would like to be put in sync with the quattor template. With a "ncm_wrapper.sh --list" you will receive all possible ncm components on the machine. Using the command "ncm_wrapper.sh <
..." will deploy the changes only for the selected components to the node (e.g. user access, sudo, etc).
/usr/sbin/ncm_wrapper.sh

  • spma_ncm_wrapper will put the machine completely in sync with the template, i.e. will execute all the ncm modules and spma
/usr/sbin/spma_ncm_wrapper.sh

Little Quattor FAQ

How to find a quattor template

The lemon monitoring framework provides links to the quattor template of each node, e.g. http://lemonweb.cern.ch/lemon-status/info.php?entity=cmslogbook. On the page you will find the link to the quattor template under the script_code.png icon, e.g. http://tpl-viewer.cern.ch/cdb-tpl-view/tpl_view.php?profile=profiles/profile_cmslogbook. The name of the template to checkout is always in the first line, e.g.

Template: profiles/profile_cmslogbook  

The browser also allows to follow links to other templates which are included in this one.

I cannot create / modify a template

In case you don't have write permissions to a certain template you can check the acls with

<cdbop@cdbserv.cern.ch: ~> acl_get profiles/profile_cmslogbook
/profiles/profile_cmslogbook: %cmsvoc -> rwa, %gridops -> rwa
<cdbop@cdbserv.cern.ch: ~> group_list %gridops
%gridops: !straylen dcollado horat lfield mbabik roiser wlapka
<cdbop@cdbserv.cern.ch: ~> 

If you are not listed as a group member you will need to ask aa admin (!) to add you.

How to install a new software package

Software packages can be added to a node with the line

"/software/packages" = pkg_add("elog", "2.8.1-1", "x86_64"); 

instead of the version number the string "DEF" can put to get the default version of a package.

How to use lemon sensors / actuators

Below is an example of a lemon sensor and exception/actuator for a web service. Whenever the alarm (exception) is raised the actuator will run and try to restart the service + apache. If it fails 3 times it will execute an alarm to the operators.

"/software/packages" = pkg_add("lemon-sensor-http","0.2-0","noarch");
include pro_monitoring_sensor_httpget ;
include pro_monitoring_metrics_httpget ;
"/system/monitoring/metric/_9514/active" = true ;
"/system/monitoring/metric/_9514/param" = list(
   "url", "https://cms-logbook.cern.ch/elog/",
   "tag00", "title",
   "content00", "ELOG Logbook Selection"
);
"/system/monitoring/exception/_33008/active" = true ;
"/system/monitoring/exception/_33008/actuator" = nlist(
   "execve", '/bin/sh -c \\" /etc/init.d/elogd restart; sleep 5; /etc/init.d/httpd restart ; 
      /bin/echo \\\"Elogd and httpd restarted by Lemon. Please check.\\\" | 
      /bin/mail -s \\\"Elogd and httpd restarted on $HOSTNAME\\\" root \\" ',
   "maxruns", 3,
   "timeout", 30,
   "window", 0,
   "active", true,
);

How to enable user access to nodes

User access can be give to a node by putting the line

"/software/components/useraccess/users/roiser/acls" = list("system-auth"); 

into the quattor template, per user. User access can also given to members of a given unix group

"/software/components/authconfig/method/ldap/nss_base_passwd" = "OU=Users,OU=Organic Units,DC=cern,DC=ch?gidNumber=1399";

egroup (you have to create the corresponding group at https://e-groups.cern.ch/e-groups/EgroupsSearchForm.do

"/software/components/authconfig/method/ldap/nss_base_passwd" = 
   "OU=Users,OU=Organic Units,DC=cern,DC=ch?memberOf=CN=cms-elog-users,OU=e-groups,OU=Workgroups,DC=cern,DC=ch";

or both group id and egroup

"/software/components/authconfig/method/ldap/nss_base_passwd" = 
   "OU=Users,OU=Organic Units,DC=cern,DC=ch?one?|(gidNumber=1399)
      (memberOf=CN=cms-elog-users,OU=e-groups,OU=Workgroups,DC=cern,DC=ch)";

How to give users sudo access to nodes

The template has to contain once the line

include components/sudo/config; 

Then for every sudoer an extra line

"/software/components/sudo/privilege_lines" = push(nlist("user","roiser", "run_as","ALL", "host","ALL", "cmd","ALL", "options","NOPASSWD")); 

has to be created, corresponding /etc/sudoers entry. The one above is the easiest to provide but can be tailored down to the specific needs, e.g. only certain commands, etc. if needed.

How to enable backups for a node via TSM

TSM backups need to be setup together with tsm.support@cernNOSPAMPLEASE.ch . An example for quattor template entries would be

variable tsmserver = "TSM64";
include { 'services/tsmclient/config' };
"/software/components/tsmclient/inclexcl" = push("Include /usr/local/elog/.../*");

where everything under the directory /usr/local/elog will be backed up.

How to install a host certificate via quattor + host-certificate-manager

The quattor template needs to contain the line

include pro_service_hostcertificate; 

On lxvoadm the host certificate can be created with the "host-certificate-manager" tool, e.g.

host-certificate-manager --username=roiser cmslogbook

and subsequently deployed on the destination node via

[cmslogbook] /afs/cern.ch/user/r/roiser > /usr/sbin/ncm_wrapper.sh sindes

Useful Links

Lemon homepage: http://lemon.web.cern.ch/lemon/index.shtml

Lemon monitoring: http://lemonweb.cern.ch/lemon-web/

-- StefanRoiser - 04-Jul-2011

  • script-code:
    script_code.png
Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r7 - 2012-08-29 - PabloSaiz
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback