Distcc as a service


distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network.

distcc should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile.

This quickstart will show you how to use the it-puppet-module-distcccern module.

Please note it supports kerberos auth only and no pump mode.

It need access to AFS for using wlcg compilers.

Why self service ?

As IT ran a pilot service for many years, it became clear that users have different needs. A cluster tuned with some settings may conflict with other users expectations.

So we decided to provide a puppet module to be able to tune/configure to your needs.

Available clusters

Openstack IT Validation Cluster

(This is a temporary cluster for testing our puppet changes if you want a production class service you will have to create your own.)

At this time we have an Openstack cluster run by IT.

You can get a list of available machines by running the following on aiadm :

aiadm$ ai-pdb hostgroup lxdistcc --plain 

Monitoring : https://meter.cern.ch/public/_plugin/kibana/#dashboard/temp/Zzfqq1xWRx2P0n-zap3Y6Q

Atlas (Self Service Openstack cluster)

You can get a list of available machines by running the following on aiadm :
aiadm$ ai-pdb hostgroup voatlasgcc/gcc --plain 
aiadm$ ai-pdb hostgroup voatlasgcc/gcc47 --plain
aiadm$ ai-pdb hostgroup voatlasgcc/gcc49 --plain 

Machine are configured per hostgroup. The idea is to have only one compiler enable (WIP).


This is not an introduction to Openstack or Puppet at CERN, you can read them first:


Enable koji repository

Latest Distcc (https://code.google.com/p/distcc/source/browse/) has been recompiled in koji and is available through koji repos.

Please note that an old version of distcc is available in slc{5,6}-extras and is still supported during the removal of the old physical cluster.

Install the new repositories for distcc6-stable on client machine (distcccern puppet module will do that for you on the server side.)

  name=distcc cern version (krb enabled) [stable] (OS)


  • You need, first, to include in your hostgroup definition the distcccern module:
        include ('distcccern')
  • Configure modules options in your hostgroup yaml file :
       lcg_compiler_dir: "/afs/cern.ch/project/linux/dev/distcc"
       afs_cachesize: 8256000
       distccd_timeout: 1200
  • lcg_compiler_dir can be setup to your own directory. Just copy the file available in the default location. this location need to be available to both clients and servers.
  • afs_cachesize can be set to AUTOMATIC please refer to afs puppet modules for details.
  • distccd_timeout in second adjust to your need. Default 1200.


  # yum install distcc 

Set client options:

  $ export DISTCC_IO_TIMEOUT="600" (Default to 300)
  $ export DISTCC_HOSTS="--localslots=8 --localslots_cpp=16 builder1/$numcore,auth"

A oneliner to generate DISTCC_HOSTS variable from an hostgroup name:

  aiadm$ LIST="--localslots=8 --localslots_cpp=16"; for i in `ai-pdb hostgroup voatlasgcc/gcc --plain` ; do LIST="$LIST $i/8,auth"; done; echo "export DISTCC_HOSTS=\"$LIST\"";
  export DISTCC_HOSTS="--localslots=8 --localslots_cpp=16 aidistcc001.cern.ch/8,auth aidistcc002.cern.ch/8,auth aidistcc004.cern.ch/8,auth aidistcc003.cern.ch/8,auth"
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2015-03-30 - ThomasOulevey
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LinuxSupport All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback