Continuous Builds

The LCGDM software development process is based on the idea of continuous builds. Whenever there is a change to the source code, in any of the components, a build system updates the installable packages in a shared directory. Anyone can install those packages, which represent the trunk release and the candidate release.

Our build system is Jenkins https://jenkins-lcgdm.web.cern.ch/

The components that are related to DPM are being incrementally migrated from SVN to GitLab. What follows are managing instructions related to both platforms, knowing that at the end of the migration only GitLab will be used.

Repositories

The builds populate repositories that are publicly available. One can configure a machine to always use the package repository that comes from the last successful build of the ''SVN trunk'' or of the ''candidate release'' version.

1. Download the appropriate repository file here

2. Or configure it by hand using something like (example for EL6):

[lcgdm-cbuilds-relcand-el6]
name=LCGDM Continuous Build Repository - Candidate release
baseurl=http://grid-deployment.web.cern.ch/grid-deployment/dms/lcgdm/repos/release-candidate/el6/$basearch
gpgcheck=0
enabled=1
protect=0

3. Add the WLCG repo:

[wlcg]
name=WLCG Repository
baseurl=http://linuxsoft.cern.ch/wlcg/sl6/$basearch
enabled=1
gpgcheck=0
protect=0

Components in GitLab

The components are available here: https://gitlab.cern.ch/lcgdm/

The used convention is that

  • the develop branch contains the most advanced code that compiles and works. It can contain experimental features thought to work
  • the master branch contains the "release candidate" code. This branch is manually rebased from master every time the code reaches stability
  • tags are made from the master branch, to document releases

NOTE: As on December 2018, the candidate release builds are made from the master branch. In order to better approach the usual release workflow it would be desirable if the build system were able to build only the latest tag. Apparently this is not as simple.

Components in SVN (incrementally deprecated)

Our builds are used both as nightlies, feeding our development cycle and as producers of releases for a well-known set of tags. For all the components that we manage we build:

  • SVN trunk. This is expected to compile and be usable. The features can be incomplete or not yet completely tested.
  • candidate release. A tag or branch is said to be part of the candidate release if the developer has marked it as such in SVN. This can be performed with a simple SVN command (see below). The candidate release build is expected to always compile and work. The features are expected to be reasonably complete and stable, and missing more deep testing, likely performed by external collaborators.

We build ''SVN trunk'' and ''release candidate'' for SL5 and SL6, hence populating and keeping up-to-date four repositories.

How we use the builds

The build of the SVN trunk is used to feed our nightly tests, and to produce feedback that is useful for the developers and maintainers of the various components. If you are a developer then you should consider the SVN trunk build to verify your work in the shortest possible time.

The build of the release candidate is supposed to be used to feed testbeds like our internal one. If you are a tester, then you may be interested in testing the features before they get released to the public, by using this repository.

How to mark a tag/branch as part of the candidate release

The basic requirement is that the component, when built, produces a packaged file (.rpm) that contains the version number in its filename. In other words, the package file must be recognizable.

Each component can have an SVN directory that acts in a way that is very similar to a symbolic link, through a feature called ''svn:externals''. If we checkout that directory, we checkout whatever the component maintainer decided to bless as candidate release.

Here is an example of how the dmlite component looks like, with its release-candidate directory.

$ svn ls svn+ssh://svn.cern.ch/reps/lcgdm/dmlite
furano@svn.cern.ch's password: 
branches/
release-candidate/
tags/
trunk/

How to create then this svn:externals link is described here:

I verify that the directory is missing:

$ svn ls svn+ssh://svn.cern.ch/reps/lcgdm/dmlite-plugins-librarian/
furano@svn.cern.ch's password: 
branches/
tags/
trunk/

And then we create the directory and link it to the tag we want with these simple commands.

'''IMPORTANT: please note that the svn:externals points to the plain http repository.'''

$ svn mkdir svn+ssh://svn.cern.ch/reps/lcgdm/dmlite-plugins-librarian/release-candidate/
Committed revision 8838.

$ svn co https://svn.cern.ch/reps/lcgdm/dmlite-plugins-librarian/release-candidate/

$ cd release-candidate

$ svn propset svn:externals 'rc  http://svn.cern.ch/guest/lcgdm/dmlite-plugins-librarian/tags/dmlite-plugins-librarian_0_5_0/' .
property 'svn:externals' set on '.'

$ svn commit

As a final test, we checkout this release-candidate directory, to verify that it checksout the tag that we chose into a subdirectory called rc. This is what our continuous build system expects.

$ svn co https://svn.cern.ch/reps/lcgdm/dmlite-plugins-librarian/release-candidate/

 U   release-candidate

Fetching external item into 'release-candidate/rc':
A    release-candidate/rc/cmake
A    release-candidate/rc/cmake/modules
A    release-candidate/rc/cmake/modules/FindDMLite.cmake
A    release-candidate/rc/LICENSE
A    release-candidate/rc/dist
A    release-candidate/rc/dist/dmlite-plugins-librarian.spec
A    release-candidate/rc/RELEASE-NOTES
A    release-candidate/rc/src
A    release-candidate/rc/src/LibrarianCatalog.cpp
A    release-candidate/rc/src/Librarian.cpp
A    release-candidate/rc/src/Librarian.h
A    release-candidate/rc/src/CMakeLists.txt
A    release-candidate/rc/etc
A    release-candidate/rc/etc/librarian.conf.in
A    release-candidate/rc/CMakeLists.txt
A    release-candidate/rc/README
Checked out external at revision 8839.

Checked out revision 8839.

-- FabrizioFurano - 2018-12-18

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2018-12-19 - FabrizioFurano
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    DPM All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback