DIRAC Release Procedures (DRAFT)


  • Initial draft (PhC): 16.02.09


This document aims at defining the procedures for a new release of DIRAC to be put together, certified and then rolled out. The purpose of these procedures is to ensure a reliable while reasonably fast deployment of new software.

A new DIRAC version may be required for several reasons. Each of these releases correspond to a change in one of the fields of the version name that is of the form v.r.p.

  • Bug fix or technical fix: change p field
  • New features added: change r field
  • Changes to interfaces or packaging (incompatible with previous releases): change v field

DIRAC Packages and responsibilities

The DIRAC packages are listed in the DIRAC3InstallProject twiki page. Each package must have a release manager, who is usually the main developer. It is up to the package responsible to decide when the package is to be released. In order to allow development to go on, tested software should be tagged in CVS by the release manager. A full DIRAC release would then be made of a set of package tags.

There are two ways that can be envisioned for putting together packages into a DIRAC release:

  1. Global tag over the list of package tags. This allows the whole DIRAC project to be tagged globally. A tag collector is highly desirable in order to prepare a new release and keep track of package tags used in previous releases. The ganga tag collector could be explored. A convention should be set up for the format of the tags.
  2. Use CMT and DiracSys for setting package versions. Each package has its own version (v.r.p.) and tag in CVS. The DiracSys CMT requirements file contains the list of versions to be put together for a given release. Its own tag is the overall DIRAC version number. All packages can be easily checked out using getpack -r Dirac <version>.

DIRAC currently contains at the same level systems that are LHCb-specific. LHCbSystem however is not at the same level in CVS but at the same level in the file system. It might be wise to use another prefix for LHCb-specific packages, or at least to treat them on an equal footing. One could consider whether they have to be released at the same pace as DIRAC pure clients?

Testing and building a release candidate

Testing should be done on the development instance of DIRAC for services and agents. Proper scheduling should be done between developers in case changes from different packages impact the same service or agent.

Whenever new versions of packages are available and it is agreed it is worth a new DIRAC release, a new release candidate should be built. Releases should be discussed at the weekly PASTE meetings and in case of emergency at the more frequent Computing Operations meeting. Obviously proper preparation should be done by eMail exchanges and use of the tag collector.

A new DIRAC release candidate must be built only from versionned/tagged packages. The built process will differ slightly depending on which of the above solutions is chosen, but it will remain fairly simple:

  • Tag the DIRAC version (global tag or DiracSys editing and tagging)
  • Check out DIRAC from CVS
  • Build what has to be built (binaries, scripts)

Certifying a release

Services and agents certification

After a release candidate has been built, it should be deployed on a dedicated certification instance of DIRAC where all services and agents would be restarted. Even services that have not been modified in this release will have to be tested as changes in e.g. the Core or Security packages may affect those services.

A series of tests should be defined that need to be systematically run for certification, e.g.:

  • Submit a few jobs from a stand-by production
  • Submit user jobs
  • Exercise DM services

Provided that conventions for the patch/bug fix releases are strictly followed and testing of the patch was possible, one could envision that these releases be more quickly put in production?

Pilots certification

Whenever a new release of the pilots is made, it should be tested against the new release of the services. Services should also be tested against the current production release of pilots. In case the new services are incompatible with the old pilots, this should be identified as a major release.

Client certification

The candidate release should be deployed at CERN on the LHCBDEV area, and a series of unit tests should be run. In particular compatibility with the current versions of the DM middleware should be tested and eventually a new release of LHCbGrid should be done in order to include bug fixes.

Deploying a release

Certified releases should then be deployed on all production machines:
  • VOBOXes at CERN
  • Online transfer node
  • VOBOXes outside CERN

The question often is raised of deploying only on machines where one believes it is needed. This is a drawback of the single release process. However we should be very careful as some changes may have affected services without them being really modified (e.g. changes in the Framework, changes in the DMS used by the WMS internally etc...). Therefore it is most likely preferable to deploy all central services and agents at once.

Special care should be taken when a new release implies a change of interface or a change of DB schema, as these deployments should be synchronised with the DB update or client release. Interface changes should imply a major release number change, and the framework should disallow incompatible connections.

-- PhilippeCharpentier - 16 Feb 2009

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2009-02-18 - PhilippeCharpentier
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback