Project Based Release Builds

Introduction

This is a braindump on Project Release Builds, based on earlier discussions and email threads.

Concept

The overall concept is to split ATLAS software into multiple projects that have well defined dependencies between them, each project having it's own development release cycle. One possible decomposition would be:

   PhysicsAnalysis
   Reconstruction
   Simulation
   Core

Each of the above (Reconstruction, etc.) would be treated as an independent project.

There isn't a strict hierarchy inplied by the above, but Simulation would depend only upon Core, likewise Reconstruction would depend only upon Core, and finally PhysicsAnalysis would depend upon both Reconstruction and Core. I'll use this set of Projects for the following discussion, but other decompositions would be feasible and might be preferable, and I discuss some alternatives later.

CMT Support for Projects

CMT provides support for projects via the CMTPATH environment variable. ATLAS already takes advantage of this and comprises three separate projects:

  AtlasRelease
  Gaudi
  LCGCMT

which correspond to a CMTPATH of:

  AtlasRelease:Gaudi:LCGCMT

CMT also provides a mechanism for "stitching" such sets of projects together so as to build CMTPATH from dependencies expressed within each project. Each project requires a package that sets up the location and explicit version of the next project, and also adds a dependency against the corresponding package within that project.

For example, the ATLAS release has a package called External/ExternalComponents (not an ideal name, but that's not relevant for this discussion). This package sets up the location of Gaudi, and the explicit version that is to be used for this release. Finally it sets up a wildcarded dependency against the package within Gaudi that does the equivalent for the LCGCMT project. The (simplified) relevant requirements file fragment is:

macro GAUDI_DIR  "${SITEROOT}/atlas/offline/external/Gaudi"
macro GAUDI_RELEASE 0.14.6.14

path_remove CMTPATH "/Gaudi/"
path_append CMTPATH "$(GAUDI_DIR)/$(GAUDI_RELEASE)"

# Setup dependency against LCGCMT Project - The LCGEnv package within Gaudi does this
use LCGEnv v*

This sets up the CMTPATH fragments correctly, and is self-contained in the sense that each project contains within itself the information about the project (or projects) that it depends upon. Furthermore, there are no redundant dependencies that could result in inconsistencies. The ATLAS release only knows about the specific Gaudi release, and it's the Gaudi release that knows about which LCGCMT release it was built against, and makes that information available to the ATLAS release.

Boundary Conditions & Constraints

  • No package can appear in more than one project.

  • Packages in each project can depend upon other packages in the same project and deeper projects(s), but not upon those from higher projects. Thus a Simulation package can depend upon packages from the Simulation and Core projects, but not from the Reconstruction project.

CVS Package Structure and Container packages

Much of the existing CVS container package infrastructure does not map well into project based builds because it is orthogonal to the proposed decomposition. Thus a Simulation project will be composed of packages taken from across multiple sub-detector containers such as InnerDetector. I propose therefore that we ignore the CVS container structure when forming projects. Ideally at some future time, these container packages (e.g. InnerDetector) should be converted back into simple directories within CVS (which only means that the cmt directory & requirements files should be removed), although there might be other uses that I'm not thinking of here.

I propose that we use a similar approach to that taken by the existing nightlies and that each project should have a single package that lists all the other packages and versions within the project release. This would eliminate the time consuming and error prone procedure of tagging container pacakges that is currently needed. The Tag Collector API could simply be used to obtain the necessary list of packages and tags.

I therefore propose the following package structure for these project packages within CVS:

offline/Projects/CoreRelease /RecoRelease /SimuRelease [etc.]

where we can decide on sensible names (e.g. Releases/ is an alternative to Projects/ and AtlasReco might be preferred to RecoRelease).

Each such package would just consist of a requirements file containing the list of packages and versions contained within the release. None of the existing container packages (such as Reconstruction, InnerDetector) would appear in these lists, other than as a part of the CVS path to the relevant leaf packages.

Project Coupling Packages

As mentioned within the Section "CMT Support for Projects", each project needs a package that provides the coupling to the project (or projects) that this project depends on. I propose the following name for this package:

offline/Projects/<Project>/<Project>Projects
Thus the RecoRelease project would have a package:
offline/Projects/RecoRelease/RecoReleaseProjects
I note that Christian had earlier proposed the name Settings for this package.

One aspect of this that might need to be improved relative to what we are using now is the way in which the site and distribution kit specific location of a project is specified, which is rather cumbersome. It's not obvious how one would checkout multiple projects to reside alongside each other.

Tag Collector

In this model, each project needs a dedicated instance of the Tag Collector in order to specify the set of packages and tags that will be used for a particular project. Since each project will have it's own timeline, and release numbering, I do not believe that a single instance of the Tag Collector will be sufficient. However, it might still be advantageous to display the relevant packages in a hierarchical fashion within each project and to allow protected tag updates within a hierarchy.

I note that thus approach does not ensure that every package is included once and only once across the projects. I don't think the Tag Collector Groups provides the required functionality for this since it also doesn't provide the provising of a release timeline per group (unless I'm missing something).

Nightly Builds

Each project must have it's own set of nightly builds with associated web pages. Presumably the nightly for each project is built against a fixed version of its immediate dependent project.

Management

Each project needs its own release coordinator. The role of each coordinator includes the role of the existing single release coordinator for that project, but also must include the decision on which version of the dependent project should be used, both for the nightly and development/production releases.

Testing and Test Packages

In general each project should contain it's own set of test packages for testing. However, such packages suffer from the same constraints as the project itself in that they cannot depend upon any other packages apart from those in the project itself, and any project(s) that this project depends upon. For example, the Core project might have a set of test packages that exercise StoreGate, which itself is part of the Core package. However, any test packages that are needed for more extensive tesing using e.g. Reconstruction packages must themselves reside within the Reconstruction project rather than the Core project. This is a problem since it's not obvious how to perform adequate testing of the lower level projects such as Core. I have a couple of suggestions:

  • Trial release builds of higher level projects against new versions of lower level projects. This is manually intensive, both in terms of understanding which release versions should be coupled, and also the release builds themselves.
  • A integration set of nightly builds that runs slightly behind (~24 hours?) the project nightly builds, but essentially uses the package versions from each project nightly builds. I would expect these to be somewhat unstable since e.g. the Reconstruction nightly builds will normally be built aggainst a stable Core project release rather than the Core nightly, but I would also expect them to decrease the overall integration problem.

Release Distribution

Distribution kits should be associated with each project.

Release Directory Structure

For each project one needs the following releases to be available:

  • Nightly builds (nightlies)
  • CERN AFS builds corresponding to the actual builds (builds). These are needed as a stepping stone to building (and rebuilding) the distribution kits but I would propose that we reduce their public visibiity and direct most users towards the following releases.
  • CERN AFS builds created from the distribution kits (installs). These are what should be used by the majority of users.

I propose the following directory structure on AFS:

/afs/cern.ch/atlas/software/projects/
                                     ProjectA/
                                              builds/
                                              installs/
                                              kits/
                                              nightlies/
                                     ProjectB/
                                              builds/
                                              installs/
                                              kits/
                                              nightlies/
                                     [etc.]
Thus each project is represented in the projects/ directory, and contains directories for the three sets of releases discussed above, and a kits/ directory for the distribution kits.

An alternative organization is certainly possible:

/afs/cern.ch/atlas/software/builds/
                                   ProjectA/
                                   ProjectB/
                                   [etc.]
                           /installs
                                   ProjectA/
                                   ProjectB/
                                   [etc.]
                           /kits
                           /nightlies
As further food for thought, the following would also be feasible (as would others):
/afs/cern.ch/atlas/software/projects/
                                     ProjectA/
                                              ProjectA-01.02.00/
                                                                builds/
                                                                installs/
                                                                kits/
                                                                nightlies/
                                              ProjectA-01.03.00/
                                                                builds/
                                                                installs/
                                                                kits/
                                                                nightlies/
                                              [etc.]
                                     ProjectB/
                                              ProjectB-02.00.00/
                                                                builds/
                                                                installs/
                                                                kits/
                                                                nightlies/
                                              ProjectB-03.00.00/
                                                                builds/
                                                                installs/
                                                                kits/
                                                                nightlies/
                                              [etc.]
                                     [etc.]
But I think I have a slight preference for the first or third since most developers and users will be interested in a single primary project. I think more thought about how these would map into AFS volumes needs to be made before a final decision is made.

As for the names of the project release directories, I propose the following:

ProjectName-ii.jj.kk
Where ii,jj,kk are two digit major id, minor id, and patch ids as we use currently. This notation differentiates the name of the package within a project that lists all other packages from the release itself. Thus:
/afs/cern.ch/atlas/software/projects/AtlasReco/installs/AtlasReco-01.02.00
will be the location of the kit installed build (using the first directory structure proposed above). Within that project, I would expect that there would be an AtlasReco package, having a tag AtlasRecc-01-02-00.

Possible Project Decompositions

Here are some alternative decompositions for discussions. They are by no means meant to be exhaustive and others might be preferable.

Project Structure 1

   PhysicsAnalysis
   Reconstruction
   Simulation
   Core
This is what's been discussed above.

Project Structure 2

   PhysicsAnalysis
   AnaEDM
   RecoAlgorithms
   RecEDM
   CommonAlgorithms
   CommonEDM
   SimuAlgorithms
   SimEDM
   Core
In this decomposition, the model is to decouple the physics event data model from the Algorithms, and to group together algorithmic tools (whether Algorithms or Tools) can might be useful for both physics analysis and reconstruction (and to a lesser extent Simulation). Thus the dependencies for Physics Analysis might be:
  PhysicsAnalysis:AnaEDM:CommonTools:CommonEDM:Core
This has the advantage that Physics Analysis is better decoupled from reconstruction, but does suffer from the difficulty in having access to Reconstruction algorithms from within the context of physics analysis.

Conclusions

Pause for thought.

-- DavidQuarrie - 09 Feb 2005

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2005-02-10 - DavidQuarrie
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback