TaskForce

Here is a summary of the activities, issues and decisions involving the integration and configuration task-force.

Initial Situation
  ARC dCache gLite UNICORE
Build units 1 single spec and control file 2 ~300 ~20 (Number of relevant pom.xml Maven project descriptors)
Packages 3 11 ~300 Each pom.xml produces a jar file. Additionally, there are bundles ("assemblies", e.g. in tar.gz format) that are delivered to end-users.
Languages C++ and Python Java, C, sh, python, ruby C, C++, sh, Java, Perl and Python Java
Build System Autotools in Mock and PBuilder (triggered by NorduGrid) Ant (triggered by Hudson) Many (integrated with ETICS) Mostly Maven for jar files and assemblies, Ant for some helper tasks
Build Space chroot externals installed as root in VM, build in user space user space user space
Number of Externals ~35 build-time + 6 runtime ~60 ~60 explicit + many OS installed ~ 50
External location YUM/APT repositories YUM/APT repositories tar.gz in ETICS repository jar in Maven repositories
External versions OS defaults OS defaults selected versions based on OS defaults selected versions
Packaging 100% OS based OS based but not 100% guidelines compliant for server, 100% OS based for dcap OS based but with custom dependency versions and not compliant tar.gz, zip OS independent. Starting to investigate rpm and deb
Platforms Fedora, Debian, Mac, Win Debian, SLC4 i386 and x86_64 and SL5 x86_64, Fedora, Solaris SLC4 i386 and x86_64 and SL5 x86_64 independent (need only Sun Java >=1.5 runtime)

Issues and solutions

Issue
External dependencies used by different middleware distributions are of different versions and are taken from different sources
  • VOMS from EPEL is not equal to VOMS from gLite
  • Globus from EPEL is not equal to Globus from VDT
  • Other externals are taken either from DAG or from EPEL and they may not be identical
  • Java dependencies are taken from Maven, JPackage and OS packages

Solution
Everybody tries to use default versions from OS YUM/APT repositories and EPEL
  • If Java components are not to be packaged for Linux distributions, they can use Maven externals. Otherwise EMI will have to maintain many Java packages to be used by Mock and PBuilder. A possibility is to use JPackage. To be better investigated.
  • Basic components such as VOMS will need to be packaged properly according to EPEL guidelines to be used by others


Issue
Different ways of installing external dependencies: ARC and dCache install RPM/DEB, gLite and UNICORE install them in user space and the build systems handle the locations

Solution
External dependencies will be installed from RPM/DEB packages. Internal components will be instead installed in user space.
  • dCache already works in this way
  • This solution gets close to what ARC is currently doing
  • ETICS will change behavior to install the external packages from YUM or APT instead of installing tar.gz in workspaces
  • ETICS will change behavior and the externals project will only contain the OS default versions of package (apart from exceptions to be considered)
  • Maven can't use OS defaults
  • If Java components are not packages for Linux distributions, they can use Maven externals, otherwise only OS defaults when available


Issue
gLite and UNICORE have tools for developer workspace setup while dCache and ARC have no interest in them

Solution
gLite developers will have to install the external dependencies in the machine when setting up workspaces executing etics-checkout
  • Only external packages will be installed, components built from source (typically internal EMI components) will not be installed but installed in user-space (i.e. staged in the workspace)


Issue
Different interest in external dependency versions: ARC and dCache focused on defaults from Linux distributions, gLite used to be focused on exact control of dependencies, UNICORE gives some freedom to developers, but leverages Maven dependency management to avoid conflicts. Dependencies are bundled in the assemblies (tar.gz, zip), so these are independent of anything installed in the OS (except Java runtime).

Solution
Not much control over external dependencies is possible anymore. Try also to minimize non-OS dependencies.
  • Users of non-OS or non-EPEL dependencies will be responsible of maintaining the configuration and packaging of that software, according to project packaging policies. No externals will be imported anymore from tar.gz
  • Again, Java can be handled separately.


Issue
Different interest on packaging for distributions: ARC first priority, interest and future goal for gLite and dCache, UNICORE is interested

Solution
As a first attempt, the integration build will be executed just to evaluate compilation/incompatibility problems without caring much about proper packaging for Linux distributions, as a second step this will be handled


Issue
UNICORE sees some practical difficulties in building everything from source (CVS or SVN), due to the versioning and dependency management scheme (SNAPSHOT versions vs. released versions). gLite is used to do it. ARC and dCache would accept it but do not consider it a priority since their integration is done at source-package level. Building from source is also necessary for the project QA process (metrics generation, software validation, etc.).

Solution
Also following the PEB directions, all software will be built from source.


Issue
ARC consider a priority to produce buildable source-packages. It is also a good practice having the release manager re-building from source any released package, before signing.

Solution
ARC and ETICS experts will evaluate how difficult is to make the ETICS packager to produce buildable source-packages. In the current ETICS implementation, the major problems in having buildable source-packages are not related to the packager itself but to the external dependency mismatch with respect to OS packages.

Progress

  • 05/08/2010 - First Meeting - Indico event with minutes
    • Each MW partner will send information on how to fragment the software in build/release units based on dependencies, languages, etc.
    • Each MW distribution will keep its own tool chain for building/releasing but will keep in synch the EMI tool chain
    • Only one repository will be used, not one per node or per service or per MW
    • Java software will be probably handled separately and independently from the OS guidelines
    • Platforms for building: RHEL5 (and flavors) 32bit and 64bit, Debian5 32bit and 64bit, investigate on RHEL6
    • External dependencies (non Java) will be taken from the OS and EPEL when possible, including Globus.
    • External dependencies for Java will be taken (freely?) from Maven repository

  • 10/08/2010 - Second Meeting
    • Outcome of the 09/08/2010 PEB about integration:
      • A 100% build from source is highly desiderable
      • EMI Resources are allocated to configure the various middleware distributions to use the unifiedintegration tool
      • Proper packaging for Linux distributions is not EMI's first priority at least for now, build integration is
    • A proposal has been discussed using ETICS (with some modifications) as the main integration tool:
      • All the external dependencies will be installed in the WN from RPM/DEB instead of being placed within the build workspace in user space
      • The allowed versions for the externals will be only (apart from exceptions) the ones available in the OS repositories or EPEL
      • Java components will be built from source with Maven triggered by ETICS and using a local repository to share the built components
      • There will be some effort to evaluate how difficult is to make the ETICS packager to produce source packages valid for Linux distribution builds
    • A document (available above) will be drafted by Lorenzo with a first integration proposal to be discussed at the EMT mailing list.

  • 12/08/2010 - Integration Guidelines ready

  • 01/09/2010 - Configuration Guidelines started
  • 10/09/2010 - Configuration Guidelines continued (Properties and Dependencies section completed)
  • 04/10/2010 - Changed the integration guidelines after SA1-SA2 meeting (Alberto Di Meglio, Alberto Aimar, Cristina Aiftimei, Francesco Giacomini, Lorenzo Dini)

Material

  • EMI_Integration_TaskForceV1 PPT PDF
  • EMI_Integration_TaskForceV2 PPT PDF
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2010-11-29 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback