EMI-0 (Zugspitze) - Activity

EMI-0 - An ‘exercise’ release designed to understand how to apply the agreed procedures, find any problem about tools and processes and in general fine tune the EMI software engineering process before the EMI 1 release. The outcome of EMI 0 is not expected to be released to external users. The goal of EMI 0 is to prepare a consistent, coherent repository of non-conflicting packages by the end of October 2010 without any specific commitment on functionality.

Mandatory Platforms

  • Platforms/architectures:
    • SL5 on 32 and 64bit architectures
    • Debian 5 on 32 and 64bit architectures

Milestones:

End of Sept, 2010 - configuration ready in ETICS:

General Guidelines - to be applied to all EMI Releases

  • A single tool will be used to perform integration builds from source (nightly builds). All build tools will be supported and integrated via this unified integration system.
  • Even though ETICS cannot be considered the optimal tool for each environment, it is the only one able to handle an integration build from source of all the four middleware distributions. Therefore it will be used to produce a first unified integration system.
  • Globus packages for EMI shall be taken from existing standard repository like EPEL whenever available. All components in EMI shall adopt this strategy
  • EMI is to use a common build, test and QA system as much as practically possible. Currently the system of choice is ETICS. All components to be released as part of EMI must have configurations in ETICS
  • EMI is released on two sets of platforms, mandatory and optional. Mandatory platforms must be supported by all components in EMI, optional platforms can be supported by individual PTs depending on needs of technical capabilities. Currently EMI 0 and 1 are going to be supported on RHEL5 and compatible platforms and Debian 5 and compatible platforms on 32 and 64bit platform
  • All components in EMI must be provided in the standard packaging formats typical of the supported mandatory platforms. Currently this means rpms for RHEL5 and compatible platforms and debs for Debian 5 and compatible platforms
  • The EMI Repository must contain always non conflicting packages and only packages that are not commonly available on standard OS repositories or other mainstream repositories such as Fedora or EPEL

EMI-0 Specific Guidelines

  • The starting platform will be Scientific Linux 5 x86_64 with the following enabled YUM repositories: SL5 X86_64, EPEL 5 x86_64, glite 3.2 X86_64
  • A new ETICS platform and Worker Node is available: "sl5_x86_64_gcc412EPEL" which will have all the RPMs required to build the whole EMI distribution already installed in the VM image. Full RPM list: EMI_SL5_x86_64_EPEL_rpmlist.txt
  • A new EMI project is available where all the EMI software will be built. All the EMI configurations required to build will be part of this project.
  • Configurations already existing in other parts of the system will be cloned and moved in the EMI project if needed
  • Product Teams which are interested in re-organizing subsystems or components are given the opportunity to change their software structure in ETICS. Please contact the ETICS Support for this.
  • ETICS dependencies are currently not explicitly required for EMI-0 because all the dependencies are already installed in the VM. They will be needed for EMI-1. If PTs want to add them also in EMI-0, then it will be one thing less to do for EMI-1.
  • It is not necessary to remove dependencies if a configuration already has them since they will be soon required again.
  • All the build configuration guidelines present in the "Configuration & Integration Guidelines" already apply for EMI-0.
  • No .DEFAULT properties are allowed in component or subsystem configurations. .DEFAULT properties are only present in the project configuration.
  • No STATIC dependencies allowed unless requested to the EMT upon emergency and approved.
  • Until a new version of the ETICS client, currently available version = 1.4.14-1, is released, EMI builds must be executed with this line in the "Append Requirements" field when submitting a build:
client_release=1.4.15-1_EMI_fixes@default
  • ETICS project-configuration to be used for EMI-0 is "emi_R_0"

Hints on using globus default variables & pkg-config (EMT phone-conf 15.11.2010)

  • Owen:
dcapBuildPackage_EPEL_i386_CO_5.4
./configure --with-globus-lib=/usr/lib/ --with-globus-include=/usr/lib/globus/include/ 
dcapBuildPackage_EPEL_x64_CO_5.4
./configure --with-globus-lib=/usr/lib64/ --with-globus-include=/usr/lib64/globus/include/ 
dcapBuildPackage_VDT_i386_SL_5.4
./configure --with-globus-include=/opt/globus/include/gcc32/ --with-globus-lib=/opt/globus/lib/ 
dcapBuildPackage_VDT_x64_SL_5.4
./configure --with-globus-include=/opt/globus/include/gcc64/ --with-globus-lib=/opt/globus/lib/

  • Mattias:
dcap EPEL specfile:
%configure \ --with-globus-include="%{_includedir}/globus -I%{_libdir}/globus/include" \ --with-globus-lib=/dummy 

  • Valery:
    • you can use pkg-config in your configure.ac too: have a look at the PKG_CHECK_MODULES(GLOBUS_GSI, ...) in configure.ac

Example for the WA use:

etics -checkout --ignorelocking --config "your_configuration" --project "emi" --project-config "emi_R_0" --runtimedeps --forcecheckout --verbose --continueonerror
etics -build  --config "emi-glite-apel_R_3_2_5_1" --verbose --target postpublish --continueonerror
platform:   sl5_x86_64_gcc412EPEL
client version:   1.4.15-1_EMI_fixes@default

Example using the etics-client:

  1. Download the new etics-client version with fixes for EMI:
    python etics-client-setup -r 1.4.15-1_EMI_fixes --volatile default
  1. Check out a specific configuration:
    etics-checkout --c <component_configuration_name> --project-config <project_configuration_name> <component_name>
  1. Locally build the code:
    etics-build --platform sl5_x86_64_gcc412EPEL <component_name> 

ETICS-configuration schedule

  • Sequence of ETICS_configurations steps:
    1. VOMS:
      • glite-security-voms-[api|api-c|api-cpp|api-clients]
      • "old" subsystem org.glite.voms
    2. CESNETSecurity:
      • gridsite-[shared|apache], glite-security-gsoap-plugin, glite-security-gss.
      • "old" subsystems: org.glite.security, org.gridsite
    3. gLiteSecurity:
      • glite-security-[lcas|lcmaps] glite-security-[trustmanager|util-java]
      • "old" subsystem org.glite.security
    4. gLiteInformationSystem:
      • bdii, glue-schema, glite-info-[templates,generic], glite-info-provider-[release|service|ldap]
      • "old" subsystems org.glite.bdii, org.glite.info
    5. DataManagement
      • "old" subsystem org.glite.lcg, org.glite.data, org.glite.lcgdm
    6. gLiteL&B, DGAS, APEL, ARGUS
    7. gLiteJobManagement (CREAM+WMS), StoRM

  • Timelines, discussed on EMT-meeting - 27.09.2010:

Task Comments Due Date
1), 2), 3), 4) in parallel + UNICORE Tuesday (28 Sept)
dCache start as soon as 4) is ready 1 Oct
5), 6) in parallel, start as soon as ( 1),2), 3), 4)) are ready Thursday (30 Sept)
ARC as soon as 1)& 5) are ready 1 Oct
7) start as soon as ( 5), 6) ) are ready 1 Oct

EMI-0 Build status

PT EMI-Subsystem ETICS Components Priority Status Worklog
VOMS emi.voms org.gltie.security.voms-api-cpp
org.glite.security.voms-clients
org.glite.security.voms-api
org.glite.security.voms-server
org.glite.security.voms-api-java
EMI.voms.javadoc
Very High
Yes / Done see VOMS - changes for EMI - important for PTs dependng on voms
see conf.nams <=> rpm in EPEL
IMPORTANT - org.glite.security.voms-api-c was replaced/included in/by org.glite.security.voms-api-cpp, details on links above
15.11.2010 - ALL voms component build, except voms-mysql (to investigate)
22.11.2010 - ALL components build
CESNETSecurity emi.gridsite
emi.jobid
emi.px
-
Very High
Warning, important communicated the subsystem/components/configs to clone
22.10.2010 - configurations imported
10.11.2010 - have to solve globus dependencies
15.10.2010 - ALL gridsite builds, except gridsite-core (globus problems); ALL jobid components fail; ALL px components fail
30.11.2010 - emi.gridsite - OK, emi.jobid - OK, emi-px - requested myproxy in externals (!)
06.12.2010 - emi.px - In progress, will be done soon
gLiteSecurity emi.delegation
emi.hydra
emi.java-security
emi.sac
-
Very High
Stop reorganization of the subsystem
22.10.2010 - configurations imported
08.10.2010 - correction required on voms-api-c dependency
15.11.2010 - builds only delegation-interface; hydra.ssss - errors on vdt.globus; emi.sac - 8 (of 30) build, the rest problems with globus & m4
gLiteInformationSystem emi.bdii -
Very High
Yes / Done communicated list of subsyst/compon/configs to clone ,
22.10.2010 - configurations imported
10.11.2010 - CVS to SVN migration - for which "Failed" emi.bdii.glite-info-provider-service, emi.bdii.glite-info-provider-release, emi.bdii.yaim-bdii, emi.bdii.glite-info-plugin-fcr
14.11.2010 - ALL components in emi.bdii are building
UNICORE Target System Access emi.unicore
  • unicore-unicorex
Normal
Yes / Done building
UNICORE Web Service Interfaces emi.unicore
  • unicore-unicorex
  • unicore-registry
Normal
Yes / Done building
UNICORE Client and APIs emi.unicore
  • unicore-ucc
  • unicore-hila-unicore6
  • unicore-hila-shell
Normal
Yes / Done building
UNICORE Service Hosting emi.unicore
  • unicore-unicorex
Normal
Yes / Done building
UNICORE Security emi.unicore
  • unicore-gateway
  • unicore-xuudb
Normal
Yes / Done 29.11.2010 - problems with gateway, caused by the specific WN were build was run
dCache emi.dcache -
Normal
Yes / Done communicated the subsystem/components/configs to clone
22.10.2010 - configurations imported
10.11.2010 corrected configuration-names, updated emi-dcache_B_HEAD, errors to correct for emi.dcache.srmclient
15.11.2010 - emi-dcache-srmclient_R_HEAD error on check-out command
30.11.2010 - dcache-server builds fail because of unsatisfied dependencies on SunJDK
09.12.2010 - ALL components build Action #18383
DataManagement emi.lcgdm
emi.lcgutil
emi.fts
-
HIGH
Stop communicated list of subsyst/compon/configs to clone
22.10.2010 - configurations imported
15.11.2010 - lfc-devel builds, but there are others failing (18 from 46 lcgdm)
ARC emi.arc arc0
arc1
Normal
Yes / Done test-builds with problem on dependencies on voms
10.11.2010 - problems with lfc, lfc-devel because of missing globus dependencies
15.11.2010 - arc0, arc1 build
gLite L&B emi.lb
emi.lbjp-common
-
Normal
Stop communicated the subsystem/components/configs to clone
22.10.2010 - configurations imported
15.11.2010 - have to correct globus dependencies
30.11.2010 - emi.lbjp-common - waiting for postgresql-devel, emi.lb - in progress
06.12.2010 - emi.lbjp-common - OK; emi.lb - mostly OK, emi.lb.server failing on emi.sac.lcas
DGAS emi.dgas -
Normal
Yes / Done 11.10.2010 - configurations present, have to update globus dependencies
30.11.2010 - ALL components build
APEL emi.apel -
Normal
Yes / Done communicated the subsystem/components/configs to clone
22.10.2010 - configurations imported
11.10.2010 - have to correct build commands, problems with java
08.12.2010 - All APEL modules built successfully in the NB Action #18382
ARGUS emi.argus -
Normal
Yes / Done configurations will be done by PT
test-builds - problems with globus 11.10.2010 - no problems
14.11.2010 components build
gLite JobManag emi.cream-ce
emi.cream-ui
emi.wms
emi.wms-ui
-
Normal
Stop Working ETICS Configurations
errors on building components:
MPI emi.mpi -
Normal
Yes / Done configurations will be done by PT
test-builds
15.11.2010 - ALL components build
StoRM emi.storm storm-checksum
storm-common
storm-.dynamic-info-provider
glite-yaim-storm
storm-backend
storm-frontend
storm-client
storm-gridftp-dsi
Normal
Stop 15.11.2010 - some components don't have dependencies
Miscellaneous Components emi.misc glite-wms-utils-classad
cleanup-grid-accounts
edg-mkgridmap
glite-info-generic
glite-info-templates
glite-version
emi.glite.build-common-cpp
emi-jobman-exception
lcg-expiregridmapdir
lcg-service-proxy
- Yes / Done components that not have "other" etics-subsystem
15.11.2010 - ALL components build, except emi.misc.classad-utils

EMI-0 nightly builds
The EMI-0 automatic builds - executing three times per day at the following time: 12:00, 16:00, 22:00

Problems encountered

  1. change of VCS checkout command from:
checkout:   cvs -d ${vcsroot} co -r ${tag} ${moduleName}
to
checkout:   cvs -d ${vcsroot} co -r ${tag} -d ${moduleName} org.glite.apel.core

ETICS-responsible persons & external-dependencies lists

  • ETICS-resposible persons

Name Component/PT ETICS-subsystems
Cristina Aiftimiei glite-yaim-core emi.yaim
emi.misc
ALL
André Giesler
Bjorn Hagemeier
UNICORE (all components) emi.unicore
Martin Savko
Jozef Cernak
Marek Kocan
ARC (all components) emi.arc
Sunil Ahn AMGA emi.amga
Cristina del Cano Novales APEL emi.apel
Valery Tschopp Argus emi.argus
Pedro Andrade
Oliver Keeble
DataManagement emi.fts
emi.lcgdm
emi.lcgutil
Owen Synge
Jan Schaefer
Christian Bernardt
dCache emi.dcache
Andrea Guarise DGAS Client emi.dgas
Paolo Andreeto
Massimo Sgaravatto
gLite Job Management emi.cream-ce
emi.cream-ui
emi.jdl
emi.misc
emi.wms
emi.wms-ui
Pedro Andrade GLiteInformationProvider emi.bdii
emi.misc
Enol Fernández gLite MPI emi.mpi
Oscar Koeroo
Joni Hahkala
Mischa Salle
Dennis van Dok
GLiteSecurity emi.sac
emi.java-security
emi.delegation
emi.hydra
Elisabetta Ronchieri StoRM emi.storm
Vincenzo Ciaschini VOMS emi.voms
Zdenek Sustr
Frantisek Dvorak
gLite L&B
CESNET Security
emi.gridsite
emi.jobid
emi.lb
emi.lbjp-common
emi.px

  • List of dependencies that are not in the OS; different versions that the one provided by OS

PT/Component dependencies-list
ARC ArcDependencies
AMGA AMGA
Argus ArgusDependenciesList
APEL APELClient
DataManagement DMDependenciesList
dCache DCacheDependencies
DGAS alligned with OS
CESNET Security CesnetSecurity
gLite Information System openldap, openldap-servers
gLite Job Management JobManPTExternalDeps
gLite L&B gLite L&B
gLite MPI GLiteMPI
gLite Security  
StoRM StoRMPTExternalDeps
VOMS VOMSExternalDependencies
UNICORE Java >= 1.6 (Sun or OpenJDK)

End of Dec, 2010 - packages ready in repository

  • List of components and their build status XLS, PDF

Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt list_externals_names_64_v7.txt r1 manage 1.4 K 2010-11-01 - 13:40 DoinaCristinaAiftimiei list of external dependencies names available on SL5/EPEL (x86_64) repos
Texttxt list_externals_names_debian_lenny_v2.txt r1 manage 1.4 K 2010-11-01 - 13:48 DoinaCristinaAiftimiei list of external dependencies names available on Debian5-stable (Lenny) repos (after Vincenzo's comments)
Texttxt list_externals_names_debian_squeeze_sid.txt r1 manage 0.6 K 2010-11-01 - 13:49 DoinaCristinaAiftimiei list of external dependencies names available on Debian5-testing (Squeeze) repos
PDFpdf ms18_tabel.pdf r1 manage 39.1 K 2011-01-21 - 23:45 DoinaCristinaAiftimiei List of components and their build status
Microsoft Excel Spreadsheetxls ms18_tabel.xls r1 manage 23.5 K 2011-01-21 - 23:44 DoinaCristinaAiftimiei List of components and their build status
Edit | Attach | Watch | Print version | History: r61 < r60 < r59 < r58 < r57 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r61 - 2011-05-26 - DoinaCristinaAiftimiei
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback