Please find below my review assessment of the DJRA1.6.1 "Integration Work Plan and Status Report" deliverable. I am sending the review as plain text.
regards,
Balazs
JRA1.6.1 "Integration Work Plan and Status Report
Reviewed document: DJRA1.6.1, date 28/10/2010, v4, pdf file
Review date: 29/10/2010
Reviewer: Balazs Konya, LU
General comments
- I accept the deliverable if a major revision is prepared. The revised version should move away from the current procedures, organizations, subtasks, tools, etc.., centric presentation and instead put more focus on describing
Morris: Major revision done, taking the following inputs into account:
a) current status of integration and interoperability of the EMI software components
Morris: This has been put as the focus (re: state-of-the-art chapter, also noted from Maria)
b) concrete objectives and actions for the integration and interoperability activity on a component level: two examples "interop testing of X and Y components wrt Z interface specification", or "integration of X, Y, Z components".
Morris: Within the agreed timeline of year 1 I have added a plan chapter (re: also stated by Maria), although as a matter of fact not many integration work is to be done between state-of-the-art, EMI-0 and EMI-1.
André: Major revision done by considering deliverable template from PO. Focus is now on intergation tasks concerning EMI-1 release including schedules and work plans. State-of-the-art is summarized in chapter 3.
- The document does not address the current integration status. The status report part is completely missing. The only integration/interop status related information i could find was the reference to the wiki matrix.
Morris: After discussions with Balazs I put the important content of the matrices into the document.
André: Status report is summarized in chapter 3. It reflects the work of the JRA1.7 subtask before EMI-1 and integration tasks that have been performed in the time before EMI or for the exercise release EMI-0. The chapter contains a snapshot of the matrix. However, status report is short, since there were no integration tasks done in EMI-0 and the matrices provide information about integration and interop relations before EMI. The next JRA1.7 deliverable will then consider EMI-1 as the current development cycle in its status report, whereas the following chapters will then concentrate on EMI-2.
- The current document does not read as a workplan, it resembles more like an internal WP (this case for the JRA1.7 task) organizational document.
Morris: as stated above the document was changed away from an organizational document towards a work plan
André: This comment has been adressed by the given template structure.
- The document is not an easy read and it repeats similar texts several times. Here is an example sentence that i could not really decode (page 4):
"This document in particular describes in these tools in the light of identifying those EMI components that are of relevance for this activity."
Morris: Section dropped after discussions with Balazs and Maria
André: same for the revised deliverable
- When it comes to concrete statements, e.g Section 7.1 "Standard Compliance tests" or Section 8.1 "Use Cases" the document is very incomplete or even controversial (see details later).
Morris: Part dropped after discussions with Balazs and Morris. After discussion with Maria, we dropped the interoperability part because the document is only an 'integration plan'
André: Interoperability is still part of the deliverable, since it is given by the template structure.Some sections provide general informations about interoperability. However, there are no concrete interoperability plans for EMI-1. However, interoperability is an issue in some integration tasks for EMI-1 since the integration of GLUE schemas for instance is as well interoperability between EMi components by using GLUE.
- The workplan is very unspecific when it comes to identifying who would actually do the work (partners, PTs). The only (strange) exception is when it refers to external collaborations under use case section. On the other hand, the workplan formulation in terms of sub-tasks, without specifying partners behind, looks like an overkill.
Morris: The state-of-the-art as well as the plan are organized along the product teams involved now clarifying inherently who does the work.
André: The deliverable tries to point out who is doing the individual work. For instance, that the appropriate PTs have to provide a Software Validation and Test Plan and unit tests as a prerequisite before the actual integration testing can be started. It is also shown what JRA1.7 should do in particular, e.g.providing integration test plans or collaborate with SA2.4 in order to perform tests in the EMI testbed. Chapter 7 provides a diagram about the relations.
- By reading the document i developed a strange feeling that the document was written by only one partner and it is not reflecting the integration plans of EMI PTs because the other partners or PTs might have not been properly included in the integration plan preparation. Please make sure that the middleware consortia through their PTs will be directly involved in the preparation of the next version and not just via providing input through the famous matrices that were found controversial by many PT leaders.
Morris: Each PT has been involved in the matrix process and now also with respect to timeline discussions. In addition, the version 4 of this document has been circulated on the JRA1.7 integration mailing list getting no commons probably due to the PT concept feeling not responsible for any work in tasks.
André: nothing to add
- The document refers to the "matrices" and puts those into the center of the work as the key tool for most of the subtasks and base for all concrete work. Nevertheless, the matrices are not attached to the document.
Morris: As discussed with Balazs the matrices will be not put at an annex (will be a book then) - but the content has been put into the document addressing the integration work (re: maria integration plan, not interoperability)
André: Yes, the matrices are not in the annex, since they are just too big. Only a snapshot is included in chapter 3. The matrices will also be revised in the next weeks. The main focus of JRA1.7 in the time after Prague was to write the deliverable and to gather schedule informations concerning EMI-1 from the product teams.
- I am not sure that the given definition of "integration" is shared by EMI partners and other EMI workpackages.
Morris: The integration term has been discussed in several bodies and the definition is now in-line with the overall project work.
André: Chapter 5 describes the aspects of integration and testing strategies. The structure was given by the template.
- All over the document interoperability is defined as "standards-based" interoperability. This restrictive definition may not satisfy EMI needs. I can easily envision interoperability of several EMI components based on "EMI agreements" or "de facto standards". I don't remember that the restricted definition of interoperability used and assumed in the deliverable has actually been discussed or agreed within EMI. In any case, the integration task workplan should cover non-standard based interoperability work as well.
Morris: as discussed with Maria and as a result of our name of the deliverable, we have removed the interoperability work.
André: As stated above it is not completely removed. However, the document is not talking about "de facto standards" anymore but is listing some typical agreed standard interfaces and schemas (OGSA-BES, SRM, IPV6, ...) which will be used in EMI (chapter 5.3 and 5.4).
- The selected initial use cases to validate interoperability and integration of EMI components are rather questionable. Was there really an agreement to use HPC-BP (GIN),
SAGA and EUFORIA fusion application as a test use case? From the EMI tech development point of view these use cases are not really relevant.
Morris: According to experience, the project has no FTEs left to work on this and as such we dropped it as an activity of JRA1.7.
André: Nothing to add
Specific comments
In general, the document needs a major revision, that would touch most of the sections, nevertheless i collected numerous concrete comments here.
Morris: as written above - major revision done.
André: Given template
- Document organization: please move Term Classification section right after the Executive summary (swap Section 3 and 4).
Morris: After discussions dropped completely.
André: Obsolete because of new template.
- Executive summary: the current lengthy text does not read as an execution summary. It reads like a WP description since most of the text describes the subtasks.
Morris: Since the sub-task description is everywhere at least the summary should state how we drive the efforts in the task including the PT overlay.
André: Obsolete because of new template.
- Page 9, "Addressing technical objectives" presents the subtask structure of the JRA1.7 tasks and relates those to some project objectives. I wouldn't mind this section being removed from a workplan document. If you decide to keep, please at least rename it as "Integration subtasks" or something similar.
And don't start the first sentence as "This overal task aims .." (which overal task?).
Morris: section removed.
André: Obsolete because of new template.
- Term clarification (starting from page 10): As mentioned above, i find the used definition of interoperability too restrictive. For me interop is not necessarily open community standard based. The definition of integration is even more unclear to me.
Morrris: section dropped.
André: As described above. A short definition with a figure example is still in the document.
- Interop example on OGSA-BES (page 11). This section is completely irrelevant for a workplan document. Please remove it. side note: the ARC related statemets are not 100% correct either.
Morris: section dropped
André: I missed to consider this note, and included the OGSA-BES example again in the document. I wanted to give a short introduction to interoperability between EMi components by using this example. My intention was to provide an understanding of interoperability to the reader. If such definitions are not required in the document I will remove it again. Maybe, its better to change then the template strucure and to combine then section 5.3 (interoperability testing) and 5.4 (compliance testing) since they are in principle the same.
- Integration example (page 13). I'd remove this as well. This does not belong to a workplan document.
Morris: section dropped
André: Obsolete because of new template.
- Streamlined component set releases (page 14): The section introduces the "JRA1 release preparation" activity. It is a kind of small "EMT" or SA1.2 within JRA1. The section is completely vague when it comes to describing how, by whom (PTs, or dedicated team?) and according to what schedule the JRA1 release team will deliver the tested/integrated "JRA1 components" to SA1.
Morris: expected milestone from JRA1 and thus of importance for this activity, but can be also dropped if needed
André: Obsolete because of new template.
- Section 6.1 "Relevant areas work for component integration" (page 16). This section presents the concrete integration workplan. It suggests integration work and testing of components that are already integrated. For example, it wants to validate/test how the ARC CE internal components are integrated. Why to do that? I find the presented integration work items more than questionable.
Morris: Section has been dropped
André: Obsolete because of new template. Chapter 6 is now presenting the concrete work plan for EMI-1.
- I think the confusion originates from the used definition of integration.
Morris: Discussion at AHM to finally clarify among SA1, JRA1
André: Obsolete because of new template.
Section 6.1 needs a complete rethinking.
Morris: done.
André: Obsolete because of new template.
- Section 7.1 "Existing standard compliance checks" (page 19). Presents the concrete workplan for interoperability tests. First of all, not only STD-based interoperability of EMI components should be tested. Then, when it comes to standard-based interop validation i find the planned work incomplete. For example, it picks OGSA-BES-JSDL (btw, to be called HPC-BP), SRM, IPV6 and LDAP GLUE only. It misses entire X509 area, SAML, gridftp, GLUE-XML, just to name a few. Then, even within a specific standard verification it incompletely lists the components related (e.g. no ARC components listed for LDAP-GLUE and not clear which GLUE version is concerned). All in all, this section is rather incomplete.
Morris: interoperability dropped in scope
André: Obsolete because of new template. Chapter 6 lists the relevant integration, interoperability, and standard tests concerning EMI-1 release in detail.
- Section 8.1 Use cases (for integration tests) page 21. This section identifies the following three high-level use cases to be used for integration/interop testing of EMI software stack: GIN (basically HPC-BP), EUFORIA fusion application,
SAGA. Although these might have some relevance for some EMI components, the three use cases are not the best to perform complex testing of EMI (e.g. HPC-BP is applicable only for "hello grid" demos, while
SAGA support within EMI components is minimal).
Morris: use cases section dropped because of lack of manpower to follow scientific use cases.
André: nothing to add