TD Deliverable Review Form

Please find below my review assessment of the DJRA1.6.1 "Integration Work Plan and Status Report" deliverable. I am sending the review as plain text.

regards, Balazs

Reviewed document: DJRA1.6.1, date 28/10/2010, v4, pdf file

Review date: 29/10/2010

Reviewer: Balazs Konya, LU

General comments

- I accept the deliverable if a major revision is prepared. The revised version should move away from the current procedures, organizations, subtasks, tools, etc.., centric presentation and instead put more focus on describing

Morris: Major revision done, taking the following inputs into account:

a) current status of integration and interoperability of the EMI software components

Morris: This has been put as the focus (re: state-of-the-art chapter, also noted from Maria)

b) concrete objectives and actions for the integration and interoperability activity on a component level: two examples "interop testing of X and Y components wrt Z interface specification", or "integration of X, Y, Z components".

Morris: Within the agreed timeline of year 1 I have added a plan chapter (re: also stated by Maria), although as a matter of fact not many integration work is to be done between state-of-the-art, EMI-0 and EMI-1.

André: Major revision done by considering deliverable template from PO. Focus is now on intergation tasks concerning EMI-1 release including schedules and work plans. State-of-the-art is summarized in chapter 3.

- The document does not address the current integration status. The status report part is completely missing. The only integration/interop status related information i could find was the reference to the wiki matrix.

Morris: After discussions with Balazs I put the important content of the matrices into the document.

André: Status report is summarized in chapter 3. It reflects the work of the JRA1.7 subtask before EMI-1 and integration tasks that have been performed in the time before EMI or for the exercise release EMI-0. The chapter contains a snapshot of the matrix. However, status report is short, since there were no integration tasks done in EMI-0 and the matrices provide information about integration and interop relations before EMI. The next JRA1.7 deliverable will then consider EMI-1 as the current development cycle in its status report, whereas the following chapters will then concentrate on EMI-2.

- The current document does not read as a workplan, it resembles more like an internal WP (this case for the JRA1.7 task) organizational document.

Morris: as stated above the document was changed away from an organizational document towards a work plan

André: This comment has been adressed by the given template structure.

- The document is not an easy read and it repeats similar texts several times. Here is an example sentence that i could not really decode (page 4): "This document in particular describes in these tools in the light of identifying those EMI components that are of relevance for this activity."

Morris: Section dropped after discussions with Balazs and Maria

André: same for the revised deliverable

- When it comes to concrete statements, e.g Section 7.1 "Standard Compliance tests" or Section 8.1 "Use Cases" the document is very incomplete or even controversial (see details later).

Morris: Part dropped after discussions with Balazs and Morris. After discussion with Maria, we dropped the interoperability part because the document is only an 'integration plan'

André: Interoperability is still part of the deliverable, since it is given by the template structure.Some sections provide general informations about interoperability. However, there are no concrete interoperability plans for EMI-1. However, interoperability is an issue in some integration tasks for EMI-1 since the integration of GLUE schemas for instance is as well interoperability between EMi components by using GLUE.

- The workplan is very unspecific when it comes to identifying who would actually do the work (partners, PTs). The only (strange) exception is when it refers to external collaborations under use case section. On the other hand, the workplan formulation in terms of sub-tasks, without specifying partners behind, looks like an overkill.

Morris: The state-of-the-art as well as the plan are organized along the product teams involved now clarifying inherently who does the work.

André: The deliverable tries to point out who is doing the individual work. For instance, that the appropriate PTs have to provide a Software Validation and Test Plan and unit tests as a prerequisite before the actual integration testing can be started. It is also shown what JRA1.7 should do in particular, e.g.providing integration test plans or collaborate with SA2.4 in order to perform tests in the EMI testbed. Chapter 7 provides a diagram about the relations.

- By reading the document i developed a strange feeling that the document was written by only one partner and it is not reflecting the integration plans of EMI PTs because the other partners or PTs might have not been properly included in the integration plan preparation. Please make sure that the middleware consortia through their PTs will be directly involved in the preparation of the next version and not just via providing input through the famous matrices that were found controversial by many PT leaders.

Morris: Each PT has been involved in the matrix process and now also with respect to timeline discussions. In addition, the version 4 of this document has been circulated on the JRA1.7 integration mailing list getting no commons probably due to the PT concept feeling not responsible for any work in tasks.

André: nothing to add

- The document refers to the "matrices" and puts those into the center of the work as the key tool for most of the subtasks and base for all concrete work. Nevertheless, the matrices are not attached to the document.

Morris: As discussed with Balazs the matrices will be not put at an annex (will be a book then) - but the content has been put into the document addressing the integration work (re: maria integration plan, not interoperability)

André: Yes, the matrices are not in the annex, since they are just too big. Only a snapshot is included in chapter 3. The matrices will also be revised in the next weeks. The main focus of JRA1.7 in the time after Prague was to write the deliverable and to gather schedule informations concerning EMI-1 from the product teams.

- I am not sure that the given definition of "integration" is shared by EMI partners and other EMI workpackages.

Morris: The integration term has been discussed in several bodies and the definition is now in-line with the overall project work.

André: Chapter 5 describes the aspects of integration and testing strategies. The structure was given by the template.

- All over the document interoperability is defined as "standards-based" interoperability. This restrictive definition may not satisfy EMI needs. I can easily envision interoperability of several EMI components based on "EMI agreements" or "de facto standards". I don't remember that the restricted definition of interoperability used and assumed in the deliverable has actually been discussed or agreed within EMI. In any case, the integration task workplan should cover non-standard based interoperability work as well.

Morris: as discussed with Maria and as a result of our name of the deliverable, we have removed the interoperability work.

André: As stated above it is not completely removed. However, the document is not talking about "de facto standards" anymore but is listing some typical agreed standard interfaces and schemas (OGSA-BES, SRM, IPV6, ...) which will be used in EMI (chapter 5.3 and 5.4).

- The selected initial use cases to validate interoperability and integration of EMI components are rather questionable. Was there really an agreement to use HPC-BP (GIN), SAGA and EUFORIA fusion application as a test use case? From the EMI tech development point of view these use cases are not really relevant.

Morris: According to experience, the project has no FTEs left to work on this and as such we dropped it as an activity of JRA1.7.

André: Nothing to add

Specific comments

In general, the document needs a major revision, that would touch most of the sections, nevertheless i collected numerous concrete comments here.

Morris: as written above - major revision done.

André: Given template

- Document organization: please move Term Classification section right after the Executive summary (swap Section 3 and 4).

Morris: After discussions dropped completely.

André: Obsolete because of new template.

- Executive summary: the current lengthy text does not read as an execution summary. It reads like a WP description since most of the text describes the subtasks.

Morris: Since the sub-task description is everywhere at least the summary should state how we drive the efforts in the task including the PT overlay.

André: Obsolete because of new template.

- Page 9, "Addressing technical objectives" presents the subtask structure of the JRA1.7 tasks and relates those to some project objectives. I wouldn't mind this section being removed from a workplan document. If you decide to keep, please at least rename it as "Integration subtasks" or something similar. And don't start the first sentence as "This overal task aims .." (which overal task?).

Morris: section removed.

André: Obsolete because of new template.

- Term clarification (starting from page 10): As mentioned above, i find the used definition of interoperability too restrictive. For me interop is not necessarily open community standard based. The definition of integration is even more unclear to me.

Morrris: section dropped.

André: As described above. A short definition with a figure example is still in the document.

- Interop example on OGSA-BES (page 11). This section is completely irrelevant for a workplan document. Please remove it. side note: the ARC related statemets are not 100% correct either.

Morris: section dropped

André: I missed to consider this note, and included the OGSA-BES example again in the document. I wanted to give a short introduction to interoperability between EMi components by using this example. My intention was to provide an understanding of interoperability to the reader. If such definitions are not required in the document I will remove it again. Maybe, its better to change then the template strucure and to combine then section 5.3 (interoperability testing) and 5.4 (compliance testing) since they are in principle the same.

- Integration example (page 13). I'd remove this as well. This does not belong to a workplan document.

Morris: section dropped

André: Obsolete because of new template.

- Streamlined component set releases (page 14): The section introduces the "JRA1 release preparation" activity. It is a kind of small "EMT" or SA1.2 within JRA1. The section is completely vague when it comes to describing how, by whom (PTs, or dedicated team?) and according to what schedule the JRA1 release team will deliver the tested/integrated "JRA1 components" to SA1.

Morris: expected milestone from JRA1 and thus of importance for this activity, but can be also dropped if needed

André: Obsolete because of new template.

- Section 6.1 "Relevant areas work for component integration" (page 16). This section presents the concrete integration workplan. It suggests integration work and testing of components that are already integrated. For example, it wants to validate/test how the ARC CE internal components are integrated. Why to do that? I find the presented integration work items more than questionable.

Morris: Section has been dropped

André: Obsolete because of new template. Chapter 6 is now presenting the concrete work plan for EMI-1.

- I think the confusion originates from the used definition of integration.

Morris: Discussion at AHM to finally clarify among SA1, JRA1

André: Obsolete because of new template.

Section 6.1 needs a complete rethinking.

Morris: done.

André: Obsolete because of new template.

- Section 7.1 "Existing standard compliance checks" (page 19). Presents the concrete workplan for interoperability tests. First of all, not only STD-based interoperability of EMI components should be tested. Then, when it comes to standard-based interop validation i find the planned work incomplete. For example, it picks OGSA-BES-JSDL (btw, to be called HPC-BP), SRM, IPV6 and LDAP GLUE only. It misses entire X509 area, SAML, gridftp, GLUE-XML, just to name a few. Then, even within a specific standard verification it incompletely lists the components related (e.g. no ARC components listed for LDAP-GLUE and not clear which GLUE version is concerned). All in all, this section is rather incomplete.

Morris: interoperability dropped in scope

André: Obsolete because of new template. Chapter 6 lists the relevant integration, interoperability, and standard tests concerning EMI-1 release in detail.

- Section 8.1 Use cases (for integration tests) page 21. This section identifies the following three high-level use cases to be used for integration/interop testing of EMI software stack: GIN (basically HPC-BP), EUFORIA fusion application, SAGA. Although these might have some relevance for some EMI components, the three use cases are not the best to perform complex testing of EMI (e.g. HPC-BP is applicable only for "hello grid" demos, while SAGA support within EMI components is minimal).

Morris: use cases section dropped because of lack of manpower to follow scientific use cases.

André: nothing to add

Please find below my (long) review assessment of the major revision of the DJRA1.6.1 "Integration Work Plan and Status Report" deliverable. I am sending the review as plain text.

The most important outcome of my review is that

"i propose to RELEASE the document to Brussels in its current or slightly improved form without too much further delay"

regards, Balazs

ps: i apologize for my delayed review. there was always something more urgent

Reviewed document: DJRA1.6.1, v0.5, pdf file generated by project office (https://twiki.cern.ch/twiki/pub/EMI/DeliverableDJRA161/EMI_Integration_Plan-v0.5.pdf)

Review date: 28/01/2011

Reviewer: Balazs Konya, LU

General comments

This is my second review of the "Integration workplan" DJRA1.6.1 deliverable. I concluded my first review as "I accept the deliverable if a major revision is prepared". The new version i reviewed is indeed a major revision therefore i took the time to read the entire document from cover to cover.

According to my understanding the purpose of this document is twofold:

a) capture and present the current integration status of EMI components, as of "EMI-0". Describe the "state of art of integration of EMI components", set a reference point so that further EMI releases could be compared against the initial status.

b) serve as a workplan, contain a "list of tasks, assignments" for EMI product teams and for the integration task personnel itself, so that in the project everybody would cleary know which integration tasks must be carried out by which dates in order to reach the EMI-1 release.

The reviewed revised version, in my humble opinion, unfortunately fails to achieve any of these:

- The "plan" part of the document can not be used as a plan, something you would give to PT leaders or T7.1 task member and they would know what to do. It is still not a workplan. A PT leader, after investing the time necessary to read through the 43 pages, will not have a clue about "what's next, who is doing what?". To be a useful workplan document, a better structured, more concrete, much shorter, more down to-the-point document should have been produced.

- The one and half page status report section (Section 3 Status Report) literary says nothing, not a single word about status. It gives (btw confusing) definitions of some terms, contains a fully irrelevant paragraph and includes a snapshot of a fraction of the integration matrix. A person reading the status section will learn nothing about current integration status of EMI. From this aspect, the document can't be given to our customers (e.g. EGI or user communities).

Generally, the integration plan document is still a bloated, rather non-specific questionable quality "plan" document. The information content density is very low, difficult to find and capture relevant important bits. The 43 pages contain lots of irrelevant, unnecessary, unrelated texts and tables. At numerous places i could not figure out why that certain paragraph appeared there, what that one had to do with the rest of the section or why was that at all included in a integration plan. The document is full of unnecessary repetitions and duplications of the same information, sometimes even same text. A simpler, shorter document with more concrete tasks would have been better.

All in all, it is unclear who would benefit from this document, who could use this workplan as part of their development activity or information source (state of art). I don't see the usefulness of the current version of the document apart from being one of those mandatory paperwork EMI must send to Brussels.

Nevertheless, after lots of thinking, i propose to RELEASE the document to Brussels in its current or slightly improved form without too much further delay. Why?

a) Because the main purpose of the deliverable was to serve as an integration plan for EMI-1 development. A plan, that was supposed to be used during the process of preparing EMI-1. The current version of the document is not such a plan. This "plan" has not been used and clearly will not be used for EMI-1 preparation regardless how many times it is going to be rewritten. Spending further time and efforts on a never-to-be-used plan makes no sense. Sending the EU commission an EMI-1 integration plan dated after or close to the EMI-1 code freeze date would put EMI in a strange position at the review: a reviewer may ask why EMI did spend time on preparing a plan that was actually never used?

b) Because, i think EMI has more important things to do than to write the missing integration status report. The integration team, instead of trying to write the missing status document, now should be directed to carry out some integration tests for EMI-1.

c) Because the value of the current document as an EMI-1 integration plan is very little and i don't see any reason to invest more efforts improving it. Instead, i suggest efforts to be invested into actual integration tests and soon preparation of proper EMI-2 integration plan.

For the EMI-2 integration plan, i'd like to see a down-to-the-point, concise, short document, a kind of "task list" that clearly identifies the integration steps and their effort/time plan.

Concrete observations

The following, by far non-exclusive comments underline the general statements made above.

- Minor but rather annoying formatting problem: there were no page numbers in my PDF version. Therefore, i can't refer to specific pages in my comments Another minor thing: wrong document date 31/07/2010

A: corrected

- At several pages references to important documents or tables are not given, only the top-level JRA1.7 (internal) wiki page referenced. Examples: In Section 3 there are no direct references to the famous integration and interop matrices. Or, in Section 4.1 The EMI Technical Development Plan reference is given as a link to the JRA1.7 top wiki page.

A: Added references to the famous matrices in Section 3 and deleted the reference in 4.1

- The document still comes with not-really relevant tables, some of them with rather unclear and messy/wrong content. See e.g. the Table in Section 4.3.1 that lists some EMI component improvements. Why these components? Why these improvements? For Example, what does dcache's new authenticated web interface have to do with integration plan? or What is the meaning of "nordugridmap for all" improvement of "arc sec utils" component (btw, there is no such component, arc sec utils is a product team name)?

A: Deleted the cross area improvments, since they are not relevant for integration tasks.

- Section 4 is mostly a repetition of the already available information of DNA1.3.1 (high level plan) and/or Technical area workplans. Very little (if nothing) extra information is presented. Very little said about actual integration. Section 4.3 lists some product changes. Unclear, not explained, how are these related to better integration of EMI components. The entire section reads like a (partial) implementation plan of some components for EMI-1 release. The integration aspect is unclear.

A: Got from template for 4.3 : "This section gives a brief overview of the EMI Products affected by the defined Technical Objectives as described in the overall EMI Software Development and Test Plan focusing on their integration and interoperability aspects." So section 4.3 tried to give an overview which component is integrating what. The detailed information is then given in section 6. For me it's not clear which other integration aspects are required here. In regard to following integration plans it needs to be clarified what exactly is expected in chapter 4. An example would be good to have for JRA1.7 that shows for one integration task the input for chapter 4 and 6 describing what is in detail expected from JRA1.7.

- Section 5 describes some methodology for integration, explains how integration should be done (pre-requisities, testing steps, test report, etc). The information presented there is correct but would rather belong to one of the EMI SA2 guidelines.

A: Without these methodology text it is hardly possible to fill section 5 with text following the targets from the template. So, it would be an option to remove section 5 from the next integration plans and transfer the strategies to some SA2 guidelines.

- Section 5.3 "Interoperability testing" claims that it "describes the scope and goals of the interop activity", i assume for EMI-1. Then it goes on talking about OGSA-BES and JSDL. NOOO. OGSA-BES and JSDL Interoperability testing is NOT within of the scope of EMI-1 preparations. Most of the 5.3 section must be removed, as i requested in my previous review. EMI has no plan to perform any OGSA-BES or JSDL interoperability as part of year1 plan.

Strictly speaking, EMI interoperability tests should cover/validate the implementations of "EMI agreements". For example, once the EMI accounting record is defined, that must be checked. Or once the EMI Execution interface implementations are available, that should be checked.

In general, i miss a very concrete list of interoperability tests JRA1.7 task wants to perform as part of EMI-1 certification. I know one thing for sure: neither BES, nor JSDL belongs to that list. (note: later in the document i found a quite hidden sentence that claims there won't be interoperability tests in EMI-1, see below)

A: Everything concerning JSDL or OGSA was deleted from section 5.3 and appropriate figure. Yes, there is no interop testing in EMI-1. The section just introduces interoperability.

-Section 5.4 "Standards compliance testing". This was the most concrete section of the entire "plan". Here, at least some concrete standards to be checked were identified. Again, OGSA-BES and JSDL should be removed. IPv6 compliance checks, if possible, should be extended to as many EMI components as possible.

A: OGSA section deleted.

Even though i found Section 5.4 as one of the best, i still find it rather incomplete and far being a proper plan. Unclear who will execute what kind of tests (is there a test suit already?) for which components.

- The integration plan is NOT presented in Section 6.1.1 "Integration plan". There is only a statement that says "each PT must develop and record plans for conducting the integration activities...." .. The integration plans should be part of the Software Development Plans..." and the reader is directed to JRA1 SDTP.

- The interoperability plan is NOT presented in Section 6.1.2 "Interoperation plan". There is only a statement that says "each PT must develop and record plans for conducting the interoperability activities...." .. These plans should be part of the Software Development Plans..." Then it says "No interoperability tasks are planned for EMI-1".

Fine, but then why was there a previous section on "5.3 Interoperabilty testing"?

A: The section was given in the template from Alberto. It is also there to give an introduction to the term interoperability. Particularly, we have asked PTs for interoperability plans within the EMI project and the PTs asked us what we actually understand by the term interoperability. True is, there is no interop testing planned for EMI-1. I've no problem to remove it, since interoperability testing and compliance testing are the same in my opinion. Concerning section 5.4, informations about test suites or who will execute tests were not yet available when writing this document.

-The standard compliance testing plan is NOT presented in Section 6.1.3. It only says that these tests are "included in the appropriate integration and testing strategies". Below that, there is a table without any explanation. I can only guess that table defines the standard compliance tests for EMI-1. due dates? test suites? test execution (who?)

A: Added a sentence for describing the table: "The standard compliance testing in EMI-1 is included in the appropriate integration testing plans. The following table lists the compliance testing tasks for EMI-1, affected components, and the associated test plan." Concerning the dates, availabe test suites or who is testing: if not listed in the referenced test plans, those informations were missing when the document was written.

- Section 6.3 is a excellent example of unnecessary duplications.

6.3.{1,2,3,4,5,6} are almost identical sections, line-by-line, annoying copy and paste paper generation! Including a typo that got propagated onto 11 (!) pages:

"The expected availability of the test implementations is: 2010-01-31"

several pages could have been saved by merging unnecessarily multiplicated the info on one single page.

6.3.{7,8} are also identical copy & paste generated pages with the same "2010-01-31" typo

A: Corrected the typo concerning the date. It is true that many information is multiplicated. But also here, I'd like to refer to the input from the template: "This section describes the expected input and output of the testing activity for each Product within the scope of the EMI 1 Integration Plans. The section is organized in subsections, one subsection for each Product. Each subsections contains a description of the integration, interoperability and standard compliance test cases and their implementation. If the information is available in external documents, it can be added by reference. If information is not available from the PT, mark it as expected, but missing." So, I followed the instructions and produced a subsection for every affected component in the integration tasks. Of course, by using copy and paste function. If it is desired, there is no obstacle to compress the information in one table. For this deliverable, I hope it is ok to leave it as it is. However, I deleted all the interoperability sections, since no tests were done in this respect in EMI-1 anyway.

- Section 8 "integration organization and resources"

This is another unnecessary useless page. The entire section is just a reference to SA2 guidelines. I would have either removed this section or at least would have provided a short extract of relevant information contained in the SA2 guidelines. Actually, i am not 100% sure that those SA2 guidelines really cover how integration testing is organized and what efforts are allocated to the task within JRA1.

A: Section 8 was removed.