Deliverable Review Form

Trivial fixes have been included directly in the document with change tracking.

General comments

The document is satisfactory. The remarks in the table below suggest some improvements.

Additional recommendations (not affecting the document content, e.g. recommendation for future work)

None.

Detailed comments on the content

Note 1: The reviewers must list here any observation they want to track explicitly and that require interaction with the authors
Note 2: These comments have to be explicitly addressed by the authors and the action taken must be clearly described

N°	Page	Section	Observations and replies	Is Addressed?
1	7	1.4	The text describing the amendment procedure will be very similar, if not identical, for all deliverables. Why don’t simply reference a document with this description? Probably it would be this document in fact. -- FrancescoGiacomini - 20-Jul-2010 Well, not necessarily. Different documents may have different amendment procedures. For example, this QA plan explicitly references the QA Manager as responsible for the amendments and this doesn't make sense for other documents. The Technical Plans may require a more complex procedure with involvement of external parties. And so on. -- AlbertoDiMeglio - 08-Aug-2010 In EGEE3 we used to write: Amendments, comments and suggestions should be sent to the authors. The procedures documented in the EGEE “Document Management Procedure” will be followed: http://project-EGEE-III-na1-qa.web.cern.ch/project-EGEE-III-NA1-QA/EGEE-III/Procedures/DocManagmtProcedure/DocMngmt.htm and I've never seen anything different. My guess is that authors will simply cut&paste from previous deliverables, probably even from EGEE3, so I keep suggesting that providing some boilerplate for the common case would be useful. Since this text can be included in the template, maybe with a note saying that one is free to change it if the procedure for that specific document ought to be different, this document can in fact ignore the issue. -- FrancescoGiacomini - 13-Aug-2010 I've made the paragraph more generic in the template and added a note that the text can be adapted if needed -- AlbertoDiMeglio - 17-Aug-2010
2	8	1.5	The acronym VV is never used -- FrancescoGiacomini - 20-Jul-2010 Removed -- AlbertoDiMeglio - 08-Aug-2010
3	10	3.1	Is "progress factor" a technical term? is so, it should be explained. -- FrancescoGiacomini - 20-Jul-2010 No, bad phrasing for 'performance indicators'. Replaced with that. -- AlbertoDiMeglio - 08-Aug-2010
4	11	4	The second paragraph is truncated after "and". -- FrancescoGiacomini - 20-Jul-2010 Added missing table references -- AlbertoDiMeglio - 08-Aug-2010
5	11	4	Second row in table 2, about Project planning and reporting: which validation method and responsibility apply to which task? -- FrancescoGiacomini - 20-Jul-2010 Made correspondence between task, method and responsibility explicit -- AlbertoDiMeglio - 08-Aug-2010
6	12	4.1.1	CVS is still used at CERN, not only SVN. -- FrancescoGiacomini - 20-Jul-2010 Added -- AlbertoDiMeglio - 08-Aug-2010
7	14	4.1.4	It would be useful to have template for risk management. -- FrancescoGiacomini - 20-Jul-2010 Ok, will be added to TWiki -- AlbertoDiMeglio - 08-Aug-2010
8	14	4.2.1	"Submission - 30" is not immediately clear. Better something like: "30 days before submission date" -- FrancescoGiacomini - 20-Jul-2010 Modified -- AlbertoDiMeglio - 08-Aug-2010
9	14	4.2.1	The TOC of a deliverable has to be validated by the task leader and WP leader in advance. It can simply be the WP leader to send it to the PEB, without the need to involve the PO -- FrancescoGiacomini - 20-Jul-2010 Slighly modified to take this into account. However, the new phrasing leaves open the issue of how long the WP leader validation takes before they send it to the PEB. It is assumed that it is a very brief delay -- AlbertoDiMeglio - 08-Aug-2010
10	15	4.2.1	Item 4), about the twiki-based review form: In practice there is no difference wrt the procedure followed in EGEE with dialog forms, apart that editing in twiki is a greater pain. Twiki is a better option only if it allows more interactivity between reviewers and the author, possibly all on the same page. Otherwise the dialog form is a better option, since the reviewer can work offline and just send the form at the end of the his/her review. Moreover the dates in the dialog form, e.g. the Review Date, should be dd/mm/yyyy or, even better, yyyy-mm-dd -- FrancescoGiacomini - 20-Jul-2010 The choice of TWiki was to allow greater flexibility and interactivity and track the comments close to the documents. However, I'm not very happy myself with this, personally I would only use Word change tracking, using the log tables at the beginning of the document for the comments. I'm open to suggestions, but I would leave them for the next revision of this QA Plan -- AlbertoDiMeglio - 08-Aug-2010 Comments in the document don't allow much iterativity either (just imagine this discussion managed with comments). TWiki can be fine, but it cannot just emulate the dialog form, especially if one has to respect the tabular format as I'm doing right now; it has to be used for example as suggested in the "hints on good style" (bullet on how to deal with a discussion). And you cannot use the log tables in the document for all the comments! Just imagine writing all this into that! I'm reappraising the EGEE dialog forms... We can certainly continue the discussion not in the scope of this document, but what should the reviewers of the next deliverables do? -- FrancescoGiacomini - 13-Aug-2010 Although TWiki is not the most comfortable tool for editing, it allows some degree of interactivity as this dialogue form shows. The detailed comments table can be used to discuss each topic as needed. It is recommended to use the TWiki style guidelines and use TWiki signatures for each comment -- AlbertoDiMeglio - 17-Aug-2010
11	15	4.2.1	A note on how internal reviewers are chosen would be useful -- FrancescoGiacomini - 20-Jul-2010 The following has been added: "The reviewers have to be chosen taking into account the skills and technical knowledge required to understand and critically assess the document within the context of EMI. Possibly they should be chosen from a different WP than the one issuing the deliverable, with the exception of JRA1 where the extent and diversity of skills may allow for independent reviews within the same WP. Whenever appropriate or necessary, external reviewers should also be considered, especially when the deliverable concerns topics of relevance in the relationships of EMI with other projects or communities." -- AlbertoDiMeglio - 08-Aug-2010 That sentence is certainly worth specifying, but I was rather referring to the procedure to choose reviewers, e.g. something like: the author suggests the WPs that should provide a review and the leaders of those WPs then identifies them. -- FrancescoGiacomini - 13-Aug-2010 The following note has been added: "The Work Package leader is responsible to coordinate the process of selecting appropriate reviewers for the deliverables. The WP leader must contact other WP leaders within EMI or knowledgeable persons outside EMI and ask for the names of possible reviewers. The WP leader is requested to make sure that reviewers are selected for each deliverable at least three months before the deliverable due date" -- AlbertoDiMeglio - 17-Aug-2010
12	15	4.2.2	A link to the template for the milestone report would be useful -- FrancescoGiacomini - 20-Jul-2010 Added at point 1 of the procedure -- AlbertoDiMeglio - 08-Aug-2010
13	15	4.2.2	What if the milestone is not achieved? probably the PEB should discuss and define appropriate actions, including change of plans -- FrancescoGiacomini - 20-Jul-2010 The following has been added at the end of the section: "If a milestone is not achieved within the expected deadline, the PD must foresee appropriate time for discussion during the regular PEB meeting to identify the issue and define appropriate corrective actions. Changes of plan can also be considered in case the milestone is not relevant anymore due to changed technical conditions. In case a relevant milestone is not achieved even after reasonable corrective actions have been defined and proposed, the PD must escalate the issue to the CB and discuss possible interventions on the Partner(s) responsible for the milestone." -- AlbertoDiMeglio - 08-Aug-2010
14	16	4.2.3	How should a WP contribution to the QR/PR be provided? Should the document be uploaded to twiki? to CDS? sent as an attachment? other? -- FrancescoGiacomini - 20-Jul-2010 Right. I forgot this. Added in the procedure that it should be uploaded to TWiki -- AlbertoDiMeglio - 08-Aug-2010
15	17	4.2.3	First item: "WP leaders provides contributions 20 days before the end of the period." I suppose it's after the end of the period -- FrancescoGiacomini - 20-Jul-2010 No, this is the procedure for the Periodic Management Reports. the contributions should describe the work done in the full previous year and have to be put together in the full official report for the EC. The entire process has to start before the end of the period, since the report has to be sent to the EC 10 working days before the Review and at the latest 45 days after the and of the period. -- AlbertoDiMeglio - 08-Aug-2010
16	17	4.2.3	End of the section, why specify here that PD and WP leaders provide presentations? Should they already be attached to the PR? -- FrancescoGiacomini - 20-Jul-2010 No. Why? The presentations for the review and the PR are different things. But I agree that this sentence makes little sense here. I just removed it, since the review process is described elsewhere -- AlbertoDiMeglio - 08-Aug-2010
17	17	4.2.4	If/when CDS supports reviews, will our review procedures change? if that's a possibility, maybe mention it -- FrancescoGiacomini - 20-Jul-2010 I thought of that, however, it is still not clear whether this is indeed the case. it seems to offer this feature, but not in a way that we may actually fond useful. In any case, I've added a sentence about this -- AlbertoDiMeglio - 08-Aug-2010
18	18	4.2.5	Who decides the authors of a publication, so that they can be all acknowledged? -- FrancescoGiacomini - 20-Jul-2010 Good question. I'm not sure I have an answer. Should the PEB or PTB be involved or should this be left to the Partners? It may be difficult to enforce specific authors for papers concerning the detailed work of individual PTs. The PEB and/or PTB could decide in case of general overview papers about EMI -- AlbertoDiMeglio - 08-Aug-2010 Whatever, but what is in the document now ("All authors have to be acknowledged") is not satisfactory. -- FrancescoGiacomini - 13-Aug-2010 I've tried to clarify this issue in the text. The main point is that the 'authors' to be acknowledge are actually the 'contributors' having produced some of the work described in the paper. in addition, all paper describing relevant EMI work have to be communicated to the PEB for verification and dissemination -- AlbertoDiMeglio - 17-Aug-2010
19	18	4.2.5	Include the exact wording on how to acknowledge the EC funding contribution -- FrancescoGiacomini - 20-Jul-2010 Added -- AlbertoDiMeglio - 08-Aug-2010
20	19	4.3	Provide more details on how documents should be versioned, not just "The version of the documents must be increased at every revision" -- FrancescoGiacomini - 20-Jul-2010 Added detailed description at the end of the section -- AlbertoDiMeglio - 08-Aug-2010 Good attempt, but there is something wrong at least with the first item: The version number is composed of two (2) numbers and has the format x.y. The three numbers are called major version (x), minor version (y) and revision number (z) respectively. (emphasis mine) -- FrancescoGiacomini - 13-Aug-2010 Fixed the inconsistency. The version number should be three, although z can be omitted if z=0 -- AlbertoDiMeglio - 17-Aug-2010
21	21	5.1	KSA1.1 should be reviewed/removed. See discussions on User Support in the PEB -- FrancescoGiacomini - 20-Jul-2010 Can it be redefined? If we remove it, we do not have any metric about the number of incidents. We should tack how many incidents we receive. We should probably replace Service Desk with a more appropriate term, but leave the KPI -- AlbertoDiMeglio - 08-Aug-2010
22	28	6.3	Aren't three rehearsals a bit too much? -- FrancescoGiacomini - 20-Jul-2010 In my experience, they are not enough. Take into account that the first is to get initial drafts and discuss about the principles, the second is to work on the details and the third is to properly rehearse timing and smoothness of delivery -- AlbertoDiMeglio - 08-Aug-2010
23	30	7	Why are deliverable review dialog forms protected? how to do this in twiki? -- FrancescoGiacomini - 20-Jul-2010 Indeed, no need to restrict them. In any case, TWiki has a protection mechanism based on login names, which is used for the Quarterly Reports for example -- AlbertoDiMeglio - 08-Aug-2010
24	30	7	There are probably missing links corresponding to "EU Quarterly and Periodical Report" and "EU review report" -- FrancescoGiacomini - 20-Jul-2010 The assumption was that CDS was going to be in place by now. For the moment I've added links to the TWiki pages -- AlbertoDiMeglio - 08-Aug-2010
25	30	7	It would be nice if https://twiki.cern.ch/twiki/bin/view/EMI/EmiKPIs or equivalent page would be generated automatically -- FrancescoGiacomini - 20-Jul-2010 Yes, but how? I will provide a template, but I don't see any way of creating the content automatically. The WP leaders have to provide numbers and NA1 fill the TWiki pages manually -- AlbertoDiMeglio - 08-Aug-2010
26	31	8.1	No ODS/OpenOffice for spreadsheets? -- FrancescoGiacomini - 20-Jul-2010 Added -- AlbertoDiMeglio - 08-Aug-2010
27	32	8.6	About EVO: mention that we have already a room (or whatever is called) and how to access and use it (or a link thereto) -- FrancescoGiacomini - 20-Jul-2010 Added. There is no direct link to the communities, as far as I know, but I've explained where to find it -- AlbertoDiMeglio - 08-Aug-2010