JRA1 Deliverable Review Form

Attach the reviewed document to the deliverable page, put here a link

General comments

ToC comments ToC comments

DSA2.3.2 Periodic QA Reports ToC Review:


1. TOC layout:

Would it be possible to keep the section titles consistent: "Report on the required documentation", "Metric reports", "Summary of SQA Deviations"?
This can be achieved either by using the naming convention from the previous report i.e. DSA2.3.1 (like "Status of document") or by establishing a new consistent convention (e.g., "Report on documentation", "Report on Metrics", etc).

Section 5 "Metric Reports": please add subsections for each metric.

Section 7 "Summary of SQA Deviations": is the term "deviation" a part of official SA2 terminology? I would suggest to use less pejorative word if possible. Subsection for each "deviation" would be nice.

Section 8 "List of actions": it is not clear to me what you intend to write there but if this should be an evolution/working plan till the next QA report then I would suggest a separate section for that.

Conclusion(s) part is missing.

2. Missing content?

DSA2.3.1 made some obligations, I am not sure if all of the issues will be addressed in the current report:
a) "Status of the Service Reference Cards per software component should be included in the upcoming QA reports."
--no such section in the current document,
b) "For future QA reports, it’s therefore foreseen to include reviews of the Technical Work Area plans."
--no such section in the current document
c) "The progress of the Software Release Plan deliverable [...] Once the document is finished it will be considered in the next QA report."
--no such section in the current document

Could you comment on that and/or include the comments in the executive summary (as explanation of what happened since last report)


3. Minors:
-Title in the page header is missing.
-Reference to the previous report should be included.
-Abstract could include the time interval covered by the current report.

• The document is very imprecise on the subject of work that need to be done with regard to who and when will do the work. There are a lot of statements like "should be discussed", "could be improved", etc. I understand that it is not a work plan but writing in such a manner leaves bad impression. I am quite sure you have some plans regarding most of such points. Maybe you can put them in the document?

Maria: OK, We don't have a work plan as such. Some ideas for improvement would need to be discussed first with the SA2 leader, so we can't take for granted that all of them will be implemented, but at least, I have summarised then in a new section called "Recommendations"

Jedrzej: "Recommendations" are indeed an improvement, I still miss some "due to" deadlines there but I consider the current status a compromise. It should provide the reviewers of the next deliverables with background what was planned/what was achieved"

Maria: OK. I've given dates for those action items that depend more on my task and where I can decide. For the others is more complicated and I've also tried to define a deadline or at last I have put a checkpoint for the next All Hands Meeting to present the status of that action

• The document is also quite imprecise with regard to reasons for failures. An example of that is the way how late documentation policies are described. What were the reasons for the delay, who is responsible for that, and how can it be avoided in the future?

Maria: Could you please be more specific on which parts of the document you are missing this information?

Jedrzej: I gave an example above. With regard to late policies for instance the only statement I found is: "SA2 expects better results for EMI 2 although in general development teams have worked very hard and done a big effort to comply with the policies.". Prove me wrong, but I don't considered it as a way of avoiding the problem in the future.

Maria: OK. I have tried to explain better the situation and also presented how it can be avoided in the future. This is now moved to section 6.

• "One important change has been the redefinition of guidelines to policies. This has helped product teams to understand that the procedures defined by SQA have to be followed and that they guarantee a successful release process." well I hardly believe that there are any policies which can guarantee successful release process. The word guarantee has a very specific, almost legal meaning, and it is misused here in my opinion.

Maria: I mean here that when SA2 agreed to change the name from guidelines to policies it was because SA2 believed it would help to make it more clear that the policies were instructions to be followed, not just recommendations that could be ignored. And I don't think "guarantee" has any legal meaning so that we can't use it in the deliverable. SA2 believes that by following the policies, a succesful release process can be achieved.

Jedrzej: I am not a native speaker but I see a discrepancy between a guarantee of a success and a belief that a success can be achieved

Maria: OK. I have removed the sentence since in any case there are more details given later on in section 3 without using the word "guarantee"

• Secondly: the report reads like this was the main (if not the only one) achievement of SA2 in the last year: changing a word from guideline to policy... and suddenly the world is saved? I got the impression that this highlight sets the crossbar very low. At the same time, there are no more challenging highlights to be reported?

Maria: This was not the only achievement. Many achievements are included in the report. I don't agree with this comment. Can you support this statement with some facts? Where does it say that this is the only achievement?

Jedrzej: The impression is mainly caused by the Executive Summary I guess. The change is there mentioned in the first paragraph, and after that we got some problems and wishful thinking: "difficulties in having the policies and procedures up to date are also presented ... and it has to be clearly defined what the channels for implementing changes are and how to perform them and notify them to the concerned audience." and some future work "A status of the documents that have to be monitored" and acknowledgements "The metrics started to have more and more relevance towards the EMI 1 deadline. The importance of useful and meaningful metrics was acknowledged by SA2 and a review exercise was organised with developers. "

I exaggerate a little bit to underline what I mean. I am quite convinced that SA2 did some good work in the first year and have bigger achievements than only changing one word in the other. The problem is that the achievements are not highlighted. This is bad for the report, bad for SA2, bad for EMI.

Maria: OK. I have removed the reference to guidelines-policy change in the executive summary.

Additional recommendations (not affecting the document content, e.g. recommendation for future work)

Detailed comments on the content

Note 1: The reviewers must list here any observation they want to track explicitly and that require interaction with the authors
Alternatively all changes must be listed in the document itself using Word change tracking features (if you use Word)
Note 2: These comments have to be explicitly addressed by the authors and the action taken must be clearly described

Presentation

-- FloridaEstrella - 06-May-2011

-- JedrzejRybicki - 27-May-2011