JRA1 Deliverable Review Form

Identification of the deliverable or milestone
Project: EMI Deliverable or milestone identifier: D4.3.2
Title: DSA2.3.2 - Periodic QA Reports Doc. identifier: EMI-DXXX-CDSREF-Title-vx.x
Author(s): M. Alandes Due date: __

Identification of the reviewer
Name: J. Rybicki Affiliation: JUELICH EMI Activity/External project or Institute: JRA1

Review date mm/dd/yyyy
Author(s) revision date mm/dd/yyyy
Reviewer acceptance date mm/dd/yyyy

Attach the reviewed document to the deliverable page, put here a link

General comments

DSA2.3.2 Periodic QA Reports ToC Review:

1. TOC layout:

Would it be possible to keep the section titles consistent: "Report on the required documentation", "Metric reports", "Summary of SQA Deviations"?
This can be achieved either by using the naming convention from the previous report i.e. DSA2.3.1 (like "Status of document") or by establishing a new consistent convention (e.g., "Report on documentation", "Report on Metrics", etc).

Section 5 "Metric Reports": please add subsections for each metric.

Section 7 "Summary of SQA Deviations": is the term "deviation" a part of official SA2 terminology? I would suggest to use less pejorative word if possible. Subsection for each "deviation" would be nice.

Section 8 "List of actions": it is not clear to me what you intend to write there but if this should be an evolution/working plan till the next QA report then I would suggest a separate section for that.

Conclusion(s) part is missing.

2. Missing content?

DSA2.3.1 made some obligations, I am not sure if all of the issues will be addressed in the current report:
a) "Status of the Service Reference Cards per software component should be included in the upcoming QA reports."
--no such section in the current document,
b) "For future QA reports, it’s therefore foreseen to include reviews of the Technical Work Area plans."
--no such section in the current document
c) "The progress of the Software Release Plan deliverable [...] Once the document is finished it will be considered in the next QA report."
--no such section in the current document

Could you comment on that and/or include the comments in the executive summary (as explanation of what happened since last report)

3. Minors:
-Title in the page header is missing.
-Reference to the previous report should be included.
-Abstract could include the time interval covered by the current report.

  • The document is very imprecise on the subject of work that need to be done with regard to who and when will do the work. There are a lot of statements like "should be discussed", "could be improved", etc. I understand that it is not a work plan but writing in such a manner leaves bad impression. I am quite sure you have some plans regarding most of such points. Maybe you can put them in the document?

Maria: OK, We don't have a work plan as such. Some ideas for improvement would need to be discussed first with the SA2 leader, so we can't take for granted that all of them will be implemented, but at least, I have summarised then in a new section called "Recommendations"

Jedrzej: "Recommendations" are indeed an improvement, I still miss some "due to" deadlines there but I consider the current status a compromise. It should provide the reviewers of the next deliverables with background what was planned/what was achieved"

Maria: OK. I've given dates for those action items that depend more on my task and where I can decide. For the others is more complicated and I've also tried to define a deadline or at last I have put a checkpoint for the next All Hands Meeting to present the status of that action

  • The document is also quite imprecise with regard to reasons for failures. An example of that is the way how late documentation policies are described. What were the reasons for the delay, who is responsible for that, and how can it be avoided in the future?

Maria: Could you please be more specific on which parts of the document you are missing this information?

Jedrzej: I gave an example above. With regard to late policies for instance the only statement I found is: "SA2 expects better results for EMI 2 although in general development teams have worked very hard and done a big effort to comply with the policies.". Prove me wrong, but I don't considered it as a way of avoiding the problem in the future.

Maria: OK. I have tried to explain better the situation and also presented how it can be avoided in the future. This is now moved to section 6.

  • "One important change has been the redefinition of guidelines to policies. This has helped product teams to understand that the procedures defined by SQA have to be followed and that they guarantee a successful release process." well I hardly believe that there are any policies which can guarantee successful release process. The word guarantee has a very specific, almost legal meaning, and it is misused here in my opinion.

Maria: I mean here that when SA2 agreed to change the name from guidelines to policies it was because SA2 believed it would help to make it more clear that the policies were instructions to be followed, not just recommendations that could be ignored. And I don't think "guarantee" has any legal meaning so that we can't use it in the deliverable. SA2 believes that by following the policies, a succesful release process can be achieved.

Jedrzej: I am not a native speaker but I see a discrepancy between a guarantee of a success and a belief that a success can be achieved

Maria: OK. I have removed the sentence since in any case there are more details given later on in section 3 without using the word "guarantee"

  • Secondly: the report reads like this was the main (if not the only one) achievement of SA2 in the last year: changing a word from guideline to policy... and suddenly the world is saved? I got the impression that this highlight sets the crossbar very low. At the same time, there are no more challenging highlights to be reported?

Maria: This was not the only achievement. Many achievements are included in the report. I don't agree with this comment. Can you support this statement with some facts? Where does it say that this is the only achievement?

Jedrzej: The impression is mainly caused by the Executive Summary I guess. The change is there mentioned in the first paragraph, and after that we got some problems and wishful thinking: "difficulties in having the policies and procedures up to date are also presented ... and it has to be clearly defined what the channels for implementing changes are and how to perform them and notify them to the concerned audience." and some future work "A status of the documents that have to be monitored" and acknowledgements "The metrics started to have more and more relevance towards the EMI 1 deadline. The importance of useful and meaningful metrics was acknowledged by SA2 and a review exercise was organised with developers. "

I exaggerate a little bit to underline what I mean. I am quite convinced that SA2 did some good work in the first year and have bigger achievements than only changing one word in the other. The problem is that the achievements are not highlighted. This is bad for the report, bad for SA2, bad for EMI.

Maria: OK. I have removed the reference to guidelines-policy change in the executive summary.

Additional recommendations (not affecting the document content, e.g. recommendation for future work)

Detailed comments on the content

Note 1: The reviewers must list here any observation they want to track explicitly and that require interaction with the authors
Alternatively all changes must be listed in the document itself using Word change tracking features (if you use Word)
Note 2: These comments have to be explicitly addressed by the authors and the action taken must be clearly described

Page Section Observations and Replies Is Addressed?
  8 2 "The second Periodic QA report describes the SQA activity after the first year of EMI." actually it mainly describes the activities in the first year. YES
  8 2 "The defined policies help SA2 to measure the software according to our defined metrics;" do they really measure the software or some aspects of the produced software? .
Maria: I'm not sure about your comment, what is it exactly the difference between saying "measure the software" in general, and "measure some aspects of the software"?
Jedrzej: IMHO it is like a difference between measuring the temperature of water in a lake and measuring water. Software just like water has many aspects which can be measured, but it is hardly measurable as such.
Maria: I have removed this sentence as the executive summary has been reorganised as suggested by PO
  8 2 "The policy arrived very close to the release deadline and many product teams showed discontent to apply it. " why they arrived so late? This report usually does not answer such questions.
Maria:This is explained in section 8.
Jedrzej: Section 9 now? I find the explanation vague and I think at least a forward reference to Section 9 should be included here.
Maria: OK. This is moved to section 6 and I have given more details.
    3 "The use of words like “MUST” or “SHOULD” was also reviewed. " Well, lot of people working in computer science are aware of the existence of RFC2119 where exactly such words are defined. You actually mentioned this RFC (in different content). So my question is: why couldn't SA2 adopt the definitions from the RFC?
Maria: These definitions are indeed adopted. This is what I mean when I say that "The use of words like “MUST” or “SHOULD” was also reviewed".
Jedrzej: Let me reformulate: why they were not used from the very beginning? What were the reasons for that?
YES. Added explanation.
    4.1 "SA2 policies have changed throughout the lifetime of the EMI-1 release." why and how they changed? More background information needed.
Jedrzej: Forward reference?
YES. Added reference.
    4.1 "The final training was about Release, Change, Testing, Certification and Documentation policies. This training tried to give a global overview of the whole release process. " why it only tried? Was it a failure? Either rephrase or provide more background. YES
    4.2 I wish the problem of "not coherent information" would be explained in such a way that a ETICS layman like me could understand what the problem is. I think the sections should also be more specific with regard to possible solutions and how they shall be implemented. YES. I have added some more explanations.
    5.4 "It’s a very good reference to understand the existing tools and the technologies involved." "very good" according to whom? Was this quality actually tested? YES. I have removed "very good"
    5.6 "We evaluated the global readability by answering the question: Is information structured in a clear way so that it is easy to find? And also the clarity of the documentation by answering the question Is information presented in a clear way?" Was this evaluation conducted by single person? I see some problem when comparing answers of different people on such an unspecified questions. Things which are clear to me might not be clear to other people, sure. But the margin of error when answering such questions with yes/no is much much smaller as when own "feelings" are involved and answers like "the information is presented is outstanding clear way". What is the value-added when answering the questions with the adjectives? Could you elaborate on the usage of multiple choice for such simple questions? And explain how it was assured that the words outstanding or poor were used properly?
Jedrzej: the problem is caused by the fact that the process was conducted by a group of people and their judgment on the quality of the documentation is not "standardized" nor comparable
YES. Comment taken into account and added as an improvement we have to consider for next review.
    5.6 "On the other hand, many development teams have already the effort and met these requirements." 1) there is missing verb here? 2) who miss the requirements? maybe you can attach the information or at leas a reference. YES
    6 "A very useful document that was also created to help developers throughout the different stages of the release process is the EMI component release check list [R14]." I assume that the document was created by SA2? otherwise it would not be very useful. Was it tested if it is really very useful? YES. I have removed "very useful"
    7 "Take into account developer’s feedback who should be the first ones benefitting from the metrics report. But also take into account other parts of the release process where metrics may help to understand how EMI releases are doing." 1) spelling: benefiting? 2) the managers benefit from metrics not the developers. A developer knows what her software is worth of, the manager does not understand the software and needs metrics to get the feeling that she understand.
Maria: 1) The spell checker doesn't mark it as wrong.
Jedrzej: my bad, American English probably
Maria: 2) SA2 doesn't agree with this statement so I won't take it into account.
Jedrzej: Prove otherwise or remove the statement
YES. Statement is now removed.


Section Observations and Replies Is Addressed?
    "presented in detail in section 6" consequently in the report section references do not start with capital letter (Section 6). The references to figures on the other hand, do. It could improve the presentation to "capitalize" both kinds. YES
    Figure 1 could be moved at the end of the section. Otherwise it cuts the text and hides the second part of the release review comments. YES
    the use of abbreviated forms (like it's didn't) should be avoided in formal documents. YES
  1.5: Page 7 Terminology: --alphabetical order? --Key Performa*n*ce Indicator (spelling) YES
  5.3: Page 15 improvements could be better presented in form of a table
Maria:I think it's pretty clear with a list.
Jedrzej: And I think that in the list one cannot see what is "DONE" and what is "MISSING".
  5 .6 : Page 18 Table 1: the caption of the middle column looks strange (floats left) YES
  5.6 : Page 20 Twiki is written here capitalized, and on other places (Page 9), please do it consequently one way of the other YES
  5.6 : Page 20 and Page 21 The Pie charts are strange: 47%+4% is not 100% and 17%+27% is not 100% YES. I have removed the %. They were indeed strange. This is calculated by google docs automatically and I don't know why it is wrong.
  8: Page 25 "Almost most" YES
  the word "publicize" is very often used in this document.
Maria: This word is used twice. I don't think twice is "very often", but of course, "very often" is something subjective.
Jedrzej: It is used three times (publicized also counts for me).

-- FloridaEstrella - 06-May-2011

-- JedrzejRybicki - 27-May-2011

Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r12 - 2011-05-28 - MariaALANDESPRADILLO
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback