Evaluation of ETICS services "on the field" and strategies for their improvement

SA2.5 - Task overview

  • Analysis of the behaviour of the etics-services "on the field" together with the users
  • Possible area of analysis:
    • Efficiency, scalability and fault tolerance and any other non functional areas impacting the usability of the system
  • Suggestion of improvements to SA1 in order to constantly improve the usage of etics services, based on real usage feedback

Process

Interaction between SA1 and SA2 teams will be realized through a wiki document. The process of interaction will consist of 5 phases:
  • Phase 1: Fact finding. SA2 interviews ETICS users
  • Phase 2: SA2 gathers all the information from the interviews and makes some analysis (about the criteria for this analysis brief discussion with SA1 team)
  • Phase 3: SA2 waits for a SA1 response about possible modifications and time frame according to the analysis
  • Phase 4: New release of one of the ETICS services
  • Phase 5: Analysis of the improvements made by SA2 and further interviews to ETICS users if possible

Process.jpg

Phase 1: Fact Finding

In order to do an evaluation of the etics services we decided to start from the opinions of ETICS users. With the help of these contributions it is possible to do a precise analysis of ETICS system and try to improve it when possible and necessary.

Generally speaking, in order to obtain information about a certain system from its users, various kind of techniques can be used and the most popular among them are interviews, questionnaires, record reviews, case tools, on-site observations and so on. Interviews consist basically in questions asked to a user of a system, questionnaires are another way of information gathering where the potential users of the system are given questionnaires to be filled up and returned to the analyst. Record reviews activity consists basically in analyzing the collection of information and data accumulated over the time by the users about the system and it's operations. The analyst can use also case tools to represent and assemble all the information and the data gathered about the system. Finally the on site observations are one of the most effective tools with the analyst where the analyst personally goes to the site and discovers the functioning of the system. For more information about these techniques
http://www.freetutes.com/systemanalysis/SA004.htm

Which techniques to adopt?
In order to extract as much information as possible from the concerned people, and to let the interviewee free to answer we decided to use the interview technique. Moreover the interview is especially suitable for a context in which there are not many interviewees as in our case because it is a time consuming technique. Actually in our case the interview technique is accompanied by the on site observations one. This is due to the fact that the analyst, during the interview, should observe personally the activity carried-out by the user. As an observer the analyst can gain first hand knowledge of the activities, operations, processes of the system and also can realize better the problems of ETICS users.

What type of Interview?
Interviews can be of two types: structured and unstructured. The structured are those where the interviewee is asked a standard set of questions in a particular order. All interviewees are asked the same set of questions.The unstructured interviews are undertaken in a question-and-answer format. This is of a much more flexible nature than the structured interview. Each of the structured and unstructured interview methods has its own merits and demerits.
We decided to follow the structured interview method in order to obtain a uniform evaluation. The questions that will be asked are listed below. Anyway in order to avoid the demerits of the structured interview, the set of questions we prepared is a mix of open-response and closed-response ones. Thus for some of these questions the interviewee is quite free to answer, in this way there may be the case that some issue might surface spontaneously. In that case the respondent can express views on that issue also.

Which users?
The ETICS users that will be involved in this evaluation of ETICS services belong to two type of "ETICS roles": the Developer and the Integrator. Both of these kind of users exploit ETICS services more than others, thus their opinions will be very precious for our analysis. In order to summarize the task of this two kind of users look at the use case diagrams below: Use_Case_Diagram__Untitled1.jpg Use_Case_Diagram__Integrator.jpg

Among ETICS users it can be taken into consideration also the Administration role. At the moment we decided to not interview people belonging to this role because the functionalities offered by ETICS Services to administrators are not mature and can be evaluated in a later time.

Questions of an interview to an ETICS user.

  1. What is your activity in ETICS as developer or as integrator?
  2. What are the most frequent tasks that you carry-out with ETICS?
    1. As a developer:
      1. checking-out, building, packaging of the code
      2. editing of advanced metadata
      3. Checking in and tagging code
      4. Registering of artefacts
      5. Editing and creation of configurations
    2. As an integrator:
      1. creating meta rpms using metadata
      2. grouping configuration together into a asubsystem/project configuration
      3. Managing of metadata and links between CVS tags, etics configurations name and package version
      4. Analyzing of conflicts between configurations using metadata
  3. In which aspects of your work do you expect to be mostly helped by ETICS?
    1. Performing which task do you think that ETICS is useful to you?
    2. In what aspects, instead, do you think that etics delays your activity?
    3. Can you suggest an ideal solution in order to solve this delay. How do you think that ETICS should work?
  4. Do you use in a prevalent way the etics-command line client or the web application?
    1. In order to perform which task do you prefer etics command line client? and what about the web application?
    2. Concerning etics command line client:
      1. Which commands do you use more frequently? What about their performance and usability?
      2. Do you use etics command line client only for integration or also for developing?*
    3. Concerning etics web application:
      1. Do you think that the placement of the various commands (buttons) is intuitive?
      2. Do you think that a different placement consisting in grouping the buttons in different panes will be more intuitive?
      3. Do you ever felt the need to have a guide that leads you in using the proper button in some situations?
      4. Do you think that it would be useful to have a sort of guidelines that explains to you how to fill the form correctly? For example dependencies, properties, or configurations...
      5. Which commands (or buttons) do you use more frequently in the web application?
      6. What do you think about the overall layout? Where do you think it can be improved?
      7. What panels are lacking in usability?
      8. What are the actions you find difficult to accomplish (e.g. too many clicks, unclear sequence, insufficient feedback, etc..)?
      9. Where do you find the WA is not presenting concepts coherently with the CLI?
      10. Is there any functionality you feel the WA misses at interface level?
      11. Is there any piece of the WA you feel useless?




* Listening to some ETICS users we understood that some of them use ETICS every new and then and only for some tasks like workspace-setup, checkout and build. Instead other users use ETICS daily in their activity

Interview Results

Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg Process.jpg r1 manage 49.7 K 2008-11-06 - 10:03 MarcoCanaparo  
JPEGjpg Use_Case_Diagram__Integrator.jpg r1 manage 110.4 K 2008-10-30 - 11:55 MarcoCanaparo ETICS Integrator
JPEGjpg Use_Case_Diagram__Untitled1.jpg r1 manage 130.5 K 2008-10-30 - 12:03 MarcoCanaparo ETICS Developer
JPEGjpg process.jpg r1 manage 49.0 K 2008-11-04 - 18:13 MarcoCanaparo  
Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r18 - 2008-12-09 - MarcoCanaparo
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ETICS All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback