Buildings NA3.2.3 - Exploitation Strategies: NA3 Software Practice Workshop 2012 Paper

The following information is available:

Workshop Information

  • Title: First Workshop on Maintainable Software Practices in e-Science (SoftwarePractice12)
  • Location: Co-located with e-Science 2012 @ Chicago
  • Date: 9 October 2012
  • Workshop Webpage

Submission Goal

  • Submission about ScienceSoft and its plans
  • Re-using material from discussions and deliverables

Initial Paper Submission

  • 2012-07-21 Submission of initial paper deliverables Paper

Acceptance Confirmation and Reviewer Comments

Dear Morris Riedel,

we are pleased to let you know that your submission to SoftwarePractice2012

On Realizing the Concept Study ScienceSoft of the European Middleware Initiative - Open Software for Open Science

has been accepted as a full research paper for the workshop, to be published in the main e-Science conference proceedings. Reviewers comments are below and you should address these in a revised version of your paper for publication.

Camera ready versions of the papers are required by * Friday 7th September *. These should follow the IEEE Computer Society proceedings style guides:

If you have queries about the requirements for the proceedings, please direct them to the proceedings chair, Zhao Zhang <>

The workshop will be held on Tuesday 9th October as part of the e-Science 2012 conference. Registration and travel instructions for the conference are available on the main website:

Please can you let us know as soon as possible who will be presenting the work at the workshop.

We look forward to seeing you in Chicago.

Best regards, Neil Chue Hong Jennifer Schopf

REVIEW 1 --------------------- PAPER: 6 TITLE: On Realizing the Concept Study ScienceSoft of the European Middleware Initiative - Open Software for Open Science AUTHORS: Alberto Di Meglio and Morris Riedel

OVERALL RATING: 2 (accept)

This is an interesting paper describing the thinking behind the ScienceSoft initiative of the European Middleware initiative, which is planned to promote and extend the use of Open Source software and provide a ‘one stop shop’ to match users’ needs and software products and services. To me, the most interesting aspect of this paper is an exploration of the current barriers to the full exploitation of Open Source software. The authors identify barriers such as the frequent lack of an active community of interacting users and developers; the lack of real usage information (so as to enable judgements to be made about the value of a particular OS initiative); the lack of a means of citing developers’ work so that they can get academic (publication) credit; and difficulties in finding software. They plan to address these problems and to promote end-to-end vertical open source communities by implementing ScienceSoft using as much current OS software as possible. They ackn owledge that similar initiatives have been made in the past, but postulate that with the rise of social networking, initiatives like ScienceSoft might have found their time.

The weak aspects of the paper lie mostly in the presentation. The title is hard to understanding and I don’t think the paper is really about ‘realizing the concept’ but about exploring the concept. Perhaps a title like ‘ScienceSoft: Open Software for Open Science’ might be better. The whole paper could do with proof-reading. While reading it, I often wondered whether the right word was being used – e.g. appliances or applications? Technical metrics or technical attributes? And I just didn’t understand the last sentence of 3B: ‘It is felt that there is excessive formalization and loss of information due to being in different projects’. One final presentation issue is that the authors use a lot of acronyms without either explanation or reference.

Two points made in the paper which were rather left hanging were: the mention in the introduction of promoting citizen science (not mentioned again); and the repeated references to collecting real usage statistics (but with no discussion as to how this can be done).

To summarise: I think this is an interesting and valuable paper in exploring how the use of Open Source software might be improved and I look forward to finding out in the future how successful ScienceSoft will be in meeting the challenges.

REVIEW 2 --------------------- PAPER: 6 TITLE: On Realizing the Concept Study ScienceSoft of the European Middleware Initiative - Open Software for Open Science AUTHORS: Alberto Di Meglio and Morris Riedel

OVERALL RATING: 2 (accept)

The paper describes the ScienceSoft project of the European Middleware Initiative. Overall it describes existing problems with scientific software ecosystem and argues that there is a need to develop open source communities, not just projects with open source licenses. The paper highlights many specific issues, based in part on interviews and presentations that the project has made (ie getting community feedback) and lays out a set of activities that the ScienceSoft project intends to undertake.

Overall I think this is a great contribution; the paper had me writing "Yes!" in many different spots, but especially when it talks about the need to gather data on usage in order to help people make the case for the value of their software contributions. The paper is a bit short on details on how to do this, but does point out that downloads are only part of the solution.

The paper does not make use of existing research on scientific software production, but that can be incorporated and this workshop will be a good opportunity for them to meet some of those undertaking this work. The authors might find looking over literature like Segal (2008), Segal and Morris (2008) or Howison and Herbsleb (2011) useful.

Some of the solutions essentially point out that the tools and infrastructure are already available; but the paper doesn't go on to ask the question of why they aren't being used. Identifying that will really help in understanding how best to move forward. In particular, I am worried that this initiative has the expiring funding problem: if this can't be done without funding doesn't that indicate a lack of sustainability right from the start?

Practically speaking, I think the best suggestion in the paper is to develop a prototype of how to organize an open project (this is pretty much what the Apache project's incubator does).

The paper is a bit distracted at the start with the market analysis and the stack figure. I think it would be best to remove that section and really focus on everything that comes after the "B Open Science Market Analysis Model".

I think this will be a very interesting paper at the workshop. References:

Howison, J., & Herbsleb, J. D. (2011). Scientific software production and collaboration. Computer Supported Collaborative Work (CSCW 2011).

Segal, J. (2008). Models of Scientific Software Development. Proc. 2008 Workshop Software Eng. in Computational Science and Eng. (SecSe 08). Retrieved from

Segal, Judith, & Morris, C. (2008). Developing Scientific Software. IEEE Software, 25(4), 20.

REVIEW 3 --------------------- PAPER: 6 TITLE: On Realizing the Concept Study ScienceSoft of the European Middleware Initiative - Open Software for Open Science AUTHORS: Alberto Di Meglio and Morris Riedel

OVERALL RATING: 2 (accept)

This paper describes and effort to provide a community resources for scientific software being developed particularly in Europe, but it can also be applicable to software being developed in the US. Particular emphasis is placed on issues of sustainability of the software products.

The paper does a good job in identifying the challenges and problems with today's scientific software and presenting possible solutions.

Although the paper mentions issues of software metrics and ratings of software, it does not talk much about these quantities. How do you measure the usage? By the number of installations? Number of jobs ran using the software, etc..

It would also be good to mention the relationship to communities such as the Apache foundation.

In the context of this paper, it would be good to also have a discussion of the sustainability of ScienceSoft over time.

It would be interesting to see how successful is ScienceSoft over time.

Fig 1 which is referred to throughout is impossible to read.

REVIEW 4 --------------------- PAPER: 6 TITLE: On Realizing the Concept Study ScienceSoft of the European Middleware Initiative - Open Software for Open Science AUTHORS: Alberto Di Meglio and Morris Riedel

OVERALL RATING: 2 (accept)

Although there have been many other initiatives around creating an open source community around discipline based and project software and just as many actual implementations, this paper offers very useful findings (through a formalised feasibility study) that will be of high relevance to delegates, not least because the EMI Science Soft case study is also part of a wider strategy in this space.

There are challenges, however - and not least addressing goals, fragmentation, motivation and investment. The real issue seems to be if there is enough interest in working together as a community to drive joint development in EMI and related open-source software as a whole.

It would be beneficial to channel the paper into a panel discussion and provide a community'critique' as the authors have suggested in their abstract as a means of informing the direction of the proposed initiative and in building on the initial findings.

REVIEW 5 --------------------- PAPER: 6 TITLE: On Realizing the Concept Study ScienceSoft of the European Middleware Initiative - Open Software for Open Science AUTHORS: Alberto Di Meglio and Morris Riedel

OVERALL RATING: -3 (strong reject)

This paper starts by describing European initiatives for open source software development and middleware development, but then devolves into a very generic discussion of requirements and solutions for better management of open source software with very little meat or value. For example, the authors dismiss as a viable solution only because it houses very few academic projects. But the site is vibrant and quite amazing, and collects many of the statistics that the authors argue for. It is working well for many existing open source projects. Why reinvent the wheel? Why not promote its use more within academic communities? The authors don't justify this with any clarity. It is hard to imagine duplicating and then surpassing the functionality of with the specified 6-month development window for ScienceSoft. Similarly, the authors reject other approaches such as Academia and ResearchGate, both of which have gotten considerable traction with millions of researchers. The authors fail to address the social mechanisms that they will use to attract and build a vibrant community of similar size. They fail to acknowledge that engaging a community is even an issue.

The paper is light on originality and difficult to read. The text in the figures is unreadable. The paper is full of typos and mistakes. Page 1, top of second column: "specific software enable" should be "specific software enables". Page 3, second column: "followed by a few question" should be "followed by a few questions". "structure that animate" should be "structures that animate". "The cases were" should be "The cases where". "On a daily base" should be "On a daily basis". Page 4, second column, "Before using an application Limited..." doesn't make sense. Missing a phrase or something. "what exist already" should be "what exists already". Page 5, first column, section H starts with the same sentence as section G, and then says "The main of foremost barrier..." which makes no sense. "Institutes" and "Companies" are bizarrely capitalized on page 5. On page 7, first column, "existing successful open source foundation" should be "existing successful open source foundations." "Science Soft" should be "ScienceSoft". On Page 8, "aims to share related and has currently" makes no sense. "We can con clude" should be "We can conclude". "the reason why using it at all" makes no sense.

REVIEW 6 --------------------- PAPER: 6 TITLE: On Realizing the Concept Study ScienceSoft of the European Middleware Initiative - Open Software for Open Science AUTHORS: Alberto Di Meglio and Morris Riedel

OVERALL RATING: 1 (weak accept)

This paper introduces the concept of ScienceSoft which is one of activities of European Middleware Initiative (EMI). The paper presents deep analysis of how software could be recognized and widely used by global communities. Unfortunately, this is an introduction of EU project rather than technical paper for the workshop. Actually, the concept has alrady presented at several international conferences and workshops, hence it is less expected to have fruitful discussions in the SofwarePractice workshop. I agree with their motivation and some interesting issues are presented in this paper. The paper should be improved before publication if this will be accepted.

Camera-ready version

  • Revision by Alberto: [ doc ]

Topic attachments
I Attachment History Action Size Date Who Comment
Microsoft Word filedoc 2012-07-20-ScienceSoft-e-Science2012.doc r1 manage 840.5 K 2012-08-20 - 11:18 MorrisRiedelExCern Initial Paper Version submitted for review (doc version)
PDFpdf 2012-07-20-ScienceSoft-e-Science2012.pdf r1 manage 685.4 K 2012-08-20 - 11:17 MorrisRiedelExCern Initial Paper Version submitted for review
Microsoft Word filedoc 2012-07-20-ScienceSoft-e-Science2012_ADM.doc r1 manage 849.0 K 2012-09-10 - 13:55 MorrisRiedelExCern e-Science 2012 ScienceSoft Camera-ready Paper Revision by Alberto

This topic: EMI > WebHome > EmiProjectStructure > NA3 > EMINA3TASK2 > EMINA3TASK2SUBTASK3 > NA3SWPRACTICEWS2012PAPER
Topic revision: r3 - 2012-09-10 - MorrisRiedelExCern
This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback