DEVELOPMENT OF SCIENTIFIC SOFTWARE: A SYSTEMATIC ...

International Journal of Software Engineering and Knowledge Engineering Vol. 23, No. 4 (2013) 463506 # .c World Scienti¯c Publishing Company DOI: 10.1142/S0218194013500137

DEVELOPMENT OF SCIENTIFIC SOFTWARE: A SYSTEMATIC MAPPING, A BIBLIOMETRICS STUDY, AND A PAPER REPOSITORY

ROSHANAK FARHOODI prairieFyre Software Inc., Ottawa, Canada [email protected] VAHID GAROUSI Department of Electrical and Computer Engineering University of Calgary, Canada and Informatics Institute Middle East Technical University Ankara, Turkey [email protected] DIETMAR PFAHL Institute of Computer Science, University of Tartu Estonia and Department of Electrical and Computer Engineering University of Calgary, Canada [email protected] JONATHAN SILLITO Department of Computer Science University of Calgary, Canada [email protected]

Received 24 April 2012 Revised 17 October 2012 Accepted 9 January 2013

Scienti¯c and engineering research is heavily dependent on e®ective development and use of software artifacts. Many of these artifacts are produced by the scientists themselves, rather than by trained software engineers. To address the challenges in this area, a research community often referred to as \Development of Scienti¯c Software" has emerged in the last few decades. As this research area has matured, there has been a sharp increase in the number of papers and results made available, and it has thus become important to summarize and provide an overview about those studies. Through a systematic mapping and bibliometrics study, we have reviewed 130 papers in this area. We present the results of our study in this paper. Also we have made the

463

464

R. Farhoodi et al. mapping data available on an online repository which is planned to be updated on a regular basis. The results of our study seem to suggest that many software engineering techniques and activities are being used in the development of scienti¯c software. However, there is still a need for further exploration of the usefulness of speci¯c software engineering techniques (e.g., regarding software maintenance, evolution, refactoring, re(v)-engineering, process and project management) in the scienti¯c context. It is hoped that this article will help (new) researchers get an overview of the research space and help them to understand the trends in the area. Keywords: Development of scienti¯c software; systematic mapping; bibliometrics study; paper repository.

1. Introduction New knowledge in science and engineering relies increasingly on results produced by scienti¯c software. To build high-quality software in a cost e®ective manner for applications in science and engineering disciplines, scientists and the Software Engineering (SE) community have collaborated in the last few decades and a community mostly referred to as \Development of Scienti¯c Software" has emerged [63, 115]. In this context, the term \scienti¯c software" is referrers to software which is developed by scientists for scientists and which implements complex algorithms, e.g., to solve systems of mathematical equations or to provide simulations for physicists. Over the years, the SE practice and research has focused on techniques and concepts intended to be generally applicable to software development. However, SE best practices and research have only rarely been adopted by scienti¯c computing [131]. The motivation for the development of scienti¯c software is to conduct science. Unlike software engineers, scientists' focus is on performing science, not on writing software [132]. Also, the variation in di®erent disciplines of science, maturity of the science, and motivation of scienti¯c software projects in°uence development methods, and one would consequently expect large variations both across and within di®erent disciplines. As the community and research area of scienti¯c software has matured, there has been a sharp increase in the number of papers and results made available, and it has thus become important to summarize and conduct a systematic classi¯cation of those studies. To address this need, we have conducted a systematic mapping study and have collected bibliometric and demographic data in this area. From an initial pool of about 500 publications, we systematically selected and classi¯ed (mapped) 130 publications. Regarding the type of research, two thirds of the set of 130 publications are \solution proposals", \experience papers", and \¯eld studies". The most frequently mentioned scienti¯c discipline is \physics", and the most frequently used programming languages are \Fortran" and \Cþþ", with an emerging trend towards using more modern programming languages such as \Java" and \Python". The software development activities and techniques most frequently mentioned in our pool of selected publications are \development/coding", \architecture and design", and \testing/V&V/quality assurance". Based on an analysis of the bibliometric and demographic data (similar to previous studies such as [133136]), ten research questions regarding, for example, the

Development of Scienti¯c Software

465

frequency of relevant publications over time, the most prominent publication venues, the most cited publications, the most active researchers, and the geographical distribution of publications, have also been addressed in this study. Mapping studies and their results have several bene¯ts, for example, they can help new researchers in this area (e.g., new PhD students) gain a ¯rst overview of their ¯eld of research [137], and they can indicate directions for new research. We hope that the results of our study can provide a better understanding about the activities performed and the techniques used in the development of scienti¯c software. Our study also aims to identify current trends and directions in this ¯eld, which are valuable to both researchers in software engineering (SE) and scientists who develop software. The other goal of this SM is to identify strengths and weaknesses of stateof-the-art in the area. The rest of this article is structured as follows. Background and related work are discussed in Sec. 2. Section 3 presents the goal of the study, the research questions and research method. The article selection process and details are presented in Sec. 4. Section 5 presents the classi¯cation scheme (map) of our systematic mapping. Results of the systematic mapping are described in Sec. 6. Section 7 provides the bibliometric data and demographics of the publications. Section 8 discusses threats to validity. Finally, Sec. 9 concludes the article and indicates directions of future research.

2. Background and Related Work We present a brief background on systematic mapping studies in software engineering in Sec. 2.1. Then in Sec. 2.2, we summarize the secondary studies in the area of development of scienti¯c software. 2.1. Background: Systematic mapping studies in software engineering As a research area matures, there is often a sharp increase in the number of papers and results made available, and it becomes important to summarize and provide an overview about those studies [138]. Many research ¯elds (e.g., medical sciences) have speci¯c methodologies for such secondary studies, and they have been extensively used in for example evidence-based medicine. Budgen et al. [137] presented di®erent bene¯ts of conducting mapping studies and usefulness of their results. For example, such studies can form a useful preliminary step for new PhD students in this area. The guideline paper by Petersen et al. [138] is a great resource on conducting systematic mapping (SM) studies in software engineering, which we used as a guide in designing our study. A software engineering systematic map is a de¯ned method to build a classi¯cation scheme and structure a software engineering ¯eld of interest [138]. The analysis of results focuses on frequencies of publications for categories within the scheme. Thereby, the coverage of the research ¯eld can be determined. Di®erent facets of the scheme can also be combined to answer more speci¯c research questions.

466

R. Farhoodi et al.

Similar to SMs, but on a more comprehensive scale, Systematic Literature Review (SLR) is another secondary study method that has gotten much attention lately in SE. Brie°y, a systematic review (SLR) identi¯es existing primary studies, reviews them in-depth and describes used methodologies and achieved results related to a de¯ned subject matter topic. SLRs usually has more analytic depth than SMs. Often, SLRs include a SM as one of their elements. As the ¯rst step in our e®orts to review the existing literature on methods and techniques used in the development of scienti¯c software, the current article reports a SM (and not a SLR). 2.2. Secondary studies in the area of development of scienti¯c software In total, we identi¯ed 130 relevant publications [1130] for the period 19962011. Within this set, only two recent publications [119, 139] were secondary studies. Howison et al. reviewed three publications and interviewed the authors and developers of the projects presented in the papers [119]. Based on the analysis of the selected publications and related interviews, the authors identi¯ed several modes of how the development and maintenance of software interleaves with the way research is conducted by scientists. These modes depend on the purpose of the software and the role it can play in the system of reputation incentives for scientists. The authors conclude that the development and maintenance of scienti¯c software may either be indirectly rewarded with academic credit if the software facilitates science publications, or it may be directly rewarded if the software can be released for academic credit on its own. The latter case, however, requires a certain degree of collaboration between scientists and a readiness of scienti¯c community to accept scienti¯c software as a research asset in its own right. Sletholt et al. presented a literature review of reported agile practices and their e®ects in the development of scienti¯c software [139]. The authors identi¯ed ¯ve publications reporting on experience with the use of agile practices in scienti¯c software projects. Based on the analysis of these papers, the authors conclude that there exists evidence that using agile practices can have positive e®ects on test e®ectiveness and e±ciency.

3. Research Method In the following, an overview of our research method and then the goal and research questions of our study are presented. 3.1. Overview This SM is carried out based on the guidelines provided by Petersen et al. [138], and Kitchenham and Charters [140]. In designing the methodology for this SM, methods from several other SMs such as [141143] were also incorporated.


467

The protocol (method) that lies at the basis of this SM is outlined in Fig. 1, which distinguishes ¯ve phases that are described in Secs. 46. All the steps of the research process were carried out manually. Data collection and statistical software tools (e.g., Microsoft Excel and Minitab) were helpful during the process.

3.2. Goal and research questions The approach we have used in our study is the Goal Question Metric (GQM) methodology [144]. Using the GQM's goal template [144], the goal of this study is to systematically review the state-of-the-art in utilizing software engineering (SE) techniques, and methods in the development of scienti¯c software, to classify the articles in this area, to explore the bibliometrics trends in this area, to identify the weaknesses and strengths, and to ¯nd out the recent trends and directions in this ¯eld from the point of view researchers and scientists in this area. In simpler terms, the goal is to understand existing scienti¯c software research area with regard to several attributes (e.g., type of papers, applications, and systems under analysis). We are also interested in understanding how the research has been changing over-time with regards to those research attributes. Based on the above goal, the following research questions are raised. To extract detailed information for each of the questions, each question is divided into subquestions. Metrics used to answer each of the questions are discussed in Sec. 5. .

RQ 1 – Systematic mapping: What is the research space of scienti¯c software development? This question aims at conducting a systematic mapping (classi¯cation) in the area. We raised the following sub-questions for RQ 1 which are similar to other SM studies [138]. ○ RQ 1.1 – Type of paper: What types of papers have been published in the area? This question will enable us to answer whether papers are technical in nature or propose opinions. ○ RQ 1.2 – Scienti¯c disciplines: What scienti¯c disciplines (e.g., chemistry, mathematics, and physics) have been covered more in the articles in this area? SS development is an interdisciplinary research area and by extracting the disciplines which are frequently demanding the development of software systems, more rigorous discipline-dependent practices and frameworks can be suggested to better ¯t this demand. ○ RQ 1.3 – Attributes of the software systems studied: What are the attributes of the systems under study in the papers, such as LOC or programming language? ○ RQ 1.4 – SD/SE activities/techniques: What SD/SE activities/techniques are addressed/discussed in each paper?

.

RQ 2 – Bibliometric data and demographics: What are the trend of publications over years, demographics of the authors/countries and also the

468

R. Farhoodi et al.

bibliometrics data of the papers? Bibliometrics is a set of methods to quantitatively analyze scienti¯c literature. We raised the following sub-questions for RQ 2 which are similar to other bibliometric and demographic studies [133–136]. ○ RQ 2.1 – Publication count by year: What is the annual publication count? ○ RQ 2.2 – Publication count by venue type: What is the annual publication count in di®erent types of venues (conference, journals etc)? ○ RQ 2.3 – Citation count by year: What is the citation count for papers from di®erent years? ○ RQ 2.4 – Top cited papers: What are the top cited papers in this area? ○ RQ 2.5 – Citation count by venue type: What is the citation count for di®erent publication venue types? ○ RQ 2.6 – Citation rates of top venues: What is the citation count for most frequently used venues? ○ RQ 2.7 – Top venues: Which venues had the highest publication count? ○ RQ 2.8 – Top authors: Which authors have been most active? ○ RQ 2.9 – Author a±liation: How many authors belong to academia or industry? ○ RQ 2.10 – Publication count by country: Which countries have been more active?

4. Article Selection For the article selection phase of our SM, we explain the following steps in order: .

Source selection and search keywords (Sec. 4.1). . Exclusion criteria (Sec. 4.2). . Inclusion criteria (Sec. 4.3). . Final pool of articles and the online repository (Sec. 4.4). 4.1. Source selection and search keywords To ¯nd relevant papers, we searched the following online article search engines: (1) IEEE Xplore, (2) ACM Digital Library, (3) Google Scholar, (4) Microsoft Academic Search, (5) CiteSeerX, and (6) Science Direct. In order to ensure that we are including as many relevant publications as possible in the pool of selected papers, we identi¯ed all potential search keywords regarding the focus of each of our research questions. First, we classi¯ed these keywords into two sets of search strings as show in Table 1. The ¯rst set of search strings contained keywords related to scienti¯c software. The second set of search strings contained more software development related keywords. The strings from the ¯rst group were combined with the strings from the second group to form a ¯nal set of comprehensive search strings. We produced combined search strings via concatenation using the AND and OR operator.

Development of Scienti¯c Software Table 1.

469

Our two sets of search keywords.

Set one \scienti¯c software or \computing software or application \scienti¯c computing" application or or program" program" \computational science" \scienti¯c software development" \scienti¯c computation" Set two \requirement" \design" \programming" \validation" \documentation"

\requirement elicitation" \maintenance" \testing" \architecture"

\analysis" \implementation" \veri¯cation", \software engineering"

With these search strings we found 452 publications which we considered as our initial pool of potentially relevant publications. If multiple papers with the same (or similar) title by the same author(s) were found, the most recent one was included and the rest were excluded. This was especially applied for extended journal articles and their earlier conference versions. However, if titles of papers by same author(s) and also their contents were substantially di®erent, we included all of them. Only publications written in English language and only publications which were electronically available were included in this pool. Note that we performed the literature search during the Summer (July until August) 2011 and thus all the publications available by the end of August 2011 were included in our pool. To keep our search and ¯ltering process manageable, we decided to set year 1996 as the lower bound on publication date of papers, and thus our SM study reveals the trends in the last 15 years. 4.2. Exclusion of irrelevant publications The next step in our article selection process was to exclude those papers that were in our pool of 452 papers, but did not relate to the goals of our study. Speci¯cally, our focus is on papers that discuss the application of SE techniques, methods and tools in the development of scienti¯c software, as well as speci¯c conditions and issues observed in the development of scienti¯c software. For inclusion, we considered papers that validated their ¯ndings empirically, and also those which were peer-reviewed. As a result we excluded other kinds of papers, such as papers that only describe a piece of scienti¯c software (e.g., a library or a tool to solve a speci¯c set of di®erential equations). To apply this exclusion criteria to the initial pool, the ¯rst three authors of this article inspected the publications in the initial pool and assigned a vote on a 4-point scale to each publication, with \4" indicating a strong opinion in favor of not excluding a publication, and \1" indicating a strong opinion in favor of excluding a publication. Thus, the maximum vote on a publication could be 12 marks. We decided to use a threshold of 8 marks for the decision on publication exclusion, i.e., publications with cumulative votes of less than 8 marks were excluded. The speci¯c threshold value of 8 marks was chosen since it represented a signi¯cant level of

470

R. Farhoodi et al.

agreement. We reviewed the votes of the authors and threshold value of 6 or 7 did not seem to be good threshold values. To mark each paper we reviewed its title, abstract and keywords. If not enough information could be found in those sources, a more in-depth evaluation was conducted. Based on the results of the joint voting, the size of the pool of selected publications decreased from 452 to 118. We should note that, to increase the preciseness of our study and its results, we utilized systematic voting process among the four authors in the article selection, exclusion and inclusion phases. 4.3. Inclusion of additional publications To decrease the risk of missing relevant publications, similar to previous SM studies and SLRs, we searched the following potential publication venues manually: .

Personal web pages of active researchers in the ¯eld of interest. . References found in publications already in the pool. . Speci¯c venues. Speci¯c venues included journals and proceedings conferences and workshops in the area of scienti¯c software, for example: the Elsevier Journal on Advances in Engineering Software, IEEE Journal on Computing in Science & Engineering, Springer Journal of Scienti¯c Computing and International Workshop on Software Engineering for Computational Science and Engineering. All publications found in the additional venues that were not yet in the pool of selected publications but seemed to be candidates for inclusion were fed into the voting mechanism described in Sec. 4.2. If a candidate publication surpassed the threshold of 8 or more votes, we added the publication to the pool. 4.4. Final pool of articles and the online repository After the initial search and the follow-up analysis for exclusion of irrelevant and inclusion of additional publication, the pool of selected publications comprised was ¯nalized with 130 papers [1, 130] (refer to Sec. 10.1 for the full list of primary studies). The ¯nal pool of selected publications has been published as an online repository using Google Docs, and is accessible publically online at URL [145]. We plan to update the online repository at least once each year and to add relevant material published in the future. The classi¯cations of each selected publication according to the classi¯cation scheme described in Sec. 5 are available in the online repository [145]. 5. Classi¯cation Scheme (Map) Iterative development of our classi¯cation scheme (map) is discussed in Secs. 5.1 and 5.2 presents our metrics and data extraction approach.


471

5.1. Iterative map development To develop our classi¯cation scheme (map), as it is shown in Fig. 1 (discussed in Sec. 3.1), we analyzed the papers in the pool and identi¯ed the initial list of attributes. We then used attribute generalization and iterative re¯nement to derive the ¯le map. To increase the preciseness of our classi¯cation scheme, we utilized the \observer triangulation" method [146] in designing the classi¯cation scheme. As papers were identi¯ed as relevant to our research project, we recorded them in a shared spreadsheet (hosted in the online Google Documents system) to facilitate further analysis. The following information was recorded for each paper: (1) paper title, (3) authors, (2) publication venue (e.g., IEEE Software), (4) year of publication, (5) home country of authors, and (6) type of institution authors are a±liated with (governmental, academic, industry or a combination). With the relevant papers identi¯ed and recorded, our next goal was to categorize the papers in order to begin building a complete picture of the research that has been conducted in the scienti¯c software domain. Though we did not a priori develop a categorization scheme for this project, we were broadly interested in: (1) the methodological techniques reported on by the papers, (2) the scienti¯c software domain of the research, and (3) the software engineering activities and techniques covered in the research reported in the papers. We re¯ned these broad interests into a categorization scheme using an iterative approach that involved all four of the authors of this paper. The ¯rst author of this

ACM Digital Library

IEEE Xplore

Application of exclusion criteria

Relevant articles found in databases (452) Microsoft Academic Search

Articles by browsing personal web pages

Google Scholar

Filtered set (118)

Referenced articles

Application of inclusion criteria

Articles from specific venues

Final selection (130)

Science Direct CiteSeerX

Article selection (Section 4) Attribute Generalization and Iterative Refinement

Final Map

Initial Attributes

Attribute Identification

Classification Scheme/Map (Section 5) Systematic mapping

Systematic Mapping results

Systematic mapping (Section 6) RQ 2 Bibliometrics of the research space

RQ 2 Bibliometrics and Demographic Analysis

Demographics of the research space

Trends, Bibliometrics and Demographics (Sec. 7)

Fig. 1. Our research process.

RQ 1

472

R. Farhoodi et al.

paper conducted an initial pass over the data, and based on (at least) the title, abstract and introduction of the papers created a set of initial categories and assigned papers to those categories. As a group we discussed and reviewed the results of this ¯rst analytic pass and re¯ned the categorization. Next the rest of the researchers conducted a second pass over the data, to revisit the categorization. When the assignment of papers to categories could not be clearly determined just based on the title, abstract and introduction, more of the paper was considered. In this process both the categories and the assignment of papers to categories were further discussed and re¯ned. In the end, every paper was reviewed by at least two researchers. The ¯rst set of categories in our scheme deal with the nature of the research reported on by the paper. These categories were in°uenced by categories described by Petersen et al. [138] and our aim is to provide insights into the empirical foundation being developed by the body of research. The \research type" categories include: .

.

.

. .

. .

. .

Solution Proposal – A paper in this category proposes a solution to a problem. The solution can be either novel or a signi¯cant extension of an existing technique. The potential bene¯ts and the applicability of the solution are shown only by a small example or a good line of argumentation. Validation Research – Involves the investigation of novel techniques or tools that have not yet been implemented in practice. Empirical techniques used are, for example, laboratory experiments. Evaluation Research – These studies go further than \Validation Research" by implementing and evaluating techniques or tools in practice. These studies demonstrate how the technique is implemented in practice and explore the consequences of the implementation in terms of bene¯ts and drawbacks. Philosophical Papers – These papers sketch a new way of looking at existing things by structuring the ¯eld in form of a taxonomy or conceptual framework. Opinion Papers – These papers express the personal opinion of the author(s) around whether a certain technique is good or bad, or how things should been done. They do not signi¯cantly rely on related work or research methodologies. Experience Papers – Experience papers explain how something has been done in practice, based on the personal experience of the author(s). Field Study – A study that aims to gather data about software engineering in real scienti¯c software environments. Examples data collection methods used include questionnaires, interviews, focus groups, and participant observation. Secondary Study – A study that uses data from secondary sources. Examples include systematic literature reviews, literature surveys, and systematic mappings. Other – A catchall category in the event that the work reported in a paper does not ¯t into any of the above research types.

The second set of categories in our scheme deal with the scienti¯c software domain under study, with the goal of comparing how the body of research is distributed


473

across various disciplines. The following is the list the disciplines we identi¯ed in the papers we have reviewed: . . . . . . . . . . .

Math Physics Biology Chemistry Medical Bio-medical Engineering Geo (Earth) Science High Performance Computing (HPC) General Other

The ¯nal set of categories in our schema deal with the software engineering activities and tools covered in the research reported in the papers. We were interested in the programming language used in the scienti¯c software systems investigated (if any) and more especially in what SE activities have been well studied in the scienti¯c software context and which have been, comparatively, neglected. Through iterative re¯nement, we identi¯ed the following list of SE activities/techniques as the best category for this aspect of our mapping: . . . . . . . . .

Requirements Engineering and Analysis Architecture and Design Development (Programming) Testing, Veri¯cation and Validation and Quality Activities Maintenance and Evolution Process/Project Management Agile Development Practices Con¯guration Management Run-time Performance

As an example, one of the papers in our pool is \ The T experiments: Errors in scienti¯c software" [100] which has had the highest number of citations (83) as of this writing (April 2012) among all papers in our pool. This paper was categorized as follows: .

Research type: categorized under evaluation research (empirical study) since they study presents comprehensive empirical study with details statistic, etc. . Scienti¯c discipline: engineering and geo- (earth) science . Software systems studied and their attributes: LOC: 5,233,639, number of systems: 123, programming languages: C and Fortran . SE activities/techniques covered/used in the paper: testing, V&V, and quality

474

R. Farhoodi et al. Table 2. Metrics extracted from the papers. RQs

Metrics

RQ 1 Systematic mapping RQ 1.1 Type of paper RQ 1.2 Scienti¯c discipline RQ 1.3 Software systems studied RQ 1.4 SD/SE activities/ techniques

Research type Scienti¯c discipline discussed in paper LOC, number of systems, programming language SD/SE activities/techniques used in paper

RQ 2 Bibliometric data and demographics

publication year

RQ 2.1 Publication count by year RQ 2.2 Publication count by venue type RQ 2.3 Citation count by venue type RQ 2.4 Citation count by year RQ 2.5 Top venues RQ 2.6 Citation rates of top venues RQ 2.7 Top authors RQ 2.8 Author a±liation RQ 2.9 Publication count by country

publication Venue Citation count, publication Venue Citation count, publication year publication Venue Citation count, publication Venue Names of authors Author a±liation (university, government, industry) Countries of authors' a±liation

By extracting the size of the software systems in the publications, a general overview on the scale of scienti¯c software in the publications can be achieved. This can motivate the development of speci¯c practices to ¯t scienti¯c software development based on the context in which they are expected to be used, e.g., large-scale software systems need di®erent considerations in their development process compared to mid/ small-scale systems.

5.2. Metrics and data extraction Based on our map, we were then able to de¯ne the metrics we need to extract to answer each of the RQs. The metrics/data we extracted from papers are listed in Table 2. To extract data, the papers in our pool were reviewed with the focus of each research question and the required information was extracted. To increase the preciseness of our classi¯cation scheme, we utilized the \observer triangulation" method [146] in designing the classi¯cation scheme and data extraction (mapping) phases.

6. Results of Systematic Mapping (RQ1) This section presents results related to our ¯rst research question (RQ1). It aims at characterizing the research space related to the development of scienti¯c software


475

and, thus, RQ1 contains the systematic mapping component of our study. The subquestions under RQ1 are described in the following one by one.

6.1. Type of paper (RQ 1.1) Figure 2 shows the classi¯cation of the 130 selected publications [1130] according to the type of research they report. Exact paper references have also been provided in the right side. Since some publications deal with more than one system or application or type of study, in some cases publications were categorized in multiple categories, yielding a total of 148 di®erent instances of research type classi¯cations. More than one third of all classi¯cations (55 papers) relate to presentations of solution proposals without further validation or evaluation. A similar number of classi¯cations (56 papers) relate to presentations of either the authors' experience with various approaches to the development of scienti¯c software (28 papers) or results of ¯eld studies (28 papers) reporting such approaches applied by others in real-world development organizations. The remaining classi¯cations (33 papers) relate mostly to opinion papers (18 papers) or to reports of validation research (10 papers) and evaluation research (5 papers). The numbers of secondary studies (3 papers) and philosophical papers (only 1 paper) are very small. The distribution of publications with regards to type indicates that research about the development of scienti¯c software is not yet mature, since the number of publications that report about validation and evaluation of solution proposals is rather small. On the other hand, we found a substantial number of experience papers, ¯eld studies, and secondary studies, indicating that some work has been done to better understand the speci¯cs of developing scienti¯c software. We assume that publications presenting solution proposals are to some degree based in an understanding of the speci¯c conditions and problems related to the development of scienti¯c software.

Fig. 2. Frequency of research types mentioned in publications.

476

R. Farhoodi et al.

Fig. 3. Trend of research types mentioned in publications over time.

Figure 3 shows how the publication years of papers included in our study evolved per type during the past 15 years. The numbers indicate that interest in research about software development practices for scienti¯c software strongly increased beginning with year 2004. 6.2. Scienti¯c disciplines of scienti¯c software (RQ 1.2) We categorized the publications with respect to their scienti¯c disciplines. 29 publications did not mention a particular scienti¯c discipline. From the 101 publications in which at least one discipline was mentioned, the distribution is shown in Fig. 4. Since several publications mentioned more than one discipline, the numbers in Fig. 5 add up to 139. By far, the most frequently researched scienti¯c discipline is physics (37 papers), followed by mathematics (20 papers), biology (14 papers) and highperformance computing (HPC) (14 papers). 6.3. Software systems studied (RQ 1.3) Where a given primary study provided information about the software system(s) under study, we extracted the following three pieces of information: (1) Number of scienti¯c systems discussed in each paper, (2) Size in LOC, and (3) Programming languages used for development of the system(s). Characterizing the primary studies by the number and size of scienti¯c systems discussed in each paper will enable to assess their level of evidence and experience. For example, a given paper which quantitatively and qualitatively discusses the challenges of requirements engineering


477

Fig. 4. Frequency of scienti¯c disciplines mentioned in publications.

in the context of scienti¯c software with experience based on 100 large-scale systems will provide higher degree of evidence than another paper which reports similar challenges based on one system only.

6.3.1. Number of systems discussed Only 28 publications explicitly inform about speci¯c software scienti¯c systems they are discussing. In eleven cases, publications discuss more than one software system. Figure 5 shows whether selected publications discuss one or several systems. As can

Fig. 5. Number of systems discussed in publications.

478

R. Farhoodi et al.

Fig. 6. Frequency of programming languages mentioned in publications.

be seen, the majority of publications (17) discuss exactly one system. One publication mentions 123 di®erent systems from 73 di®erent organizations, and two publications mention 7 di®erent systems. When looking at these numbers, one needs to keep in mind, however, that some publications are referring to software systems that are also mentioned in other publications. Hence, Fig. 6 contains duplicates.

6.3.2. Programming languages used Of the 130 selected publications, 97 informed about the programming language that speci¯c systems used or for which speci¯c solution proposals were made or other kinds of analyses were conducted. In total, we found 128 occurrences where programming languages used in scienti¯c software were mentioned in the selected publications. Figure 6 shows the frequencies with which di®erent programming languages were mentioned. According to this data, the majority of scienti¯c software systems were written in Fortran followed by Cþþ, C, Python, Java and Matlab. In the category \other", the languages Perl and Visual Basic (VB) were mentioned most frequently. We were interested in ¯nding trends related to the usage of programming languages in scienti¯c software systems. Thus, we plotted the distribution of programming language occurrences in the selected publications over time. Although there are no clear trends, Fig. 7 indicates that both Fortran and Cþþ are mentioned throughout the whole period of consideration (19962011), while programming languages such as Java, Matlab and Python (with one outlier in 1997) have only


479

Fig. 7. Trend of programming languages mentioned in publications over years.

been mentioned since 2003. The programming language C has not been mentioned during the last two years (20102011).

6.3.3. Size in LOC For those publications that mentioned the size of the software systems discussed, we calculated the average lines of code (LOC) per system. Figure 8 shows the average number of LOCs of systems mentioned per publication over time. We had to take the average values since many papers did not provide the exact LOC of all systems they had analyzed, but rather reported a LOC range, e.g., [50] reported analysis of 5 systems ranging between 100K500K LOC. Thus, for example, if a publication mentions LOC data of two systems, then the average LOC data is presented in Fig. 8. More recent papers are more frequently reporting LOC data, and the average LOC data reported per publication is less than 1 million LOC. One publication reports on a system with more than 10 million LOC [79]. The system is called ATLAS, software developed for particle accelerator experiments at the Large Hadron Collider (LHC) located at CERN, Geneva, Switzerland. The system is written in Cþþ and Python. Since the same paper mentions another system of size 140,000 LOC, the related data point in Fig. 9 reports an average of 5,070,000 LOC. One publication reports about two sets of systems with a total of more than 5 million LOC [100]. For two programming languages, Fortran and C, many code packages were collected from many companies, adding up to 3,305,628 LOC Fortran code and

480

R. Farhoodi et al.

Fig. 8. Tend of the size (in LOC) of the software systems under study over years.

1,928,011 LOC C code. The Fortran and C codes stem from 47 (for Fortran) and 26 (for C) di®erent organizations, with package sizes ranging from 806 to 770,444 (for Fortran, total of 55 packages, average package size 60,102 LOC) and 361 to 431,655 (for C, total of 68 packages, average package size 28,353 LOC). Due to the large number of di®erent systems, the average size shown in Fig. 9 is only 42,550 LOC (for both programming languages taken together).

Fig. 9. Frequency of SD/SE activities/techniques mentioned in publications.


481

6.4. Software development/engineering activities or techniques (RQ 1.4) We also tried to understand which software development (SD) and software engineering (SE) activities and techniques were addressed in the selected publications. Figure 9 shows how often speci¯c SD/SE activities/techniques were mentioned in the 130 selected publications. Most of the publications discuss aspects related to architecture/design, development/coding and testing/veri¯cation/validation/ quality. A smaller share of publications mentions aspects related to requirements engineering and software maintenance/evolution. Generally, management activities are underrepresented as well as modern development techniques (i.e., agile development) and speci¯c topics such as software performance. In Ref. 63 the authors found that there is a weak correlation between the size of a scienti¯c software development project and the importance that is attributed to project and product management. Since most of the projects discussed in the publications contained in our repository are small, this might explain why management-related activities are underrepresented. What surprised us is that \performance" is only mentioned three times. One possible explanation for this might be that performance is so much at the core of scienti¯c software development that it is not explicitly mentioned. Note that, to keep the article size reasonable, we have grouped the SD/SE techniques discussed in the papers as shown in Fig. 9. Interested readers can access the online repository to read more details about the exact SD/SE techniques used/ proposed in each paper (e.g., the types of architecture and design techniques), as the authors have extracted and provided those details online.

Fig. 10. Trend of top three SD/SE activities/techniques mentioned in publications over years.

482

R. Farhoodi et al.

Figure 10 shows how the mentioning of the top three SD/SE activities/techniques evolved over time. The data seems to indicate that the proportions between the occurrences of the top three activities/techniques have not signi¯cantly changed in the period 1996l2011.

7. Trends, Bibliometrics and Demographics (RQ 2) This section presents results related to our second research question (RQ2). It aims at characterizing demographics of the selected publications as well as bibliometric data. Bibliometrics comprises a set of methods to analyze scienti¯c and technological literature [147]. In particular, it utilizes quantitative analysis and statistics to describe patterns of publication within a given ¯eld or body of literature. Bibliometric data such as the ranking of authors according to the number of their publication in the ¯eld of scienti¯c software development can potentially be used by researchers and grad students as an entry point for literature search and establishing promising contacts and, perhaps, collaborations [56]. The answers to sub-questions under RQ2 are described in the following one by one.

7.1. Publication count by year (RQ 2.1) Figure 11 shows the distribution of the selected 130 publications over the period 1996 to 2011. The data clearly shows that the number of publications in the second half of the period is signi¯cantly higher than in the ¯rst half. This indicates that the topic \development of scienti¯c software" has gained popularity and importance.

7.2. Publication count by venue type (RQ 2.2) Figure 12 shows the distribution of the 130 selected publications over types of publication venues. Almost one third of the selected publications (35) are journal articles. The majority of publications, however, are workshop papers (42) and conference papers (30). Smaller numbers of publications appeared in magazines and proceedings of symposia.

Fig. 11. Frequency of publications over time.


483

Fig. 12. Frequency of publication per venue type.

7.3. Citation analysis 7.3.1. Citation count by year (RQ 2.3) Figure 13 shows the distribution of citations of the 130 publications over the period 1996 to 2011. The citation numbers were taken on January 25, 2012 with Google Scholar. There were two publications [19], and [45], which have been cited an astonishingly 3,712 and 448 times, respectively. They will be discussed in Sec. 7.3.2. In order to make the charts more readable, the data reported in Fig. 13 does not include these \outlier" points. Figure 14 shows the histogram of number of citations to 128 papers in the pool, excluding the two highly-cited publications [19, 45]. We can see that 26 articles have

Fig. 13. Citation count of selected publications over time.

484

R. Farhoodi et al.

25

Frequency

20

15

10

5

0

0

15

30 45 60 75 Number of citations (as of Jan. 25 2012)

90

Fig. 14. Histogram of number of citations to 128 papers in the pool (excluding two highly-cited publications [19, 45]).

had no citations. 11 and 24 articles were cited 1 and 2 times, respectively. The average number of citations to the pool articles by excluding and including the two highly-cited publications were 9.60 and 41.46, respectively. 7.3.2. Most cited papers (RQ 2.4) The list of top 10 cited papers are shown in Table 3 (ordered by number of citations). The mapping of each article based on our classi¯cation scheme (Sec. 5.1) is also shown. The 2004 paper [19] written by 25 authors is cited 3,712 times. This paper reports an open software development platform called Bioconductor (www.bioconductor. org) for computational biology and bioinformatics. It is a collaboration among researchers from the Fred Hutchinson Cancer Research Center (located in Seattle, Washington, USA) as the core team, and researchers from various US and international institutions. The goals of the project include: fostering collaborative development and widespread use of innovative software, reducing barriers to entry into interdisciplinary scienti¯c research, and promoting the achievement of remote reproducibility of research results. The authors describe details of their aims and methods, identify current challenges, compare Bioconductor to other open bioinformatics projects, and provide working examples. Bioconductor has been operational since 2001 and since then it has become a prominent software project for computational biology and bioinformatics. The authors of [19] argue that the success of the project is due to many factors. These include the choice of R [148] as the main programming language, the adoption of standard practices of software design and a belief that the creation of software infrastructure is an important and essential

3712

448

81

64 62 60

48 47

42

41

[19]

[45]

[100]

[53] [84] [108]

[30] [94]

[93]

[12]

Article

Number of citations

2005

2007

1997 2004

1997 2005 2003

1997

1999

2004

Year published

Field study

Solution proposal Experience paper, lessons learned Series of ¯eld study

Validation research Solution proposal Experience paper

Empirical study

Solution proposal

Validation research, experience report

Research type

Manufacturing, signal processing, product performance Space

Physics (combustion, micro-tomography) 40 application areas (e.g., nuclear engineering, chemical engineering, civil engineering) General HPC Mechanics (hyper-velocity vehicles) Math Physics

Biology

Scienti¯c discipline

Cþþ, Fortran, Matlab, Python, Perl Not reported

5/317,800

Not reported

Cþþ Not reported

Cþþ, Fortran Veri¯cation Fortran, Ruby

C, Fortran

C, Fortran 77

R

Programming language(s)

Not reported 7/509,710

Not reported Not reported 1/500,000

123/5,233,639

Not reported

Not reported

Number/and Scale (LOC) of system(s) studied

Table 3. Table 3-top 10 cited papers.

Requirements, architecture, development, testing

Process, project management Architecture and design Quality engineering, project management Veri¯cation and validation

Development, performance

Testing/ V&V/Quality

Architecture, design, development, Prototyping capabilities, Designing by contract Architecture, design

SD/SE activity/technique(s)

Development of Scienti¯c Software 485

486

R. Farhoodi et al.

component of a successful project of this size. In terms of SD/SE activities and techniques, the authors explain the architecture and design of the frameworks and provide examples on the development, prototyping capabilities, and design by contract in Bioconductor. The 1999 paper [45] written by 8 authors is cited 448 times. It reports a work-inprogress e®ort to develop a common component architecture for high-performance scienti¯c computing. It is a collaboration among several American governmental laboratories (e.g., Sandia and Oak Ridge National Laboratories) and universities (e.g., Indiana University and University of Utah). This research stems from growing recognition that the scienti¯c community needs to better manage the complexity of multidisciplinary simulations and better address scalable performance issues on parallel and distributed architectures. Driving forces are the need for fast connections among components that perform numerically intensive work and for parallel collective interactions among components that use multiple processes or threads. This paper focuses on the areas we believe are most crucial in this context, namely, an interface de¯nition language that supports scienti¯c abstractions for specifying component interfaces and a ports connection model for specifying component interactions. The third-highly cited paper (81 citations) is a 1997 paper [100] and has a single author. It is a statistical work in the context of scienti¯c software. As the author reports, between 1990 and 1994, with the help of a collaborator and the cooperation of many commercial software vendors, he conducted two large experiments to test whether science and engineering software really does, accurately, what it is supposed to do. Together, he called those investigations the T experiments. Experiment T1 measured defects without running programs. Speci¯cally, it gauged the formal consistency of several million lines of scienti¯c software written in C and Fortran 77, from many di®erent industries and application areas, by a technique called static deep-°ow analysis [149]. Experiment T2 measured, within one of these industrial application areas, the extent to which several independent implementations of the same algorithms acting on the same input data with the same parameters-gave the same results when the authors ran the programs. This article [100] was a result of analyzing a large code-base (in total 5,233,639 LOC) consisting of 3,305,628 LOC of Fortran 66/77 and 1,928,011 LOC in C, from 73 organizations. Experiment T1 showed that C and Fortran code, independent of the application area, are riddled with statically detectable faults [100]. For example, calling-sequence inconsistencies (also known as interface inconsistencies) occurred at the average rate of one in every 7 interfaces in the Fortran code they tested, and one in every 37 interfaces in C. The authors also found that Fortran components are typically 2.5 times as large as C components, but interestingly, also had 2.5 times as many arguments; and that roughly 30 percent of the Fortran code analyzed and 10 percent of the C codes would be deemed untestable. Experiment T2 was even more disturbing. Scientists like to think that their results are accurate to the precision of the arithmetic used. In this study, however, the level of agreement among the results


487

Fig. 15. Citation count by venue type (excluding two highly-cited publications [19, 45]).

from similarly implemented code gradually degenerated from 6 signi¯cant ¯gures to 1 during the computation. The authors concluded that the reasons for this disagreement can be laid squarely at the door of software failure, as other possible causes are considered and rejected. 7.3.3. Citation count by venue type (RQ 2.5) Figure 15 shows the distribution of the aggregated citation count of the 130 selected publications over types of publication venues. Publications in journals and magazines have the highest total citation numbers, closely followed by workshop publications. This indicates that publications in magazines have the highest since the total number of publications for magazines is substantially smaller than for journals and workshops. This observation can also be made based on the data shown in Fig. 16 where the average citation rates per publication in each of the venue types are shown. The citation numbers were extracted on January 25, 2012 from the Google Scholar service. As mentioned above, two publications, [19] and [45], have been cited an astonishingly 3,712 and 448 times, respectively. In order to make the charts more readable, the data reported in Figs. 15 and 16 does not include these \outliers".

Fig. 16. Average citation count by venue type (excluding two highly-cited publications [19, 45]).

488

R. Farhoodi et al.

Fig. 17. Citation count by venue type (including two highly-cited publications [19, 45]).

Figures 17 and 18 correspond to Figs. 15 and 16 with the only di®erence that they include the two highly cited publications.

7.3.4. Citation rates of top venues (RQ 2.6) Figure 19 shows the distribution of citations over the period 1996 to 2011 for those venues with at least two publications from our selection of 130 publications. The ranking of the venues is according to the number of citations per publication. It should be noted that the publications with the highest citation numbers, [19] and [45], are not included in Fig. 19 because each has been published at venue at which no other publication of our selection of 130 publications has been published. Figure 19 indicates that the magazines IEEE Software and BMC Bioinformatics have the highest citation rates per publication and the highest total numbers of citations of all venues with at least two publications. Figure 20 shows the distributions of citations per venue type. Note that again only venues with at least two publications are included in each category and thus, again, the two publications with the highest citations are not included. Based on the data shown in Fig. 20, magazine publications have the highest impact in terms of citations, followed by journal publications. It is interesting to see that workshop papers

Fig. 18. Average citation count by venue type (including two highly-cited publications [19, 45]).


489

Fig. 19. Citation count of selected publications distributed over venues with at least two publications.

have slightly higher impacts than conference papers. Note that the upper and lower boundaries of the boxes in the box-plots represent 75% and 25% quartiles, respectively. The line inside the boxes represents the 50% quartile (median), which is di®erent from the average citation numbers (means) shown in Fig. 16.

Fig. 20. Distribution of citation counts per venue type.

490

R. Farhoodi et al. Table 4.

Top venues (with at least two papers in the pool).

Venue name

Acronym

Number of papers from this venue in our pool

International Workshop on Software Engineering for Computational Science and Engineering International Workshop on Software Engineering for High Performance Computing Applications IEEE Magazine on Computing in Science and Engineering International Conference on Software Engineering IEEE Software ACM Transactions on Mathematical Software International Conference on Computational Science BMC Bioinformatics Working Conference on the Architecture of Scienti¯c Software Proceedings of the ACM Conference on Computer Supported Cooperative Work Procedia Computer Science (Conference Proceedings) Scienti¯c Programming - Complexity in Scalable Computing Advances in Engineering Software Journal of Computational Science

SECSE

18

SEHPCA

17

CISE ICSE IEEES TOMS ICCS BMCB CASS CSCW

9 9 8 4 4 2 2 2

PCS SPCSC AES JCS

2 2 2 2

7.4. Top venues (RQ 2.7) Table 4 shows the distribution of selected publications over publication venues that attracted at least two publications. The most popular publication venues are the \International Workshop on Software Engineering for Computational Science and Engineering" (SECSE) and the \International Workshop on Software Engineering for High Performance Computing Applications" (SEHPCA). The SECSE workshop is linked to the \International Conference on Software Engineering" (ICSE) which also attracted a substantial number of the selected publications. The most frequently used magazines are \IEEE Computing in Science and Engineering" (CISE) and \IEEE Software" (IEEES).

7.5. Top authors (RQ 2.8) Figure 21 shows the ranking of researchers who authored (or co-authored) two or more of the 130 selected publications. Diane Kelly, a±liated with the Royal Military College of Canada is the most active author with a total of ten publications [6, 61, 80, 83, 89, 91, 92, 98, 99, 103], followed by Judith Segal of Open University, UK, with seven papers [10, 12, 96, 110, 111, 125, 129]. Richard Kendall and his colleague Douglas E. Post, from the American Los Alamos National Laboratory, with six publications each [20, 81, 93, 94, 112, 114, 150] stand in the 3rd rank. It would be worthwhile to review the major research areas of the top researchers. We have summarized the main research areas of the top three-ranked authors in Table 5. It seems that among the top active areas of the top authors are software process, testing, quality, and project management in the context of scienti¯c software.


491

Fig. 21. Most active researchers.

7.6. Author a±liations (RQ 2.9) Figure 22 shows the distribution of authors of the 130 selected papers over type of a±liation. We de¯ned four categories of a±liations: (1) governmental research groups such as NASA, (2) universities, (3) corporate research groups and laboratories in industry, and (4) collaborative work (joint research published by groups of

492

R. Farhoodi et al. Table 5. Main research areas of the top three-ranked authors.

Rank

Researcher

1

Diane Kelly

2

Judith Segal

3 (tie)

Richard Kendall

3 (tie)

Douglas E. Post (a co-author of Richard Kendall)

Areas

Paper references

Testing Risk analysis Evolution Testing, quality Process, project management Design Requirements Process, project management Testing, quality Design Performance Same areas as Richard Kendall Process, project management

[61, 80, 83, 89, 91, 92, 98, 99] [6] [103] [10, 12, 96, 125] [110, 111, 129] [10, 12] [10, 12] [20, 81, 94, 150] [81, 93, 94] [20] [114] [20, 81, 93, 94, 150] [112]

Fig. 22. Author a±liations by type.

authors a±liated with two or three di®erent categories). The ¯gure indicates that publications where all authors are a±liated with universities are the most frequent in our set of selected publications, followed by publications written by author groups having mixed a±liation categories. Figure 23 shows the distribution of types of author a±liations in the period 1996 to 2011. Generally, the proportions of a±liation types vary much from year to year but we couldn't ¯nd any speci¯c pattern of change. It seems, however, that in the second half of the period, i.e., beginning with year 2004, the portion of publications produced by authors exclusively a±liated with universities is larger than in the ¯rst half.


493

Fig. 23. Trend of the types of author a±liations over years.

7.7. Publication count by country (RQ 2.10) Figure 24 shows the distribution of the authors' a±liations by country. Since authors of a publication can have a±liations from di®erent countries, the sum of the numbers shown in the bar chart is larger than 130. As Fig. 24 indicates, authors of the selected publications are a±liated with organizations from 17 di®erent countries. The USA, Canada and the UK are the top 3 contributing counties, followed by Germany, Australia, Switzerland and Norway. The majority of publications stem from North America and Europe and from Anglo-Saxon countries generally. In this regard, our sample may be biased as we only searched for publications written in English.

Fig. 24. Researchers' a±liations by country.

494

R. Farhoodi et al.

Seven papers are the result of international collaborations between the following countries: . . . . . . .

UK, US [14] Norway, Sweden [139] US, Switzerland, Denmark, Germany, Australia [19] US, Norway [22] Canada, UK [59] Norway, Canada [63] Switzerland, US [32]

8. Threats to Validity The discussion of threats to validity is important to judge the strengths and limitations of our SM study with regard to the validity of the results. For our study, the following issues may induce threats to validity: selection of search databases, de¯nition of search terms and time frame, researcher bias with regards to exclusion/ inclusion, and incorrect data extraction (classi¯cation). Using the standard classi¯cation scheme of validity threats suggested in [151], we discuss these issues in relation to four types of threats to validity: (1) conclusion validity, (2) construct validity, (3) internal validity and (4) external validity.

8.1. Conclusion validity In order to present results that are reasonable and can be reproduced by other researchers, detailed information regarding the concrete search term and the databases used is presented. Furthermore, we describe the process of exclusion and inclusion of relevant publications and the categories used for the classi¯cation of publications. Since the decision which articles to include or exclude as well as to how they should be categorized strongly depended on the judgment of the researchers conducting the study, we applied strategies to increase the reliability of our decisions. With regard to exclusion/inclusion decisions, in order to reduce researcher bias, we had three researchers looking at the candidate publications independently and used the voting mechanism described in Sec. 4. With regards to data extraction we split the ¯nal pool of publications in three equally large sub-pools and assigned one researcher to each sub-pool plus an additional reviewer. In case of disagreement about a speci¯c classi¯cation decision, a second reviewer was involved. In all cases, this procedure yielded consensus. With these procedures in place, we trust that a replication of our SM study will not result in a signi¯cantly di®erent set of relevant publications nor do we expect major deviations from our classi¯cation decisions.


495

8.2. Construct validity Construct validity re°ects to what extent the object of study really represents what the researchers have in mind when investigating their research question. In order to obtain as complete as possible set of relevant publications, the search terms chosen need to be appropriate. As described in Sec. 4, we tried to be as inclusive as possible using a systematic procedure to de¯ne di®erent terms for scienti¯c software as well as for software development activities and SE techniques, methods and tools. This resulted in a relatively high number of false positives which had to be removed manually by the researchers but it makes us believe that we did not miss too many relevant publications. More problematic than the de¯nition of search terms was the de¯nition of the categorization scheme for the data extraction. In order to ¯nd a joint understanding of the various terms in our classi¯cation scheme we had several review rounds involving all authors of the study. This, and the fact that all of us have been working as SE researchers for a long time makes us trust that we de¯ned the classi¯cation scheme in an unambiguous way that is conforming with the terminology typically used in this ¯eld of research.

8.3. Internal validity Internal validity is concerned with drawing correct conclusions about causal relationships between study treatment and outcome. Since we only present descriptive statistics we think that threats to internal validity are small.

8.4. External validity External validity is concerned with the question whether the results of our SM study can be generalized. Since we do not intend to generalize our ¯ndings beyond the ¯eld of scienti¯c computing, we don't think that any threats exist in this regard. However, we must keep in mind that we restricted our search to publications written in English and where an electronic text ¯le was readily available online. Thus, the population from which we selected publications excludes potentially relevant databases containing publications written in other languages as well as repositories of gray literature (i.e., publicly available texts that have not been quality-controlled by commercial publishers such as, e.g., technical reports) and books. On the other hand, we think that the population we chose to search from is large enough as to be representative for all types of literature on the development of scienti¯c software worldwide. We have no reason to believe that accounts about the development of scienti¯c software would contain signi¯cantly di®erent information if reported by researchers and professionals living and working in non-English speaking communities and cultures.

496

R. Farhoodi et al.

9. Summary, Conclusion and Future Work In our mapping study, we answered four detailed research questions (RQ 1.1 to RQ 1.4). The main results are as follows: .

RQ 1.1. Regarding type of research, the 130 selected publications can be grouped into three clusters of approximately equal size. The ¯rst cluster contains \solution proposals". The second cluster contains \experience papers" and \¯eld studies" (both equally frequent). The third cluster contains \opinion papers", \validation research", \evaluation research", \secondary studies", and \philosophical papers" (with decreasing frequency). . RQ 1.2. The scienti¯c discipline most frequently mentioned in our pool of selected publications is \physics". . RQ 1.3. The programming languages most frequently mentioned in our pool of selected publications are \Fortran" and \Cþþ". . RQ 1.4. The software development activities and techniques most frequently mention in our pool of selected publications are \development/coding", \architecture and design", and \testing/V&V/quality assurance". The collection of bibliographic and demographic data helped us answer nine detailed research questions (RQ 2.1 to RQ 2.10). The main results are as follows: . . .

. . .

. . .

RQ 2.1. In our search period (1996–2011) we observed a sharp increase of relevant publications beginning with year 2004, reaching a peak in year 2008. RQ 2.2. The type of publication venue most frequently occurring in our pool of selected publications is \workshop". RQs 2.3 and 2.4. The two most frequently cited publications in our pool were cited 3,712 times [19] and 448 times [45], respectively. The rest of the publications were cited on average between 19 times (magazine articles) and 5 times (conference papers). We used Google scholar to collect citation counts. RQ 2.5. In total, 17 out of 130 publications were cited more than 25 times. RQ 2.6. The publication venue with both the highest number of citations and the highest number of citations per publication is the magazine \IEEE Software". RQ 2.7. The publication venues most frequently occurring in our pool of 130 selected publications are \International Workshop on Software Engineering for Computational Science and Engineering" (18 publications) and \International Workshop on Software Engineering for High Performance Computing Applications" (17 publications). RQ 2.8. The most active author in the ¯eld is Diane Kelly, Royal Military Collage of Canada, with a total of ten (co-authored) publications. RQ 2.9. Most of the authors in our pool of selected publications are a±liated with a university (43.6%). RQ 2.10. The authors' most frequent a±liations are located in the US (approximately 50%), followed with a large distance by Canada and the UK.


497

As mentioned in Sec. 3.2, one of the objectives of this SM was to provide an overview of existing approaches in this area in order to identify existing gaps and to establish future research directions. The main stakeholders in developing scienti¯c software which would bene¯t from the results of our study are (1) software engineering researchers and (2) developers of scienti¯c software and scientists (those working in scienti¯c areas considered in the domain of this study, e.g., physics, mathematics, etc.). For example, by the results of the analysis done on the SD/SE activities/techniques covered in papers, the trends seem to denote that the area such as maintenance, evolution, refactoring, re(v)engineering, process/project management of scienti¯c software have not been explored as much as other area. Thus, these areas need more work by researchers and scientists. Less than a third of the papers in our pool (33 out of 130) were based on experience and/or analysis of actual software systems. Although opinion papers based on experience are valuable, however it is important that in the articles in this area, researchers and scientists report the details of the actual software systems they have worked on. In terms of scienti¯c discipline, physics, math and biology were the top three disciplines with the highest number of publications in this area, which denotes that SE researchers and scientists in those areas have already good scale of collaborations. However, the three disciplines: bio-medical, geo (earth), and medical science had the three lowest numbers of publications with 6, 6, 4 papers only, respectively. As there are vast numbers of software applications developed in these disciplines, it is imperative to expect to see increasing level of collaborations between SE researchers and scientists in those disciplines in future. The author a±liation data showed reasonable share of joint works between academia and industry (36 of 130 papers). However, the higher this number, the better. There were only 8 articles where all authors were industry. It is important to attract and encourage more industry authors to publish their work in the venues in this area. In conclusion, the contributions of this article were two-fold (1) the results of a SM study regarding the use of software engineering techniques and activities in the development of scienti¯c software, and (2) a collection of bibliographic and demographic data of the publications selected in our mapping study. From our point of view, the most interesting result of the SM is the classi¯cation of publications regarding the types of scienti¯c software and the types of development activities and techniques they mention. The results of our study seem to suggest that many software engineering techniques and activities are being used in development of scienti¯c software. However, there is still a need for further exploration of the usefulness of speci¯c software engineering techniques, e.g., regarding software maintenance, evolution, refactoring, re(v)-engineering, process and project management.

498

R. Farhoodi et al.

In the near future, we plan to conduct a Systematic Literature Review (SLR) on the use of selected software engineering techniques for the development of scienti¯c software, which will comprise a more-in-depth analysis compared to this SM. Acknowledgments Vahid Garousi and Roshanak Farhoodi were supported by the Alberta Ingenuity New Faculty Award No. 200600673 through the grant entitled: \Engineering Intelligent Software Systems for Improving the Operational E±ciency of Oil Pipeline Networks". The Alberta Ingenuity agency is now called Alberta Innovates Technology Futures. Vahid Garousi was also supported by the Visiting Scientist Fellowship Program (#2221) of the Scienti¯c and Technological Research Council of € Turkey (TUBİTAK). References Primary studies 1. S. Smith and W. Yu, A document driven methodology for developing a high quality Parallel Mesh Generation Toolbox, Advances in Engineering Software 40(11) (2009) 11551167. 2. S. Smith and L. Lai, A new requirements template for scienti¯c computing, in International Workshop on Situational Requirements Engineering Processes Methods, Techniques and Tools to Support Situation-Speci¯c Requirements Engineering Processes, 2005. 3. D. W. Kane, M. M. Hohman, E. G. Cerami, M. W. Mccormick, K. F. Kuhlmman and J. A. Byrd, Agile methods in biomedical software development: A multi-site experience report, Bioinformatics 7(1) (2006) 273285. 4. H.-t. Li, An Ontology-based process model for scienti¯c research software development, in International Conference on Software Engineering and Data Mining, 2010, pp. 299304. 5. W. S. Smith, J. Carette and J. McCutchan, Commonality analysis of families of physical models for use in scienti¯c computing, in International Workshop on Software Engineering for Computational Science and Engineering, 2008. 6. R. Sanders and D. Kelly, Dealing with risk in scienti¯c software development, IEEE Software 25(4) (2008) 2128. 7. M. A. Heroux, J. M. Willenbring and M. N. Phenow, Improving the development process for CSE software, in International Conference on Parallel, Distributed and NetworkBased Processing, 2007. 8. S. Larson and B. Morrison, Managing software requirements in the context of the scienti¯c enterprise, in IEEE Aerospace Conference, 2000. 9. C. J. Roy, Practical software engineering strategies for scienti¯c computing, in AIAA Computational Fluid Dynamics Conference, 2009. 10. J. Segal, Some challenges facing software engineers developing software for scientists, in Workshop on Software Engineering for Computational Science and Engineering, 2009. 11. S. Smith, Systematic development of requirements documentation for general purpose scienti¯c computing software, in Proc. of IEEE International Conf. on Requirement Engineering, 2006, pp. 205215.


499

12. J. Segal, When software engineers met research scientists: A case study, Empirical Software Engineering 10(1) (2005) 517536. 13. J. P. Kenny, C. L. Janssen, M. S. Gordon, M. Sosonkina and T. L. Windus, A component approach to collaborative scienti¯c software development: Tools and techniques utilized by the quantum chemistry science application partnership, Scienti¯c Programming 16(4) (2008) 287296. 14. R. H. Fogh, W. Boucher, W. F. Vranken, A. Pajon, T. J. Stevens, T. N. Bhat, J. Westbrook, J. M. C. Ionides and E. D. Laue, A framework for scienti¯c data modeling and automated software development, Bioinformatics 21(8) (2005) 16781684. 15. I. Gorton, D. Chavarría-Miranda, M. Krishnan and J. Nieplocha, A high-performance event service for HPC applications, in International Workshop on Software Engineering for High Performance Computing Applications, 2007. 16. L. Gross, H. Mühlhaus, E. Thorne and K. Steube, A new design of scienti¯c software using python and XML, Pure and Applied Geophysics 165 (2008) 653670. 17. M. Sala, An object-oriented framework for the development of scalable parallel multilevel preconditioners, ACM Transactions on Mathematical Software 32(3) (2006). 18. F. Hernandez, P. Bangalore and K. Reilly, Automating the development of scienti¯c applications using domain-speci¯c modeling, in Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications, 2005. 19. R. C. Gentleman, V. J. Carey, D. M. Bates et al., Bioconductor: Open software development for computational biology and bioinformatics, Genome Biology 5(10) (2004) R80.116. 20. R. P. Kendall, D. E. Post and A. Mark, Case study of the NENE code project, IEEE Magazine on Computing in Science and Engineering, 2010. 21. M. C. Smith, R. L. Kelscy, J. M. Young and A. Ginger, Creating a °exible environment for testing scienti¯c software, in Proceedings of SPIE The International Society for Optical Engineering, 2004, pp. 288296. 22. T. Cickovski, T. Matthey and J. u. A. Izaguirre, Design patterns for generic objectoriented scienti¯c software, in International Conference on Software Engineering, 2005. 23. D. W. I. Rouson, H. Adalsteinsson and J. Xia, Design patterns for multiphysics modeling in Fortran 2003 and Cþþ, ACM Transactions on Mathematical Software 37(1) (2010). 24. D. C. Arnold and J. J. Dongarra, Developing an architecture to support the implementation and development of scienti¯c computing applications, in Working Conference on the Architecture of Scienti¯c Software, 2000, pp. 3955. 25. D. Woollard, C. Mattmann and N. Medvidovic, Injecting software architectural constraints into legacy scienti¯c applications, in Workshop on Software Engineering for Computational Science and Engineering, 2009, pp. 6571. 26. A. Sharma and S. S. Sarangdevot, Investigating the application of AOP methodology in development of bioinformatics software using Eclipse-AJDT environment, in International Conference & Workshop on Emerging Trends in Technology, 2011. 27. D. E. Bernholdt, R. C. Armstrong and B. A. Allan, Managing complexity in modern high end scienti¯c computing through component-based software engineering, in Workshop on Productivity and Performance in High-End Computing, 2004. 28. B. A. Allan, B. Norris, W. R. Elwasif and R. C. Armstrong, Managing scienti¯c software complexity with Bocca and CCA, Scienti¯c Programming 16(4) (2008) 315327. 29. Donna Kaminskyj Long, Kiemele Liam, Celina Gibbs, Andrew Brownsword and Y. Coady, Mind the Gap! Bridging the dichotomy of design and implementation, in International Workshop on Software Engineering for Computational Science and Engineering, 2011.

500

R. Farhoodi et al.

30. H. P. L. Are Magnus Bruaset, Object-oriented design of preconditioned iterative methods in di®pack, ACM Transactions on Mathematical Software, 1997. 31. M. Urbano Cuadrado, M. D. Luque de Castro and M. A. Gómez-Nieto, Object-oriented techniques for design and development of standard software solutions in automation and data management in analytical chemistry, Trends in Analytical Chemistry, 2006. 32. M. Sala, K. S. Stanley and M. A. Heroux, On the design of interfaces to sparse direct solvers, ACM Transanctions on Mathematical Software 34(2) (2008). 33. K. Morris, D. W. I. Rouson and J. Xia, On the object-oriented design of referencecounted shadow objects, in International Workshop on Software Engineering for Computational Science and Engineering, 2011. 34. A.-S. S. Hai-Tao Li and Y.-X. Meng, Parallel development and independent integration model for scienti¯c research software engineering, in International Conference on Computer Science and Software Engineering, 2008. 35. C. Blilie, Patterns in scienti¯c software: An introduction, Computing in Science and Engineering 4(3) (2002) 4853. 36. R. Neely, Practical software quality engineering on a large multiDisciplinary HPC development team, in International Workshop on Software Engineering for High Performance Computing Applications, 2004. 37. Rune Møllegaard Friborg and B. Vinter, Rapid development of scalable scienti¯c software using a process oriented approach, Journal of Computational Science, 2011. 38. X. Jiao, M. T. Campbell and M. T. Heath, Roccom: An object-oriented, data-centric software integration framework for multiphysics simulations, in International Conference on Supercomputing, 2003, pp. 358368. 39. G. Rodríguez-Gómez, J. Muñoz-Arteaga and R. Fern andez-Del-Busto, Scienti¯c software design through scienti¯c computing patterns, in International Conference on Modelling, Simulation, and Optimization, 2004. 40. B. Norris, Software architecture issues in scienti¯c component development, in International Conference on Applied Parallel Computing, 2004. 41. D. Woollard, T. Berger and W. Yamada, Software engineering for neural dynamics simulations: A case study, in International Workshop on Software Engineering for High Performance Computing Applications, 2004. 42. T. Q. Andrea Lani, Dries Kimpe, Herman Deconinck, Stefan Vandewalle and Stefaan Poedts, The COOLFluiD framework: Design solutions for high performance object oriented scienti¯c computing software, in International Conference on Computational Science, 2005. 43. P. V. D. Velde and G. D. Mallinson, The design of a component-oriented framework for numerical simulation software, Advances in Engineering Software, 2007. 44. S. M. Gallo, R. Miller and C. M. Weeks, The design of a portable scienti¯c tool: A case study using SnB, in Proceedings of the ACM/IEEE Conference on Supercomputing, 1996. 45. R. Armstrong, D. Gannon, A. Geist, K. Keahey, S. Kohn, L. McInnes, S. Parker and B. Smolinski, Toward a common component architecture for high-performance scienti¯c computing, in IEEE International Symposium on High Performance Distributed Computing, 1999, pp. 1324. 46. G. Scherp and W. Hasselbring, Towards a model-driven transformation framework for scienti¯c work°ows, in Procedia Computer Science (Conference Proceedings), 2010. 47. D. W. I. Rouson, Towards analysis-driven scienti¯c software architecture: The case for abstract data type calculus, Scienti¯c Programming, 2008.


501

48. C. Macaulay, D. Sloan, X. Jiang, P. Forbes, S. Loynton, J. R. Swedlow and P. Gregor, Usability and user-centered design in scienti¯c software development, IEEE Software 26 (1) (2009) 96102. 49. I. Gorton, C. Sivaramakrishnan, S. White, S. Purohit, M. Madison and K. Schuchardt, Velo: Riding the knowledge management wave for simulation and modeling, in International Workshop on Software Engineering for Computational Science and Engineering, 2011. 50. L. Hochstein and V. Basili, A case study of large-scale parallel scienti¯c code development, IEEE Computer, 2008. 51. O. Marques and T. Drummond, Building a software infrastructure for computational science applications: Lessons and solutions, in Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications, 2005. 52. P. K. Chilana, C. L. P and A. J. Ko, Comparing bioinformatics software development by computer scientists and biologists: An exploratory study, in Workshop on Software Engineering for Computational Science and Engineering, 2009. 53. J. R. Cary, S. G. Shasharina, J. C. Cummings, J. V. W. Reynders and P. J. Hinker, Comparison of Cþþ and Fortran 90 for object-oriented scienti¯c programming, Computer Physics Communications 105(1) (1997) 2036. 54. B. J. Henz and D. R. Shires, Coupling scienti¯c applications within an object-oriented programming framework: Improving development time and software quality, in International Conference on Parallel and Distributed Processing Techniques and Applications, 2004. 55. D. Hudak, N. Lubdan, V. Gadepally and A. Krishnamurthy, Developing a computational science IDE for HPC systems, in International Workshop on Software Engineering for High Performance Computing Applications, 2007. 56. R. Arora, P. Bangalore and M. Mernik, Developing scienti¯c applications using generative programming, in Workshop on Software Engineering for Computational Science and Engineering, 2009, pp. 5158. 57. R. Kendall, J. C. Carver, D. Fisher, D. Henderson, A. Mark, D. Post, C. E. R. Jr and S. Squires, Development of a weather forecasting code: A case study, IEEE Software 25(4) (2008) 5965. 58. M. Hatlo, F. James, P. Mato, L. Moneta, M. Winkler and A. Zsenei, Developments of mathematical software libraries for the LHC experiments, IEEE Transactions on Nuclear Science, 2005. 59. S. M. Easterbrook and T. C. Johns, Engineering the software for understanding climate change, Computing in Science and Engineering 11(6) (2009) 6574. 60. D. M. Beazley and P. S. Lomdahl, Feeding a large-scale physics application to Python, in International Python Conference, 1997, pp. 2128. 61. D. Kelly, D. Hook and R. Sanders, Five recommended practices for computational scientists who write software, Computing in Science and Engineering 11(5) (2009) 4853. 62. E. V. Wyk, M. Heimdahl and Y. Saad, From models to e±cient code: It's all in the middle, in International Workshop on Software Engineering for High Performance Computing Applications, 2004. 63. J. E. Hannay, C. MacLeod, J. Singer, H. P. Langtangen, D. Pfahl and G. Wilson, How do scientists develop and use scienti¯c software? in Workshop on Software Engineering for Computational Science and Engineering, 2009, pp. 18.

502

R. Farhoodi et al.

64. M. Umarji and C. Seaman, Informing design of a search tool for bioinformatics, in International Workshop on Software Engineering for Computational Science and Engineering, 2008. 65. V. Basili, T. Craveiro, D. Cruzes, K. Despain, B. Dorland, L. Hochstein, N. Zazworka and M. Zelkowitz, Large e±cient table-top tera°op computing, in International Workshop on Software Engineering for Computational Science and Engineering, 2008. 66. P. F. Duboi, Maintaining correctness in scienti¯c programs, IEEE Magazine on Computing in Science and Engineering, 2005. 67. T. E. Oliphant, Python for scienti¯c computing, IEEE Magazine on Computing in Science and Engineering, 2007. 68. J. J. Marshall, R. R. Downs and S. Samadi, Relevance of software reuse in building advanced scienti¯c data processing systems, Earth Science Informatics, 2010. 69. W. Yu and S. Smith, Reusability of FEA software: A program family approach, in Workshop on Software Engineering for Computational Science and Engineering, 2009, pp. 4350. 70. Y. K. G. Iwai, T. Saski and Y. Watse, SAGA-based user environment for distributed computing resources: A universal Grid solution over multi-middleware infrastructures, in Procedia Computer Science (Conference Proceedings), 2010. 71. S. Faulk, E. Loh, Michael L. Van De Vanter, S. Squires and L. G. Votta, Scienti¯c computing's productivity gridlock: How software engineering can help, IEEE Magazine on Computing in Science and Engineering, 2009. 72. D. Woollard, N. Medvidovic, Y. Gil and C. A. Mattmann, Scienti¯c software as work°ows: From discovery to distribution, IEEE Software 25(4) (2008) 3743. 73. K. S. Ackroyd, S. H. Kinder, G. R. Mant, M. C. Miller, C. A. Ramsdale and P. C. Stephenson, Scienti¯c software development at a research facility, IEEE Software 25(4) (2008) 4451. 74. P. Luksch, U. Maier, S. Rathmayer, A. M. Weidmann and F. Unger, Sempa: Software engineering for parallel scienti¯c computing, IEEE Concurreny, 1997. 75. D. S. Mark Vigder, Janice Singer, Software automation in scienti¯c research organizations, in International Workshop on Software Engineering for Computational Science and Engineering, 2008. 76. M. Erwig and Z. Fu, Software reuse for scienti¯c computing through program generation, ACM Transactions on Software Engineering and Methodology 14(2) (2005) 168198. 77. A. Dall'Osso, Using computer algebra systems in the development of scienti¯c computer codes, Future Generation Computer Systems 19(2) (2003) 143160. 78. T. L. Dahlgren and P. T. Devanbu, Adaptable assertion checking for scienti¯c software components, in International Workshop on Software Engineering for High Performance Computing System Applications, 2004, pp. 6469. 79. H. Naguib and Y. Li, Applying software engineering methods and tools to CSE research projects, in International Conference on Computational Science, 2010. 80. D. Kelly and R. Sanders, Assessing the quality of scienti¯c software, in Workshop on Software Engineering for Computational Science & Engineering, 2008. 81. D. Post, R. Kendall and W. Earl, Case study of the falcon code project, in Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications, 2005. 82. M. G. Cox and P. M. Harris, Design and use of reference data sets for testing scienti¯c software, Analytica Chimica Acta 380(23) (1999) 339351. 83. R. G. Diane Kelly, Yizhen Shao, Examining random and designed tests to detect code mistakes in scienti¯c software, Journal of Computational Science, 2011.


503

84. G. Avrunin, S. Siegel and A. Siegel, Finite-state veri¯cation for high performance computing, in International Workshop on Software Engineering for High Performance Computing Applications, 2005. 85. R. Palmer, S. Barrus, Y. Yang, G. Gopalakrishnan and R. M. Kirby, Gauss: A framework for verifying scienti¯c computing software, in Proceedings of the Workshop on Software Model Checking, 2005. 86. S. Asgari, L. Hochstein, V. Basili, J. Carver, J. Hollingsworth and F. Shull, Generating testable hypotheses from tacit knowledge for high productivity computing, in International Workshop on Software Engineering for High Performance Computing Applications, 2005. 87. M. Heroux, Improving CSE software through reproducibility requirements, in International Workshop on Software Engineering for Computational Science and Engineering, 2011. 88. T. L. Dahlgren and P. T. Devanbu, Improving scienti¯c software component quality through assertions, in International Conference on Software Engineering, 2005. 89. R. Graya and D. Kelly, Investigating test selection techniques for scienti¯c software using Hook's mutation sensitivity testing, in International Conference on Computational Science, 2010. 90. S. A. Vilkomir, W. T. Swain, J. H. Poore and K. T. Clarno, Modeling input space for testing scienti¯c computational software: A case study, in International Conference on Computational Science, 2008, pp. 291300. 91. D. Hook and D. Kelly, Mutation sensitivity testing, IEEE Design & Test 11(6) (2009) 4047. 92. D. Kelly, S. Thorsteinson and D. Hook, Scienti¯c software testing: Analysis with four dimensions, IEEE Software, 2011. 93. J. C. Carver, R. P. Kendall, S. E. Squires and D. E. Post, Software development environments for scienti¯c and engineering software: A series of case studies, in International conference on Software Engineering, 2007, pp. 550559. 94. D. E. Post and R. P. Kendall, Software project management and quality engineering practices for complex, coupled multiphysics, massively parallel computational simulations: Lessons learned from ASCI, International Journal of High Performance Computing Applications, 2004. 95. C. Morris, Some lessons learned reviewing scienti¯c code, in International Workshop on Software Engineering for Computational Science and Engineering, 2008. 96. J. Segal, Some problems of professional end user developers, in IEEE Symposium on Visual Languages and Human-Centric Computing, 2007, pp. 111118. 97. H. Remmel, B. Paech, C. Engwer and P. Bastian, Supporting the testing of scienti¯c frameworks with software product line engineering A proposed approach, in International Workshop on Software Engineering for Computational Science and Engineering, 2011. 98. D. Hook and D. Kelly, Testing for trustworthiness in scienti¯c software, in Workshop on Software Engineering for Computational Science and Engineering, 2009, pp. 5964. 99. R. Sanders and D. Kelly, The challenge of testing scienti¯c software, in Conference for the Association for Software Testing, 2008. 100. L. Hatton, The T-Experiments: Errors in scienti¯c software, IEEE Computational Science & Engineering 4(2) (1997) 2738. 101. S. Booth and D. Henty, Veri¯cation strategies for high performance computing software, in International Workshop on Software Engineering for High Performance Computing Applications, 2004.

504

R. Farhoodi et al.

102. E. Loh, M. L. V. D. Vanter and L. G. Votta, Can software engineering solve the HPCS problem? in Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications, 2005. 103. D. Kelly, Determining factors that a®ect long-term evolution in scienti¯c application software, Journal of Systems and Software 82(5) (2009) 851861. 104. M. Wagner, Evolution from a scienti¯c application to an applicable product, in European Conference on Software Maintenance and Reengineering, 2007. 105. Y. Li, Reengineering a scienti¯c software and lessons learned, in International Workshop on Software Engineering for Computational Science and Engineering, 2011. 106. Je®rey Overbey, Spiros Xanthos, Ralph Johnson and B. Foote, Refactorings for Fortran and high-performance computing, in International Workshop on Software Engineering for High Performance Computing Applications, 2005. 107. J. Pitt-Francis, M. O. Bernabeu, J. Cooper, A. Garny, L. Momtahan, J. Osborne, P. Pathmanathan, B. Rodriguez, J. P. Whiteley and D. J. Gavaghan, Chaste: Using agile programming techniques to develop computational biology software, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 366(1878) (2008) 31113136. 108. W. A. Wood and W. L. Kleb, Exploring XP for scienti¯c research, IEEE Software 20(3) (2003) 3036. 109. D. Kane, Introducing agile development into bioinformatics: An experience report, in Proceedings of the Conference on Agile Development, 2003, pp. 132140. 110. J. Segal, Models of scienti¯c software development, in International Workshop on Software Engineering for Computational Science and Engineering, 2008. 111. J. Segal, Software development cultures and cooperation problems: A ¯eld study of the early stages of development of software for a scienti¯c community, Computer Supported Cooperative Work 18(56) (2009) 581606. 112. D. E. Post, The challenge for computational science, in International Workshop on Software Engineering for High Performance Computing Applications, 2004. 113. J. Willenbring, M. Heroux and R. Heaphy, The Trilinos software lifecycle model, in International Workshop on Software Engineering for High Performance Computing Applications, 2007. 114. M.-S. Wu, J. L. Bentz, F. Peng, M. Sosonkina, M. S. Gordon and R. A. Kendall, Integrating performance tools with large-scale scienti¯c software, in IEEE International Parallel and Distributed Processing Symposium, 2007, pp. 18. 115. M. Vigder, End-user software development in a scienti¯c organization, in ICSE Workshop on Software Engineering Foundations for End User Programming, 2009. 116. T. Epperly, S. R. Kohn and G. Kumfert, Component technology for high-performance sceinti¯c simulation software, in Working Conference on the Architecture of Scienti¯c Software, 2000, pp. 6986. 117. J. P. Kenny, S. J. Benson, Y. Alexeev, J. Sarich, C. L. Janssen, L. C. Mcinnes, M. Krishnan, J. Nieplocha, E. Jurrus, C. Fahlstrom and T. L. Windus, Component-based integration of chemistry and optimization software, Journal of Computational Chemistry 25(14) (2004) 17171725. 118. V. Maxville, Preparing scientists for scalable software development, in Workshop on Software Engineering for Computational Science and Engineering, 2009. 119. J. Howison and J. D. Herbsleb, Scienti¯c software production: Incentives and collaboration, in Proceedings of the ACM Conference on Computer Supported Cooperative Work, 2011.


505

120. S. Samadi, N. Almaeh, R. Wolfe, S. Olding and D. Isaac, Strategies for enabling software reuse within the earth science community, in International Geoscience and Remote Sensing Symposium, 2004. 121. C. Halverson, C. Swart, J. Brezin, J. Richards and C. Danis, Towards an ecologically valid study of programmer behavior for scienti¯c computing, in International Workshop on Software Engineering for Computational Science and Engineering, 2008. 122. V. R. Basili, D. Cruzes, J. C. Carver, L. M. Hochstein, J. K. Hollingsworth, M. V. Zelkowitz and F. Shull, Understanding the high-performance-computing community: A software engineer's perspective, IEEE Software 25(4) (2008) 2936. 123. J. F. Cremer, R. S. Palmer and R. E. Zippel, Creating scienti¯c software, Transactions of the Society for Computer Simulation International 14(1) (1997) 3749. 124. S. M. Baxter, S. W. Day, J. S. Fetrow and S. J. Reisinger, Scienti¯c software development is not an oxymoron, PLoS Computational Biology, 2006. 125. J. Segal, Scientists and software engineers: A tale of two cultures, in Proceedings of Annual Meeting of the Psychology of Programming Interest Group, 2008. 126. R. Baxter, Software engineering is software engineering, in International Workshop on Software Engineering for High Performance Computing Applications, 2004. 127. Y. Alexeev, B. A. Allan, R. C. Armstrong and D. E. Bernholdt, Component-based software for high-performance scienti¯c computing, Journal of Physics 16, 2005. 128. R. A. Bartlett, Integration strategies for computational science & engineering software, in Workshop on Software Engineering for Computational Science and Engineering, 2009, pp. 3542. 129. C. Morris and J. Segal, Some challenges facing scienti¯c software developers: The case of molecular biology, in IEEE International Conference on e-Science, 2009. 130. L. Nguyen-Hoan, S. Flint and R. Sankaranarayana, A survey of scienti¯c software development, in International Symposium on Empirical Software Engineering and Measurement, 2010.

Other studies 131. D. F. Kelly, A software chasm: Software engineering and scienti¯c computing, IEEE Software 24(6) (2007) 118119. 132. V. K. Decyk, C. D. Norton and H. J. Gardner, Why Fortran? Computing in Science and Engineering 9(4) (2007) 6871. 133. R. L. Glass and T. Y. Chen, An assessment of systems and software engineering scholars and institutions (19982002), Journal of Systems and Software 68(1) (2003) 7784. 134. R. L. Glass and T. Y. Chen, An assessment of systems and software engineering scholars and institutions (19992003), Journal of Systems and Software 76(1) (2005) 9197. 135. V. Garousi and T. Varma, A bibliometric assessment of Canadian software engineering scholars and institutions (19962006), Canadian Journal on Computer and Information Science 3(2) (2010) 1929. 136. V. Garousi and T. Varma, A bibliometrics analysis of Canadian electrical and computer engineering institutions (19962006) based on IEEE journal publications, Canadian Journal on Computer and Information Science 5(4) (2012) 124. 137. D. Budgen, M. Turner, P. Brereton and B. Kitchenham, Using mapping studies in software engineering, in Proceedings of Annual Workshop of the Psychology of Programming Interest Group, 2008, pp. 195204. 138. K. Petersen, R. Feldt, S. Mujtaba and M. Mattsson, Systematic mapping studies in software engineering, in International Conference on Evaluation and Assessment in Software Engineering, 2008.

506

R. Farhoodi et al.

139. M. T. Sletholt, J. E. Hannay, D. Pfahl, H. C. Benestad and H. P. Langtangen, A literature review of agile practices and their e®ects in scienti¯c software development, in International Workshop on Software Engineering for Computational Science and Engineering, 2011. 140. B. Kitchenham and S. Charters, Guidelines for performing systematic literature reviews in software engineering, in Evidence-Based Software Engineering, 2007. 141. V. Garousi, Classi¯cation and trend analysis of UML books (19972009), Journal on Software & System Modeling (SoSyM), to Appear. 142. F. Elberzhager, J. Münch and V. T. N. Nha, A systematic mapping study on the combination of static and dynamic quality assurance techniques, Information and Software Technology 54 (2012) 115. 143. S. Ali, L. Briand, H. Hemmati and R. K. Panesar-Walawege, A systematic review of the application and empirical investigation of search-based test-case generation, IEEE Transactions on Software Engineering 36(6) (2010) 742762. 144. V. R. Basili, Software modeling and measurement: The Goal/Question/Metric paradigm, Technical report, University of Maryland at College Park 1992. 145. R. Farhoodi, V. Garousi, D. Pfahl and J. Sillito, Systematic mapping of developing scienti¯c software, online repository on Google Docs, http://www.softqual.ucalgary.ca/ projects/2012/SM CSE/, Last accessed: April 2012. 146. P. Runeson, A. Rainer, M. Host and B. Regnell, Case Study Research in Software Engineering: Guidelines and Examples (John Wiley & Sons, 2012). 147. B. D. Nicola, Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics: (Scarecrow Press, 2009). 148. R. Gentleman, R Programming for Bioinformatics (CRC Press, 2009). 149. Y. Liu and A. Milanova, Static analysis for inference of explicit information °ow, in Proceedings of ACM Workshop on Program Analysis for Software Tools and Engineering, 2008. 150. R. Kendall, D. Fisher, D. Henderson, J. C. Carver, A. Mark, D. Post, C. E. Rhoades, Jr., and S. Squires, Development of a weather forecasting code: A case study, IEEE Software 25(4) (2008) 5965. 151. T. D. Cook and D. T. Campbell, Quasi-Experimentation Design and Analysis Issues for Field Settings (Houghton Mi±n, 1979).