Portability Testing of Scientific Computing Software Systems Rahul Nori, Nitin Karodiya, and Hassan Reza University of North Dakota School of Aerospace Sciences Grand Forks, ND 58201 USA
[email protected] Abstract— Software testing is considered to be the ultimate obstacle before software can be released. Traditional approaches to testing the software revolves around capabilities of a system using various types of testing (e.g., black-box) at various level of software development process. Testing the software for system-wide qualities such as portability is very rare. The research work in this area is very limited. In this work, we discuss the importance of portability testing and attempt to explore a testing approach to test the portability of Scientific Computing (SC) software using different platforms. The main incentive for testing of SC software is that this class of software is challenging because of the elements such as computing environment, computational mathematics, languages, optimization; these elements play significant roles in success or failure of these systems. Keywords-component; Sceinfic computing, software testing, portability testing, software engineering.
I.
INTRODUCTION
Portability of a system refers to the ease by which a software system can be made to operate on different platforms. Portability is a collective term composed of other quality attributes such as Adoptability (Recompile-ability), Installability, Replaceability, and Coexist-ability. Portability is an essential quality attribute of a system [11][12], and the software products irrespective of their types, size, and complexity are benefitted by their ability to operate on different platforms (software, hardware) with minimum changes. The approach to realize portability includes the standardization that is perceived as the most effective approach to realize the portability of a software intensive system. However, standardization is heavily context sensitive and hence it can be applied under specific conditions. The Portability testing is perceived as an alternative to standardization; it is a testing method to show failures attributed to porting computer-based systems. More specifically, the goal of portability testing is to reveal defects stemming from porting the elements of ported systems to the target platform. Examples of those elements include data, hardware, and software.
Portability plays an important role in the area of scientific computing. Scientific computing performs mathematical modeling of some physical phenomenon; it involves quantitative numerical and theoretical analysis of data. An example of a scientific computing is a computer simulation. Computer simulation, for most parts, is used to predict a physical behavior under a set of assumptions. Most of these calculations are computationally intensive depending on how large is the size of the dataset is. To solve complex computational problems High Performance Computation (HPC) systems such as Supercomputers, Distributed and Grid Computers and Graphical Processing Units (GPU) are made use. Scientific software is a kind of software that involves high computation and modeling of some physical phenomenon. The results given by this scientific software provides data for making some decision or conclusion about the physical phenomenon or physical problem addressed by the software [23]. There are mainly three types of risks associated with the scientific software called Theory Risk, which has 3 types: (a) Risk of Cognitive Complexity: It is the difficulty faced in the understanding of the problem or physical phenomenon. (b) Risks to Validation Testing: It is associated with the correct representation of the physical phenomenon or problem using a mathematical model or software. (c) Risk from approximation to continuous models: If the problem have the continuous model then transforming it to mathematical model always requires some sort of approximation. The assumption made for this approximation can increase the risk in validation the software. The “Implementation Risk” also consists of 3 types: (a) Risk to correctness: It is associated with the correctness of calculation in the code, (b) Risk from Poor code documentation: If the documentation of the code is poor then it can again increase the risk of the software because it can be used properly and efficiently, and (c) Risk to verification testing; there are various definitions of verification testing for scientific software. Blochinger et. al [21] designed and specified a programming environment that acted as a conditional compilation constraint that that allows the code to select a particular code that remains to be independently executable by itself (i.e., Platform independent). They tried to merge two distinct concepts in the sense that they tried to compile
platform specific applications (programs) using a platform independent model that would then make the necessary decisions to compile the code on that particular domain. Software diversity has been considered better than single version software because on average diverse software fails less often as compared to the single version software [25]. In fact, the relationship between the reliability and diverse design is very difficult to assess. The dependency between the diversity of design over the performance and reliability is not clear and it cannot be said that using a particular design gives a particular amount of gain in reliability than using an alternate design. Scientific computing relays on mathematical models that can be very complex. The models are applied along with the numerical approximation to solve the problem and this process is called Simulation. The simulation and actual results differ in practice. The simulation itself introduces poses some uncertainty due to the approximation and transforming of models [26]. This uncertainty must be quantified and reduced to fulfill the purpose of scientific computing adequately. In [26], the authors have discussed a framework for verification, validation and uncertainty quantification in scientific computing. The authors have also discussed the types of uncertainty. The uncertainty is of 2 types: (a) Aleatory Uncertainty: This uncertainty occurs due to randomness in the distribution and is also known as irreducible uncertainty. Research oriented, scientific software development often stays isolated from the modern techniques of software development and testing that is generally associated with commercial software [16]. Scientists develop software in their laboratories; perform formal testing of the mathematical models and run tests on their institutional systems. It is not common to validate these software systems on multiple software and hardware architectures. Most of the software developed in the scientific applications utilize the component of the shelve (COTs) in the sense that they reuse existing components from pre existing research software. Examples include C++, BLAS and Linear Algebra Package (LAPACK) [8], [10], [9]. II. RELATED WORK It is very rare that a software package built entirely complete scratch and hence SC is no exception. Therefore, it is imperative to test such open source, scientific software that can be deployed across various platforms before they are made available to other users. Required documentation needed to perform portability testing include: a list or model of software package dependencies, compiler and various versions of that compiler together with the instructions to compile them (e.g., make files, shell scripts, etc.).Generally most of these software packages are designed on a platform that is most accessible and well known to the developer [15]. Testing these software packages on different architectures can often be considered as a part of
development process. Approaches to test such software uses a formal method in the sense that the results tend to vary depending on the system architecture and the programming language being used. A formal approach to validate the results has to be adopted to nullify discrepancies. As a test case, we intend to expand upon the work done by Forlemu et. al [1], by to exploring portability in scientific Software and the mathematical model discussed. To this end, we designed a model called the dipole interaction model programmatically taking two routes or approaches of calculating Circular Dichroism (CD) in of proteins comprised of alpha helices and beta sheets. The dipole interaction model [2], [3], is a model in the classical electromagnetic theory that has been applied by the authors of the current work to reproduce the experimental CD, theoretically using the programmatic approach for the pi-topi* transitions in proteins and peptides. The Dipole interaction model was described in [4]. The program has been written in FORTRAN, C, and C++; it allows the treatment of whole proteins unlike the existing work of CAPPS which was unable to treat all the possible proteins and Protein Data Bank (PDB) [5] files that are not adhering with its standards. In their work Forlemu et al. came up with high performance computing software named Dipole Interaction Model (DInaMo) [1] [6]. Although the package produced verifiable results, but not all components were completely portable across various platforms. The components in general were designed using programming languages such as FORTRAN, C and C++ and worked on platforms such as SGI UNIX, Red Hat Linux and Mac OSX. In work by [7] [17], researchers tested the scalability and portability of computational chemistry programs across heterogeneous systems and came up with a model approach. In the paper on portability [13], the authors have proposed a framework to implement portable system. In general, the software development methodologies do not incorporate the any tactic to achieve portability. More attention is paid on the reuse than on achieving portability and also there is no acceptable metrics for measuring portability of a system [14]. Software interacts with its environment using numerous interfaces. To achieve portability, all the interfaces of the software should appear identical across all platforms. The authors have discussed various strategies to increase portability during software development. They are: (a) Control the Interfaces: It requires the identification of all interfaces and standardizes them in coding. (b) Isolate Dependencies: it requires that the dependency among the components must be identified and modules should be made so that the dependency is minimized. (c) Think Portable: It requires continuous awareness of the developer to make decisions considering the possibility of future portability requirements. Making use of these basic portability approaches and the scientific computing software, DInaMo, we explored facets
of portability testing across platforms such as Hardware dependencies, software dependency (e.g., compiler, Operating System (OS), and the programming languages) used to solve the mathematical model presented in the preliminary work. We verified the formal validity of the model by testing the results across various platforms after porting the software to different platforms. III. METHODOLOGY As discussed in [26], when it comes to formal models like the one we are considering, there are sources of uncertainty such as model inputs, numerical approximations and model form. Model Inputs include the parameters used in the model and also surrounding environment factors like boundary conditions etc. Numerical approximation is the approximation used to create the model, assumptions, abstractions, etc. Model form includes the resulting model after the model inputs and approximations. The uncertainty framework is used to predict the uncertainty. So analyzing our computational model involved the following steps as a start up to understand the various issues pertaining to its testing: a)
Identify all sources of uncertainty: The uncertainty in model inputs by all potential sources should be identified. b) Characterize uncertainties: Every uncertainty must be characterized i.e. a mathematical structure and associated parameters should be given for each uncertainty. c) Estimate uncertainty due to numerical approximation: The uncertainty due to numerical approximation should be estimated. As we have a mathematical structure for each uncertainty we can estimate the degree of uncertainty. d) Propagate input uncertainties through the model: The sampling and probabilistic distribution can be used to propagate the uncertainty through the model. This helps in better estimation of the uncertainty. e) Estimate model form uncertainty: Estimate the model form uncertainty. This can be estimated using the process of model validation. f) Determine total uncertainty in the system response quantity: The estimated total uncertainty by each step should be collectively estimated and quantified in term of system response quantity. By estimating the total uncertainty in terms of system response quantity, the validation and verification on the current system was necessary to be performed. To conclude, this verification, validation and uncertainty quantification we needed to come up with a framework that provides the techniques that help the decision making process in the scientific computing which improve the results of the physical phenomenon or problem addressed by the software.
Usually various scientific software are developed and used by the scientists in their lab. This software fails because they are implemented in an unstable and non-robust way. Proper testing for this software is also not carried out in the labs [20]. They are mainly concerned with data input and the output results. And, if the software is not tested well then the results given out by the software cannot be trusted. So, a general approach or some form of black-box testing is required so that it can be applied to testing of any scientific software. One needs to ensure that it conforms to the specification of the problems they solve. To solve such considerations we developed a small test set amino acid complex that had five peptides, called Cyclo. We made use of a standard software called InsightII [36], for building this molecule and also made use of the dimer structure of Human Insulin (PDB Code: 3INC) to verify the validity of the software under development. These two molecule sets acted as our test sets for our formal testing approach. This approach also helps in developing the reference data sets corresponding to the different problems which vary in their level of difficulty and also corresponding to the application or domain specific properties. Our testing approach had two components to it that are discussed here. A. Formal Testing To test the basic functionality of the formal model, we had to build an input structure file that was considerably small. For this purpose we built an input test set using Insight software [36], a peptide called Cyclo that comprises of the amino acids: Glycine-Proline-Glycine-DecaAlanine-Proline in its chain. This is relatively a very small structure and contains just these five amino acids and around about fifty atoms in the structure. The software takes as input various residue parameters such as the elements in the chain and the angle between them, building together to form the overall structure that can be saved as a standard PDB file. We then made use of this structure file and computed CD across various platforms across which the code was ported. We were able to validate the results obtained from the CD calculations by looking at the computed model and also comparing the results with the previously established results. This formal approach acted as a standard for us to test out program after being able to run the software using various Operating systems, compiler versions and programming languages. We were also able to compare the theoretical and experimental CD Spectra for a comparatively smaller protein like Insulin which is a much larger structure than the peptide, but was able to produce the expected results. Therefore, the input PDB files were passed through various stagesof DInaMo program package and the output result that is the CD spectrum values were compared from runs across all the different platforms. We performed a Root Mean Square analysis for the computed results in order to
check how much the output CD values varied from one platform to another. We also superimposed the various structures with their counterparts using Pymol software [37] to cross check the conformational changes noticed and observe the changes pictorially. B. Portability Testing Portability testing is a standard testing approach which depicts the ease with which software can be run across various computing platforms with minimal changes. For example, it attempts to answer questions like: Is it possible to run the application on Linux? Can the application be minimally changed to run on other platform such as Mac OSX and Windows? Some of the basic variations include platforms that we considered were different but limited set of operating systems, compiler and their various versions, programming languages, computer hardware, etc. We tested three variations of Operating Systems: SGI Unix, Linux, and Mac OS X. We then tested our software package for portability across these three operating systems while neglecting windows systems, even though we had working versions in windows OS for some components of the software. To this end, we made this assumption from personal experience and observing the trends across various research groups across campus that Unix based systems are widely used in Scientific Computing software. We also tested the FORTRAN version of the application across two compilers versions available g77 and gFORTRAN. Most of the applications before porting were compilable using g77 compiler which uses the FORTRAN 77 programming standard. By porting this code to gFORTRAN we were able to not only use a standardized compiler version, but also make use of extended version of FORTRAN 90. Toward using of gFORTRAN, we further assume that it should make it easier for other researchers to install access and use the software. The third aspect of porting that we considered was a programming language. Although we were not able to port all the components across the various languages considered, we were able to make significant progress. We ported CDCALC and COREUL applications from FORTRAN to C and C++. This helped us in making use of features like dynamic memory allocation and object oriented programming. There still needs to be a significant amount of work done to port the code completely over to C and C++ but the initial results was promising. In regard to hardware portability, we were able to achieve significant performance gain in porting the system from SGI Unix to regular Linux machines that supports a much faster processor on board. We are interested in making the system operate in parallel so that it can achieve higher level of concurrency and hence performance. There are some basic portability requirements [15] that need to be considered. The criteria include:
a)
Installability: Testing the system to find out how difficult it is to install the software on a target environment. b) Coexist-ability: Testing the system to identify wether the operation of this particular system can be coupled with other existing software on the target system. c) Adaptability: Test the ability of the software to work on various OS platforms without applying a significant amount of effort. This, in turn, requires how easy it is to install the system on different operating systems. Having good documentation and system builds that account for various operating systems improves software adaptable. d) Replaceability: Testing the system to find out that it can replace existing software performing a similar functionality with improved quality (e.g., better visualization or user interaction). Two main aspects of our portability testing include: 1) testing on Build system, and 2) testing on Target systems. The build system is one on which the developers work. While working on the build system the most important task is to identify all the dependencies of the system under development. The code reused generally occurs in the form of using pre developed libraries or adding functionality to some of the legacy code. In general for a developer, it is difficult to identify all the dependencies of the system without having developed the whole system on his own. Having required documentation allows to identifying the source of dependencies. To this end, we decided to detect main dependencies of the software. There are many approached to identify dependencies. For example, Operating System Version that is required for the system to run can be identified by Error Analysis. Error analysis process attempts to identify errors that are generated when a user tries to run the program on an alien system. Making a note of these errors, and hence making necessary changes to the code and the way program is compiled can eliminate these errors. For example, errors such as “ld: warning: -macosx version min version not specified, assuming 10.7” were found to be very common in most of our compilation trials. Analyzing these error messages and warning signs are necessary to understand and debug the problems. Using error analyze, we can also identify hardware dependencies. But using this method can be tricky as most of the compilers are not able to successfully identify the hardware. Therefore, it is up to the user to test the software on various builds of the system and identify the required changes. Identifying the Programming Language used to implement the system or a library component is rather straightforward as long as the developer has good knowledge of various programming methodologies. Porting code from one language to another becomes easy depending on the necessities of the project.
We were able to identify these needs for the project and attempted to port the code from FORTRAN to C. Compiler versions and software libraries act as a basic dependency for any software system. Identifying these can be challenging. To accomplish this, we made use of CMake files. Presented below in following figures: Codelblock 2, and 3. The codeblock 2 figure is a sample extract from one of our CMake files ; it shows as an output to the user what kind of FORTRAN compiler can be found on the system and uses the necessary compilation flags needed to execute the program. After the dependencies are identified we analyzed each dependency for feasibility of porting with respect to the software and ported the necessary components across the target platform by changing the code.
As mentioned earlier, the second aspect of portability is testing on the target system. Target system in this case can be any system other than the one, on which the system was initially built. Therefore, we made approaches to test the system on the various target systems that have different combinations of platform parameters like operating systems (Linux, Unix, OS X), programming language (FORTRAN, C & C++) and FORTRAN compilers (g77, gFORTRAN). Although it is difficult and time consuming to test each and every one of the possible combinations, but this can be done on the many such systems by identifying the dependencies. The process to perform the portability testing of target platform includes the following steps: 1) Test the CMake files written on the build systems, now on the target system (like the one shown in Code block 2. This step helps in looking up the existing compiler version and libraries on the target system. Depending on the available compilers and analyzing the missing compilers, one can install the necessary ones. This step is done manually, but it can be automated
but it requires the user to have access right so we left that part of the implementation as a manual process. 2) Compile and run the software on the target system. After the program runs we performed the formal testing approach as discussed in the previous section on all of the target systems and then presented the results in a later section. 3) Document all the dependencies of each of the component of the system and come up with Makefiles for each of the components so that a cross platform build system can be developed. We standardized each of the installable files into separate folders so that they can be delivered to each system build type separately. A sample Make file is shown in the code block 3 below. As it can be seen in the code block the necessary line with operating system specific flags can be commented or uncommented depending on the system being used.
IV. RESULTS By performing the portability across the systems with our formal approach we were able to answer a few questions, those are: a)
Does the Coordinate values from PDB files change across different platforms? NO b) Do CD Spectra values change by porting code across different versions of FORTRAN compilers? NO c) Do CD Spectra values change across different Operating System platforms? YES (minor change) d) Do CD Spectra values change by porting code from FORTRAN to C? YES. e) Is the CD Spectra calculated theoretically valid? YES. f) Does the system provide an optimal match between experimentally and theoretically calculated CD? NO.
g) Is there a difference in performance (time) across various platforms? YES. h) In regard to the formal analysis of the calculated CD across various platforms, we calculated the root mean square value of the CD spectra value at various wavelengths. Table I shows the results.
These results are shown in Table II. Using our approach in porting across hardware of systems, we were able to make some basic observations that make it faster hardware components. The results were compiled as an average of a few runs performed on various systems, and thus are represented as a range of results. Also using method, we tried to answer the most important questions: ”What is the significance of Porting code across compiler versions and programming languages?” Tabulated in Table III are a few such reasons and the various versions of each of the software components currently available. Our assumptions were that the compiler versions we used are standardized and they are easily available and installable. One of the major motivations for exploring portability was to draw an observation as to how some of the approaches we applied were successful across some of the platforms explored. We made use of CMake and Makefiles to test the system for its dependencies and hence to test the target systems for what all changes need to be made to make the software operate successfully on that system. Depending on our understanding of how we used them for making the software portable and some general guidelines, some of the advantages of using CMake and Makefiles include: a)
We were able to compare the root mean square difference between two sets of structures by super imposing them in Pymol [37]. The software overlaps corresponding structure files after calculating the CD values and then shows a pictorial representation of the difference as well as a numerical RMS difference. Figure 2 depicts the difference between the insulin structures from running the package on Linux systems versus running it on Mac OS X systems. By porting the various components of the package across various platforms we were able to increase the performance of the software.
They are generally known to Aid Cross-Platform Software Development which was one of the major goals of our implementation. b) They introduced an ability to write platform checks withCMake. c) They helped in finding existing libraries and various compiler based dependencies They also helped to create a cross platform build system. d) They help in packaging the project for use by others. e) They helped in finding and using existing frameworks on various operating systems considered. f) They aid compilation and deployment of software across various platforms. g) They acted as a means to implement and test the application across various platforms. We were able to identify some of challenges in porting software across platforms. The challenges we faced with porting code across platforms, we noticed that a significant time and effort needed to incorporate changes to existing software. At the occasions, it became difficult to assess the compatibility and efficient use of other existing software that were made use of the software being considered. We present the advantages in terms of some portability metrics as discussed previously (Section III). Some of the advantages to porting code were a significant Improvement in Execution Time of the software from older platforms to the newer version platform. We were able to port the software to a standardized compiler version. This helped in increasing the Adaptability and Installability of the system. In doing that we realized that porting code across programming languages resulted in
ability of software to be parallelized. Parallelization of software increases replaceability, performability, and availability of the system. Making the software available on various operating systems increased Installability, Co-existability, and Adaptability of the system. Our changes helped in reducing the overall effort which increased the Adaptability of the system. There was an improvement in the robustness which allows the system to be used efficiently and effectively. We were able to conclude that making the system operable across these various platforms aided in garnering better support for system from developer and user communities in the field as they are now much more standardized than before. V. CONCLUION and FUTURE WORK Predicting the hardware and software configuration of target systems proved to be one of the biggest challenges to overcome in the development process and hence requires further work. A significant effort was invested in eliminating errors produced by porting existing software, which were nonexistent prior to the changes. There is always need to retain the validity of the mathematical model used to implement the functionality while porting code. There is a basic necessity to understand the tradeoff between the need to port software versus adding new functionality to the existing software. It is difficult to strike the right balance between these two, but we assumed porting code had significance at the current stage of the project. In this work, we attempt to show the significant of portability testing of scientific computing system. Toward this end, we showed some progress, but still a significant work remains to be done in order to make meaningful contribution toward portability of scientific computing applications. Future work, we would like to port all the current components of our software into either C, C++ or a combination of both so that they can be parallelized easily using MPI on multiple cluster nodes or CUDA on processors with GPUs. REFERENCES [1] Neville Forlemu, Boris Sango, Sandeep Pothuganti, Rahul Nori, Yvonne Bongfen Mbote, Kathryn Thomasson. ”Introducing DInaMo: A Package for Calculating Protein Circular Dichroism Using Classical Electromagnetic Theory”, International Journal of Mass Spectrometry, IJMS 2012 (Submitted). [2] Bode, K. A.; Applequist, J., Globular Protein Ultraviolet Circular Dichroic Spectra. Calculation from Crystal Structures via the Dipole Interaction Model. J. Am. Chem. Soc. 1998, 120, 10938-10946. [3] Applequist, J., Theoretical pi-pi* circular dichroic spectra of helical polyglycine and poly-L-alanine as functions of
backbone torsion angles. Biopolymers 1981, 20, (2), 387397. [4] Applequist, J. In Calculation of Electronic Circular Dichroic Spectra by a Dipole Interaction Model, Chirality and Circular Dichroism: Structure Determination and Analytical Applications; 5th International Conference on Circular Dichroism, Colorado State University, Fort Collins, CO, 1993; pp 152-157. [5] Berman, H. M.; Westbrook, J.; Feng, Z.; Gilliland, G.; Bhat, T. N.; Weissig, H.; Shindyalov, I. N.; Bourne, P. E., The Protein Data Bank. Nucleic Acids Res. 2000, 28, (1), 235-242. [6] Whitmore, L.; Woollett, B.; Miles, A. J.; Klose, D. P.; Janes, R. W.; Wallace, B. A., PCDDB: the protein circular dichroism data bank, a repository for circular dichroism spectral and metadata. Nuc. Acid. Res. 2011, 39, (Database issue), D480-D486. [7] Victor Lotrich, Norbert Flocke, Mark Ponton, Beverly A. Sanders, Erik Deumens, Rodney J. Bartlett, and Ajith Perera. 2009. An infrastructure for scalable and portable parallel programs for computational chemistry.In Proceedings of the 23rd international conference on Supercomputing (ICS ’09). ACM, New York, NY, USA, 523-524. [8] Demming Robert, Duffy Daniel J. (2012) Introduction to the Boost C++ Libraries - Volume 2 - Advanced Libraries. Datasim. [9] Anderson, E.; Bai, Z.; Bischof, C.; Blackford, S.; Demmel, J.; Dongarra, J.; Du Croz, J.; Greenbaum, A.; Hammarling, S.; McKenney, A.; Sorensen, D. (1999). LAPACK Users’ Guide (Third ed.). Philadelphia, PA: Society for Industrial and Applied Mathematics. [10]”http://www.netlib.org/lapack/lug/node145.html” LAPACK Users’ Guide. [11] Schach, S.R. Classical and Object-Oriented Software Engineering (3rd ed.). Richard D. Irwin, Chicago IL, 1996. 9. [12] Sommerville, I. Software Engineering, 5th ed, AddisonWesley, Reading MA, 1996. 10. [13] Mooney, J.D., Bringing Portability to the Software Process, Technical Report TR 97-1, Dept. of Statistics and Computer Science, West Virginia Univ., Morgantown, WV, 1997. [14] Song, X., and Osterweil, L. Toward Objective, Systematic Design- Method Comparisons. IEEE Software, 9, 3 (May 1992), 43- 3. [15] ISTQB GUIDE,”What is Portability testing in software?:http://istqbexamcertification.com/what-is portability-testing-in- sftware/” [16] Chris Wilcox, ”A Survey of Portability in Scientific Computing:http://www.cs.colostate.edu/saxs/researchexam/ ResearchExam.pdf”. [17] Jan Tretmans. 1999. Testing Concurrent Systems: A Formal Approach. In Proceedings of the 10th International Conference on Concurrency Theory (CONCUR ’99), Jos C. M. Baeten and Sjouke Mauw (Eds.). Springer-Verlag, London, UK, 46-65.
[18] Barton P. Miller, Gregory Cooksey, and Fredrick Moore, An empirical study of the robustness of MacOS applications using random testing. In Proceedings of the 1st international workshop on Random testing (RT ’06). ACM, New York, NY, USA, 46-54. [19] Jarek Nieplocha, Ian Foster, Rick A. Kendall, ”ChemIo: High Performance Parallel I/o for Computational Chemistry Applications”, International Journal of High Performance Computing Applications September 1998 vol. 12 no. 3 345363. [20] M. G. Cox, P. M. Harris, ”The design and use of reference datasets for testing scientific software”, International Conference of Optical Spectrometry: Applications and Instrumentation into the 21st Century, Royal Holloway College, UK, 28 June - 2 July 1998. [21] Blochinger, W., and Kchlin, W., Cross-platform development of high performance applications using generic programming. In Proceedings of the 16th International Conference on Parallel and Distributed Computing Systems (PDCS), August, 2003, pp. 654-659. [22] William Langdon, OpenGL Example http://www.cs.mun.ca/ blangdon/ opengl/glDrawPixels.html [23] Diane Kelly, Rebecca Sanders, ”The Challenge of Testing Scientific Software”, CAST 2008: Beyond the Boundaries July 14-16, 2008 Toronto, Canada. [24] S. McConnell, ”Open Source MethodologyReady for Prime Time?” IEEE Software, vol. 16, no. 4, 1999, pp. 68. [25] Les Hatton. 1997. N-Version Design Versus One Good Version. IEEE Softw. 14, 6 (November 1997), 71-76. [26] Christopher J. Roy, William L. Oberkampf, ”A Complete Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing” 48th AIAA Aerospace Sciences Meeting 4 - 7 January 2010, Orlando, Florida. [27] A. Szabo and N. S. Ostlund. Modern Quantum Chemistry. McGraw-Hill Publishing Company, New York, NY, 1989. [28] Hatton, L.; , ”The T experiments: errors in scientific software,” Computational Science & Engineering, IEEE , vol.4, no.2, pp.27-38, Apr-Jun1997.
[29] H. Dachsel, H. Lischka, R. Shepard, J. Nieplocha, and R. J. Harrison. A massively parallel multireference configuration interaction program - the parallel COLUMBUS program. Journal of Chemical Physics, 18:430, 1997. [30] O. Vahtras, J. Almlof, and M. W. Feyereisen. Integral approximations for LCAO-SCF calculations. Chemical Physics Letters, 213(5, 6):514, 1993. [31] B. P. Miller, L. Fredriksen, B. So, An Empirical Study of the Reliability of UNIX Utilities, Communications of the ACM 33, 12, December 1990, pp. 32-44. Also appears in German translation as Fatale Fehlertrchtigkeit: Eine Empirische Studie zur Zuverlassigkeit von UNIX-Utilities, iX (March 1991). [32] B. P. Miller, D. Koski, C. P. Lee, V. Maganty, R. Murthy, A. Natarajan, J. Steidl, Fuzz Revisited: A Reexamination of the Reliability of UNIX Utilities and Services, University of Wisconsin-Madison, 1995. Appears (in German translation) as Empirische Studie zur Zuverlasskeit von UNIX-Utilities: Nichts dazu Gerlernt, iX, September 1995. [33] J.E. Forrester and B.P. Miller, An Empirical Study of the Robustness of Windows NT Applications Using Random Testing, 4th USENIX Windows Systems Symposium, Seattle, August 2000. Appears (in German translation) as Empirische Studie zur Stability von NT-Anwendungen, September 2000. [34] A. Hertzfeld, Revolution in the Valley, O’Reilly Media, Inc., Sebastopol, CA, 2004, pp. 184-185. [35] Weirong Zhu, Yanwei Niu, and Guang R. Gao. 2007. Performance portability on EARTH: a case study across several parallel architectures. Cluster Computing 10, 2 (June 2007), 115-126. [36] Manual for instructions on using InsightII on Linux based systems: http://benedick.rutgers.edu/softwaremanuals/insightIImanual.pdf. [37] Pymol: http://pymol.sourceforge.net/newman/userman.pdf. [38] Ctest Module: http://www.cmake.org/cmake/help/v2.8.8/ctest.html.
