2014 8th Malaysian Software Engineering Conference (MySEC)
A Review of Software Operational Profile in Software Reliability Engineering Malik Muhammad Ali-Shahid1,2, Shahida Sulaiman2 1
Faculty of Computer Science COMSATS Institute of Information Technology Vehari, Pakistan 2 Faculty of Computing Universiti Teknologi Malaysia 81310 Skudai, Johor, Malaysia
[email protected],
[email protected] Abstract—Software reliability engineering is evolving in tandem with the growth of software engineering research and practice. The software operational profile has proven its merits in reliability and quality enhancement. It has become essential tool and guidance to testers. Simultaneously, it saves cost, time, and effort. Software operational profile development requires improvements with the evolution of interrelated technologies, growing scales of software, incorporation of new frameworks and matured requirements of the customers. This research focuses on the review of a software operational profile, its extension, enactment, and shortcomings. In the literature review procedure, a study selection criterion is shaped to guide the literature search process. The research summarizes the outcome of 30 main (51% of the total 58) studies that are filtered and selected. The findings of the research is an initial step leading to the construction of a comprehensive software operational profile eliminating common issues in it and enhancing software performance overall.
it is fundamental to SRE [2], [6]–[11]. It also helps to increase productivity, reliability and speed up software testing by prioritizing and allocating testing resources efficiently. SOP guides allocation of such resources to the most required functions to be tested in conjunction with their future probabilistic usage [7], [9].
Keywords—Software operational profile; Software reliability engineering; Software reliability based testing
The vital components of a software application with respect to operational profile are operations intrinsically implemented as tasks of the underlying system, which are called functions at the design level. A comprehensive set of operations arranged and sorted with matching usage probabilities shape up software operational profile [12], [13] usually called “operational profile” as described in Fig. 2; operational profile itself is further divided into subprofiles. For software under test sub-profiles are used to classify customers, map user environment, enlist system modes, analyze functions and evaluate the operations in details [7], [15]. SOP was devised as a guide for testers to safeguard the required reliability in case of early delivery enforcement [14]. With the inclusion of different studies, the prime objective of the research is to determine the share, significance of SOP in the context of SRE and to perform the analysis and explore limitations or issues that still exist. To accomplish this review process was carried out under the guidelines of the subject experts [16]–[19]. The goal is to explore future research problems in the aforementioned context.
I.
Fig 1. Stepwise software reliability engineering process integerated with software operational profile
INTRODUCTION
Reliability is a key indicator for the safe functioning of modern technological systems [1]. Software reliability analysis is indispensable for contemporary systems, significant amount of system functionality is now software reliant and software is exposed to failures [2]. Many frameworks, models and guidelines have been laid down to achieve the target reliability. Fig. 1 lists the core process steps required for the reliability achievement. The foundation of Software Reliability Engineering (SRE) is Software Operational Profile (SOP) [3], [4]. SOP is used to evaluate, foresee and enhance software reliability [5], being a probabilistic characterization of software usage [4]. Continuous research, expert recommendations and demands of software testers are requiring more to be performed by SOP [4]. The orthodox SOP model and its protracted variants have been studied to spot their issues limitations, highlight possible research problems and categorize the nature of research done on the subject. SOP is a measurable portrayal of system usage in the real world, thus
978-1-4799-5439-1/14/$31.00 ©2014 IEEE
In this paper, Section II formulates the literature selection process of the studies, Section III encompasses data extraction, and Section IV includes the related work that
1
explicitly contribute to SOP in the context of SRE. The review also excluded the prefaces, editorials, and summaries of tutorials, panels and poster sessions. Furthermore, for duplicate articles on the same study that exist in many books, journal papers, conferences and workshop papers, the study only considers the most complete version while the rest are excluded (See complete inclusion-exclusion criteria in Appendix-1 table 1).
reviews operational profiles. Finally, Section V concludes this research.
B. Search Process The search process carried out with the help of the search keywords as listed partly in Table I and with a search framework as depicted in Fig. 4 (phase 1). TABLE I. Fig 2. Complete inner architecture/breakdown of SOP [7]
II.
SEARCH STRINGS USED IN SEARCH PROCESS (KEYWORDS IN PHASE-1)
Keywords examples (continuous refinement occurred after each search) SOP based USE of SOP based Other areas of SOP Software reliability Software operational Operational profile for engineering with profile reliability operational profile enhancement Enhanced software Limitations of Review of operational operational profile operational profile profile Scenario based Software testing using Application of operational profile operational profile operational profile in software
STUDY SELECTION PROCESS
Initially few papers were handpicked seeing the titles and abstracts. Then, a citation based on forward snowballing strategy was adopted [16], computing inclusion and exclusion criteria and examining search statistics of the focused domain (the existing research is shown chronically in Fig. 3). In subsequent stage, specialized search queries were formed to gather the studies that satisfied the inclusion-exclusion criteria and its match relevance.
Fig 3. Annual publication and citation flux shown with count of studies over timeline with 1553 total studies
A. Review Protocol The research included only studies written in English language. Included studies were taken from quality journals (covering the related domain), conference proceedings (with sufficient content quality), and taken from genuine or reputable sources known such as ISSRE (International Symposium on Software Engineering) or ICSE (International Conference on Software Engineering). The studies published in last thirteen years show the publication flux in citation and publication statistics produced by search databases that were used for this research (as shown in Fig. 3; drawn using Microsoft academic search).
Fig 4. Phase-1 showing search process on the base of keywords and general queries
Types of studies such as journal papers, conference proceedings, workshops and tutorials were generally fetched by search engines using keywords. However, after focusing the search to only authentic research databases and adopting the customized search process, configuring advanced search forms by injecting refined keywords using specific search operators and search fields, the study obtained better results.
The most significant studies of year 1972 (even old but highly related) and onward till 1997 are also included, then studies from emphasizing years where studies started to roll out are included. The review excluded the studies that do not
The search sources focused are given in Table II with their customized query patterns. It presents the results of the search
2
process by calculating the percentage of search source databases contributing in total core studies. TABLE II.
having ISI-indexing, Scopus indexing or having good ERA ratings. Five studies (16% of the core studies) from conferences or symposium with high quality and relevance are included from ISSREW[21], Reliability and Maintainability Symposium[2] or Reliability Maintainability and Safety[12]. Content from workshops, book chapters, presentations and reports were rejected by filtering (82% of total searched). In the next step, the relevance of 58 studies is analyzed using expert review. Hence, 26 studies (44% of total filtered so far) are rejected. In the last filtering step, the remaining 32 studies are checked for duplication and two more (3% of total filtered) studies are rejected as the subject of the work [6] was also covered in another study of the same author and similar case occurred in another work though authors are not the same [5]. Fig. 6 shows the share of publishers or search databases in the core studies which are accepted for review process (feed-in for phase-2 is shown in Fig. 5). Fig. 6 shows the number of studies coming from a journal or a conference and share of publishers (details in Appendix-1 table 2). It is evident that IEEE is the largest publisher (46%) for the core studies that contribute to this research and ISSRE is the largest source (20%) of the core studies accepted for the review.
SHARE OF SOURCES/SEARCH DATABASES FOR STUDIES FILTERED AS CORE STUDIES
Source Name
Share (%age)
IEEExplore
21
ACM ELSEWHERE
21 11
SpringerLink
14
CiteSeerX
16
WileyOnlone
10
WASET Others
0.4 22
Customized search (enforced keyword = operational profile, optional keyword= software operational profile”) (("Publication Title”: enforced keyword) OR "Abstract":keyword2) (Abstract: ":enfrcd kywrd”) and (title:”kywrd2”) pub-date > 1999 and TITLE-ABSTR-KEY(optionalkywrd "enforced keywrd2") or (opt2 "enfrcd kywrd3") 'Optional kywrd1 AND optional kywrd2 AND "enforced kywrd1" AND NOT (not-required-keyword1)' “title:(enforced ) AND abstract(software operational profile AND year:[start TO end]” Enfrcd kywrd in Publication Titles AND optional kywrd in Abstract between start-years and end-year title: “enforced keyword1”+ using + optional keyword1~ allintitle: “enforced keyword1” OR intitle: optional keyword1+ allintext: other related keywords
C. Quality Filtering A step-by-step filtering process is adopted as shown in Fig. 5 (phase 2) justified through [16]-[19]. Each step only filters certain amount of search results, which are fed in to the next step for more focused filtering. The first step aimed to achieve the most recent or significant studies over the total timeline of SOP/SRE.
Fig 6. Contribution of journals and conferences for core studies (count) with share percentage of publisher/sources
III.
DATA EXTRACTION AND SYNTHESIS
The final analysis of the filtered studies required a thorough reading and synthesis of the studies towards the contribution to SOP in terms of analysis, implementation, extension or specialization for particular scenario and their rating based upon quality factors (see bubble chart given in Fig. 7). In theory, SOP always exhibits the increase in reliability. Hence, it promotes productivity by allocating testing resources to the essential elements of the software product [8], [11]. Nevertheless, synthesis of the core studies shows that this is not always true for all scenarios [22]–[24]. Synthesis also depicts that software is tested for all operations sequentially in a controlled manner before it is released and the optimal testing policy is determined through nonlinear programming formulations [9], [25]. The synthesis also reflects that the delivered reliability of software is dependent on operational profile closer to the
Fig 5. Phase-2 Showing filtering process for searched studies
The filters check the searched results over the date and due to the filtering process only 20% of the studies remain. The most significant studies that have good citations are selected even published before 1999 (26% of core studies). After this year, the fluctuation of citation or publication became high as depicted in Fig. 3. Hence, studies that are relevant in this era (17% of the searched studies) as per their title/abstract to SOP are also filtered. Note also one of the [20] was filtered previously but here it is rejected on basis of relevance. Next filter passes the remained content on the basis of quality of journals containing those studies (55% of relevant studies)
3
TABLE III. LIMITATIONS OF SOP AS DEPICTED THROUGH SYNTHESIS USING
perception of user’s requirements; specifying how the users will use the modules as well as on the number of defects that still remain in each module [26]. Moreover, the synthesis also reflects that testing driven by an operational profile approach increases the reliability rapidly while reduces the intensity of failures per unit of execution time [27].
LITERATURE REVIEW
#
Inputs are not diverse or not like real environment inputs used for SOP testing
[15],[37]
2
Conventional SOP covers generality of software not much effective for specialized software with non-average features Conventional SOP and many extended SOPs do not consider the complexity of the software under test Conventional SOP is no more suitable for large software and large amount of code, extended profiles only covers specific scenario Traditional SOP does not consider the impact of non-users’ factors on the software input SOP models make unrealistic restrictive assumptions to ensure tractability and solvability (data completeness, complete defect removal)
[6], [36], [38], [39]
4
5 6
Hence, it proves the importance of SOP as failures that occur most frequently are caused by the repetition of faulty operations [6]. If a design flaw, especially in large-scale software system is detected in the implementation or testing phase, it will be more difficult and more expensive to rectify than at the architecture design stage [28] and this also establishes the practical need of SOP. In order to develop SOP, several other profiles are also developed [7]. Synthesis describes developing an operational profile in order to guide testing that is not an easy job [7], [22], [29]. This involves progressively narrowing perspective from customers down to operations [6]. SOP is further used to speculate probabilities of operations in customers’ use of the software and to focus the testing more on the operations which contain higher probabilities trying to minimize the chances of failure after rapid delivery [7], [30]–[33]. Below are the strong points of SOP and motivations to use it for reliability improvement: • SOP remained most applicable and handy to enhance the reliability in the domain of SRE [34] • SOP improved the testing time, effort required and overall cost of the software [7], [13], [21]–[23], [35], each study come up with a different improvement percentage. • SOP has become a framework for design and requirement analysis of software with detailed profiles, usage estimation, risk analysis, test guidance, software scheduling and reliability estimation [3], [8]. • Guiding the testers is time saving while using SOP [2], [7], [36].
Reference
1
3
Fig 7. Bubble chart is categorizing filtered studies in type-classification and comparing all studies on quality factors associated
Limitations of SOP
IV.
[29],[31], [37] [14],[37]
[12] [5], [10], [40]
Research problem How to generate realistic inputs to be used for SOP based testing? How to increase the software coverage and diversity for SOP? How to increase complexity handling capability of SOP? How to reform SOP to test and support large sized software? How to extend SOP for non-user factors? How to use SOP while being realistic to actual software environment?
RELATED WORK
The study reflects that SOPs have become mature over the time with research and advancements in supporting areas, but many research issues still exist. If specific environment inputs exist that may lead to failures with severe consequences and cannot be confidently estimated using conventional techniques, then it may be necessary to develop new techniques for these specific scenarios [41]. The ability of a fault to hide is a crucial characteristic of software complexity [37]. SOPs are still not using realistic inputs drastically effecting required reliability [15]. With the change in nature, real-time-ness and complexity of software the need for SOPs arises while SOP staying general cannot improve the reliability [6], [36], [38]. Environmental changes, non-user and non-interactive factors, automatic test generation and complexity coverage are given less treatment even in extended versions of SOPs [12]. SOP being statistical model uses the input data and parameters with many assumptions which are unrealistic and at end the estimated reliability will not be the actual reliability as required by the customer or company [5], [10], [40]. V. CONCLUSION The study has reviewed the studies with regard to SOP in the context of SRE. Besides, the study also explored the limitations, key issues, significance, and future research areas related to SOP. This paper contributes as an insight on possible research questions and issues in the refinement of SOP. Generally, the study shows that i) the selection of test cases is not covered, ii) no specified identification for the operation of criticality is suggested [37], iii) the automatic generation of operational profile is not considered for all
The weaknesses or limitations found in SOP are summarized in Table III with the help of core studies as filtered above.
4
scenarios [6] and iv) The extended SOP based upon AI-based meta-programming architecture is also not scaling well as it imposes a high maintenance cost [5], [23].
[20]
ACKNOWLEDGMENT
[21]
The authors acknowledge FRGS, Ministry of Education (MoE) Malaysia (cost center 4F26) for its partial support. The main author also thanks Universiti Teknologi Malaysia for motivation and support along with COMSATS Institute of Information Technology Pakistan for the financial support.
[22]
REFERENCES
[24]
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]
[18] [19]
[23]
E. Zio, “Reliability engineering: Old problems and new challenges,” Reliability Engineering & System Safety, vol. 94, no. 2, pp. 125–141, Feb. 2009. S. Arora, R. B. Misra, and V. M. Kumre, “Software reliability improvement through operational profile driven testing,” in Reliability and Maintainability Symposium, 2005. Proceedings., 2005, pp. 621–627. J. P. Fu and M. Y. Lu, “Develop software operational profile with uniform design,” in 2009 IEEE International Conference on Industrial Engineering and Engineering Management, 2009, pp. 842–846. C. Smidts, C. Mutha, M. Rodríguez, and M. J. Gerber, “Software testing with an operational profile,” ACM Comput. Surveys, vol. 46, no. 3, pp. 1–39, Feb. 2014. K. S. Kumar and R. B. Misra, “Software operational profile based test case allocation using fuzzy logic,” International Journal of Automation and Computing, vol. 4, no. 4, pp. 388–395, Oct. 2007. S. Ramanujam, H. Yamany, and M. Capretz, “An Agent Oriented Approach to Operational Profile Management,” International Journal of Intelligent Technology, vol. 16 2008–04, no. 1, pp. 334–344, 2006. J. D. Musa, “Operational Profiles in Software Reliabilitv Engineering,” IEEE Transactions on Software Engineering, no. March, 1975. F. Urem and Z. Mikulic, “Developing operational profile for ERP software module reliability prediction,” MIPRO, 2010 Proc. 33rd International Convention, pp. 409–413, 2010. S. Özekici and R. Soyer, “Reliability of software with an operational profile,” European Journal of Operation Research, vol. 149, no. 2, pp. 459–474, Sep. 2003. C.-G. Bai, “Bayesian network based software reliability prediction with an operational profile,” Journal of System and Software, vol. 77, no. 2, pp. 103– 112, Aug. 2005. S. Sharma and M. Ingle, “Ontology based Specifications for Software Reliability Advancement,” International Journal of Computer Applications, vol. 43, no. 13, pp. 18–26, Apr. 2012. Q. Ao, J. Ai, M. Lu, and F. Zhong, “Scenario-based software operational profile,” in The Proceedings of 2011 9th International Conference on Reliability, Maintainability and Safety, 2011, pp. 700–704. H. Koziolek, “Operational Profiles for Software Reliability,” in Seminar “Dependability Engineering,” 2005, pp. 1–17. J. D. Musa, “Software reliability-engineered testing,” Computer (Long. Beach. California)., vol. 29, no. 11, pp. 61–68, 1996. M. Gittens, H. Lutfiyya, and M. Bauer, “An Extended Operational Profile Model,” in 15th International Symposium on Software Reliability Engineering, 2004, pp. 314–325. B. Kitchenham and P. Brereton, “A systematic review of systematic review process research in software engineering,” Information Software Technology, vol. 55, no. 12, pp. 2049–2075, Dec. 2013. P. Brereton, B. a. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Lessons from applying the systematic literature review process within the software engineering domain,” Journal of System and Software, vol. 80, no. 4, pp. 571–583, Apr. 2007. K. A. Sedek, S. Sulaiman, and M. A. Omar, “A systematic literature review of interoperable architecture for e-government portals,” 2011 Malaysian Conference on Software Engineering, pp. 82–87, Dec. 2011. B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, “Systematic literature reviews in software engineering – A
[25] [26] [27] [28] [29]
[30] [31]
[32] [33] [34] [35] [36]
[37] [38] [39] [40]
[41]
5
systematic literature review,” Information Software Technology, vol. 51, no. 1, pp. 7–15, Jan. 2009. J. C. Munson, “Software faults, software failures and software reliability modeling,” Information Software Technology, vol. 38, no. 11, pp. 687–699, Nov. 1996. Q. Li, L. Luo, and J. Wang, “Accelerated reliability testing approach for high-reliablity software based on the reinforced operational profile,” 2013 IEEE International Symposium on Software Reliabilitly Engineering Workshop, pp. 337–342, Nov. 2013. P. G. Bishop, “Rescaling reliability bounds for a new operational profile,” ACM SIGSOFT Software Engineering Notes, vol. 27, no. 4, p. 180, Jul. 2002. K.-Y. Cai, C.-H. Jiang, H. Hu, and C.-G. Bai, “An experimental study of adaptive testing for software reliability assessment,” Journal of System and Software, vol. 81, no. 8, pp. 1406–1429, Aug. 2008. J. Voas, “Why testing software under expected operational profiles is not sufficient,” Software Focus, vol. 2, no. 1, pp. 13–14, 2001. C. Stringfellow and A. A. Andrews, “An Empirical Method for Selecting Software Reliability Growth Models 1 Introduction 2 Background,” Empirical Software Engineering, vol. 7, no. 4, pp. 319–343, 2002. P. Cao, Z. Dong, K. Liu, and K. Cai, “Robust Dynamic Selection of Tested Modules in Software Testing for Maximizing Delivered Reliability,” Computing Research Repository, vol. 1309.3052, p. 16, Sep. 2013. T. Takagi, Z. Furukawa, and Y. Machida, “Test Strategies Using Operational Profiles Based on Decision Tables,” 2013 IEEE 37th Annual Computer Software Application Conference, no. 1, pp. 722–723, Jul. 2013. W. Wang, D. Pan, and M. Chen, “Architecture-based software reliability modeling,” Journal of System and Software, vol. 79, no. 1, pp. 132–146, Jan. 2006. M. Nagappan, K. Wu, and M. a. Vouk, “Efficiently Extracting Operational Profiles from Execution Logs Using Suffix Arrays,” in 2009 20th International Symposium on Software Reliability Engineering, 2009, pp. 41– 50. A. Asthana and K. Okumoto, “Integrative Software Design for Reliability: Beyond Models and Defect Prediction,” Bell Labs Techology Journal, vol. 17, no. 3, pp. 37–59, Dec. 2012. B. Cukic and F. B. Bastani, “On reducing the sensitivity of software reliability to variations in the operational profile,” Proc. ISSRE ’96 7th International Symposium on Software Reliability Engineering, pp. 45–54, 1996. B. D. Jensen, “A software reliability engineering success story. AT&T’s Definity PBX,” in Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE’95, 1995, pp. 338–343. J. Musa, Iannino, Anthony, Okumoto, and Kazuhira., Software reliability : measurement, prediction, application. New York; Montrial: McGraw-Hill, 1987. P. Jalote and B. Murphy, “Reliability Growth in Software Products,” 15th International Symposium on Software Reliability Engineering, pp. 47–53, 2004. S. K. Tipu and S. Zia, “An Effort Estimation Model for Agile Software Development,” vol. 2, no. 1, pp. 314–324, 2012. A. E. Hassan, D. J. Martin, P. Flora, P. Mansfield, and D. Dietz, “An Industrial Case Study of Customizing Operational Profiles Using Log Compression Categories and Subject Descriptors,” Proc. 30th International Conference on Software Engineering, pp. 713–723, 2008 J. Voas and J. A. Whittaker, “Toward a more reliable theory of software reliability,” IEEE Computer Society, vol. 33, no. 12, pp. 36–42, 2000. J. Voas, “Deriving Accurate Operational Profiles for Mass-Marketed Software,” Reliable Software Technologies, NSF, 2000. J. H. Poore, G. H. Walton, and J. a. Whittaker, “A constraint-based approach to the representation of software usage models,” Information Software Technology, vol. 42, no. 12, pp. 825–833, Sep. 2000. K. Kanoun, M. Kaaniche, and J.-C. Laprie, “Experience in software reliability: From data collection to quantitative evaluation,” in Proceedings of 1993 IEEE International Symposium on Software Reliability Engineering, 1993, no. 6362, pp. 234–245. L. Luqi, V. Berzins, and P. Dailey, “Driving Automated Open-Architecture Testing: An Operational Profile Model-development Strategy,” Monterey, 2009.
APPENDIX-1 TABLE I. INCLUSION AND EXCLUSION CRITERIA IN DETAILS USING SYSTEMATIC METHOD Exclusion Inclusion L1. Language L2. Focus L3. Subfocus
L4.Year Range
Content available in English or translated in English. Content mainly focusing SRE techniques in general and SOP in particular. Content covering new/extended/proposed software operational profile for generalized/specialized software. Content covering SRT using SOP. Content covering software reliability estimation inside SOP. Picking early content of high significance and high relation from 1975 to 2000 Picking most of the related content from 2000 to till date Content focusing upon software testing alone Content focusing upon software reliability estimation without operational profile Content focusing upon statistical models of reliability engineering
E1.Nonrelevance
Content focusing upon software testing alone Content focusing upon software reliability estimation without operational profile Content focusing upon statistical models of reliability engineering
E2.Type filter
E3.Duplicates E4. Chronicle
Content focusing upon operational profile used only for any other entity other than software (one example is operational profile for hardware, operational profile for network or operational profile for quality of service etc.). Prefaces, editorials, and summaries of tutorials, panels, presentations, non-authorized reports and poster sessions The study already published or part of book or any other article or incomplete in current document(s) The content published before 1975 Content published between 1975 to 2000 related but less significant Content published between 2000 to date unrelated or not in library search
#
1 2 3 4 5 6 7 8
TABLE II (THE COMPLETE RESULTS OF FIG. 5 AND 6 IN THE MAIN CONTENT OF THE PAPER Search process (Phase1) Filtering process (Phase2) L4(old-sig) but not E4(outdated Title based Enhanced Keyword or insignificant) keyword L3 but not Abstract Not E2 result Filter search E1 (Focused based (Type (Seed) Source (L2) Irrelevance) (L3) filter) Loose queries Custom L4A(bef2000)+L query 4B(aft2000)-E4
Not E3 (Duplication)
IEEE Xplore
3243
429
362
57
47
20
14
14
ACM
7520
352
338
24
22
5
4
4
ELSEWHERE
12572
5120
189
39
28
8
5
5
CiteSeerX
21209
2188
275
65
47
2
1
1
Wiley Online
1809
267
169
46
31
5
2
2
Springer Link
425
321
239
35
33
9
4
2
WASET
28
28
8
7
5
2
1
1
Others
7874
200
77
62
54
7
1
1
54680
8905
1657
335
267
58
32
30
TOTAL (Columnar)
6
