2017 IEEE/ACM Joint 5th International Workshop on Software Engineering for Systems-of-Systems and 11th Workshop on Distributed Software Development, Software Ecosystems and Systems-of-Systems (JSOS)
HEAL ME - An Architecture for Health Software Ecosystem Evaluation Iuri Carvalho¹, Fernanda Campos², Regina Braga², José Maria N. David², Victor Stroelle², Marco Antônio Araújo² Computer Science Postgraduate Program Federal University of Juiz de Fora Juiz de Fora, Brazil ¹
[email protected] ²fernanda.campos, regina.braga, jose.david, victor.stroele, marco.araujo{@ufjf.edu.br} of a SECO, its strength and longevity are named "health" [4, 19]. Considering this scenario, we can observe the risk of losses due to investments made in SECOs with low health conditions. For this reason, defining and controlling the health of a SECO are important factors [5]. This control can be accomplished by defining metrics and evaluating them. Since it is a systematic activity, it is also interesting that this control be as automatic as possible to accelerate the process and reduce any errors. With SECO health control, we can ensure that the investments made are not wasted, increasing the influence of the platform and, consequently, of its products and services. As a proposal solution, this paper describes an architecture, named Health dEfinition Architecture ontoLogy-based froM sEco (HEAL ME). This architecture performs the analysis and definition of the health assurance of a SECO based on metrics and their control. The architecture captures the SECO data, in a semi-automated way and from the use of pre-defined metrics, can analyze and present the SECO health scenario. This activity is performed through the semantic analysis of its components such as users and developers’ relationship and data. As part of the solution evaluation, the proposed architecture was used in the context of the scientific E-SECO platform [10]. The E-SECO platform uses the scientific SECO approach and considers all stages of the life cycle of an experiment [16]. As a result, the health of this platform was analyzed and defined, based on the defined metrics. Regarding the evaluation of the SECO health, based on a systematic review, we found some works that deal with SECO health. We highlight two of them as the most detailed ones. The first [7] presents a quality model for evaluation of open-source SECOs. This work presents metrics for characteristics evaluation that indicate the solidity of a SECO. The second [6], presents a conceptual framework for the evaluation of health indicators such as robustness, productivity, and niche creation. Comparing these works with the proposal presented in this article, we can say that these works do not use all the health indicators presented in our work. Besides that, they evaluate the health using manual procedures. They do not have a service that gathers information from SECO environment. In this first prototype,
Abstract—Software Ecosystems are complex environments. Several actors are grouped on a software platform, facilitating investments in research, development, and reuse of components. However, such investments may be lost if the ecosystem is not solid. Therefore, it is important to assess the health of an ecosystem. This work presents HEAL-ME architecture, which main goal is to evaluate the health of software ecosystems. A pilot case study using a scientific software ecosystem is also presented. Enhancements will be provided for a complete SECO health assessment. Keywords-component — SECO Health; Software Ecosystem; Architecture; Ontology; Quality.
I.
INTRODUCTION
Software Ecosystem (SECO) concept has been widely discussed in Software Engineering. According to Bosch and Bosch-Sijtsema [1], a SECO can be defined as "a software platform, a set of internal and external developers and a community of domain specialists serving a community of users that make up relevant solution elements to meet their needs". Some advantages of adopting the SECO approach are often addressed in the specialized literature. According to Bosch [2], the companies that maintain the platforms gain by reducing the cost in research and development, which are often carried out by co-development companies. Accelerating the innovation process is another advantage of adopting this approach. Also, the knowledge diffusion and the reuse of software are facilitated within a SECO. Information exchanges, components, and other artifacts are common activities on these platforms. Thus, the greater the openness offered by the platform, the greater will be the distribution of knowledge among the involved actors [3]. The solidity and longevity of a SECO are critical factors. Solidity refers to the fact that SECO integrity is not affected by factors such as problems related to design, management, and technology changes. Longevity is the ability of SECO to maintain itself over a long period of time [6]. The criticality of these factors is due to the involvement of several companies, and the high effort and financial investments made. In order to avoid the loss of these investments, there can be no risk of SECO death or failures to meet the expectations of those who adopt it [11]. The well-functioning 978-1-5386-2799-0/17 $31.00 © 2017 IEEE DOI 10.1109/JSOS.2017.13 10.1109/JSOS.2017..13
59
HEAL-ME architecture performs the evaluation of five main health indicators, in a semiautomatic way, using semantic analysis. This paper is structured as follows: section 2 presents the theoretical background, where the SECO health and the technologies related to our proposal are discussed; Section 3 presents some related works; Section 4 presents the architecture; Section 6 discusses the application of the architecture in the E-SECO platform; finally, Section 5 presents the final considerations. II.
world models [8]. In addition, their formal representation allows the use of reasoning services and easy extensibility. Using ontologies for metric analysis is already in the literature [8]. However, the use of logical rules, aiding in the combination of indicators is a solution not yet explored in the reviewed literature [13]. III. RELATED WORKS From a systematic mapping1 carried out, it was possible to identify that there are few studies that deal with SECO health. We found 12 articles that directly or indirectly address SECOs health. Among them, three articles were highlighted, where health indicators, their metrics and evaluation methods are described. The work described by Jansen [3] focuses on defining the existing quality attributes and the characterization of software platforms. Through a case study, the author sought to confirm that platform extenders have architectural problems when selecting it. It also sought to verify if the platform architecture influences the development behavior of these extenders and restricts the support provided by the SECO platform itself. At the end the results are presented, pointing out the great influence of the platform characteristics directly on the performance of the solutions developed for extending the platform. In this way, the need to measure the health of the “keystones” was evidenced. Lessons learned on the structure and functioning of SECO were also listed from the perspective of their extensors. In this work, the health of SECO is analyzed indirectly, analyzing its integrity and membership. Such characteristics are linked to sustainability. However, this analysis is done partially, disregarding details of its composition. A SECO can be viewed from the perspective of a network of actors, organizations, and companies; or by technical focus and social aspects, defining it as a group of software projects and their communities. From these perspectives, the work presented by Franco-Bedoya et al. [7] proposes to access the quality of SECO in a broader sense. Through a Systematic Review of Literature (SRL), FrancoBedoya et al. [7] collected the most important characteristics for the structure of a SECO. As a result, the model, named QuESo, was generated. This model was composed of two types of elements: quality characteristics and measurements. The model has the following indicators and characteristics: sustainability, maintenance, process maturity, health of the network and health of resources. They are important for SECO health definition. In this way, the proposed model assists in the assurance of these quality dimensions, inherent to SECO. However, the work presented by Franco-Bedoya et al. [7] has a restricted focus on open-source SECOs. In addition, other health indicators commonly cited in the literature, such as diversity and productivity, are not addressed. Finally, this work presents the QuESo model only as a theoretical
BACKGROUND
SECOs are complex environments, composed by diverse actors interacting in a distributed software development environment [5], with a platform, where these actors give technological support to the established environment. In addition, independent development and users’ communities are created, in addition to the existing SECO development companies. Also, new technologies emerge all the time, as well as new needs and requirements of software that deserve to be treated [12]. All these factors impact in the SECO health. According to Santos et al. [5] SECO health is the ability of a SECO to remain stable and expand, meeting everyday demands. In other words, measuring the health of a SECO is to define the degree to which it offers opportunity for the employees and the actors who depend on it. It is important that the network of relationships of a SECO be strong, and that it does not diminish or end up. This is due to the degree of investment that exists both on the part of the company that maintains the platform, called "keystone", and by companies and development communities [2]. Thus, the health of a SECO is a decisive factor. The literature describes five main aspects of SECO health. Dhungana et al. [4] discuss two of them, named: sustainability and diversity. These aspects are linked to the SECO network of relationships, their stability, and their ability to involve different collaborators. Jansen [6] discusses three other aspects: productivity, robustness, and niche creation. These are directly linked to the production and the possibility of SECO expansion. For the operationalization of the developed research, the aspects were called health indicators. This approach was necessary for the decomposition of health feature thus allowing its analysis. For the analysis of the indicators, Franco-Bedoya et al. [7] proposed to divide them into characteristics which are defined by metrics. To exemplify, the sustainability indicator has the characteristic of heterogeneity, which can be defined by the metric of geographic diversity, among others. Another way to analyze the indicators is to directly use a set of metrics [6]. It is through the process of analyzing these metrics that the definition of characteristics and health indicators can be carried out. The use of ontologies and inference rules can aid in the analysis of these data and relationships [8]. Ontology can be defined as a modeling of explicit information consisting of properties, concepts, classes, and their relationships. In contrast to databases, ontologies allow us to work with incomplete knowledge because of their support for open
1
Systematic Mapping of SECO Health Available at: http://www.ufjf.br/nenc/files/2008/09/SecoHealthMapping. pdf
60
data from communities, users and developers, platform visibility data and their products.
framework, not presenting automation of its operation. The HEAL-ME architecture includes all of these items. The work presented by Jansen [6] aims to analyze and define the operation of Open Source SECOs and their health, based on three indicators: productivity, robustness, and niche creation. The author suggests that the health of a SECO can be defined by two factors: longevity and growth capacity. Another observation described by the author is the difference between SECO health and software projects health. The software project health is defined through metrics such as: commits consistency, tracking and error correction, number of releases and number of downloads. The definition of SECO health depends on factors such as: relationship network, and capacity for expansion. The author presents such factors as complex to be measured. The author's approach provides an overview of the metrics needed to define the health of a SECO. As a result, a framework called Open Source Ecosystem Health Operationalization (OSEHO) was generated. The work presented by Jansen [6] presents metrics and processes for the definition of productivity, robustness, and niche creation indicators. However, sustainability and diversity indicators, important for the longevity of a SECO, are not addressed. In addition, the OSEHO framework is conceptual. Therefore, this work does not include the automation of the suggested metrics. The studies cited in this section present models and some metrics, which are the state of the art for SECO health definition [6, 7]. These papers also present the importance of the health definition, and the impact that this scenario has on the daily work of SECO component developers. However, none of them addresses all aspects of health cited in the literature. They do not present also a proposal for process automation. HEAL-ME architecture proposes the use and application of the metrics, as well as the automation of the health quality evaluation process. IV.
Figure 1. HEAL-ME architecture.
Data about the platform itself are also captured, such as number of employees, technologies, among others. For these connections, the architecture uses APIs repositories, such as GitHub API, BioCatalogue and myExperiment. Through these APIs, the project data is captured directly from the repositories. Data that are not present in the repositories are captured directly from users’ interface. Other important data is perception data. The users of the platform also provide these data, to parameterize the evaluation using the metrics. Finally, this layer also includes the interfaces analyses, through which the analyzed data is available for consumption by platform applications (services), content managers, users, and other SECO participants. In the service layer are the services responsible for data flow. There are services responsible for capturing the data coming from the interfaces and then for the connection to the analysis service. This service is responsible for inserting the collected data into ontologies, that are responsible for their interpretation. The analyzed data can be queried, if necessary. Finally, the analysis service has the function of returning the data to the communication layer, so the data can be processed. The persistence service is responsible for communicating with the data repository, storing and retrieving data already processed. The knowledge discovery layer encompasses ontology, along with the evaluation rules. In this layer the data repository is also present. This repository stores the data already processed by the ontology and definitions already made on the health of the evaluated SECO. These data are stored for future use by project managers or by keystone themselves. This storage is useful for comparison with new platform evaluations.
HEAL-ME ARCHTECTURE
This section presents the HEAL-ME2 architecture, whose main goal is to evaluate the SECO health using semantic rules applied to its data. The analysis is made in data from SECO, its products and environment. Metrics are applied to these data, according to each of the five health indicators discussed above. After the processing of this information, it is possible to evaluate the health conditions of the SECO platform, as well as its constrains. Figure 1 presents an overview of the architecture. The architecture is divided into three layers. One of the main components is the ontology, specified in OWL 2.0 language [17], where the SWRL rules that implement the automatic evaluation of the metrics are expressed. User interfaces were built as web applications. The communication layer is responsible for the coupling of the architecture to the SECO structure and its applications. Through communication interfaces, this layer is responsible for capturing SECO components data. These include project 2
Available at: https://github.com/pgcc/HEAL-ME
61
Figure 2. OntoHealth.
As already mentioned, to analyze the SECO health using HEAL-ME, an ontology named OntoHealth was built. It models the SECO domain, its network of collaborators, actors, applications aggregated to the platform and the keystones. Figure 2 presents the OntoHealth overview. By populating this ontology with data collected from SECO and using rules expressed in Semantic Web Rule Language (SWRL) [18], it is possible to present an initial overview of the platform health, by evaluating its characteristics and indicators. To perform the evaluations, the ontology uses 27 SWRL rules 3 . These rules process the metrics available in the literature to analyze each of the health indicators, and their respective characteristics. The sustainability indicator, for example, has five characteristics: heterogeneity, regeneration ability, effort balancing, expertise balancing and visibility. The robustness indicator has four characteristics, which are: interrelationship, information consistency, grouping and financial consistency. The metrics analysis defines the indicators diversity, productivity, and niche creation. For the first version fifty eight metrics were generated and included in the ontology. The metrics for sustainability indicators and for the characteristics of interrelationship, consistency of information and grouping were extracted from [7, 15]. The metrics for the diversity indicator were extracted from [9]. The metrics for defining the productivity and niche creation indicators, and for the financial consistency characteristic of the robustness indicator were extracted from [6]. Table 1 contains the list of the metrics. The divisions represent the indicators, numbered from 1 to 5. For each division, the characteristics that make up the indicators are displayed. The metrics are grouped within their respective indicators or characteristics, and are signaled by the letter M.
Table 1. List of metrics. [Indicator 1]Sustainability Characteristic: Heterogeinity (M1) Is there more than one country/state/city in the SECO partnership? (M2) How is the semantic closeness between services in the SECO? (M3) Which types of nodes are in the SECO peer-to-peer network? Characteristic: Regeneration Ability (M4) How long do all members of each community work together? (M5) How many new members have been added? Characteristic: Effort Balance (M6) What is the number of commits per developer? (M7) How many hours did each developer spend on each project? (M8) How many communities have at least one active developer? (M9) In how many releases did each developer participate in each project? (M10) In how many total releases did each developer participate? (M11) How long did all members of each community work together? Characteristic: Expertise Balance (M12) How many files are modified by each developer in a month? (M13) Which types of files are modified for each developer in a month? (M14) In what events did each community member participate? Characteristic: Visibility (M15) How many people are in each project event? (M16) How many jobs advertisements were divulged? (M17) How many downloads were made from the main site or mirrors? (M18) How many users are subscribed in the mailing list? (M19) How many passive users are in the community? (M20) How many readers does the community have? (M21) How many scientific publications were generated by the community? (M22) How many hits does the project get in the social media and blogs? (M23) How many web page requests were received? [Indicator 2] Diversity (M24) How many different developers exist in the community? (M25) How many users’ groups exist in the community? (M26) How many different programming languages are supported by the platform? (M27) How many hardware devices support the platform? (M28) Is there a collapse plan for the SECO? [Indicator 3] Productivity (M29) How many new projects were added in the last 30 days? (M30) How many events occurred in the community in the last 30 days? (M31) How many KLOC were added in the repository in the last 30 days?
3 Ontology, rules and web services are available at: https://github.com/pgcc/HEAL-MEWS
62
definitions. As a result, the SECO is classified as having a particular indicator, or not. The presence of the five indicators demonstrates the platform health, since these are mutually dependent features. Finally, the analyses interface shows the results of the evaluation process, and informs the status of the platform according to each characteristic, indicator and finally displays the overall health assessment. HEAL-ME was developed to evaluate the health of a SECO in a generic way. To demonstrate its operation, this architecture was applied to a scientific SECO platform named E-SECO [10]. Figure 3 shows simplified overview architecture of the E-SECO platform. E-SECO aims to provide service recovery and to support the specification of scientific workflows, facilitating reuse [10]. The blue diamond highlights the HEAL-ME architecture and its interaction with the platform.
(M32) How many artifacts were added in the repository in the last 30 days? (M33) How many messages were transmitted in the communication channels in the last 30 days? (M34) What is the average time to fix bugs? (M35) How many partnerships were added in the community in the last 30 days? (M36) How many users have used the platform in the last 30 days? (M37) How much time on average do users spend on the platform? [Indicator 4] Robustness Characteristic: Interrelatedness (M38) How many connections have each peer-to-peer node? (M39) What is the maximum connection capacity of the peer-to-peer network? (M40) What is the number of connections ratio to the maximum capacity of the peer-to-peer network? (M41) Which peer-to-peer network node has the highest number of connections? (M42) How many external partners does the platform have? Characteristic: Information Consistence (M43) Does the platform have a glossary of terms to be used in the project? Characteristic: Clustering (M44) How many product types are there on the platform? (M45) Which project received contributions from the largest number of communities? (M46) How many active projects does the platform have? (M47) What is the total number of produced files? Characteristic: Financial Consistence (M48) How many partnerships does the platform have? (M49) How many commercial patronages does the platform have? (M50) What is the total value of the contribution received by the platform? (M51) How many active contributors does the platform have? (M52) How many frequent users does the platform have? [Indicator 5] Niche Creation (M53) Does the platform have documentation? (M54) What are the types of contributors? (M55) What are the types of project applications? (M56) Does the platform support natural languages? (M57) What are the types of technologies supported? (M58) What are the types of development technologies supported?
Figure 3. Simplified Overview of E-SECO Platform with HEAL-ME architecture.
V.
.HEAL-ME IN ACTION
This section presents a pilot study of the use of the HEAL-ME architecture in the context of the E-SECO platform. This scenario has the purpose of detailing the operation of the architecture to evaluate its feasibility and applicability, from the point of view of software project managers, in the context of a scientific SECO, more specifically E-SECO. When HEAL-ME architecture is initialized by an agent (human or software), in this specific case study, a project manager, the data repositories belonging to the E-SECO platform are accessed through the communication layer. The data from development, versioning, products, and services of the platform are collected through the GitHub API and populate the ontology and database4. Additional data, such as communication and personal data from the development team were previously informed by the platform developers, including external developers (with specific authorizations). In this way, companies that have closed platforms can use it without modifications in the architecture.
Through the communication layer, the data extraction interface captures the platform data. These data are inserted into the ontology by the persistence layer. The communication layer also has the perception extraction interface. The user informs through the parameterization interface the data for the evaluation. These data will be used in conjunction with the automatic captured SECO data, to process the rules to analyze the health. Through these parameters, the metrics evaluate the SECO data, considering the characteristics and each of the health indicators. For this reason, it is important that the manager who uses the architecture has the perception of the SECO environment and its components [14]. To assure that the result reflects the real situation of a SECO, it is interesting that project managers, or keystone developers, make use of the architecture. With these data loaded in the ontology, the inference engine is processed. This action processes the rules using the loaded data and the defined parameters, generating the health
4
It is important to emphasize that only new data are collected, i.e., the database and ontology only should be updated with new data
63
Figure 4. Parameter inclusion.
The user can then analyze the E-SECO health, using the button “Analyze” in Figure 4 or consume the method “analyze” of the communication layer, in the case of a service requester. Platform data is captured automatically. However, for a proper execution of the architecture, it is necessary that the project manager complete the evaluation parameters. Figure 4 shows the data that were used as parameters for evaluation of the E-SECO platform. After setting the parameters, it is possible to perform the initial health analysis. The informed data are then processed by the ontology rules, and the reasoning machine processes the definitions related to the platform. This process occurs in the Services layer. At the end of the process, the service returns the results to the project manager. The data are returned defining the characteristics and indicators, through the metrics implemented by the rules, as presented in Figure
5. As an example, Figure 6 presents one of the SWRL rules used in defining the heterogeneity characteristic.
Figure 6. One of the SWRL Rule used for defining heterogeneity characteristic.
The results presented in Figure 5 demonstrate the definition of the Heterogeneity characteristic. These results indicate that there is a distribution of the developers, there is a semantic distance between the services provided by the
Figure 5. E-SECO Results of Health Analyses using the Heal_ME architecture.
64
[3]
platform and there are different types of nodes in the peer-topeer network. According to M1, M2 and M3 metrics presented in Table 1, these results indicate the presence of Heterogeneity characteristic. This is used to define the sustainability indicator. Likewise, the other indicators and characteristics are evaluated and the results are presented in the overview, demonstrated by Figure 5. As it can be seen in Figure 5, it was possible to verify some evidence of the health of the E-SECO platform. The feedback received from the platform managers indicates the effectiveness of the results obtained, as well as the utility to identify potential problems. In addition, evidence of the effectiveness of the HEAL-ME architecture can be observed in this specific context, i.e., a scientific SECO. However, a formal and controlled experiment is required to confirm the results presented, as well as their applicability in other SECO types, contexts and domain. VI.
[4]
[5]
[6]
[7]
FINAL REMARKS
SECO are a union of developers, communities and software users who have as a common core one software platform. The opening of this platform allows the reduction of investments by the “Keystone”, since research and development are carried out by external companies. Also, increasing the speed of innovation is one of the attractions for adopting this type of approach. This scenario allows the number of employees and users to grow. For these reasons, the effort and financial investment made by business and development communities is great. Therefore, the health of a SECO is a critical factor. Its ability to keep and expand ensures that the investments return the expected results, and attract new adepts. This work presented the HEAL-ME architecture, which goal is to semantically analyze the health of a SECO. This work also described its application on a scientific SECO platform, the E-SECO. The first results show the utility of the architecture, which was designed to analyze the health of SECO in a generic way. However, it is important to note that the HEAL-ME architecture is still a prototype. Enhancements are still required for a complete SECO health assessment. The list of metrics will be improved and expanded, and its effectiveness for other SECO types will be analyzed. Automating the process was only a first step. In addition, experiments are needed to evaluate the effectiveness of the analysis.
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
ACKNOWLEDGMENT This research has been partially supported by CAPES, CNPq, FAPEMIG, and UFJF.
[17] [18]
REFERENCES [1]
[2]
J. Bosch and P. M. Bosch-Sijtsema. “Softwares product lines, global development and ecosystems: collaboration in software engineering”. In Collaborative Software Engineering, pages 77–92. Springer, 2010 J. Bosch. “From software product lines to software ecosystems”. In Proceedings of the 13th International Software Product Line Conference, SPLC ’09, pages 111–119, Pittsburgh, PA, USA, 2009. Carnegie Mellon University.
[19]
65
S. Jansen. “How quality attributes of software platform architectures influence software ecosystems”. In Proceedings of the 2013 International Workshop on Ecosystem Architectures, WEA 2013, pages 6–10, New York, NY, USA,2013. ACM. D. Dhungana, I. Groher, E. Schludermann, and S. Biffl. “Software ecosystems vs. natural ecosystems: Learning from the ingenious mind of nature”. In Proceedings of the Fourth European Conference on Software Architecture: Companion Volume, ECSA ’10, pages 96– 102, New York, NY, USA,2010. ACM. R. Santos, G. Valença, D. Viana, B. Estácio, A. Fontão, S. Marczak, C. Werner, C. Alves, T. Conte, e R. Prikladnicki. “Qualidade em ecossistemas de software: Desafios e oportunidades de pesquisa”. In Proceedings of VIII Workshop on Distributed Software Development, Software Ecosystems and Systems-of-Systems, pages41–44, 2014 (in portuguese). S. Jansen. "Measuring the health of open source software ecosystems: Beyond the scope of project health." Information and Software Technology 56.11 (2014): 1508-1519. O. Franco-Bedoya, D. Ameller, D. Costal, and X. Franch. “Queso a quality model for open source software ecosystems”. In Software Engineering and Applications ICSOFT-EA), 2014 9th International Conference on, pages 209–221, Aug 2014. Schugerl, P., Rilling, J., Witte, R., & Charland, P. (2009, September). A quality perspective of software evolvability using semantic analysis. In Semantic Computing, 2009. ICSC'09. IEEE International Conference on (pp. 420-427). IEEE. Jansen, S., M. A. Cusumano, and Brinkkemper, S. Eds. Software Ecosystems: Analyzing and Managing Business Networks in the Software Industry. Edward Elgar Publishing, 2013. Freitas, V., et al. An Architecture for Scientific Software Ecosystem. In 9th Workshop on Distributed Software Development, Software Ecosystems and Systems-of-Systems (WDES). 2015. Belo Horizonte 41- 48 (in portuguese). van den Berk, I., Jansen, S., & Luinenburg, L. (2010, August). Software ecosystems: a software ecosystem strategy assessment model. In Proceedings of the Fourth European Conference on Software Architecture: Companion Volume (pp. 127-134). ACM. Manikas, K. (2016). Revisiting software ecosystems research: a longitudinal literature study. Journal of Systems and Software, 117, 84-103. Axelsson, J., & Skoglund, M. (2016). Quality assurance in software ecosystems: A systematic literature mapping and research agenda. Journal of Systems and Software, 114, 69-81. Stefanuto, G., Spiess, M., Alves, A. M., & Castro, P. F. (2011, November). Quality in software digital ecosystems the users perceptions. In Proceedings of the International Conference on Management of Emergent Digital EcoSystems (pp. 85-88). ACM. Bedoya, F., Hernán, Ó., Ameller, D., Costal Costa, D., & Franch Gutiérrez, J. (2016). QuESo V2. 0 a quality model for open source software ecosystems: List of measures (Technical Report). Belloum, A.; Inda, M. A.; Vasunin, D.; Korkhov, V.; Zhao, Z.; Rauwerda, H.; Breit, T. M.; Bubak, M. and Hertzberger, L. O. “Collaborative e-science experiments and scientific workflows”. IEEE Internet Computing, v. 15, n. 4, 39-47. 2011. World Wide Web Consortium. (2012). OWL 2 web ontology language document overview. Rivolli, A., Orlando, J. P., Yamamoto, C. H., & Moreira, D. (2010). Regras SWRL: Análise de similaridade e detecção de erros. In WebMedia, 11. Manikas, K., & Hansen, K. M. (2013). Reviewing the health of software ecosystems–a conceptual framework proposal. In Proceedings of the 5th International Workshop on Software Ecosystems (IWSECO), pages 33-44.
