The Intersection of Continuous Deployment and Architecting Process: Practitioners’ Perspectives a
a
a
b
Mojtaba Shahin , Muhammad Ali Babar , Liming Zhu CREST - The Centre for Research on Engineering Software Technologies, The University of Adelaide, Australia b Data61, Commonwealth Scientific and Industrial Research Organisation, Sydney, NSW 2015, Australia
[email protected],
[email protected],
[email protected]
ABSTRACT Context: Development and Operations (DevOps) is an emerging software industry movement to bridge the gap between software development and operations teams. DevOps supports frequently and reliably releasing new features and products– thus subsuming Continuous Deployment (CD) practice. Goal: This research aims at empirically exploring the potential impact of CD practice on architecting process. Method: We carried out a case study involving interviews with 16 software practitioners. Results: We have identified (1) a range of recurring architectural challenges (i.e., highly coupled monolithic architecture, team dependencies, and ever-changing operational environments and tools) and (2) five main architectural principles (i.e., small and independent deployment units, not too much focus on reusability, aggregating logs, isolating changes, and testability inside the architecture) that should be considered when an application is (re-) architected for CD practice. This study also supports that software architecture can better support operations if an operations team is engaged at an early stage of software development for taking operational aspects into considerations. Conclusion: These findings provide evidence that software architecture plays a significant role in successfully and efficiently adopting continuous deployment. The findings contribute to establish an evidential body of knowledge about the state of the art of architecting for CD practice
CCS Concepts • Software and its engineering ➝ Software architectures
Keywords Software architecture; DevOps; continuous deployment; empirical study.
1. INTRODUCTION Development and Operations (DevOps) is expected to help increase the speed of developing and delivering software features without compromising quality [1, 3]. Continuous Deployment (CD) and Continuous DElivery (CDE), treating operations teams as first class stakeholder, and infrastructure as code are some of DevOps practices for bridging the gap between development and operations teams for deploying software into production continuously [3]. CD practice is aimed at deploying frequently software or new changes into production environments by providing as much automation support as possible [3, 24]. CD practice extends and implies both Continuous Integration (CI) and Continuous DElivery (CDE) practices, while the reverse is not Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ESEM’16, September 8–9, 2016, Ciudad Real, Spain. Copyright 2016 ACM 1-58113-000-0/00/0010 …$15.00 DOI: http://dx.doi.org/10.1145/12345.67890
correct [15, 18]. That is why this paper mainly focuses on CD practice rather than CDE or CI. Given the increasing popularity of DevOps, several research efforts are being geared towards understanding how to efficiently move to CD practice, for example in terms of choosing appropriate tools and technologies [23], and promoting team mindset and expertise [21]. The other area of interest in CD research is to support (semi-) automate continuous testing [11] and increasing awareness and transparency in continuous deployment environments [5]. According to [3], among DevOps practices, continuous deployment and treating operations team as first class stakeholder may necessitate the architectural modifications. Recently, a couple of academic and white papers voiced this concern, stating that software architecture should take lead in moving to CD practice [1, 4, 6, 25]. It has been recognized that different domains and contexts bring new challenges for architects. A given context may impact considerably both on their (architectural) design decisions and their organizational responsibilities. It has been discussed that an architect should make a software system’s design as much simple and fine-grained as possible and try to remove those architectural elements that can be obstacle to automatically deploying software on the continuous basis [24]. However, there is a little empirical research on how CD practice actually changes architecting process. Architecting is a process of designing and evaluating software architecture of a system; essentially it’s a decision making for devising and assessing design solutions. Both conceptual solutions (e.g., architectural patterns and tactics) and technology solutions (e.g., tools and programming languages) are considered as architectural solutions and subject to architectural decisions [28]. This study considers both types of architectural solutions. We assert that a critical dimension of CD is to explore and understand the role of software architecture in transition to CD practice, i.e., how software should be (re-) architected to enable and support CD principles (e.g., frequent and reliable deployment). A sound architecture is expected to help ensure a desired level of quality attributes (e.g., deployability), and reduce the feedback cycle time including instant feedback from operation time [4]. This empirical study is aimed at filling this gap by systematically collecting, analyzing and classifying architectural challenges and principles for enabling and promoting CD practice. We carried out an exploratory case study involving interviews with 16 practitioners who had experience in software architecture or implementing CD practice on projects that had used continuous deployment and delivery, deployability, or microservices. This case study has investigated following three research questions: RQ1. What types of architectural challenges exist in moving to continuous deployment? RQ2. What are architectural principles to enable continuous deployment?
RQ3. How adoption of continuous deployment practice changes the importance of operational aspects? The main contributions of this study are the followings: (i) We identified three types of architectural challenges regarding continuously deploying software: highly coupled monolithic architecture, team dependencies, and ever-changing operational environments and tools. Challenges that pertain to the highly coupled monolithic architecture received more attention. (ii) According to the practitioners, architectural principles such as, small and independent deployment units, not too much focus on reusability, aggregating logs, isolating changes, and testability inside the architecture should be considered in architecting an application for adopting CD practice. (iii) Gathering and considering operational aspects of the software system make software architecture much more operationally friendly. The rest of the paper is organized as follows: In Section 2, we provide background on CD practice and related work. Section 3 details the research method. The results are described in Section 4. The Section 5 closes the paper with discussion and conclusion.
2. BACKGROUND AND RELATED WORK 2.1 Background Software architecture (SA) is important in understanding and managing large and complex software systems [2]. SA can be described and viewed from multiple perspectives. Structural and decisional viewpoints are two main viewpoints on SA [26]. The structural viewpoint focuses on components and connectors and considers SA as a high-level software structure of a system [2]. The decisional viewpoint defines the SA as a set of (architectural) design decisions including their rationale [19]. This viewpoint mainly focuses on decisions made during architecting process rather than on the end products (e.g., components and connectors) of architecting process. Continuous practices (e.g., continuous integration, continuous delivery and continuous deployment) are emerging practices in software engineering [12]. They aim at reducing the time between committing a change to a system and placing the changed code into production [3]. Continuous Integration (CI) encourages a software development team members to integrate and merge their code frequently, e.g., multiple times per day [12]. CI is expected to bridge discontinuities between development and deployment teams [12]. This practice includes automated software building and testing [21]. Continuous DElivery (CDE) refers to ensuring that software is always ready for release [15]. It has been argued that having CDE practice requires CI practice. Continuous Deployment (CD) practice goes a step further and automatically and steadily deploys software to production [18, 29]. Defining and differentiating between CDE and CD practices have been discussed by researchers and practitioners [12, 15, 18]. What distinguishes CD from CDE is production environment (i.e., actual customers). Whilst CDE practice aims at always keeping an application at deployable state, CD practice automatically deploys a working version of software to a production environment [12, 15, 18]. Whilst CD implies CDE practice, the converse is not true [18].
2.2 Related Work Whilst a few researchers have studied the adoption of CD in the software development industry [7, 21, 23], there has been little
work on empirically studying the role of SA in adopting CD practice. Claps and his colleagues studied the technical and social challenges of adopting CD in a single case software company and reported the mitigation strategies have been adopted by the case company [7]. Based on the analysis of the data gathered from 20 interviews with software practitioners in the case company, they identified 20 challenges such as team experience, team coordination, continuous integration, and infrastructure. The findings of this study reveal that adopting CD necessitates changing team responsibilities and involves several risks. Two other studies [21, 23] reported the benefits that CDE and CD practices provide for software development organizations such as: (i) getting more and quick feedback from the software development process and customers; (ii) improving customer satisfaction and product quality; (iii) automating and streamlining the release cycle. Furthermore, the studies reveal that there are a number of obstacles and challenges to adopt CDE and CD, which are in line with the reported challenges in [7]. To the best of our knowledge, there are only two studies, which have explored the role of SA as a contributing factor when adopting CD [4, 6]. Chen reported an experience of architecting 25 software applications for continuous delivery [6]. The study indicates that the continuous delivery creates new challenges for architecting software applications. It is revealed that CDE practice heavily influences a set of Architecturally Significant Requirements (ASRs), such as deployability, security, modifiability, and monitorability as well as they need to be met in order to achieve the maximum benefits of CDE. Another empirical study has been conducted on three projects adopted continuous integration and delivery to understand deployability goals of those projects [4]. The study has revealed that most of the decisions made to achieve the desirable state of deployment (i.e., deployability quality attribute) were architectural ones. Then a deployability tactics tree has been formed based on deployability goals and tactics collected from three projects. Our study has significant differences to [4, 6]: (1) our data comes from interviewing 16 practitioners from different and diverse organizations rather than one practitioner’s own observations from a single case company. (2) While the study [4] focuses on deployability as a new quality attribute at three projects, the study [6] only reveals what quality attributes are more important for CDE based on the experience of a single practitioner (i.e., author) without any investigation how these quality attributes contribute to CD practice. The 16 practitioners from 14 international organizations in our study shared the architectural challenges and principles with concrete example from 19 different projects to support CD practice.
3. RESEARCH METHOD One of the key decisions of empirical study is to select a suitable research method based on the nature of the studied problem and the research questions to be answered [10]. Since architecting for CD is in its nascent phase, we decided to conduct an exploratory case study to investigate how CD practice influences software architecting process. We developed a case study protocol and followed it strictly for conducting this study. We interviewed 16 practitioners in multiple small and large software development organizations. Our study mainly dealt with the “How” and “What” questions (see Appendix A). Table 1 summarizes the background and the affiliated organizations of the interviewees of our study.
3.1 Protocol We used interview as the main data collection method. All interviews except one (i.e., email-based interview) were conducted via Skype. Since exploring the impact of CD on SA was a new topic and we did not want to restrict the answers of the interviewees in advance, we did the semi-structured interviews, which involved 30 open-ended questions. In order to have more fruitful interviews, we sent the interview questions to the participants before conducting interviews. It is worth noting that the questions of interview evolved as the research progressed and feedback emerged from the participants [30]. All the interviews were conducted by the first author and had the character of a natural conversation. This enabled the participants to openly and freely share their experiences. The interviews lasted between 60 and 90 minutes per participant. The interviews started with a short explanation of the research’s goals, and demographic questions (e.g., role and experience years). Then, the participants were asked to share the challenges, successes, pitfalls, and the changes that CD may have brought to architecting process; the participants were also asked about their team’s structure and their daily routine works. All the interviews were audio-recorded and transcribed for an in-depth analysis. During the interviews, we asked participants to select at least one of their projects that had adopted or was adopting CD practice. For example, deployability, CD, CDE, or microservices had to be the major focus in the project to be used a reference point for the interviews. The interview guide is partially described in the appendix A.
3.2 Participants The participants for this study were recruited using purposive sampling in order to include those who either had valuable experiences in (re-) architecting for CD practice (e.g., architect) or were closely involved with or influenced by architecture (e.g., program manager or operations engineers). In general, the participants were gathered through industry contacts, our personal network and through direct contact with relevant potential interviewees. Particularly, we purposively selected the participants that had high competences of software architecture or CD practice (e.g., the (keynote) speakers in practitioner-based conferences on DevOps/CD/SA) and were quite interested in our study. As discussed in Section 3.1, we sent participants the interview questions in advance. This allowed participants to decide whether or not they would suitable for participation in our study. To provide a broad and rich understanding and characterization of the implications of CD practice on architecting, we selected participants from different organizations with divergent domains and of varying business sizes, e.g., from small organizations ( 1000 employees). We also used “snowballing technique” to ask participants to introduce suitable candidates for participation in our study [17].
3.3 Analysis The collected data (e.g., the interview transcripts) was analyzed using thematic analysis [9], a qualitative data analysis method for identifying, analyzing, and reporting patterns (themes) within collected data. We analyzed the data to identify what implications CD had on architecting, the challenges that the participants faced when designing the architecture succeeded with CD, architectural principles and solutions they employed to support and promote CD practice. We used the qualitative data analysis tool, NVivo,
for supporting the systematic analysis of the data (http://www.qsrinternational.com/). We followed the five steps of the conceptualized thematic analysis method for software engineering as detailed below: (1) Extracting data: all interviews were transcribed and we separated the parts that were relevant to each of the research questions for this study. (2) Coding data: we tried to read and examine the interview transcripts to form the initial ideas for analysis. Most of our questions were “How” questions, therefore, extracted data was explained in few sentences rather than being abstract (i.e., couple of words). This enabled us to extract the initial lists of architectural challenges, practices and contributing factors in moving to CD. (3) Translating codes into themes: for each interview transcript, we tried to combine different initial codes generated from the second step into potential themes. (4) Creating a model of higher-order themes: the challenges, practices, pitfalls and changes identified from third step were checked against each other to understand what themes had to be merged with others or dropped (e.g., lack of enough evidence). (5) Assessing the trustworthiness of the synthesis: through this step, we defined clear and precise names for each of the challenges, practices, pitfalls and changes in order to answer the research questions.
3.4 Limitation Whilst we strictly followed the guidelines provided by [20] to conduct this exploratory study, the findings of this study may have been affected by the following threats: (1) the study only collected data using one data collection method (i.e., interview). That means we did not use method triangulation, however, we believe this is not a significant limitation because the participants were from different organizations from divergent domains and of varying sizes. (2) The participants may have the tendency to overstate the important role of SA as a contributing factor to successfully introduce CD. In order to minimize this threat, we also recruited non-software architecture practitioners, such as developer and DevOps engineer. We also included the participants from different roles in the same organization to help reduce any personal bias. (3) The obtained findings in this study heavily relied on the participants’ statements, which might be subjective, and it can negatively affect the results of this study. To mitigate this threat, we only reported findings that have been confirmed by data of multiple participants (e.g., at least two participants). Furthermore, we clearly defined the concepts of deployability and continuous deployment practice for each of the interviewees before the start of each of the interviews. We are confident that this practice helped us and the participants to gain a common understanding of the concepts deployability and continuous deployment for the interviews. (4) A large part of the data analysis step was conducted by the first author; in the case of any doubt, continuous discussions were organized with the second author for carefully checking the findings. Furthermore, we strictly followed a pre-defined case study protocol described in Sections 3.1 to 3.3 during conducting this exploratory case study.
Table 1. Summary of participants’ details Participant Background ID
P1
Current Role
Architect
Organizational Attributes Experience in current Role
11
Experience as Architect
11
Country
Australia
Research Institute
Project Domain
Type
Team Size
Project 1: Cloud-based system
Greenfield
10
Project 2: Data integration system
Maintenance
7-8
Internet Technologies
Network monitoring system
Greenfield
7
US
Large
Services and Consulting
Financial system
Greenfield
20
0
Australia
Large
Telecommunication
N/A
Greenfield
25
8
12
India
Small
RFID
Supply chain system
Maintenance
40
5
20
US
Large
Financial Services
Commercial credit software
N/A
15
Small
Network Security
Network access management system
Maintenance
14
Large
Telecommunication
Telecommunication system
Greenfield
10-15
Project 1: Transport system
Maintenance
15 -20
Project 2: Banking system
Greenfield
5
Project 3: Financial system
Maintenance
6 teams with about 7 people
Developer
P3
Senior Consultant for DevOps
3
0
P4
Program Manager
2
P5
Director of Engineering
P6
Vice President of Development
P7
Manager/Chief Architect
2
10
US
P8
DevOps Engineer
1.5
0
Australia
IT Consultant/ IT Architect
Small
Domain
Large
P2
P9
Size
Project(s) Attributes
7
1.5
0
10
China
Netherlands
Medium
Services and Consulting
P10
Continuous Delivery Consultant
3
12
Netherlands
Medium
Services and Consulting
Insurance website
Maintenance
26 people for software development; 4 people for continuous delivery
P11
Architect
10
10
Germany
Medium
Services and Consulting
N/A
Maintenance
10
P12
Technical Lead/Architect
7
7
England
Large
Services and Consulting
Content management system
Greenfield
50 people in 4-5 different teams
P13
Architect
12
12
US
Large
Services and Consulting
Financial system
Maintenance
25
P14
Architect/ Independent Consultant
3
8
Latvia
Medium
Financial Service
Financial system
Greenfield
3 distributed teams each 50 people.
P15
Architect/ Consultant
7
8
Latvia
Small
Services and Consulting
Financial system
Greenfield
15
P16
Operations Engineer
1
3
Finland
Small
Startup (Game)
Website application for selling game
Greenfield
7
4. FINDINGS In the following subsections, we report the findings obtained from the synthesis and analysis of the interviewees’ data.
4.1 Overview We interviewed 16 practitioners (P1 to P16), all men with 0 to 20 years of professional experience as software architect (average of 7.7 years), and from 14 different organizations (7 participants worked in large organization, 4 in medium-sized organization and 5 in small one). Out of the 16 participants, 8 were currently in the role of architect (e.g., IT architect or solution architect) with 7 to 12 years of experience (average of 9.5 years). The participants came from 9 different countries. The types of projects varied from green-field to maintenance projects. Table 1 presents a summary of the demographic details of the participants, their projects and their affiliated organizations. Table 2 shows the frequencies of releasing software internally (i.e., not deployed to production) and externally (i.e., deployed to production), along with the level of continuous deployment pipeline automation in the interviewees’ organizations. These criteria can be used to determine the level of CD practice adoption by software development organization [21]. Furthermore, Table 2 indicates that the degree of software readiness (e.g., being in deployable state) to be delivered internally [21]. We found that the interviewees’ organizations had better capabilities in terms of the frequency of deploying software to production and developing and using fully automated deployment pipeline compared to the organizations studied in [21]. While 7 out of 14 our interviewees’
organizations used fully automated deployment pipeline, the results in [21] revealed that none of the studied organizations used automatic chain to deployment. We also observed that in spite of having (semi-) automated CD pipeline, there are still confounding factors (e.g., non-technical factors) that may limit the organizations to continuously and automatically deploy the software to production and delay the software release process. Table 2. The speed of release and CD level automation ID
Frequency of release internally
Frequency of release externally
Automated CD pipeline?
P1
Project1: Weekly Project2: Monthly
Project1: Weekly Project2: N/A
No
P2
Weekly
Monthly
No
P3
Multiple times a day
6 months
No
P4
Fortnightly
Based on customer’s request
Yes
P5
Twice a week
Monthly
No
P6
Daily
Every three weeks
No
P7
Weekly
Monthly
No
P8
Multiple times a day
Fortnightly
No
P9
Fortnightly
Project1: Monthly Project2: Fortnightly Project3: Monthly
No
P10
Multiple times a day
Daily
Yes
P11
N/A
Monthly
No
P12
N/A
Fortnightly
Yes
P13
Multiple times a day
Multiple times a day
Yes
P14
Multiple times a day
Daily
Yes
P15
Multiple times a day
Every three weeks
Yes
P16
Multiple times a day
Two times a day
Yes
4.2 Architectural Challenges (RQ1) This section reports the architectural challenges that the participants experienced during the design of their systems to enable continuous deployment. It should be noted that based on the data collected from the interviews, we found that the participants also faced other challenges for adopting CD practice, such as lack of team skills and expertise, and lack of leadership’s support, which were out of the scope of this paper. Following are the main categories of the architectural challenges that were reported as bottlenecks to continuously deploying software.
4.2.1 Highly coupled monolithic architecture Our analysis has revealed that highly coupled architecture presents the biggest challenge when organizations are going to move to continuous deployment practice. As a continuous delivery consultant stated that “at the same time I do not know they (architects) are aware that; they create kind of monolithic applications and monolithic application contains a large functional of domains would be hard to use in the large-scale organization if you want to have continuous delivery” P10. Such kind of systems have a lot of cross-functional dependencies, in which every dependency needs to be managed in the CD pipeline for deploying software [16]. As the complexity and size of these applications grow continuously, it becomes unmaintainable as well as changing the components or services independently would be problematic. If a component needs to be deployed into production environment, all dependencies are expected to deploy as well. “And when you say, we have software component X and I would like to deploy that to the customer; do I need to deploy other software components as well; do I really need it or not? And these are design issues that we really face” P9. Our study reveals that not only applications can be monolithic from the code perspective but can also be monolith from database stand point. One core database in large-scale application can create operational bottleneck and it gradually becomes an undeployable unit. It is mainly because changing any functionality in the application necessitates touching and incorporating database scheme as well. “And one of traditional is that front end and database for big application they used one database, then you know for every piece of functionality that you change, you also need to incorporate database, you need to test database so these are sort of single point of these big organizations financially they cannot continuously deploy database. So they need to refactor and do changes the software architecture” P9. Furthermore, we found that dependency at application level is also an inhibitor to introducing CD in the studied organizations. For deploying a piece of software on the continuous basis, it is needed to continuously deploy all other applications which are dependent on. This brings the challenges of coupling and refactoring for big legacy system in software-development organizations.
challenges reported by the participants was that the dependent teams working on monolithic large-codebase were not successful in adopting and implement CD practice. While many teams are trying to push their changes at the same time into production, this would lead a lot of friction in deployment pipeline. Albeit one team may have perfect control on its deployment process, have quality-driven build and be able to release to the production constantly, they still depend on the release of other teams. Therefore, they can easily break the changes that other teams are working on. This situation results in significant challenges for team members to make appropriate decision for deploying software continuously. One of the participants reported this issue in these words: “we had multiple teams working on the same codebase. They were working on different functionalities and they would be somebody working on search team and somebody working on other features as part of content management system and things like that. We started noticing that it was really affecting our ability to deploy the software” P12.
4.2.3 Ever-changing environments and tools Another challenge reported at architectural level was the influence of ever-changing environments and tools on architecture design to enable CD practice. Our analysis of the data revealed that it was challenging for a couple of practitioners to design applications for different operations environments, in which they may have had difficulty to make consistency in a set of heterogeneous operations environments in order to seamlessly transfer and deploy the application in all production environments. Furthermore, architecting process in those heterogeneous environments is not only architecture in terms of software application but it should be more about architecting the whole environments and artifacts (e.g., database and test data). Architect is expected to see architecting process as a larger entity to find solutions to drive the consistency after the application is deployed and upgraded (e.g., versioning the whole environment rather than only versioning the source code). For example, a manager and chief architect commented that “it is not architecture things in terms of source but it was more about architecting the deployment as a larger entity …. So if we require new customer we deploy, we might upgrade the existing customer, how to maintain consistency across those environments because they are on different servers. It is not cloud system, in one multi-tenant environment” P7. Another issue regarding operations environment is that the participants did not have any idea and solution to determine to what extent operations requirements should be collected and considered. For example, taking knowledge of operations environment too much makes the architects probably design the system too specific for that environment. Through this, they may make some specific design decisions, and if operations environment slightly changes, those specific decisions will no longer be valid and relevant. However, if they do not consider most of the characteristics of the operations environment, the application may not be really operation friendly. “They [operations team] had purchased some infrastructure and some hardware and what they wanted us that solution should be in compliance with their already purchased infrastructures… that all added the challenges” P5.
4.2.2 Team dependencies
4.3 Architectural Principles (RQ2)
Conway’s Law indicates that organizations produce designs which usually represent their internal structures [8]. The architecture of an application largely determines the structure of the software development team that works on that architecture. One of the
CD is all about deploying high quality value frequently and reliably with as much automation as possible. It has been reported by both architecture practitioners and non-architecture practitioners that software architecture should be taken into
account, and should be changed and optimized during the journey to adopting CD practice. In this section, we describe a set of high level architectural principles that should be applied to increase actionability of the software architecture to be compatible with the sprint of CD practice (e.g., frequent and reliable release). An architectural principle can be further described into several architectural tactics, patterns, and styles.
4.3.1 Small and independent deployment units Architecting for deployability requirements in mind from day one has been reported by many interviewees as first and simple practice for better adopting CD. Similar to practitioners’ blog [22], we found that an important principle for successfully adopting and implementing CD practice in a given organization is that small and independently deployable units (e.g., service, component, and database) should be considered as a foundation for architecture design. This is expected to minimize both the semantic and operational couplings1 of software units. However, a few interviewees reported that it was much easier to manage operational coupling than other types of couplings. Through this, the units can be brought to production independently. Although many participants emphasized that the need of simple, modular and lego-based architecture with a set of independently deployable units is necessary for moving to CD, but they did not apply any concrete framework or criteria to break down (monolithic) applications into the small and independent units. We could not find how much a service or a component should be small to succeed with it in CD practice. For example, for one of interviewees small meant the maximum amount of work for coding and testing a single feature lasted three-day effort for one person. As another example, the participant P13, an architect, shared that in order to not encounter with the level of service granularity, it is needed to start with small number of large services first, rather than having too many fine-grain services first and then decompose those services further. “If you are trying to go from monolithic to something like servicebased or microservices, don’t try to break it down into little tiny pieces first of all right away. You’ve got struggles of service level, structurally you have to look at how coupled your components are” P13. Microservices: In the last few years microservices architecture style has emerged as a new architectural style for designing finegrained applications as a set of small services [22; 24]. It has been said that microservices architectural style is the first architectural style for CD practice [13]. This architectural style tries to break complex systems into small, autonomous, independently deployable services. We found that some interviewees’ organizations have successfully implemented this architectural style for enabling CD practice. However, there is evidence that indicates that adopting this architectural style involves difficulties as it may bring negative results for software development organizations. That is why some studied organizations ignored to adopt this style or failed to achieve the real anticipated benefits of microservices style. For example, since microservices style puts a lot of operations overhead in terms of like monitoring these services, and administrating them, it is needed to have mature operations team to adopt it. This issue forced the participant P12, a technical lead and architect, to split a monolithic code and make them as binary dependencies, instead of implementing microservices for this purpose. We also found that adopting microservices style necessitates changing the organization style 1
http://www.michaelnygard.com/
and also team structure. Moreover, team maturity level should be ready to successfully implement this architectural style. Interviewees’ organizations adopting microservices realized that it was easier to structure their team look like the microservice architecture style and have one communication structure (e.g., through well-defined interfaces). “… company struggles with microservices architecture and … they implemented microservices and they slow down because they are not ready to change the organization to support this microservices architecture, because microservices conditionally interaction don’t reflect this structure that they have and they slow down …” P14 Vertical layering: Another noticeable practice implemented by a couple of interviewees to improve the level of independency among team members was splitting out modules or components of the applications into vertical layers rather than employing horizontal layered architecture and then letting each team to be responsible for their own module or component. Through this, each team is completely responsible for the deployment of that component, without being bothered about what other teams are working on and responsible for that. Furthermore, it hampers the ripple effects resulting from any changes. However, it has been argued that it was hard to achieve this goal because there is a tendency to group people with similar skills together. For example if you have specific developers who are doing frontend development task and other developers who do business logic layer development, it would not easy to form end-to-end development team by them. “We had separation in layer in architecture, so you have presentation layer to talk to business service layer … if you have three teams responsible for one of this layer, all three teams need to work together to bring a new piece of functional live, … to make better, basically we swap the organization and now we do not have team who only does frontend and another team does business logic layer. Just makes the team and want to end-to-end development and another team doing the same” P10. Database as continuously deployable unit: In Section 4.2.1, we discussed that monolithic (i.e., core) database brings challenges for continuously deploying an application. A few participants emphasized that not only the deployability of database should be improved for CD practice by splitting core database into smaller and less complex individual databases; they also recommended that it is vital to consider database as easily deployable unit like other software components. It is mainly because they are part of deployment process of software components. By incorporating the database into deployment process as the application, it is expected to potentially prevent unexpected issues that database updates may introduce at the point of deployment. One of the architects described the role of the database in the deployment processes in these words: “… we are talking about any sort of database that your system depends on as part of deployment. … so you expect when you test the deployment of that [software component] and it also spins of what other dependencies are needed to incorporate in that ecosystem” P13.
4.3.2 Not too much focus on reusability Another strategy that was mentioned in several interviews was that focusing too much on reusability can be a huge bottleneck to continuously deploying software. Reusing at the component level as well as reusing packaged software within application may involve two side effects for CD: (i) it increases dependencies between teams, at which all teams depend on single and shared
unit. In addition, there is a need to spend huge amount of time to maintain and obtain agreement from all teams before evolving a unit. This hinders teams to autonomously deploy components and applications. (ii) Another side effect of reusability is on testability. For example, by reusing the well-known packaged software, we start losing the testability of their configurations over time. From the CD stand point, it is vital for application developer to be able to write tests that validate the configurations of packaged software used within application. So (re)-using packaged software may reduce ability to write the unit test and ability to put them on continuous deployment pipeline. Therefore, it is recommended that in the CD process everything should be decentralized. To give an example, the participant P14, who was in the role of independent consultant and architect and helped organizations adopt CD practices, commented that “there are a lot of things between them we reused like utilities and even some part of domain … and we spent amount of time to maintain agreement between teams how we went to evolve them … and actually it went down significantly and we decided to fork them just go your own way whatever you want and just leave us alone. …. we have to understand duplication is not evil” P14.
4.3.3 Aggregating logs Our analysis of data revealed that an important principle for the success of CD implementation is proper logging. As the speed of deployment increases, it potentially may increase the number of errors, and the chance of unavailability of the system. Therefore, it is absolutely critical to collect the operational data in order to both monitor the current state of application in the production and to fix issues and problems [3]. Therefore, log aggregation, analysis and monitoring are becoming increasingly important for deploying software on the continuous basis. A number of participants emphasized that log aggregation and monitoring should be an integral part of continuous deployment environment. For example, the participant P5, who worked as director of engineering at small company, noted that “we have everything based on log…; I think that aspect of logging which we can probably see we already have as part of our framework” P5. According to practitioners, the continuous deployment tools do not provide sufficient monitoring and logging capabilities. Therefore it is needed to provide capabilities to produce sufficient amount of logs and also aggregate and abstract the log data overtime. In the CD context, such capabilities can be provided by two ways: (i) external tools and (ii) software application itself (e.g., as P16 stated that loggability should be embedded into the code). Once the application has been deployed into production system, operations team need to monitor application to be able to hypothesize and run experiments to see how to make something better, and also to assess and ensure the functionality and performance of application in the production environment. This necessitates the use of tools that can facilitate application’s performance and address operational issues. Participant P6 who was in the role of vice president of software development stated that they were able to address performance problem using the log data: “You build log capability such they can be turned on and off in order to eliminate those performance problems. You know always logging you get switched in the code in a way to determine logging on and off” P6. Another participant emphasized that having sufficient and good logging mechanism is expected to eliminate the need of rollback. “We needed much action on information as possible for logs to be able to quickly figure out, because if you get solved quickly right
then you do not need to get asked to rollback. In order to not rollback, you really need good logging” P7. We found that there is an increasing trend in the interviewees’ organizations to have a log-driven, log-specific architecture in the middle of the applications. Software architectures of those applications continuously collected operational data at the appropriate levels and more importantly they made it easy to aggregate logs, convert them into appropriate format and make them searchable. It should be noted that such architectures provide continuous monitoring as well. There are two issues that still needed to be addressed according to the interviewees. First it needs to determine to what extent log data can be produced by an application. As P7 noted “… internal meeting discussions about how much we can effort to capture [logs] per system; you know how much with level of details appropriate”. Secondly, the logs should be readable by all stakeholders. Since most of the logs written by developers for developers, they may not be employed by operators to perform diagnosis and operations tasks.
4.3.4 Isolating changes Another category of architectural principle is related to how applications should be architected to isolate changes and minimize the impact of changes. In our interviews, we asked the participants to explain what other criteria besides deployability they had in mind when designing the applications or splitting them into smaller components. Some interviewees noted that they always considered that software units (e.g., component or service) had to be small enough to be modified, replaced and run independently in productions environments. As an IT architect with a consulting company commented: “If you don’t have dependency, you can isolate your changes and as a team you can independently do your lifecycle and of course your lifecycle also means bringing it to production” P8. We found that a couple of practitioners applied domain driven design2 and bounded context3 concepts to separate the architecture of their system based on business capabilities rather than functionality. Those concepts were primarily adopted to actually identify autonomous business capabilities of software architecture, which were needed to be developed, evolved and deployed into production independently of the others [27]. Furthermore, it has been discussed that these techniques reduced the dependencies among team and provided less coupling. As a participant pointed out: “So you had to reorganize business structure to see sort of separate value stream because in the system I am talking about there are different money sources, … based on that you had to reorganize the business like based on business capability and then forms teams around them” P14. A few of interviewees referred the feature toggle4 and branch by abstraction5 patterns to make incrementally changes to an application. While others often referenced similar practices for implementing incrementally changes with different names (e.g., P7 noted this as side by side execution of components). It seems they are becoming as best practices to deploy the applications on the continuous basis. “And to kind of facility to do that, there are some design patterns, like branch by abstraction; we need to know what feature toggles 2
http://domainlanguage.com/ddd/ http://martinfowler.com/bliki/BoundedContext.html 4 http://martinfowler.com/bliki/FeatureToggle.html 5 http://continuousdelivery.com/2011/05/make-large-scalechanges-incrementally-with-branch-by-abstraction/ 3
like… We can also make that much more elegant way by using branch by abstraction when you have implementation details separated out by using interfaces and the child of implementation has based on toggles” P12. Another practice for making changes relatively easier at operational level was commented by the participant P10 as “if your WAR file is completely stateless and if I want to deploy new version and I can simply deploy new version on the top of it and there is no state to keep migrate to new version” P10.
4.3.5 Testability inside the architecture One of the most prominent obstacles reported by several interviewees in moving to continuously deploying the software, were the challenges associated with testing phase. To end this, to some of interviewees, testability in mind influenced their architectural and design decisions directly to explicitly build testability into the system. Furthermore, participants shared that building testability into the system facilitated deployment process of that architecture. However, the practitioners also made lots of technological decisions to improve testability by choosing the right tools and technologies. According to practitioners, the tools cannot solve all problems associated with testing. In fact, most of practitioners chose moderate perspective on testing (i.e., choosing right tools and testability inside the architecture). Apart from choosing appropriate testing tools, the interviewees adopted following strategies in order to improve the testability of their system: Improving test quality: Instead of testing everything which may bring huge overhead on continuous deployment pipeline, some practitioners tried to improve the quality of test (data). This includes having right number of test cases, improved quality of test data, and right time for testing. Long running tests can increase the cycle time (i.e., the time needed to transfer code from code repository to production) of continuous deployment pipeline as well as slow down the feedback cycle in deployment pipeline. That is mainly because tests are poorly designed and written. For example, a participant reported that they were able to address the slow cycle of feedback from continuous deployment pipeline by applying two approaches: (i) creating much smaller unit tests, in which gave them immediately feedback; (2) reducing the number of functional tests. That enabled them to get the same level of feedback about whether the applications are working for the users but it did not consume much time. So it is necessary to make informed (architectural) design decisions to improve the testability of a system by adopting more suitable, simpler test methods and decreasing test cycle times. Making code more testable: Our analysis of the interviewees’ data also uncovered that a couple of practitioners emphasized that the code should be written to be much more testable. For example, services and components need to be tested as stand-alone thing; in the meanwhile, it is expected to be tested within a larger integration and deployment environment. Testing those services and components within large integration and deployment environments can increase dependency. A typical solution for this problem was dependency injection which facilitated more flexibility in testing. As a participant commented that “from architectural stand point, we very often build projects that use dependency injection mechanism like for example Spring or Angular so that we can inject one set of dependency is a test time and another service dependency is runtime, which makes the code much more easily testable” P13.
Test automation: Interviewees argued that the architecture should support automating testing. All the participants believed that they were able to automate the units and integration tests to a large extent, but automating user acceptance test remains a big challenge and it takes a lot of workload in the company “… they often have a really challenges to get [automated test] done, for acceptance tests most of time it is not easy to fully automate” P9.
4.4 Operational Requirements (RQ3) For many years, operations stakeholders were treated as separate entities in software development process. Operations teams and their requirements have been regarded as something that they cannot be easily controlled and, at which software development organizations may not be able to do anything about them. DevOps advocates higher collaboration between development and software operations teams. In the DevOps culture, operations team should be regarded as first class stakeholder from requirement viewpoint [3]. We found that the organizations adopted different strategies to consider the operational requirements:
4.4.1 More responsibilities for developers Our analysis has discovered that the first and foremost change brought by DevOps and CD practice into the interviewees’ organizations is that developers have more responsibilities about their code in production environments. In fact the principle of “You build it, You run it” is getting more popularity in the studied organizations. Some activities (e.g., troubleshooting) that previously done by operations teams after the application has been deployed into production environment, now they are handled by developers: “…getting access to monitoring data and getting access to logging information, being able to start and stop the services themselves, all these capabilities moved and responsibilities moved to application team” P10. Our analysis of the data also revealed that while in some organizations listed in Table 1, development and operations tasks were performed by exactly the same staff; however, there are organizations that had still separated development and operations teams. For those organizations, we found that they placed operations team next to developers (e.g., at same office/location) and encouraged them to have more collaboration with other team members. One participant shared that in spite of having separate teams for development and operations, they were successful in adopting and implementing CD practices in many complex projects: “I think you do need to have close collaboration to be successful but you do not need to have teams to do both operation and development” P11. Additionally, a few participants partially referred to on-call developers to support their applications 24x7. In both cases, although most of responsibilities of operations have been transferred to developers, but operations and infrastructure staffs are needed to support and mentor the developers to write the operations side of the code. The interviewees asserted that deploying software on the continuous basis requires shared responsibilities and close collaboration among developers, testers, quality assurance team, and operations stakeholders.
4.4.2 Considering operational aspects According to [3], operations team has a set of concerns (e.g., logging and monitoring). Operations stakeholders care about fast failure detection, prediction, diagnosis, and recovery. Since, most of logs written by developers for developers, they may not be meant by operators. We found that one of emerging trends in the studied originations was that requirements of operations team
have been considered as part of requirements gathering and architecture design processes. A software architect in large consulting company reported that “only recently we have realized that operations is really fundamental part of architecture but if you take that as part of scope of architecture and treat them as first class citizen and then allows you to build architecture that is much more flexible” P13. Engaging operational staffs in decision-making process at the early stage of software development process (i.e., designing process) was also another strategy adopted by interviewees’ organizations in order to embrace CD practice. This gives more flexibility to organizations and enables them to deploy faster. Involving operations team in the development of requirements ensures that they have equal voice in the software development process. One participant featured this trend as “…when people do the design, they normally do not engage with operations teams or for the operation support team; so, this is an area we are trying to improve when we finalize the design, we are getting the operation teams and operation support to sign-off of design” P4. During the interviews, participants commented that thinking about the operational aspects of the architecture from day one was a really useful and growing exercise. This is expected to force the architects think much more upfront about the problems that may happen at the deployment of application.
5. DISCUSSION AND CONCLUSIONS Our study has been motivated by the growing importance of software architecture to successfully enable and support Continuous Deployment (CD) practice. We discuss some of the main findings from our study and draw some conclusions. The findings from this study have provided initial evidence that software architecture plays a significant role in successfully and efficiently adopting continuous software deployment. We have identified a range of recurring architectural challenges (i.e., highly coupled monolithic architecture, team dependencies, and everchanging operational environments and tools.) and five main architectural principles (i.e., small and independent deployment units, not too much focus on reusability, aggregating logs, isolating changes, and testability inside the architecture). These principles may not be bounded only to CD practice, but highly complex and challenging nature of CD practice makes these principles vital for successfully adopting and implementing CD practice. This study has provided empirical evidence to support the recommendations of involving operations’ staff at an early stage of design process and considering operational aspects of software system for making SA readily supportive of CD. Some of the participants shared that initially leveraging architectural dimension for CD practice was not directly targeted in their respective organizations; but once they realized that the architecture had an important role for adopting CD practice, they tried to redesign and optimize software architecture in order to isolate business capabilities, functionalities, and to have individual deliver teams to get more benefits of CD practice. Our research has explained the concept of “Continuous Deployment Units”, which it does not only refer to software components or services to being as independently deployable units, but also refers to every entity (e.g., database and operating system) that application depends on, should be a unit of deployment in the deployment process. Our results have shown that ignoring this principle can usually lead to severe bottleneck to continuously deploying software. Every dependency is to be incorporated into deployment process.
We observed that due to the ever-changing and unreliable production environments and tools coupled with the increasing rate of changes in the projects that had adopted CD practice, the participants often were unable to make so many upfront (architectural) design decisions. Instead they only focused on an initial set of core decisions. Since there was a little up font architectural work, a lot of architectural decisions were made to simplify the changing and evolving the application on daily basis. It has also been discussed that sometime the decisions were made unconsciously. So software architecture in CD practice should be extremely adaptable to unpredictable changes. This is challenging because it means an architect has to make it lightweight, legobased and high quality. Additionally, an architect should invest not only in architectural simplicity but also in design simplicity like improving code readability. However, a few participants explicitly emphasized that they were using or have planned to apply evolutionary architecture [14, 24] rather than big upfront architecture for CD practice, our analysis of the interviewees’ data revealed that the evolutionary architecture concept is becoming more popular in CD practice. By evolutionary architecture we mean decisions should be made as late as possible. It is expected that adopting evolutionary architecture instead of up-front, heavyweight architectural designs would be more suitable to implement CD practice [14, 24]. Our findings have also made it clear that CD practice introduces some overhead cost for operations processes and operating staff. It is mainly because with the increased speed of deployment, there are more environments, more variables, and more things to manage. Therefore, it increases the complexity of deployment process, which necessitates designing and implementing sophisticated logging and monitoring mechanisms. For example, CD practice requires every team to build a deployment pipeline for each component or service rather than having one deployment pipeline for the whole application. Although having simpler deployment pipeline for each component or service can bring a lot of benefits but the requirement of a dedicated pipeline needs extra effort to set up the dedicated pipelines for the first time. Some of the participants reported that they were employing automation technologies such as Docker6 to simplify the deployment process. By employing Docker, they could isolate deployment processes, which reduced the complexity of deployment. For example, using virtual machine for Docker container for each component makes components pretty well isolated. However, at the same time, it means if each component runs as a separate virtual machine, and so obviously it is needed to have memory, and CPU. We observed that there has been always a trade-off between the ease of deployment and the cost of deployment. According to some of the participants, this trade-off impacted on their architectural and technology decisions directly. According to our results, the interviewees’ organizations struggled with bringing CD practice into monolithic application architectures. With the increasing number of organizations that view software architecture as an important factor to adopt CD practice, in future we plan to develop architecture-driven framework to help practitioners (re-) design CD-driven software architectures. The framework will leverage the architectural principles, patterns, styles and guidelines needed for successfully adopting and implementing CD practice. Furthermore, we plan to conduct a survey to quantify the architectural challenges and principles of CD practice identified in this study.
6
https://www.docker.com/
6. ACKNOWLEDGMENTS The authors would like to thank all participants of interviews for their time and efforts. This work is partially supported by NICTA. NICTA is funded by the Australian Government through the Department of Communications and the Australian Research Council through the ICT Centre of Excellence Program.
7. APPENDIX A. INTERVIEW GUIDE Participant background What is your role and responsibilities in the project team? How long have you performed that role? As architect? Please talk about size and domain of your organization. Project’s Description What was domain, type, team size of project? In this project, are/were the development and operations tasks performed by the same team? Could you describe your team structure before and after adopting CD practice? How adopting CD could change your daily work activities? How often did you deploy software internally and into production? Was the deployment pipeline fully automated or not? Operations team and their requirements How did you consider operations requirements in design process? Architecture and Continuous Deployment Practice How would you describe the relationship between architecture and CD practice? How the adoption of CD practice changes architecting? How did you design the architecture of your system to enable and better support CD practice e.g., improving deployability? How did you predict and evaluate the deployability of your software system during the design process? Can you provide one example of architectural decisions you made for improving deployability? What architecture practices (e.g., patterns, styles, and tactics) you employed in order to promote/support CD practice? What challenges did you experience during designing application’s architecture for CD?
8. REFERENCES [1] Puppet Labs. 2015. 2015 State of DevOps Report. Available at: https://puppetlabs.com/2015-devops-report [Last accessed: 5 October 2015]. [2] BASS, L., CLEMENTS, P., and KAZMAN, R. 2012. Software architecture in practice. Addison Wesley. [3] BASS, L., WEBER, I., and ZHU, L., 2015. DevOps: A Software Architect's Perspective. Addison-Wesley Professional. [4] BELLOMO, S., ERNST, N., NORD, R., and KAZMAN, R. 2014. Toward Design Decisions to Enable Deployability: Empirical Study of Three Projects Reaching for the Continuous Delivery Holy Grail. In Dependable Systems and Networks (DSN), 2014 44th Annual IEEE/IFIP International Conference on, 702-707. [5] BRANDTNER, M., GIGER, E., and GALL, H. 2015. SQA-Mashup: A mashup framework for continuous integration. Information and Software Technology 65(9//), 97-113. DOI= http://dx.doi.org/http://dx.doi.org/10.1016/j.infsof.2014.10.004. [6] CHEN, L. 2015. Towards Architecting for Continuous Delivery. In Software Architecture (WICSA), 2015 12th Working IEEE/IFIP Conference on, 131-134. [7] CLAPS, G.G., BERNTSSON SVENSSON, R., and AURUM, A. 2015. On the journey to continuous deployment: Technical and social challenges along the way. Information and Software Technology 57, 0 (1//), 21-31. DOI= http://dx.doi.org/http://dx.doi.org/10.1016/j.infsof.2014.07.009. [8] CONWAY, M.E. 1968. How do committees invent? Datamation 14, 5. [9] CRUZES, D.S. and DYBA, T. 2011. Recommended Steps for Thematic Synthesis in Software Engineering. In Empirical Software Engineering and Measurement (ESEM), 2011 International Symposium on, 275-284. [10] EASTERBROOK, S., SINGER, J., STOREY, M.-A., and DAMIAN, D. 2008. Selecting Empirical Methods for Software Engineering Research.
In Guide to Advanced Empirical Software Engineering, F. SHULL, J. SINGER and D.I.K. SJØBERG Eds. Springer London, London, 285-311. [11] ELBAUM, S., ROTHERMEL, G., and PENIX, J. 2014. Techniques for improving regression testing in continuous integration development environments. In Proceedings of the Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (Hong Kong, China2014), ACM, 2635910, 235-245. [12] FITZGERALD, B. and STOL, K.-J. 2015. Continuous Software Engineering: A Roadmap and Agenda. Journal of Systems and Software. DOI= http://dx.doi.org/http://dx.doi.org/10.1016/j.jss.2015.06.063. [13] FORD, N. 2015. Architecture is abstract until operationalized, http://nealford.com/memeagora/2015/03/30/architecture_is_abstract_until_ operationalized.html. [14] FORD, N. and PARSONS, R. 2016. Microservices as an Evolutionary Architecture, Available at: https://www.thoughtworks.com/insights/blog/microservices-evolutionaryarchitecture [Last accessed: 20 March 2016]. [15] FOWLER, M. 2013. Continuous Delivery. Available at: http://martinfowler.com/bliki/ContinuousDelivery.html [Last accessed: 21 October 2015]. [16] GIBSON, S. 2015. Monoliths are Bad Design... and You Know It, Available at: https://www.thoughtworks.com/insights/blog/monoliths-arebad-design-and-you-know-it [Last accessed: 4 March 2016]. [17] GOODMAN, L.A. 1961. Snowball Sampling. Annals of Mathematical Statistics 32, 1, 148-170. [18] HUMBLE, J. 2010. Continuous Delivery vs Continuous Deployment, Available at: https://continuousdelivery.com/2010/08/continuous-deliveryvs-continuous-deployment/ [Last accessed: 1 March 2016]. [19] JANSEN, A. and BOSCH, J. 2005. Software Architecture as a Set of Architectural Design Decisions. In Software Architecture, 2005. WICSA 2005. 5th Working IEEE/IFIP Conference on, 109-120. [20] KITCHENHAM, B., PICKARD, L., and PFLEEGER, S.L. 1995. Case studies for method and tool evaluation. IEEE Software 12, 4, 52-62. [21] LEPPANEN, M., MAKINEN, S., PAGELS, M., ELORANTA, V.-P., ITKONEN, J., MANTYLA, M.V., and MANNISTO, T. 2015. The Highways and Country Roads to Continuous Deployment. IEEE Software 32, 2, 64-72. DOI= http://dx.doi.org/10.1109/MS.2015.50. [22] LEWIS, J. and FOWLER, M., 2014. Microservices: a definition of this new architectural term. Available at: http://martinfowler.com/articles/microservices.html [Last accessed: 05 January 2016]. [23] LIANPING, C. 2015. Continuous Delivery: Huge Benefits, but Challenges Too. IEEE Software 32, 2, 50-54. [24] NEWMAN, S. 2015. Building Microservices. O'Reilly Media, Inc. [25] NORTHROP, L. 2015. Trends and New Directions in Software Architecture. Available at: http://resources.sei.cmu.edu/asset_files/webinar/2015_018_100_438676.p df. Software Engineering Institute, Carnegie Mellon University Pittsburgh. [26] POORT, E.R. and VAN VLIET, H. 2012. RCDA: Architecting as a risk- and cost management discipline. Journal of Systems and Software 85, 9 (9//), 1995-2013. DOI= http://dx.doi.org/http://dx.doi.org/10.1016/j.jss.2012.03.071. [27] SOKHAN, B. 2015. Domain Driven Design for Services Architecture, Available at https://www.thoughtworks.com/insights/blog/domain-driven-designservices-architecture [Last accessed: 10 January 2016]. [28] SOLIMAN, M., RIEBISCH, M., and ZDUN, U. 2015. Enriching Architecture Knowledge with Technology Design Decisions. In Software Architecture (WICSA), 2015 12th Working IEEE/IFIP Conference on, 135-144. [29] WEBER, I., NEPAL, S., and ZHU, L. 2016. Developing Dependable and Secure Cloud Applications. IEEE Internet Computing 20, 3, 74-79. DOI= http://dx.doi.org/10.1109/MIC.2016.67. [30] Waterman, M.G. 2014. Reconciling agility and architecture: a theory of agile architecture. Doctoral Thesis, Victoria University of Wellington.
