How is Performance Addressed in DevOps? A Survey ...

1 downloads 0 Views 564KB Size Report
formance is addressed in industrial DevOps settings. In particular, we were interested in the frequency of executing performance evaluations, the tools being ...
How is Performance Addressed in DevOps? A Survey on Industrial Practices

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

Bezemer1 ,

Eismann2 ,

Ferme3 ,

37 38 39 40 41 42

ABSTRACT

1

DevOps is a modern software engineering paradigm that is gaining widespread adoption in industry. The goal of DevOps is to bring software changes into production with a high frequency and fast feedback cycles. This conflicts with software quality assurance activities, particularly with respect to performance. For instance, performance evaluation activities — such as load testing — require a considerable amount of time to get statistically significant results. We conducted an industrial survey to get insights into how performance is addressed in industrial DevOps settings. In particular, we were interested in the frequency of executing performance evaluations, the tools being used, the granularity of the obtained performance data, and the use of model-based techniques. The survey responses, which come from a wide variety of participants from different industry sectors, indicate that the complexity of performance engineering approaches and tools is a barrier for wide-spread adoption of performance analysis in DevOps. The implication of our results is that performance analysis tools need to have a short learning curve, and should be easy to integrate into the DevOps pipeline.

DevOps is a modern software engineering paradigm that aims to reduce the time between changing software and delivering these changes into production with high quality [27]. This reduction in delivery time is achieved through organizational changes that bring together development and operations teams and processes with a high degree of automation, e.g., via continuous delivery (CD) pipelines and quality gates [16]. One of the most important quality aspects of a software system is performance. The performance of a system can be described as several system properties that concern the system’s timeliness and use of resources [17]. Common performance metrics are response time, throughput, and resource utilization. Performance requirements for software systems are typically defined by setting upper and/or lower bounds for these and other metrics. In order to ensure that such performance requirements can be met, several activities are required during the development and operation of these systems [8]. A common distinction is made between model-based activities, such as prediction using performance models [11], and measurement-based activities, such as load testing [18] and monitoring [14]. Historically, performance-related activities in software development and operations were tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams [9]. In our prior work [9], we discussed how existing solutions could support this integration, as well as open research challenges in the area of performance evaluation in DevOps. Despite the widespread adoption of DevOps practices and technologies, there are still many unanswered questions about DevOps. In this paper, we discuss our survey on the current state-ofpractice of addressing performance concerns in industrial DevOps applications. Prior empirical studies show that the adoption of DevOps correlates with positive software quality outcomes [26]. Also, in the open source community, the usage of DevOps and continuous integration (CI) leads to more frequent releases [15]. However, these studies do not present the current practice of performance engineering in DevOps applications. Our survey is the first to focus on performance engineering practices in a DevOps setting.

ACM Reference Format: Cor-Paul Bezemer1 , Simon Eismann2 , Vincenzo Ferme3 , Johannes Grohmann2 , Robert Heinrich4 , Pooyan Jamshidi5 , Weiyi Shang6 , André van Hoorn7 , Monica Villaviencio8 , Jürgen Walter2 , Felix Willnecker9 . 2018. How is Performance Addressed in DevOps? A Survey on Industrial Practices. In Proceedings of (Preprint). ACM, New York, NY, USA, 6 pages. https://doi.org/ 10.1145/nnnnnnn.nnnnnnn

45 46 47 48

51 52 53 54 55 56 57 58

62

Heinrich4 ,

of Alberta (Canada) 2 University of Würzburg (Germany) 3 Software Institute, USI-Lugano (Switzerland) 4 Karlsruhe Institute of Technology (Germany) 5 University of South Carolina (USA) 6 Concordia University (Canada) 7 University of Stuttgart (Germany) 8 ESPOL (Ecuador) 9 fortiss GmbH (Germany) [email protected]

44

50

61

1 University

43

49

60

Cor-Paul Simon Vincenzo Johannes Robert Pooyan Jamshidi5 , Weiyi Shang6 , André van Hoorn7 , Monica Villaviencio8 , Jürgen Walter2 , Felix Willnecker9

35 36

Grohmann2 ,

59

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Preprint, , © 2018 Association for Computing Machinery. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn

INTRODUCTION

63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115

1

116

Preprint, , CP. Bezemer et al. 117

Table 1: A summary of the implications of our findings

118

175

119

1. The complexity of performance engineering approaches is a barrier for wide-spread adoption by practitioners. 2. Performance engineering approaches must be lightweight. 3. Performance engineering approaches must smoothly integrate with existing tools in the DevOps pipeline.

120 121 122

Section 5.1 Section 5.2 Section 5.3

123

In particular, we focus on the following aspects:

125 127 128 129 130 131

134 135 136 137 138 139 140 141 142 143 144 145 146 147

relatively new and far from being applied widely in industry, as also reported by Logz.io [1] and Erich et al. [12]. CA Technologies [3] discusses the findings from an audience of 1425 senior IT and line-of-business executives and reports on the most critical DevOps demand drivers and tools, along with DevOps benefits and the factors that are driving DevOps. It is interesting to notice that improving the quality and performance of the applications is the top driver, with 42 % of the participants agreeing on this. Tool-wise, application performance management and monitoring (APM) [14] tools are perceived as the most important tools for DevOps by 38 % of the participants, while 37 % of the participants consider performance testing tools as critical. KMS Technology [4] surveyed 200 IT practitioners who were involved in transitioning to DevOps, and reported that 51 % had a very positive impression, and 79 % had achieved their desired goals. They also reported that the most significant challenge during the transition was the limited skill set and knowledge about DevOps among in-house IT staff (28 %). The second biggest challenge was a lack of support from the executive staff (23 %), followed by an inability to agree on and/or articulate the goals of the transition (18 %). In addition, prior surveys of practitioners targeted the industrial adoption of performance testing [6] and CI [15]. The report by TechBeacon [6] is indirectly related to DevOps because the survey assessed performance engineering practices throughout the software development life cycle, and reported that 62 % of the participants agreed that performance engineering is important for DevOps. Hilton et al. [15] studied the barriers that developers face when using CI, and reported that the complexity of CI tools and flaky tests are important barriers for effective DevOps integration.

(1) How often are performance evaluations of applications developed using DevOps conducted in industry? (2) Which performance evaluation tools are being used in the CD pipeline? (3) What is the granularity of the analyzed performance data? (4) Are performance models used in the CD pipeline?

126

133

Our study reveals that automatic performance evaluations are usually not integrated into automatic delivery pipelines and not performed regularly. In addition, performance modeling is not applied in most companies. In this study, we observed that diagnosing performance issues is typically performed based on “human intuition” [19]: engineers investigate hypotheses about what might have gone wrong in the system using data analytics to draw a conclusion about the observed performance issue. The remainder of this paper is structured as follows. Section 2 provides an overview of related work, focusing on surveys about DevOps practices. Section 3 presents details about our methodology, including the survey design. The main results of our survey are discussed in Section 4. Section 5, discusses the main implications (which are summarized in Table 1) of our study. In Section 6, we discuss the threats to validity of our study. In Section 7, we conclude the paper.

148 149

2

150

Others have performed prior surveys to assess the state-of-practice of DevOps in industry. Several of these surveys were conducted by corporations that sell DevOps solutions to companies. While some surveys touched briefly upon the topic of software performance in DevOps, none of them focused on getting an in-depth overview of how performance engineering is applied in DevOps. Prior surveys on the organizational impact of applying DevOps in industry [1, 2, 12], assessed the DevOps adoption over different years [2] and the types of tools and techniques used in DevOps pipelines [3]. These surveys concluded from practitioner responses that DevOps has an increasingly large impact in industry. These prior surveys focused on the used tools, and underline how these tools are usable to optimize certain businesses and technology goals, such as improving software performance. In particular, software performance is discussed as one of the main drivers for using DevOps [3, 7]. Other drivers of the DevOps movement are: “more efficient time-to-production for new software; a better collaboration between IT and other lines of business; and more consistent and higher quality software deployments” [4]. Overall, the surveys conclude that the DevOps trend is substantial and long-term. Puppet [2] collected responses from 3200 surveyed practitioners, and reported that the percentage of teams that use DevOps (compared to other IT-related teams) increased from 16 % in 2014 to 27 % in 2017. As these percentages show, DevOps can still be considered

151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174

177 178 179 180

124

132

176

RELATED WORK

181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211

3

METHODOLOGY

This section describes the design of the survey, the way in which it was advertised, and the profile of the participants.

212 213 214 215 216

3.1

Survey Design

The survey design follows the guidelines for conducting surveys in software engineering by Linåker et al. [21]. We designed our web survey to answer how industry addresses performance in DevOps processes. Our survey contained 58 questions, divided into three parts: 1) questions about the participants’ professional information (11 questions); 2) questions about development process models and team organization (30 questions); and 3) questions about performance assessment and evaluation (17 questions). Based on the four aspects that are specified in Section 1, we defined the target audience for the survey mainly as DevOps engineers, software architects, software developers, software operation engineers, software performance testers, and software consultants 2

217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232

Preprint, , How is Performance Addressed in DevOps? 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250

with a focus on performance engineering at software vendors and consultant companies worldwide. We developed a set of initial hypotheses, such as on the frequency of performance evaluations, the applied tools and the acceptance of performance models. Based on the set of hypotheses, we derived a questionnaire plan, consisting of survey goals, such as “Measure capabilities of monitoring tools” or “Measure the completeness of the continuous delivery pipeline”. Each goal is composed of a set of concrete questions by which we want to answer the corresponding goal. Additionally, the survey design aims not only at describing “what” happens, but also at answering “why” it happens in order to conduct an explanatory study as opposed to just being descriptive. In order to enable comparison, we aimed at minimizing free text questions and introduced single and multiple choice questions as well as Likert scales as often as possible to order the choices. Questions with ordered choices are less difficult to answer for participants and easier to analyze for researchers than unordered ones [10, 13, 24].

291

depends on the project

3.2

253

We advertised the link of the survey through industry-related mailing lists such as the SPEC (Standard Performance Evaluation Corporation)1 mailing list, social media, related events such as DevOpsDays2 and links in online computer magazines and blogs. In addition, the request for participation in the survey was spread via the authors’ network of industry contacts. The data collection was conducted between May 2016 and March 2017. By the time this article was written, 26 full responses (all questions answered by participants) and 108 partial responses (a part of the questions answered) were gathered. The following sections of this paper are based on the 26 full responses only.

254 255 256 257 258 259 260 261 262 263 264

267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290

3.3

42.3%

294 295

dedicated

296

26.9% 0%

10%

20%

30%

297

40%

Figure 1: Responsibility for performance evaluation

298 299 300 301 302

applied less frequently. Continuous integration is often (38 %) applied in combination with agile processes, such as Scrum. Most participants (54 %) use real-time data for process improvement.

303 304 305 306

4

Survey Context and Advertisement

THE MAIN RESULTS OF OUR SURVEY

307

In this section, we present the main results of our survey. The complete questionnaire, raw response data, and a more detailed analysis are publicly available online [5].

308

4.1

312

Performance evaluations are not regularly conducted in most companies

Approximately one third of the participants conducts performance evaluations on a regular basis (19 % continuously, 8 % daily, and 8 % weekly). The other participants conduct performance evaluations monthly (8 %), quarterly (27 %), yearly (12 %), less than yearly (8 %), or never (12 %). In addition, 50 % of the participants spend less than 5 % of their time, and only 20 % spend more than 20 % of their time on performance. 26 % of the participants report that performance evaluations are assigned to dedicated persons or teams; 41 % report to be in charge themselves (see Fig. 1).

265 266

293

self-responsible

251 252

292

30.8%

309 310 311

313 314 315 316 317 318 319 320 321 322 323 324

Survey Participants

4.2

The collected responses cover a wide range of education levels, processes, roles, work experiences and company sizes. Approximately 85 % of the participants have a university degree (i.e., a Bachelor’s degree (35 %), a Master’s degree (25 %), or a Ph.D. (25 %)), while the other 15 % of the participants hold a high school degree. There is a variety of job positions represented in the sample; however, more than a half of the participants describe themselves as software developers, and less than 10 % as DevOps engineer or performance engineer. Most (56 %) of the participants have 1 to 3 years of working experience in their current position, while 22 % have 3 to 5 years of experience, and 22 % have 5 or more years of experience. The participants work in companies that have between 100 and 999 employees (42 %), between 10 and 99 employees (31 %), and between 1,000 and 9,999 employees (19 %). The remaining participants work at companies that have less than 10 employees or more than 10,000 employees (8 %). Most participants apply continuous integration (54 %) while continuous deployment (12 %) and continuous provisioning (4 %) are

Jenkins is by far the most widespread CI solution

There exists a wide variety of tools that support the continuous integration pipeline. Not surprisingly, version control systems (VCSs) are used by all surveyed practitioners. The vast majority uses Git (77 %) and/or SVN (38 %) as VCS. Jenkins is the most popular “endto-end” solution for CI. A majority of 77 % of the practitioners use Jenkins for continuous builds and 65 % of the practitioners use Jenkins to deploy their software. Surprisingly, 50 % of the practitioners use SSH as a deployment system, beating Puppet (31 %) at the third place. The relatively heavy use of SSH suggests that CI solutions such as Jenkins cannot yet fulfill all wishes of practitioners, e.g., because such solutions are not capable of working with legacy code. To monitor performance, practitioners tend to rely on lower level system tools (35 %), such as top, or Nagios (35 %). APM tools (which are advertised as tools that support CI) are hardly used by practitioners (see Fig. 2).

4.3

Application-level monitoring is mostly done in an ad-hoc manner

Even though 70 % of the participants reported to have access to monitoring data, the responses on how their systems are monitored were

1 https://www.spec.org/

2 https://www.devopsdays.org/

3

325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348

Preprint, , CP. Bezemer et al. 349 350

System

32.1%

Nagios

32.1%

general-purpose data analytics and visualization components (e.g., logging and Graphite) to set up custom monitoring infrastructures.

351 352 353 354 355 356 357 358 359 360

7.1%

New Relic

21.4% 0%

10%

362

30%

20%

361

Figure 2: Employed performance evaluation tools

363 364 365 366

Instruction Level

3.8%

5

367 368 369 370 371

Operation Level

42.3%

372 373 374 375 376 377

System level

50.0%

0%

10%

20%

30%

40%

50%

380 381 382 383 384 385 386 387 388 389 390 391 392

Heard about Queueing Petri Nets

5.1

19.2%

Have heard about performance models

76.9%

Would like to use performance models

69.2%

Do not use performance models 0%

88.5% 10%

20%

30%

40%

50%

60%

70%

80%

90%

Figure 4: Adoption of and attitude towards performance models

393 394 395 396 397 398 399 400 401 402 403 404 405 406

410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433

Figure 3: Granularity of system monitoring

378 379

IMPLICATIONS OF OUR FINDINGS

As discussed in Section 4.1, most surveyed companies do not regularly conduct performance evaluations. In prior work, Leitner and Bezemer [20] showed that in most open source projects performance evaluations are not conducted on a regular basis either. These findings suggest that there is a mismatch between what the plethora of performance engineering research delivers, and what practitioners are really looking for. Below, we discuss the most important implications of our study for researchers.

23.1%

Application Level

Few practitioners use performance models, despite widespread interest

The results of our survey reveal that performance models are currently not used in industry and there appears to be no trend towards their adoption either (see Fig. 4). Our survey shows that 88 % of the participants do not apply models for performance management, even though 18 (almost 70 %) of them state that they would like to use such models. While most participants are aware of performance modeling formalisms, their knowledge seems to be shallow, since our results show that only 5 (19 %) of the participants have (some) knowledge about queuing networks, i.e., the most well-known performance modeling formalism.

7.1%

Other

408 409

4.4

Dynatrace

407

surprising (see Fig. 3). While monitoring system-level (and infrastructure) metrics is common, hardly any monitoring is conducted at higher levels, in particular, at the application-level (e.g., using application-internal metrics). The lack of application-level monitoring is reflected by both the reported granularity of measurements and the used tools. The granularity of monitoring is mentioned with decreasing occurrence from system level (50 %), over application level (42 %) and operation level (23 %), to instruction level (4 %). Typical system-level monitoring tools such as Nagios and Munin, or those provided by the (operating) system were mentioned (73 %). As opposed to this, only 15 % of the participants reported that they are using a (commercial) APM tool. Three participants reported about self-developed tools, which seems to be a current trend to use

The complexity of performance engineering approaches is a barrier for wide-spread adoption by practitioners

Software performance assurance activities are complex tasks by nature that require much knowledge of various aspects of the entire software life-cycle. As a result, performance engineering approaches, which are often highly complex, are not straightforward for practitioners to adopt and understand. For example, performance modeling is a widely leveraged technique in research that can be particularly suitable in a DevOps context. As performance tests can be conducted much faster on performance models than on real applications, performance models could work well for applications that release many times per day. Unfortunately, Section 4.4 shows that the application of performance models is rare in industry. The lack of participants’ knowledge is the most likely cause for not having a clear opinion about the underlining science of such models. Performance modeling techniques, being mostly research prototypes, often lack documentation and require expert knowledge to be leveraged, which makes their integration for non-experts tedious. Hence, the valuable outcomes of the performance models may be difficult for practitioners to interpret, digest, or even trust.

5.2

Performance engineering approaches must be lightweight

Our findings highlight the need for more lightweight performance engineering approaches, which still retain the necessary accuracy. A step towards such approaches might be automating aspects of existing approaches and hiding their associated complexity from the practitioner. The high amount of required effort upfront to 4

434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464

Preprint, , How is Performance Addressed in DevOps? 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479

construct and calibrate a performance engineering technique (e.g., performance modeling) may be an extra barrier for industrial adoption. While academic studies show the benefits of performance models for reasoning about design decisions and trade-offs [23], industry may fear the high upfront cost. In addition, automated and systematic performance engineering approaches, e.g., creating and updating performance models, may facilitate the adoption of such techniques in industry. While automated extraction approaches approaches already exist [9], there is still no “one-click” solution, which would significantly reduce the entry barrier. One important step to enable more lightweight performance engineering approaches. An example of tools that aim at reducing the entry barrier are APM tools. Unfortunately we did not observe wide-spread adoption of such tools by practitioners, yet.

and analyze [10, 13, 22, 24]. Hence, we focused on closed-ended questions.

5.3

482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522

Performance engineering approaches must smoothly integrate with existing tools in the DevOps pipeline

A possible explanation for the low adoption of performance engineering practices in DevOps could be that performance engineering approaches are typically not designed with the consideration of DevOps as a general context. On the other hand, existing tools that are used in many DevOps settings, such as Puppet and Docker, do not integrate nicely with existing performance engineering processes in industry. For example, many practitioners still rely on old-fashioned tools, such as SSH and system tools, to deploy their applications and monitor performance. In addition, we observed that even though many participants conduct application level monitoring, they do so without the use of specialized tools (such as APM tools). Our recommendation for performance engineering researchers is to ensure that their tools integrate smoothly in existing DevOps pipelines. For example, we observed in the survey responses that Jenkins CI is by far the most popular CI tool. Hence, we recommend that researchers provide plugins that allow an easy integration of their performance evaluation tools in Jenkins.

6

524 525

7

480 481

523

CONCLUSION

526

In this paper, we highlight the results of an independent survey that focused on performance engineering practices in DevOps. We found that two third of participants do not conduct performance evaluation on a regular basis, and among the ones that conduct performance evaluations, 50 % of the participants spend less than 5 % of their time on them. For what concerns the applied practices in DevOps, most participants perform continuous integration, while continuous deployment and continuous provisioning is seldom implemented. Tool-wise, Jenkins is the most used end-to-end tool for implementing DevOps practices. We also found that the use of performance models by practitioners is very low. One explanation for the low adoption of performance engineering practices in DevOps could be that the DevOps movement is still in its infancy, and developers are still getting used to the opportunities that this movement offers in terms of automation of performance engineering processes. Our survey shows that even though the adoption of DevOps is relatively widespread in industry, performance engineering practices are lagging behind. Future research should focus on assisting software developers and performance engineers to convert their existing performance engineering practices into the DevOps pipeline.

527

ACKNOWLEDGEMENTS

549

This research was conducted by the SPEC RG DevOps Performance Working Group.3 We would like to thank all survey participants for their responses. The authors have benefited from discussions with various colleagues during community events such as the Dagstuhl seminar on “Software Performance Engineering in the DevOps World” [25].

528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548

550 551 552 553 554 555 556 557

REFERENCES [1] “The 2017 DevOps pulse,” https://logz.io/wp-content/uploads/2017/07/devops_ pulse_2017_final.pdf, accessed: 2018-06-01. [2] “2017 state of DevOps report,” https://www.ipexpoeurope.com/content/ download/10069/143970/file/2017-state-of-devops-report.pdf, accessed: 201806-01. [3] “DevOps: The worst-kept secret to winning in the application economy,” https://www.ca.com/content/dam/ca/us/files/white-paper/ devops-winning-in-application-economy-2.pdf, accessed: 2018-06-01. [4] “KMS technology survey: DevOps enjoying its moment in the sun,” https://www.kms-technology.com/press/ kms-technology-survey-devops-enjoying-its-moment-in-the-sun, accessed: 2018-06-01. [5] “Spec rg devops survey — supplementary material,” https://spec-rgdevops.github. io/specrg_devops_survey/, accessed: 2018-06-01. [6] “State of performance engineering,” https://techbeacon.com/sites/default/files/ gated_asset/state-of-performance-engineering-2015-16_final2.pdf, accessed: 2018-06-01. [7] “xMatters Atlassian DevOps Maturity,” http://info.xmatters.com/rs/ 178-CPU-592/images/atlassian_devops_survey.pdf?_ga=2.20537722.378918857. 1526304278-746614242.1525703839, accessed: 2018-06-01. [8] A. B. Bondi, Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice. AddisonWesley Professional, 2014. [9] A. Brunnert, A. van Hoorn, F. Willnecker, A. Danciu, W. Hasselbring, C. Heger, N. Herbst, P. Jamshidi, R. Jung, J. von Kistowski, A. Koziolek, J. Kroß, S. Spinner, C. Vögele, J. Walter, and A. Wert, “Performance-oriented

THREATS TO VALIDITY

In this section, we discuss the threats to validity of our study. Internal validity. Threats to internal validity relate to the participant bias and errors. A first internal validity threat concerns the possible selection bias for survey participants. To avoid such bias, we advertised the survey in a wide variety of channels (see Section 3.2). However, some of these channels (e.g., the SPEC mailing list) may target a specific audience. Hence, the results of our survey may be biased. In addition, our survey targeted industrial projects, which are mostly closed-source. Hence, our findings do not necessarily extend to open source projects. Future studies are necessary to further explore how performance is addressed in DevOps in other companies and in open source projects. Construct validity. A threat to the construct validity of this study is that our survey consisted mostly of closed-ended questions. As a result, the richness of the responses may be affected. However, we felt that the advantages of closed-ended questions outweighed the disadvantages: closed-ended questions are easier to answer

558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578

3 https://research.spec.org/devopswg

5

579 580

Preprint, , CP. Bezemer et al. 581 582 583 584 585 586

[10]

587

[11]

588 589 590 591 592

[12] [13] [14]

593 594 595

[15]

596 597 598

[16]

599

[17]

600

[18]

601

DevOps: A research agenda,” SPEC Research Group — DevOps Performance Working Group, Standard Performance Evaluation Corporation (SPEC), Tech. Rep. SPEC-RG-2015-01, Aug. 2015. [Online]. Available: http: //research.spec.org/fileadmin/user_upload/documents/wg_rgdevops/endorsed_ publications/SPEC-RG-2015-001-DevOpsPerformanceResearchAgenda.pdf A. C. Burns, R. F. Bush, and J. Nash, Basic marketing research: using Microsoft Excel data analysis. Pearson Prentice Hall, 2008. V. Cortellessa, A. D. Marco, and P. Inverardi, Model-Based Software Performance Analysis, 1st ed. Springer Publishing Company, Incorporated, 2011. F. M. A. Erich, C. Amrit, and M. Daneva, “A qualitative study of devops usage in practice,” J. Softw. Evol. Process, vol. 29, no. 6, pp. n/a–n/a, Jun. 2017. A. Fink, How to conduct surveys: A step-by-step guide: A step-by-step guide. Sage Publications, 2012. C. Heger, A. van Hoorn, M. Mann, and D. Okanović, “Application performance management: State of the art and challenges for the future,” in Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE ’17). ACM, 2017, pp. 429–432. M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig, “Usage, costs, and benefits of continuous integration in open-source projects,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). IEEE, 2016, pp. 426–437. J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Addison-Wesley Professional, 2010. R. Jain, The Art of Computer Systems Performance Analysis. John Wiley & Sons, 1991. Z. M. Jiang and A. E. Hassan, “A survey on load testing of large-scale software systems,” IEEE Transactions on Software Engineering, vol. 41, no. 11, pp. 1091–1118,

2015. [19] J. Kaldor, J. Mace, M. Bejda, E. Gao, W. Kuropatwa, J. O’Neill, K. W. Ong, B. Schaller, P. Shan, B. Viscomi et al., “Canopy: An end-to-end performance tracing and analysis system,” in Proceedings of the 26th Symposium on Operating Systems Principles (SOSP 2017). ACM, 2017, pp. 34–50. [20] P. Leitner and C.-P. Bezemer, “An exploratory study of the state of practice of performance testing in java-based open source projects,” in Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE). ACM, 2017, pp. 373–384. [21] J. Linåker, S. Sulaman, R. Maiani de Mello, and M. Höst, Guidelines for Conducting Surveys in Software Engineering, 2015. [22] U. Reja, K. L. Manfreda, V. Hlebec, and V. Vehovar, “Open-ended vs. close-ended questions in web questionnaires,” Developments in Applied Statistics (Metodološki zvezki), vol. 19, pp. 159–77, 2003. [23] R. H. Reussner, S. Becker, J. Happe, R. Heinrich, A. Koziolek, H. Koziolek, M. Kramer, and K. Krogmann, Modeling and simulating software architectures: The Palladio approach. MIT Press, 2016. [24] P. Salant, I. Dillman, and A. Don, How to conduct your own survey, 1994, no. 300.723 S3. [25] A. van Hoorn, P. Jamshidi, P. Leitner, and I. Weber, Eds., Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World, 2017. [26] B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov, “Quality and productivity outcomes relating to continuous integration in github,” in Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (FSE 2015). ACM, 2015, pp. 805–816. [27] L. Zhu, L. Bass, and G. Champlin-Scharff, “DevOps and its practices,” IEEE Software, vol. 33, no. 3, pp. 32–34, 2016.

639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658

602

659

603

660

604

661

605

662

606

663

607

664

608

665

609

666

610

667

611

668

612

669

613

670

614

671

615

672

616

673

617

674

618

675

619

676

620

677

621

678

622

679

623

680

624

681

625

682

626

683

627

684

628

685

629

686

630

687

631

688

632

689

633

690

634

691

635

692

636

693

637

694

638

695

6

696