Cooperation between Developers and Operations in Software Engineering Projects Bjørnar Tessem
Jon Iden
Dept. of Information Science and Media Studies, University of Bergen Postbox 7802, 5020 Bergen, Norway phone: (+47) 555 84 103
Dept. of Strategy and Management, Norwegian School of Economics and Business Administration Helleveien 30, 5045 Bergen, Norway phone: (+47) 55959690
[email protected]
[email protected] also ensure that non-functional requirements [9], like response time, availability, and security, are satisfied. In contrast, software developers will work with defining the functional requirements of software, designing, programming, and documenting it, and perhaps also educating the users.
ABSTRACT In this paper we discuss how the cooperation between developers and operations staff is practiced. We have analyzed data collected from a focus group of experienced software engineers and project managers, as well as interviews from two case studies. Our position is that well performed cooperation between the development team and the operations team is crucial for successful deployment and operations of a new or extensively revised software system. The data shows that cooperation can be improved in several development activities like requirements engineering, system design, documentation, testing, training, and deployment planning. Likely consequences of poor cooperation in these activities are lower productivity in development and operations, as well as unsatisfied users.
Bringing new, tailor-made software into production brings about changes in the operational environment, both technologically and organizationally. Operations may need new equipment, larger computing and networking capacity, make changes to work procedures, enter agreements about service levels and pricing, and train its personnel. It is therefore crucial that operations are involved early in the development process in order to make the necessary changes prior to putting the new software into production. Successful deployment rests on a mutual understanding among the two teams, and the ability of the development team to inform the operation team about the characteristics and requirements of the new software, and the operation team’s ability to implement the technical and operational changes needed.
Categories and Subject Descriptors K.6.1 [Management of Computing and Information Systems]: Project and People Management – Life cycle, Systems development.
In a position paper discussing the transfer of software from development to operations [11], the authors found that there is a lack of research on issues in this area. Existing research mainly restricts itself to discussing issues like installation procedures, and software for doing installations [3, 4]. The SWEBOK Guide [10] for example, only discusses these issues from a technological angle. Technology is of course relevant, but issues like organizational arrangements, roles, management, and how developers and operations should cooperate, also need to be approached. In the consultancy literature, the Ambler and Constantine’s book [1] on the transition phase in Unified Process stands pretty much alone in this respect. Other sources that include approaches to these issues are Microsoft’s Microsoft Solutions Framework [6] process and the Applications Management process in ITIL [7].
General Terms Management, Human Factors.
Keywords Cooperation, Development Team, Operations, Empirical Study
1. INTRODUCTION There are several important stakeholders in the development of software. The roles of and the relations between the user and the developers are highly emphasized in software methodologies. A second, less emphasized relationship is the one between the development team and the operations team, which is the staff that is going to operate the software after deployment, i.e., the operations or IT services organization.
Even though literature is scant on the topic, practitioners of course need to organize the cooperation between development team and operations team. Anticipating the outcome of this study, it is likely that the approaches used to handle this cooperation will range from ad hoc and informal solutions to well defined approaches and procedures based on many years of experience. We expect the same variety of maturity and quality as we find in the development practices followed by various software engineering teams.
The operations team’s daily work consists of system surveillance, manual routine tasks, maintenance, user support, and generally they
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHASE ’08, May 13, 2008, Leipzig, Germany. Copyright 2008 ACM 978-1-60558-039-5/08/05…$5.00.
Our current research topic is therefore to get an initial understanding of how cooperation between development and operations is enacted. In this paper we analyze cooperation in these activities using qualitative data from a focus group study as well as from two case studies.
105
has operations as a separate unit in the project organization. They have participated from the start” But often this is mainly a coordination activity as it seems as if developers are doing the main work in constructing the deployment solutions, and operations are only receiving those to do the installation. Also there is some codecision taking place in this phase concerning the establishment of routines for operating the new software in the existing software environment. But the practice is not unambiguous. For instance, one of the project managers told that in his company the developers do the final installation themselves as they do this for all versions of the software. He said: “The last step, the installation of the production version itself, operations ought to have responsibility for that. But we believe that the level of competence needed is so high that people working with this on a daily basis should perform this function. This creates some wonder and discouragement among in the operations team.”
2. THE DATA We have analyzed data from three different sources, one focus group and two case studies. The data from the case studies are only semi-structured interviews, whereas the data from the focus group is email interviews and a group discussion. The focus group consisted of six experienced software development project managers, each professionally engaged in the interplay between software development and operations. We first conducted email interviews with the group focusing on the activities involved in the transfer of software to operations. After a preliminary analysis of this data, we invited to a 2.5 hours workshop where we discussed cooperation between development and operations. The six experts came from various sectors like finance, telecommunications, IT consultancy, and the public domain. They had experience with various methodologies including Unified Process and agile methods, as well as traditional waterfall like methods.
Testing is an activity which is central in the last stages of the development process, but surprisingly operations do not participate to a large extent in this activity. Their responsibility is mainly to set up the test environments and then stay away. One of the project managers said that in his company they don’t want to include the operations team in the early test rounds. As he said: “We don’t want them there. They create a lot of noise”. Early testing seems to be mainly on functional aspects of the system, and only in the last test, the acceptance test, the non-functional requirements are considered. Acceptance testing is often the only test activity where operations cooperate with developers in order to assess how the software works.
The first case study was conducted in a development team in a pension insurance company. The team’s task was to construct a web portal providing customers with information about their pensions, and also for offering new products to the customers. Operations were outsourced to a large national company offering such services. The development team followed an iterative process with deliveries every 3 months. The interviewees were the project manager, two developers, a test manager, a requirements engineer, a system architect, and a local responsible for operations. These data have previously been analyzed by Braadland [2]. The analysis in this paper is conducted independently of that work.
Documentation The experts agreed that system documentation is essential for the operations team. This documentation is normally produced by developers. The experts suggest that operations should be more involved in this activity. However, there was some uncertainty about this among the experts, as they fear that involving operations in writing the documents, would lead to more work on the developers’ side in order to deliver documentation at the precision level wanted by operations.
The second case was conducted in an IT-company that both develops and operates software for its customers. They are delivering services to many types of businesses, and run a weakly defined process for development, like you often find in young software companies. The case application was a web solution that would make it possible for a customer to easily add news to a web site. The interviewees were four from the operations team, but also three developers. These data have previously been analyzed in a study by Halleraker [5]. The analysis in this paper is conducted independently of that work.
Separate organizations As software operations have matured it has evolved from being an activity to a large extent done by the developers themselves, to an activity carried out by a separate organizational unit within or outside the user organization. This evolution has had great impact on the relation between developers and operations, and among the developers in the expert group we encountered some doubts about this division. The organizational divide could lead to a situation where the operations team does not gain enough knowledge about the system. They may be less dedicated, and may feel less ownership to the software. One of the experts said: “Where I’m now, they outsourced the whole operations to a foreign actor. It is now much longer distance to the operations organization. Previously operations knew the system in and out. They understood the architecture of the system. Now they just know how to turn off and on the box. It has been a real degradation.” On the other side, some of them argued, more formal organization is beneficial, because one avoid mixing up development and operational tasks during daily work. Specialization is needed.
3. ANALYSIS Cooperation is a topic that has been subject to many studies in different contexts. However, the perspective we have chose to guide our search for cooperative practice and challenges related to that, is based on Schäl and Zeller’s [8] perspective that cooperation comes in three forms. The first is collaboration, where people work together in the execution of a certain activity. Secondly we have coordination, where individuals with different tasks working towards a common goal need to synchronize their activities with those of others in order to be maximally productive. The last form is co-decisions, where the individuals contribute to take a joint decision. We analyzed the data using a qualitative approach looking for text elements that relate to one of these three cooperation forms, and also relate to any of the activities identified above as candidates for cooperation.
Resources Although there was an agreement that bringing operations systematically into the development project, there was also awareness that getting the resources for this could be difficult. The operations organization may not want to prioritize cooperation with the development team during development unless funded, and
3.1 The focus group Deployment In today’s practice, the experts explained, the developers and operations usually cooperate in planning the deployment and daily production. One of them stated that operations’ participation was well defined: “The project I’m in now
106
practices here. They suggested more, earlier, and more direct cooperation with operations, also indicating a need for formal procedures to organize the cooperation. One developer said: “They are too little involved, I think, in the project itself. They just get orders to do this and that, and then it works or doesn’t work. It would have been much more favourable if they had participated in the project in a different way
if so, they want to see a clearer value from their involvement. This may be an area where savings are put into effect when the budget gets tight Involvement in development There was a clear agreement among the experts that operations need to be more involved in development at an early stage in the process. For example, non-functional requirements need to be defined early, need to be considered when defining the software architecture, and need to be tested before the final acceptance tests to avoid problems at deployment date. The experts proposed a range of different approaches to handle the issue of cooperation, from the strictly formal with communication plans and systematic involvement, to short releases with almost continuous feedback from operations to development. In sum, the experts argued that operations need to be able to participate in most activities of software development, and they believed that a generic framework for organizing the involvement of operations in the development is needed.
3.3 Case no 2 Reduced cooperation The second case is from a software company that recently, as a result of business and staff growth, had been divided into one development and one operations department. Data indicates that they were still in a period of organizational transition. The developers behaved much like they did as previously when the two groups were co-located, whereas operations had got more specialized and did not know really much about the development projects anymore. Interview data reveals that the operations team seldom was involved in the development process or in organized project meetings, and that this was felt as a problem, especially for operations. As one interviewee from operations explained, “generally, we are starting to cooperate when we get the deployment package at our hands”. As a consequence, it was pointed out, operations were not able to make plans for deployment activities and they did not obtain any system knowledge before the software was put into production. Thus, making the deployment plans were the sole responsibility of developers. One given explanation to the lack of cooperation was that development projects were not able to bill customers for meetings with operations.
3.2 Case no 1 Limited cooperation The data from this case is gathered from the development team only. In this organization, cooperation between development and operations was very limited. Mostly, the two teams worked independently of each other. As a result nonfunctional requirements were not defined in the project, no systems documentation was produced, non-functional aspects of the system were not tested, and operations had little knowledge of the software. Communications mediator One type of cooperation we could see was in the coordination of deployment. This coordination was handled by one single person with the title applications operator. A somewhat critical developers stated: ”Then we have a person at Application Operations who is the insurance company’s person there. It is this department that really should operate the software. But this is really not the case. But they talk to the operations contractor, to say it like that.” An example how this mediating role was involved was in the writing of the deployment documentation. This was originally written by the developers, and then passed on to the applications operator, who fixed errors in the document or returned the document to the developers. When it was accepted by him it was submitted to operations. They were allowed to give feedback on the documentation, which then again might take the way back to the developer team.
Non-functional requirements Although no formal procedures regulated development-operation cooperation during the development process, we found examples of ad hoc cooperation. Requirements engineering is an example. Developers had little knowledge of non-functional requirements, and they meant that this was operations’ responsibility. The development team, however, did not really care too much about these requirements, as one developer stated: “We don’t give a damn about infrastructure! We have no relations to hardware, nothing. And this is how it is supposed to be”. Further, operations were seldom involved in testing. “We do not have anything to do with testing. The product must be finished when it gets in our hands.” However, although they didn’t have any responsibility for the testing, operations personnel sometimes felt “obligated” to participate.
Acceptance testing The tasks of the operations team were mainly to run acceptance test of the software, checking if it functioned according to very vaguely specified non-functional requirements, as well as installing, running, and look after the daily operations of the software. Functional acceptance tests were not run. This in fact became the responsibility of the users, who were routed to the development team when they reported errors. Users thus were quite dissatisfied with the response they got.
Some formal coordination Short of formal procedures, the cooperation was done informally on the daily basis through discussions in the offices, telephone calls, or email communication. One formal coordination mechanism was identified though. After the new organizational arrangement was introduced, the operations department had created a formal system for request management in order to protect their daily routines from too much disturbances. The developers did not value this arrangement, as they found it too bureaucratic. In spite of deficiencies in the developer-operation cooperation, and the impediments made by organizational arrangement, the culture was positive, characterized by a good tone among the two teams.
Access to production systems The developer team applied an iterative process, but had to order acceptance tests and releases weeks in advance, and this had to go through the applications operator. The only form of collaboration we found between the two teams was that after a release a developer would sometimes work for a short time with the operations team to handle upcoming errors. After this period, and if errors were reported, the developers were not allowed to have access to logs and other parts of the production systems, due to security considerations by the operations organization. The developers showed some frustration with the
4. DISCUSSION Poorly handled cooperation As stated, we have taken the position that well performed cooperation between development and
107
knowledge about this area will give software organizations more and better options in defining an approach to developmentoperations cooperation that fits their particular situation
operations teams throughout the development process is vital for the successful deployment and operation of new software, a position supported by the ITIL framework for running IT services[7]. From the three data sources we can see that there is a lot left to be done to achieve this. The two cases confirm that the cooperation is poorly handled, and we must admit that we did not suspect that the cooperative practices would be so few and so ill-defined in real software development projects.
6. ACKNOWLEDGMENTS We would like to thank Stine Braadland and Hanne Halleraker for giving us access to the data they collected as part of their master thesis projects.
Perceived problems The teams struggle with several problems. The developers need better information from error logs to do their error fixes, but are not given access. Non-functional requirements are often not taken care of, not defined, and not tested. Members of the operations team have little knowledge about the software and this may lead to a reduced feeling of ownership for the successful operation of the software. This may again be caused by lack of sufficient documentation and training, as well as lack of opportunities to participate in planning of deployment and production.
7. REFERENCES [1] Ambler, S. W. , Constantine, L.L. (2001) The Unified Process Transition and Production Phases. Lawrence, KS: CMP Books [2] Braadland, S. (2007) Cooperation between Development Team and IT Operations Staff - a case study. Master Thesis, University of Bergen. [3] Dearle, A. (2007) “Software Deployment, Past, Present and Future”. In 2007 Future of Software Engineering, Workshop at International Conference on Software Engineering. Washington, DC: IEEE Computer Society, pp 269-284. DOI= http://dx.doi.org/10.1109/FOSE.2007.20
Well-defined participation needed We believe that there should be room for formal participation for the operations team in software process methodologies. In the literature on modern, agile methodologies and variants of Unified Process, little is said about the role of the operations team in development. Our opinion is that operations’ role in development should be as well-defined in each project as the roles of developers, users, testers, customers and others. And one should be willing to assign resources to this participation. Operations team members may for instance collaborate in specifying and testing requirements, and writing documentation. There is need for coordinative activities in planning and enacting deployment and production. And development and operations should be able to co-decide in choosing hardware and software architectures, and also in process improvement activities.
[4] Gentleman, M. V. (1996) “Challenges in Deploying Software: Rollout, Field Support, Upgrades”. SEKE 1996: Lake Tahoe, NV: Knowledge Systems Institute, pp 426-433 [5] Halleraker, H. (2007) A Case Study Of How a IT Operations Team Cooperates with a Development During a Systems Development Life Cycle. Master Thesis, University of Bergen. [6] Microsoft (2007) Microsoft Solutions Framework. http://www.microsoft.com/technet/solutionaccelerators/msf/def ault.mspx [7] OGC (2002) Application Management. London: The Stationary Office.
Consequences As for consequences of the low level of cooperation we do not have much data, but we may speculate that productivity in either organization is not optimal. Lack of communication and well defined routines can cause developers to spend unnecessary time to find needed information about operations problems, information they need to improve the software. We also heard that user support is not well organized, which potentially might lead to dissatisfied users. Missing non-functional requirements mean that the software and the hardware configurations may become suboptimal for the customer’s needs, and this may in the long run be costly for all stakeholders.
[8] Schäl, T., Zeller, B. (1991), "Design principles for cooperative office support systems in distributed process management", in Verrijn-Stuart, A.A, Sol, H.G., and Hammersley, P. (Eds), et al. Support Functionality in the Office Environment, NorthHolland, Amsterdam, pp.85-101. [9] Sommerville, I. (2007) Software Engineering. 8th ed. Harlow. England: Addison Wesley. [10] SWEBOK (2004) Guide to the Software Engineering Body of Knowledge: 2004 Edition. http://www.swebok.org/pdfformat.html
5. CONCLUSION In this work we have tried to shed some light on how developers and operations teams cooperate in software engineering projects. The data material shows that this cooperation is not handled optimally, and this has most likely consequences both for productivity of the developers and operations, the software’s quality, and finally the service to users. Therefore, we think it is worthwhile studying this area to greater depth to suggest improvements to existing software process methodologies. More
[11] Tessem, B and Iden, J, (2007) Issues in the Transfer of Software from Development to Production. In Habib, L., Molka-Danielsen, J., Moe, C.E., Graumann, C., and Larsen, T. J. (eds.) NOKOBIT 2007. Norwegian Conference on the Use IT in Organizations. pp. 15-28, Tapir, Trondheim, Norway.
108
