Session 1B: Software Engineering and Design
ITiCSE '17, July 3-5, 2017, Bologna, Italy
Learning Agile with Tech Startup Software Engineering Projects Kevin Buffardi
Colleen Robb
David Rahn
Computer Science Department California State University, Chico Chico, California, USA 95929-0410 (+1) 530-898-5617
Department of Management California State University, Chico Chico, California, USA 95929-0011 (+1) 786-229-9760
Department of Management California State University, Chico Chico, California, USA 95929-0011 (+1) 530-588-5907
[email protected]
[email protected]
[email protected]
software in industry. Agile emphasizes incremental development with continuous delivery of the product to the customer with feedback that drives future revisions [2]. This quick cycle accentuates the ability to rapidly adapt to changing needs, especially in contrast to Agile’s predecessors that rely on rigidly documented requirements.
ABSTRACT The Tech Startup model is an approach to learning software engineering methods by partnering with students studying entrepreneurship to collaborate on real software products. Agile software development methods align with Lean Startup practices so that students in sister classes experience leading contemporary practices in their respective fields. This paper describes a pilot study of interdisciplinary Tech Startup projects with a heuristic evaluation of software engineering realism and formative assessment of students’ surveyed experiences. The study found several similar student outcomes to other project models; however, it also identified limitations in the pilot with corresponding recommendations for future implementations.
Following the Agile process and learning associated tools techniques can logically fit in software engineering courses meant to prepare students with contemporary software development skills. In particular, software engineering courses commonly incorporate semester-long team projects to approach emulating the real world software development process and environment. However, while students may enjoy the freedom of defining their own software products, studies [19] have recognized that such “toy projects” lack the pressure of delivering well-designed and tested software to real users and consequently, “Students know their code matters only as much as they might find our assignments interesting, or as much as it counts toward their grades” [18]. To properly learn Agile development, students’ team projects require business people (or customers) as external stakeholders.
CCS Concepts • Social and professional topics~Software engineering education • Social and professional topics~Computing and business • Software and its engineering~Agile software development
Keywords Software engineering; Agile software development; Lean Startup; interdisciplinary collaboration; entrepreneurship; team projects
Meanwhile, the United States Bureau of Labor Statistics projects that software publishers will be among the fastest growing industries through 2024 [14]. In the USA alone, fifteen to twenty technology companies are founded each year that eventually surpass over $100 million dollars in revenue [15]. Correspondingly, there is potential for computer science students to pursue technology startup business (tech startups) and follow in the footsteps of the founders of Google and Facebook, whose entrepreneurial ideas and development began while they were studying at university [13][21].
1. INTRODUCTION As many computer science students shift from academic to professional software development careers, upper-division software engineering courses may provide a unique opportunity to usher a smooth transition. Educational programming assignments often have limited scope and rigid designs for each individual student to complete in the matter of a couple weeks while real software development projects involve teamwork to design and produce more sophisticated products. Consequently, software engineering courses can expose students to development methods and skills that they will need if they choose to pursue the popular career path of software engineering.
In this paper, we introduce a “Tech Startup” approach to teaching Agile software development in a software engineering course by leveraging collaboration with an entrepreneurship class. We outline the partnership between the sister courses and explain strategies to address challenges such as managing intellectual property. Finally, we evaluate initial findings from students’ experiences from the first semester of implementing the tech startup model and compare the outcomes to other contemporary approaches to software engineering education.
Since its emergence during the turn of the century, Agile software development has grown as a prominent approach to building Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. ITiCSE '17, July 03-05, 2017, Bologna, Italy © 2017 ACM. ISBN 978-1-4503-4704-4/17/07…$15.00 DOI: http://dx.doi.org/10.1145/3059009.3059063
2. BACKGROUND 2.1 Lean Startup Lean Startup methodology arose from principles of Toyota’s innovative Lean manufacturing and supply chain management which eliminated risk by shortening development cycles and
28
Session 1B: Software Engineering and Design
ITiCSE '17, July 3-5, 2017, Bologna, Italy Specifically, Nurkkala and Brandle [19] identify six ways academic software engineering projects differentiate from real, professional projects, including: the lack of a real product, relatively short duration, high personnel turnover, low sophistication of software, no software maintenance, and no customer. They explain that external pressures from customers also help address projects focus on creating a real product.
testing business hypotheses by regularly producing new releases and measuring results to drive business decisions [25]. Specifically, Lean Startup methodology encourages releasing a version of the product that provides the greatest opportunity to discover and validate information about customers that also takes the least amount of effort to produce, known as the Minimum Viable Product (MVP) [23]. By releasing MVPs, studying customer response, and adapting to the observations for the next MVP, Lean Startups enter a “build-measure-learn” loop and continuously optimize both the business model and product.
An emerging remedy to engage students in more realistic projects is to involve them in Free and Open Source Software (FOSS) [1]. FOSS is a model for software development where the product’s code is publicly available online and licensed permissively so that anyone can use or modify it. Consequently, people from across the world may contribute to the same project. Meanwhile, others have leveraged the FOSS model to engage students in projects for social good by contributing to humanitarian (HFOSS) software [10].
2.2 Agile Software Development Similarly to principles of Lean Startup, Agile software development focuses on continuous delivery of software products in small increments with the ability to respond and adapt quickly to changes in requirements [2]. Moreover, Agile asserts that selforganized teams of “Business people and developers must work together daily throughout the project [and] the most efficient and effective method of conveying information to and within a development team is face-to-face conversation.” To also continually refine the team’s practices, “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly” [2]. The Agile paradigm does not prescribe specific methods; instead, various tools and techniques have established widespread use based on their compatibility with Agile philosophies.
Nevertheless, FOSS model’s strength in gathering remotely distributed collaborators also makes it impractical for applying Agile principles of frequent face-to-face communication. In acknowledging both the advantages and disadvantages of the traditional FOSS model, in Fall 2014, we founded the Chico Open Source Consortium (COSC), an organization to foster collaboration between local software professionals and computer science students on a FOSS project. Since its inception, COSC has worked on BossyUI [3] each semester and organized weekly face-to-face meetings. Because the team’s co-location differs from the traditionally remotelocated FOSS model, we distinguish it as a localized (LFOSS) model [4]. A previous study found advantages to LFOSS in its ability to adhere more closely to the Agile paradigm and by providing professional networking and employment opportunities for students [5]. Likewise, more direct modes of communication are positively correlated with software developers’ perceptions of being held accountable [6].
For example, Scrum [24] is a framework that facilitates face-toface conversation by arranging scrums—or daily standup meetings—and organizes intervals of rapid, incremental development and deployment into sprints. In correspondence with Lean Startup’s emphasis on MVP’s, Scrum sprints can focus on creating working functionality that delivers the most value with the least effort required. Instead of dedicating a distinct phase to verifying software functionality, Test-Driven Development (TDD) involves writing unit tests—automated tests that exercise a small unit (e.g. function) of code to check that it produces the correct functionality—incrementally as small units of code are developed [12] so they can be verified immediately and errors can be corrected before moving on. However, while methods and techniques like Scrum and TDD are commonplace in industry, Agile software development methods usually reflect a significant departure from how students are accustomed to working on programming assignments.
On the other hand, we previously compared FOSS and LFOSS projects to students working on propriety software sponsored by industry (hereafter referred to as industry for brevity) and selfcontained (SE-only) projects. While the vast majority of SE-only projects lacked external clients, we observed that students dedicated more hours [5] outside of class to SE-only projects. While increased hours worked on a project could possibly be attributed to a number of different factors, based on our anecdotal observations, we believe it may be at least partially credited to students’ greater motivation.
2.3 Software Engineering Projects
Others have involved external customers in software engineering projects, such as for service-learning projects [17], but doing so is difficult to scale for large software engineering courses with multiple teams. On the other hand, there is no literature on software engineering projects collaborating with entrepreneurship students to create viable commercial software products.
Accordingly, understanding and experiencing the process of software development is vital to software engineering education. Consequently, Problem-Based Learning (PBL) is a popular constructivist approach by which students discover issues and learn about a subject through the process of solving a problem [8]. By working on designing and developing a novel software product, students can follow PBL by exploring and discovering software engineering techniques that help them deliver quality software. Therefore, semester-long team projects are common in software engineering courses to expose students to experiential learning.
3. METHOD 3.1 Course Organization & Collaboration In the Fall, 2016 semester at California State University, Chico, professors from the Computer Science and Management departments planned and implemented a collaboration between students in upper-division courses in their respective programs: Software Engineering and Web Entrepreneurship. The latter course concentrated on acquiring experience and skills for running e-businesses, including: Lean Startup methodology; minimum viable products (MVP); customer and market development; web-
However, when software engineering students generate their own ideas for their assignment, their projects are often “thrown away,” and never used by real customers nor maintained by a development team beyond the end of the semester. Without the pressure to produce well-designed code that will have a longer lifespan and have to satisfy customers, students’ software development experience will not be particularly realistic.
29
Session 1B: Software Engineering and Design
ITiCSE '17, July 3-5, 2017, Bologna, Italy compared to other forms of project collaborations relative to how they address the six gaps between academic and realistic professional software engineering [19]. Specifically, comparisons were made based on the following needs: real product, long duration, continuity, design/program sophistication, software maintenance, and real customer. We refer to “sophistication” rather than Nurkkala and Brandle’s “complexity” to avoid confusion with cyclomatic complexity, the software metric.
based sales and marketing; social media; search engine optimization; and web analytics [7]. Before the semester began, we reached out to the entrepreneurship students to introduce them to the opportunity to collaborate with software engineering students. Those who expressed interest provided a brief questionnaire to describe their project idea, agree to dedicate at least 2 hours per week for interacting with and providing feedback to the software developers, and arrange a time for them to pitch their idea to the software engineering class. In addition, contacts from an industry partner, a local LFOSS collaborative project, and a remote FOSS project (as represented by a student who had interned with active development leads for the project) each pitched their projects.
Likewise, we compared the Tech Startup model to LFOSS as well as remote FOSS and Industry projects with regards to their compatibility with Agile principles. Specifically, we addressed the needs: iterative development, frequent delivery and feedback, face-to-face communication, business people and developer interaction, and reflective adaptation. We combined both assessments to form a set of criteria for heuristic evaluation.
Likewise, software engineering students with their own entrepreneurial ideas were given the opportunity to pitch their idea to the Web Entrepreneurship class to solicit a (student) client to focus on the business needs of the software product. Before approving any ideas, professors from both departments vetted the projects to make sure they were appropriate for the courses.
Separately, we evaluated the execution of the Tech Startup model’s pilot semester based on surveys of students’ attitudes and experiences. Along with with pre-project survey that collected students’ project preferences, students also completed the Foss2Serve pre-course instrument [11], a questionnaire with Likert-type items regarding computing and software engineering that has been used by multiple institutions [9] to evaluate students’ experiences with FOSS projects. Items exclusive FOSS projects were omitted and any questions that referred to FOSS or HFOSS were reworded to address “software engineering projects” more broadly. The following 5-point (1 “Strongly Disagree” to 5 “Strongly Agree”) Likert-type items were included in the instrument specifically to address students’ confidence in a breadth of software engineering topics (as adapted for wording):
All projects that garnered interest from both courses were compiled into a list along with brief descriptions and posted on the learning management system website for students’ reference. At the beginning of the second week of class, all software engineering students completed a pre-project survey, which included identifying students’ top three preferences (in order) for projects. The software engineering professor reviewed student preferences and assigned students to teams according to their preferences, while keeping into account students’ schedules and team sizes. As a result, the software engineering students (n=58) were divided into ten teams—one LFOSS (n=7), two industry (n=12), one FOSS (n=6), and six Tech Startup (n=33) teams.
SE01. I am comfortable that I could participate in the planning and development of a real-world software project; SE02. I can list the steps in the software process we used in the software engineering project; SE03. I can use a software process to develop a software engineering project; SE04. (omitted as HFOSS-specific question) SE05. I have gained some confidence in collaborating with professionals from a variety of locations and cultures; SE06. I can describe the impact of project complexity on the approaches used to develop software; SE07. I can describe the impact of project size on the approaches used to develop software; SE08. I am confident that I can maintain a software engineering project; SE09. I can describe the drawbacks and benefits of software engineering to society; SE10. I can use all tools and techniques employed in my software engineering project; SE11. I can participate in a software development team’s interactions; and SE12. Participation in a software engineering project has improved my understanding of how to behave like a computing professional.
Software engineering students had two, one-hour lectures and two, one-hour labs per week. Lab hours were dedicated to project teamwork where teams began each lab with a scrum (daily standup meeting) and occasionally reported team progress to the professor in a “scrum-of-scrums.” Meanwhile, teams coordinated with their respective clients outside of class. Software engineering lectures covered the Agile paradigm first, but also included learning applicable skills: collaborative version control with Git and GitHub, scrum, test-driven development (TDD), developer operations (DevOps) for continuous integration, software design patterns, and analyzing software quality using software tools and metrics. The team projects accounted for 75% of the course grade while the remaining 25% included assessments of mostly in-class assignments to practice the aforementioned skills. Software engineering students performed an end-of-term team presentation that demonstrated their product and reported their self-assessment of the software’s quality and adherence to Agile. In lieu of a final exam, teams (along with some of their clients) exhibited and discussed their work in a public “tech showcase,” where teams set up booths to show their work to the general public. External, impartial judges—recruited by the professors to represent tech and business professionals—rated each group for technical innovation and market viability; prizes (sponsored by local companies) were awarded to the highest-rated teams in each category.
However, we also wanted to gauge student confidence in applying specific skills taught in lecture as they applied them to software engineering projects. Consequently, we supplemented the questionnaire with the following software engineering skill items: SK01. I can write automated software tests to verify that the software reliably behaves as was intended; SK02. I can set up and manage a development environment and development operations (DevOps) tools for collaborating on a software development team;
3.2 Evaluation For formative assessment of the Tech Startup model [7], we evaluated the different categories of projects using a mixedmethods approach. First, we assessed how the Tech Startup model
30
Session 1B: Software Engineering and Design
ITiCSE '17, July 3-5, 2017, Bologna, Italy
SK03. I can identify and use software design patterns appropriately; SK04. I can analyze and evaluate the design of large software projects for strengths and weaknesses
4.1.1 Real Products and Customers The Tech Startup model evolved from exclusively software engineering student (SE-only) projects that often had no contact with customers and their software was usually discarded and unused at the end of the semester. Through collaboration with entrepreneurship students, Tech Startup projects are designed to benefit from external stakeholders who focus their attention on the product’s market as well as business and end-user needs. Even though these projects begin from scratch like SE-only, the entrepreneurship clients should help steer products with clearer goals of real world use.
Accordingly, we collected a post-project survey that used Foss2Serve’s post-semester evaluation instrument along with the supplementary software engineering skill items {SK01…SK04}. To investigate the impact of the students’ projects on their confidence in software engineering, we categorized students by project category {Tech Startup, LFOSS, FOSS, Industry} and compared the changes in their responses from pre- to post-project surveys.
LFOSS and FOSS projects have arguably more tangible evidence for having real products and real customers as long as their software is already published and adopted by users. However, the LFOSS project began (in Fall 2014) in a similar manner to Tech Startups, where target customers and products had been identified but the execution was yet to be seen. Likewise, our Industrymentored projects have always begun with comparable ideas for products and customers but no evidence of either until the product has been released. In terms of project realism, though, external pressure from a client to create a real, functional product provides an element of real world realism. In this perspective, each project category is similar but the biggest difference is that Tech Startups have business-minded students as clients while other projects tend to have professional, technology-minded mentors.
The post-project survey also included items for students to report their ratings on their project interactions, as previously used to evaluate FOSS project interaction [5] (5-point Likert-type items, unless otherwise noted with response options in italics): INT01. Outside of lecture and lab, estimate how many hours you spent per week on your project (free response) INT02. I directly communicated with the mentor/customer for my project... Choose one: {I didn’t have a mentor or customer; I had a mentor or customer but only other team members directly communicated with them; Less than once a month; Each month; Each week; Each day} INT03. My predominant contact with my mentor/customer was… Choose one: {I didn’t have a mentor or customer; I had a mentor or customer but only other team members directly communicated with them; In person; Synchronous communication (Phone, video messaging, instant messaging, etc); Asynchronous communication (Email, bulletin boards, messaging at different times, etc) } INT04. My mentor/customer was prompt in replying to me INT05. My mentor/customer’s communication was helpful to my progress on the software development project INT06. My mentor/customer held me accountable to completing my work well and on time INT07. Interacting with my mentor/customer was valuable for my professional networking
4.1.2 Project Duration and Continuity Real software is not complete within the limited timespan of a semester. Therefore, any project that begins and ends along with the semester lacks that element of realism. FOSS benefits from an open and public design that allows for projects to be continued, even when there is turnover in the development team. Likewise, by leveraging the FOSS model and the momentum of local software professionals carrying the project from semester-tosemester, LFOSS has also demonstrated strength in this dimension. On the other hand, none of the Industry projects we have observed lasted for more than a semester. Granted, the short duration is not inherent to Industry-supervised projects; it has just been the decision of our industry sponsor to change projects each semester.
Finally, the students answered two free response questions to provide qualitative feedback: “Explain what was most helpful about working with your mentor/customer” and “Explain what was most difficult about working with your mentor/customer.”
By its design and dependence on teams comprised entirely of students, continuity of Tech Startup projects may be its biggest challenge. However, we believe the challenge can at least partially addressed by involving entrepreneurship students, who are motivated to gain experience and success with a real startup business. Even if software engineering students become inactive at the end of the semester, we have required their work to be maintained on an online repository, which could then be passed on to new developers who will continue development.
Software engineering students received participation credit for completing each survey but their responses were not reviewed until after credit was awarded. As an option to opt-out of including their responses in this research, students were offered an opportunity to write a short essay instead but 100% of students (n=58) consented to include their responses in this research.
In cases of continuing with an entrepreneurial project despite team member turnover, intellectual property should be addressed to avoid legal repercussions of inadvertent partnerships [16]. In anticipation of this concern, the entrepreneurship professor held an intellectual property workshop for all Tech Startup teams at the beginning of the semester. Dynamic equity sharing, such as the Slicing Pie model, establishes a flexible relationship where different contributions—such as developing a particular feature— earn pre-defined amounts, as agreed upon by the entire team [22]. Consequently, if a contributor leaves a Tech Startup project that continues and later earns revenue, they still receive compensation, as previously agreed upon.
4. RESULTS 4.1 Heuristic Evaluation We have incorporated LFOSS, FOSS, and Industry projects in the Software Engineering course for multiple semesters, since Fall 2014. Meanwhile, each project category continued during the Fall 2016 semester, when we piloted the Tech Startup model in the same course. Based on our each project category’s inherent design as well as our observations of them in practice, we evaluated the Tech Startup in comparison to the others according to the software engineering project realism and adherence to Agile paradigm principle criteria outlined in the methods section of this paper.
During this preliminary investigation of the Tech Startup model, it would be premature to report on whether it is conducive to project
31
Session 1B: Software Engineering and Design
ITiCSE '17, July 3-5, 2017, Bologna, Italy
continuation and realistic software duration. However, it is worth noting anecdotally that three of the pilot Tech Startup teams expressed interest in recruiting more developers from the following semester and one team reported (unsolicited) that they plan to legally incorporate the business in the immediate future.
only Tech Startup and LFOSS models inherently involve inperson meetings between students and their clients, by design. The analysis of student surveys also shed light into project interactions in practice.
4.1.3 Software Sophistication and Maintenance
In the post-project survey, students identified their predominant form of communication with their client (INT02), which we ranked from 1 (Asynchronous communication) to 3 (In Person). To compare ordinal data from the Tech Startup group to those of other groups, we performed Wilcoxon-Mann-Whitney tests and used the Bonferroni method to adjust the critical value (α=.017) to account for multiple one-way tests. Despite the Tech Startup (M=2.38, sd=0.86) design for intended face-to-face communication, many students communicated most often through online messaging and only one team’s members unanimously reported face-to-face as the primary method of communication (n=3, M=3.0, sd=0.0). Even so, the Tech Startup group used more direct forms of communication approaching significance (p=.04) than Industry (M=2.0, sd=0.0) and FOSS (p=.12, M=2.0, sd=0.0), while less direct forms on average than LFOSS (p=.06, M=3.0, sd=0.0) approaching significance. For more nuanced understanding of combination of modes of communication, we plan to use survey items that report frequency of use for each level of communication in future studies.
4.2 Survey Analysis
Sophistication follows product continuity and maturation. While designing a product from scratch exercises useful software design skills, understanding, maintaining, and extending other programmers’ design and implementation is a distinct software engineering skill of its own. Given that software development jobs at established employers (with existing software) probably outnumber those for startup businesses, class projects that begin with pre-existing, sizable software generally expose students to more common real-world experience. FOSS and LFOSS projects again have an advantage in this dimension. However, if Tech Startup projects manage to continue for multiple semesters, the difference in sophistication and maintenance may become trivial for continuing projects of any category.
4.1.4 Agile Iteration The Agile process of developing in small intervals, delivering and receiving feedback, and adapting to changes is not particularly inherent to FOSS projects. However, from our observations, most active FOSS projects review code via pull requests—submissions of developers’ requested software revisions, such as bug patches or addition of new features. Accordingly, those reviewing pull requests usually do not want a laborious task of reviewing a lot of changes at once. Small, iterative pull requests are encouraged and the feedback (as comments or pull request rejections) establishes a sort of de facto Agile-like process. However, this iteration usually requires students to be more proactive about seeking feedback from FOSS projects, unless there is a dedicated mentor actively managing students (as we had for our FOSS project). In that regard, LFOSS tends to foster iterative development more naturally since local software professionals mentor the students in person with weekly meetings. Industry projects operate similarly.
Likewise, we ranked the reported frequency of communication from 1 “I had a mentor or customer but only other team members directly communicated with them” to 5 “Each day” and compared Tech Startup to other groups using Wilcoxon-Mann-Whitney oneway tests with Bonferroni-corrected critical value (α=.017). Tech Startup (M=3.19, sd=.91) groups reported less frequent communication with their clients than Industry (p
