Building Better Software Better: Software Project Management. 2 ... Month. He
later wrote an essay, “There is no silver bullet,”1 which caused many comments.
BUILDING BETTER SOFTWARE BETTER:
S O F T WA R E P R O J E C T M A N AG E M E N T WHY PROJECT MANAGEMENT IS A DISASTER FOR SOFTWARE
David B. Black, Technology Partner, Oak Investment Partners One Gorham Island, Westport, CT 06880 USA www.oakvc.com
[email protected] ph: 201 670 6952 Rev 2.1, May 20, 2008 © David B. Black
Building Better Software Better: Software Project Management
I N T RO D U C T I O N
I started writing software in 1966, during a computer programming course in high school. I spent most of the years between then and now with software. Much of the time I wasn’t personally writing code, I have been heavily involved in practical computer programming through more than a dozen years working in the venture capital community with companies based on software technology. So I’ve been around and active in software for roughly two thirds of its total life, since it emerged around 1950. While much has changed, it’s amazing how little some important things have changed. Here are a few basic facts of (software) life: •
Most initiatives to write new programs fail, and are cancelled prior to completion. • Most programming projects that “succeed” are actually over budget, late, or miss their functional targets; frequently all of the above. • Most efforts to re-write software fail. • The advances that most people see in computers from the outside are mostly due to the computing, storage and networking hardware getting amazingly faster, cheaper and more reliable. • While the tools and environments for writing software have gotten richer, the ability of programmers to understand and trouble-shoot the complete environment has gotten poorer. This isn’t news, and it isn’t much talked about. People who have experience with software have various suggestions to make about how to make the situation better, but all agree that there is tremendous room for improvement. Fred Brooks wrote a great book on this subject more than 25 years ago, The Mythical ManMonth. He later wrote an essay, “There is no silver bullet,”1 which caused many comments at the time. In the twentieth anniversary edition of the book, he included the essay and some updating comments in which he basically says, software is hard to write – we know lots of reasons why it’s hard to write and many of the reasons that things go wrong, but after all this time, there is no “silver bullet” which will make software projects meet their expectations. I advise anyone interested in software to read this book. Brooks writes: “Most [responses to the essay] agree with most of the arguments …, but then go on to assert that there is indeed a silver bullet for the software beast, which the author has invented.”2 I agree with the arguments, and further agree that there is, indeed, no silver bullet for the software beast. I say this with the humility of having time and time again been convinced that there was indeed a silver bullet, and that I had it! That having been said, it isn’t as though all software development efforts are the same, or yield the same results. There is wide variability. The good news is, we really don’t need to compare ourselves to some made-up, abstract ideal of perfection in computer programming. In many practical situations, the main issue is to be relatively better at getting the job done 1
Reprinted as Chapter 16 in The Mythical Man-Month, 1995
2
Frederick P. Brooks, Jr. The Mythical Man-Month, Boston, 1995, p. 208
2
Building Better Software Better: Software Project Management than others who are attempting roughly the same thing. It’s like when a group of people are trying to escape from a people-eating tiger – they don’t need to out-run the tiger, they just need to out-run the other people. The purpose of this series of documents is to explain the ideas, techniques and characteristics I have observed that enable software groups to “run faster” than the competition. I don’t claim uniqueness, completeness or perfection for these prescriptions. Most software projects are doomed from the start because of the set of guiding concepts used to organize them. These concepts, copied from manufacturing and similar physical engineering disciplines, have led to nothing but trouble in software, and a clear pattern of success by exception. It is the purpose of this paper to deconstruct the orthodoxy of project management in order to clear the way for more effective methods of building software. Introduction..........................................................................................................................................2 The Process Aspect of Software Development...............................................................................4 Project management ............................................................................................................................4 The Evolution of software efforts.....................................................................................................6 When it works ..................................................................................................................................6 Bringing in adult supervision .........................................................................................................8 The inmates are in charge.............................................................................................................10 We can do better............................................................................................................................11 The “project management” Paradigm of Software Development..............................................12 Project management is good for some things ...........................................................................14 Unfortunately, software is not among them..............................................................................15 But is there an alternative? ...........................................................................................................15 Project Management: the Dirty Laundry ........................................................................................17 Comparing apples and warthogs.............................................................................................17 Problems with Project Management in Software .....................................................................19 Objective standards ..................................................................................................................20 Expectations as the standard...................................................................................................22 First time vs. repeats.................................................................................................................24 The difficulty of doing new things .........................................................................................25 Requirements and their spawn................................................................................................27 The consequences of change...................................................................................................28 The “planning” mind-set and the attitude towards variation .............................................29 Unanticipated obstacles: “Gosh darn” variations ................................................................30 Additions to the goals: “Scope creep” variations.................................................................34 The focus on form and process over content ......................................................................35 The difficulty and complexity of computers in general.......................................................36 Mental and physical work ........................................................................................................37 The difference between physical things and digital things..................................................40 Problems with Project Management in General .......................................................................41 Making project management work..............................................................................................43 Back to apples and warthogs...................................................................................................44 The Requisite Table ..................................................................................................................46 Beyond project management .......................................................................................................48
3
Building Better Software Better: Software Project Management
T H E P RO C E S S A S P E C T O F S O F T WA R E D E V E L O P M E N T
So how do we build software better? There are many ways to address this question. Let’s start with a simple scheme of categories: people, process and tools. The involvement of people is pretty obvious. The people organize themselves in a wide variety of ways, and engage in heated discussions concerning the best way of organizing themselves and their work. Therefore, process is also a perennial favorite. Since programmers use a wide variety of ever-evolving tools to do their work, it is the final category. It is essential to look at software construction from all three of these perspectives. While these are all important topics, in this document we will limit our attention to process. When things aren’t going well in a software project, and people begin to talk about “chaos,” how things are “out of control,” and wonder out loud how these well-qualified and well-equipped people can possibly be expected to deliver good results with such an ineffective organization or methodology, you can be sure that the focus is on process. Why is process so hard? You’d think we’d have it all figured out by now! In fact, many people, including most of in authority, think we know all about software process, but simply and inexplicably fail to execute it. What is it exactly about software process that leads to such frustration? Why, after all these years of experience, is so much of the time we spend building software like trying to make a room warm by vigorously breathing out on the furniture? I suggest the answer has something to do with our adherence to the set of concepts and practices known as “project management.” “Project management” is as effective at guiding software projects to success as hopping and grunting is at helping pool balls to drop in the intended pockets – it may be entertaining to watch, but it has no constructive impact on the outcome. More important, to the extent that we focus on our hopping and grunting technique, we fail to pay attention to what really matters – hitting the ball correctly with the queue. Similarly, in software projects, the more things get off track, the more we seem to focus on project management hopping and grunting activities, so much so that the shaking floor actually makes things worse. The fact is that the current reigning paradigm for how to organize software projects, “project management,” leads to bad results, and the more vigorously it is pursued, the worse the results get. Sadly, while nearly everyone knows that poor results are the norm in software projects, few people ascribe the blame to project management. P RO J E C T M A NA G E M E N T
“Project management” is the leading approach for understanding and controlling the software development process. While there are nearly as many flavors of project management as there are authors and managers, they share a set of common concepts. But as many people (except project managers) have pointed out, it seems that no matter how
4
Building Better Software Better: Software Project Management much effort goes into it, software projects seem to be either incredibly over-resourced and have extremely modest goals, in which case they work out just fine (assuming you’re patient and rich), or they are resourced according to the best judgments of the people involved and roughly aligned to business goals, in which case they are usually late, miss their goals, or both. When things go badly, what is the expert diagnosis? Poor project management; the project management you had wasn’t good enough, you need more of it! What’s wrong with this picture? Why does no one seem to question the appropriateness and effectiveness of “project management” itself as a method for assuring satisfactory results from software efforts? Many people will say that the bad outcomes are due to using the wrong brand of project management – but I never hear about flaws in the concept of “project management” itself. I think a deep examination of this concept, contrasted with healthy and constructive experience of building software, will lead you to conclude that “project management” (not just this flavor or that variety) comes out of bad thinking and inappropriate conceptual metaphors. This paper will attempt to show you how and why this is so, necessarily along with some explanation of the “right” way of doing things. Let me say right away that I’m not proposing a Fred Brooks-style “silver bullet” here that, if used, will make all our software problems go away; but I am saying that the whole style of thinking embodied in project management comes from people who don’t know how to write good software well. They did their best to impose order on the chaos they saw, but we can do a lot better. I’m going start by sketching the evolution of software efforts in their most typical form. Possibly after an initial phase, software efforts usually end up importing or having imposed on them one or more forms of “project management.” So naturally, I’ll spend some time explaining what project management is. I will start by doing this as “straight” as I can manage, presenting it in the best possible light. I want to make sure we’re all on the same page about what project management is in general, and frankly build some credibility, before exposing fang and claw and going for blood. We’ll see that the disease of project management has two forms: one of them is obviously virulent, characterized by grotesque and disfiguring symptoms; the other is chronic, characterized by an extended period of listlessness and torpor produced by internal corruption, but is nonetheless fatal in the end. Finally, in the spirit of a celebration at a funeral, we’ll talk about some of the alternatives to project management. If you’re a teenager in a world of arranged marriages, the world of marriage for love can seem wild, undisciplined and scary; similarly, the world of software without project management can be scary, but in the end, like marriage for love, I hope you’ll see the choices as being rich with opportunity. I recently spent some time with the seasoned, non-technical leader of one of our portfolio companies, and some of his lead technical people. We discussed one of their most successful products. The CEO described how he got involved with a couple customers who had a problem that no one could solve, how he promised them a solution and got his programming team to throw something together that sort of worked. They then scrambled, fixing problems and coming out with a flurry of new releases, always listening to the customer and evolving their code until things settled down, the customer’s needs were met and the company had a new product line.
5
Building Better Software Better: Software Project Management “Of course,” said the CEO, glancing over at his technical people, “that was the wrong way to do things. Later, we settled down and got back to proper project management.” Of course – the CEO had to intervene and make sure something important actually got done. Later, “project management,” i.e., doing very little but trying hard to do that little on time and on budget, could be allowed to return. It is my ultimate goal to make it so that future CEO’s don’t feel the need to apologize in situations like that, and so that future generations of technical people lead the way to more effective ways to build software than can be accomplished under the misguided orthodoxy of project management. T H E E VO L U T I O N O F S O F T WA R E E F F O RT S
Every once in a while, a small group of bright and motivated people with the right kind of education and experience get together and try to solve a problem with software. Sometimes they’re working on the problem directly, and sometimes they’re consciously trying to create a solution that’s better than the ones that are out there already. There is nearly always some good fortune involved when it works – it has to be the right time, the right place, have the right backing, the right connections, etc. But this doesn’t mean it has to be in an elite center with elite people, in fact frequently the places that are consciously set up to create software break-throughs are unable to do so. In any case, sometimes everything comes together and it really works. In my experience and opinion, this happens more than you might think. You don’t hear about it because one or more of the elements of the complete package for a big success are missing, or simply because it worked, everyone is happy, end of story!
When it works When software efforts work well, and I have been part of a few of them and observed others, they are a true pleasure. They are effective, efficient, satisfying and frequently lots of fun. It’s not that nothing goes wrong with projects that work – it’s just that mistakes happen quickly, are recognized quickly, are corrected and people move on. Things are built that aren’t needn’t – but the excess baggage is unceremoniously and unsentimentally dumped. Things that are needed aren’t built – but as soon as the lack is recognized, people focus on it and get it done. There are mis-communications – but they are surfaced, corrected and people move on. There are obstacles and gotchas that arise at inopportune times – but they are confronted head-on, overcome and people move on. Notice that there’s a theme here: move on. Find it, fix it and – move on! In projects that work, there’s a sense of aliveness, a sense of listening, learning, responding and creating. Things are happening in real time. It’s like an improvisational theatre group – they’re skilled actors with lots of experience, but they’re there, in the moment, feeling their way to a successful performance, working with the material that comes up. In projects that don’t work, it’s like one group writes a play for an audience they’re unfamiliar with and another group does their best to stage the play and act it out according to script. They know it’s not working with the audience, but this is a play, not improvisation, it simply isn’t possible to change the rules in mid-stream. 6
Building Better Software Better: Software Project Management Here are some successful software projects that I have participated in and/or observed. Today’s internet is a huge network spanning the globe, and it grew out of a small, successful software project at BBN. Roughly half a dozen people working over a remarkably short period of time created the first IMP (predecessor of today’s routers and switches). They overcame many obstacles without lots of fanfare and built a platform whose eventual success is sufficient to demonstrate its worth. That was a successful software project. My own first project after college was to build a FORTRAN compiler, in assembler language, for an early display computer3. The conditions were ideal – I had enough basic education in compiler theory to know the principles, enough experience in programming to have the skills, the direction from the head of the company to get the job done, done quickly and done well, and I was so young that I didn’t know what I was taking on and therefore too ignorant to realize I’d been given a very challenging assignment. With lots of enthusiasm, I simply dove in and wrote a compiler that passed all the tests designed by the standards committee in about six months -- In raw assembler, in 4K words, with only 6 level paper tape for storage. No one told me it couldn’t be done in that environment with those tools, that it would take a whole department if decent tools in a good environment were available, that the QA alone would take 3 months, etc. Ten years later, the compiler was still in regular production use. That was a successful software project. A small company had a successful piece of software for back-office processing of bank credit cards (like VISA and MC). They tried to modify it a number of times to meet the needs to retail card processing (like a Sears card), without success. The leading posting programmer got together with a couple of hard-core programmers at a leading retailer’s computer shop, understood what was really needed, and wrote a new posting program, from scratch, and had it in production in under a year. That program now processes hundreds of millions of accounts worldwide and is the leading such program in its industry. That was a successful software project. A young man doing consulting projects in the health care world recognized an opportunity to handle all the patient interaction and billing needs of physician practices via an internet-based service. There were a number of products already available that had credibility, installed base, etc. He wanted his service to do everything that both practice management and billing companies did and more, and to make it easier to use at the same time, and do it reliably over the internet. Naturally, the only way to get such a tall order filled was to talk his recently graduated brother into doing it. His brother persuaded a couple of friends to help him. They used methods and tools that are looked down on by industry “experts.” They got it done and the company is now transforming its field. That was a successful software project. An entrepreneur with experience in the automotive industry got excited by the possibilities of the internet for transforming the way dealerships sell and drivers buy cars. He For anyone interested in computer history, here: http://www.blinkenlights.com/classiccmp/imlac/ is information about the machine. It was a kind of machine we don’t see these days: two different processors sharing a common memory. The “computer” processor was a 12-bit version of the PDP-8. The “display” processor executed “display lists” over and over, the result of which was drawing a picture on the display.
3
7
Building Better Software Better: Software Project Management got some dealers and programmers together and quickly had something that worked. A tight cycle of interaction developed between the dealers, the handful of programmers and the entrepreneur. Before long, the company had traction and credibility and was the industry leader. That was a successful software project. In reality, there have been many successful software projects. The number of people involved tends to be on the small side, though not always. There is a high degree of interaction. Lots of mistakes are made, but the mistakes are made, recognized and corrected quickly. The people involved are very “in the moment,” reacting to what’s in front of them. They tend to feel like the time is passing quickly, but at the same time last week seems like ages ago. They feel like they’re skating on the edge between structure and chaos. It tends to be an exhilarating experience for the participants, and fun!
Bringing in adult supervision Whether or not the initial software project was successful in the way I have described, the next stage inevitably follows. The initial group probably saw themselves as brave pioneers forging new paths through the trackless wilderness, braving untold privations, hardships and danger. Particularly if the effort is successful, they see themselves as fulfilling a difficult but worthy mission in the best way that was possible given the circumstances. They are among the first to scale Everest. Those who follow them, regardless of the project’s outcome, tend to view them as wild, unruly cowboys, capable of eking a living at the fringes of society, but fundamentally undisciplined and unreliable. It’s time to bring law and order to Dodge City! The pioneers, instead of being viewed as manly men and tough frontier gals, are viewed as outlaws and ruffians. Frequently, to switch images, they are viewed as toddlers in a sandbox, badly in need of adult supervision. The keywords at this point are order, control and predictability. What’s interesting to me is how frequently this change of regime takes place, and how independent of the actual circumstances it actually is. Imagine the improvisational theatre group: maybe things don’t click quite as well as usual for a couple of nights, or maybe even it’s a disaster. In either case, people gets knots in their stomachs about the risks of improvisation, the unpredictability of results, how you don’t even know what the riffs are going to be about, much less what the actors will say! The call goes out to bring in an “experienced, senior” manager, or a “proven” consulting firm. This business is too important, too serious to leave things to chance. There’s too much money at stake. We need some adults to give us confidence (which is what it’s actually all about). Regardless of what people say, however, a couple things are certain at this stage. Costs will go up. By a lot. Lots of bad things will be said about the prior regime. Good things will be taken for granted. Lots of time will be needed “to clean up the mess.” There will be lots of meetings and lots of talk about discipline and control. Standard project management methods will be introduced. There’s a big emphasis on doing things “right.” In this context, “right” means more people, more money spent on just about everything, and more time. There’s a big emphasis on process and procedure. But there is optimism, because it’s really nice to wear grown-up clothes and drive a car instead of riding a bicycle.
8
Building Better Software Better: Software Project Management So what happens when “the law” arrives? Are you familiar with the phrase “Monday morning quarterback?” Well, it applies. There are lots of things that can be said about this stage, but probably the most important is that the focus shifts from work products to work process. The people who really care about work product tend to leave at this point – what a surprise! The organization no longer really cares about what they care about, so why shouldn’t they leave? Also, no big surprise, the people who leave tend to be the ones who actually wrote most of the code. There are many software projects that have been subjected to adult supervision. But it’s hard to find out about them if you’re not in the industry and experience them personally, because they generally don’t get written about. The people who helped create the disaster certainly don’t write about it; in their opinion, all they were trying to do was bring order to the chaos, and were either foiled by poor circumstances, unsupportive management, a bad business environment, or they defined what they did (whatever it was) as “success.” Yes, the output was poor, the productivity unconscionable, and so on, but predictability was achieved. The people who bailed out have better things to do than write about bad experiences, what good would it do anyway? The vast majority of people involved in the new regime just think that’s what software is, and they can barely write English, much less software, so no enlightenment can be expected from that direction. Even I have trouble writing about some of the juiciest examples, because some of the participants are still alive and might recognize themselves from the story, and I have no desire to add humiliation to an existence that must be, by any reasonable measure, nearly worthless already. But I’m going to suck it up and choke out a couple of examples, mostly because there are software groups I care about who are still in the “heroic” stage of development, and out of their desire to do the right thing, are listening to the received wisdom and considering bringing in adult supervision to “help them scale” or to “bring them to the next level” or to “clean things up” or because the board of directors is getting concerned about the lack of experience in the engineering leadership. When I hear something like this, my heart quakes with fear, and the only hope is that enlightenment will strike before the disaster that inevitably seems to come with adult supervision does. An entrepreneur conceived a break-through in how to manage physical engineering projects. While there are many tools available for this purpose, the variations among organizations and the intrinsic differences among the products being designed are so great, that implementations tend to become drawn-out and very expensive. He realized that the existing tools were designed as fixed products instead of configurable, programmable component-based tool kits. He solved this problem in an excellent way, expressing any engineering group’s product design elements and workflow in the form of easily editable meta-data. He got a group of theoretically-minded programmers in a Slavic country to do most of the programming. Early adopters were enjoying rapid implementations, short timeto-benefits, and easy customization. The product worked, the customers were happy. So the company decided it was time to – choke! Gasp! – scale up. They brought in a “seasoned engineering manager” to clean things up and build the foundation for rapid growth. The result was nine months of no new releases and a detailed study that called for putting the company on hold for an additional year while the software was re-written from the ground up, using proper enterprise class tools, instead of the toys used by the existing team. There 9
Building Better Software Better: Software Project Management was a major loss of investor confidence, and the once-promising company was put on lifesupport. A small company had a good position in the field of document imaging and workflow that was emerging at that time. Like many small companies, this one was chasing opportunities and stretching to leap-frog the competition, while inventing innovative features and satisfying customer demands. Productivity and results were high. Major features were going from conception to customer implementation in weeks. As usually happens in this kind of environment, goals were re-adjusted frequently and mistakes were made. An experienced new CEO was brought in, who was appalled at the lack of discipline and elementary project management skills. He compared the current schedule to one from a month ago, and couldn’t believe the differences. He moved decisively to fix the problem. He purchased expensive project management software, required all developers to attend the three day training class, and brought in management that was accustomed to this way of operating. Meanwhile, employees responded taking company time to write, rehearse and perform a play on “quality,” fashionable management books were bought and distributed by the score, and the time that it used to take to actually deliver a feature became the time to plan the project that might eventually result in the feature being built. Leaving out details, the product fell behind the competition and the company was pumped up and sold, and shortly withered away. The desire to bring in adult supervision is so nearly universal, and so much the common wisdom that it’s hard to realize in a software context how lame-brained it really is. Think about a horse trainer, for example. The horse he’s trained runs the Belmont Stakes. The horse comes in tenth. The trainer shouts with joy. He tells you proudly that his excellentlytrained horse came in exactly where he predicted – what could be better than that??!! Is it possible that, as the horse’s owner, you don’t fully share the trainer’s perspective, and start thinking that perhaps it’s time for a new trainer? It is possible that you really didn’t care about fore-knowledge of the tenth-place finish, but actually wanted to win the race??
The inmates are in charge The adult supervision phase rarely last indefinitely. After law and order has come riding into town, there are upset people. There may be a showdown at noon on Main Street. Gunfights may erupt in saloons. After the noise stops and the dust settles, pretty much everyone has signed up for the new regime, or evinces an air of superficial conformity while displaying a quiet, sarcastic detachment to those he deems the discerning few. Everyone agrees that schedules are important and that commitments must be met. Everyone wants things to be “under control.” Everyone agrees on what is required to achieve this, and sooner or later, it all happens. At beginning, the adults are clearly in charge. As time passes, the original adults sometimes think they’re still in charge. But slowly and inexorably, the power shifts from the adults to the “people.” There is usually no great drama. But ever so gradually, the inmates take charge of things in a completely “bloodless” revolution. Here are some of the things that are said, each full of logic and common sense in its own way, and following is what results in practice. “After you work a certain number of hours, your efficiency simply goes down. You can’t concentrate as well.” Æ Everyone slips out starting at 4 p.m. 10
Building Better Software Better: Software Project Management “This pushing to make unreasonable deadlines just exhausts everyone. They fight with each other, don’t get any better results, and then, after, everyone is so beat that nothing gets done for weeks.” Æ Deadlines get established so far in the future that earthquakes, floods and fires can take place, and they can still be met. The meaning of the word “deadline” evolves from “I’ll make it or die trying” to “a dead person could meet the target.” “People who do nothing but work are unpleasant, narrow, one-dimensional people. They don’t work well in teams, and before long they just crack up and become unproductive anyway.” Æ HR people start roaming the halls at 4 p.m. suggesting that people go home and spend more time with their families. Anyone who seems stressed is invited to take a week’s vacation. “Our products are sophisticated, complex pieces of technology. It takes a long time to understand everything. It’s hard to find enough experienced people to do the work.” Æ The “go along to get along” people are promoted and salaries of senior people advance rapidly. “There is a huge amount of value in the intellectual property we have built up here over the years and embodied in our people and processes.” Æ Suggestions for change are rejected without serious consideration. “Rodney isn’t really a team player. He makes other people uncomfortable, and sometimes says things that make other people feel bad. He’s bad for morale.” Æ Rodney is smart, hard-working and wants to build a great product. But he doesn’t fit in well, and is asked to look elsewhere for employment. “Our customers have come to depend on our reliability and commitment to quality. We have to continue to pay attention to what we do, and continue to do it extremely well.” Æ Don’t bother suggesting doing more, working in parallel, or getting things out more quickly. “Our customers don’t have the money to chase after every cool-sounding new idea that comes along, most of which fail anyway. They depend on us to incorporate new technology for them once it actually becomes useful and proven.” Æ We will continue to defend our approach, using words like “practical” and “proven,” even after it has become dead-and-buried obsolete. “Our customers pay our salaries. Their needs come first.” Æ I really liked the customer I met a couple years ago. Of course, there are always unreasonable people out there who care about no one but themselves. Am I supposed to get ulcers worrying about what people like that think? Who exactly is that going to help? AARRRRGGGHHH! Get me out of here!!! What’s really scary is that if you live in the asylum long enough, surrounded by this syrupy-sounding “common sense,” it begins to make sense to you; it feels comfortable and nice, in a warm and sleepy kind of way.
We can do better After the “adults” come in and “clean things up,” every once in a while someone in a position of power who cares notices that things are going terribly wrong. There is a big shake-up. Everyone gets upset. Heads roll. People leave. And in the best of circumstances, 11
Building Better Software Better: Software Project Management there is a renewed period of building great software, with the needs of the customer and the success of the business squarely in focus. In the ideal case, there is a good combination of the sense of urgency and focus on results characteristic of early-stage good software efforts, with the maturity, control and sense of perspective that comes from experience. The team is larger and doesn’t always work all weekend, but it still gets to the top of Everest, but now with better equipment and oxygen bottles and more maturity about when to call off an ascent due to bad weather. How to go about this is, of course, a huge topic. The good news is that there’s more than one way! I’m going to keep this short (for now), because we’re mostly talking about how not to do it. Here’s something to think about. Everyone knows “ready, aim, fire.” The joke that’s made about “out of control” software efforts that are severely lacking process is that they operate according to “ready, fire, aim” or “ready, fire, oops.” What almost always happens is that everyone agrees that missing the target when you fire is the worst thing that can happen. Therefore, more and more time and effort is expended on getting ready, and the originally-simple “aim” becomes “prepare to aim, get ready to aim, check your aim, adjust your aim, …” By the time you finally fire, the target has moved, died, composted and turned into a garden, or evolved to a higher life-form. One really good way to “do it better” is to implement an effective “aim, fire, repeat” strategy. Don’t screw around, aim quickly and fire as best you can! Look at where the bullet ended up, revise and repeat. If customers are involved, find a way to avoid having all of them looking over your shoulder while you’re doing this (or standing near the target), but have a few of them checking your judgment of how well you’ve hit the target, and even whether you’re aiming for the right one. There are good technical ways to accomplish this; see for example, my paper on software quality automation. You will quickly find that this method only works well with people who can actually pick up a gun, load it, aim, shoot, and see how they’ve done. Specialists who are good at loading but nothing else, holding the rifle but nothing else, great at ejecting the spent cartridge but happen to be blind, etc. just aren’t of much use in this environment. Sorry. They should find employment at a software shop that values their skills. T H E “ P RO J E C T M A NA G E M E N T ” PA R A D I G M O F S O F T WA R E D E V E L O P M E N T
“Project management” is normally the method that is brought in to bring order to the chaos and to cure whatever ails a software development effort. It’s the main thing the “adults” in the room are likely to think about. “Project management” is a paradigm for software development. It is an epidemic that infects groups of people who are supposed to be developing software, and is particularly virulent in places where little actual, effective software is produced. To hear people talk about it, you would think it is the only conceivable paradigm for software development. “What do you feel like breathing today?” “Oh, I think I’ll just stick with the old tried-and-true, air.”
12
Building Better Software Better: Software Project Management “Really? I guess I’m feeling adventurous; I’m going for the mud. It looks good today.” Of course, people don’t make choices like that for air. People who are into project management think of it as being like our “choice” for breathing. They think there are kinds of air, like sea air, high mountain air, polluted air, and so on, but they’re all air – all project management, the only choice for breathing, the only choice for managing software processes. So what is “project management?” This is a question that tends to inspire passionate responses, because people take the question to mean “what is the best (most effective, etc.) kind of project management?” Most of the books on the subject, one way or another, are polemics in response to this question, or attempts to establish the author as embodying the received wisdom and “best practices” in the field. There are lots of books like this, and you’re welcome to spend all the time you’d like reading them and absorbing their “wisdom.” Fortunately, there are a number of software products available to help people manage their projects using these techniques. By studying them, we can understand pretty easily what are the core concepts shared by all approaches. There are products that look at individual projects, and there are products that help executive managers oversee a portfolio of projects. I suspect that anyone reading this is already familiar with them, so if you’re really that impatient, you will probably skim ahead even if I don’t invite you to, but here’s a brief summary. “Project management” is a methodology that is applied to scheduling, coordinating, tracking and managing complex efforts involving multiple people and tasks. It has been widely accepted and applied to many disciplines. It may work well in those other disciplines, for all I know; here, our focus is software development. All approaches to project management assume there is a planning phase and a tracking phase to project management. Planning is where you set the expectations; tracking is where you get the results and compare how you did against the expectations you set. In planning, the general idea is to define the work that needs to get done in terms of tasks that take inputs, consume resources and produce outputs that become inputs to later tasks. If one task requires the output of another task, it is said to have a dependency on the other task. Frequently, each task is associated with entry criteria, which are the conditions that have to be met for the task to begin; similarly, there may be exit criteria for the task to be completed. In software projects, the resources are typically people who have various skills. Each task is then associated with so many resources having such and such skills. Project planning software typically supports entry of the actual people who may be available, the skills each person has, and the calendars of the individual and the group to account for vacations and holidays. Large tasks may be defined in terms of smaller sub-tasks. When some of the tasks can be performed at the same time, there is parallelism, assuming there are enough resources to staff the tasks. When a project has lots of parallelism, it is common to identify the critical path through the tasks, which is the sequence of tasks in which a day of delay in the task translates into a day of delay for the project as a whole. People tend to like to view project
13
Building Better Software Better: Software Project Management plans graphically, with time stretching to the right, and tasks represented as boxes, with lines connecting those that have dependencies with each other. During the course of a project, the people involved normally enter the actual time they spend on tasks and when milestones have been achieved. This enables the software to track the actual progress against the budget. As the project goes on, people like to look at the variance between actual and budget, this project and other similar ones, and other things. If someone gets sick or otherwise becomes unavailable, the project planner puts the change into the software and performs a what if analysis to understand the impact of the change, like with a spreadsheet. As the project proceeds, hitting the targets is typically emphasized. “Good project planning” is normally equated with the actual completion of the project being close in time to the budgeted completion. Project planning can incorporate any portion of the actual expenditure of effort towards the overall goal. The largest variable tends to be requirements. Determining the requirements to a degree of exactness can take a large portion of the overall time spent on the project, but the rest of the project normally depends to a large degree on these requirements. Therefore, whether or not they are officially part of the managed project, making requirements comprehensive and exact is normally emphasized.
Project management is good for some things When you lay it out like this, project management sounds like a perfectly normal, natural thing. Programmers can and have gotten excited about it. Even math guys have figured out how to apply what they do to it, particularly when you throw in probabilities, expected returns, the value of the results, etc. As a set of concepts, it makes sense. People in the field just accept it. But it is completely and utterly nuts! It’s not nuts for everything. In fact, it’s quite nice for processes that are (1) completely understood; (2) variations in the process are minor and have well-understood ways of being corrected; and (3) repetitive. There are lots of important things that meet the criteria, from mortgage processing to building cars. So project management is quite useful, where it is applicable. In fact, the concepts of project management are so powerful that they can be helpful even in situations where some of the criteria are not met. Think of the selling process. One way or another, most sales systems are project management systems. What you’re selling and how you’re supposed to sell are (or are supposed to be) well understood. There may be really important variations (you’re selling to someone new all the time), but at least you’re always selling basically the same thing to people who will buy it (if they buy it) for a similar set of reasons. And selling is definitely repetitive. So while you can’t predict with certainty the outcome of any particular sales effort, you can normally predict the odds of a successful conclusion at any stage.
14
Building Better Software Better: Software Project Management
Unfortunately, software is not among them I’m sure you’re wondering when I’m going to consider the question of whether project management, a wonderful tool in general, is applicable to software. The time is now, and the answer is no way, not at all, nada, are you kidding, duhhh, and many similar expressions. Obviously, the reason I think it’s not applicable is because: (1) any one particular software development task is not completely understood before you’ve done it, and maybe not even then; (2) what you encounter when you dive into a project can vary extremely widely and completely surprisingly; and (3) software development is not repetitive. When I go to work on an assembly line, my work is repetitive – I apply the same widget to the same assembly using the same tool during the same period of time all day long. In software, if every day you go in, you write the same line of code, fix the same bug or solve the same design problem, you’ve got a problem so deep that project management isn’t going to fix it; you better just find your way to the nearest fast food joint and see if you can qualify for an entry level position there, because programming isn’t for you, sweetie. Project management doesn’t work for software because it makes you think about software as a mechanical process. You have a picture in your head that is like a factory turning out many copies of a particular object; in fact, “software factory” is a term some people use when they want to convey that they can produce higher quality software more quickly. The methods that they implement correspond to this model. The model is like a production line (think of a big PERT chart) with inputs coming in (requirements or code produced by earlier steps), having actions performed against it at various workstations by people and/or programs, and finally the software pops out the end of the production line. The last few work steps frequently are ones devoted to assuring that the software has the desired quality. It is as though the original requirements are the drawings produced by the CAD workstation, and the original designs are the tapes produced for the numerically-controlled milling machines, and then everything else is just the mechanical steps required to actually build the software. This is, I suppose, a nice dream, but it doesn’t match any software project I’ve ever been part of, excepting of course the rhetoric. But it sure is a persistent dream. We seem to want a “reliable” software development process that turns out good quality software on time. We seem to think that better trained people, superior processes and, sometimes, better programming tools can achieve this. But whatever we focus on, the model is a mechanical one. Yup, those managers can sit back in their chairs and dream about their tight, efficient software factory, churning out quality code that gets the job done, just like clockwork. And just like clockwork, their “factory” fails to live up to their expectations.
But is there an alternative? So if mechanical project management isn’t a productive way to think about software, what is? Well, we can’t really know until lots of people worth listening to agree to it, but it seems to me that the answer is found in those unusual software projects that really work. One way to understand them (we’ll talk about others later) is as an organic process. Yes, you start out with a pretty good idea of what you want to end up with and how you’re going to get there, and you give that idea a respectable-sounding name that probably includes “architecture” or “design.” But then you set out building the thing. As you are building it, you learn all sorts of things you didn’t know before starting, things you couldn’t possibly 15
Building Better Software Better: Software Project Management learn without actually doing the work. People who have a choice make a wide variety of minor course corrections and changes as they are building, the result of which is superior software. It’s like taking a walk rather than riding on train tracks. When you’re on a train, we all know what results from even a minor deviation from the tracks – we call it a “train wreck.” When you take a walk, you know the destination and the general route, but there is no need to plan every step in advance, in fact it’s best if you leave such decisions to the time. We want our trains and our software projects to “run on time.” But if we continue to treat deviations from the original plan as disasters to be fended off at all costs, we’ll not only be late, but we’ll drop our passengers off at the wrong location. Unlike taking a train, when we build software, we’re actually exploring the proposed route and laying the tracks. One way to think of organic software development is like creating a baby and then growing it into an adult. When you start with the baby, it’s little, helpless and can’t do much beyond eat and breathe; this is kind of the equivalent of a “Hello, world” program. But it’s alive! With babies, we constantly monitor their health. We insist that growing a baby into an adult is a process that is additive and continuous. This means that, as the baby grows, it learns new skills (walking, talking, tying its own shoes, etc.) that add to the skills it already has. It is completely unacceptable to think that, for example, the baby forgets how to walk as a consequence of learning how to tie its shoes. Each acquired skill increments the accumulated base of acquired skills. In addition, the growth is continuous (although not necessarily at a uniform pace). At no time does the pre-adult die, go into hibernation, or otherwise cease its basic functions. Once it is alive, it stays alive, with all functions mastered to date. Similarly, the ideal method of program construction starts with a program that is alive but can’t do much. The program may not change much in a day, and when functions are first added (think walking), they may not be very effective, but the program “stays alive” and keeps getting new functions and refinements or extensions of old ones. In contrast, the mechanical model normally gets us to think in terms of fixed “subassemblies” that are constructed at different stages of the “assembly line.” (This is like taking a long time to build adult-style arms, legs, etc.) The different sub-assemblies are built to a master plan, and when they are all ready, we bolt them all together into a complete system. But we know the likely success of this from repeated experience, so we plan for a period of “integration testing,” which is a long, unstructured period of time during which everyone is trying to get something that is the software equivalent of “Frankenstein’s monster” to act like it’s alive – something which it has never previously done. Fixing one thing breaks another, implications of the design that had never before been understood come to light, last-minute drastic compromises are made in the spirit of getting to the finish line. While the phrase “death march” can aptly be applied to any phase of a software development project, the phrase “death march in quick-time” is particularly apropos of this “final” stage of development. How much more pleasant it is to “grow the baby.” This approach doesn’t guarantee that everything will go wonderfully, of course. But there are advantages. Our baby (or toddler or child or pre-teen, etc.) is always alive and always presentable for what it is. Instead of hoping it comes alive at adulthood, it’s always alive.
16
Building Better Software Better: Software Project Management Regardless of how we implement it, the principle of organic development is clear. While mechanical development attempts (rarely successfully) to carry out a pre-set plan, organic development starts with a “baby” version of what it wants to end up with, and “grows” it to adulthood, with continual enhancement, feedback and learning taking place along the way. Obviously, there’s a lot to ask about organic development. But now is not the time or place. For now, it’s sufficient to have an “existence proof,” an example (to be elaborated elsewhere) of another way of thinking about software development. If there is another way, then mechanical project management is not like air, it’s more like dessert at a restaurant – there are a number of choices, and the choices are meaningfully different with different consequences. P RO J E C T M A NA G E M E N T: T H E D I RT Y L AU N D RY
A lot people know and love project management. It’s the best they have. They feel complete and fulfilled with it, in spite of all the evidence. Why should you pay attention to a possible alternative when you like what you have? So I think it’s time to lay out in more detail just why it is that the mechanical project management approach to software deserves our derision and scorn.
Comparing apples and warthogs People who believe in project management for software tend to rely on how much sense it makes in general, and how strong the correspondences are between its non-software uses and its software applications. So let’s start by examining the analogy. Is it really apples to apples? Suppose you are the planning manager for a road construction company. You are looking at a request for bids for some road widening project. The request, naturally, tells you exactly where and how the road is to be widened. You can go and inspect the current road yourself. The request tells you what you have to do with manhole covers, sidewalks, traffic during construction, the specifications of the materials to be used for re-paving, and many other things. Your job is, first of all, to produce a winning quote. A winning quote should be compliant with the requirements, for example when you must start by and by when the job must be complete; your bid should also be the lowest compliant bid. If your bid is accepted, you should be able to actually meet the requirements closely enough to avoid getting into trouble, and make money on the job. So you have to plan out the amounts of various materials, the availability of various machines and men, and the time it takes to perform the various tasks involved in the project. You have to allow for things going wrong, things like another job running late, bad weather, and so on. Fortunately, this isn’t your first day on the job, and this isn’t your first job of this kind. You know all about projects like this, and could probably, in a pinch, fill in and do any of the jobs. You’ve done many projects like this. They all involve the same elements, in roughly the same order. The physical location and condition of the job site varies, of course, but you’ve learned to scope out the important variations ahead of time so you can account for them. On any one job, you may be delayed more days than you allowed due to weather or fewer days, but since the crews are active most of the time, it evens out in the end. In the vast 17
Building Better Software Better: Software Project Management majority of cases, you can bring the jobs in on time and on budget, so that the customer is satisfied, your company makes money, and you get that year-end bonus. It’s pretty much the same thing with software, right? There is the existing road that is too narrow (the existing code), the requirements to make it wider (what more capabilities the users want), surveying the job site (inspecting the existing code and functionality), planning for the required materials (new lines of code), machines (development and test systems and software) and workers (programmers), and finally organizing the work in a sensible way. Based on experience and the scope of the work to be done, the time and resources for each step can be estimated in advance, and then tracked during execution. Results in software should be as predictable as they are in construction, right? This is the kind of analogy that gives heart to believers in project management. I kind of wish, for their sakes, that the analogy made sense. Sadly, it does not. Or, rather, the analogy makes sense as stated, but it’s really not comparing apples to apples, but apples to warthogs. Here is why: •
•
• • •
•
•
In construction, the main thing you’re working with is physical, material substances; if I stooped to that kind of humor, I might even call them “concrete” things, which in any case are easily quantified. In programming, you’re dealing with intellectual constructs which avoid measurement, and for which quantification tends to be misleading; for example, most mathematicians prefer proofs that have fewer steps, while the value of a road goes up strictly according to its length. In construction, the obstacles are pretty normal things that are easily identified and dealt with by standard means. Programming is more like doing construction in a war zone, in which a shovel might hit a land mine at any moment and blow the whole thing up. In construction, the maker of the underlying ground never gives you a “new release” of the ground which is suddenly titanium-laced granite, while in software the environment constantly changes in devious and perverse ways. In construction, it is common to have requirements that are clear and unchanging. In software, the buyers are constantly dropping by the job site and demanding things like adding a jogging path, or adding in provision for a strip mall along the way. Imagine what it would be like if, in construction, the users came along a month after the project got underway and said that things have changed in the last month, there’s so much extra traffic that the whole road needs to be made freeway style, and otherwise it’s worthless. Imagine that in construction, in any one town, there is a mixture of dirt roads, concrete roads, macadam, light rail for personal rail cars and channels for rocketpropelled vehicles, with fervent proponents of travel by foot, horse, model T, Ferrari, personal rail vehicles and rocket cars. Your workforce and machines are a mish-mash of the above. Now imagine that the requirements were a compromise between the leading groups favoring travel by horse and by rocket, with a few points favoring bicycles. How would things go? Imagine that the tools and materials you work with advance in power and efficiency every year the equivalent of 20 year’s advance (minimum!) of motor vehicles, and that you have to build something that will still make sense five years from now. 18
Building Better Software Better: Software Project Management •
Imagine that the construction materials used for the roads people are using today actually carry the vehicles, but in spite the overcrowding, delays, and never-ending traffic jams, the roads “work.” Imagine that, for the revised set of roads, everyone wants the best new construction materials and techniques to be used, but roads built using them constantly collapse, erupt in geysers that last for hours, and get meterdeep potholes that appear in seconds and sometimes cure themselves minutes or hours later, and other times grow until the whole section needs to be replaced. • Imagine that a typical job is to replace a dirt road with a guideway for rocket cars, and the people writing the specifications have read about rocket cars, talked with friends who liked them, but never been in one. They do the best they can, but of course the requirements are full of holes, contradictory and vague. Worse, the people building the guideway for rocket cars were just hired for the purpose because they just left a company where they ate in the same cafeteria with people who read a lot about them, and where the company claimed to be good at building such things. • Suppose that all the construction equipment, including the most important and powerful, had controls that were documented in a novel combination of church latin and classical sanscrit, and only someone who had those backgrounds who had spent significant time in gothic cathedrals and Siva temples had a chance of operating them as intended. And suppose that, even when they were operating correctly, the machines were invisible and the results of their actions could only be seen at the end of the project? • Imagine that standard operating procedure were to assemble the new sidewalks, roadways, etc. off-site, test them in big open spaces miles away from the construction site, and to test each “part” (the traffic lights, the pedestrian crossings, the right turn lanes, etc.) completely separately, and then to assemble all the “components,” and finally, at the end, replace the old roadway with the new one. Do you think that the integration and installation portions of this process would typically go well? • Finally, imagine that the construction industry had gotten used to this state of affairs. If you wanted to develop a reputation for reliability, you would probably focus on providing estimates that enabled earthquakes to take place in the middle of the project and still finish on time, insist on requirements that couldn’t be changed without exorbitant costs, and generally get everyone to think about the process rather than the reality. So I’m quite happy with a construction industry analogy for software, so long as we agree to change those nice, understandable, comforting facts in construction to correspond to software realities. We’ll examine each of these issues in more depth in the following sections.
Problems with Project Management in Software The problems with project management as applied to software development include: •
The lack of objective standards for measuring work and difficulty
•
Expectations as the standard
•
First time vs. repeats 19
Building Better Software Better: Software Project Management •
The difficulty of doing new things
•
Just what are requirements?
•
The consequences of change
•
The “planning” mind-set and the attitude towards variation
•
The focus on form and process over content
•
The difficulty and complexity of computers in general
•
The difference between mental and physical work
•
The difference between physical things and digital things
Now, I admit right away that none of these things sound like awful sins. It’s not like I’m saying something self-evidently awful, like “project managers consistently steal money,” or “project management directly results in cruelty to children,” or even the ever-popular insult, “project managers are Nazis who hate their mothers,” (although I’ve met people who think that last description fits pretty well). Instead, we get namby-pamby sounding complaints like “the difficulty of doing new things.” So what? That’s what “new” is all about – why should doing something you haven’t done before be easy? All I can say is chill out and read on. Once you really understand what these problems mean, you will dash off, retract all known copies of your resume, and edit it to disavow any involvement in, connection with or even any knowledge of project management. “Project management” as applied to software will be on the same list, in your mind, as such gems as “malfeasance” and “childish fantasy.”
Objective standards Probably the first thing to be said about project management is to point out the lack of absolute standards in software development. For relatively simple things with objective measures of success that are done over and over again by many people, there are extremely clear, objective standards. Think of carefully controlled things like track and field events (the high jump), or less well controlled things like your commute to work, and the commutes of other people to the same place. Say you tell a friend your commute this morning took half an hour. Your friend will immediately be able to react to that information by saying, “whoa, was there an accident?” or “how come they didn’t catch you, lead-foot?” Your friend has a clear idea of the range of reasonable times for a “normal” commute, and can use that knowledge to judge on his own whether the one this morning was fast or slow, and by a lot or a little. Now suppose a friend tells you that you’ve just finished designing a database, and that it took a week. Lots of people have designed databases. Say you’ve designed lots on your own. How do you respond? Is your friend a superman, for designing the database in record time, or a slug, for doing “as fast as molasses in January” one better? The fact is, without knowing a whole lot about the details of the database to be designed, you have no idea. Even if your
20
Building Better Software Better: Software Project Management friend tells you some basics like the number of tables and columns, you still really have no idea. Was there a starting model? Are there special performance, size, interface or compatibility requirements? Were there a bunch of conflicting interests to account for? Depending on lots of details like these, even knowing the “raw” size of the database that was designed, you still have no way of judging. There have been lots of attempts to introduce objective standards into computers over the years. But every objective standard has ended up introducing more problems than it has solved. A favorite standard is “lines of code (LOC).” The good thing about LOC is that it truly is objective. There are a couple things wrong with it. One tiny little problem is that it’s really hard to predict in advance, even approximately, how many lines of code are actually needed to solve a problem. The second problem, which unfortunately is worse than the first, is that LOC is a perverse, counter-productive measure. Suppose you’re building a brick wall. You plan out its dimensions, get the bricks and the cement, and start laying rows. You can predict pretty closely the number of bricks you’ll need. A bigger wall requires more bricks, more bricks cost more and have more value. If brick wall number 2 is 10 feet longer than brick wall number 1, and otherwise the same dimensions, we accept the fact that brick wall number 2 is worth more. So lines of code are like bricks, right? More lines of code is better and means more value, right? Sadly, the answer is wrong. Think of a sort routine. Is a sort routine that has more lines of code better than a sort routine that has fewer lines of code? By the bricks analogy, the answer is clearly yes. But any sensible person knows that we don’t care how many lines of code are in a sort routine – we really don’t! We care about things like its flexibility, the amount of time it takes to sort a given set of items, the amount of work space required beyond the additional items. If anything, there are likely to be circumstances in which we positively want the routine to have fewer lines, and to require less additional work space. It’s like the old, “sorry for the long letter – if I’d had more time I could have thought about it more and been briefer.” Bricks sit there and take up space; lines of code mostly exist to do things, and you care about how well they do the things they’re supposed to do. If you’ve ever been in an environment where they use LOC, guess what, everyone responds to the need to be productive by the LOC measure, so they merrily go around adding to the total LOC at every opportunity. The trouble with code, of course, is that, once born, someone has to “care” for each line of code for its whole “life.” So what we really should want is to continually abstract our code, to have fewer lines of better thought out code that gets more done. Every measure you can think of has the same couple of big problems – (1) the measure is nearly impossible to predict until you’ve done so much work that the job is practically done, and (2) in most cases, having fewer of the thing measured is actually better in the long run than having more. This is true for LOC, function points, classes, tables and just about anything else you can think of.
21
Building Better Software Better: Software Project Management Is this because software is weird or something? Well, probably, but we can explain the facts just as well by noticing that nice things like brick walls are passive, static things, they are assembled out of standard components, have been built essentially the same way for years, and have simple arithmetic relationships between the components and the results. Programs are not any of the above. What this means is that, when you’re looking at a task or set of tasks and trying to figure out whether the time allocation is reasonably accurate and achievable without lots of padding, unless the task is something safe like “take a 2 hour nap” and you’ve allocated 2.5 hours for it, neither you nor the person who made the estimate has much of a sense of whether it’s too much, too little, or just right.
Expectations as the standard The lack of objective standards is tied closely to the peculiar importance given to meeting expectations rather than some objective measure of achievement. The people who look at your project plan typically don’t have a clue – not even a foggy idea – of whether the times and resources assigned are reasonable. What the approver typically wants above all is confidence that, whatever resource and time commitments are made, are kept. So project management, in this context, is all about having lots and lots of highly technical looking stuff with charts and diagrams and fancy reports to justify completely outrageous resource and time commitments. No one looking at them has any way of knowing that they’re outrageous. So they get approved. The expectations are eventually met. And everyone’s a hero! What this means is completely bizarre. It means that project plan is judged not by comparing it to what would be the optimal plan, as objectively determined, but by comparing it to the expectations as set by the person who made the plan. Imagine that you tell your teenager to make his bed. Right now. The teen goes off to his room and you don’t hear anything for five minutes. You shout up, “what’s going on – are you done yet?” The teen responds, “give me a break! I just got started! It’s going to take me seventy minutes of continuous effort. I’ll let you know when I’m done.” If you walk up to the room, say something like “don’t give me that b#$%^” and ask what’s going on, then you have an objective idea of how long making the bed should take, and care more about comparing your teen’s performance to this objective measure than anything else. What you care about are your expectations, which you determine from your own sense of how long the project should take. If, on the other hand, you say, “OK, looking forward to hearing from you,” then you set your expectations completely on what you’re told, and care more about whether those expectations are met than anything else. What you care about is whether the expectations set by the person doing the work are met. You’ll be happy if your teen gets the bed made in an hour. You can run your family any way you want. Your kids can leave their beds unmade, for all I care. But what’s amazing is that people actually run projects like the 70 minute wonder teen in the story I just told! When this attitude towards expectations is extended to a group of people, the results are, if possible, even stranger and more perverse.
22
Building Better Software Better: Software Project Management Imagine we’re on a race track and there are some people running the 100 meter. Most of the people are outstanding runners, so they’re always competing with each other. In every race, all but one of them is a loser, and they tend to set aggressive goals of the times they’re going to make, which they frequently miss by a little. So the typical project manager would look at them as a bunch of unpredictable losers! They usually miss their goals, after all, and they practically always lose the contest they enter! Now someone sidles up to the disappointed project man and says that he’s a different breed. “When I say I’m going to do something, I do it” is the line. So our runner says, see that 100 meter track over there? I’m going to go up to it within the next 15 minutes, and I’m going to tell you when I’m starting. You time me. I’m going to cross the finish line with 60 seconds. And I’m going to try to exceed your expectations. Naturally, the project man is impressed. This is what he’s been looking for, a winner who does what he says he’s going to do. Ten minutes later, the guy goes to the starting line, announces his start, and 55 seconds or so later, crosses the finish line, ahead of schedule! Wow! He started ahead of time and took even less time than he told me – this guy is a star! If you think about projects you’ve been involved with, you will probably realize that, after the project gets going, everything is about comparing actual results to the expected results – you’re a hero if you’re early and a jerk if you’re late. Once the expectations, however unreasonable or silly, have been accepted, those expectations are the gold standard. What if a particular person or group consistently comes in early? In almost every project I’ve been involved with, this is not an occasion for happiness and celebration; it’s an occasion for keeping it quiet and possibly for consternation. The discussion that pretty much everyone involved wants to avoid is the reasonableness of the expectations themselves. This attitude makes it clear: if you’re on a months-long project and you come in a week early, you look good; if you finish in half the time, you raise the uncomfortable question of your estimating and planning skills. If that’s the rule, what does any sensible, non-self-destructive person do? He sets expectations that are as expensive and as long as he can possibly get away with, and then manages the results, slowing things down when necessary, to come in a little on the early side. Do you want someone who is too stupid to realize that this is how things work to manage your project? If you have someone smart enough to run it this way, why exactly is playing the “meet your expectations” game to your advantage? By focusing on expectation setting and meeting, the project planning approach to software becomes a con game that depends on the fact that people who are paying for the project are completely ignorant of the work that needs to get done, could never do it themselves, and therefore trustingly turn over their pocketbooks to “experts” who “know what they’re doing.” The “good” ones do indeed know what they’re doing – they know how to get you focused on the reliability of the results instead of the reasonableness of the work plan, and thus make you actually happy to have people “running” 60 second 100 meter races and pay them bonuses for doing exceptionally well (compared to your 70 second expectation), when there are lots of people out there who can do it in well under 15 seconds, but who are unable or unwilling to play the expectations game.
23
Building Better Software Better: Software Project Management
First time vs. repeats I can’t tell how many times I have heard people talk about “software factories” that turn out software reliably, predictably, and on-time. It seems painful, pedagogical and obvious to point it out, but there is an irreconcilable difference between real factories and fantasy-land software factories. Real factories have the job of taking in raw materials and components and building/assembling them into many copies of a finished product. Before the first copy rolls off the assembly line, working versions of the product have already been built by engineers. After the prototypes have been built and approved, there is then a whole “manufacturing engineering” process that goes on to transform the original prototypes into easily manufacturable and maintainable versions of the product. Then and only then is an attempt made to “manufacture” (i.e., to create many copies, effectively, efficiently and consistently) the product. When the first one rolls off the assembly line, there may be oohs and ahhs, but none of the viewers are thinking “so that’s what it looks like” or “darn, I thought it was going to be blue on top.” They’re all (hopefully) going “good – exactly what I expected” or “phew, no glitches” or “so far, so good, now let’s test to make sure.” Real factories make repeat copies of a known thing. Success is defined by things like how close the copy is to the ideal model. Sounds obvious, doesn’t it? Software isn’t like that at all. People may like to think in terms of “requirements” going into the factory as the raw materials, designers, programmers and QA people as the software factory workers, and the various stages from high level design through integration and final test as the work stations in the factory, with software popping out the end, exactly embodying the requirements that went in the front. Like anyone else, I can see the analogy. But the fatal flaw in the analogy is that the software (whatever it is, a whole product or a couple changes to existing software) that you want from the factory has never existed before in any form; it is (almost by definition) being built for the first time. The point is, making something for the first time is really hard. Think about the process of designing a new car, typically a new design of an existing model. Even that incremental job takes car companies years and hundreds of millions to billions of dollars. But the “new” car is practically the same as the old one! It has basically the same parts, the same controls, mostly the same materials, etc. As anyone involved would tell you, making all of those minor changes is still a lot of work, and then even more work goes in to making it manufacturable. The good news (for car makers and car buyers) is that, once all this work is done, cars of acceptable quality and cost pop off of that assembly line at a good clip. Once the design work has been done, making the repeats is relatively easy. But that’s only compared to the design! Actually, manufacturing consistent, high-quality copies of a physical thing has enormous challenges all of its own – whole generations of thinking have gone into this problem. From it, we’ve gotten statistical quality control, theories of continuous improvement, etc. All good stuff, in its way! But to the extent that it’s applicable to software, it’s applicable to making copies of the software, which is for software of what a factory is for cars. Making the master copy of the software is not like what a factory does – it’s like what the car designers do when they’re figuring out how to design the new model. Every time you build a piece of software, you’re building it for the first time! You’ve never built it before!
24
Building Better Software Better: Software Project Management Unfortunately, this little detail is not something that novice programmers get wrong, but authorities in the field soon correct. Authorities in the field of software actually have created standards for how best to build software (for the first time), explicitly modeled on theory and practice that was developed for how to make copies of physical things reliably! For example, one of the most widely followed theories is the Capability Maturity Model (CMM) for software4, which is supposed to lead an organization into creating software dependably, repeatably, and even can get better and better at doing so. Maybe all those years devoted to writing working code has addled my poor brain, and I’m missing something really basic here. Is it really possible that our most-publicized efforts to enhance software quality and productivity are based on simple-minded translations of methods originally devised for and applicable to factories making copies of physical things? Software, as I hope we all know: (1) isn’t physical; (2) making copies is trivial; (3) what we do when we create software is make the “master copy.” If anything, the factory analogy would be applicable to taking the design specifications of the software (the source code) and turning into an executable (the object or executable); this is the analogy of when the experts in the engineering design machine shop take the designs and hand-build a working model of the design. This is at least hard! The project management approach is designed (and even appropriate for) repeating a known process. This is a flaw so gross, so egregious, that it alone is enough to put the stake in the heart of the project management approach to software! That’s why there’s all this emphasis on documenting the process, making it predictable and repeatable, etc. And that is exactly why it is completely, utterly and totally inapplicable to software development, which by definition is building (designing) software you don’t already have.
The difficulty of doing new things Huge amounts of effort have gone in to automation, and that effort is mostly about performing some task over and over again. The task performed over and over again is done either for its own sake (for example, a traffic signaling system) or as part of making copies of some object. It turns out that creating accurate copies of a thing, consistently and reliably, is not such an easy task. Think how much more difficult it is, then, to create something for the first time. Let’s first think about something artistic, like writing or painting. The “tool box” is pretty simple in either case; in the case of writing, it’s the mechanism of writing and the available words and grammar of the language in which you’re writing. There are a finite and countable number of words; they’re even listed in a dictionary. There are rules that allow you determine whether any particular sequence of such words “makes sense.” Both the dictionary and the grammar evolve, very slowly, over time. A writer rarely starts with a truly “blank” sheet of paper. A writer is always trying to produce something in some genre, whether it’s a journalist writing a story about a press For example, see: http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr24.93.pdf. You can read about how CMM was explicitly derived from the excellent work of Deming and later maturity models, and how, without skipping a beat, work that was based on and relevant to making copies of physical things was translated to work making originals of digital (not really physical) things. 4
25
Building Better Software Better: Software Project Management conference, a magazine author writing a profile, or someone trying to write a short story. If you think about the number of pieces written covering high school football games, murder mysteries, or puff pieces about actresses, you realize that each writer is actually trying to produce a variant of a piece that has already been written countless times. The variation has to be enough so that the details of the particular situation are reflected and no accusations of plagiarism can be leveled, but minor enough so that the piece clearly fits in with others of its kind. In spite of the number of pieces that are written, the number of people capable of writing acceptable pieces is actually very small. This is in spite of the fact that the standards are actually very loose. There are huge number of acceptable “answers” for any particular writing assignment, which perhaps explains why we’re only willing to pay a small amount for a large collection of them in a newspaper or magazine, why most of them are so forgettable, and why anything approaching “literature” is so rare and valuable. The case of software is remarkably similar in many ways. The dictionary and grammar are tiny by comparison, but the concept of “data store” (into which values can be placed and retrieved) is added. Because the purpose of software is to process data, the criteria for judging whether a program produces correct answers are strict, and the grammar is completely unforgiving. Just as in natural language writing, there are genres. It takes a great deal of time and effort to reach proficiency in any one genre; and the fact that you are good at one doesn’t necessarily mean that you are good at another. Other creative, “first time” endeavors are similar in many ways. Moving from the artistic realm, making a scientific advance or creating a patentable invention are also remarkably similar. In no case is anything done completely “out of the blue.” There are long years of training and specialization. You become immersed in a genre. While you know that what you are doing is quite new and lots of people would be glad to have such an achievement under their belts, from the outside it is easy to say, “oh, it’s just another one of X.” So of the sequence of experiments that first established that there are specific opiate receptors on certain human cells, from a certain point of view it’s just another bunch of cells and liquids and measurements and little machines, running tests and finally getting all excited when one set of numbers is different than another set of numbers. And then they do the same thing all over again with a different set of stuff taken from different places, and somebody wins a Lasker Prize! Big deal, right? Shouldn’t we be able to apply project management principles here, clean things up and get some predictability? I mean, it’s pretty outrageous, don’t you think, that these cowboys just keep demanding more and more money and then have the nerve to tell us that not only don’t they know when they’re going to be able to cure AIDS, they don’t even know whether they’ll be able to do it!! What I’m getting at here is, why don’t the people who think that software can be produced on command if you have good project management, think that you can produce good literature or science or inventions with the same regimen? Actually, some of them do! But somehow, the rascals soon get chased out in those other fields, where common sense seems to spend a higher fraction of time in control than it does in software. The idiot’s view of “it’s just another set of tables and screens” seems to prevail more often, where no one would dare say “it’s just another set of notes” or “it’s just another sequence of words.” Leaving out for the moment the IQ of the people making such comments and being 26
Building Better Software Better: Software Project Management sympathetic (for a change), it must be admitted that no one has a clue what the programs are really doing or even what they are, while most people at least have the experience of reading words and listening to music.
Requirements and their spawn One of the biggest laughs I get when I look at nice, professionally done project plans is when boxes on the big diagram say things like “gather requirements,” “validate requirements,” “establish requirements traceability matrix,” “rank-order requirements,” and similar things. It’s kind of like the big “take vacation” project plan, with lots of details about where you’re going to be and what you’re going to do there, and a few boxes at the beginning: “determine where we’re going,” “determine how long we’re going to stay,” and “validate vacation goals.” My mother lives about half an hour from me, and her house is on a little lake, which is lovely in the summer. There are also nice places to stay on the southern tip of New Jersey, Cape May, and there are some very attractive beach areas just south of Chennai, India. Do you think the choice of vacation “requirements” has a little affect on my “take vacation” project plan? Of course, anyone would admit that all “take vacation” project plans have some elements in common, and that taking advantage of the commonalities can be useful. But suppose I’m choosing between (1) driving over to my mom’s house and (2) flying to Argentina during my summer to do some skiing during their winter. Once you’ve gotten past the requirements, how much do you suppose the bulk of the plans really have in common, beyond silly things like “sleep at night?” So, as anyone involved in project management knows, requirements are the key to everything. You simply can’t have a meaningful project plan unless that plan is based on a set of good, well-specified, detailed requirements. So let’s agree right away that project plans that include anything that has to do with determining requirements are completely bogus. Very simply, this is because everything that comes after the requirements is completely determined by the requirements. So if a project plan includes determining requirements, of what possible value is anything that comes after the requirements phase in the plan? Of course, such a plan can’t have any value. Nonetheless, you’ll see 12 month projects of which the first two are spent determining the requirements. What is this about? What people actually do during the initial two months? In my experience, what the planners do is a combination of determining user/market desires/needs, and making rough estimates of the degree of difficulty of various features and capabilities. Given that the total time and effort have already been roughly bounded, their job is to select the things that will get done among the larger set of what various people want to get done, and (above all) to make sure that the selected items can be built with the allotted time and resources. That way, everyone agrees to feel good about the project – it met its requirements, and it came in on time and on budget! In my opinion, this is a real emperor-has-no-clothes scenario. The “planning” effort is all about reducing what gets done, and making sure that, even if people screw up, are lazy, quarrelsome and minimally productive, what little ends up getting committed to does, indeed, get delivered. Instead of the main focus being “how can we get it done; how can we do more, and do it faster,” the main focus in a project planning environment is “severely control and prune our goals, so that above all, we are sure to deliver what little we got away 27
Building Better Software Better: Software Project Management with committing to deliver.” Instead of our satisfaction being tied to how much we accomplish and how productive we were, we are happiest when we deliver our commitments “on time and on budget,” totally ignoring how laughably modest the goals were, and paying no attention to the inefficient and minimally productive efforts that went into accomplishing them! I know this is hard to imagine, but it gets even worse when people get serious about the requirements process. Some project management-run efforts end up being not too bad, because outside of the “management” context, a bunch of people somehow discover what the business actually needs, care about it, and go out of their way to try to deliver what is really needed. In other words, the project gets “out of control” with all sorts of unplanned and unrequired activities taking place. Well, suffice it to say, a competent project manager would never let that happen! So what “good” project managers who “exercise control” do is impose a regime of “requirements traceability.” What this means is that each activity in the post-requirements portion of the project, during which the vast bulk of the work typically takes place, is examined to see which requirement it is fulfilling. This is done both to make sure that the requirements are being met, and to assure that only work directly coming out of the requirements is being performed – there’s far too much to do, after all, it would be foolish to miss our deadlines because we spent time on non-essential tasks! Better you should go home and rest or refresh yourself with some non-work activity than for you to exhaust yourself performing something which isn’t required – yes, I do understand that we have some anecdotal evidence that some individual users would really like certain things to be included, but part of being a successful professional is learning when the right thing to do is to say “no.” It’s a matter of professionalism, discipline, commitment to the success of the organization, and resisting the temptation to be like a kid and play hero to a bunch of unimportant people. Do you want to have a future in this organization, or do you want to jeopardize the success of everyone involved here by going off-sides and getting some ego strokes from some people who are, after all, losers? I wonder if the serpent used this kind of smooth language and quiet tones when he talked with Eve about that apple?
The consequences of change In the world of construction, you don’t normally have to build bridges between islands that shift, irregularly and unpredictably, 10 meters a day; the islands don’t have volcanoes that erupt, abruptly and prodigiously and often; there aren’t landmines in the ground and random torpedoes coursing through the sea; the substances that you use aren’t different than the ones you used last year, and untried and unproven at that; the machines you use aren’t constantly coming out with radical new versions, with few parts in common with their predecessors, and with capabilities so great that you’ll be behind the competition if you fail to use them, but are so complex and abstruse that only a few of the most talented and dedicated can possibly manage them; the systems you connect with aren’t replaced frequently with new ones that don’t work the way the old ones did, when they work at all; you don’t have to cope with customers who demand smooth connections to existing, completely incompatible construction, but who also demand the aggregate of the best new things they’ve heard about, and who change their minds as they see the construction in progress and hear yet new things from their friends; and at the same time a completely unsympathetic and unforgiving environment that expects you to build your project as thought none of this were happening and that you had perfect knowledge and perfect 28
Building Better Software Better: Software Project Management foresight and a staff of talented people for whom none of the above presented any challenges whatsoever. And yet, this constant, unrelenting, unpredictable and radical change of most relevant variables is the norm in software. The components and tools available for any project are undergoing constant change; the components and tools you end up selecting for any project have multiple levels of uncertainty (there will be bugs, problems and incompatibilities with the components and tools you choose), and will change with patches; the people available for a project are under a wide variety of changing pressures; the people directing the project undergo change; the business circumstances that drive a project evolve constantly; the priorities and pressures of the business environment change constantly; internal and external politics intersect with objective circumstances to create sudden gales from new directions. This unrelenting change makes it amazing to me that any software project gets to completion successfully. In my experience, “successful completion” takes place mostly because enough powerful people involved decide that it has; the idea that any objective goal has been reached is completely incidental. Does “project management” help this situation? What “project management” does is say, real life is endlessly variable, messy and complex. I can’t deal with it. Let’s pick some people and go into this special enclosed space. While we’re in this space, we’re going to take on roles that are only vaguely related to real life. We’re going to agree on how we act, individually and as a group. We’re going to agree who goes where, what they do, who says what to whom, when the action starts and when it finishes. Meanwhile, messy, constantly changing real life is taking place outside the special enclosed space. But the rules say it stays out there. In here, we have our own rules, and things change only when we say. If we say it’s a year ago, it’s a year ago. Finally, we reach the end, declare success, and leave the enclosed space, happy and proud of what we’ve accomplished, and eager to tell anyone who will listen how great it was. What have I just described: a project team defining and executing a project with great vigor and discipline? Or I have I described a theater company going into the theater and acting out a play? When you get right down to it, what’s the difference – they’re both a group of people withdrawing from reality, defining their own rules, ignoring the outside world, and declaring their own success while the world marches on.
The “planning” mind-set and the attitude towards variation There are two kinds of variation we’re going to pay attention to here: 1. Variations in the nature and difficulty of the tasks that need to get done in order to meet the project’s goals, commonly expressed as “oh, foo-ey, where did that come from,” or “gosh darn, I didn’t think about that.” These are variations that affect the project costs, tending to increase them. 2. Variations (typically understood as additions) in the goals and deliverables of the project, commonly expressed by people “inside” the project as “scope creep.” These are variations that affect the project goals (deliverables), tending to increase them. The theory of project management is all about controlling variations and their effects; a good deal of project management practice in general, and in software in particular, is about 29
Building Better Software Better: Software Project Management the early identification of potential variations, and limiting them (in the case of cost changes) or eliminating them (in the case of goal changes). Reducing variation to a minimum is supposed to be one of the key achievements of a “mature” software development organization, according to the widely-accepted CMM (Capability Maturity Model). People who are as good as it gets in project management talk about identifying the highest-risk elements of projects and executing them first, in order to reduce the risk, i.e. the change, as much as possible. In any case, since project management’s highest value is predictability, setting expectations and then meeting them, obviously anticipating everything that might happen and accounting for it, and then letting nothing happen that wasn’t anticipated, is a great deal of the art of doing it “well.” Of course, this is pernicious and destructive, but you can judge for yourself from the details.
Unanticipated obstacles: “Gosh darn” variations Every software project beyond the trivial encounters things that weren’t anticipated at the beginning. In addition, the estimates given by the programmers most often turn out to be unrealistic. For as long as I’ve been around software, the rule of thumb has been to add up all the estimates provided by programmers and double or triple them to get something that is vaguely realistic. The larger the organization, the more people in the hierarchy there are who think this way. So the manager may sum up the estimates he’s received and double them before passing them to his director. The director will sum up his managers’ estimates and triple them before giving them to his VP – there’s no way any project is going to come in later than scheduled under his watch. And sure enough, when the VP talks about the project at the executive staff meeting, the times get padded. I’ve often wondered, with all the padding on top of doubling and tripling that goes on (and some people have the nerve to throw around terms like computer “engineering” and even computer “science” – hah!), how any project can possibly come in late? As the project goes on, people encounter the time-sinks that cause the “add up all the programmers’ estimates, triple the result and then pad it” estimate to become, in retrospect, outrageously optimistic. When each time-sink is encountered, it is experienced as a “gosh darn.” But it’s worth distinguishing between the “whoa, where’d that come from” type and the “gosh darn, somebody [other than me] should of thought of that one” type. There is also the mysterious, always-present but never-seen elephant in the room, the “this doesn’t count” variation.
Whoa, where’d that come from? This is the kind of variation that project planning is supposed to find and flush. Pretty much everyone is aware of it. The wise people who preach project management like to say things like “the sooner you start, the later you finish.” The idea is that more time spent planning results in greater reduction of unknowns, and therefore fewer unanticipated tasks to be handled in the course of the “main” part of the project. There should be no “surprises.” The ideally-planned project is one that puts its participants to sleep with boredom, while somehow maintaining high morale. So the “where’d that come from” reaction is evidence of planning failure.
30
Building Better Software Better: Software Project Management These things happen. They’re not great. But I suggest that “gotcha’s” of this kind have little to do with the way projects actually go. In fact, since they are so widely understood and yet so relatively trivial, I suggest that the vast majority of gotcha’s are just red herrings – they’re the equivalent of the local underclass doing a little shoplifting while certain powerful CEO’s practice larceny on a grand scale. Shoplifting, which certainly exists, causes us to focus on certain environments and targets, leading us to completely ignore the places where really big money walks out the door.
One of you should of thought of that one There’s another kind of gotcha that happens to software projects. When someone encounters it, there’s usually no personal embarrassment, because the person who uncovers the problem usually feels it wasn’t his responsibility; in fact, the person often feels injured, because he’s worked hard and well, and now his progress is threatened by this unanticipated obstacle that “someone” should have caught. This can be understood by the fact that software projects are carried out by a team of people with varying skills and specialties. The skills and specialties are not measured on a single scale, like IQ, but in multiple greatly different categories, like a basketball team. While the team has a single goal (score points and prevent the other guys from scoring), the specialties among the five people on a team vary dramatically. At least one person should be able to dribble and pass. When certain big players attempt to dribble, a groan emerges from the fans, what’s he doing, trying to dribble? A person who is great at rebounding but little else is nonetheless a valuable player. Similarly, in programming, the work is usually divided along functional lines, so one person is good at user interfaces, another at communications, another a DBMS specialist, and so on. Just as the dribbler spends little time learning to box out under the boards, so do the UI person’s eyes glaze over when the DBMS person yaps on about what to turn into a stored procedure; it’s not part of his world, why should he care? It’s just a waste of neurons. Just as those basketball specialists all end up touching the ball at some point in the play, there are things, we call it “data,” that flow among otherwise non-conversant software specialties. Users see information on their screens, they enter some data, press a button, and the data flows from browser to web server to applet to application server, to a web services call, to a remote server (traversing multiple networking boxes along the way), through more application and operating services layers, to a DBMS, spawning a trigger and some replication functions, and all the way back again, with copies and side effects going in various directions. You can see the software “coach” pacing the sidelines nervously as the data-pass is started by the user pressing the button; he screams out “FIVE!!” hoping all his players near and far can hear the “play” he called above the din of the crowd. At some point, a player makes a lead bounce pass to a teammate who turns in a different direction at the wrong moment, and isn’t there when the ball is. In basketball, a turn-over, in software, a classic “somebody should have thought of that one” design flaw. In software, there are more ways for things to go wrong than there are in basketball. At least basketball is based on there being a single ball that always has the same size and bounce. In software, there are data packets being passed all the time among many players, and the packets range from little goodies the size of M&M’s to trailer trucks. The sender and receiver have to agree exactly on the size, weight, shape and handling characteristics of each unit being transferred. 31
Building Better Software Better: Software Project Management If the receiver expects a raisin, you’d better send a raisin, a grape won’t do. If you send a carton of ice cream and the receiver expects something the size, weight and shape of the ice cream carton, but doesn’t know it requires refrigeration, the results in the world of software will be at least as bad as the sticky mess you know you’ll have after a short while in the world of ice cream. When there’s a turn-over in basketball, who’s at fault? The passer thinks “he should-a been there,” while the receiver thinks “I was here and the ball wasn’t, what am I supposed to have, ten-foot arms?” In software, the players usually feel similarly innocent and aggrieved. Each sits firmly in his area of responsibility thinking “I sent what I was supposed to send” and “I didn’t get what I was supposed to get.” But software is actually less like a clean court with a single ball than a third-world traffic square with a dozen inlets, jammed with trucks, cars, bicycles, animals and pedestrians – when something goes wrong, it goes horribly wrong, and the domino effect can reverberate for hours. “Planning” is supposed to orchestrate this tangle so perfectly that nothing goes wrong. But in practice, things go wrong all the time. If it were just the problems and fixing them, it would be bad enough. But you know, if we were more of a team, and communicated more, this kind of thing wouldn’t happen…
If only X had… Those of you who have seen basketball or football games are probably familiar with the receiver who is down court or down field, waving his arms, ready to receive the pass; if only the ball could be thrown to the receiver, the open guy could then make the score. Meanwhile, he’s just standing around. Much is made of this phenomenon in project management circles. Pedro Martinez is a great pitcher, but anyone should have seen that he was getting tired, and leaving him in for that extra inning cost us the game. The coach deserved to lose his job. Or there you had Jerry Rice, the greatest receiver in the game, breaking clear of the defender and tearing down field. Don’t you think it would be a good idea to get him the ball? All of this is easy to see from the stands, and not so easy to pull off on the field of software development. From my perspective, I’m a role player in a carefully worked-out set of projects, and when something I need isn’t there, there just isn’t much I can do about it. I can wait around, jump up and down waving my arms (probably not a great idea in an office and it gets tiring after awhile anyway) or try to do something useful. It’s probably not a great idea to get too vocal about the situation, because the other guy has his problems too – I hear one of his key players quit abruptly, and the new guy is pretty well qualified but taking some time to come up to speed.
Where’d the day go? I’m supposed to go out with some of the team for drinks after work, because some people think that if we got to know each other better in a casual setting, things would be smoother at work. I understand the logic, but I’m really tired and just want to go home. I’m not sure why I’m so tired -- it’s not as though I’ve gotten much done today. When I got in, there were a pile of e-mails that took forever to go through. A bunch of them were alerts from the new source management system we’re trying out. Now I get told when someone 32
Building Better Software Better: Software Project Management has checked in something that might affect me. It seemed like a good idea after we spent all those hours for all those days last week hashing over last month’s schedule slips. So many of them seemed to come from other people checking in code with surprises we didn’t find until we did integration debug. But I’m not sure this is saving me much time, and Randy had to take days from his schedule to implement the alert system, not to mention the help he needed to pull in from other groups. Well, I’m not going to say anything, he worked so hard, and he’s bummed that all his tasks will be late, and then all the people who can’t start theirs until they get his – I don’t want to think about it. Then we had the code walk-through’s. I know we’ve been having trouble with nonconforming code, and it is true that when we only have four specialties represented in the review that lots of important stuff seems to slip by, but when you’ve got a dozen people in a room, and you fiddle for twenty minutes to get the projector to work, and then you pass around copies of the code, just getting started seems like an achievement. Then it’s time for the database guy to start raving! The next process review meeting I’m really going to stand up and make my voice heard. Each database guy they send seems to have a different set of hot buttons, and all the hot buttons seem to amount to that we want to change their %^* precious schema and we don’t know what we’re doing. Well, hello! If I wanted to be in databases, I’d be in databases – of course I don’t know what I’m doing, that’s why you’re there, dingbat, just give me the fields I need and get out of my way! And take your incessant blathering about normalization or ab-normalization or whatever and cram it. Then when I finally get my field, it’s the wrong type and it’s somewhere weird and I need a new query. So I put it in. Then the next &*()&* database guy comes in and starts in on how this “incessant multiplication of queries” is going to kill performance, when it was the last guy who made me re-write sixteen screens and put the new queries in! This can’t be a good way to run the place, so I’m definitely reserving an hour’s slot to discuss it at the next “review the process” meeting. At last, we could leave and it was time for lunch. Management has been griping about all the long, off-campus lunches we’ve been having, but this is Mindy’s going-away lunch, and I’m not going to stand her up. We’ll be back by 2 o’clock for sure – 2:30 latest. I knew I had to be back for my mentoring session, anyway. I think the project guys are right on this one, it really does help make things go smoother when the new people have some real, regular, scheduled time to spend getting their questions answered about how things work and how to get things done. I’m glad to spend some time helping somebody out, and I’m sure it makes things go faster in the end. Then one of the business team guys came along and begged me to join this meeting they had going with these French guys who were supposedly doing something like us. It sounded reasonable; we’re all working for the same company, after all. It took awhile, but finally we all realized that while there were lots of similarities, we had enough different requirements that we still needed to keep our projects separate. But we agreed to set up meetings every couple weeks or so to explore the opportunities for code sharing. They’re behind, just like we are, and we figure that what with all this object stuff we’re doing, if we have two groups doing practically the same thing, there’s bound to be some object re-use we’ll get out of it, so that one of us can do a little less work, and maybe pull in our schedules by a bit. So it’s worth the extra work. I feel good about that one! But then I looked up at the clock and it was 4:30 already, time to start to think about packing up and getting out of here. I’m 33
Building Better Software Better: Software Project Management definitely not going to check my voice mail. I just don’t have the oomph at this point, and the e-mail, forget it. But there was that thing I wanted to pick up. I wonder if it’s on Amazon and whether they have any reviews or something? Anyway, maybe I’ll just stop by and have one little drink, just to be social, after I check this out…
It’s the system, stupid! Project management applied to software is like pre-Copernican astronomy. It just gets more and more complicated and implausible. As the failures pile up, ever-more sophisticated mechanisms are devised to fix things up. We have equants, eccentrics and epicycles, all to create an illusory universe of theoretically-demanded uniform motion, when anyone can see that planets just plain go backwards against the background of the stars. The break-through was not incremental, it was conceptual. In fact, Copernicus’s heliocentric solar system didn’t even lead to better predictions than Ptolemy’s at the beginning; but it was simpler, cleaner and much closer to reality, and helped people think about the problem in more fruitful ways. When Kepler applied Tycho Brahe’s more accurate observations, particularly of the motions of Mars, the clean heliocentric concept allowed him to devise new but still simple formulas that made predictions of unprecedented accuracy. Today’s software project management methodology is a system that is vastly more complicated than Ptolemy’s epicycles, and every bit as detached from reality. While Ptolemy’s system at least yielded accurate predictions, software project management has a solid track record of proven disaster. Whenever there’s a failure, instead of questioning the system, the prescription is – more project management! Just as the great advance in astronomy was based on a change of perspective, putting the Sun in the center of the system instead of the earth, so will software get better only when we focus on the software itself, leaving behind the chimera of “software factories” and all the mechanical fantasies associated with them.
Additions to the goals: “Scope creep” variations Given how hard it is to get something done (see other sections), you can imagine that the absolute worst thing that can happen in a project is additions or changes to the goals. The farther you are into the project, the more planning and coordination that has been done, the more bringing up the remote possibility of a goal or requirement change makes people react to you as though you’d proposed tying up Nell and placing her on the tracks in front of an express train. People who get into project management tend to be fairly even-keeled types. They’re generally into pouring oil onto the troubled waters (or at least that was what they were into before that image conjured up environmentally evil acts committed by heartless global energy corporations). But if there is anything that will cause a true-blue project management type to hyperventilate, it’s the threat of the dreaded, the beyond-evil “scope creep.” Given how awful scope creep is, it’s truly wonderful that business and technical situations change so rarely, and that we understand them so well, that we’re so rarely even tempted to make changes of this kind…
34
Building Better Software Better: Software Project Management
The focus on form and process over content Now it’s completely understandable that the “buyer” of a project would want to know how long it’s going to take and how much it’s going to cost before committing to going ahead. Suppose you talked with someone about building an addition to your house. Naturally, when things are wrapping up, you ask what you both know are the key questions: what’s it going to cost; when can you start; and when will you be out of here? The builder pushes for more time, more money and giving you less; you push for starting when it’s convenient for you, finishing as soon as possible, paying as little as you can while getting everything you want. If you end up going ahead, you agree on the time and money, and measure the builder by his ability to deliver the promised results at the promised time. You’ll do things “by the book,” of course – you’ll have a contract, milestones, payments tied to progress, the whole nine yards. That’s how projects work; everyone knows it, and what is this guy doing complaining about it? Well, I’d like to point out that “this guy” isn’t complaining about the project management approach to home additions, where it is not only accepted practice but delivers pretty good results, but about its applicability to software. But before we get to software, let’s finish with the addition. Suppose the builder isn’t very good, has people suddenly quit on him, is pressed for cash, is unscrupulous, or any number of other unpleasant characteristics. What will happen? He will start the job right away so he can get your initial payments and keep you hooked. After your place is a mess and you are quite committed, he disappears and tries to put out other fires. Meanwhile, you’re living in chaos for no good reason. Some people show up every once in a while and it’s pretty obvious that they’re lazy or drunks or bozos or a combination. They re-arrange the chaos, add some more to it and leave. Leaving out lots of drama in the middle, the reputable builders who actually agree to visit your site shake their heads, cluck their tongues and basically blow out of there as quickly as they can. You’re stuck. Could this, possibly, by any long chance, remind you of a software project you’ve experienced? With home additions, we all know that the nightmare situation I just mentioned has nothing to do with whether or not good project management practices were followed. It has to do with the content of the work, which is closely related to the people doing the work. But the interesting thing about software is that somehow we nearly always imagine that “management” or “project management” is at the root of the issue – that good management would have prevented the problem from taking place, or at least quickly caught and fixed it. We all know that, with home additions, everything depends on whether you actually end up with a team of people who are available to work when you need them, have the right skills and experience, and want to work hard individually and as a group to get the job done. Project management can certainly help such a group by arranging that the right tools and materials arrive in a timely manner, and so on. But can good project management get good results from a bunch of schlubs who are no good, don’t show up and have bad attitudes when they do? Of course not. It’s all about who does the work and how they do it. It’s the content, not the process. Bad process can slow down good content, but without good content, you’re doomed. But somehow, when the subject is software, we leave our heads at the door. Yeah, we hired a bunch of jerks. They couldn’t get the job done on a good day. So we hired more of 35
Building Better Software Better: Software Project Management them and gave them more time. But morale was bad, and, well, we missed our targets. Sorry. But you know, if we had better project management, we could have avoided these problems. Sure. And I have a bridge you may be interested in…
The difficulty and complexity of computers in general Obviously, “project management” is insane in general, and drives me nuts in particular. But the fact that computer programming is difficult work and simply beyond the ability of most people, even most people peripherally associated with it, makes a bad situation worse, and is one of the factors that make project management concepts disruptive instead of helpful for software. Think about what people have to go through to learn a new human language, like Mandarin, Telugu or American English. Children, of course, learn their first human language quite early, and they appear to be able to learn other languages without much visible effort, given sufficient exposure. The fact to be noted in all cases of humans learning human languages is that human language is firmly rooted in human experience, internal and external. Languages provide us a way to speak and write about what we see, hear, feel, think and do. When we try to learn a new language as an adult, it’s considerably more difficult, and without complete immersion, most people never develop fluency. This difficulty is in spite of some considerable advantages: the new language is about all the same things as the language we already know, i.e. it’s all about things we experience, think, do and feel. With a few exceptions, there are fairly exact correlates between the new language and the one we know; this is why dual-language dictionaries are possible. Even though we’re just learning a new vocabulary and a somewhat different grammar, we find it difficult. The key thing about computers (this may sound obvious at first blush, and it is, but we usually don’t think about it or what it means) is that they aren’t humans! Duh, I think I knew that, thanks a lot. The other key thing about computers … get ready for another gee-whiz, duh moment … is that they are programmable! The programming is expressed in – a language! – surprise, surprise, and that language is completely rooted in and about life as a computer! There are eerie similarities between human and computer languages, partly because all computers and all computer languages have been invented by humans, but the most important things are: human languages “speak” from “within” human experience; computer languages “speak” from “within” computer “experience;” while there are clear analogies between the two, any two human languages are vastly more similar to each other than any human language is to any computer language. There are actually multiple worlds of experience inside computers. This is reflected in the fact that there are languages and protocols that are specialized to one environment or another (e.g., networking, applications, databases); it is unusual for a human to be truly proficient in more than one of these environments. If a human knows more than one computer language, it is most likely that all the languages are of the same “type.” Most people who become truly proficient in one of these languages completely internalize the environment and the experience. You can readily hear this in the way they speak – they tend to take on the perspective of the computer itself, saying things like “I read the disk, calculated the return values using the wrong offsets and blew myself up!”
36
Building Better Software Better: Software Project Management There are people who become completely immersed in the world of computers. They see great vistas containing tremendous variety and detail. They visualize and recall things in great detail from all the time they have spent “inside” the computer. They can enthusiastically describe similarities and differences the way a world traveler would compare Vancouver, Hong Kong and Cape Town. They’ll actually have a broader perspective than many travelers in physical space, because you’ll hear from them how each striking site came to be, how it could have been different, and how the traveler would have preferred to have created or evolved it. So: humans try to learn computer languages as adults; when humans try to learn human languages as adults, they rarely attain proficiency; attaining proficiency usually requires immersion; immersion in computer languages is challenging; learning computer languages is qualitatively more difficult than learning human languages, because the computer “point of experience” is distant from any human one; becoming a true computer master requires attaining proficiency in multiple “types” of computer environments. Given all this, is it surprising that computer mastery is a rare skill, and, regardless of their degrees or certification, that few people truly attain it? Even those few people who run the gauntlet of computer education, which usually makes them as qualified in computing as, say, high school kids who have a year or two of a foreign language, those few people get into the work force, try doing actual, real-life programming for a couple of years, and then decide that there’s got to be an easier way of earning a living. How about I use my people skills and become a manager? I know enough now that I could get paid more to write requirements for projects that other people would have to implement. The fact is, we don’t have masters of programming doing our software work – if we did, we wouldn’t be talking about the idiocy of project management. We have waves of young kids coming out of barely-relevant educational programs, trying to program for a year or two, and then sliding away from the actual work. As a result, whatever might be said about it, software project management is all about people who rarely have any computer literacy getting people who have marginal computer literacy to create extensive programs in one or more computer languages at which they are novice speakers, at best. To the extent that the people who don’t actually program do anything at all, it’s usually “contributing” the longobsolete knowledge they accumulated many years ago, during their brief stint of actual (more or less) programming. Because of the inherent difficulty of really learning computers, all too many software projects amount to groups of people who see dimly, supported by people who have been blind from birth, and led by people who sometimes remember what it used to be like to see faintly, but who are now mostly into exercising power and authority, and keeping the expectations of “management” so low that even their stumbling band of handicapped idiots can sometimes, as though by chance, get the job kinda sorta done.
Mental and physical work Whenever we think about something that we understand, like construction and unlike software, the planning activities are primarily mental, while the execution activities are primarily physical. The planning is usually done by the people wearing nice clothes, who are 37
Building Better Software Better: Software Project Management generally smart and educated. The execution is done by, well, construction or factory workers, who wear clothes meant to get dirty and who have enough intelligence and education to get their very physical job done, not to put too fine a point on it. Software is different. The planning activities are still primarily mental. There are differences between the people who do software planning and those who actually do the work, but they are more subtle – both types are highly educated people who work primarily in offices. While some programmers become planners, transitions in the opposite direction are rare. Planners tend to be more socially oriented and interested in process and relationships, while programmers tend to be more technically oriented and interested in the internals of program construction. But these differences are subtle, compared to the gap between construction workers and planners. There are many implications of the fact that the planning phase of projects is always primarily mental, while software stands nearly alone among project types in that the execution phase is also primarily mental. The problems come because most project management concepts are derived from mental planning, but physical execution. One way of starting to understand what this means is the experience of trying to explain to someone who doesn’t have a clue what you want them to do, and you say something like “move aside, I’ll do it myself – it’s quicker to do it than to explain how to do it.” Suppose we’re talking about something physical, like building a three foot high, hundred foot long stone wall. Would you be tempted to make such a “get out of my way” statement in that context? I don’t think so. No matter how skilled you are, building that wall is going to take a lot of time and effort. And you’d darn well better plan it out right, because “adjusting” it after it’s laid just isn’t going to happen. When what we’re building is mental, the situation is quite different. The hard work is always getting it right in your head. There are huge differences between people here. Turning the work in your head to “reality” in the computer is relatively quick and easy. Furthermore, the distance from “planning” to “doing” is minimal. Someone who’s going to do it, and do it well, had better have all the planning work in his head. Another key point is that, when the “doing” is physical, yes there are some physical skills involved, but nearly every planner could fairly easily do the “doing,” while most of the “doers” could never acquire the mental skills and discipline to be planners. Oddly enough, the opposite is the case in software. Most planners never wrote code or wrote it for a little while and then schemed to get out of it, and in any case never embraced it and became really good at it. On the other hand, good programmers often consider the planning function to be trivial and annoying, something that ignorant people impose on them in an attempt to control them, when they really don’t and don’t have a clue (see the section on expectations). It’s actually quite laughable to see these mental midgets try to put together reasonable plans; it’s like watching a bunch of high school students playing at planning a nuclear power plant. Sometimes there is a useful relationship between planners and doers in the software world, just like sometimes there are novels written by teams, one of whom does more
38
Building Better Software Better: Software Project Management character development and plotting while the other does most of the dialog and actual writing. But, like in the world of writing, it’s rare. Once the writer gets the idea fleshed out in his mind, the natural thing is to write it – that’s what writers do! Writers look at what they’re writing as they write, and make all sorts of modifications and course corrections, some minor and some drastic, as they go along. A writer who does this isn’t a jerk; that’s just how writing works. One thing to understand about code is that each line or small group of lines represents a thought of some kind, expressed in an excruciatingly exact language. If a programmer spends time repeating these thoughts, or blocks of code, he’s making a serious error. There is always a way to abstract out the repeating blocks and refer to a single instance of the master block. If there are variations of the common blocks that are used, there are ways of stating the common parts a single time, and naming the variation, just as the conductor of an orchestra will ask for a passage to be repeated, “only this time, a bit faster, and I want a sharper entrance by the horns in measure 14.” Planning a piece of software is figuring out what it’s going to do. This is best done by people who can actually do it, who are thinking in terms of the software itself as they are thinking about what it’s going to do, just as the people who plan a stage show are visualizing it as they plan the various aspects of it. In stage show planning, when you’re done, you write down an exact description of how it’s going to go (the lighting, the singing and dancing, the costumes, etc.) and then the real work begins. In software, once you write down that exact description of any part of it, you can get the computer to run it and see what it looks like. In fact and in general, once you write down that exact description, you’re done! Let’s go over this again, because it’s important. If you’ve written exact instructions for a theater set, you have to get the space, get the materials, do the cutting, joining, assembling, painting, etc. If you’ve written exact instructions for a garden landscape, you have to work the soil, get the plants, etc. In any case where the realization of the plan is physical, the bulk of the work is the physical doing, which often require different skills than the planning. With software, once you’ve written exact instructions, which normally you can do easily once you’ve understood what’s needed, you’re all done. People can go off and run your program to their heart’s content. Normal project management is like a human brain, planning and overseeing tasks that the body is to perform. Normally, the brain takes a small amount of time and effort, while it’s the body that burns all the calories, takes the time, etc. In this context, it makes a huge amount of sense to think before you do; doing your work with foresight and care yields better results achieved with less effort, often by a large margin. Software is similar only in that you should think through your code before you type it in. Typing and clicking is a trivial amount of effort, goes quickly, and generally is the easy part – in fact, the typing and clicking amounts to little more than recording what you already have in your head. The brain part is the work. Project management was designed for, and is actually appropriate for, mental planning of physical work. Software is not physical. Software is itself mental in nature. The planning function and the “building” function, such as it is, merge into one another, so that, in software, the exact instructions that naturally result from the planning activity are the ultimate results you want.
39
Building Better Software Better: Software Project Management
The difference between physical things and digital things To understand why project management is just the wrong paradigm to apply to software development, it helpful to understand that not only is the substance of the work of any software project mental (and therefore, not physical), but it is that peculiar type of mental “thing” that we call “digital.” There are lots of things that are mental but not inherently digital. Dreams, for example, are clearly mental events, and not digital at all. There are also mental things that can be digitized, for example a JPEG version of a mental image. While many mental things are easily given digital form, only some things are, like software, inherently digital. One of the consequences of this is that software people use software to build software. While this seems like a nice, clear distinction, you begin to realize the power of it when you realize that originally, programmers used templates for flow-charting and coding sheets that were keypunched and then entered in to the computer, and they used “core dumps” for debugging. That’s now ancient history. As computers got more powerful and less expensive, programmers gradually wrote programs (we call them development tools) to help them write the programs they are actually supposed to produce. How many guys in the construction industry design and build backhoes? But in software, it’s natural for programmers to be drawn into tool-building, first of all, because they’re programs like any other, and second, because having powerful tools and making effective use of them is absolutely central to building software quickly and effectively. What is it that makes the “doing” part of physical projects hard, long and expensive, thus requiring the extreme measures of project management to reduce errors and increase predictability? The main factors are these: The “stuff” you work with may take time and effort to acquire, and costs money; therefore, you want to make sure you get exactly the right amount of the right stuff, apply it correctly and not waste any. In software, the stuff you work with is arrangements of bits; they take zero time and effort to “acquire,” they’re effectively free, the wrong bits can easily be transformed into the bits you want any time with little effort, and there’s no need to think about wasting them. Lots of physical doing involves taking time and effort to perform similar operations to similar objects, for example, building a brick wall. In software, any such repetition is eliminated at the design level by centralizing common elements, or minimized at the tool level, by automating any sequence of repetitive operations, however context-sensitive or complex. As my local building inspector said, on viewing the foundation of the garage that had been built a century ago but which violated current set-back requirements: “That sucker ain’t goin’ nowhere. Let her be.” The cost, time and trouble of changing physical things, once they’ve been built, can easily exceed that of doing them in the first place. The cost, time and trouble of changing already-built software can be huge if you have a band of drooling idiots stumbling through the code, and can be longer than you’d like if it was built by morons in the first place, but in any case, code is inherently modifiable, and tools enhanced by macros enable you to make amazing, extensive modifications effectively and efficiently. Everything about physical things costs money and takes time. The effort of muscle and machines is required to move it, shape it and place it. This effort takes the vast majority of the time and money spent on typical physical projects. In software, while the programmers take time and (in some cases, unreasonably) demand payment for their services, what they are doing is mentally conceiving of the work to be done, and getting that mental conception, whether it is code itself or modifications to existing code, into an exact format. Once the 40
Building Better Software Better: Software Project Management exact form of the conception is realized, the work is done. Either the code is written, or the existing code has been appropriately altered and augmented. It must be admitted that not all code is simple. There is something in software that corresponds to parts of physical endeavors. Some physical projects involve special devices or components that the people involved in the project don’t understand, but can apply if given good instructions; think of a fuel cell or a telephone switch. While lots of the “stuff” of physical projects is simple, like concrete or cloth, some things are mentally deep and intense. Similarly in software, most software is pretty simple stuff, moving things around, calling functions, “glue” code of one kind or another. But a tiny amount of software is algorithmic in nature. Usually, a small number of lines of code is involved, but each line is fraught with meaning and effect; tiny changes have huge implications. Obvious examples are sort routines and search algorithms, but there can be “high leverage” code nearly anywhere in a program. This is a little like building a skyscraper and failing to take the effect of the winds into account; the building may not fall down, but it sways so much that people are uncomfortable being on upper floors. The good news is that, although the skyscraper “ain’t goin’ nowhere,” software, being digital, can have its very infrastructure replaced and fix the problem. If you’re building a hydro-electric dam, the power-generating turbine is likely to be a huge piece of equipment that is built off-site, transported to the dam at great cost, and put in place with great care. Every step of the process had better be worked out in advance and done correctly; what a disaster it would be to have the thing tip and fall while being placed in position! I think such projects are admirable, and it’s a testimony to the skill and care of the people involved that truly awful things almost never happen. In the digital world of software problems, hundreds of thousands (or millions) of lines of code easily equals the turbine in semantic complexity. Software engineers can easily “transport and install” such a module into a project with a couple of keystrokes. If it’s wrongly “placed,” they can move the code with a couple more clicks. They can then cause “water” to pour through the huge, complex “turbine” with a few new lines of code. They can scale up the dam to a hundred streams of water pouring through a hundred turbines with a few more lines of code. With a little more code, they can make the number of turbines in action be proportional to the amount of water available for flowing. The mental, digital world of software is different than the world of physical objects, and different techniques are needed for getting good outcomes; project management, as I think you know now, is not among the techniques that are appropriate.
Problems with Project Management in General Naturally, I’ve focused on the “limitations” of project management as applied to software. I’ve been kind of vague about project management applied to other things. I do think there are merits in the project management way of thinking; those merits simply aren’t applicable to or help you with software, any more than a spoon helps you consume a hamburger. Still, I think it’s appropriate to at least point out some of the major issues with project management in general, because these issues also raise their ugly heads in software environments, even though there’s nothing in them that is particular to software.
Gamesmanship
41
Building Better Software Better: Software Project Management All organizations have rules that determine success and failure. When the rules are fully understood, it is clear that individuals and groups can experience success or failure independent of the success or failure of the organization as a whole. Smart, ambitious people often like to “succeed” in life, and it is natural that they devote their considerable talents and energy to learning the rules (the real rules, not the official rules) and navigating them to advancement for themselves and, if required, doom for others. Naïve people who identify with the success of the organization can be mystified or disconcerted by the gamesmanship played by such sharks navigating their way to personal advancement. Others retreat into cynicism, try to ingratiate themselves with the winners and in any case keep their heads down.
Hijacking success The focus in games-oriented organizations has nothing to do with creating success. If something that could be perceived as success appears on the horizon, most of the organization’s active minds become devoted almost entirely to getting identified with that success. This is not easy, because so many are trying to do the same thing, and the appearance of trying to claim the success almost certainly foils having it be granted.
Avoiding Blame Even in a truly functioning organization, bad things happen; in a game-driven atmosphere, the frequency rises. So no one takes seriously getting the organization to avoid failure. The only question is, when the sirens sounds and the searchlights scan for the guilty, how can you remain out of the spotlight while others are reamed? This is a wide-spread phenomenon – how many times have you been to the doctor, and at some point the light dawned that the doctor really isn’t focused on helping you with your problem – he gets paid the same regardless of how things turn out for you – but his main concern is avoiding future trouble if your case turns out badly and an investigation tries to find someone to blame for the problem. This practice, decried by many doctors (but widely practiced nonetheless), is called “defensive medicine.” Similarly, in project management situations, you know things are going to go bad. Why put effort into trying to create success, which buys you nothing even in the remote event that you pull it off, when most likely things are going to go badly, a search for the guilty will take place, and innocents will become the victims? You know it’s going to happen, and you know that the fickle finger is as likely to end up pointing at you as the next guy. You’d better start taking steps now to assure that, when the time comes, your defense/alibi is rock-solid. Given a choice between building a solid defense and contributing to success, the choice is obvious – any success would be stolen by someone else anyway, better to have the only thing that is always yours, that no one can pre-empt: a good excuse.
The Feudal system One of the unwritten rules that most projects have is the pre-eminence of personal loyalty and the unimportance of “merit.” Naïve people who focus on the work the organization is supposed to accomplish disdain org charts and titles. They maintain that these things create walls and barriers that get in the way of the work and the human cooperation required to get it done effectively. They are right. They are also oblivious to what transpires under their very noses. Most organizations function the same way Europe did under the feudal system. Most people are just peasants. Their role is to accept a minimal standard of living, no future, and give allegiance to the local lord. The aristocracy is caught 42
Building Better Software Better: Software Project Management up in a never-ending swirl of alliances, low-level hostilities and open wars. Smart players identify the winners and attach themselves to them, regardless of their behavior, productivity, or contribution to the overall well-being. As a group they take more and more power, riding to the top on the backs of competing groups and, always, the peasants. The people with the most power and influence are people who are extremely capable and skilled … at acquiring power and influence.
Group behavior In this environment, you quickly learn what matters. Mostly, your job is to be invisible. To the extent that you do anything, you have to ingratiate yourself with your local lord. If you are smarter and know more than the lord, it is important to hide it, because the lord will take your strength to be a threat rather than an asset. Since the lord’s main job is to avoid failure, that’s your main job. Among other things, that means you have to take as little responsibility as possible, avoid making commitments, and if commitments can’t be avoided, then you have to control the definition of success. When a task comes to you, the most important thing you can do is find fault with the work of those incompetent ne’er-do-well’s who last worked on it. If you don’t carefully examine everything they did and document the flaws and omissions, someone might blame them on your group, and that’s simply unacceptable. Their screw-ups completely explain your own group’s late and/or poor delivery, assuring that, whatever you actually do, it’s not your fault.
High stakes You might think that, when the stakes for a project are high, people would put aside their games and unite to get the job done. For example, think about war time and delivering proper materials to the soldiers who are in harm’s way. It does sometimes happen that group insanity strikes and people really do work together towards an important common goal. But more often, the more important the situation, the more these behaviors take place. It’s pretty simple. In the average situation, people just don’t care. Success? Failure? No big deal either way. After all, I’m OK. But when the stakes are high, success still means nothing to you, and in the likely event of failure, what matters is when the spotlight of blame starts its search for the “guilty,” it doesn’t stop on you. In that environment, you care even more about the local lord’s protection, and any sensible person begins and ends his day by making sure that his defense against potential future blame is flawless.
Making project management work OK, you say, I get it. Project management has real problems, and as currently understood doesn’t work well for software. But, gee, I guess I knew that already. What am I supposed to do? Give the programmers blank checks? Ask them to let me know when they’re done? Actually, sometimes, the way they act, you’d think I did give them blank checks, and when I bring up the deliverables subject, they make me feel like I just asked “Are we there yet, daddy? Are we going to be there soon?” Sometimes, when they’re feeling nice, they mutter what they think are re-assuring noises and obviously hope I’m falling asleep and will stay asleep until the end of the “trip.” Is this supposed to be better than at least attempting to impose at least a modicum of order and control on these tattooed cowboys?
43
Building Better Software Better: Software Project Management This is what project management frequently ends up being in practice, all too often: a hopeless attempt to put a veneer of order and control on a cloud of chaos. Can it get any worse than this? It can. And does. Frequently. Be calm, draw in a breath and let it out slowly, and try not to take anything that follows too personally. Or rather, do take it personally, but don’t take it out on me, personally. Organizations, tired of being tired with failure, frustration and waste, find a way to “eliminate” the failure and frustration and gain something that seems to satisfy them: predictability.
Back to apples and warthogs This point is illustrated by an experience I recently had driving through Troy, New York, the self-proclaimed home of “Uncle Sam.” Now this place is a hell-hole. The residents are, for the most part, dull and lifeless slugs. The municipality’s way of raising extra taxes is to stop cars with out-of-state plates for obviously bogus traffic violations. “I’m sorry to inform you, sir, but you were driving 35 miles an hour in a 40 mile an hour zone. Yes, I clocked you with my radar. Yes, that’s a traffic violation in Troy. Now, you can choose to drive from your comfortable home which is more than 100 miles from here and come to court here for 15 minutes on each of three consecutive odd-numbered Tuesdays to discuss the situation with my cousin, the judge, or you can mail your $300 fine in small, unmarked bills with abject apology, apology optional, to the address given here. Or we’ll take away your license. The choice is completely yours. Have a nice day, and next time, watch your speed.” The last time I drove through Troy, I discovered that the City thugs – err, fathers – had decided to “improve” the main road through town, just to vary the monotony. Two lanes squeezed into one, with lots of lane switching and orange traffic barriers. I waited for five minutes at one point, at the only place in the whole mile-long stretch of torn-up road at which anything resembling “work” was taking place. Actually, let me be more precise. It was the only place where people dressed as construction workers were visible. It took these six guys five minutes to stretch a line from point A to point B, about 10 feet apart, and take down a measurement. When they were done, after consulting the other two guys who were there, probably as literacy backup specialists, it took them another minute to kick the measuring line off to one side of the road; it took this long because the line wasn’t easy to move by kicking it, and it appeared to be against the rules to bend down and pick it up. The traffic was then allowed to move again. Now, what do you think – was this road project in Troy, NY being run according to project management techniques? Did it have phases and entry criteria and milestones and all that? It probably did. And everyone who bid to get the job knows the rules and certainly doesn’t want to rain on their collective parade. So the road is torn up for months, six guys take five minutes doing one minute’s worth of work by two guys, and then it’s break time, and so it goes; the road costs a million dollars a mile, but, by golly, the work is done on time! Aren’t we good! But that isn’t the good part. The good part is when you take everything you know about road construction, including human nature, and apply it to software. In Troy, everyone could see six guys straining to do the work of two, at a fifth of the pace. It was in broad daylight, with lots of witnesses. And no one seemed the least embarrassed. Why should they be? Why, 44
Building Better Software Better: Software Project Management they’re construction workers. They’re working, collecting a paycheck. In their minds, what they were doing was normal, while to me and hundreds of other people perfuming the air of Troy with our idling vehicles, what they were doing was outrageous. Because we could see it, and easily imagine how it could have been done. What if you couldn’t see the construction workers? What if you have never done what they do, not even anything like it? When we watch an Olympic runner, we’re impressed because we’ve all run at some point in our lives, and so we have personal experience to understand and appreciate how amazing it is to run as quickly as any of them can. We’ve all done things resembling construction work. But what about something where you can grasp the results but have never seen it being done? What if you can’t imagine what it must be like to do it because you’ve never done anything really resembling it yourself? That’s programming. There are two strikes against understanding programming: you’ve never done anything like it, and it’s an essentially invisible activity. “Phase I of the project involves performing object-oriented analysis and design of the interactive transaction processes, and should result in UMT-compliant models with supporting use cases. All methods and sub-classes should be clearly identified along with their inheritance relationships, and standard template techniques should be employed wherever applicable, with the relevant templates used being clearly identified. Polymorphism should be avoided, and dynamic typing used only when unavoidable.” Now how many people understand that? How many people can pick out the elements that are plausible but don’t really fit, and the parts that aren’t really industry terminology? How many people can fix it, improve it, and then actually make it happen? Would you want to be friends with any of those people? Now, imagine the software project as a road project in Troy, NY. The “road” is completely torn apart. The “road” isn’t visible. There are lots of people assigned to the project, but most of them aren’t even present at the job site. You can’t tell which is which. The ones who are there are working at a pace that makes it feel to any rational observer that the time spent should be counted as vacation time. You have no way of judging the pace. Of the people on-site, most of them aren’t actually contributing to constructive results. You can’t tell the difference. The ones that are actually doing something do the software equivalent of kicking the line, while anyone watching is thinking “just bend over and pick the &*() thing up!” You can’t tell the difference. Thousands of people are put through countless hours of completely avoidable wasted time and misery, not to mention the outrageous costs, because of the way things are done. If the project comes in “on time,” it’s all the same to you. The fact is, software projects are a whole lot more like road construction in Troy, NY, than anyone is willing to admit or wants to talk about. Most of us don’t realize it because we can’t relate to the work, we can’t imagine doing it ourselves, we’ve never done anything truly resembling it, and therefore the work is essentially invisible to us. And the irony of it all is, the normal approach to project management tricks us into thinking that situations far worse than Troy, NY are actually big success stories.
45
Building Better Software Better: Software Project Management
The Requisite Table There have been lots of metaphors and rhetoric about the ills of project management. But it can sometime be difficult to cut through to what the real issues are. Here is my attempt to boil down the difficulties of software project management to the bare essentials:
Software Project management issue
Explanation
Objective standards
Given that the software you want to build hasn’t been built yet, and given that there are many ways to build it, objective measures of it (like lines of code) tend to be trivial or misleading, particularly prior to building.
Expectations standard
as
the Because software that hasn’t yet been built is so hard to “see” (unlike “see that hill over there? Climb it.”), people focus on how much effort people say it will take to build something. There’s no good way to tell if what they say is wildly unrealistic, reasonable, or intolerably unambitious. People nonetheless tend to focus on the expectation that was set instead of how long something should take to build.
First time vs. repeats
Software project management was modeled on a methodology that was designed for repeating a known thing in the physical world and making sure the repetitions come out on time with good quality. But building software is, by definition, building it for the first time!
The difficulty of doing new Not only is software always being built for the first time, things it’s a very hard first time thing to do, as demonstrated by the great trouble we have getting it done. Just what are requirements?
The consequences change
The requirements for un-built software are particularly problematic, and are nearly impossible to create both realistically and helpfully prior to building.
of In most real-life software projects, everything is changing – the business environment, the requirements, the underlying software and systems you’re using. Even when it isn’t explicitly changing, it still has bugs and other disasters.
The “planning” mind-set The whole attitude of project planning comes from and the attitude towards manufacturing, where the goal is to do something variation repeatedly, and have minimal variations in the results. This attitude is completely inappropriate for software, and leads to multiple bad consequences.
46
Building Better Software Better: Software Project Management The focus on form and Software, like, say, physics, is an intellectually challenging process over content endeavor. Project management was developed for processes that are physical and require no intellect. It is completely appropriate to focus on process for physical manufacturing, because that’s the challenging thing. In software, effort put into process distracts from effort put into the real problems. The difficulty and The difficulty of computers is one of the problems that complexity of computers in project management is intended to solve. But it actually general solves nothing, adding a new layer of concern and bureaucracy to an already-difficult task, like adding blowing soap bubbles to riding a unicycle. Mental and physical work
Project management concepts apply reasonably well to physical work on physical objects. But computer work is entirely mental. Finding a bug can take a second or it can take a week. Mental work of this kind isn’t linear, and isn’t measurable the way project management assumes.
Physical things and digital Digital mental things are even more non-physical in nature things than other mental things that have physical correlates, making the project management concepts even more inappropriate. General project There are a host of well-known problems with project management problems management that are not specific to its application to software, that plague software just as vigorously as they plague other endeavors. Here is my attempt to contrast the software project management approach to a better way of doing things: The Project management approach
A better way
Avoiding errors
Finding and fixing errors quickly
Setting expectations and then meeting Making goals them Process orientation
Content orientation
Good management is key
Good workers are key
Control and predictability are supreme
Speed and agility are supreme
Planning
Doing
47
Building Better Software Better: Software Project Management Meet those expectations
Continuous improvement
Standards
Creativity
Software factory
Software jam session
A scripted play
Improvisation
Central control
Distributed control
Strict hierarchy; command and control
Loose hierarchy; collaboration
Synchronous advance of coordinated Semi-autonomous groups asynchronously New requirements will hurt the project
groups
working
Adopting to new understanding helps the business
Beyond project management So how can we make software better? How can we make better software? Naturally, I have some ideas. The scope of the ideas goes well beyond this edition of the paper, which focuses mostly on the process part of how software is built, ignoring for the most part people and tools; furthermore this paper pays little attention to building better software, i.e., the content that makes good software differ from bad software. Still, it’s only fair to end on in constructive spirit, with the following notes on the world beyond software project management. In my experience, talent and intellectual strength beats experience and time in service in computer jobs. Successful hiring selects for talent, strength and drive, hoping to get a person who can learn what needs to be learned, build what he needs, and adapt to changing circumstance to make a success out of the job. My first really successful hire was a guy with a physics degree who had done a little work with computers in a lab course. I gave him a manual to read, and he came back confused by a paragraph on page 105 – which, when I read it, I realized was confusing because it was wrong. I hired him on the spot, he tore through everything he needed to know, and quickly became self-reliant and wildly productive. In a “mature” software development organization, the people near the top are the ones who have the talent and the drive to get to the top of such of an organization, and stay there. This is not the same thing as developing good software that helps the business! When such people hire employees or decide who to encourage, their natural instincts will be to hire stable, experienced people who are good at doing a certain thing at a certain level of the organization – in other words, people who are likely to achieve specific modestly-defined goals, but pose no threat to the position of the person doing the hiring. So “experienced, seasoned, professional managers” (with rare exceptions) tend to impose the bad practices
48
Building Better Software Better: Software Project Management identified in this document, and hire the kind of people who “work well” in those perverseincentive environments. Efforts do well to the extent that they become aware of repetitive activities, and apply automation techniques to minimize the repetition. This applies to code itself (if sections of code look similar, identify the similarities and factor it out), development cycles, testing and integration (my paper on QA focuses on methods to achieve this) and all other aspects of the time and effort spent to get working code out. If this is done well, you can actually eliminate most of the hierarchy and process that bogs down development organizations as they try to grow. Healthy software development embraces change of many kinds. It emphasizes frequent check-points. The people involved should welcome healthy change, since it means the group is learning and evolving as it goes. One of the biggest things successful software projects do is acknowledge that you don’t know important things about where you need to go, and that your users won’t be much help until you take them there. So the faster you can make results visible, the quicker you’ll get meaningful feedback. One of the biggest complaints that experienced software builders yelp about is not being given good enough requirements and not being given enough time to perform a “proper” design. They should get over it. I have myself made these mistakes so many times it makes me sick to think about it. And I still have the impulse! In decades of work, my sad conclusion is that “good” up-front requirements are nearly impossible to produce, and that most effort spent on “design” ends up creating flexibility that isn’t needed, and creating structures that prove to be obstacles. What looks like good design ends up being after-thefact, and amounts to eliminating repetition. Some of the biggest bloopers of large software projects tend to take place in the testing, integration and deployment phases. Awful experiences tend to result in severe penance being imposed in the form even greater overheads and process. Instead, testing, integration and deployment should be continuous and automated processes as described in the QA white paper. In conclusion, I wish you excellent software experiences, ones in which really good software is built efficiently and quickly, and plays a significant role in raising the level of the organization for which it is built.
49
