In part 1 of this post, I introduced the notion of software inventors, which I believe captures fairly well what we software professionals do for a living. Being software inventors doesn’t mean that we’re inventors in the traditional sense. We work in a software industry that requires systems to be developed and delivered. Being software inventors in that context comes with challenges and implications.
Inventors suck at estimation
It’s hard to estimate how long it will take to invent something, because you’ve never done it before. If you ask an inventor when the new thing she is inventing will be completed, she will probably answer, “How the hell should I know?” which is in fact the correct answer. It’s not an answer software inventors can give. It’s difficult to estimate how long it will take to deliver a software project because it’s probably something that we haven’t done before, and perhaps no one has done before. Yet we have to provide an estimate.
Inventors suck at delivery
Let’s talk about WD-40. It’s a great product. It has been said that you need only two things in life: WD-40 to make things go, and duct tape to make them stop. WD-40 is a penetrating oil that was intended to be a rust-prevention solvent, and also serves as a lubricant and moisture displacer. Its name is significant. “WD” stands for Water Displacement, and the number 40 represents the 40th attempt at getting the formula right. Taking 40 tries to successfully develop a formula was fine for the Rocket Chemical Company, which invented WD-40, and few people remember the previous 39 attempts. However, in the software industry, we don’t get 40 attempts, or even four attempts, to deliver a working system. We need to get it right the first time.
The aforementioned duct tape is also a great product. But what’s the one thing duct tape is not good at? Taping duct. Come on, duct tape, you had one job! But it can be used for so many other things. When we develop software solutions for our customers, we need to develop the system they want, not the one we think they should want or the one we really, really wish they wanted. If the delivered software doesn’t satisfy their requirements, it won’t be of much consolation to them if we tell them about all of the other cool stuff that software does.
Software invention is not traditional invention
Overcoming these challenges is aided by the fact that software invention is not invention in the traditional sense. We software inventors know that it is possible to build a software system that will meet the requirements we are given. If we don’t know whether that is true, we’re in the realm of research and development (R&D), not software invention.
We also generally know or can determine the elements that will comprise our software solution. For example, if the requirements for the system we are to build include taking certain actions when we detect that specified events have occurred, our solution will probably include a messaging or data streaming framework of some sort. We may have to combine these solution elements in a way that hasn’t been tried before, but at least we know or can determine what these elements are.
These aspects of software invention give us a significant advantage over traditional inventors, and those advantages enable us to estimate and plan software projects. That being said, there are also some implications that arise from being software inventors.
Estimation
Estimation involves not just estimates of effort, but creating schedules. In my experience, I have found that there are two kinds of scheduling. I call them cable dude scheduling and yard dude scheduling. They look quite different, but they are actually two views of the same thing. An estimate given by a cable dude, for example when they tell you when they’re likely to arrive, is in the form of a range. I’ll be there between noon and 6 pm.
The yard dude1 gives you a specific time. When I would ask my yard dude when he was going to come to trim my shrubs, he would say something like Thursday at 2 pm. They’re not going to show up on Thursday at 2 pm. Maybe they’ll come on Friday at 3 pm. It used to drive me crazy until I figured out the system. The time the yard dude gave me was never the intended time, it was an example of a time at which they might show up. So Thursday at 2 pm really means a day during the latter part of week in the afternoon. It’s a range like between Wednesday and Friday from 1 pm to 5 pm. Friday at 3 pm is in that range.
Thus, cable dude scheduling and yard dude scheduling are really the same thing viewed from different perspectives. So what does this have to do with project scheduling?
Most if not all projects have a schedule with milestones and a delivery date that is determined and agreed upon prior to the start of the project. Any deviation from this schedule is considered a major problem and is often addressed by meetings. And the meetings will continue until morale improves. The project delivery date and milestones are yard dude estimates. There is roughly a zero percent probability that the project will meet those dates with the expected deliverables (well, a 2.5 percent probability according to the PriceWaterhouseCoopers survey). This isn’t a cop-out. It’s reality. How do we deal with this reality? That’s what the iron triangle is about. How can we create estimates and create benchmarks to check progress against those estimates in this reality? I’ll talk about that in a bit.
We are optimistic estimators
First, I want to talk about how we estimate. Humans are optimistic estimators. We estimate tasks based on nominal conditions. Think about your average commute time on days when you go to the office, in the form of a range such that 80 percent of your commute times fall within that range. The average commute time in Houston (where I live) is reported to be 30 minutes. So if you’re an average commuter in Houston, you might tell people that it takes 30 minutes plus or minus five minutes to get to work. Is that really a good indicator of your commute time?
No, it isn’t. You know this because if you have a meeting at 8 AM and you absolutely have to be there on time, you’re going to leave well before 7:30, because there may be accident that slows your commute or an abnormally high volume of traffic. When we give a range estimate for the time required for an activity, we tend to assume that the activity duration has a bell-shaped curve (that’s where the “plus or minus” comes from) and we pick a range that’s too narrow.
If you were to plot your commute times over a few months, you would see not a normal (or bell-shaped) distribution, but a lognormal (or skewed) distribution. You would also likely see that your 30 minute estimate is actually closer to the median, not the mean. This means that you travel to work within 30 minutes only 50 percent of the time. Your mean commute time may be 35 minutes. That five minute difference between your median commute and mean commute may not sound like much, but over the course of a year, it really adds up.
That original 25 to 35 minute estimate, which you thought encompassed about 80 percent of your commutes, is really more like 25 to 50 minutes. If you want a 90 percent confidence interval, it may be something like 24 to 55 minutes. Now you have a much better idea of when you need leave home to reach the office for that 8 AM meeting.
Software inventors are optimistic estimators
We do a similar thing when we estimate the effort required to complete a unit of software work such as a user story. We assume that things will go nominally because until we start the work, we don’t completely know what we’re up against. We need to push against that tendency to assume that things will go well. I’ve summarized this idea as follows:
Bishop’s First Law of Software Development
Sweat the small stuff, because the small stuff will kick your ass.
For example, maybe I need to move some data from a Sybase database to a Mongo database, and enrich it a little on the way. No problem, I can use Kafka to do this. I’ll use a Sybase source connector, write a simple KSQL streams app and get a Mongo sink connector. One or two days max. Then I find out that there is no source connector that works with the version of Sybase my client is using. I need to write a custom Sybase connector, and that two day estimate is now three days too small. That’s the small stuff I’m talking about, and we need to take it into account when we estimate.
The ramifications of Bishop’s First Law of Software Development are that the differences between the median and mean times for software development tasks tend to be significantly greater than for more predictable tasks like commuting to work. Thus, software development can take much longer than expected.
How to estimate software invention
To estimate the effort and time required for a project powered by software invention, we need to create a cable dude estimate based on our knowledge of how the lognormal distribution applies to estimation. At a high level, here’s how it works:
The project is broken down into software development stories. These may be user stories, or perhaps they’re closer to epics, but it doesn’t matter. We need to estimate, given what we know before the project starts, what stories must be done in order to deliver the project.
For each story, we estimate how long we think it will take in some unit of time (for example, days). That datum alone doesn’t tell us very much. We also need to estimate the uncertainty level of our time estimate, for example, none, low, medium or high. No uncertainty means that you’ve delivered this story or one very much like it at least several times and you know exactly how long it will take. In other words, this story is software practice, not software invention. For stories that call for software invention (probably the vast majority of the stories in the project), there will be uncertainty in the estimate, and if you’re having a hard time choosing between two uncertainty levels, choose the higher one.
Since we know that the time range of a story estimate takes the form of a lognormal distribution, we can interpret our estimate as the mean and use the uncertainty level to choose a standard deviation, and from that we can build a distribution against which we can perform a Monte Carlo simulation. From the results of that simulation, we can determine an estimated time for the story in terms of confidence intervals (e.g., 70th percentile, 80th percentile, etc.). By adding the story time estimates together, we can calculate the total estimated time of our software stories.
We also need to estimate how many of the project’s stories are known. For example, if we think that about 80 percent of the stories are known, then we’ll need to add about 25% of the current total story time estimate (20% unknown stories divided by 80% known stories) to our overall story time estimate.
There will also be effort required in project startup and completion tasks that will have to be added to the overall project estimate. In addition, there are “glue tasks” such as Scrum ceremonies and meetings. Estimate how much time you anticipate spending in these glue tasks as a percentage of your work time. For example, if you think you’ll be spending 8 hours per week in meetings and Scrum ceremonies, that’s 20 percent of your work week, so add 20 percent to your project estimate.
Having determined our total effort estimate (and we’ll use staff months as the unit), we can determine the nominal schedule in months as follows:
Schedule_{nominal} \approx 3 \times \sqrt[3]{Effort}
This nominal schedule strikes a good balance between cost and completion time. We also have to determine how many people we need on the project. In general, the average number of people required during a project is determined by simply dividing the effort estimate by the desired calendar time of the project. However, compressing the schedule by adding more people is not a simple matter. For example, you can’t necessarily cut the schedule in half by doubling the number of people on the project. Years of industry research has shown that the relationship between schedule change and effort change isn’t linear, it’s a PNR curve. According to the PNR curve, compressing a schedule to less than 75% of the nominal schedule is impossible.
Upon completion of this exercise, will we have project effort and calendar time estimates at multiple confidence percentiles. With this data, we can provide a cable dude estimate. We just need to choose the confidence percentiles that we want to use for the lower and upper values of our range. For example, we might choose the 70% confidence percentile as the low end of the estimate and the 90% confidence percentile as the high end.
How to determine the status of software invention
We still need to know how we are progressing in our project, whether we are doing well or struggling, and be able to communicate that status to people who need to know it. One way to do this is the “red-yellow-green-light backlog.” Here’s how it works (for projects using Scrum).
We need to calculate two values: the mean and standard deviation of our sprint velocities. It may take a few sprints before we can get good values for these variables. Then we can produce a plot like the one below, where the Y axis represents points and the X axis represents time.
The vertical hash mark on the right end of the X axis is a given date against which we are assessing status. The line labeled µV represents the mean velocity per sprint, µV + σV is the mean velocity plus one standard deviation, and µV – σV is the mean velocity minus one standard deviation. When we extend these lines to a particular point in time, we see how many points we can deliver by that date at that velocity. For the remainder of this section, let n be the number of sprints remaining until the date represented by the vertical hash mark on the X axis.
The circles represent stories. The dark green circles are stories that have already been delivered. We take the stories from the top of the backlog whose cumulative points add up to but do not exceed (µV – σV)n, and color them light green. These are stories we are confident we will deliver by the given date. Then we take the next stories from the backlog whose cumulative points add up to but do not exceed ((µV + σV) – (µV – σV))n and color them yellow. We will deliver some of these stories by the given date, but probably not all of them. The remaining stories are depicted in red. They will not be delivered by the given date unless something changes, like project scope or resources.
This status will change from one sprint to the next. Therefore, we should do this assessment at least once per sprint, and it can be presented at the sprint review. By doing so, we will have solid actionable data with which we can discuss the status of the project with stakeholders. For this assessment process to work, it is imperative that the backlog remains prioritized.
Conclusion
There has long been a perception that the software industry has consistently fallen short of expectations in terms of delivering on the promise of the iron triangle: software projects delivered on-time, within cost and with all requirements implemented. These expectations are based on a misunderstanding of what it is that we do. Our industry is not a software engineering industry with the higher level of predictability associated with engineering projects. We are software inventors, and we work in a domain that is much less predictable. However, by understanding the nature of this domain in which we work, we can provide reasonable estimates of the time required to deliver a software project, and assess progress against these estimates in a way that yields actionable data, while continuing to deliver outstanding software applications.
1 This depiction of a yard dude is based on my experience with a particular yard dude. Your experience with yard dudes may differ from mine. Any resemblance to yard dudes living or dead is coincidental. No yard dudes were harmed in the making of this blog post.