Down the Rabbit Hole of Suck, Part 1

As a software professional, I have striven throughout my career to do the best job building software that I can. I’m sure all of you in the software profession who are reading this have done the same. The general standard for doing a good job building software has been illustrated by the iron triangle, which is shown below.

The image shows the three main constraints of a software project: the requirements that comprise the solution, the amount of time taken to implement the solution and the cost of the implementation. These constraints are interrelated (e.g., if you increase the scope, cost and time will also increase). From the time I was a wee lad just out of college, it was drilled into me that a software project is a quality software project if and only if all requirements within the original scope are delivered on time and within the agreed-upon budget.

How have we as a profession done compared to this standard? Research on 5,400 IT projects conducted by McKinsey and the BT Centre for Major Programme Management at the University of Oxford found that, “large IT projects run 45 percent over budget and 7 percent over time, while delivering 56 percent less value than predicted.” [1] According to PriceWaterhouseCoopers, out of approximately 10,000 projects surveyed, only 2.5% were delivered within the original scope, time and cost. [2] Those surveys are several years old, but I see no reason to believe that the situation has gotten dramatically better since then. Only one conclusion can be derived from this data.

We suck. Big time.

The thing is, we’ve gotten a lot better at what we do. When I first started out in the software profession, we were building software systems using Structured Design within a Waterfall life cycle. Now we’re using paradigms like Object-Oriented Design, Functional Programming, Reactive Programming and Domain-Driven Design within an Agile/Scrum or Kanban process. We’re following Continuous Integration, Continuous Delivery and Continuous Deployment to get software solutions into the hands of our customers much more rapidly. We’re building scalable, distributed, robust applications. And yet according to the iron triangle, we still suck.

This perception of suckitude (and yes, that’s a real word) is one of the biggest drivers of the low trust that exists between IT organizations and the business organizations they work with. I believe that one of the root causes of this perception is a fundamental misunderstanding of exactly what it is we software professionals do for a living. This misunderstanding is on us, because I don’t think we’ve done a very good job of telling others what we do.

We’ve had (and still have) many different names for the jobs that we do. We’ve been called programmers, systems analysts (though I don’t recall ever analyzing a system while I had that title) and developers, among other things, but software engineer is the job description that seems to stick more than the others. The term has been around for a long time, having been coined by Margaret Hamilton back in the mid 1960s, but it started to become popular in the 1980s.

The idea that we apply engineering principles to the process of software development is quite attractive. However, if we are truly software engineers, we should be able to deliver on the promise of the iron triangle with some degree of consistency. The data cited above indicates that we cannot do that. What kind of profession would we have to be in order to deliver on that promise?

Real engineering professions, such as electrical engineering, mechanical engineering, etc., can meet the demands of the iron triangle orders of magnitude better than we can. Take an electrician, for example, which for the sake of this post I’m calling the coding discipline of the electrical engineering profession. An electrician can come to your house, diagnose the issue, figure out what needs to be done, tell you with reasonably good precision how long the task will take and how much it will cost, and generally deliver that task within the stated time and cost. We software professionals can’t come remotely close to doing that. (And don’t tell me that software development is much more complex than electrical engineering. Electricity is magic. Prove me wrong.)

How do electricians do it? Electrical engineering, like other engineering professions, has a documented body of knowledge that is known and commonly accepted across the industry. Electricians entering the profession progress through multiple levels of maturity as they learn the body of knowledge over time and practice it daily. Most of what electricians do are things they have done before, and as they gain more experience, they become more adept at handling any boundary cases that may arise. There are certification and licensing processes in a real engineering profession. Engineers who are licensed are generally assumed to be more skilled and knowledgeable (and more expensive) than those who are not.

The software profession doesn’t have these things. Yes, there is a Software Engineering Body of Knowledge. Have you read it, or part of it? Few software developers have; thus, it is not known or commonly accepted across our industry. Many states in the U.S. have certification and licensing processes for software engineers, but a software engineering license doesn’t have the importance that it has in a real engineering discipline. How many companies with which you have interviewed have called for a licensed software engineer as part of the job description? By my personal count, it’s zero. Other than some superficial similarities, our profession just doesn’t have much in common with engineering professions. We are not software engineers, and we never were. So what are we? What is it that we software professionals do?

Through discussions with my colleagues regarding this question, I learned that there are three aspects to what we do as software professionals:

• We are learners. We must learn the business model, business processes, pain points and mission of our company and client. We need to understand these things in order to deliver appropriate software solutions for them.
• We are software practitioners. Parts of our software solutions involve things that we have done before, perhaps many times. If we haven’t done those things, one or more of our colleagues has and we can ask them for help. Examples include standing up an authentication solution using Active Directory, and building an Azure DevOps pipeline.
• We are software inventors. Most of what we do as software professionals is aimed at solving problems that we have never solved before, and that probably no one has solved before. The solutions we develop contain aspects of software practice like those mentioned above, but the bulk of these solutions consist of software that we invent. If the entire solution had already been developed before, customers would simply buy that solution. We are called in when no such solution exists for our customer. They need us to invent one.

In part 2 of this post, I’ll focus on the idea that we are software inventors, and the implications that has on how we plan and execute software projects, as well as what reasonable expectations of a software project are.