Measuring software risk: the forgotten development constraint

Software risk haunts every development project. And yet we almost never talk about it. Every project manager focuses on the same big three constraints when putting together a proposal: scope, timeline and budget. For all three, you’ll get fancy graphs and analytics, a full, quantified representation.

Then software risks come in and blow it all apart.

In software development, risk is the forgotten constraint. This is especially galling because the risks in software development have such a big impact on the other three constraints it’s almost criminal not to address it. If a feature is at risk of being altered, it changes the scope and affects the timeline and budget. If a feature is at risk of being overly complex, it changes the timeline, which also affects the budget and pushes on the scope. And if a feature is largely undefined, it can put all three in jeopardy.

We always say that you’ll never know less about a project than you do when you start it. A software engineering risk analysis allows you to identify how little you know, and make a plan for finding out more. At Table XI, the custom software company where I’m CEO, we’ve developed a system to measure and visualize the risk of each software project we start. It gives us a better sense of what we’re tackling, and it allows us and the product owner to de-risk the project and build more accurate estimates. Just as we evaluate whether we’re on time and on budget over the course of a project, we evaluate how well we’re managing software risk.

Here’s how we make the risk in software development a tangible part of our projects.

What is software risk, and how is it affecting my development projects?

Software risk is the thing that keeps product owners up at night. Starting your software engineering project with risk analysis and management gives you a much clearer answer to that question. When most development teams, including Table XI, start scoping and estimating a project, we break the work down into stories — features or tasks that need to be accomplished. Those can be grouped into functional areas, like all the stories necessary to create a login/signup process. Once we know what needs to happen, we can apply software risk analysis methods to understand how likely each story is to derail that part of the project.

This is when we get to take the power back. Instead of worrying about unseen software project risks, now we have everything out of the shadows. With the right software risk identification process, we can put numbers around those concerns. We can identify which stories and functional areas pose the biggest threat. And then we can use design sprints, development spikes and other techniques to learn more about them and ultimately de-risk them.

So here’s how to perform a software risk analysis you can use.

Qualitative software risk analysis

A full accounting of project risk starts with qualitative risk identification. That helps us get as much information from product owners as possible, so we can take what they know into account when we perform our quantitative risk assessment. For each story, we ask the product owner to define three software design risks: volatility, complexity and completeness. Each gets assigned a score of high, medium or low.

Volatility is how likely something is to change. We like to understand from the product owner how likely the business might change its mind about a feature or its technology requirements. If a story or functional area is up in the air, it can pose a larger risk than features that are more defined and understood by the team.

Completeness is a variable we can help assess. We let the product owner know how complete our understanding of the feature is, and they can fill in any blanks, like giving us access to an API we may need to work with.

Complexity is the last variable. Simple things we’ve done a million times get a low score, while stories dealing with complex security restrictions or difficult system integrations would get a high score. There’s not always much we can do to reduce this, short of rescoping the story, but we need to know where we’re entering complicated terrain.

Doing the qualitative portion of the risk assessment provides a chance for us to go back and forth at a high-level with product owners, so we can figure out what’s making them anxious and what we can do about it. The three categories and three levels gives them an understanding of risk analysis in software engineering. Then we can analyze the results and make a software risk management plan.

See how we visualize qualitative software risks

Quantitative software risk evaluation

After going over the qualitative risks with the product owners, we turn to our designers and developers. Now we can introduce the numbers. The product owner gives us a gut feel, but the software delivery teams give us an estimate.

At Table XI, we use story points. Each story gets assigned a minimum, a likely, and a maximum number of story points that represent the relative complexity of each task. So one story’s min/likely/max might be 1/3/5 while another’s might be 1/1/1 and another’s might be 3/3/3. (A three point story will likely take three times as long as the 1 point story).

You can start to see right away how risk is represented. A 3/3/3 and a 1/3/5 average out to the same amount of story points: Three. The variability in the 1/3/5, however, indicates that there’s a lot more we don’t know about that story and a potential for different outcomes.

What a software risk assessment allows you to do

Having both the qualitative and quantitative scores gives us a full picture of the software development risks. Now we can do some neat analysis. We can filter the quantitative scores like we did the qualitative ones, and begin identifying opportunities to de-risk the project. We can also compare the two sets of scores. If a product owner is indicating there’s a lot of risk in one functional area, but our developers don’t see it, there’s a good chance it’s because the owner knows something we do not. A complete software requirements risk assessment gets all the information out on the table, so we can start as transparent and clear-eyed as possible.

Once those risks are known, you can compare them, and that allows you to prioritize.

Building a resilient software development team

We staff our teams with software project risk management in mind. As part of the visualizations we do, we map out the story points assigned to each functional area — and how much variance there is in each estimate — so we have a clear picture of the amount of effort it will likely take our team to get the work done. We can choose to staff for that likely level, or we can choose to staff with risk in mind.

As we continue to adjust the risk visualizations throughout the project, we can roll people off early if the project is looking less risky, or keep them around to manage lingering unknowns.

Starting in the right place for software risk management

To visualize where the biggest risks lie, we plot each functional area on a 2x2 risk matrix for the software project, where priority is on the x-axis and risk is on the y-axis. We aggregate the stories in each functional area, using the overall story points to determine the size of the bubble, then plotting it based on its average risk score and priority rating. The result let’s us see the work by timeline, priority and risk all at once.

Learn how to visualize your projects with our software risk management pdf

If a functional area is in the high-risk, low-priority quadrant, we can table it for now, because nothing’s going to be affected if it’s cut from scope. The high priority, high risk quadrant, on the other hand, is where the dragons are. We don’t have a choice, we can’t cut it from scope. So we have to find a way to de-risk these concepts. We can do that with a short development spike, where we dive into the code needed for a particular feature. We can also do more user research, or facilitate a Design Sprint, whatever it takes to get a clearer picture. Then we can move to software risk management.

Building a software risk mitigation plan

Software development risk management is all about getting as much information as possible. Instead of procrastinating on the risky work, we pull it to the front and immediately get started. The best way to de-risk something is to start doing it. The closer you get to done, the more the risk goes down. In our case, we actually visualize that by shrinking the size of the balloon on the matrix as the timeline moves along.

Starting with the riskiest projects lets us figure out early on how much of the contingency we’re going to need. That way we can reestimate and add features, or set an earlier end date if the project turns out to be less risky than believed.

For product owners worried about the amount of variability in an estimate, a way to measure software development risk is a godsend. Many teams handle software risk mitigation by just building in a bigger contingency plan. If you’re looking at an estimate with hundreds of thousands of dollars and weeks of work between the high and low end, it’s not because your development team is trying to scam you. It’s because there are too many unknowns. Product owners can close that gap by running a risk assessment for the software, then targeting the problem areas with design sprints and other de-risking tools. With a focused week of work, your developers and designers will know enough to meaningfully reduce the risk of a project.

Risk can be measured and tracked like anything else. If you’re going to take the time to forecast a budget, you should absolutely take some time to put numbers around your risks so you know how likely they are to upend that budget. The goal of assessing timeline, budget and scope is to determine what needs to be done. The goal of a software design risk analysis is to determine how likely it is to actually get done. If you want fewer sleepless nights on your software development projects, it’s time to face your risks head on.

Download the equations we use to quantify and visualize software risk.