Success in Software Engineering means doing things that help increase sales or retention in the case where the product is being created, or help improve some part of the pipeline when software is meant to support business operations.

How then, do we measure success?

In the case of supporting the business, it would be by looking at the overall improvement in flow through the system (i.e. how much can the business create and sell?).

In the case of creating the product, the same is true, but it’s more focused on sales increase or customer retention.

Flat sales or declining sales means that the value created by software engineering is not sufficient to compete in the marketplace. Something is not working.

These are trailing indicators, and we also discussed leading indicators broadly.

But the leading indicators can vary widely across industries and companies and business models. Facebook, for example, uses time and attention metrics (which explains why it seems to want to drain the life out of us). Netflix may use hours watched. A budgeting app may use a metric about how many transactions are uncategorized or ignored.

These metrics are business and model specific indicators. But are there also broad metrics that tell you something about the ability of the software engineering organization in itself?

What metrics would be widely useful to track, to properly diagnose problems or simply find opportunities for further enhancements in company effectiveness?¹

Widely Applicable Metrics

Previously we identified small quick iterations as useful for verifying hypotheses and delivering value quickly (to increase rates of return).

Learning quickly and responsively making course corrections will powerfully affect your outcomes.

This, it turns out, is well-trodden territory.

In Accelerate, the authors put forward four metrics. They also make a strong case that these metrics correlate strongly with high-performing organizations with leading bottom-line performance.

These metrics are:

1. Deployment frequency
2. Lead time for changes
3. Mean time to restore
4. Change failure rate

These metrics are borrowed, with slight modification, from LEAN principles of manufacturing (a major inspiration for agile philosophy and methodologies).

Deployment frequency is actually an approximation of batch size. It’s very hard to know how big a batch is in software, as each commit can have any number of changes, and complexity of changes is hard to measure from lines changed and other surface level metrics.

Lead Time for changes is how long it takes to go from code committed to code deployed. This ignores the highly variable product development lead times, as those have indeterminate length and are hard to compare (and additionally are a different bottleneck in the process that can be separately diagnosed).

Mean time to restore is how long before service is back to normal after an incident. In our fast-moving world, problems will happen. Acknowledging that, the next question is “How fast can we make a problem a non-problem?” This metric measures the impact to team (unplanned work) and customers (time with degraded or no service).

Change failure rate is how many deployments lead to failures in production. “Failure” must be defined, and a further breakdown into major/minor/etc may be useful in focusing efforts to improve.

Why are these metrics good metrics?

Frequent small iterations are good (as we’ve established). Stability is also essential, as instability leads to a lot of issues internally and with customers.

Learn fast, break less and less over time, change rapidly (when needed).

The first two metrics are about changing quickly. The second two metrics are about stability of function.

Given that a software engineering org has a responsibility that it alone possesses, to maintain the ability to continue delivering software, these metrics are especially important in gauging this core competency.²

Measuring Performance

If these are good metrics for measuring the core competency of software delivery, then we have a common baseline for looking at the health of Software Engineering teams (and organizations).

This doesn’t tell us if rapid delivery is being well-directed to maximize learning and market opportunity (i.e. getting better and selling more).

Software engineering is just one competency of a business, and if it’s an organization in the business, it also has to be good at existing in that context. But if the business is healthy, then a great software engineering organization will help it blow past the competition.

If the parts aren’t working well together, then very little good will come out of being exceptional at software engineering.

However, it tends to be the case that companies that engage in transformational efforts in one department often find the rest of the culture begins to move in the same direction, and even better things start to happen.

1. Note, not efficiency. Efficiency is doing stuff with as little work as possible. Effectiveness is doing the right stuff, and is a much better paradigm to operate from. ↩︎
2. There are likely situations where these metrics are not exactly applicable. Embedded systems without networks make deployments (getting code to existing customers) very challenging, and out of direct control. Some applications, such as flight systems, need to have practically zero major flaws, because people could die. You don’t want your pacemaker developer to “move fast and break things”. So there are places where you need to weight stability very heavily. For most software organizations, however, there is probably a way to create some sort of extra wall between creation and deployment, so that internally the developers can move confidently and quickly. But every situation needs to be evaluated on its own. ↩︎