2 What Engineering Excellence Really Means

Engineering excellence isn’t about perfection, right? A lot of teams and organizations, especially organizations with more than one team, try. They have this ethos. Oh, we’re going to move to a new way of doing things and I want to understand all of the facets of that before we move. I want to understand all of the choices we’re going to make. I want to understand why we’re making all those choices. I want to make a big plan of what we’re going to do. And that’s not reality. That’s not how, when you actually start implementing things. And this is true for implementing things with people, implementing things in systems, implementing things in software.

The needs that you discover are different than the needs that you thought of right at the beginning. So there’s not a huge amount of value in planning out all the things at the start only to have that plan destroyed. So we still need planning. Absolutely. I’m not saying shoot from the hip and make stuff up. That’s definitely not what I’m saying. The plan itself, that every little detail is the irrelevant part.

I think there was a US president from years ago. Might have been a general or a US president, might have been Eisenhower, said, plans are irrelevant. Planning is everything. I think it was a World War II thing, right? It’s about the planning, the discussions, the what do we think’s going to happen? What do we want to try and do? What are our options when these different things happen? And discussing them and disseminating that information, but the plan itself is irrelevant.

I think isn’t there a Sun Tzu? Maybe no plan survives contact with the enemy, right? As soon as you start doing something in your organization, as soon as you start following the plan, it’s irrelevant, right? The first step, the feedback from that first step in your plan probably destroys the entire plan. So if your goal is to move towards engineering excellence, it’s not about perfection. It’s about instilling those ideas, modern software engineering ideas of continuous, right? Usually you stick continuous in front of everything. Continuous delivery, continuous quality, continuous adaptation, continuous learning. It’s that continuous or emergent nature that the things you’re going to need, the knowledge you’re going to have to learn, the things you’re going to find out are going to emerge over time.

So we need to create a system within which that emergent knowledge adapts the system as it goes to what is happening. So that we end up getting, delivering working products, getting high quality output. And although the goal is not agility, being fast and nimble and delivering for your customers is, right? And that can be a definition of agility for sure.

So there are loads of tools and capabilities, practices that are known to enable this. And every time I work with a lot of customers and we talk about some of these practices. I was talking yesterday about trunk-based development and the customer came back with five or six reasons why trunk-based development wouldn’t work for them, right? Or we couldn’t do it that fast, but you know, it could be this long.

And my immediate response was, well, why are those things blocking your ability to do that modern engineering practice? Where did those things come from and how do you fix them, right? And some of it could be a long-term fix. We’ve caused some legacy products and it’s going to take time to get to that point. Right? If you’re currently manually building out your infrastructure for your products, if you’re currently doing manual portions in your build process, right? It’s not an automated CI/CD process. There’s stuff somebody has to go do in between, in different spots in order to make it work. Then it can be really daunting to have this idea of, well, we’re going to do trunk-based development. We’re going to have a topic branch spin-off. It’s only going to exist for a few hours. It’s going to have a full environment. We’re going to run all our tests and then it’s going to come into main in a few hours. How is that even possible, right? Because of the frictions that are holding us back.

But if we want fast feedback, right, then we have to get to our customers as quickly as possible with high quality, right? Not delivering them crap because then the feedback will be it’s crap, right? We want to deliver high quality, usable, working product on a continuous, as quickly as possible, as often as possible basis. And for that, we’re going to need to have clean code, right? You can’t do this with poor quality code. And you might be starting with poor quality code. So there’s something to go work on. You need observability in the way you do things and you may not have observability today. You may not have telemetry coming through your product. You may not have those things built in that allow you to see what’s going on, the knowledge that you gain on how users are using your system but also what’s happening in your system while users are using it. It can be invaluable, right?

You’ve effectively got brittle, I’m going to say poorly architected, but I don’t mean that as a direct criticism. Maybe atrophied code. It’s not usually a deliberate effort. We want to build bad code and we want to build bad products and we’re bad people because we built bad products. That’s not what we’re talking about. It’s something that happens slowly over time. It’s like a drift. It’s like when your tires on your car are not aligned properly, when the tracking is not aligned properly, right? Maybe it’s just a little bit out. Maybe it’s just a degree. That could be quite a lot actually. Maybe it’s just a fraction of a degree out. And that means the wear on your tires is unnatural, right? And you maybe have to replace your tires more often. So, it’s costing you more money. And you’re trying to figure out why is this costing me more money? Well, and it’s £100 a tire and you’re doing that every, let’s say every year you’re having to replace these tires. You’re like, what’s going on?

But if you paid £40 to get the tracking aligned, then we don’t have that problem. Maybe our tires last a lot longer. Maybe they last 50,000 miles and someone, that’s actually a lot for tires. Maybe they last 20,000 miles instead of 5,000 miles, right? So that’s the type of thing we’re talking about. How do we fix the tracking on our software, right? Well, we need clean code. We need observability. We need CI/CD ideas. We need to automatically build our product. We need automated testing. We need to test as much as possible. This is that whole idea that fixing the tracking is the idea of also shifting left. You might have heard that terminology, but shift left is about moving things as close to the developers writing the code to figuring out that it doesn’t work.

I always use the example of the Azure DevOps team because they started with a massive legacy product. Very low, I’m going to say low code quality. I think they probably agree with that. Not great code quality. And very poor testing capability. Most of the testing was long-running system tests, like things like Selenium tests today, but they had their own system. And that’s just not good enough anymore because they take too long to run. It takes too long to find out, for the developer to find out that they’ve broken something. And if it takes too long, they’ve lost the context. And if they’ve lost the context, they have to relearn that context. And that’s a massive cognitive effort that could have been used to work on other things, right?

So we want fast tests and they were, I think they were either 48 or 72 hours to run their full test suite against the stuff they were building as Azure DevOps. Not in Azure DevOps actually, Azure DevOps being building that product. And they spent a bunch of effort. It took four years because they’re doing a little bit at a time, right? They’re paying back a little bit of that friction every sprint. And over four years, which I think is 80 sprints. They did three week sprints, so you can probably figure it out. But over those 80 sprints, they paid back a little bit of time and they went from 48 to 72 hours to find out whether they’d broken anything to three and a half minutes.

So a developer can run the entire test suite for the product in three and a half minutes on their local workstation. And that includes having a full environment running to be able to run those types of, not just the fast unit tests, but that includes some integration tests as well. And that is what makes things better. That’s what makes things faster. The quicker, shortening the feedback loops. This is the fundamental premise of DevOps, of agile, is shortening the feedback loops. DevOps is about shortening the feedback loops in engineering. Agile is about shortening the feedback loops in business, right?

So if you want your teams to build better and deliver faster, you need to start with engineering excellence.