If you want to be able to ensure that your software performs well and creates a great user experience, you’re going to have to put in a lot of effort in making that happen. It is not going to magically happen. You’re not going to have awesome software with an awesome experience and fast and responsive without actually putting in the effort to figure out how to do that. Performance engineering has a huge impact on user satisfaction and the business goals you’re trying to achieve. Without a focus on performance engineering, and this is also shifting left, right towards where the team is doing the work, without that, you’re not going to be able to build a product that suits your customers. Customers are going to leave because it doesn’t perform well, or customers are going to leave because it doesn’t do what they need it to do, or customers are going to leave because it’s buggy and you’re not getting things there fast enough.

Performance engineering is where we’re looking at how quickly users are able to do things. Some of that could be user experience based, taking steps out of the process, and some of it can be does the software actually function at speed? Does it function at load? While some of these things can be load and stress tested, I’m going to quote one of my favourite product managers, Brian Harry, who said, “There’s no place like production.” No matter how much testing you do prior to production, you’re going to have problems that only exist in production. You can’t simulate real users; it’s not possible. You can only synthetically simulate users. So no matter how much of that stuff you do, you’re going to have problems when you hit production. They can be performance problems; they can be the way people use it problems, right? The order that people do things or the amount they do this particular thing is different from what we had in the simulation, and that’s where you run into a lot of your major performance problems.

My philosophy is that we want to get into production as quickly as possible so that we can figure out the impact of the changes that we’ve made and analyse the load and stress test within the context of real usage, real users using it for real. This is the story of testing in production, which doesn’t mean you’re not testing before production. Holy moly, no! Don’t ship stuff that’s not tested to production; that’s a bad idea. But we need to get into production as quickly as possible, which means we need to automate and create fast, sleek automated testing for all of the normal stuff that we need to test. Does the software actually function the way we expect it to? Then we want a ring-based or audience-based deployment model so that we can control the exposure of these new capabilities and features, and we do load and stress testing effectively in production.

So what that might look like, I’m going to use an example because I love the way that they’ve done this. I’ve worked a lot in the past with the Azure DevOps team at Microsoft. Seeing how they do things has been really enlightening because they’ve got quite a big product; it’s quite a big scaling problem. They’ve got millions of users, and what they effectively do is they have some of the rings that you’re deploying to are physical rings, like environments. But even once you’ve deployed a new capability to all of your production environments, let’s say you have six areas around the world, you’ve deployed to them all. Those features that are deployed might not be accessible by people, right? So they’re not impacting production in any way; their feature flag is off.

But then the team that owns that feature that’s deploying that feature wants to ensure that that feature is load tested, stress tested, is the right thing, right? Is the right thing that resonates with customers, provides the right capability, but also works load tested, stress tested? We agreed, well, I agreed, that we can’t do load testing and stress testing very well, not in production. So what we need to be able to do is we need to be able to limit the number of users that are accessing this new feature and then expand it over time. There are various ways to do that. What they would do is their first ring that they would create, and this is actually a ring for each feature. So they’re creating an audience-based deployment model per feature if it’s big enough, right? You might have features that you just ship; you might have changes that you just ship. You don’t do this with everything; this is expensive, right?

But what they do is the team will go, “Ah, we want to start getting real users into kicking the tyres on this feature.” It’s just us so far, just internal people. So what we’re going to do is we’re going to publish a blog post. They publish a blog post that says, “We’re working on this new feature; it’s going to provide this capability. Here’s what it currently looks like, and if you would like to help us kick the tyres, send an email to this email address, and we’ll enable it for you.” You give them the data of who you are, and then they enable it for you. You can then go in the tool; you would then have a little feature flag that you can turn on and off. When you choose to turn it off, you get asked for a comment, and it goes to the team, right? So they can find out why you don’t like it, right? Why you’re not using it. It’s slow, it’s this, it’s that. Then they can go look at the telemetry.

But that’s a very private preview, right? People have to actively opt in. Once they have enough data to say that they think this is a good feature, they think it performs well, at least with the small number of users, once we have enough telemetry, they then open it out. They’ll turn it off for everybody that’s using the service but with the feature flag enabled for everybody, so everybody can go turn it on. They’ll do another blog post that says, “We’re now ready for more people to try this. You don’t have to email us anymore; you can just go switch it on, go try it out, and tell us what you think.” So that encourages a bunch of people to go turn it on, and then they get a bunch of telemetry for people using it. Either people then continue to use it, or they turn it off because they don’t like it, right? Maybe it overrides the existing functionality, and they don’t like the way it is because something’s missing, so they got to turn it off. They’re asked what the problem is; they type in, “I don’t like it because it’s slow,” “I don’t like it because there’s features missing,” “I don’t like it because I don’t like my cheese being moved,” whatever the reason is, right?

And then they’re collecting more telemetry. If they get enough data, they maybe go to the next stage, or maybe they need to do more iterations on the capabilities, improve the performance. Maybe they need to turn it off because it’s not performing well, and then they need to do some performance improvements and then turn it back on again. Eventually, if they’ve got enough telemetry, they’ll turn it on by default for everybody with the option for people to go in and turn it off. So everybody gets it forcibly turned on, and then we’re collecting data and telemetry from the people that turn it off to find out why they don’t like it. Why do they go turn it off? What do we need to do to get those folks on board? Once they have enough telemetry, they see there’s not so many people turning it off. There are always going to be people that don’t like what you’re doing, right? So you can’t—it’s not a unanimous thing. They then have it on by default; you can’t turn it off, right? They maybe disable the option to turn it off, see what complaints roll in for people that want to be able to turn it off, and then eventually they get rid of the feature flag, and it’s on for everybody in production.

That’s a modern software engineering implementation of continuous delivery to production and then an audience-based rollout and expansion and testing story to allow you to do load testing and performance testing and stress testing in production. That takes discipline; that takes effort. It definitely has a cost, right? There are a lot of things that had to happen there and things that had to be organised and things that had to be done. But Azure DevOps, using that capability that they built into the system, using that story, they created a massive following from their users. Their users expected excellence, expected new things, expected those new things to work. As those new things rolled out, because you have an expectation if you opt into a private preview that there might be some things that don’t work quite as well or we don’t know yet, right?

So you’re not testing in production with users that haven’t chosen to be part of that test story, right? You’re soliciting for people to come in and help you, and because they want to help you, one, they’re more forgiving, and they’ll give more feedback, right? Because they’re choosing to be there. And that’s how you create a story of performance improvements and enhancing user experience in a progressive, modern engineering excellence manner. This is something that Naked Agility can help you build within your teams, build within your product. Don’t expect it to be magically overnight; these things take effort. They take discipline. Sometimes mistakes are made, but with a focus on delivering high-quality, usable, working product continuously to our users, we can build some of the best products you’ve ever seen.