I recently penned a post titled “Stop Hiding Behind Complexity and Start Delivering Continuously,” and I want to expand on that idea here. The crux of my argument is straightforward: if anyone in your organisation claims that your software is too big or too complex for continuous delivery, they are either lying, being lazy, or simply don’t know how to make it happen. More often than not, it’s the latter—they just don’t understand how to transition from their outdated, cumbersome systems to a more agile, continuous delivery model.
The Path to Continuous Delivery
Let’s be clear: achieving continuous delivery is not an insurmountable challenge. It requires a genuine organisational commitment to improving both the quality of the product and the speed at which features are delivered. This isn’t about throwing more people at the problem; it’s about getting your product into shape.
- Assess Your Product: If your software resembles a lumbering, 500-pound gorilla, it’s time to rethink your approach. A bloated product cannot adapt quickly to new demands.
- Commit to Change: Just as you would if you were trying to get fit, you need to commit to changing your software’s architecture and processes. This means investing time and effort into making your product leaner and more responsive.
Drawing Parallels with Personal Fitness
Consider your own health. If you aspire to achieve a physique like Hugh Jackman in Wolverine, you wouldn’t expect to get there without significant lifestyle changes. You’d need to:
- Revamp Your Diet: Eating healthier is crucial.
- Exercise Regularly: A consistent workout routine is essential.
- Adjust Your Habits: You might need to change your daily schedule to accommodate your new lifestyle.
The same principles apply to your software. If you want it to be agile and capable of rapid feature delivery, you must put in the work to ensure it reaches that state.
The Challenge of Legacy Systems
I often see organisations, particularly in industries like aviation, struggle with legacy systems. These systems are like old mainframes that have been patched over the years but never truly updated. For instance, I’ve heard stories about American Airlines having to pull retired engineers out of retirement homes to fix their outdated systems. This is a clear sign of a knowledge gap and a lack of organisational will to modernise.
- Recognise the Risks: When systems are neglected, they become ticking time bombs. The recent outages at major airlines highlight this risk. They often stem from a failure to maintain and upgrade core systems, leading to catastrophic failures when external issues arise.
- Invest in Resilience: To avoid these pitfalls, organisations must invest in robust application lifecycle management. This means not just building a system and walking away but continuously improving and maintaining it.
The Importance of Continuous Improvement
The reality is that software development is an ongoing process. Just as maintaining fitness is easier than achieving it, keeping your software in good shape is less challenging than getting it there in the first place. However, it still requires consistent effort and investment.
- Learn from Others: Take a page from the Windows team’s playbook. They transformed their delivery process from a six-year cycle to continuous delivery by addressing long-standing issues in their codebase. This required a commitment to change and a willingness to tackle the technical debt that had accumulated over decades.
The Call to Action
So, what’s the takeaway? No matter how complex your systems may seem, you can move towards continuous delivery. It’s not just an engineering problem; it’s a cultural one.
- Foster Organisational Will: Leaders must recognise the value of investing in their systems. This means allocating resources—time, effort, and money—to fix existing problems and prevent future ones.
- Don’t Ignore the Signals: If you notice signs of trouble, such as frequent outages or slow delivery times, take them seriously. These are indicators that something needs to change.
In conclusion, the complexity of your software should not be an excuse for stagnation. With the right mindset and commitment, you can overcome these challenges and achieve a state of continuous delivery. Don’t hide behind complexity—embrace the work required to simplify and improve your systems. The rewards will be well worth the effort.
So I just wrote a post called stop hiding behind complexity and start delivering continuously. The premise of the post is that any person in any organisation working on any software anywhere in the world that says our software is too big and too complex to be able to deliver continuously is either lying, lazy, or they don’t know. Probably they don’t know how. The most likely scenario is they think they don’t understand how to get from where they are right now with their mainframe-backed, old, crusty software to continuous delivery.
We don’t know how to get there, but you can get there if there is an organisational will to improve the quality of the product. If there is an organisational will to improve the speed of delivery of features in the product, right? Because improving the delivery of features in the product is not about adding more people; it’s about getting your product in shape.
If you’ve got this huge, lumbering, obese product, it’s not going to be able to do the things quickly that you want it to be able to do. People aren’t going to be able to get it to do the things that you want it to do quickly. It’s lumbering; it’s the 500-pound gorilla in the room, right? But it doesn’t have to be.
How would you, as an individual, if you were not where you wanted to be health-wise, how would you get healthy? How would you get to that goal that you wanted? Perhaps you want to look like Hugh Jackman in the Wolverine, right? How would you get from where you are right now to there? Because it is possible. I mean, you wouldn’t look exactly like him because everybody’s body type is different, but how would you get that type of physique, that level of body fat?
Well, you’re going to have to work. You’re going to have to change all aspects of your life in pursuit of that goal, right? You’re going to have to eat differently. You’re going to have to exercise differently. You’re going to have to change when you do things, how you do things in order to get to that physique you want.
If you want your software to get to that physique so that it’s easy peasy to add features to your product, then you’ve got to do the work to not just ensure that it gets there. That’s the tough part, right? If you’re incredibly unfit, getting to that fitness point is really hard. Maintaining it is easier, right? Once you are fit, let’s call it, once the product is fit, maintaining it in that position is definitely easier. It still requires investment and effort, and time to do things, but you’re not necessarily doing the same things that you had to do in order to get there. It should be easier.
And that’s where you see a lot of organisations and software struggling. I think the one that gets me is every so often the airlines falter. Someone once called it a scrumble, right? I know they may not be doing scrum, but it’s like a cross between scrum and stumble. You effectively trip over your feet, you fall on your face because something’s wrong with the way you’re doing, the way you’re running, the way you’re building your product, and you fall on your face.
If you just pick yourself up and start running again, you’re in danger of having exactly the same thing happen. You see that with the airlines, right? They have these old mainframe systems behind, under the covers, deep in the bowels of their systems, but they don’t understand. They don’t have people that can understand anymore.
I think American Airlines, I heard stories about pulling people out of retirement, going to find the last person who could fix the system, and they were in a retirement home, and having to pay them exorbitant amounts of money to come out and take them five minutes to fix it, right? Knowing where to poke because they’ve lost that knowledge, they’ve lost that capability, yet they keep that system. They’ve lost understanding of a system, and they keep it around, which is massive amounts of risk.
You see that American Airlines has been down, Delta’s been down, British Airways has been down, and those are the big ones that I can think of because they hit the news because they’re massive airlines. But the reason they’ve been down is because they’ve not cared for their application. They’ve done a one-and-done, right? You build it, you get to the end of the project to build it, and then that project is done. There’s no more funding; therefore, the product then starts to stagnate, atrophy, and decompose.
That’s what happens, right? I mean, the system keeps doing what it was always doing, but the systems around it are changing. The needs of the business are changing. The world has changed, and then you have to layer on more and more stuff to take care of the fact that this core of the system doesn’t necessarily work exactly like you need it to work, but you can’t change it anymore, or you feel like you can’t change it anymore.
And all of that is crap. It’s all crap. You absolutely can change it. You absolutely can fix it. You absolutely can make those problems go away. You can make it so you never go down again. You can make it so that when, what was it? It was CRCH strike happened, right? Wait, failure of DevOps, let’s not talk about that one. But when Crowstrike happened, I think it was Delta Airlines that was taken out by it, and it took them a week, even though Crowstrike fixed the problem within hours, right?
So theoretically, Delta was only out of action from the external source for hours. The implication of that outage was that they really struggled to get a whole bunch of systems up because they didn’t have good resilience, didn’t have good disaster recovery, didn’t have good stories around these things that were practiced and manageable and able to be delivered quickly.
So it took ages of people running around with their hair on fire to get stuff working, and I think some of the stuff didn’t come up and running for more than a week after the outage. That’s not good enough, right? That’s a business risk. That’s a time bomb waiting to happen in your organisation. Don’t have those systems. Those are not the right systems for if you’re running a multi-billion dollar business.
Those are not the right systems, man. If you’re running a little business, right? Because risk is what’s it called when it matches your size? If it’s a 10% risk that something’s going to go down, if you’re a small business, that’s a massive risk to your business. If you’re a huge multinational, it’s the same amount of risk, right? It’s 10% of your business.
So proportional, that’s the word I was looking for. Fix it. Get off your ass and fix it. And this is not just an engineering problem, right? Because you’ll probably find that there are engineers around this system. There are managers of those engineers around this system that know the problem. They know what it’ll take to fix it, right? They know those things, but there is no organisational will to fix it because it’s going to take time, effort, and money.
Time, effort, time, and effort cost money. It’s going to take time, effort, and money to deliver on those changes to fix those problems, and organisations don’t want to pay for it because they don’t see the value, right? They throw their toys out of the pram when something goes wrong and demand it be fixed and demand it never happen again.
But the only way for it to never happen again is they get out their checkbook and they write a check big enough to make sure it doesn’t happen again for the engineers to go fix the problem, to remove those systems, to replace those systems, to upgrade those systems, and to continuously upgrade those systems. They don’t have a good application lifecycle management, right? They don’t have it.
And that’s why I feel like they’re hiding behind the complexity, right? “Oh, this is too big and too complicated, and it’s too difficult to do.” No, it’s not. It’s really not. Just fix it. The Windows team went from delivering to production once every six years with massive amounts of legacy code. I know somebody who worked on the Windows team who was talking about what they were working on, and they’re the ones that were responsible for, do you remember when you used to copy a file or group of files from one folder to another in Windows? If there was one file that had an error, it would just crap out the process and stop, and you’ve copied half the files, and you don’t know which is which.
They went in and changed the file copying protocols so that it had resumability and it had the retry button and all of those things. That was their work, and the code that they were changing hadn’t been changed since 1985. This was maybe 10 years ago, right? So what’s that? 20 years had changed in 20 years. That’s not good enough.
And that’s why Windows became this continuous delivery story, because that’s how you build modern products. Continuous delivery forces you to deal with those problems because they magnify. They start magnifying, right? If you’re trying to do continuous delivery and you’ve got some slow-moving mainframe system in the back end that the team can only deploy once a month, and you’re deploying many times a day and you need changes from them, they’re not going to be able to keep up.
They’re just not. They’re going to become the bottleneck. They’re going to become the thing that you need to go fix. So go fix it. Don’t just look at it and go, “Yeah, I’ve got a problem.” I’ve got another blog post about ignoring the signals, right? That’s a signal in your organisation to say there’s something wrong here. You need to fix it.
And yes, it might take investment of money, time, and effort. Azure DevOps, when it was TFS, took about, I guess they went to deploying Azure DevOps overnight, getting it up and running within a couple of weeks. But to really pay back the technical cruft, the long years of just fire-and-forget code, right? To really start to pay that back took them almost four years of continuous effort.
Now they’re delivering features, and in fact, they were ramping up and delivering more and more and more features throughout that time, but they still had to continue that effort, and they continue that effort to this day to start breaking up some of those monolith pieces that make sense to break up. Because there’s nothing wrong with a monolith, but break them up so that they can deliver some things continuously and other things take longer.
And what you need is a business need for it. I can’t imagine that American Airlines is sitting there saying we’re okay with this ticking time bomb of risk that could happen at any moment, at any time, on any day. It could happen on Christmas Day; it could happen on our biggest sales day of the year, and we’re okay with that.
Nobody’s sitting there doing that. They just don’t want to pony up the dough and get off their ass and fix the problem. No matter how much complexity you have in your system, you can positively move towards continuous delivery.
