So I first encountered DevOps as a developer at Marl Lynch, but I experienced the frustration that created the need for DevOps in many jobs beforehand. I started my career back in the early 2000s, 2001, and I was working for what was then called New Media agencies, which are just web development companies these days building websites. We did all the things you shouldn’t be doing. We deployed from our local machine, we edited in production. We did all of those crazy, nasty things because at the time, continuous integration wasn’t really a thing. Oh, it was around right at that time in 2001. I think continuous integration had been around for about nine years, but most people hadn’t really heard of it. Most people weren’t really doing it around then. Your experience will absolutely vary. There’ll be lots of people that were doing it, and I experienced the frustration of mistakes, of deployments that failed, of overwriting things you shouldn’t be overwriting, all of those kind of things.
When I moved to Marl Lynch, which is a big investment banking company, the tools were different. You didn’t have access to production. Once it’s production, you only had access to specific environments. You had to have specific deployment packages. You couldn’t make changes to certain types of systems like SQL Server. You had to give them a patch to be able to apply. When you’re in those restrictions and you try and do the same thing that you were doing before, because when I arrived at the part of Maryland that I was working at, they didn’t have any automation. They didn’t have any of these things. You experienced even worse frustration because you give somebody who doesn’t care one jot about your product, some DBA in I don’t know where they were. I know that Marl Lynch had five and a half thousand DBAs in the organization who you would give scripts to, and they would go run it against the databases, and nobody else was allowed to go near the databases.
But it meant that you would give them it. It would take them a half day to get to it, and they would tell you that the script didn’t work, and then you would have to go tweak the script and give it to them, tweak the script and give it to them, and it would just be a nightmare. It would take ages to get anything deployed. So what we started doing was thinking about how we could do automation, how we could create a more slick process where at the very least we were able to get into production with less hiccups. There wasn’t a lot of DevOps going on back then. It was called application life cycle management, and application life cycle management was both good ways of doing things and bad ways of doing things, but just under that guise of we’re going to be actively managing these applications.
It’s the term that then morphed into DevOps a little bit later on once DevOps was coined. So what we did was we introduced tools, we introduced ways of doing things, we introduced automation to our story, and I started using at the time Team Foundation Server as part of that integration automation. Everything linked together, traceability story for application life cycle management. We made it a little bit more professional. But while that was my kind of baptism into that very different environment where there were a lot of controls, we still didn’t do it very well. It really wasn’t until I started working more heavily in the Team Foundation Server community and I became a Microsoft MVP that I started engaging with other people who were in that category and really working towards making things more effective.
We started talking about how we could do the automation, how we could enable getting from code all the way to production without having people in the way, messing things up or other stories in the way, getting in, you know, other biases in the way, getting in the way of the thing you’re trying to deploy. That was really my introduction into proper DevOps. It was as a Microsoft application life cycle management MVP, working with other ALM MVPs within that category to really focus on that story. I found that I seemed to have an affinity for that, an affinity for engaging with development teams and helping them get better at that continuous delivery, continuous integration tools. Tools are not all of DevOps. Tools support DevOps, but they’re definitely a common route into towards DevOps because I really think of DevOps as the other side of the agile coin.
I.e., agile was something that was created by some people who were definitely engineers. You think of the signatories of the Agile Manifesto, but they were also much higher-level engineers. They were managers, and they were working on big products and projects, so they weren’t just software engineers building products. It was a much bigger thing, and they really thought of, came up with this idea of agile from that perspective. Some things seemed to be a little bit missing that were only implied in the Agile Manifesto but weren’t actually explicitly stated. This is the thing you need to worry about, and that for me, this is how I feel about it. I think that’s why DevOps was born. That’s why application life cycle management became DevOps, is because we really wanted folks from the engineering perspective who were trying to get to the same thing that the folks from the Agile Manifesto perspective were trying to get to as well.
That engineering focus means that there’s a lot more reliance, engagement, and support on tools, on processes, on systems because that’s how core engineering folks kind of think. That story kind of culminated in me co-authoring a book for Rocks called one of the big red books, engineering books called Professional Application Life Cycle Management with Visual Studio 2013. That kind of solidified my progress towards supporting that story. I don’t remember when it came about. It might have been around that time. It might have been a little bit later. A great DevOps person called Donovan Brown, who did eventually work for Microsoft, came up with a definition of DevOps that I think makes the most sense. It’s a little bit tools-focused, but not as well. I quite like it.
DevOps is the union of people, processes, and products to enable continuous delivery of value to our end users. So when you think about where DevOps came from, application life cycle management, the engineering side of delivering products, and that move towards realizing, and this was a big realization for me as well, is that tools—sorry, let me rephrase that—tools don’t solve problems; people do. People, with folks like myself with an engineering focus, very much focus on the tools and the code and the engineering, and we sometimes forget that the people are part of the story as well. We need to bring the people and the processes and the products together to create our system that allows us to build effectively, build products, and continuously deliver products to our end users.
There are kind of three primary challenges in that story of DevOps, which is what I experienced as I moved through them myself. I think the best way to experience them is to move through them yourself. We learn by doing. The first one is that culture and collaboration shift. If you think of the Agile Manifesto, people and interactions are more valuable than processes and tools, and that’s absolutely true even on the DevOps side. But DevOps, to a much greater degree, acknowledges that processes and tools support the people in doing the things that we need to do. I keep thinking of the Night Capital Group, right? That story about the investment firm listed on the New York Stock Exchange, $450 million cash in the bank to support their business.
They had a failed deployment because they were manually deploying. They had poor quality code. They had a poor quality deployment with just one person, and something went wrong that they couldn’t figure out. They started hemorrhaging money the moment they did their deployment and ended up, by the end of the day, they managed to figure out what the problem was with their deployment and fixed production, but by then it was too late. They drained their capital reserves. They didn’t have any cash flow anymore, and they had to file for bankruptcy. The reason we know so much about why they failed is because the bankruptcy filing has to list why they’re going into bankruptcy, and it detailed the problems that they’d figured out they ran into. They’d been reusing code that they thought wasn’t used for anything. They had not been focusing on their architectures and building up software quality, and they had one person doing the deployment with no backup or support to validate what it was that they were doing.
The person accidentally deployed to seven out of the eight servers that they were deploying to. That’s roughly the story. So that cultural shift on the people side of getting people on board with this idea that while tools and processes don’t solve your problems, they’re there to serve the people’s needs and enable them to make things that are really complicated and difficult, like deploying software, can often be really complicated and different. I build and maintain a product still, and I think there’s 20 steps in building the product and maybe five or six steps in deploying the product, and they all have to be done correctly in order for the output to work. I don’t want to have to walk through that list every time. Not only would it be time-consuming, but I’m likely to miss stuff. I’m likely to have my eyes go from step two to step four instead of one, two, three.
That happens all the time. I did training years ago for a bunch of testers, and I had made the assumption that if anybody in the world was able to follow a set of steps that they created to validate that a thing happens the correct way, it would be people who were trained as testers. But as it happens, that was not true. We had labs that they were doing in this training against a DevOps, and they would constantly go, “Wow, this isn’t working. I can’t, this doesn’t work. This is broken. Your lab’s wrong,” when in fact they missed a step. It’s even professionals whose job it is to follow those steps when you’re running things manually also miss those steps. It’s just the way the human brain works. It’s the way things happen. We’re not going to be able to fix it by focusing on the people, but we can build automation to help support that cultural shift.
But people have to accept that the people that are part of your organization have to accept that automation is the key, that automation in defining those processes is the key to supporting that story. Lots of organizations still prevent us from being able to deploy directly to production using automation. What we ideally want to do is go from developer checks a line of code into source control. It ends up in our deployable branch, whatever that is. We want to go from there all the way to production and rolled out to everybody without any human intervention because that’s our production line. The key shift here is from development to delivery. Think about the way car companies build cars. They’ve, and Toyota kind of pioneered this.
Toyota pioneered the ability to deliver cars. They’re taking something that they’ve designed and created and that works and delivering lots of it into production. That’s their production line. We can apply those ideas and mentalities to our DevOps production line, which is from we’ve created the thing and we validated that it works all the way through, and we can actually do the validation as part of that. But all the way through to getting it in the hands of real users. That’s not only that culture shift but also the toolchain integration and automation that needs to happen in order for that to work. Once you’ve got that toolchain and automation in place, we’re able to do this thing. We then need to figure out how to do it better. How do we optimize that production line to make it more effective?
The story I always think about is the Azure DevOps team, right? When they were the TFS team and they were moving towards continuous delivery from their waterfall mindset, the head of that group, Brian Harry at the time, decided to run a little test. They were creating a version of their product, and they had shipped their product all the way to production. It had gone through the gauntlet of lawyers and validation and signing and all of the gobbin that has to happen in between. It took them, I don’t know, 10 weeks, 20 weeks, whatever it was, to get their product to a state where it would pass the entire gauntlet that it had to go through, and then they had a deployable version.
He called it a null build. There were no changes to the code base. We’re just doing the same thing again. How long does it take to go through the process? You know upfront every one of those checks, so they had a concerted effort to look at their toolchain, to look at how they did the build and process, and what of these checks could they automate into the process so that they can then just give the artifacts and they sign them off? Or even if we have those artifacts and we pass these checks, will you automate that sign-off? Will you, as the person who has to sign off over here, just say, “I’m happy if you followed this process. I’m happy it all worked. All I was doing before was checking that you’d followed the process manually. If you can automate that, I’m good to go.”
They worked through that story inside of the organization of convincing all of those different people, and the build started to chunk down to a much smaller capability. I think they got it down that way to 48 hours was their automated pipeline. Then they started thinking about the advanced stuff. If your biggest problem is going from eight weeks to 48 hours, focus on that story. Getting all the approvals, getting all the engagement, bring it down. Now you’re at this 48 hours. What’s the next biggest thing we need to deal with? That’s part of that cultural shift. Not only have we got the toolchain involved, but we’ve got that cultural shift of what’s the biggest problem? What’s the most length of time that our process is taking, and how can we make it shorter and engage in that story?
Then you start thinking about more advanced stuff like our advanced automation and orchestration. The next thing that particular team focused on was why does it take 48 hours to run our tests? Well, we need the whole system up. We need to feed it with test data, and then we’re orchestrating across the entire system to do those tests. Is there a different way? Perhaps we can convert all those long-running tests that need data loaded, and maybe we can convert as many of them to unit tests as possible. So they worked for another four years getting rid of flipping that pyramid, right? From having most of their tests being the long-running automation and a few unit tests, flipping that pyramid until, in fact, after four years, all of their tests were unit tests, and they got that 48 hours down to three and a half minutes.
That’s three and a half minutes for a developer to validate that the product that they just made changes to still works the way it’s intended to work. That’s a total game-changer for organizations. Think about if you make a change and you don’t know for 48 hours whether it was the right change or not, or it worked or not. Do you work on other stuff and perhaps introduce more breaking changes, or do you stop and wait for that result and then continue on? That delay is a huge amount of waste in the system. Again, lean product delivery, right? You’re thinking about how you do that. Once you do get there and you’re confident you can get things into production, what about monitoring, logging, and observability?
Are you able to see what’s going on in production? Are you able to head off and find and trace issues before they become customer problems? Are you able to go and find out what happened when something goes wrong? Do you have the transparency that you need in your product? That’s a huge part of that toolchain and automation that it takes many, many months or years of effort to get your product to a state where you even have time to focus on that stuff. So monitoring and logging is really important. That’s telemetry, right? How much telemetry do you get out of your system? I only know the stats from about five years ago, but about five years ago, Visual Studio, the big Visual Studio product, was sending 7 and 12 terabytes of telemetry to Microsoft every day.
That’s telemetry on what features developers are using, how long the things that we’re using take, how long it takes to load, what buttons we click, all of those things so that they can build a better product. Because then you start thinking about, well, if we’ve got our advanced orchestration, we’ve optimized our pipeline, we’ve also optimized, we can see what’s going on because we’ve got telemetry and logging, and we can observe in almost real-time what’s happening in the system, then we can start talking about optimizing for scalability and reliability. These are all part of that DevOps story.
Then there’s the third thing. So we had cultural shift and collaboration. We had toolchain integration and automation. The third thing is continuous learning and skill development. None of that stuff I just talked about comes along overnight, and none of that stuff everybody already knows in your organization. Even if you start hiring in people that have DevOps skills, what do they really know? What do they really understand? We need to build these skills inside of our organization. Don’t think you can buy DevOps and install it, right? You can’t just bring in a consulting organization, tell them to install DevOps in your organization, and expect that to be the outcome.
The people in your organization have to have gone through the journey and had the battle scars to understand why they do things differently. They’re then incumbent in your organization, in your organization’s culture, and they’re able, as new people come in, to reduce the number of scars that the new people need inside of your organization in order to get on the same track. Does that make sense? It’s kind of like why we train recruits going into military organizations. You train them so that they don’t make the mistakes that people made in the past, things that we already understand aren’t going to work out well. That’s why we train them, so that having that incumbent knowledge in our organization and then having people come in and train and become part of that organization and that culture in our organization is really important to that story.
That’s kind of where I got to over, I guess, six or seven years. From 2006 to 2013, 2013 is around when I did the application life cycle management book. That was my story to that point. But then, as you learn new stuff, sometimes we do things because we’re told to do things, and we don’t know why we’re doing things. In order for true learning to happen, we have to understand the theory. That’s actually why training theory is so important inside of organizations, right? Because in order for the people in your organization to have the knowledge and skills to be able to make the best decision in circumstances they’ve never encountered before, because we’re doing complex cognitive work, we need to understand the theory of why those things are true.
Things like Little’s Law and the theory of constraints are part of that lean story. If we understand those underlying theory and concepts, we can start doing stuff and learning. We feed that learning loop through our theory, and it solidifies our understanding because the theory is the thing that helps tailor our way towards knowledge and understanding. The thing in DevOps, if you’ve not heard of it, is something called the three ways of DevOps. These are kind of the core tenets, I guess. They’re probably not universally agreed, but these are the core tenets in the way I think about DevOps. This is that perspective shift to what’s the theory that we generate, the body of theory we generate that we then build all this other stuff on top of because we’ve seen it working in organizations.
So then what are the commonalities? What are the theories that we can understand that will then work in everywhere else as well? Because everywhere else is different, so the theories are the bit that connects it all together. The first of the three ways of DevOps is systems thinking, amplifying feedback loops, and a culture of continual learning and experimentation. Systems thinking is really, really important. We can’t think about systems in isolation. If you think about your software as the system, and that’s what I’m working with, what about the people that are doing things around your system? What about the automations that are in place that build and deploy your system? The system is your entire organization and everything that impacts on your system.
So we have to look at all of those things. We have to take the blinkers off and raise that impact because if we don’t look at systems thinking as a wider view as part of DevOps and Agile as well, right? If we’re not thinking about systems and we think about just one small part, we might make optimizations for the part that don’t, in fact, result in a better outcome for the entire system. We end up making incorrect choices that maybe inhibit the rest of the system. I’m not sure that I’m explaining that well. Let me think about how I can explain that better. There are definitely things that I’ve done where I’ve made a change to the way I do things on the small scale that means that the bigger scale of things is harder to do.
An example of not thinking of the big picture in the IT world is where organizations have a centralized security department. The security department, in order to optimize their world of security, decides that everybody has to use, for example, a virtual machine. In order to maximize our ability to be secure, and this makes perfect logical sense, we want to 100% secure all of our stuff. Okay, nobody’s allowed to do work remotely. In order to interact with any of our systems, you have to be on our network, on our hardware. Therefore, we’re going to prevent anybody from doing anything else. Or you could do what Microsoft does, for example. When you’re working with Microsoft, you can buy your own physical hardware, join your own physical hardware to their domain. They then take control of that hardware and validate that it has all of the security requirements that they need.
They can remote wipe it, and suddenly you’re able to work with any local machine on their systems and be able to interact with a local machine. You’ve got full low latency access to everything, moving stuff around, that kind of thing. That is more effective. That’s a let’s take security with a wider picture of everybody needs to be able to do the job that the business requires rather than focusing on security as a blinkered view of just that thing. That’s an example of systems thinking. It’s widening that view and looking at the wider scope of things. The second one is amplifying feedback loops. We need to be able to not only have feedback loops but how do we maximize the value in those feedback loops?
How do we increase our ability to do stuff with those feedback loops? A great example is when you’re using telemetry. Do you have the right telemetry? Not just do we have telemetry from our product seeing how people are using it, but do we have the right telemetry with the right timings of the right things? How do we enable ourselves to get the right data to then make different choices on the way we build and deliver our applications to customers? Those feedback loops are hugely important. In Scrum, we have loads of feedback loops implementing that empirical process control system.
The third thing is that culture of continual experimentation and learning. How do we enable people to not get blamed for mistakes? We want to take accountability for our actions that we’re going to be doing as much of the right thing as possible, but in order to push the boundaries, in order to experiment, not all experiments are successful, and that’s normal. It’s normal that not everything goes well. So how do we create an environment within which we’re able to not just punish people for things going wrong but explore what was the root cause of things going wrong? How do we change the way our system works in order to enable less of those things that go wrong to happen?
How do we change the way we do things in order to minimize our recovery time and maximize our learning from the outcomes? These are the most important things in application life cycle management, which became DevOps, and that story of the other side of the coin from Agile. Agile coming from the business perspective, DevOps coming from the engineering perspective, both focused on trying to solve that same problem. At Naked Agility, we love DevOps. DevOps is how we engage with our customers. It’s how we enable them to be more effective. It’s how we get people working together, a combination of DevOps and Agile practices, which quite often are the same practices.
It enables us to help teams because there are well-known metrics that we can use in the DevOps space. If you look at the DORA report, you’ll find a bunch of metrics that you can use to get an understanding of where we are as an organization. Are we focused on the right things? Are we looking at holistic systems, or are we looking at isolated things? Are we amplifying the feedback? Can people even amplify their feedback loops, or is everything restricted and locked down, enclosed, and there’s no experimentation, no learning, no ability to change anything? No ability to do anything with those feedback loops, in which case people will stop using them.
Those are the things which enable organizations to not only understand where they are right now, organizations and teams, to understand where they are right now, but what’s the next most important thing they could be working on? Or what’s the next thing on their backlog of increasing their capability to successfully and continuously deliver products of the highest possible value? What’s the next thing they should be focused on to enable that to happen much, much quicker? If you’re trying to integrate DevOps into your team or organization, then Naked Agility can help you focus on the three ways and enable you to get the value that you and your customers deserve. Get in touch below.
