The Common Challenges of Adopting DevOps Practices

Published on
6 minute read

When organizations embark on the journey of adopting DevOps practices, they often encounter significant challenges. One of the most common is what I like to call “regression” – the frustrating experience of making two steps forward only to fall five steps back. Let’s dive into these challenges and explore how to effectively navigate them.

The Evolution of Risk Management

From Traditional to Modern Risk Profiles

In the traditional software development world – or as I sometimes refer to it, “the old days” – we had a very structured approach to managing risk. We would spend a substantial amount of time designing the product, defining the architecture, listing out features, and then marching toward a release date. This was followed by rigorous testing, usually performed by a separate test team, and finally, a handoff to an operations team for deployment. This approach was heavily reliant on a waterfall model, where each phase followed the previous one in a linear fashion.

However, the risk profile in this traditional model is vastly different from what we face today. In the old model:

  • Design and planning were extensive, often leading to a long development cycle.

  • Testing was a separate phase, with dedicated teams responsible for ensuring the product’s quality.

  • Deployment followed a strict sequence of environments, often starting with staging, then UAT, and finally production.

This method had its merits, but it also had significant drawbacks, particularly in today’s fast-paced environment where flexibility and adaptability are key.

The DevOps Shift

DevOps changes the game entirely. It demands a new approach to risk management, one that prioritizes:

  • High quality: Ensuring that every release meets the highest standards.

  • Flexibility: Adapting to changes quickly without sacrificing quality.

  • Adaptability: Continuously improving based on real-world feedback.

In the world of DevOps, we no longer have the luxury of long development cycles. Instead, we’re working in a continuous delivery environment where speed is of the essence. This brings us to one of the biggest challenges in adopting DevOps: managing risk in a way that doesn’t slow us down but rather enhances our ability to deliver value rapidly.

The Importance of Feedback Loops

Closing the Loop: The Key to Success

One of the most critical aspects of DevOps is the concept of closing feedback loops. In the traditional model, feedback often came too late, after the product was already in the hands of users. This delayed feedback was costly and inefficient.

In DevOps, the goal is to:

  • Identify feedback loops: Determine where and how feedback can be gathered throughout the development process.

  • Eliminate waste: Streamline processes to ensure that feedback is gathered as quickly and efficiently as possible.

  • Close the loop: Act on feedback in a timely manner to continuously improve the product.

A great example of the importance of closing feedback loops can be seen in the development of Windows 8. Microsoft invested hundreds of millions of dollars in user experience studies, labs, and testing, yet the product still failed to resonate with consumers. The lesson here is that no matter how much testing and validation you do in controlled environments, the only place to truly validate a product is in production.

“There’s No Place Like Production”

As one of my favorite people, Brian Harry, who used to run the Azure DevOps team, famously said, “There’s no place like production.” No matter how much testing or validation you do, especially in a service-oriented world with thousands of users, you can’t simulate production. The real test of your product comes when it’s in the hands of your users, operating in the real world.

This is why closing feedback loops is so crucial. It’s not just about collecting data; it’s about acting on that data to improve the product continuously. In a DevOps environment, feedback needs to be immediate, actionable, and directly tied to the product’s success in the market.

The Challenges of Modern Risk Management

Adapting to a New Risk Profile

In today’s DevOps-driven world, the risk profile is continuously evolving. We no longer have six months or a year to test a product before release. The Windows team, for example, used to operate on a six-year delivery cycle. Imagine that – six years to build, test, and release a new version of Windows! But those days are long gone.

Now, we need to:

  • Understand our risk profile: Recognize that the risks we face today are different from those we faced in the past.

  • Mitigate risk effectively: Use modern tools and techniques to manage risk without slowing down the delivery process.

  • Adapt quickly: Respond to changes in the market and user feedback in real-time to maintain a competitive edge.

A colleague from Boeing once shared an insight that resonated with me: “Boeing doesn’t build quality in; they test quality in.” This aggressive testing approach is necessary in some industries, but in software development, we have the advantage of being able to release products quickly and gather real-world feedback.

Embracing Rapid Iteration

In software, we don’t have to suffer from the same challenges as industries like aerospace. We can:

  • Release quickly: Get the software out the door as fast as possible while maintaining the necessary level of quality.

  • Test in the real world: Validate the software in production to see how it performs in the market.

  • Adapt based on feedback: Use market feedback to determine whether the product is increasing or decreasing market share, and adjust accordingly.

This rapid iteration and adaptation are at the heart of DevOps. It’s about closing feedback loops and using that feedback to continuously improve the product. It’s a mindset shift from the traditional model, where testing was a separate phase, to a modern approach where testing and validation are ongoing processes.

Overcoming the Common Challenges

Closing Feedback Loops

As we’ve discussed, one of the most significant challenges in adopting DevOps is closing feedback loops. To overcome this challenge:

  • Prioritize feedback: Ensure that feedback is gathered at every stage of development, from initial design to post-release.

  • Act on feedback: Don’t just collect feedback – use it to make informed decisions and improve the product.

  • Integrate feedback into the workflow: Make feedback a natural part of the development process, not an afterthought.

Managing Modern Risk

Another common challenge is managing risk in this new, fast-paced environment. To do this effectively:

  • Understand the new risk profile: Recognize that the risks we face today are different from those in the past.

  • Use modern tools and techniques: Leverage the latest DevOps tools to manage risk without slowing down the delivery process.

  • Adapt to changes quickly: Be prepared to pivot based on real-world feedback and market conditions.

Conclusion: Embrace the DevOps Mindset

Adopting DevOps practices is not without its challenges, but the benefits far outweigh the difficulties. By closing feedback loops and managing modern risks effectively, organizations can achieve:

  • High-quality releases: Deliver products that meet the highest standards of quality.

  • Flexibility and adaptability: Respond to changes in the market quickly and efficiently.

  • Continuous improvement: Use real-world feedback to improve the product and increase market share.

The journey may be challenging, but with the right mindset and approach, your organization can successfully adopt DevOps practices and reap the rewards. Remember, there’s no place like production – so embrace the feedback, manage the risks, and keep moving forward. 🚀

I think the most common challenge that organisations face when trying to adopt DevOps practices is regression. I think that’s probably the best way to describe it: you make two steps forward and five steps backwards. Part of that is how do we maintain our ability to control risk within this new context because the risk profile is different.

Because the risk profile is different, we need different tools and techniques to manage that risk. In the old days—I’m calling it the old days; it might be the current days for lots of organisations—but in the old days, we would spend a bunch of time designing our product. We would design the architecture, we would decide what we’re going to build, we would list out all of the features, and then we would work towards some kind of release date of our product.

Once we got close to that release date, once we delivered a body of features that made sense, we were going to test those features. We had a different group of people who were testing and validating those features. I mean, it is still quite common to have separate test teams from engineering teams rather than a combined engineering force. Then, once they had done those tests, it was probably handed off to some kind of operations team who were going to deploy it to an environment.

Then something like UAT was going to start, where you had some kind of additional validation of what’s going on in the product. Once all of those things were successful, then it moved on to maybe deploy to production or staging, or you might have other environments. The traditional, most common model is deploy by environment, and that fundamentally doesn’t work within the context of DevOps. It’s going to slow you down.

It’s not going to get you the key thing that DevOps is trying to bring to your business: high quality, flexibility, adaptability, and your ability to validate assumptions. We have an assumption that this feature is going to be valuable, but in order to actually validate it, no matter how much stuff we do on paper or in labs or in studies, the only place to really validate that feature is in production.

A great example of that is Windows 8. Microsoft spent hundreds of millions on user experience, on labs, on flying people into labs to video them using the product, performing certain tasks, and then getting feedback from them in interviews or sitting with people that are using the product. They did all of the things that you’re supposed to do, and still, they ended up with a product that bounced off their consumers from a usability perspective.

That is because of that production problem. One of my favourite people, Brian Harry, who used to run the Azure DevOps team, made this comment that I loved: “There’s no place like production.” No matter how much testing, no matter how much validation you do—especially if you’re in the service world where you’re building a service that’s got thousands and thousands of customers using the same service—no matter what you do, you cannot simulate production.

We can maybe do some stuff, but that common challenge is how do we address these things as that world is changing? We’ve no longer got six months or a year. The Windows team used to be on a six-year delivery; that’s how long it would take them to get a new version of Windows out the door, from starting to write the code to it actually being released to production.

They had six years to do testing. I had a colleague from Boeing, and he talked about one of the things that he saw as a problem: Boeing don’t build quality in; they test quality in. You’ve got all of this aggressive testing that’s happening as part of building something, and then you’re testing it. “Oh, right, it failed.” So what do we have to do differently?

That is quite often what you have to do with some manufacturing stuff these days. You’ve got simulators, so you can do a lot more, but in the software world, we don’t have to suffer from that problem. We don’t have to spend lots of time building a rocket to put it on a rocket pad and launch it, and then it explodes, and we look at the telemetry and figure out what went wrong.

We don’t have to do that. We’re building software; we can get that software out the door as quickly as humanly possible with the level of quality that we need to maintain our business brand, protect our business, protect our consumers, and protect our producers. We can get things out the door as quickly as possible, test it in the real world, test it in the market, find out how accepted that thing is in the market, whether it’s increasing our market or decreasing our market, and then adapt around that.

That’s part of that DevOps story: closing those feedback loops, not just identifying those feedback loops and eliminating waste in the process to get it to go through the process as quickly as possible, but actually closing the feedback loops, not just collecting the data.

The two big common challenges that I see are, one, closing the feedback loops, and the other one is actually getting how do we change the way we understand our risk profile and how we mitigate and organise around risk within that context. Then, how do we adapt to the things that we see in a timely manner? Those are the two most common challenges, and those are the things that we can help teams, products, and your organisation deal with and figure out how to do better.

People and Process Software Development Software Developers Market Adaptability Operational Practices Pragmatic Thinking Resilience and Change Product Delivery Value Delivery Complexity Thinking

Connect with Martin Hinshelwood

If you've made it this far, it's worth connecting with our principal consultant and coach, Martin Hinshelwood, for a 30-minute 'ask me anything' call.

Our Happy Clients​

We partner with businesses across diverse industries, including finance, insurance, healthcare, pharmaceuticals, technology, engineering, transportation, hospitality, entertainment, legal, government, and military sectors.​

Slicedbread Logo
Emerson Process Management Logo
Cognizant Microsoft Business Group (MBG) Logo
Freadom Logo
Epic Games Logo

CR2

Bistech Logo
YearUp.org Logo
Deliotte Logo
Sage Logo
Philips Logo
Flowmaster (a Mentor Graphics Company) Logo

NIT A/S

Healthgrades Logo
Boeing Logo
Illumina Logo
Slaughter and May Logo
New Signature Logo
Ghana Police Service Logo
Washington Department of Transport Logo
New Hampshire Supreme Court Logo
Royal Air Force Logo
Department of Work and Pensions (UK) Logo
Nottingham County Council Logo
Lean SA Logo
Epic Games Logo
Graham & Brown Logo
Illumina Logo
Teleplan Logo
Boxit Document Solutions Logo