DevOps: Elevating Your Organization’s Performance Through Bespoke Solutions

Published on 16 August 2024

Written by Martin Hinshelwood

6 minute read

Link copied to clipboard!

In the fast-evolving world of software development and operations, no two organizations are the same. Every company, every team, and every goal is unique, requiring a tailored approach to DevOps that aligns with specific needs and objectives. In this post, we’ll explore how understanding your current state, streamlining practices, and adopting the right tools can lead to higher quality, more frequent deliveries, and reduced friction in your software development process.

Understanding the Current State with DevOps Reports

Before setting out on any journey, it’s crucial to know where you are. The same principle applies to your organization’s DevOps journey. Understanding your current state is the first step in determining the direction you need to take. This is where a State of DevOps Report comes into play.

Why Is It Important?

Baseline Assessment: The report helps you understand your current practices based on modern engineering and DevOps standards.
Informed Decisions: Knowing your starting point allows you to make informed decisions about where you want to go and how to get there.
Alignment: It ensures that everyone in the organization is on the same page regarding the current state and future direction.

Key Areas to Evaluate

When conducting a State of DevOps Report, consider the following areas:

Release Processes: How are products being released? Are there bottlenecks or inefficiencies that need addressing?
Tooling: What tools are currently in use? Are they the right fit for your needs, or is there redundancy that could be streamlined?
Ground-Level Practices: What’s happening on the ground, not just at the leadership level? This involves understanding the real processes used by the teams.

Example: I once worked with an organization that had 13 different source control systems in use for a single product. The waste was staggering, and it was clear that streamlining this process would lead to significant improvements.

Evaluating and Streamlining Current Practices

Once you understand where you are, the next step is to evaluate and streamline your practices. This process often reveals dysfunctional behaviors—what I prefer to call “opportunities for improvement”—that can be corrected to enhance efficiency and quality.

Common Issues and How to Address Them

Branching and Merging: One organization I worked with had a team of 30 engineers who understood branching but not automatic merging. They were manually copying code between branches—a process that was not only inefficient but also prone to errors.
Custom Tools: Another company had developed a custom tool that allowed product managers to create builds with selected features. While it seemed like a good idea from a business perspective, it was highly problematic from an engineering standpoint. The tool led to inconsistencies and a lack of cohesion in the final product.

Steps to Streamline

Solo Interviews: Conduct solo interviews with team members to uncover the real issues. People are more likely to speak honestly when they’re not in a group setting.
Automated Processes: Embrace automation where possible. For example, understanding and implementing automatic merging can save countless hours and reduce errors.
Tool Rationalization: Assess your current tools and eliminate redundancies. Streamlined tools lead to streamlined processes.

Overcoming Compliance Misconceptions

One of the significant challenges many organizations face when adopting DevOps practices is compliance, particularly with regulations like Sarbanes-Oxley (SOX). A common misconception is that compliance requirements necessitate a separation between development and operations, which can hinder the adoption of DevOps practices.

The Reality

SOX Compliance: It’s a myth that SOX compliance requires a strict separation between developers and those who release code. In reality, it’s possible to meet SOX requirements while still embracing DevOps principles like continuous integration and delivery.
Education and Coaching: Part of our role in helping organizations adopt DevOps is to educate and coach both technical teams and business leadership on how to meet compliance requirements without sacrificing the benefits of DevOps.

Example: Automatic deployments are often resisted due to compliance fears. However, with the right controls in place, it’s entirely possible to automate deployments while staying compliant with SOX and other regulations.

Benefits of DevOps: Ownership and Quality

One of the most significant benefits of adopting DevOps is the shift towards ownership. When teams are responsible for the entire lifecycle of an idea—from development through deployment and monitoring—it leads to higher quality and more frequent deliveries.

Why Ownership Matters

Continuous Delivery: DevOps enables more frequent, higher-quality deliveries by fostering a sense of ownership among teams.
Reduced Friction: When teams are responsible for the entire process, there’s less friction in getting software out the door.
Closing Feedback Loops: Ownership ensures that feedback loops are closed, leading to continuous improvement.

Practical Steps to Enhance Ownership

Empower Teams: Ensure that teams have control over the entire lifecycle of a feature or product.
Monitor Quality: Implement monitoring tools to track the performance and quality of deployed features.
Deploy Strategically: Avoid deploying on Fridays. Instead, aim for Monday mornings when the entire team is available to address any issues that arise.

Practical Steps for Effective DevOps

To successfully implement DevOps in your organization, it’s essential to adopt practical steps that align with your specific needs while maintaining quality and compliance.

Tools and Compliance

Compliance Tools: Use tools that help you meet compliance requirements without sacrificing the agility of DevOps.
Quality Assurance: Shift your focus from User Acceptance Testing (UAT) as a cost center to ensuring high-quality delivery from the start.

Case Study: Learning from CrowdStrike

A recent incident with CrowdStrike highlights the importance of controlling the blast radius during deployments. When deploying new features, it’s crucial to start small and gradually expand to avoid widespread disruption.

Controlled Deployments: Deploy to a small, low-impact group first to monitor performance before rolling out to the entire organization.
Strategic Timing: Deploy on Monday mornings, not Fridays, to ensure that any issues can be addressed promptly.

Example: Even organizations with high-quality products, like Windows, control their blast radius by deploying to a small group before a full rollout. This approach minimizes risk and ensures a smoother deployment process.

Conclusion

DevOps is not a one-size-fits-all solution. Every organization is different, and the approach to DevOps must be tailored to fit specific needs and objectives. By understanding your current state, streamlining practices, and adopting the right tools, you can unlock the full potential of DevOps—leading to higher quality, more frequent deliveries, and a more efficient software development process.

🔍 Key Takeaways:

Know Your Starting Point: Use a State of DevOps Report to understand where you are before setting out on your DevOps journey.
Streamline Processes: Identify and address dysfunctional behaviors to enhance efficiency and quality.
Embrace Ownership: Empower teams to take full ownership of the development and deployment process.
Stay Compliant: Meet compliance requirements without sacrificing the agility and benefits of DevOps.

By following these principles, your organization can successfully navigate the complexities of DevOps and achieve a more streamlined, efficient, and effective software development process. 🚀

We’ve been doing DevOps for quite some time. It’s really difficult to provide an overview of our DevOps services because our DevOps services tend to be bespoke for every single customer that uses them. There are some patterns, but there are not that many formulas that work everywhere. Every company is different, every group of people is different, and every outcome that they’re trying to achieve with the philosophy that is DevOps is different.

But there are some patterns that do kind of make sense. One of those is a state of DevOps report. The idea is you want to understand, as an organisation, where we are based on modern engineering and DevOps practices. Where do we sit? Because it’s difficult to understand where you want to go unless you know where you are right now. That’s the first thing you do when you have a map and you’re trying to figure out how to get somewhere; you have to find out where you are so you can then figure out what the direction to go is.

So that kind of state of DevOps, state of agile, where are we right now in our understanding of or our application of DevOps practices? We might talk about things like how products are being released. We might talk about what sort of tools are being used within that context of releasing products. I worked with an organisation that had 13 different source control systems in use for one product. There’s a lot of waste there that can be identified.

What are we actually doing? How do we take the ideas that we’ve created in our product, and how do they actually get into production? Not just what leadership thinks is happening, but what’s actually happening on the ground. We tend to interview a bunch of people in the organisation, usually trying to do solo interviews because people are more willing to speak when there’s not somebody else present. This kind of anonymous information allows us to find out all sorts of things.

For example, I worked with an organisation where their team of 30 engineers understood branching, but they didn’t understand automatic merging. So they were manually copying code with a text diffing between branches in order to merge code, which is just mental. I worked with another organisation that had built a custom tool because the business had asked for a capability. They built a custom tool that allowed product managers to pick features and create a build of the product that only had those features merged together into the product.

This might sound like a good idea if you’re on the business side, but from an engineering standpoint, creating a cohesive, valid, high-quality, usable, high-value product is totally insane. There are all sorts of non-dysfunctional behaviours; I tend to call them opportunities for improvement because it’s a little bit more positive. But the reality is they’re dysfunctional behaviours that are really common in the industry and have been for many years because people are pushed towards solving a problem, and they don’t necessarily know that there are known logical ways to solve that problem already.

Even if they do know about those non-logical ways to solve that problem, somebody in the business has an objection to it. A great example is automatic deployments. The objection might be that we’re under Sarbanes-Oxley audit; we’re Sarbanes-Oxley compliant, therefore we need a separation of developers from the people who do the releasing of the code. That means we can’t have developers with access to production.

That’s actually not true with Sarbanes-Oxley; not true at all. But because somebody believes it is, you then end up with these ideas where development and operations are separate. One of the biggest values that we bring, the biggest benefit that we bring to organisations when we’re talking about the DevOps philosophy, is that ability for higher quality, more frequent delivery, and less friction in your ability to get software out the door.

Some of it’s about ownership. If it’s you, you have an idea, you’re going to build that idea, you’re going to deploy that idea, you’re going to monitor that idea, and you’re responsible for the upkeep of that idea, the support, and the continued value add of that idea. That should all be within the control of the group, the team.

So, IDE all the way through to deployment and closing those feedback loops should be part of that story, and that’s very difficult to do within a lot of organisations. So what are the steps that you can take to get towards that? What are the tools that you can use to ensure that you meet the compliance that you need to meet in order to continue to meet your compliance but do these new things? How do you maintain quality?

How do you maintain the level of quality? If, for example, you’re not doing UAT because UAT is a cost centre, not a value centre for software engineering, if your UAT is a value centre because it finds stuff and protects your business, then we’ve not got high enough quality delivery of our product. But if we create that high enough quality and UAT becomes a cost centre, we want to get rid of those cost centres.

How do we ensure that we don’t end up in the same situation as CrowdStrike? There’s a great example of an organisation recently in the news that caused massive disruption. I think they’re going to be sued by Delta for 500 million dollars that Delta has lost over the week that they had a problem, which is partly their own problem with DevOps.

What could CrowdStrike have done differently? One of the things they didn’t do was control the blast radius. You don’t just deploy to production; you don’t just deploy to 100 machines. I’m not sure exactly how many machines run at CrowdStrike, but you don’t deploy to all of them at the same time. That’s insanity. Even Windows, who have very high quality in the product that they ship, deploy to a small number of people first.

They control the blast radius. What are our lowest impact customers or customers that would be impacted the least by an outage? Deploy to them first. Perhaps deploy to our own systems first and run it for a few days to see if there are any problems, monitoring that feature and that capability that’s being deployed.

Not deploying on a Friday; the best time to deploy is Monday morning when everybody comes in because you get a whole week to fix any problems. Deploying on Friday is probably not the best idea. There are all kinds of things that make logical sense and are within the context of this DevOps story that we can help educate, coach, and engage with your team, your engineers, as well as with your business and your leadership to understand what these things are, what value you get from them, and how you maintain your ability to control risk within this new world of continuous delivery to production.

Pragmatic Thinking Software Development Software Developers Value Delivery Product Delivery People and Process Deployment Strategies Practical Techniques and Tooling Deployment Frequency Operational Practices

Link copied to clipboard!

Connect with Martin Hinshelwood

If you've made it this far, it's worth connecting with our principal consultant and coach, Martin Hinshelwood, for a 30-minute 'ask me anything' call.

Our Happy Clients

We partner with businesses across diverse industries, including finance, insurance, healthcare, pharmaceuticals, technology, engineering, transportation, hospitality, entertainment, legal, government, and military sectors.

CR2