A Journey to DevOps Success
I’ve been asked to share my favourite DevOps consulting outcome, and this question immediately led me straight down memory lane to a unique project that went from chaos to one engineering system with an exceptional client. 🎯
So, today, I’ll share the story of my favourite experience of one of my most rewarding DevOps outcomes with a long-term client.
The question that triggered this trip down memory lane is, “What’s your favourite DevOps consulting outcome?
Well, let me explain how we navigated complex challenges and fostered a transformation journey - a tale you might find familiar if you’ve ever tried to manage a multi-billion-dollar software project!
I guess you could call this experience a journey of simplification and unity, and I hope it’s an experience that you find value in. 😉
1. Setting the Stage
Our story starts with a software project that began with a complex start and completely fragmented teams. This particular client was knee-deep in crafting a piece of software that was a multi-billion-dollar product.
With a profit margin running into billions, the leadership’s knee-jerk solution to most problems was, “Why not just hire more people?”
But as we all know, this isn’t always the most efficient solution.
The project landscape was vast and fragmented. Imagine 80 to 90 teams scattered across 13 locations in 9 different countries!
To make matters worse, each team operated independently with their own source control system. You could find everything, from Git to Subversion to even a homegrown Source Control System - a true testament to their problem-solving creativity.
2. The Build Puzzle - An Intricate Dance of DevOps
This particular puzzle of builds turned out to be an intricate dance of DevOps, where creating one working version of their product was like conducting an orchestra with 10,000 builds every day.
A fleet of heavy-duty servers and a dedicated team were constantly at work to ensure smooth operations. The process was intricate, to say the least, and was crying out for simplification. 🔧
3. Introduction of Agile and DevOps
My journey with this client started at this high point of complexity. My initial advice to this hardworking team was simple: restructure their processes. I proposed a simplified, streamlined process. I nudged them towards making their process easier, faster, and slicker.
To pinpoint the areas in need of improvement and to find a way to make it faster, easier, and less convoluted, as well as provide clarity, I curated a ‘State of Agile and DevOps’ report.
This report spotlighted their ‘opportunities for improvement,’ with their complex build problem taking the top spot. 🔄
My role was to guide, consult, coach, and help the people involved to cut through the complexity. However, I was there to consult, and they were the ones who had to get their hands dirty.
They were in the driver’s seat, understanding their code, the intricacies of their work culture, and the hurdles they needed to jump to bring about a change and set them straight on the path to transformation.
And boy, did they rise to the occasion! 🚀
5. A Unified Front - Triumph Over Fragmentation
Over the course of a marathon of four years, a period they jokingly referred to as a ‘company record’, we successfully moved everyone onto Azure DevOps. They now use Team Foundation Version Control, a single source control system.
This was undoubtedly a significant milestone.
This unification allowed everyone to work within the same structure, creating a harmonious symphony of codes instead of cacophony. 🎯
6. Building Towards ‘One Engineering System’
This transformation was only the first step towards migrating them onto GIT and optimising their processes further. The first hurdle they needed to cross.
The next mission?
To adopt the ‘One Engineering System’ concept, first coined by Microsoft. The idea was to create a system that allowed everyone to use the same basic procedures, even while the specific details of each product varied.
We needed to ensure everyone was on the same platform.
7. Azure DevOps is a Key Player
Azure DevOps became the unsung hero in our journey and this is precisely why I love Azure DevOps so much. This tool perfectly embodies and captures the essence of the ‘One ES’ philosophy.
It makes collaboration seamless, minimises confusion, and increases efficiency levels.
Again, this is why I love Azure DevOps and why it’s my go-to tool. ⚒️
8. Key Takeaways and Next Steps
This project was an educating adventure that underscored the significance of simplification and system unification in DevOps.
The best part?
Well, lessons were learned and the direction of future steps were set in place.
This experience of DevOps transformation in action is one that I’m proud to recount and it taught me valuable lessons about the territories of simplification and system unification.
And the beauty of this journey is that all organisations, regardless of size or industry, can embrace this approach, experience its transformative power, and reap the benefits. 🎓
Feeling inspired to embark on your Agile and Scrum journey?
Explore my Agile and Scrum courses, and let’s empower your team and revolutionise your organisation’s productivity together!
So the question is what’s my favourite DevOps consulting outcome and I really have to go to actually one of my favourite and longest customers. They were building a piece of software. I’m gonna try and not name them but they’ll know who they are when I’m talking about them. They were building a piece of software that was massive multi-billion dollar products. So ultimately leadership didn’t care about efficiency so much right when your profit margin is in the four to five billion dollars and you’ve only got 650 people working on it your answer to a lot of problems might be can’t we just hire a bunch of people to solve that right which is not always the best way to solve a problem.
And we were in the position that it was a very fragmented product so there was 80 to 90 teams working in 13 locations in nine different countries. They all did their own thing so each team picked their own source control system ran their own source control system locally. There was everything you could possibly imagine there was git but not very much of it there was team foundation version control there was subversion oh there was just everything there was even a source control system that they built themselves right because when they started building this product there were no good source control systems out there and they still had people in the company using their homegrown source control system lots of fun.
And in order to build this product because it was one product that was deployed to a machine right so you’ve got one not one executable but it for one I’m old right it was burned onto a CD and then you know that’s your install right so it’s one install for your product. In order to get that to exist they had to run a bunch of builds to go pull that code from all the different locations right pull it into some central location then actually do all of the builds that they needed to do and that was hugely complicated because every team did different stuff right so you had different tools different capabilities it was a multi multi-hour fandango to go do it and what they ended up having to do was about this is about 10,000 builds a day in order to create one working version of the product.
So they had a set of 20 between 20 and 40 build servers. All in order to build this product had to be the most unbelievably powerful machines you could possibly imagine as many cores as you could possibly fit as much memory as you could possibly fit it was just a monolithic product. And it ten thousand builds a day to have a nightly build of the product which needed a team of people to run this.
So I started working with them that was the state they were in kind of when I started working with them and my overriding advice was you need to simplify this process you need to make it easier faster slick because it’s just too hard to find out something’s wrong to find out you’ve broken something and one team breaks another team right all of those things.
So I did a state of agile state of DevOps report for them which is basically here’s what did I how did I phrase it here are the opportunities for improvement right that you have in your system and if you only fix three things here’s the top three things right and the top one was that build problem get them all into one source control system one set of builds and those kind of things and that effort because they’ve got to go do it right I’m the consultant I’m there to help them do it and I also help them with a whole bunch of other stuff as well right but day to day they’re the ones that have got to go do it I don’t understand their code I don’t know where all the bodies are buried and who they need to politically talk to in their company to get stuff done they need to go do all of that so I was there a little bit of coaching mostly consulting on other stuff while they’re doing that and then coming back around and having discussions and being part of me all that kind of stuff and it actually took and this was a record in this company four years to get them onto one source control system which was team foundation we got them all into Azure DevOps one source control system all on team foundation version control everything on one set of branches and everybody working in the same area so they’ll be able to clone the same clone what was it called create the same workspace in TFVC and I’ll work together and not poop on each other still had lots of branches still had 80 90 active branches but at least they were all related to each other and you could do the branching and merging properly and all that kind of stuff.
And that was the first stage to getting them onto getting them over onto git optimising their processes you know doing all of the other things that need to happen in order to have I there’s a phrase I love it it’s called onees. If you’ve not heard this phrase one engineering system it’s from the first time I heard it was from Microsoft and their one es strategy is part of the reason that Azure DevOps exists in the first place and that it’s a cloud platform and that’s that they had this problem internally and they wanted to you don’t want an A person who works for you to go work from one team and they move to another team and suddenly have they have to re-figure out how everything works they have to re-figure out where to store their work items they have to where all the work requests are how they build their code where they build their code where the code comes from all of that stuff shouldn’t be that should be like that’s the default stuff that we all understand.
And then the actual build that happens is different for every product in every platform but it’s in the same place right between those lines is different but the wrapper the how you get it done how what you need to install all of those things should be part of the organisational standards and that’s what one engineering system was all about was getting everybody onto one platform everybody onto one not way of working that’s the wrong phrase but one engineering system that can do all of the things that everybody in the company needed to do so that nobody’s confused about where their crap is and where they need to go to go change stuff and do stuff and collaborate with each other and that was the ethos of Azure DevOps and it met it absolutely fantastically.
Which is why I love that tool that is my go-to tool is Azure DevOps although I use other tools as well but just that vision from Simon Guggenheimer of that one engineering system of that Elm DevOps experience is just phenomenal.
Thanks for watching the video. If you enjoyed it please like follow and subscribe. I always reply to comments and if you want to have a chat about this or anything else agile scrum or DevOps then please book a coffee with me through Naked Agility.
