Mastering Agility: Balancing Engineering Excellence and Effective Processes in a Rapidly Changing Business Landscape

Published on
4 minute read

As I sit down to reflect on my recent experiences in the world of Agile and DevOps, I can’t help but think about the rapid changes we are witnessing in the business landscape. The velocity of change is staggering, and it’s not just about adopting new processes; it’s about mastering the balance between engineering excellence and effective processes.

The Importance of Engineering Excellence

Let me share a story that illustrates this point. A few years back, I came across a case involving Knight Capital Group, a company that, despite having nearly $400 million in the bank, went bankrupt in just 45 minutes due to a failed deployment. They attempted to implement a new order handling feature but ended up deploying to only seven out of eight servers. The result? A catastrophic failure that cost them $460 million. This incident serves as a stark reminder of the importance of robust engineering practices and the dire consequences of neglecting them.

In my experience, many organisations struggle with technical debt and legacy systems. It’s crucial to continuously refactor and focus on engineering excellence. Without this, we risk building the right thing but not doing it well. Conversely, if we focus solely on processes without the necessary engineering skills, we may end up with a product that doesn’t meet the needs of our users.

The Agile Mindset: More Than Just a Process

Agility is often misconstrued as merely a change in how we build software. In reality, it’s a fundamental shift in how we run our businesses. I’ve seen organisations cling to old ways of working, which stifles their ability to embrace agility. This is where the Agile mindset comes into play. It’s about fostering a culture of transparency, collaboration, and continuous improvement.

One of the key challenges I’ve observed is the lack of effective product ownership. Many product owners come from a business background and may not fully grasp the technical aspects of their products. This disconnect can lead to poor prioritisation and a backlog that doesn’t reflect the true needs of the users. To combat this, I recommend investing in training for product owners, such as the Professional Scrum Product Owner course. This investment can significantly enhance their understanding of value delivery and accountability.

Embracing Change: The Path to Agility

As we navigate this journey towards agility, it’s essential to remember that there’s no one-size-fits-all solution. Each organisation is unique, and what works for one may not work for another. This is why I advocate for an experimental approach—test, learn, and iterate.

In my practice, I often refer to the concept of “skin deep agile.” This term describes organisations that superficially adopt Agile practices without truly understanding or implementing the underlying principles. To avoid this pitfall, we must focus on building a culture that values collaboration, transparency, and continuous learning.

The Role of Metrics in Agile Transformation

To measure our progress, we need to establish meaningful metrics. It’s not enough to track output; we must focus on outcomes. This means understanding how our work impacts the business and our customers. I’ve found that using evidence-based management (EBM) metrics can provide valuable insights into our performance and help us make informed decisions.

Conclusion: The Journey Ahead

As I look ahead, I’m optimistic about the future of Agile and DevOps. The challenges we face are significant, but they also present opportunities for growth and improvement. By focusing on engineering excellence, fostering an Agile mindset, and embracing a culture of continuous learning, we can navigate the complexities of today’s business environment.

If you’re interested in exploring these ideas further, I invite you to connect with me. Whether through my upcoming training sessions or simply reaching out for a conversation, I’m here to support you on your journey towards agility. Remember, the path to agility is not a destination but a continuous evolution. Let’s embrace it together.

I’m gonna meet a lock off I’ve got caught by a quarantine yourself pass me hide knock off to be the fancy intro music as ISO News pain simply the based on Twitter says girly yeah I don’t know it work through my head sensor I do have some fancy intro music I just can’t push it through through teams teams is not zoom zoom yes it’s it was a ranger RPS but just nobody use them I’m happy I can’t

[Music]

Before singing sunshine only their banana flies ah did you not see that Donald it was just a nice idea I guess they were just emulating what was going on in Italy today mm-hmm to be brutally honest it’s not like it’s not like you have anybody never to go to see him by the way at seven o’clock so anybody please

What’s the door probably going mutes go get mark throw that from his end control you and thank you let’s can record it yes it is going out live on LinkedIn Facebook YouTube Twitter eagerly push but I’ve started the live feed okay but just excellent it’s on it’s coming soon

Nice I think well I will give out a few minutes why not raro buttes you are you gonna host an intro then SAS yes what’s that you’ve shared you know posted in their digit Donald

I’ve got about four hours with the content is that okay guys you’ve got there you go I know Ellen you’ve always got loads of content loads and loads we could do another one later in the year hopefully hopefully our face-to-face a nicer they knew that be them pizzas you know you’re being well is that I think you’re ever hopeful they’re sad 2020 stuff there’s always there’s always there’s always room for optimism yeah I reckon I actually I actually don’t know if the stream can hear you as well probably just visiting and you and me can change facial expressions I’ll be me me talking to myself I think is what everybody can hear right Phil enough yeah I’ve got a unexpressive face that’s the chat as well people saying everybody can get okay oh here comes go yes we can hear both of you Bob thank you for that

[Music]

And see you if that’s supple really great right there it’s good okay all right a couple more minutes shall we gents and they’ll kick off no worries

[Music]

A mask off anything for attention is there any community and in Spencer instill sooner lines I’m gonna do the end mr. Coburn I work what you not know I’ll do that yet the posted was a falling off conclusion the time of the pod right he thinks he’s got planned but he hasn’t done our web conference with me before distant future and I think I’ll make a start right it’s ok I think we should start thanks money to class yeah that’s given a revealer chance to jump back on right welcome everyone thanks for joining us let’s say second event future work in Scotland available a year this one obviously we’d originally planted to do APA well then you have used before but the circumstances obviously speak it remotes got there no thanks for being with us this one has a little bit of other significance actually because it’s doing it in collaboration with the VCS agile methods group that’s a phosphor date not without she done as amethyst because because our klieg and I both a part of the BCS agile methods committee and this is a foster this is Nigel methods event we’ve done in Scotland so are the significance for some of us so that’s happened all cold air will do more over the year over the rest of the year time to come hopefully some place to face was late in the year but to Martin thinks I’m being optimistic we do have a code of conduct and those of you who have attended some other events familiar with it we do have one so please do familiarize yourself with it it’s available on the hive in short things and they will do will do some community showcase for some other father they start coming up over the coming weeks as well all confined to our homes Q&A so just quickly we will do a Q&A at the end once Martin’s fellows presentations so I think we have a chat facility right here so if he can start you know putting any questions that come up as Marcus doing his presentation and Donnell denial can sit through them and we’ll reel him back to Martin’s week and he can answer the questions that come up at the end so hopefully that’s a decent format for everyone and with that I think just introduce our speaker so tomorrow and I for a few years now in fact I did Martin’s a scrum divorce a professional scrum cleaning a few years ago now in fact I think it may have been one of the first ones in Scotland so Martin does what’s come to or training or what consultancy Joe and DevOps and yeah I’m really looking forward to is presentation tonight giving one of its personal experiences doing an agility in the enterprise so with that I’ll hand over to Martin Martin folks hopefully this is going to going to go well it’s it is kind of being recorded it’s streaming live on lots of different platforms

I’m not sure how good the the Sundays or whether anybody could hear SAS that format but yeah we’re all here anyways so it doesn’t it doesn’t matter I’m both the professional scrum trainer likes that said but also our Microsoft MVP as well in DevOps and I kind of kind of split that difference a little bit between the technical skills required to deliver deliver software it’s skill as well as the the the process ideas of you know keeping it simple like we all try and do so I create this presentation a little while ago it had various names sling the dragons and how to successfully descale that skill is just the most recent one I think it was unicorns last year taming the unicorns but the idea is the same that we really need to China talked about various things that are not just a problem at scale but things that we know work pretty well that you can try so nothing here I’m going to say is you must do it this way absolutely not you need to figure out what works best with in your organizational context and that can be that can be difficult depending on on what it is you’re doing so let’s start off and if anybody wants to get in touch with me later please do any way you like I’m pretty much everywhere I’m in Glasgow that’s where I’m based and that’s where I am currently confined like everybody else so if you do do need something I’m here and in the in the time zone and I did see there’s a few folks I know from other countries logged on and I saw Anna an organ did were logged on so it was a few few people from further afield European time zones

The thing that’s behind what I’m talking about is that that idea you’ve probably seen this before firms today experience a much high velocity of business change business is changing a lot more quickly and we need to keep up and it’s not just about process it’s a balance between process and engineering without good engineering skills we might build a lot of the right thing but it will not be very good or not meet the need and without the process skills were maybe not build the thing we’re supposed to build so I’ve got a little story first and that I’ve used a couple of times before I I find it quite well it’s kind of amusing cannon art but it’s the story of how our company with nearly 400 million dollars in the bank went bankrupt in about 45 minutes because of a field deployment some of you might have heard this story it’s from the knight Capital Group and in in the US and there are fortune 500 or not fortune 500 was a Gulf New York Stock Exchange company and I need a new feature they were deploying to production it was some kind of new order handling feature that changed the way that worked and and while they were replacing all old stuff with knew they had Cain Anna am a big legacy and you know what it’s like in in in your teams today there’s usually anything that’s been around for a while there’s a lot of technical debt in the system a lot of spaghetti code that ends up sitting around for ages on last year teams continuously refactor and focus on engineering excellence so this idea is they hadn’t can a nine year old unused code and they decided to repurpose a flag in the system you know there’s this flag sitting there it’s not doing much let’s just use that in order to turn this new code on and and when they did a deployment they were doing manual deployments and the technician only deployed to seven of the eight servers so you can imagine when soon as they turned the flag on something weird started happening and it actually took it took it took the whole system then it wasn’t operating correctly so they weren’t getting the the orders through they were expecting and because the system was down and they were losing about a hundred and seventy thousand dollars per minute and as you can imagine that set off a lot of alarm bells and they go - ever trying fix it but when they try and fix it the the they can’t figure out what the problem is because it’s the amount of code they deployed is too much the flag had unintended consequences and the the deployment to seven of eight the eight servers even the technician that did the work didn’t really understand that that’s what they’ve done so there was no visibility or transparency into that because it was a manual deployment a step or a couple of steps or something was skipped and not noticed and I’m the ended the day four hundred and sixty million dollars down and had to file for bankruptcy and I think the important question there is what would be the impact in your organization of a critical system that folks write code for or that deployed being being down for for a day for two days some big examples recently where some airlines that was British Airways went down American Airlines went down for three days because of a mainframe fault that you can’t get any more technical debt the mainframe sitting there and in this particular case because it was a public company because it’s on the stock exchange they have to do an SEC filing for bankruptcy and you know that’s how we know so much about it you can go and look up this filing and see exactly what the what the problems were and 2013 is not really that long ago they really should have known better and even around that in fact later than the 2013 timeframe I worked with an organization in the Netherlands around 2014-2015 and they were an international bank and they are real time banking transaction system the servers five servers around the world that govern this weren’t there so their code was not under source control and the team they organized it believed that source control would slow them down and so in order to make changes to this real-time banking transaction system for an international bank they would log on to one of the servers open up their IDE make changes to the code and compile it on the production server I did ask how do you share that code across all five servers and the response was that’s why this five of us so each of the servers are owned by one of the engineers the engineer goes on to that individual server that’s their server and they write and maintain the code on that server so they’ve got five servers all doing the same thing that all have different code and different setup and for me that was a huge business risk but for the business the bigger risk or at least for management of that group the bigger risk was that this team was the only team that understood what to do so they get to write the rules and that can often be the kids little fiefdom at cropping up as well so how do we not have this happen and we need to get need to get better at it you need to practice do it often do it all the time that’s kind of the moral of that story automate everything you should have any manual practices in your in your process your delivery from developer committing code all the way through to being in production should be totally automated you might have approvals along the way but the approval should be you know clicking a button ticking a box and not actually following any technical steps so can you can you think of any other at really big crazy failures like multibillion-dollar failures because I can think of two the first one was a failure of quality so this was a massive product that instead of taking three years to deliver to production ended up taking six and they cut 80% of the features they were trying to deliver and their problem was technical debt and that was Windows Vista and anybody that used it when it first came out will have felt some of that pain so that was a quality issue and Microsoft worked very hard to build up an engineering excellence after that but the problem was that they still didn’t and they still were building the right thing so they ended up with a mismatch - customer desires which was indeed Windows 8 which was a massive mismatch to that idea so it was after Windows 8 and that they started focusing both on the engineering excellence and on the getting the right features into production which means you have to get tighter feedback loops which is where Windows 10 was born and Windows 10 effectively from our perspective Windows 10 ships to production every 30 days but from the team that writes on it four and a half thousand engineers and they ship to production daily the code that an engineer writes today is running on the CEOs laptop tomorrow everybody inside of Microsoft gets the internal build and they can choose to install their own machine and then not get the built in case anybody’s on here that Microsoft and knows there is a there is an escape hatch if you don’t like that but firfer for the majority of people there they’re taking that at faster build and for those of you in the know a machine I’m on just now is in the Windows 10 insider preview so I get weekly dev builds from the Windows boot from the dev branch to the windows team that that is there’s a 17 million people that do that so it’s a lot of people catching the tires and I think that’s the important thing is getting stuff into production as quickly as possible and so I think this idea of learning is something that you’re all familiar with and I want to just posit some kind of kind of problems and challenges that I think and many people seem to have had and I particularly like this one because you do a front release and then you either get a smiley face back or some strange characters but usually well that this being Scotland it would just be swear words but in most of the world it’d probably just be swear words anyway and what is this pile of crap that you’ve I should I so the the three things that I think posed this problem and for for agility is what how and who and if we break that down a little bit the what challenges some of you will be familiar with this data is from the Standish Group and Boston that do the chaos report and so no focus on on product strategy and an effective prioritization and I believe very few organizations that say they’re doing agile actually have a prioritized backlog so if you don’t require this backlog the engineers are just gonna make up whatever it is that they they want to do to work on it um those 65% of the functionality that we build is not used by our customers that’s average you could be worse or better most people that think they’re better usually end up and worse when they look at the data and 35% of your requirements were change I’m sure you’re familiar with that figure as well and but weak product owners poorly defined runs responsibilities and product management are key parts of that challenge so they focus instead of focusing on a product and making the right decisions for a product they focus on time limited short-term goals called projects and those projects may or may not provide benefit to the overall product as the person running the project is not measured generally by the overall success of the product the measured by the short-term success of the project and in those worlds you tend to have a long time to market can a long cycle times I I would personally say anything longer than 30 days time to market is too long and there’s another organization that agrees with me I did a presentation recently on detecting agile [ __ ] which is a Department of Defense article you finding that did that last last week people find that as well where I talk about some of those issues I’ve got a slide from it further on in the presentation for you guys and the who challenges who’s doing the work and apparently this is where we lose a lot we lose 50 percent for ineffective collaboration now I know many of you work with teams that are trying their best to be effective collaborators and you’re probably your number is probably better than 50% an average across all something like 70,000 IT projects worldwide but that idea and you might be not far off that 50% depending on the way your organization is is managed an effective servant leadership that that idea of people being managers instead of leaders inside the organization is kind of key there and is a big struggle and not having good objectives you know that misalignment of where we’re going with people’s understanding of what it is is is fairly fairly common and then we end up with these hierarchies of organizational structure that are based on a hundred and thirty year old terroristic practices that are really out more deep for the modern modern workplace subs the who there is also the how in the how and we’re really seeing that growth of technical debt it’s it’s systemic in our in our industry and even big companies like like Microsoft and who whose job it is to build software and end up with significant amounts of technical debt I like to use data I’m not sure if I have that data in here I can get it for anybody who asks for it and but I have data around and how many features the azure DevOps team in Microsoft was shipping to production eight years ago when they were still a waterfall team and how many features they’re shipping to production now when they were a waterfall team mounting technical debt only deploying once every two years they were shipping about twenty five features to production each year that’s with six hundred and fifty people and with the same number of people now having spent eight years paying back their technical debt so moving that needle from fighting the technical debt struggling with complexity towards adding new features they now add more than two hundred and seventy features to production each year so that just shows if they were spending 90% of their time struggling with complexity pay back that technical debt and you get a lot more done because your teams aren’t struggling with that so much and that’s a that’s a common growth of technical debt is one of the the biggest blockers to that so you really need to focus on that from an engineering perspective but also I mentioned anything that’s automated sorry anything that’s manual you want to automate everything so manual testing manual environment provisioning manual building integration manual deployment manual anything is a problem I did work with a team in Athens that was using a product called starting to do their stars control don’t get me started we’ve moved them off star team but and one of the features of star team is branching and merging as you would expect from our source control system but the team didn’t know they knew about branching but they didn’t know about marriage so there was a 30-30 i between 30 and 60 people in this organization that when they wanted to bring changes from a branch across to the main line they were manually copying and pasting those changes across there are still teams out there who don’t understand some of these fundamental concepts and need to start focusing on anything this manual is a bad idea if we take all of that that data that we saw there for the different different challenges the thing I like is this little graph here if your input was a hundred euros what would your output be based on 35 percent of your time spent building the right features so then we’ve only got 35 percent left so I’d have a hundred that’s 35 euros and if we lose 50 percent for collaboration we’re down to 17 and a half euros out of 100 and if we spend 30 percent of our time building new features and 70 percent of our time struggling with complexity we end up spending 5 euros 25 on adding new features and out of a hundred that that’s that’s a horrendous ROI and but it’s the reality for many organizations out there but just to put that in perspective for a million pounds spend a million pounds on a project that’s fifty two thousand five hundred pounds of value that’s that’s just not not good enough and that’s not just engineering that’s business processes that’s know focus on collaboration these are these are super super common industry-wide I know some of your teams might be better than that but you also work inside of wider organizations that may still have some of those problem so we want to be collaborative to be more effective and that’s why you know that’s why this little document exists the idea of the agile manifesto was to try and focus on the right things rather than the wrong things that organizations have been focusing on but it’s still taking time to get there and one of the things that I see I know this is the way we do things around here is the common thing was it the cup culture eats agility for breakfast our culture meets changed for breakfast or culture just eats everything for breakfast that’s part of that that problem and so does this journey that organizations undertake companies talk about an agile transformation I don’t really like that phrase because transformation implies some kind of completion which i think is just fundamentally not not true and I usually think about it more as an agile evolution as an organization you’re going to be constantly changing towards an effectively an unknown goal and that you can have an idea of where you would like to go next where would you like to go in the next few steps but ultimately where you end up is going to be somewhere different and I usually use this kind of diagram I know the Snyder model is not I’m not selling you on the Snyder model for sure and it’s just for illustrative purposes that if we have break our organization up into into these kind of kind of sectors where would you plot your company would your company be in the more at controlling space or in cultivating people space is it based on competence or collaboration what what percentage of each of these would it be based on there’s no 100% anything but if your organization was in a particular place and you say you want to go somewhere else it’s not actually a straight path like that it’s more of our wibbly-wobbly Road and I like to think of it as orienteering you go off in a particular direction you hit an obstacle you figure out how to get round it and then you figure out what’s how do we reorient take back to that important direction that we’re trying to get to and see where we get to so it might not quite be the same place you thought you were going to get to hopefully it would be a better a better place and so one of the things that I think is really important is you you can’t I think I’ve said a lot of important things and every organization is is is unique you start from a unique spot even if you use Schneider you start from a new unique spot you end up in the unique spot and however you want to do that everybody is unique so the idea that you can take a methodology that somebody else has built and apply it to your organization is really a fallacy as its it’s not going to work what works for you might not work for another organization for what works for them might not work for somebody else an experimentation and iterating towards these ideas is really important but that iteration towards ideas that acceptance that our current processes are imperfectly defined is fundamentally it fundamentally means that we can’t just use somebody else’s process I have two processes that I’m thinking of I’m sure you’re familiar with both of these and one is safe skill agile framework I know the newer versions of safe my picture is v4 of safe remember what they’re on now there may be as high as V but the ideas original idea of safe was Dean didn’t living well implemented in an organization how to do everything and then documented it and published it as a framework that folks could folks could use and it quite often failed in in fact mostly it failed in organizations that it was implemented afterwards it might have worked in the first organization because that was custom-built for them but then failed in future organizations same with the Spotify model and the Spotify model is not really a Spotify model I mean the folks at Spotify would kind of laugh if you said you implemented the Spotify model because what is written in that white paper is not what those teams do today and the white paper and the presentations that the Spotify team did where a point-in-time presentation a point in time understanding of what what they were doing at that point in time they’ve moved on since then they’ve changed their practices to suit what worked and what didn’t some of the practices that are described in that white paper lasted a couple of months some of them lasted a couple of years some of them lasted five minutes it just depends what what things make sense and I think that’s the important thing because really is something I like to say there’s no such thing as best practices only adequate practices for the situation at hand and there’s there’s no guarantees everything’s going to be different and I really enjoyed I don’t know if anybody saw if anybody’s read and the the Department of Defense how to detect agile BS but it’s an article that was written a white paper sent out to all of the procurement officers within the Department of Defense to help them figure out whether their vendors are actually telling them the truth about being agile or they’re just saying they’re agile because you know that’s the thing you have to say now in order to get the contract and something happened recently those that know and might have seen this already but I have a little US Air Force I have the original memorandum that came with this this was a clarification of that that memo and I’m the the CSO I can’t remember what CSO stands for chief something officer I can’t remember anyway the the senior officer in charge of IT and for the US Air Force is quite a progressive thinker and and he likens safe to waterfowl it’s a large bloated prescriptive framework that tells you how to go do something and you won’t find any large successful commercial organizations doing anything that looks like safe they’ve all built their own custom process based on the way they do things now I’m not saying that safe doesn’t have value I think it does have value in certain circumstances and if you see the like the fifth bullet point down safe might be potentially might be used a useful framework for teams who do not use DevOps but a key principle of DevOps is to decouple work in teams and only synchronization required about donors bhagavad I mean that’s not quite right but know what I mean and that idea of having that monolithic process I think really really works and that said lots of organizations that are not yet ready for DevOps not yet ready for true agility um get a little way of the way there but I think there’s definitely uh what’s it called when you you think you’re you’re there but you’re not there there’s a false sense of security maybe mmm I’m not sure what the word is and but you have this false sense they you’re doing something agile when you’re using safe when safe is in fact the complete antagonistic opposite of agility it’s a prescriptive framework and I know that new versions of safe are less prescriptive but that’s its source that’s its routes and you may as well be doing PMI prints two versions of agility and provide the same benefit I think start from something that works a little bit a little bit better so hopefully everybody’s read there’s this idea of five dysfunctions of a team it’s an awesome awesome book and I have that presentation on detecting agile BS that was into a little bit of that but the thing that we tend to do in organizations is we we we have this complex world and I think however however you-you-you understand it I think this is the idea of software development being product development rather than product delivery you know a car production line is product delivery Toyota developed the the lean Toyota Production system in order to deliver cars not design cars the design team for Toyota don’t use the name Toyota Production system because it’s a different sort of work in the software world which I am that’s my background our production line is our DevOps pipeline that’s our from the time a developer commits codes to the repository till it gets into production that’s our production light or everything before that the writing of the code the merging everything together those are those are creative endeavors that sit in this complex world of more is unpredictable than predictable with emergent answers and many competing ideas I’m sure you’ve you’ve seen these before I know I’ve talked to Satpal about these quite often but organizations tend to use a different management style the leaders job they tend to use is best practice you know I hear that all the time in organizations this idea of best practices and establishing patterns and optimizing to them and you wouldn’t believe the number of organizations that still have a command and control at hierarchical outlook to the world it’s it’s very difficult to get to get em organizations to treat complex work like complex work so in a complex environment you’re supposed to create a bounded environment for action so we want to have some boundaries so everybody’s going in the same direction but inside of those boundaries we don’t really know how to solve the problem and so we need folks to kind of get there themselves and obviously we want to increase communication because we’re talking to each other more we come up with better ideas so generating those ideas servant leadership obviously to get things out of the way for those people to be able to do the work and do it effectively create an environment for that to work and so we need to start somewhere and where we need to start and is probably some kind of loose framework and my background is a scrum I’m not opposed to replacing every time I mentioned the word scrum with Kanban I’m just using this for illustrative purposes again and I like scrum I like scrum with Kanban I teach the professional scrum with Kanban class from scrum dark I’m co-teaching with Daniel Vacanti we’re gonna do an online class Ferb well it was supposed to be an in-person class in Edinburgh but obviously it’s going to be virtual but I’m going to be teaching with Daniel Vacanti in a couple of months and we’re going to dive into how do we take those and complimentary Kanban ideas the core practices of Kanban and apply them into the the scrum world and so I don’t think I mentioned that in this presentation but it’s pretty good you can go and get the cat the Kanban guide for scrum teams which is a free website so we have this idea of potentially using some kind of framework I’m using the scrum framework for a product inside an organization so we maybe have one team working against this product and and instead of being part of the hierarchy they kind of exist within a separate bubble I’ve seen this done in organizations where they’ve actually created a separate company for the engineering teams to work in so they can work in a different way I with the hierarchy of the rest of the organization and I’ve seen this done in KLM where they created a studio for that work to happen and move the software and so for teams into that and it used services from the rest of the organization so those shared services might be infrastructure QA might be HR might be any of those those things outside of that and so you basically get a small pocket of scrum inside your organization a lot of organizations try to do a massive transformation they maybe do a small pilot project and a massive transformation and generally those fail usually they end up at something using something like safe because they need to figure out how to do everything all at once and that just tells you that so this scrum framework is just a - I mean the the scrum is just a tool to help us create an empirical process control system within which we can inspect and adapt towards creating a more optimal system that’s really what is designed for some famous Kanban except a little bit more make sense so do scrum well or do cab Manuel is your first step am i I do it well I mean that people are really not good it even doing scrum at all or doing Kanban at all I got two teams all the time who say they’re doing Kanban and then you know they’ve just got bored with stickies that move across it that’s not Kanban where your whip limits where your your concrete processes where are they not decided any of those things so some interesting data here which is actually from Forrester Research and is only 22% of organizations that say they’ve adopted agile and are actually doing short iterations if you’re not doing short iterations you’re not very agile only 13% do retrospectives if you’re not inspecting and adapting what what are you doing and only 16% of prioritized backlogs though these are appalling numbers that are that are only only getting worse as more organizations move towards agility and more snake oil salesmen and charlatans convinced them to do the wrong thing and I actually worked with some some folks in organizations and the their organization is sold unsafe so they create something that they can sell is safe but it’s actually much more agile than say if it’s a custom process for them so some organizations that say they’re doing safe actually aren’t they’re doing their own thing and that’s okay I’m okay with that just some extra data if we fix these problems we can improve a lot of things in our organization and try not have just skin deep I like that expression skin deep agile I think Ken and Jeff call it flaccid agile I like that one as well I usually call it mechanical I call it mechanical scrum we’re just mechanically following it we’re not actually doing the things those are pretty effective so step two is you need to enhance it you need to do those things like create empirical systems have our engineering practices coding guidelines user experience emergent architecture all of those things and and once we’ve got good engineering practices for me that’s the engineering practices parts you get mechanical scrum engineering practices and then introduce the values behind the processes how do you build a system within which you can have transparency that you see the truth and these are trying to trying to create that and so whatever your you’re doing you need to focus on all of those things to get to this idea of doing something professionally rather than just doing something and I think that’s that’s the fault are in many organizations is they don’t focus on professional scrum because we don’t have time to do testing let’s just get stuff done or we don’t have time whatever the things we don’t have time for to focus on that professionalism and then we end up with with some kind of small loose framework around which we can build our own custom processes that’s the idea I’m looking at so we can then add multiple teams into this bubble and that all work by different rules use these shared services and we’ve upskilled the individual teams because the reality is that skilled teams are 224 percent more likely to be successful than unskilled teams that seems like it makes common sense but it doesn’t necessarily make common sense and a lot of organizations they believe that scrum or Kanban is just this unicorn that’s able to solve all of their problems and but it ends up being you’ve heard the term scrum er fall or many waterfalls and everything ends up being a bit of a car crash that’s our very common thing I actually do have the the Kanban stuff in here they’re mate and so one thing that I always get teams to do once they get to that professional scrum level is they need to start looking at their floor and optimizing for it folks on the call who are agile practitioners who work in these big organizations you’ll know what I’m talking about if you take a bunch of amateur teams and try and get them to look at their floor it’s an even bigger car crash you have to get professional teams first get them all all going in the same direction delivering a working increment every iteration once you can get there start introducing flow start optimizing for smaller batches and focus on that idea and I mentioned the professional scrum Kanban the command guide for scrum teams and it talks about the five core practices visualize limiting work-in-progress actively manage items make polls explicit and it proved collaboratively and those are the core varieties from camera and so I’m not talking about the Kanban method which is from lku I’m just talking about those those core Kanban practices and something that can said recently was the only foundation to scale is professional teams you need to get the professional teams first before you can scale and I’m sure you’ve seen a bunch of unprofessional teams shoved together and trying to build some software it’s always an amusing car crash and but Steve Porter who was one of the co-creators of the Kanban guide for scrum teams added that manage flow the only foundation for skill is professional teams that manage flow because the reality is of scaling is expensive it’s hard it’s wasteful ultimately don’t scale scaling it is an anti-pattern of software development you need to avoid it at all costs because scaling doesn’t go linearly it has this very quick tail off as you add more people and then at some point you add enough people that it gets too complicated and you end up with something that is just not a lot of fun this is I think this is our P I planning yeah this is the last P I plus the video from the last pipelining that I saw so you can scale continuously as long as you do certain things and but you need to do them all the time you need to identify and remove impediments i–sorry dependencies dependencies are going to be your killer so anytime you can create vertical slices so feature teams rather than horizontal teams obviously and but integrating work all the time across all levels one of the downfalls of office at 2012 was that they weren’t integrating every iteration they had most of their stuff integrated but not everything so then they ended up nine months later with a big problem and then create and inspect those increments regularly obviously make sure your teams have the tools and skills necessary to do things at scale it’s going to take more tools it’s going to take that more time it’s going to be harder to beat to get get on board with that and obviously keep keep inspecting and adapting frequency frequently scrum dark noticed that this scaling thing was a problem they were very late to the scaling framework party and the reason for that was they wanted to take their time and make sure that they had something the in keeping with scrum is the absolute minimum you need to make sure you maintain the levels of communication that you need to create an empirical process control system so they came up with something called the the nexus framework which is a lot smaller than a lot of the other frameworks that are out there um I’m thinking of a scrum at scale I’m thinking of safe I’m thinking of other frameworks because it’s not going to tell you how to go solve your problems it’s just going to deal with those communication issues between teams and I’ve seen Nexus and I’ve seen Nexus implemented quite a few times with organizations and it’s very easy and quick for people to understand because it’s only a little bit more than scrum and it doesn’t add a huge amount of complexity or overhead you’ve effectively got this idea of um you know you’re gonna have to do refinement now whereas if you get one scrum team with three people you maybe don’t have to refine but if you’ve got nine scrum teams of 10 people working together men and you’re gonna have to make sure you have all of your backlogs in a good state and then you need little teams of teams to get together to make sure that you have good communication so you can see the Nexus sprint planning has a little Nexus that gets together it’s a team of teams it’s the right people opens our open space rules and from the scrum teams that get together to figure out some dependencies to figure out how we’re going to move forward for the next sprint and take it back to the rest of the scrum teams which may or may not be in the same time zone it’s okay to have multiple as long as the individual members of your team are on the same team on in the same time zone in the same cool ok they don’t have to be co-located but it’s more optimal then more optimal is having those teams together and then multiple teams in multiple locations or in the world can work together so I asked Nexus sprint planning might run over 24 hours because you’ve got that time lag we just need to deal with that problem we need to figure out how to do that obviously the closer people are the better but there’s the realities of business and and then we need a view on all of our work so the Nexus sprint backlog with a way to view individual teams backlogs we only have one review because we’ve got one product I think that’s important but even on a daily basis we maybe have that team of teams get together to discuss dependencies and issues and then at the at the whole Nexus level and then take that back to the team’s daily scrums again just a 15-minute before everybody else’s daily scrum representatives that make sense from the other teams get together figure it out take that information back same in the retrospective you’re going to have but you’re gonna have kind of you’re gonna have to one of the beginning one at the end because you might have a bunch of things that need to be input into the team’s retrospectives and then the teams might come up with all sorts of clever ideas to go solve those things so that team of teams gets together to bring all that information back that’s the idea from Nexus it’s just meant to be the minimum I’m not saying it’s the right framework for you but it can be a useful a useful tool that has those very simple added extras to help you get that increment at the end I didn’t mention the Nexus integration team because it tends to be a very poorly understood the integration team is not your DevOps team it’s not your build team it’s a team of teams it’s a group of people who get together to solve whatever problem is inhabiting the team’s ability to get a working increment there’s no point in the scrum masters getting together if the problem is merging code they’re not going to know how to solve that problem you need the right people so effectively a nexus integration team is just representative some various teams the right people maybe they’re scrum masters maybe they’re engineers it just depends on what the problem is they’re trying to solve it’s not a permanent team they’re gonna go back and the teams the individual scrum teams inside the Nexus to the do the work that’s the idea behind Nexus and having these breaking teams so then we can bring in a much larger product into our story and you’ve got lots of little small products use no service overseas and then we bring in a larger product into that into that world so as more than three teams working together I don’t know where that animation came from never mind so that that essence of scaling is we need to anticipate problems by focusing on minimizing dependencies that’s your your your almost most expensive concern is minimize dependencies and then make working software you’re still a team of people working together to make a single product so if you don’t have a working increment at the end of every iteration we’re not there yet we need to get that working integration so just inspecting and adapting towards that world and you’re probably talking about more engineering practices for absolute focus on minimizing technical debt automated testing continuous build and delivery very very very important some examples of teams that are using Nexus Capital One has delivered some products with Nexus and I like the comment they are to less than an hour to explain access to teams that are already familiar with scrum it’s not much more than scrum so Capital One Kathy Pacific Airlines and also use Nexus as well and that that really helps so while I’m I think this is a very miss Atwell I think this is a misleading quote but small teams of 32% relate to succeed large projects are 600% and that’s because large projects are such a big mass that they can get 600 percent improvement and they probably both get a similar improvement is just much more obvious in a large project and I think small project here is 50 people or less large project is more than 50 people Gail’s reports that but what that is and so we we came up with this kind of bubble just back to my bubble of scrum in my case or bubble of Kanban or even you could have one team doing can add another one doing scrum it doesn’t matter you know these shared services there’s bubble in which people work differently and and potentially you need to at some point you’re gonna have enough going on in there that you need to start focusing on that leadership ideas and and that’s where you start to look at how do we get the CEO CIO involved in that and provide agile functions of leadership like HR finance and admin there are things that the whole team can use in an agile fashion rather than just using the shared services from other projects um how do you build this new type of organization that is flatter structure or not based on hierarchy and inside of that and then this idea of bringing in more products you might have some bigger products that have multiple Nexus again the windows team has four and a half thousand people working on it and the office I think has 300 scrum teams 300 scrum teams now they’ve come up with their own way to solve that problem that’s okay too this is just a framework that you could use to help you get started but it’s it’s interesting to note the the characteristics that I’m showing here are things that these big organizations that are doing agile well whatever they call those things or how they implement them they have something that kind of fits this bill and and the next thing we need to understand is how do we know it’s all working so something that I think is very important is understanding whether you’re making improvements the organization organizations spend a lot of money with us agile practitioners to come in and help them get better at delivering software how do we know we’re making things better well we need some kind of metrics to monitor scrum that arc has some example metrics and I’m really stressing that super a heavily example metrics in particular categories but I think the categories are the thing that’s important we’ve got to look forward to towards the future with market value let’s you get the current value you’ve delivered in your product and how you monitor and measure that so you’re gonna need some metrics around current value how do we understand that and then we’ve got unrealized value value we don’t know that we need to have yet how do you look at the market how do you look at market trends and analysis and figure that out build that into ability to innovate how quickly can we innovate how much time will we spend innovating in our product so we need some metrics of own around our innovation so I might be capex darzee’s our pecs is a super simple one but I think it gets cloudy you’re gonna need some other measures in there and then time to market how quickly can we actually get our product in front of users and I’ll use the example of the windows team again which gets their software every thirty days in front of 600 I think it’s is it not a billion machines now yeah it was used to be six hundred million machines it’s closer to a billion machines now worldwide that’s pretty impressive engineering practices to get there I don’t know why that animations coming in I have some example measures in the deck I’m not going to go through them all and I’ll just flick across but you’ve got to figure out leading and lagging indicators some of them are more important than others and these are just examples they might not work for your organization I don’t know why there’s animation again and it’s doing it on all of them but unrealized value time to market so even um cycle time release frequency leak time these are a lagging indicators there are there are leading indicators frequency of build success might be a leading indicator make sure you’ve got understand your measures effectively and this time EBM metrics evidence based management EVM is something that actually comes out of the medical profession about a hundred years ago or actually a hundred and twenty ish years ago the medical profession decided that it would be a good idea to makes decisions on your health based on you know data like your blood pressure and you know collecting some data and figuring out how to solve your problem rather than just making up an idea and hoping it works so this evidence-based management is for managers and organizational leaders to look at actual data make changes see how those changes impact the data are we taking the organization in right direction like project cost per iteration time to market innovation rate how much time do we spend innovating there see struggling with complexity satisfaction of our customers and employees is that trending up or is it trending down product revenue and costs these are all things that are measures of organizational and product performance and and we want to be optimizing those things we want to making those numbers look a bit a little bit better and we can tell whether we’re making headway if we have balance across all of those things that’s why I had four different categories of metrics because we could have asked awesome time to market but we’re delivering a bunch of crap to our customers so we need to make sure and that we balance all of that out yeah just keep things simple and while my diagram might look super complicated now you saw it’s literally just it’s just scrum are we to scale scrum very simply and then a way to to measure it and that’s really the idea the SDK in there is just and let’s share some data so it’s probably a community of practice across all of our teams to help share some of those good ideas and maybe create a starter pack for any new team starting up where should they start within the context of this organization so every company will build something different and frameworks are just a starting point and it’s okay for them to start with Kanban start with scrum and start with and whatever whatever makes sense for them I had just some final suggestions where is it there yep and make sure you measure and focus on value and outcomes rather than output and progress and we can output a lot of crap and not get to the outcomes that our organization needs to get to celebrate success improve the results get create a holistic view of what you’re doing and try and improve those measures come up with a hypothesis measures move figure out if you’re doing the right thing maximize learning and really just do the best you can I think Scott had a good comment is sometimes you’ll try something and you’ll just be wrong and that’s okay we need to do the best we can with the information we have and see where we get to that was kind of what I wanted to cover and so just say I covered a bunch of stuff around scrum engineering excellence with the Kanban guide next is framework so don’t scale pretty much is my advice and really I’ve already at all costs if you can and then evidence-based management actually make sure you look at data and you can get a copy of the slides from that link at the bottom and that’s already been published for you and I guess we move over to questions yeah we’ll take questions I’ve got a type was I talking to him about any country no no no we don’t any come with chat so yeah if there are questions you have we’ve got Tim to take some yeah no II really enjoyed it it’s a really good insight I think just on the nexus stuff because as you know I’ve kind of been delving into the yes grommets kill stuff lately yes in the year and connect we enjoying the same sort of ideas Nexus where you know you want a really good kind of well-oiled well so function and scrum team that’s doing it really well and then you’re kind of stealing it Oh Tiffin I take the view that scrum at skill kind of is similar to Nexus in that you know it doesn’t really add much more than just you just got lots of scrum teams then you’ve got some additional teams are just supporting you as you scale up so you start on the scrums and then some of the government’s teams but Nexus those scenes are even more simpler than that do you see any in Challenge have you ever so implemented Nexus somebody coming in and Nexus is a framework I think you think Nexus lacks so yeah there are tons of things that Nexus lakks it’s supposed to like them it’s it is just a framework just like scrum lacks what is your definition of done yeah if you don’t have a good definition of done you’re scrums not going to be very good if you you don’t know where your backlog comes from and how to get that and create a backlog scrum doesn’t tell you how to do that and Nexus is the same it’s trying to be um it’s trying not to be prescriptive on your business practices because the the the reality is that the way sorry I’ve said that quite a lot the reality is I’ll strain stop doing that and every organization is unique the success or failure of an organization is in that uniqueness so the idea that and i know i’m i was pooping unsafe a lot but the idea that you can have every organization use the same blueprint and be just as successful is is why Nexus and scrum at scale and scrum are very compact and the minimum to solve the the empirical problem you want to create empirical process control system so you can expect and adapt to the best possible outcome or at least the most relevant okay bad idea and I think ken and Jeff almost totally agree between Nexus and scrum at scale and there’s just the some differences in the terminology but definitely there’s a little bit more in scrum at scale that can just didn’t want to be prescriptive about and I think that’s that’s okay I think there’s room for all of the frameworks and I will say if if you’re using a scrum at scale and you’re going to have exactly the same outcomes as you might have if you use Nexus as long as you see it as a starting point and a foundation and not here’s what we’re gonna do does that make sense yeah I think so I think as you see the best Fremont’s are probably ones that it’s exactly in the definition of a framework right it kind of give some shape but it doesn’t try to be too prescriptive and it shouldn’t be because then it’s good yes it’s dictating too much to unit it’s going to be restrictive right and constrain you and every as you say every organization is unique so you need to have but room to maneuver to a little needs and then that’s a good thing yeah it did any questions come in I’m gonna get good ones here yeah I like this first one so Mark Martin is asking because I often have this conversation myself so Mars is interested in soft product owner opinions so if we pull them from they always just don’t some other ones have common one say if we pull them from the business you know people gets appointed you know into the rule and they tend to be economics bag is there any ways of any good ways of can I help and get them into shape in your experience cuz you know what the different types of deals right they come in different flavors what are your thoughts on helping app you get to the place you might want them to so that it affected yeah I miss equation I’m definitely I’m definitely going to punt the professional scrum product owner and so you know I don’t teach that one so I’m putting somebody else these things but I think that that many product owners when when they first start and even if they’ve been approached or for quite a long time they have a very narrow view of what a product owners job is very narrow and if you think about a product owner as being fully fiscally accountable what they choose to spend money on then that idea of accountability inside of scrum which comes with that product owner rule accountability for value is is is key to that so if they don’t understand that accountability or they don’t enact that accountability so they need to be looking at the market market trends they need to be looking at and how users are actually using the product and a lot of or quite a lot of UX comes under the product owner world because that’s how users interact with their ideas these are all business problems to go solve the technical part actually hit that tends to not be the biggest problem it’s getting those product owners up to speed and I do have a company I worked with in Australia a consulting company and and they they themselves would pay for the potential product owners from their customers to go on the professional scrum product owner class so that they knew what was expected of them and they found that that I guess 2000 Australian dollars that two thousand dollar investment made a massive difference to the success or failure of their projects with those customers because they really understood the scope of that product product owner role leti that comes with it I’ll tell you something I’ve been through the part-owner class and and I figured out that I wouldn’t ever want to be a product owner and I’ve done it I’ve done the PSP you and I think you’re right I think if you’re really doing it there’s a lot for one person to be accountable for and just you know the different dimensions to the role yeah you know that there are so alongside you to piece in some ways you and a bit like they were you know we talked about the the Nexus mchugh the product yeah exactly so the scrum at scale you know if you’re the chief product owner you might likely actually be the chief exec and you’re cleaning that room some organizations in it’s like that so yeah it’s so I would say just as a final product owner thing is your product owner is the one that’s accountable to the business for value delivery not the development team so if your development team are have good engineering practices and are delivering high-quality working increments every iteration but they’re building the wrong thing the buck should stop at the product owner not at the development team because they don’t decide what they go work on having some of those ideas engrained into that role and will help if people feel that they’re they’re responsible for spending money they’ll up their game misuk anyone yeah not sure they get that’s an excellent put got another one for you here oh yeah just about something so cake leagues I guess it’s coming back to what you were talking about earlier by we talked about three more simple prescriptive they are so if we avoid being prescriptive do we risk never being able to set a direction of travel ie doing scrum badly as they accept it norm I think that is an interesting point so the question is if we’re being less prescriptive are we in danger of off doing doing scrum badly as just attempting that yeah um I I would say no because what what are what are the transparencies in scrum give you three transparencies in scrum oK you’ve got transparency of the future in the product backlog do we know where we’re going and how we might start to get there that’s the part of the owners I came to so if you don’t have an ordered backlog if everybody in the organization that needs to it doesn’t understand what’s in that backlog and why it’s there you’ve got a problem you’re not doing scrum yeah that rule of product owner is not doing its job and you’ve got the the Sprint backlog which provides transparency for the present what you’re doing now that has its own things but not aware that just no and then you get transparency of the past which is the usable increment so in this in software terms you need to have working software that can be deployed to production at the end of every iteration including the first iteration does your team have that if not they’re doing scrum badly they’re doing agile badly yeah you need to get that minimum bar I and I think it’s often overlooked the minimum bar for scrum is a usable increment you have to have working software at the end of your sprint otherwise that wasn’t a sprint is then you have no transparency you can’t inspect and adapt you don’t have an empirical process control system I think in in this group we can have this conversation but when you’re talking with customers you have to be a little bit more pragmatic and say well okay you’re doing scrum because you tried really hard to get a working increment and but I wouldn’t expect a team to be on Sprint 30 and still unable to get a usable increment every single sprint that should be a no-brainer by then yeah there’s your DevOps team at Microsoft which was 40 teams working together on one product took about 15 15 was a 15 maybe less or maybe 10 10 to 15 Sprint’s together craft together but that was six hundred and fifty people working together on one product but since then every sprint of the last hundred and fifty odd Sprint’s eight years of sprinting has been deployed to production that’s pretty impressive that’s that’s like your minimum bar so so Craig I I think that’s the difference between mechanical scrum actually doing it focus on the the values and principles building trust the transparencies and good scrum will come out of that and but it is difficult it’s a long journey and you need to uplift the skills on your team and your product owner along the way to get there and I would definitely create look at the Kanban guide for scrum teams that’s the bat size and flow optimization that you might be looking for to take your not so good looking scrum to the next level that’s a question but I guess you know but you know this whole idea kind of starting somewhere right we get it right you know please you do I was gonna say you look you know not feel the good place I yeah yeah you might be building the rain I don’t but you’re in a fairly good Wow comes by thought you said LA you wanna add value to the customer all right well not many other questions coming through so I’m just conscious entertain some minute um do you want to see a couple of stuff you’ve got coming up I’ve got coffee some things you’ve got the paper in and all you’re gonna have to move to virtual and you’ve probably done a bit of work to do on that front but you want to just if we’re we’re scramble here yeah we’re scrambling around a little bit trying to figure out how to do this not just from you know flip that switch and make it virtual but how do we can actually deliver classes virtually um yeah so I’ve got a bunch of classes that I had a pre set up for Edinburgh and I think in two weeks I have a PSM and a provincial scrum master and a professional scrum with UX class and they’ll both be moving to virtual but they’re pretty near nobody signed up for them yet mainly because of all the things that are going on and so if anybody’s interested please let me know about that and then Daniel Vacanti who was one of the original Kanban method founders a Corbis and who co-founded the LGU with David Anderson it is going to be teaching professional scrum with Kanban with me meant to be an Edinburgh class again but we’ll be doing it online and I think after that we have our Apple II professional agile leadership essentials which focuses that that middle management layer in an organization that doesn’t work directly with on a scrum team that works with scrum teams how do they move their game from management towards leadership that’s the idea so we’ve got some stuff coming up you can find it on my website you can find it on the scrum that org site as well my website is nkd agility calm and if anybody has any questions Martin and KD agility comm also mr. hench on Twitter and wherever else you can find me if you google for me you’ll find tons of stuff excellent brilliant fried tomorrow that’s great right just talk so just a couple of committee announcements [Music] what’s gonna be hopefully these vervain so next week Thursday the 26th our friends at the heart of agile we talked I’ve got an amazing event actually they’ve got a nice soft cuny for charleen we aren’t gonna be hosting Jeff Watts I’m sure many of you here have his books and how it’s gonna be that’s gonna be awesome I’m signed up for that one there’s me really good it’s a beast I’m on the way yes yeah are you know I was in there fast so yeah eighty folks lined up for a starts gonna be brilliant and then our next event is in me know for that 57th that’ll be the right to left if any of you’ve read that so that’s by our friend Mike Barroso he’s gonna be doing a remote event talk about some of the key ideas and not so that’s one for your diary if you’re interested not you’ll find all these on Meetup so yeah sorry not before they fill up and I think that’s it really if there’s any others yeah we can start elected on to the LinkedIn group if anybody’s interested in joining laughs you know Kieran keep talking a favor socially didn’t swing you know it’s it’s not be distant keep the conversations going and thanks for joining us evening and yeah stay safe look after yourselves and your loved ones billion thank you thanks guys okay everyone south and it slowly thank you BCS is

Events and Presentations

Connect with Martin Hinshelwood

If you've made it this far, it's worth connecting with our principal consultant and coach, Martin Hinshelwood, for a 30-minute 'ask me anything' call.

Our Happy Clients​

We partner with businesses across diverse industries, including finance, insurance, healthcare, pharmaceuticals, technology, engineering, transportation, hospitality, entertainment, legal, government, and military sectors.​

Kongsberg Maritime Logo
DFDS Logo
Big Data for Humans Logo
Qualco Logo
Workday Logo
Hubtel Ghana Logo
ProgramUtvikling Logo
Boeing Logo
Illumina Logo
New Signature Logo
Milliman Logo
YearUp.org Logo
Higher Education Statistics Agency Logo
Capita Secure Information Solutions Ltd Logo
Teleplan Logo
Sage Logo
Lean SA Logo
Healthgrades Logo
Washington Department of Enterprise Services Logo
Department of Work and Pensions (UK) Logo
Royal Air Force Logo
New Hampshire Supreme Court Logo
Nottingham County Council Logo
Washington Department of Transport Logo
YearUp.org Logo
Higher Education Statistics Agency Logo
Emerson Process Management Logo
Epic Games Logo
Akaditi Logo

CR2