When it comes to migrating data in Azure DevOps, the process can be overwhelming, especially if you’re dealing with legacy systems or older versions of Team Foundation Server (TFS). As someone who has been through this many times, I can tell you that there are countless pitfalls, but with careful planning and attention to detail, you can avoid most of them. In this post, I’ll share some of the common issues, critical steps, and personal experiences that can help you successfully navigate the complexities of Azure DevOps data migration.
Common Issues with Older TFS Versions
One of the biggest challenges you’ll face when migrating to Azure DevOps is dealing with older versions of TFS. I’ve encountered situations where clients were still using Visual SourceSafe, a system that went end-of-life in 2005. Before you can even think about upgrading TFS, you need to import any old data from systems like Visual SourceSafe. If you skip this step and upgrade TFS first, you risk losing critical historical data since Visual SourceSafe imports were only supported up to TFS 2015.
Key Takeaways:
The Importance of Order in Migration
Order of operations is crucial in any migration process. Whether it’s moving from TFVC (Team Foundation Version Control) to Git or making process template changes, the sequence in which you perform these tasks can make or break your migration.
What Can Go Wrong?
Process Template Changes: Make these changes on-premises before pushing to Azure DevOps.
Source Control Migration: Migrate your source control to Git before moving to Azure DevOps.
Account Alignment: The most critical issue is ensuring that identities align correctly during the migration. If not handled properly, users may end up with duplicate identities, leading to confusion and lost work.
Personal Experience:
I once worked on a migration where a customer moved from a workgroup environment to a domain-joined environment without proper account alignment. This resulted in users having multiple identities in TFS, leading to a situation where they couldn’t find their assigned work because it was linked to their old identity. This issue is not just a technical glitch; it’s a workflow nightmare that can severely impact productivity.
Critical Identity Alignment
Identity management is one of the trickiest parts of migrating to Azure DevOps. TFS handles identities in a way that can cause significant issues if not managed correctly.
How Does It Work?
TFS Identity Wrapping: TFS wraps your actual identity (whether a workgroup account or Active Directory account) in its system.
Migration Risks: If your accounts don’t match when moving environments, TFS might create a new identity instead of mapping to the existing one. This results in users not being able to see their work assigned to their old identity.
Case Study: Complex Migration
In a particularly challenging migration, we had to move a TFS environment from inside Microsoft to another company after a business sale. The process involved seven separate migrations, each with different scripts, validations, and legal requirements. The biggest headache was maintaining identity alignment throughout the entire process. If even one identity didn’t map correctly, it could have caused significant issues down the line.
Database Size and Cleanup
Another critical aspect of migration is managing your database size and cleaning up old data. Azure DevOps has limitations on database size, so it’s essential to clean up unnecessary data before migrating.
What to Watch Out For:
Legacy Systems: Older TFS versions might have accumulated a lot of “baggage” over the years, including half-committed changes, outdated scripts, and beta versions that were never fully supported.
Database Cleanup: Make sure to clean up your database before attempting the migration. This includes removing obsolete data, fixing old errors, and ensuring everything is in order.
Backup Procedures and Disaster Recovery
Backup procedures are often overlooked, but they are crucial to ensuring a smooth migration. Most operations teams will use standardized backup tools that might not be compatible with TFS or Azure DevOps. This can lead to incomplete backups and, in the worst-case scenario, a failed migration.
Best Practices:
Use TFS-Documented Backup Procedures: Always follow Microsoft’s documented procedures for backing up TFS or Azure DevOps. Using non-standard tools can result in incomplete or corrupt backups.
Mark Transaction Logs: Ensure you mark transaction logs across all databases before backing up. This ensures consistency across multiple databases during the backup process.
Real-World Example:
In one case, an operations team used their standard SQL Server backup tools for TFS. Unfortunately, this approach didn’t account for the complexity of TFS’s multiple databases. During a disaster recovery scenario, they were unable to restore the databases properly, leading to significant downtime and the need for Microsoft engineers to intervene.
Ensuring Successful Migrations
Despite the challenges, migrating to Azure DevOps doesn’t have to be a nightmare. With the right approach, you can avoid many common pitfalls.
Key Steps:
Plan the Order of Operations: Ensure you follow the correct sequence for all migration tasks.
Align Identities: Double-check that all user identities are correctly mapped during the migration.
Clean Up Your Database: Remove any obsolete or unnecessary data before migrating.
Follow Backup Best Practices: Use TFS-documented procedures and mark transaction logs.
Final Thoughts:
90% of the time, migrations go smoothly if you follow these guidelines. However, when something does go wrong, it often requires deep expertise to fix. The Azure DevOps team has done a great job of making the migration process easier, but there’s always a chance that something might slip through the cracks.
Conclusion
Migrating to Azure DevOps can be a daunting task, especially if you’re dealing with legacy systems or outdated processes. However, with careful planning, attention to detail, and a thorough understanding of the potential pitfalls, you can ensure a successful migration. Remember, the key is in the preparation: plan your order of operations, align your identities, clean up your database, and always follow best practices for backups. With these steps, you can avoid the common pitfalls and ensure a smooth transition to Azure DevOps. 🚀
When you’re migrating data around in Azure DevOps, there are a hundred thousand things that can go wrong. There are some really big things that people run into. If you’re using the Azure DevOps data migration service that Microsoft provides to take your whole collection up, big things are we’re on an older version of TFS that’s not supported. I did one recently and discovered during the discovery that the customer still had Visual SourceSafe. So we had to do a Visual SourceSafe import first. Visual SourceSafe I think went end of life in 2005, maybe 2005, so it was stuff that had been around for a long time. We had to bring that in first because if you don’t bring it in first and you try and upgrade your TFS first, the VSS import was only supported up to TFS 2015 because after that it’s ten years after VSS was end of life. Microsoft stopped having that functionality in TFS, so if you don’t bring it in before that, you can’t bring it in, not with full history and all of the things. They discontinued those tools.
One of the most common issues that people run into when they’re doing migrations is doing things in the wrong order. If you push up to Azure DevOps and you’ve done things in the wrong order, you might want to do process template changes. You want to do them on-prem before you push them up. You might want to bring in other source control stuff; you need to do that before you take it up. You might want to move from TFVC (Team Foundation Version Control) to Git; you probably want to do that before you move up, but you can probably get away with it.
The biggest one, and this is the one that there’s absolutely no way back from, is account alignment. This is just mental. The way TFS does identity is TFS has an identity, and it wraps whatever the actual identity is. In the old days, this was either a workgroup account if you’re familiar with Windows authentication, or it was an Active Directory account. The issue is that if you take a workgroup and then move it into a domain-joined environment and convert it to a domain-joined, but the accounts don’t match up, TFS will automatically, when you move between environments, try and find the account that matches the one that already exists and just swap out the bit inside of the TFS identity. But if it can’t find it and it’s already added to the systems, it will create a new identity.
So you’ll end up with two TFS identities. If you’ve got Bob in your company and Bob has the old TFS identity and the new TFS identity, when he logs into TFS and he searches for “show me all work that’s assigned to me,” he won’t find anything because it’s assigned to old Bob, not new Bob. That’s an order of operations issue, and it’s really hard in Active Directory because quite often there are groups and things that add people into services before you want them to be added, and then you end up with this problem.
Azure DevOps combats that a little bit when you move stuff around because even once you’re in Azure DevOps, you can move from tenant A to tenant B. Your Microsoft intra-tenant identity management can be swapped out, and it does the same thing. But it’s going to ask you, “Here’s a list of all the users that we couldn’t match up; do you want to match them up?” If you click that skip button at the bottom, which if you have somebody who doesn’t know what they’re doing, people are apt to do. I find so many times things go wrong because people click the skip button because they think they know what it says or they don’t care what it says, or they just want to do it, or it’s not their job, or whatever reason.
There is no way going back to the old way because now you’ve got both those identities existing, and there’s no way to merge those identities. If you want everything to align through your upgrade process, then all of those things need to be taken into account. Probably the absolute worst case of that having to be managed was years ago when Microsoft sold a part of their business to another company. We had to get all of this work right—the TFS environment from inside of Microsoft to inside of this other company.
As you can imagine, with legal and compliance and all kinds of things, it wasn’t that easy. When you work within a company, you take dependencies on lots of things that maybe your company owns, but then you sell this thing and you’re not selling these things that you’re taking a dependency on. So how do you get it out? I think we had seven migrations that we had to do with this single environment. There were three outgoing, one outside neither environment, and then three incoming into the new company. At each of those levels, there were different validations, different scripts, different things that had to be run against those environments to ensure that we don’t lose anything and we don’t get anything we’re not supposed to.
It was an absolutely horrendous nightmare of convoluted legalese to get it all done. The biggest issue was maintaining those identities all the way through because the people were going as well, and they wanted all of the identities to match up on the other side. You have to ensure that you map all of the identities correctly all the way through so that when you get to the other side, people open it up and go, “Oh, there’s all my stuff,” rather than, “Oh, where’s all my stuff gone?” Then they have to go find it, and they have to go create custom queries and all kinds of stuff. It’s just a pain in the butt.
Those are some of the bad things that can happen with identity management. There’s a lot of gotchas in there. Database size moving up to Azure DevOps, the size of the database and what’s in there and how you clean it up—that’s a bunch of gotchas in there. If your company, which you should know developers are apt to do, you might find that your TFS environment has been around for a really long time, and it didn’t used to be run by your operations team with control. It maybe used to be run by the developers before it was handed off to operations.
Before they knew that in the old days, operations often thought that anything that developers did was non-production, even though all the code is an organisational asset and we’re storing it in there. The build environments and build agents are all organisational assets because we can’t deliver our product if we don’t have it. They were all considered not production, so developers could have the keys to everything and do everything for so many companies like that.
The problem is that because they can do whatever they want, they go, “Oh, there’s the new beta version of TFS. Oh, it’s supported by Microsoft, so we’ll install that.” But it’s mostly supported by Microsoft. I’ve had environments where it’s a major enterprise company, but in the midst of time of that environment, it’s had a beta version of TFS deployed that was maybe supported way back ten years ago. It causes knock-on errors because there were a couple of things that weren’t taken care of that mean it can function the way it is, but it ain’t going to Azure DevOps without fixing all of those things.
If your server is old enough, there are all sorts of buggy things from TFS 2010 that they fixed in 2012. If your server’s old enough to have been in 2010, there’s just a whole nightmare of stuff left over that you have to go fix before you can go to the cloud. You can have half-committed stuff because you had to do two commits back in the day to do a rename, but nobody told anybody, and there was no error message.
In Service Pack One, they brought in the error message, and then in 2012, they fixed it, but there’s still a bunch of stuff in there that’s messed up. If you imagine a product that’s been around for 18 years—no, not quite 18 years, 16. Where are we? It was created in 2005, shipped in 2006. Man, that’s a long time, right? Nearly 20—nearly two years off 20, 18 years. My math sucks. Eighteen years of engineering work has gone into this product. Eighteen years of different people working on it. Eighteen years of forgotten stuff. Eighteen years of bodies being buried underneath some of the classes and methods.
Those are probably where the biggest difficulties are. The easy stuff, the stuff that can go right—man, I did mention the migration up to Azure DevOps. This was really early on in Azure DevOps. It took less than an hour. This was a government agency in the UK. It was a council, so not government government, but local government. We ran the tool against it; it said everything was good. The database was a couple of gig, tiny, so we just sequenced the D pack, did a dry run up into Azure DevOps. That took about 15 minutes, less than 15 minutes, and it was there. They looked at the dry run; they thought, “Oh, this looks awesome. This is perfect. Everything’s where it’s supposed to be. We see all the stuff. We trust Microsoft.”
They were just reasonable folks. We just went, “Okay, do you want to just do the production one then, or do you want to wait and validate it some more?” They were like, “No, just run the production one.” So we ran the production one, and we were done. Turn off TFS. That’s one headache off their plate, one headache off their local admin’s plate.
Hold on, we need to go back to what’s the worst—the absolute most appalling issue with TFS or Azure DevOps Server. Running it on-prem, especially if your developers are running it, this will be done right. If your operations team is running it, I guarantee you it’s wrong, and that’s backup. Most operations teams want to use their standardised tools, so they just see a SQL Server and they want to plug their standard backup tool into Azure DevOps and suck out the data, store it off-site. If they’re doing that, that does not guarantee a restore.
The problem with systems like TFS—it’s not really a problem with TFS; it’s a problem with having people that don’t know what they’re doing running it or that don’t care running it. It’s actually the same issue for SharePoint and a lot of other systems. You have multiple physical databases inside of the system. When it runs updates, you could have an update that straddles multiple databases. In TFS’s case, you’ve got a configuration database and then you’ve got a bunch of collection databases.
You can have an identity update, for example, that will hit the collection and the configuration database. The configuration database is small, so that bit will be done really quickly, and then it needs to complete on the other one. They run it as a transaction. It’s absolutely possible to hit the point in time of your backup that it’s completed and committed on one database and not completed on the other one. So it will roll back because the transaction has crossed multiple databases, and this one’s finished and it’s moving on to this one. The transaction allows you to roll back the whole thing, but not if you restore a backup without that transaction being in existence mid-transaction.
There are things that you’re supposed to do that are all documented by Microsoft to enable you to do marked transaction logs. You actually put a mark—you use the transaction system to put a mark across all of the databases, and then you back up. The transaction completes, all of the other transactions are completed, this transaction completes and has this mark, and it might be at different timestamps across the different databases. Then you back up that mark, and when you restore that mark, it’s exactly the right point.
But most people don’t do that, and they validate things by doing a trial restore. It’s totally possible that that restore will work. Probably in most circumstances, that restore will work. But you know what Murphy says: what’s the worst thing that could happen? It will happen at the worst possible time. So in an actual disaster, when you need to recover your database, that’s the time when you’ll hit that transaction log. If you hit that, you are not going to be able to restore your databases, and you’re going to have to call up Microsoft, and they’re going to have to have an engineer log into your servers and run scripts to fix the databases.
Then, because they’ve run scripts to fix the databases, that’s another thing that can end up being a future problem when you go to move to Azure DevOps, and something in the database is not quite right when they do the validation, and it fails the import. My advice is never, ever let an operations team run TFS. That’s my advice. If you have to let an operations team run TFS, ensure that they must use the TFS documented backup procedures, not their own stuff. That won’t work and might end up with you in the hole.
Yeah, so that’s the big one. Ninety percent of the time, things go perfectly fine. Upgrading TFS is super easy. It used to be an absolute nightmare, but the Azure DevOps team did a bunch of work to validate everything, check everything, and that’s part of their ethos. If you run it, it will tell you, “Here’s a problem; here’s how you fix it.” You fix it, it runs, and you can do the upgrade. Totally cool. They’re super easy. But when something isn’t right, that’s when you need that expertise to go, “What is the problem? How do I fix it? Where does it come from?”
The documentation is awesome, but there are still those things that slip through the cracks. So that’s really the thing. There are a lot of things that can go wrong and a lot of things that can go wrong that are really important, but 99.9% of the time, everything’s going to be fine. You’re going to be able to upgrade; you’re going to be able to migrate, and everything’s going to go great.
