We are going to take a look at how to do a migration with the Azure DevOps migration tools. I’m going to take a very straightforward simple migration and walk you through it. We’re also going to take care of any exceptions or issues and all those little things that come up, at least the common ones that come up. So let me switch to my desktop. There we go.
So first thing I’m going to do is install the Azure DevOps tools. We need to get them first. If we go to GitHub, the tool repo, you’ll find just down on the right-hand side a little list of all of the releases, and the latest release will be at the top. So you can use this release here. Whatever the latest one is, is the one you should be using. We only support the latest one. We have one going through the pipeline at the moment, but if you open up that release, if you’re in an environment where you can’t, you don’t really have internet access, you don’t have any of those things, then you can use this version of the file here. You just download that, and you will be able to unzip it to any location, any folder, and then run the migration from that folder.
You can do that. I’m not going to do that today because if you check out the documentation, which if you scroll down a little bit, you will find a link here to the documentation. In the documentation, it talks about how you install and configure the tools, and there’s a little getting started link up here in the top left. It talks about the install. Now we deploy to Winget and Chocolatey. So if you are on a Windows Server, you need to use Chocolatey. You can use Chocolatey anywhere, in fact, that you’re running Windows, but we also have Winget as well for those of us that are on Windows 10 or Windows 11.
So let’s put this in the other mode. I’ll switch over here. There we go. So what I’m going to do is I’m going to open a command prompt, and one thing to note is that you want to use a non-admin command prompt. There’s actually a bug in Winget that if you use an admin command prompt, it doesn’t add the app folder to the path. You don’t have to worry about what that means, but they don’t add it to the path, which means that you get a missing file.
So I’m just going to find the package. If you do Winget search, then I’m going to do Azure DevOps and pull up the list of tools that are available for Azure DevOps. I’m just going to copy that ID there, and I’m going to do Winget install that tool. You’ll find that will shoot off and go do that. It’ll take a minute or so. It’s going in downloading the package, verifying it, and then unpacking it and setting up some stuff. But again, it is just a portable setup.
While that’s running, you can go to… do I have the Chocolatey link in here? No, I maybe don’t have the Chocolatey link anymore because we want to use the new stuff. So there we go, starting the install of the package, and it’s set it up. So now if I switch to C temp and do dir, you’ll see that I’ve got a config file in there. I’m actually just going to overwrite it and create a new one by calling the tools.
So if I go CLS, right, so if I want to use the tools, I just call devops migration. That will initially show you, because you’ve not selected like what do you want it to do, it will say what the options are. There’s init, and execute. Execute is for executing a migration, and init is for creating config files. So if I do init and then do help, you’ll see it’s got some helpful options there for what things are available.
But we’re just going to call init, and it will… there we go, it’s updated and created this config file in C temp. So if I just pop that open and go to temp, there we go, I have that configuration file here. I’m going to open it in Visual Studio, and there we go, I’m all set up with this is the default out of the box configuration file.
We’ve got some changes to the default configuration file coming down the pipeline just now. They’re not quite here yet, so there’s a couple of fixes that I have to make. So if you have an older config file or you’ve generated one before, there’s some fixes that you’re going to want to make. The first fix is going to be to remove all of the field maps here. Field maps are great for when you want to move from one process to another process, but they get in the way, especially the default ones do get in the way.
So we’ve changed that. Although we’re on 2.23, I think if you do 2.2.3, which has already been updated while the package is being deployed, then you will have these things already removed. So you need to remove that, and then down here, there’s a skip to final revised work item type that should be false by default. It’s currently set to true.
The other one is these node paths just get in the way, so the best way is just to delete them and replace it with null, and then they are gone. We don’t have to worry about them. Those are the two things that you have to… that will be the default what you see here will be the default by tomorrow.
So what we’re going to do is we’re going to set up this for us. So we have a source project, and we have a target project. Okay, now the source project, we’re going to use prompt to connect to it, and we’re going to connect to an Azure DevOps instance in preview, but not this project. That one doesn’t exist, so I’m just going to open that and go to migration source. This is my default migration source project here.
Something to note here is that you will need both the URL of the collection and the URL of the project. So this one here is the project, and this here is the collection. So if I take… I’m just going to copy the project because I already have the collection in there, and you’ll see here’s the collection, and I’m going to update the project.
Now for the source, you don’t actually have to worry about reflected work item IDs. It’s not required for you to update this. It can be anything you like. It’s not going to use it on here, and you can leave this as prompt because we’re only going to read from the source. We never write to the source; we only read from it. So I’m just going to use prompt there because it’s just easier just now.
Then we need a target project. So what I’m going to do is I’m going to create a brand new target project, and I’m going to call it migration target three because I have lots of ingenuity on what to call things. So I’m going to use the same process. So the source is also this process, and I’m going to create that project here.
While this goes off and creates the project, this is going to create an empty project. It is worth noting… there we go. So this has no work items, nothing here, recently updated, nothing, recently created, nothing. Okay, but it does have the process here, and if you’re looking for the process, I just clicked the project settings on the bottom left, and then once you’ve got that open, you will see the process that’s connected to your project in here.
If you click on it, that will go into the definition of the process so you can take a look. Even if you can’t edit this, you will be able to read it, but you will need to be able to edit it on the target environment for sure. Okay, because every work item type that’s in scope needs a special field on it so that we can track which work items we’ve created and what they’re related to in the source environment.
For that, we actually add a field called reflected work item ID. If you are migrating work items, restructuring your Azure DevOps, or just need some help, my team at Naked Agility can help you, or we can help you find a consultant or expert who can. You can set up a free consultation using the links below, and don’t forget to like and subscribe.
So I’m just going to add a new one just to show you what it would look like for you. So I’m going to create a new field ID. I’m just going to call it reflected work item ID, because I think I already created one called reflected work item ID 3. It’s a single text line; it can go anywhere, no default value, and it can go anywhere. So I’m just going to stick it in details. I don’t care.
Okay, so there it’s added. If you go to edit again and go to options, you’ll see what the name of that field is. Okay, so this is the ref name rather than the friendly name. This is the friendly name here, and then the under-the-covers name is the custom reflected work item ID 3. If you’ve added or had already got a reflected work item ID in your system like I do, would you remove that one, which is this one? You can go check in options what the ref name is, and you can see this was created quite some time ago, and Azure DevOps has changed the way this works. Yours will probably say custom if you’re adding it recently.
If you’ve added a long time ago, it will say the name of the process within which it was added. So that’s M NK Scrum. So I’m just going to copy that, go back in here, and now I can update the target element to M KD Scrum that reflected work item ID.
The other thing that’s worth noting is that in the target, if you’re using Azure as your target, you will need to fill out a personal access token. There are certain capabilities in ADO that require us to do some weird patchy things that require a PAT token. So all you do is on your environment, click on the little person cog with personal access tokens and create a personal access token. You see I created one earlier, so I’m just going to regenerate that one.
There we go, I’ve got my personal access token. Now you’ll notice I have it set up as full access. You can try other stuff, but we tend to use full access when creating these. So now all I need to do is put in my personal access token in there. If you want to only use the PAT token and not use prompt, you can say access token in the authentication mode, and it will just use the token.
So we can set that up here. So we’re going to use prompt for the source and access token for the target. Prompt just pops Azure Active Directory authentication, and that’s easy peasy. You don’t need to worry about these language maps. If you have… you may need to worry about it in the source if you’re migrating from, let’s say, a German installation of TFS where the database is set up as a German collation, then you’ll need to fill out whatever the German is for area and whatever the German is for iteration because that’s how the system stores it, and then that will make sure all the mappings work.
So you just fill out your language-specific terminology in here, but 90% of the time, you’re not going to need to do that at all. So then everything else is kind of not needed for a simple migration. If you’re changing the process or you want to move data from one field to another, I’ve done things like I had a customer who had the version number as year dot number. I think it was year dot month in their source system, and in the target system, they wanted two fields, one with year and one with month.
So I just added a field map to regex those out into the individual fields. No problem at all. The documentation has a list. I jump to the docs. The docs have a list on the left-hand side under V1, under ref V1. You’ll find a list of field maps here. So there’s the regex field map, and all I did was set up a source field being my version, target field my version year only, pattern match the year out, and put it in the new field.
Right, that’s pretty straightforward. So you would just copy this, obviously change it to what you need, and put it into the field map. Okay, so it’s just a list of field maps, and the field maps are run in the order that they’re listed there. Right, so you can control the order a little bit.
If you have links from your work items to Git repos, then you will need to have those links fixed so that in the new system, they will match up. It’s worth noting that we only fix the work item connection to the Git repo. We don’t fix the Git repo connection to the work item. So in your Git commit text, you might have IDs or work items that no longer exist. They’re not links; it’s kind of invalid; it’s just text.
But the link from the work item to the Git commit will be fixed with the tool. So if you’ve changed the name of the Git repo, you just create a name list of name-value mappings in here, similar to the language map, and have the original Git repo and the target Git repo under here. You only need it if you’ve changed the name of your Git repo during the migration. If it’s the same name on both sides, it doesn’t matter; you don’t have to do anything in there.
Okay, so then the main body in here is the processor. That’s the bit that’s got all the goings for actually doing the migration, and we’re actually pretty close to being able to run a migration at the moment. So we’ve set up the connection, and then in here, we have the migration. The important bits are what type of processor it is, and then you’ve got your work item query bit here. That’s one of the most important sections here.
We’re going to run into a few problems when we do our migration. We’re going to fix them in here, so I can show you what that looks like. So what I’m going to do is just show you this query. So in this query here, you can see it excludes certain work item types. These are work item types that we do not migrate at this point using this tool. There are things you can do with shared steps and shared parameters, but that is definitely more advanced.
Feedback requests, test plans, and test suites should never be migrated with the work item migration processor. There’s a separate tool for doing that stuff. So in here, this is this bit at the front, and closed date equals nothing means that it will only migrate things that aren’t closed. But I can just delete that, and I will just not migrate those types of work items. Everything else will come across.
That’s basically everything. This is great if you have a really big migration. If you’ve got more than 10,000 things that come back from your query, you’re going to get an error. What you’re going to have to do is shape your query so that you bring back chunks of smaller than 10,000 work items. So you might do that with date range. You know, let’s migrate everything that’s been changed in the last 90 days. Migrate everything that’s been changed between 90 and 120 days, and then keep making those chunks in order to migrate it across.
And yeah, so that’s it. We’ve set up our query, we’ve set up our connection, we should be good to go. So I’m going to run the migration. So I’m going to switch back to my codes. There we go. And what we’re going to do is we’re going to do devops migration execute minus C for the config file, and it’s c m.
Is it JSON? I actually can’t remember. What is that configuration? There we go. So I’m going to run that. There we go. So it’s loaded the file, and it’s done nothing. So this is the first thing that will happen when you try and run it because you have forgotten something in the config file. You’ll notice that it has processor enabled equals false. We need to change that to true for it to do anything.
This is so that you can actually set this up so that you can have multiple configs in there and then enable and disable them as needed. I tend to have multiple files just to make things less confusing. But now I’m going to go back and run it. So this… boom. Okay, so now you can see, oh, it went quite quickly there, and I got an unhandled exception. I maybe messed something up, so I will need to check that.
So it connected here to the source. Grant it. Then it tried to connect to the target with access token, and then I got access granted. Oh, look what I did. See, there we go. What I did was I forgot to update the project name. Always something that Martin does that messes things up. There we go. Let’s go back to… so it’s migration target three was our empty project. There it is, migration target three.
So there we go. I forgot to update that. So clear. There we go. Let’s run it again and see what happens. So connected to the source, connected to the target, and then it’s doing a node migration. So it’s creating the area and iteration nodes in the target. All of these little things here, and they have to match.
We use a thing called rules. Overrides, can’t remember what it’s called, but rules bypass. We bypass the rules engine, which means that all of the areas and iterations that are on the work items must exist. So the first thing we do is we go through… if I make that smaller, there we go. We go through, and this is checking and validating all of the areas and iterations that are in the source and checking that they’re in and valid in the target.
But what we also do is after we’ve loaded the work items, and you see there’s 10 work items, we actually go through the full history of all of the work items, and we pull out all the area and iteration paths, and we check that they all exist. We check that they all exist because in the history, there are two things that might have happened. You might have an area or iteration path that’s from a previous project if somebody has moved a work item from one project to another.
That only happens in Azure DevOps, but it’s possible. The other issue is that you might have an area or iteration path that has subsequently been deleted. So then it doesn’t exist anymore in the source. So when we create the main set, it doesn’t exist. So what we’re going to have to do is we’re going to have to create a mapping for these. The system will stop and say there’s a missing iteration. There is one that is missing that we need to deal with. Actually, there’s two, right?
Because what it’s actually done here is it’s created these two, which didn’t exist before, so it’s generated them on the fly. But we’re still missing this one called Sky Point Cloud because it doesn’t recognize what to do with it. Right? It’s from an older project. So if I open up work item 414 in here, so if I just go 414, it should let me just… oh, I might need to go to the source project.
414, there it is. So here’s 414, and you can see, well, it looks like it’s got perfectly valid iterations. But if I go in here and all the way back to the past, you’ll see that the team project that it was created in was Sky Point Cloud. That is not this project. This is migration source. So it’s been moved from one project to another, but that data is still in the history.
So it’s had history, and then it’s been changed and moved to this project. So we need to go deal with that in the config. So the config has been helpful and just said, “Here’s the missing thing you need to add.” So I’m going to grab that because that’s what we need to create and go into the config.
So down in the config under the processor, you’ve got area maps and iteration maps. I’m going to give you some secret information as well that you need to add. So effectively, we have an iteration map that we need to add for this text. We want to match this text and map it to something else.
So I’m going to give you some secret sauce. First is that in JSON, you need to escape slashes. That’s why that’s doing that weird underline. So if I put another slash in, we’ve escaped it. But because this left-hand side is a regular expression, we need to escape the escape slashes. So we need another two slashes to escape. So when this goes through the JSON, it will turn into two slashes to go into the regex, and the regex will turn it into one slash for it matching.
I know, madness, but there you go. And what we want to do is we only… you know, we might have an iteration path that has Sky Point Cloud in it, like old project SL something/Sky Point Cloud/Sprint One, and that would match that. So just to protect ourselves, it’s better to put a hat at where you want the start of the string, i.e., this has to be at the beginning, Sky Point Cloud, and then you can put a dollar at the end to say that is the end of the string. Don’t match anything after that.
So we only match this exact text, and then our target project is migration target three. So where we want it to go is migration target three. We need to escape that slash, and then Sprint One maybe, I guess it could be Sprint One. Let’s check.
So migration target three, let’s go to project iterations. There’s a Sprint, and there’s an iteration that’s all messed up, but you know, that’s what there is. So we’ll put it in Sprint One. And then just to give you a super secret thing, there’s a bug at the moment where it doesn’t list the root as a problem.
If you remember from viewing the history, you’ll note that the area path was also Sky Point Cloud. So we actually need to match just Sky Point Cloud on its own and map it to migration target three for the area maps. Okay, so you just need to remember at the moment that if you’re doing a project to project map, sorry, a project rename or change, then you need to move these in.
Okay, so now if I go back and run this migration, you’ll see it will skip past. There’s loading the work items. It’s getting a list. It’s going to check it twice. There we go. Now it’s migrating work items. So it’s iterating through each of the work items, creating them. We iterate through each work item in turn.
Let me make that smaller so it looks nicer. We iterate through each work item in turn. So when you’re doing work, if work item A links to work item B, we’re going to process work item A first, and we’re not going to be able to create the links. You’ll see on the first work item, it will actually usually say, “Migrating link, skip, skip, skip.” Right? That’s absolutely fine because the other side, the target side, doesn’t exist yet. The target work item has not been migrated yet.
But as it goes through, you’ll see skips, and then as we go further down, we’ll start to see creates. Right? It’s going to create them. So when it creates the other side of the link, it’s going to link it back together. If I go down to the bottom, it’s probably pretty close to finished. It is finished.
So we have just migrated all the work items and links for those work items, and we did it fairly quickly. It took 48 seconds. There we go. So let’s take a look at what we created. So when our migration target… I’m going to load up work items. Migration target three, I’m pretty sure that’s what I called it. So let’s have a look.
There we go. They are there. It was just a caching issue. If I go back, can I… there we go. Here are the work items that we just created. You can see there’s the one that was moved from Sky Point Cloud, and in the history, all the way back in the source, it now just says migration three, right? Because we did that core mapping.
So we got all the work items. If I go to the backlog, close that, you’ll be able to see that it’s linking things in. If I change this to look at appearance as well, you’ll see there’s the feature, the backlog item, and the task coming in as well. And we did have some other work items on there, right? We had PPIs, we have some tasks and stuff, so we’re not particularly worried.
And there’s all the stuff there. So there you go. We just migrated our first set of work items. So this operated fairly quickly. It brought them all across. If you want to run it again, you can run it again. And what it will do is it will go check everything twice.
Thinking about how we might resolve that, but it checks everything twice. So it’s going to go through each of the… it’s actually not going to check all of those work items again. It’s just going to say that all work items were found. But what will happen is at some point, somebody will go to the source, and they’ll go onto the board, and they’ll drop that item from resolved to active, right? Because it regressed.
So now when we run this, we actually want it to migrate that extra change, but it will not because at the moment we’ve got a thing that will remove all found work items. So what you need to do is go in here, filter work items that already exist in target, false. Once you’ve turned that to false, let me do clear, rerun the migration, and what it will do is instead of just not trying to run all those work items, it will run through each of the work items.
It will check the number of revisions, and it’s found one work item that has one revision that needs to be updated. And there you go, it has moved that revision across. It’s going to check the links as well and make sure they’re all up to date, and it skips it because they’re already existing. So it’s checking every revision of every work item that exists, that it’s correct, that all the links work.
So it will eventually find that work item. There we go, run the whole thing. And if I go back to… so that was the source. We go to the target, go to the board. It should have moved across. There you go, it’s now moved across. So we rerun the migration. Well, that’s… oh, there’s a bug. It shouldn’t do two of those, but that’s the different migration run that we can turn that off in the config.
But there’s that additional change that came across. We changed the state, and then it’s adding this text, which it shouldn’t do. But in here, my migration comment, it won’t do that again. It can be handy if you want to do it for the first run because in the actual work item, it creates a link back to the old work item. So you can quite easily just click it and go, “Oh, what was the old work item again?” “Oh, it was this one. Does it match?” “Yes, it matches.”
Right? So people can find that. So there you go. That was a pretty straightforward migration between two Azure DevOps projects, moving the work items.
