My name is Martin. I work for a company called Naked Agility in Scotland doing digital transformations, DevOps, agile with lots of customers. I want to talk a little bit about the VSTS sync migration tools, but the first thing I want to mention is that there are more than one option for migrating from TFS to VSTS. If you want to take your entire collection, so you want to migrate your entire organization from TFS to VSTS, then the primary option you should be considering is the VSTS migration services from Microsoft.
This tool is a full fidelity service. If you go to the URL at the bottom of the screen, you will be able to see the caveats for what they do, but they basically take your entire database, your collection, and import it into VSTS. This means that you will have a little bit of downtime depending on the size of your collection. I’ve done small collections from, I don’t know, 50 gig, 58 collection that took a couple of hours to do the migration, and I’ve also done a two terabyte, two and a half terabyte collections which took 40 odd hours to do the migration. The bigger the data, the more complicated it is moving that amount of data around because you have to get it up onto Azure, and then Microsoft have to pull from that location into their service which then processes your data into VSTS. But it is as close to full fidelity as you will get, and Microsoft are working to make that full fidelity migration as powerful as possible. So if you have an option, take the VSTS migration services option. That is your primary go-to.
If, however, you have a different scenario, maybe you don’t want to move your entire collection just now. You only want to move a few team projects up to VSTS, or you have five projects in TFS and you want to end up with one team project in VSTS, or you have one project in TFS and you want to end up with five in VSTS, then the VSTS migration service from Microsoft is not going to be able to fill that need. That’s one of the reasons that I created the VSTS sync migration tools. It’s scoped by query, so basically if you can query it, there are certain things that are the bounds of the team project. We’ll talk about those as we go through, but it’s generally scoped by query, and it is not even close to full fidelity.
I’ve written a bunch of processors. I’ve had help from many contributors to the project to add additional processors that are importing data from one TFS/VSTS location to another. Because the tools don’t really care where your data is and where it’s going to, you can do TFS to VSTS, TFS to VSTS, VSTS to TFS, you can split team projects, you can merge team projects, you can pull a subset of the data out of one and push it into another. You can do bulk updates; you can do all sorts of things. I have done bulk updates where the customer wanted to migrate from the agile template in TFS to the scrum template in TFS. So one of the tools that I used to do that was this one with the bulk update. I have a bulk update processor in there, and you can give it a bunch of field mappings, and it allows you to process the data from one format into another format in place on your TFS server. So you can basically migrate process templates; you can do all sorts of things. It’s a pretty versatile toolset.
Unfortunately, with a versatile toolset comes a lot of complexity, hence there’s been a lot of asks from users of the system for maybe a little bit of an intro walkthrough to figure out how to move this through. So just a word on support for these tools: who supports them? Well, ultimately I do mostly, but there are a bunch of Microsoft MVPs that are part of the community. Some other TFS/VSTS consultants out there have been contributing, and then just people who are interesting, people who have that problem to solve. If you’re in a company and you’ve been told to go figure out how to move this data from one place to another, you might end up with my tool, and you’re going to figure out how to use it.
We are currently on, we’ve had 133 releases of our product, so we are on a continuous delivery pipeline. So basically that might be five releases in a week, or it might be five releases in a month, or it might be no releases in a month. It depends on what changes people are making, what pull requests they’re submitting. The source code is all in VSTS, so we build the product in VSTS. So source code, build, release, all of those things are there. However, we mirror that code to GitHub in an automated process, so we accept pull requests from both locations. If you’re a trusted contributor, we might give you the keys to the VSTS team project, and that’s a smaller number of contributors in there, but we’ve had 54 pull requests inside of VSTS. We also take general public non-trustee contributions, and if you make a number of submissions on GitHub and we like your code, maybe we’ll make you a contributor in VSTS. But at the moment, we have 12 people who have contributed reasonably regularly to the project, and you can find all of that information on GitHub apart from the VSTS specific stuff. But we publish all of our releases to GitHub and to Chocolatey, so I’ll show you how that works.
So I’m going to do a quick run-through. I want to show you how to find the tools, how to install the tools, just a quick overview of how to run the tools and how to get some kind of meaningful execution.
Okay, so first, how do we find the tools? Well, the first thing you’re going to do is you’re going to search for VSTS migration, and your first result is always going to be the Microsoft full fidelity migration. That’s the one you want to use if you can. But if I put tools on the end, you’ll see that VSTS migration tools pulls up the VSTS sync migration tool on the Visual Studio Marketplace. So you can go in there and you can get some basic information about the tool. There’s links to some documentation, some external review information, links into the GitHub repo, and what version we’re currently on.
So there’s some pretty useful things there, and to get started at the top there, it will actually take you through to the Git repo which actually has some of the same information. But if you want to get the tools, the recommended installation is from Chocolatey.
Okay, so there’s been over 3,000 downloads on Chocolatey. This is by far the most common way to install it, and we push out the bits automatically here, and so you can dynamically update your install pretty quickly when new releases go out.
So let me first… okay, so once you have a PowerShell window up, you want to go to the home page of chocolatey.org, and you can click the Install Now button, and you’ll see there are a number of different ways to install it. We’re going to install it with PowerShell, so we need to run this little command here.
So I’m just going to copy it, right-click, and run that command. Now what that’s going to do is it’s going to go and get the Chocolatey bits and install Chocolatey on your machine. Chocolatey is just a tool for installing packages, so it’s kind of like a package management system built around PowerShell. So it allows you to install any of the things that are on Chocolatey with a simple command line. This is just a simple tool that allows us to do that.
Okay, so I’m going to close that window. I’m going to go back to our sync migrator. There we go, we’ve got our sync migrator. So in order to install, never get Chocolatey installed, I’m going to open our PowerShell window, and this time I’m going to run choco install VSTS sync migrator.
Okay, so now that it’s installed on your C drive, you’ll have a tools folder, and inside the tools folder, you’ll have a VSTS sync migration folder that has all of the bits and bobs, mostly their TFS DLLs for allowing us to call into the TFS system. You can see there’s a single executable in there called VSTS sync migrator. That’s what we’re going to be calling to do all of our bits and bobs.
Okay, now that we have the tools installed, I’m going to open a command prompt, and we’re going to navigate to the VSTS migration folder. When we run the tool, you can see it pumps out a little bit of info. You’ve got the version of the product. Essentials is going to check and make sure that you have the latest version with our telemetry. Via telemetry is enabled, your session ID. So this session ID is super useful if you’re submitting a problem with the tool. If you give me the session ID, I’m able to look up some of the telemetry and find the exceptions and maybe figure out what the problem was.
And there you can see the full version number of the product as well, branch it came from, the hash, everything’s all there. And since we ran this with no very selected, so no init execute or anything, then it just output the default output. So what you can do is you can run init, and what that has done is it has created the default JSON file that you’ll see in the documentation.
So if I go back to that folder, there it is, the VSTS bulk editor JSON. I’m going to open the JSON file, and you’ll find it’s all in one line. Luckily, I’m using Visual Studio Code, so if you hold down shift and alt and press F, it will format it into a nicely formatted loop. There we go. So this is the config file that you’re going to feed the tool in the execute command that will tell it all of the things that it’s supposed to do.
Now this is the default JSON. You don’t want to run the default JSON. This has all of the options for all of the features that are in here, and you want to pick and choose which features make the most sense. So let’s go through a couple of the things. Some of the processors just need a source, and some of them need both the source and the target. So this is the where’s my stuff, the collection and team project name, collection and team project name. So obviously they should be different. If you just need the source, then it doesn’t matter, but if you need both, then that makes sense.
You’ve also got this reflected work item ID field name. This is something that we inherited from the old migration tools that the Rangers supported. One of the reasons we use it is it allows us to, if that field exists in your target team project in the process template, we will update that with the location. So the full URL project and ID of the work item from the old server. This allows us to parse off when we’re running through other processors and figure out where things came from, and then we can do additional work on that.
You can also have an update the source work item with where that work item went, but that slows down the system. It makes sense if you’ve got hundreds of thousands of work items and you want to skip over ones you’ve already done because even just the act of skipping over, because all we found it, it takes too long. So that’s the purpose of that.
And then we have a number of field maps. So there’s a number of different field mapping tools. There’s one instance of each of the field maps in here, but you can actually have multiple instances with multiple configurations. So basically you’ve been an object type. So this is the multi-value conditional map at config. So you have what work item types applies to. You can just put star and that place them all, and then the source field one and field two, and in the target fields, each of them have different capabilities. They’re all listed on the documentation.
Roughly it takes a little bit to get used to what options we’ve got. Each of them have been built to solve a particular problem, so it might, the one that you want, a problem you want to solve might not be one that we’ve solved before. These tools are built ad hoc to solve those particular problems.
So field, we’ll make a field empty, so you put in the field ref name that you want to blank, and it will empty that field of data as it goes through processing items, and we’ll talk about how that happens afterwards.
We’ve got the field value map, so you basically have a source field and a target field. So for example, if you’re moving from state in the old work item to state in the new work item, you might have a mapping from this looks like a mapping from scrum to agile. So approved converts to new, new approves, new committed moves to active, in progress gets mapped to active, those kind of things as you go through.
There’s a field to field mapping, so I just want to push the data from backlog priority into stack rank. Super straightforward, and you can do that with as many fields as you like. You can run these as many times as you like. A field to tag map, and so if you’re not going to have a field from the old work item in the new work item type, maybe you want to convert it to a state, and it supports regular, well, kind of regular expression, a format expression to map that detail.
So this is going to take whatever the state is, and if you place this mapper before the state field value map, it will take the old value, make it our tag, scrum state call on state name, and then you can easily find things that were in the old state, make sure everything’s mapped up. Maybe you have some queries that you don’t know how to do in a new system, and you can map it to the old one.
Okay, our field merit map. So this takes two fields, field one and field two, and maps them together. So for example, in the agile template, you don’t have acceptance criteria, but in the scrum template, you have description and acceptance criteria. So maybe we want to take the target field being description, and we’re going to map her in with description and then acceptance criteria. So zero and one, we’re going to map and enter the same field.
Sounds pretty cool. And then you’ve got the RegEx field maps, or we’re going to look at the source field. So there’s release, this particular reason I created this was the release was a bunch of numbers, so a version number of a product, but in the new template, they had a major and minor as separate fields. So this looks for a pattern and then replaces the value. You can run that as many times as you want to pull out different values.
So yes, you’re running the RegEx multiple times, but it’s a way to do that. Field value to tag map, I think that’s field to tag map, field value to tag map. So you can see stars field blocked if the pattern is yes. So if blocked is marked as yes, then put blocked in the resultant field, the target field.
Tree to tag map, so if you’re in your old team project, if you have a really horrible idea path hierarchy because it’s been a free-for-all in there, maybe you want to maintain some of that information. So this will actually convert each of the nodes in the tree to a tag, and it’ll say which one, how many to skip. So maybe you skipped the first three because you’re going to keep the first three in the next system, and then everything else becomes a tag.
So there are the instances of the types of field maps that we have in there, so you can configure them any way you like, you know, as many as you like. The order does matter; they’re executed in the order they’re listed in the file, so have at that.
We then have a little work item type definition. This allows you to decide what work items map to other work items. So here we’ve just got bug to bug and product backlog item to product backlog item, but maybe you have product backlog item to user story, and the system knows to create a new user story when you have a product backlog item in the source and user story in the target, and that allows you to do a live migration. I prefer in-place migration, so I actually quite rarely use that capability.
So there’s two types of processing. You can go into the data in place or as you’re moving from one system to another, and that brings us to the processors. There’s quite a number of processors. I’m not going to go through them all. I’m just going to go through the rough types. We’re going to talk about the individual processors in a little bit, but here you’ve got this is the work item migration config. It has a number of options. For example, if I want to prefix the project to the nodes, so let’s say I’m merging two team projects, team project one and team project two, and I’m putting them into team project three. If I prefix project to nodes, the new area paths will have the team project in the name. That kind of makes sense, so it will be project three slash project one slash whatever the old layout was.
Now there’s a little bit of our, these things have an order that they have to be run in, which is super important. If you just run it as is, this is going to fail. That’s kind of a good thing because if you don’t know what you’re doing, it’s probably going to make things worse. But prefix project, update creating date and update created by. If your source has not been modified since, and you like by yourself, then maybe you want to update the created date and created by, fill that into the new fields, and then update the source reflected ID. Do you want to back update the source system with the new reflected ID of the work items? So you can do that, and then you can also filter this query. By default, it filters quite a lot. You can modify this query to support whatever you want. At the moment, it’s filtering only work items that are open and only work items that have not got the reflected work item ID already filled out, and then they’re in these work item types: shared steps, shared parameters, test cases, requirements, tasks, user stories, bugs.
It’s super important that you do not migrate test plans and test suites. Do not migrate them. It’s not worth it. They’re completely worthless items. You cannot map them up again later. Do a full fidelity migration with Microsoft’s service if you want everything the same way. This has specific ramifications, and this migrates work items without history, with history. So a revision replay, it replays it to the revisions. It’s not really the same history; it’s just each work item with each revision, so the timing doesn’t always match up.
And then a work item update allows you to update work items in place, so it doesn’t use the target; it just uses the source and lets you update a bunch of work items in place. Okay, they’re the main processors. They’re the main things going on. I’m going to mention all the other ones and how they work. They each have their own quirks. Not all of them were built by the same people, but you take this config and we go back to our command line and we run execute, and execute with -config, and we put our C tools, the VSTS sync migration, the VSTS bulk editor JSON. We run that, and it will run the configuration that we’ve got. Do not run the default configuration first off; it will fail. That’s a good thing, and you need to go through it and figure out what it is that you’re doing.
Cool.
Okay, so we talked about how to find the tools, install the tools, running the tools, and a kind of overview of the execution and the execution flow in there.
Okay, so there are processors for work items. The order is important here. You have to run the node structures migration context first. That will create all of your areas and iteration paths, and that appending the team project name needs to be the same between node structures and the work item migration. If they’re not the same, it’s going to fail because when you do the work item migration, the area and iteration paths are not going to exist, and you can’t save a work item to area and iteration paths that don’t exist. It’s just the way VSTS and TFS work.
So run the node structures first; it will create them, then run at one of the migration contexts. So there’s work item migration, and there’s work item revision. There’s also the work item update, which only takes the source, but migration will move it without history. Revision replay will obviously replay the revisions as it goes through. The reason those two is somebody else added it, and they did it as a second context. We need to do some work to pull them together, so there’s only one, but it works pretty well. Both of them work pretty well; we just have to remember if we get a bug in one, we have to fix it in both.
At the moment, that’s the only bit that sucks there. After that, you can either run those two. They have to be run before you run the link migration context. Obviously, your work items have to exist to reapply the links in there. That would be super useful. So if you run a link migration context, it will go and add all of the links, the relationships to the work items in again.
And then there’s two ones for doing the attachments, the export and the import. You export all of the attachments to a folder; you then import them into the work items, and this works great. It takes some time to run depending on the number of attachments you’ve got, but it has some good filters in it to try to improve performance.
And then the last one, which I think Richard Fennell created, which is awesome, which is the work item query migration context, allows you to migrate your queries across, and it does some of the work to make sure that they work in the new location. I’ve not actually run that one myself; it was added after the last time I ran the migration, but yeah, super good.
Okay, so then once you’ve done all of the work items, which will bring the test cases across. Okay, so you have to do the work item migration to bring the test cases across. Leave out the test plans and suites. Then you can run the processors for the test cases. So the first one you run is the test variable migration, which creates the variables, then the configurations, which creates the same configurations that you have on both sites. They have to be in place first before you create the plans and suites.
Now you’re going to create new plans and suites because plans and suites are made up of two parts inside of VSTS and TFS. They have the work item, and then they have a bunch of other data that is inside of TFS. The other data is the bit that test management looks at. The work items are just some metadata. You can’t push them back together afterwards, which is why we don’t migrate plans and suites the first time around. We create new plans and suites, but we might wire it up to the existing test cases, so it will run through and do that.
There are some caveats, like queries don’t migrate well and some other things, but it’s its best case. And then we’ve got our very beta. I don’t think it works just now for test runs. We’re trying to figure out how to get test runs across as well, but that will allow you to migrate your test cases.
We then have some processors for other things. Some of these are pretty cool. We have export profile picture from Active Directory. So you can, if your corporate Active Directory has the pictures of all your employees in there, it will do an export of all of the pictures to a username, a folder with a username, with the picture name. The picture name is a username, and then you can import them all into TFS. This does not work with VSTS. VSTS is authenticated differently. Only the user can update their own profile picture, but it works well with TFS.
Done that before, you may have messed up your migration. Therefore, you want some way to delete all of the work items. The work item delete takes a query. Warning: this is super dangerous and will permanently delete all of the work items that end up in scope for it. Make sure you use that in a very clever manner.
We’ve got the fix git commit links. If you’re also migrating git, as long as you keep the same name of the repo from the old git repo to the new git repo, so in the new team project location, you can run this processor, and it will go and remap all of the links so that the links in the work items point to the new git repo, not the old git repo in the old location.
And so that will go run through and fix all of that. You’ve got to create team folders. That’s far. If you’re, let’s say you have many teams inside of your team project, and you want to create a folder for each team so that you can put queries and queries or query folders, it will go and build out that list for you so you don’t have to do it manually. If you’ve got 50 teams, it can be quite time-consuming to go build that. Run the processor; it will go do that.
Export team list literally just exports a list of all the teams for you. These are all the processors and tools. You can go get the source code on GitHub. You can look at how they work. You can build your own. You can submit pull requests if it doesn’t work the way you want. We might mention the field map again.
There’s three processors that these work with: work item migration, work item revision migration, and the work item update. So those three will use these. The first two, migration and revision, both move from source to target. Update only operates on source, so it’s an in-place bulk update.
So again, you get the field map, field marriage map, field blank map, field to tag, field value map, field value to tag, RegEx field map, and tree to tag map. Those are a bunch in there that are super useful. And again, the JSON file is more accurate than the documentation because the JSON file is the actual options, but it pulled out of each of the configuration of each of the code.
So while the documentation gets out of date, the JSON file that you get from the init command is always up to date. Well, there we go. We’ve gone through all of the options in the VSTS sync migration tools. Hopefully, that will give you some ability to get started with them and start migrating some of your work items.
So if you have any problems, head us up on GitHub, submit your issues. We have a very vibrant community, lots of people asking questions and getting answers, but it is ad-hoc support. It’s just us giving up our free time and had to support it, but you can ask.
