Mastering Azure DevOps Migration Tools: Your Ultimate Guide to Seamless Migrations

my name is Martin Henshaw wit and I created the azure DevOps migration tools you can find them on the visual studio marketplace if you search under Azure DevOps you will find the tools we’re actually up to about 50,000 migrations just over 50,000 migrations migration sessions per month this in the last 30 days we’ve migrated 340 thousand work items with 2.6 million revisions using the tool so that’s that’s pretty pretty huge I do want to stress the warning that you can see here on the page the tool is not designed for novices it is developed by and far as your DevOps consultants who have helped lots of folks do migrations around the world and you’re welcome to have a have a go this video is going to try and explain how to get started with the tool we’ve updated it a lot in the last year in fact two years since the last video we’ve merged a lot of the different tools together as well as enabled syncing for for work item migrations so if you do a migration and then somebody happens to have changed a bunch of work items in the old system between you starting the migration and finishing you can have it go and pull those additional changes across as well so it’s kind of pseudo syncing it’s not syncing between two systems but syncing from one system to the other system I get a lot of questions around what versions of azure DevOps does it support and everything that the object model API does so that’s TFS 2013 plus it does mostly work against older versions of TF but the official support is TFS 2013 and onwards all the way up as your DevOps and yes you can go from Azure DevOps to as your DevOps as your DevOps to TFS TFS to your DevOps pretty much whatever you want and so let me pop over there’s a documentation available on github and it has a lot more detail on how the Tool Works so if you click through to the documentation and you will go to this page and I know and here you’ll see detailed documentation about what the processors that are available are and the processors are the thing that does the work so the node structures migration processor will move your areas and iterations across the work item migration context will migrate your work items links revisions attachments and hopefully if we can get it organized it’s not working right now the HTML field embedded images as well and it will also fix the git commits as a number of other and tools that are available at work item delete this is deliberately crippled so that you don’t accidentally delete a whole bunch of stuff you will meet the code to be able to run this one but you can delete any number of work items and an update ur so instead of migration from one system to another if you just want to run a bunch of changes against an existing project then you can do that so for example you might have an added a new field and you want to migrate data from one field to another field or you might want to split data from one field into two fields that’s so that work item update will allow you to run against an existing project work item query migrator well obviously migrate your queries across so it will take them just need to make sure that it will skip any queries where the fields don’t match up so you need to be aware of that the fields in the new location need to match and the fields in the old location otherwise it with error right there is also a a migrator for the teams I think it’s actually not on this list yet but it will migrate the create teams and migrate all of the settings for those teams across or as much as possible at the moment there are a bunch of tools around migrating at test cases and so sweets and plans you have to migrate the variables first then the configurations and then you run our plans and sweets at migration which will pull that across so you do not migrate test cases using this tool you have to use the work item migration context to migrate the test cases but do not migrate the test plans and test Suites work items across at the same time otherwise you’ll end up with a bunch of duplicate items that’s about it for the general processors we have a number of field Maps I did mention that if you’re either working against a single team project your migration for one to another you may want to use some field maps and this is good if your source and target projects have different processes you may want to use this or if you’re updating changing the process and an existing project and you want to map data from one field to another and so there’s just a field to field map and field to field multi map so it’s a list if you’ve got a field or a list in it it’ll map data from one to another field marriage and put two fields together in a particular format blanket so you don’t want to migrate a particular field that field to tag that’s you can convert any number of fields from your source system to tags in your target system that’s good for a bunch of fields that are not available in a target system you don’t want to add them to the target system and for example if you’re simplifying a process you can run the field tag to save some of that data while not adding to the Lord of a number of fields field value map so that’s just a value mapper and then we’ve filled value to tag so if there’s a set of values you can have them two tags a reg X field mapper so that’s pretty useful for example I got this for a customer who had their version format in 2016 - in a text field and they wanted to set it in a drop-down list one with 16 in the other one with the two and so the reg X field map will let you do that and then for many organizations that have been in TFS for a long time they have a problem where all of their area paths their area tree hierarchy is reflective of their product or organizational hierarchy rather than the team’s hierarchy which is how the new system prefers things to work you don’t have to do that would prefer things to work so if you want to map all of that tree into tags so we’ll take each node and build a tag and attach it to the work items and you can do that so it’ll map area paths to tags so if you had an idea path that was seven deep you would end up with seven tags one for each of the things just allows your way to categorize that and save that work I note that we don’t migrate any code using this tool there are no good options for TFB C team foundation version control the original TFS version control system and there is the TFS integration tools which is not supported there’s a number of other tools out there as well and I would recommend doing the using git TFS a tool to migrate your TF BC code into M git repos that’s what I would recommend get out of TF BC moving to get get is really easy to migrate you just clone the repo from the source location and push it to the target location that it just works out of the box and we do I did mention earlier that those are while we are processing work items if the git repo has been moved across first then the tool will it fix up the the git commit links on the fly so in as your DevOps and TFS you can link from I get commit to a work item by tagging it so hash 1:21 will link work quite work item 1:21 and and you can link to pull requests so your pull request links will disappear because we can’t migrate pull requests but your git commit links will be fixed on the work item site so it will not be fixed in the comment of the commit because it’s it’s a totally different exercise to go to go do that but when the work item is created and the links are added and it will go and see if it can find the git repo in the new location if it can find the get repo in the new location it will fix your link so that it points to the right commit and I note that there is a way in the tool to map and your get repos if you’ve renamed them and if they’re the same name then that’s super easy you don’t need to do any any mapping and it will just find it and you can run this and after using the fixed commit a fixed git commit obsolete processor so you can run it on its own and but it’s better to run it as part of the whole thing builds and releases don’t get migrated automatically and you will need to export and import each of your builds each of your releases and and then all of the secrets that are part of that will need to be read and that’s just an exercise in going through them all and doing that there’s not a good way to do that I if you are open to working on that you could very easily build a processor that could do that we just haven’t had the occasion to do that yet I know aren’t contributing we are happy to take any contributions that you have as long as they’re relevant for everybody rather than just for your organization and there is no official support for the azure DevOps migration tools apart from community support because this is an open source project and I am happy to help folks out if they’re having trouble and if you have significant trouble and then that may be something that’s more related to a consulting engagement rather than just at me pointing out where where where the configuration needs to be customized and ultimately if it’s gonna take me an hour to solve the problem then I’m good maybe help you right and if it’s going to take three days to solve the problem then we might like to have a conversation and there are other contributors to the project who are also happy to help there are many conf users we’ve had over 40 contributors go look up any contributor you want have them help you out we’re going to add some more recommendations here so that you’ve got somewhere to start and I’m definitely got two more folks that are going to end up on that list very very quickly so let me jump into how you use and interact for the tool the first thing I recommend and I do a lot of migrations between Azure DevOps and Azure DevOps and that’s mainly for customers that have sold part of their organization or and they want to merge many team projects into one project or they want to split a project into more than one project that’s that works as well so the easiest way to do that is to run the tool on a machine that has access to both environments you want to migrate to so I don’t generally recommend running it on your local machine mainly because your local machine gets turned off at the end of the day and you want to go home and migrations might take a lot longer than that so I recommend and especially if all of your target and source environments are in Azure that you create in Azure and a dev test lab and the reason I suggest the dev test lab is because it can automatically turn the machines off and on as needed so I’m just gonna create a new dev test lab so create that so I’m going to leave auto shutdown enabled you want to use the location and of your both hopefully both your source on the target are in the same location and if not I recommend creating this in the same data center as your target environment because we’re going to be doing a lot more communicating with the target environment than we are with the source environment because we just load all the work items and then run against the target and so in order to see where that is if you open up your organization go to organization settings and you will see on the overview page the region and this one is in West Europe so I’m going to create that in West Europe which is my default because that’s where I am right now I’m gonna create that okay okay once your azure DevOps that lab has been created you need to create a machine inside that a virtual machine I’m going to add a virtual machine and I’ve had a lot of trouble with the Windows 10 versions of if so if I type in I’ve had a lot of problems with the the on Windows 10 at once they basically had not been starting I don’t know if that’s fixed so I’m not gonna do that just now so I’m just gonna pick Visual Studio to a nineteen enterprise on Windows Server 2019 so I migration I’m just gonna put in a password well I can do that have to look for that later so let me copy that into notepad so I don’t have to show you all of my passwords there we go you are quite welcome to steal back because I woke up deleted this machine at the end of the video I’m not gonna use the standard e3 because it is tiny I’m gonna map that too and I want at least four processors there’s four processors 16 Giga RAM so a d4 sp3 I’m generally I have the auto off set true so that at the end of every every day like maybe 7 o’clock every day the machine turns off and shuts down when I do the actual migration I’ll turn that off and then I’ll go turn it back on again and after the migration is completed so I’m gonna use premium SSDs as well so this would be an expensive machine if it was running for a long period of time and I’m not gonna setup any of this because sometimes it doesn’t work so well and I’m going to create that machine okay so once your machine has been created and it’s running we just need to log into it so let me do that there’s my password that we set up yes okay so now that we are into the machine and I will still give it a few seconds before I try and start clicking stuff for it to get organized no we do not want it to be discoverable thank you very much it will start a bunch of default stuff that I’m not really that interested in let me get rid of that there we go okay so I have visual studio I have all the bits I need installed unfortunately I have to use IE because it is server so the first thing I want to do is I need to install the tool from chocolaty whatever’s so first thing I’m going to do is install chocolaty this is very easy to do copy the script open PowerShell in admin mode oh well that’s paste in my password there we go yes although access now I’ve got it and that will install chocolaty on my machine chocolate is a package manager so it’s basically going to handle the install and uninstall of the tools so that’s it all set up so if I validate that it’s working there we go so working okay so now if I go back to the first page so in order to install our tools there’s your DevOps migration tools we just need to type yes yes always all artists into the clan line and that will go and download the bits needed to install it including any dependencies which there shouldn’t actually be any dependencies which is pretty good because we’re running on a machine with fields to do installed there we go it’s all been installed so now you will find in your C Drive there’s a tools folder in that tools folder there’s a VST s sync migration folder so if I go in there you’ll see there’s loads of files but the main one is the migration dot exe so we’re gonna call migration DXE in order to set this all up so let me kill ye because we don’t need to IE anymore let me show you what’s going on and then we’ll move some stuff to a folder that makes it easier for us to see so tools migration and does a lot of a lot of stuff in there the first thing we need to do is create a template at migration file so if you run there we go oh let’s go one back migration exe and we type in it it will create a simple work item tracking based file so if I go up here I may need to refresh configuration here so let me let’s open a new folder and pop that just onto the temporary storage there we go you there we go so we have our JSON file there now this file is just work items so if I just do top work films here we go and we can take a look at the conflict file and see what is what so this is the first time visual studio has been started on this machine so it may take a minute or so I guess while we’re waiting I can open a notepad and we can take a look while it’s loading so what I’ve done is I’ve created our default file I’m gonna go through some of the options in the files so that you can um see what is in here now maybe later you like the dark theme okay that’s the first thing is it has a version this matches the version of the to at least the first two digits of the version and its purpose is to make sure we don’t accidentally run a car an old con fake against the new version of the tool and see if there’s a new version of the tool and it changes a major version check for any different season maybe update this the two will not run if you’re using an older version of the config file but it might just be a case of changing not three four four and then everything’s good ah telemetry trace is by default set to false really don’t don’t turn this on this can be used for some debug scenarios and it will push all of the telemetry out to application insights so you don’t want to do that by default and the the base set of metrics are sent to a pin sites so we’ll say how many work items have been processed how long they’re taking exceptions those kind of things are passed to up insights if you enable the trace then every thing that you see going past on the command line screen will also be sent to a pin site so you don’t want to do that unless you you’re working with one of the consultants who has access to app insights and you want to UM have a have a discussion around some of the telemetry or try something push some things out ah this was a workaround for the azure DevOps team breaking the api’s should by default we set to false we just haven’t removed it yet and then the two really important things are the source and the target so source and target both are the same format and you’ve got the collection so what is the URL of the collection that you’re connecting to so that could be the TFS connection URL or the azure DevOps connection URL or the name of the project inside of that it goes under name and the reflected work item ID field name is the name of the field that you’ve added to that project into the process where we’re going to store the reflective work item ID now if it’s TFS you get to set the whole thing if it says your dev ops when you create a new field you only get to set this bit on the end and it used to be if you’ve added this to an older process it will be processed name dot and but if you go add one just now to Azure DevOps it will be custom don’t um I don’t know why they changed it or how that’s meant to be documented but it just is you can use there is a tool to export processes that you can use even the the inherited processes you can export that and then go look at what the fields been called but generally it’s one of those two things it’s either whatever you’ve said it as if you’re in TFS if you have an XML process in your DevOps then it will be the same and if you have an inherited process and it’s an old process then it might be called my process name dot reflected work item ID if not it’s if it’s newer it’s just custom oh so you may just need there’s only three possibilities so you could just do trial and error and the tool will check to make sure that what you’ve typed in there is valid so that’s pretty good the field maps is empty you don’t need to add any field maps by default the tool will map same field name to same field name or same field ref to same field ref if you want to change any of the data on the fly on the way across then you would need to setup field maps but this is just the easy setup and work item type definition now does not need to contain anything it can be empty but if you have for example in the source if you’re using user story and in the target you’re using for a backlog item then you would fill those differences out in here otherwise it will just process and all the existing ones and I mentioned before the the git repo mapping so if you need to map get people names from old name to new name a new location then that would be in here as well again in this case it is null so then we come to the processors there are two processors oh let’s go to the color coded version now we get to the processors and there are two processors set up here for us to see the first one is the node structure migration config this will migrate your areas and iteration paths across you can set base paths if you want to filter that but please note that if you try and save a work item when you’re doing the work item migration to an idiopathic not exist there will be an error generated and the thing will bomb out the area paths need to exist so what I normally do is just set this to empty you don’t need to have any base paths at all and it will just migrate all of your work items if there are no bats all of your area and iteration passes there are no base paths present there is one feature on here and actually needs to be consistent across all of the processors that you run if you’re going to use it and that’s prefix project two nodes which means that it is going to if you had a team project one and you were migrating it into team project to and you wanted team project two to have an area path of team project one with all of the team project one area paths underneath it then it would do that this is a feature for if you’re merging team projects so if I have team project one two three and four that are all merging into team project five team project five will have root area paths of team project one two three and four under that of the work items all the data and so that you’re not just merging it together into a big mess and you can take some action afterwards to clean it up and figure out what are those dupes and all that kind of thing generally you want their set to OFF unless you’re doing a team project marriage and even then you need to make that choice all right so this one needs nord structures needs to be run first in order to have that work you’ve then got a work item migration config the work item migration config that sets up the migration for the work items processors are set to false by default so nothing will happen if you run it I generally run them one at a time to make sure everything’s good so you can set that enabled to true and and then there’s a number of things that you might want to have so first off you’ve got the prefix project two nodes again set to false it must be consistent across the project you’ve got so got the update created date and they created by in most cases this should be set to true which is why that’s the default and if you’re having a problem because you are a particular version of TFS that does not allow you to edit the created date or created by you can turn these to false update source reflected ID it should be set to false it’s set to true or a little fix that the default should be false this will actually also go and edit the source work item with the reflected work item ID of the new location we’re doing away with this feature because there’s no point in editing the the source if there’s another way to do it and I figured out another way to do it if you’re replaying revisions you do not need build field table so if you’re just doing a tip migration then you want to set this to true and it will build a field of all build a table of all the existing fields in the source work item and save it into the comments of the new york item so that you can at least go and search for any old data that might not have been migrated across into fields and then there’s a signature which you can set to or false we’ll probably get rid of this but it’s set to false just now so that brings us to the replay revisions replay revisions will either replay each of the changes made to a work item if it’s set to true or it will just take the latest tip of the work item if set to false if you’re doing a sync ie there’s been changes made since the last revision change and it actually filters the list based on and the exact date time of the save so you’ll have maybe ten revisions and the first thing as it does is it looks at the target work item and takes out all of the identical updated dates and times out of the the list and then picks the top one if we play revisions are set to false and we plays them if it’s set to true so that’s just that’s how that part works and the way it knows which work items the Lord is you have a query bit and generally you can’t have more than 20,000 work items as the result of a query so you may need to stage this if you have more than 20,000 work items and you never want to migrate test Suites and plants there may be other work item types that you don’t want to migrate so you can add them to that not enlist and what I generally do is I focus the initial migration on the open work items so the ones where closed date is not set to anything and then I will rerun it again with the opposite being true with cause date populated to fill out all the the closed items that way people can move over and start working in the new system and everything’s hunky-dory and there’s the order bit from the query I usually do change date and so that it goes from latest change date to oldest so the most recently touched work items are updated first and then it gets into the past where it’s less likely to cause a problem and then while it’s migrating each work item there are a number of features that one is the link migrator but by default should be true and this will attempt to add all of the existing links so it’s going to add all of the different link types apart from pull request links because we can’t migrate pull requests and this will migrate all of the attachments you can you will need to pass it a temporary folder for the working path where it’s going to download and upload the attachments the default should suffice on 99% of systems it will clean up itself afterwards there may be some things left over when it can’t delete folders because files are still in use but it will generally clean up after itself and then the fix HTML attachment that links is for in HTML fields you can paste an image that image has an absolute path to the file and that will not work in the new system when the old system was offline unfortunately that’s not working just now we have some authentication issues with that but what it’s supposed to do is pool basically download each of the attachments add it as an attachment to the work items you’ll have extra attachments and then insert the the replace they let fix the links for you again it doesn’t doesn’t work just now we’ve got authentication issues with that communication we’ll get it fixed at some point there’s a number of people who are who are looking at that you occasionally get errors just communicating towards your DevOps and if you’re migrating 30,000 work items with a hundred revisions each and a bunch of attachments and links you’re going to be hitting the usurer DevOps API is pretty heavily you’re probably going to get slowed down and occasionally that system will glitch it’s on the Internet it could be for any reason between your environments and so there’s a reach automatic retry and it will wait if it gets our 503 which is a server unavailable exception it will wait for one second then retry then wait for two seconds and retry all the way up to 5 seconds and then retry and if it fills out the fifth attempt it the tool will stop cool so that’s what it is i don’t recommend making that number that much higher but if you said 215 the last one will be waiting 15 seconds and then we trying as well this is a really important one filter work items that already exist in the target there is a mutually exclusive nature here that I need to mention so what it will do is it using the same query bit okay it’s going to load the source work items load the target work items and remove any target work items from the source list that already exists and so if you’re trying to sync things that have been changed that are newer that will not work because it will filter them all out because they already exist by default you want this set to true because if you’re migrating 30,000 work items and it got cheese after 5,000 you don’t want to skip or have to Lord and skip over each of those 5,000 work items so that will filter those out and so I recommend getting a full run with this enabled and then using a date based query so see if your close date and I would do a full migration and then do cheat me spell it right change date and have that set to greater than date of migration so let’s say it was 2019 1016 because we start the migration yesterday it’s finished and now I want to rerun anything that was changed after we started the migration and you would need to start with the date of the migration because you can’t put tines in there and that that will work super awesome and then it will just go and pull the 300 work items that were changed yesterday and today and Rioch date them and but you would need to change this to false so that you can migrate and it will skip anything that doesn’t need any changes but it will need to do a bunch of stuff around that which is why you want this generally set true okay that was a brief overview of the azure DevOps migration tools let me know if you have any issues on github create an issue on github and somebody will help you