Automatically migrating your builds to Azure DevOps

calendar May 2, 2021 · 10 min read · Azure DevOps Automation DevRel DevOps Build Management ALM Enterprise  ·
Share on: linkedin copy

Overview

:right

A few weeks ago, I wrote how we migrated our build system towards TeamCity (this migration was a few years ago). If you want to know more on how we did this, then I gladly refer you to this post

Today, I'm writing about the migration back to Azure DevOps. I'll explain the reasoning behind why we move back and what we use to leverage this migration.

why we move back to Azure DevOps

Well, it's really easy: Azure DevOps offers an integrated environment and it places your build pipelines at your fingertips, next to your code repositories (Compared to a separate build orchestrator (eg TeamCity) where you have to login into and manage your builds in a separate platform.) This integration has quite some benefits for developers, but also for the other members of the development teams can be constantly (and thus faster) up to date on the status of builds in their context.

In my post on "moving to TeamCity" , I also told that the migration to TeamCity was because of a lack of templating support in Azure DevOps. This has now been solved with the Azure (yaml) pipelines and (yaml) templating support. This combined with the adoption of cake in our environment , we were able to switch back without a lot of effort.

Finally, changes the structure of the network made it harder to work in an efficient way with TeamCity and if there is one thing that development teams need, then it is efficiency. In that context, the choice became even easier!

How we move back

In the context that we are migrating, we are migrating build pipelines of multiple teams (each team manages multiple products, each with multiple pipelines. Asking the teams to do this work all by themselves might work, but there are some drawbacks:

  1. Each team has to invest time (and effort) in the migration (this includes the learning path, figuring out what to do, etc.)
  2. each team might approach a manual migration (a little) different and this would result in a situation where each build pipeline is set up differently. For management and support reasons. This would really be a step backwards.

What do we have?

The setup in TeamCity was as follows: We provided the teams with a build config template that facilitated the teams in setting up a build config. In that build config , all they had to specify was:

  1. Azure DevOps TeamProject
  2. Azure DevOps Git repo
  3. The name of the cake file (remember: we work with cake build in our environment
  4. A (limited) set of parameters that can be specified on build config level, aimed to override specific cake arguments
    1. Build Config (Release/Debug)
    2. Company specific context info
    3. ...
  5. Cake Target (allows you to execute specific logic in the cake file) (more info on cake targets )
  6. Cake (logging) verbosity

With this, we actually have everything we need in a (rather) structured way to be able to automate a migration from one build system to another. On top of this, knowing both systems well and what they need, helps in providing the 2 systems with all the info to make of this migration a real success!

what do we need

I'm not going to write on how the info can be extracted from TeamCity. I already wrote about that integration last month and I'm sure that you'll be able to figure it out if needed 😉

I'm writing this automation in .net framework (C#), but I'm sure that you should be able to reproduce it in other languages too.

To get started, you'll need to add a few Nuget packages:

1Microsoft.VisualStudio.Services.InteractiveClient
2Microsoft.TeamFoundationServer.Client

And these "usings" will be needed in your code:

1using Microsoft.TeamFoundation.Build.WebApi;
2using Microsoft.TeamFoundation.Core.WebApi;
3using Microsoft.TeamFoundation.SourceControl.WebApi;
4using Microsoft.VisualStudio.Services.Client;
5using Microsoft.VisualStudio.Services.Common;
6using Microsoft.VisualStudio.Services.WebApi;

Then you need to set up a connection and create the necessary "clients" that will be used in the interaction with the Azure DevOps API:

 1//1. Setting up the connection
 2var pat = Environment.GetEnvironmentVariable("MY_PAT"); // I created myself a personal access token to facilitate the integration
 3if (string.IsNullOrEmpty(pat))
 4            {
 5                Console.WriteLine("You need to specify an env variable 'MY_PAT' wit an azure DEVOPS PAT as value. I don't find one, so there is nothing I can do...");
 6                return;
 7            }
 8const String c_collectionUri = "https://dev.azure.com/<your account>";
 9//set up the credentials for the connection
10 VssClientCredentials clientCredentials = new VssClientCredentials(new VssBasicCredential("tim the migrator", pat));
11
12// Connect to VSTS
13VssConnection connection = new VssConnection(new Uri(c_collectionUri), clientCredentials);
14
15//2. Creating the needed clients
16
17var projectclient = connection.GetClient<ProjectHttpClient>();
18var gitclient = connection.GetClient<GitHttpClient>();
19var buildclient = connection.GetClient<BuildHttpClient>();

At this point, you have everything that you need to get started with your migration... Except for one thing, which is the yaml file. Because we work with cake, the azure-pipelines.yml file that is typically created when setting up a build pipeline, would always have the same look and feel. This is because the yaml file is basically a mechanism to trigger the cake build: all it needs to do is collect the needed info to be able to start the cake build. If you know me a bit, then you will know that I always try to find a way of working that makes it easy for the end user, but also for me (in the context of management). Because of that, I came up with an approach that does just that. It would bring us too far, but I'll write about it soon. All we need to remember is that I found a way to work with an azure-pipelines.yml that can be the same in all git repo's that we are going to define build pipelines for. The element that will allow a pipeline to be different, will be stored in the pipeline variables! (more about this soon!)

Based on the info above, we need some "basic info" to be able to create a Azure pipeline via the API:

1var buildname = "The buildname of the exported build"
2var azdoproject = "DemoProject";
3var repo = "DemoRepo"; 
4var cakefile = "build.cake"; //Default
5var caketarget = "Default"; //Default
6var cakeverbosity = "Diagnostic"; 
7var buildconfig = "Release";
8var pipelinesSuffix = "-pipelines.yml";

Then we need to retrieve the Azure DevOps Project and the git Repo

 1var project = projectclient.GetProject(azdoproject).Result;
 2if (project==null || project.Id == Guid.Empty)
 3    {
 4        Console.ForegroundColor = ConsoleColor.Red;
 5        Console.WriteLine($"The project {azdoproject} does not exist in azure devops!! Cannot migrate build {buildname}!!!");
 6        Console.ResetColor();
 7        return;
 8    }
 9var repos = gitclient.GetRepositoriesAsync(project.Id, true).Result;
10var repo = repos.FirstOrDefault(x => x.Name.Equals(repo, StringComparison.InvariantCultureIgnoreCase));
11if (repo == null)
12    {
13        Console.ForegroundColor = ConsoleColor.Red;
14        Console.WriteLine($"This repo ({repo}) in project {azdoproject} does not seem to exist! not migrating this build!!");
15        Console.ResetColor();
16        return;
17    }

And to go further, we need to better understand what is going on in the background when you manually create a new build pipeline in Azure DevOps. On repo level, you have a blue button which says "Create new pipeline". When clicking the button, a set of steps is set in motion and we need to "replicate" these if we want to automate our pipeline creation process. In essence, 2 things happen:

  1. A yaml file with name azure-pipelines.yml is created in your GIT repo (next to your code). This file will describe the build process and is thus important
  2. The yaml file is being linked to a new pipeline (which also contains metadata, such as variables) and together they "are" the pipeline.

This means that we need to replicate this:

Creation of the azure-pipelines.yml file

First, we're going to create the yml file in the git repo. We're going to use the Azure Repo's git API for this:

 1var pipelinefile = "azure-pipelines.yml";
 2var builddefcanbecreated = true;
 3Console.WriteLine($"{repo.Name} - {repo.DefaultBranch}");
 4var branch = gitclient.GetBranchAsync(repo.Id, WithoutRefsPrefix(repo.DefaultBranch)).Result;
 5var commit = gitclient.GetCommitAsync(branch.Commit.CommitId, repo.Id).Result;
 6var tree = gitclient.GetTreeAsync(repo.Id, commit.TreeId).Result;
 7if (
 8    tree.TreeEntries.Any(relpath => relpath.RelativePath.EndsWith(cakefilesuffix))
 9    && !tree.TreeEntries.Any(relpath => relpath.RelativePath.EndsWith(pipelinesSuffix)))
10{
11    Console.WriteLine($"This Repo has cake support and has no files ending with {pipelinesSuffix}: ok to migrate the builds for repo ({repo.Name})");    
12    //1: define yaml file that needs to be created (with variables to support build-time flexibility)    
13    if (!cakefile.Equals("build.cake", StringComparison.InvariantCultureIgnoreCase) && !string.IsNullOrEmpty(cakefile))
14    {
15        var specificCakeDistinguisher = cakefile.ToLower().Replace("build", "").Replace("cake", "").Replace(".", "");
16        pipelinefile = $"{pipelinefile}";
17    }    
18    //2. create change & commit procedure to support adding yml file(s) to repo
19    GitRef gitref = gitclient.GetRefsAsync(repo.Id).Result.First(x => x.Name.Equals(repo.DefaultBranch, StringComparison.InvariantCultureIgnoreCase));
20    GitRefUpdate gitrefupdate = new GitRefUpdate()
21    {
22        Name = $"{repo.DefaultBranch}",
23        OldObjectId = gitref.ObjectId,
24    };    GitCommitRef newcommit = new GitCommitRef()
25    {
26        Comment = $"Add yml file for building {repo.Name} with azure devops",
27        Changes = new GitChange[]
28        {
29                new GitChange()
30                {
31                    ChangeType = VersionControlChangeType.Add,
32                    Item = new GitItem() { Path = $"{pipelinefile}" },
33                    NewContent = new ItemContent()
34                    {
35                        Content = System.IO.File.ReadAllText(@"pipeline.txt")
36                        ContentType = ItemContentType.RawText,                    
37                    },
38                }
39        }
40    };    
41    // create the push with the new branch and commit
42    GitPush push = gitclient.CreatePushAsync(new GitPush()
43    {
44        RefUpdates = new GitRefUpdate[] { gitrefupdate },
45        Commits = new GitCommitRef[] { newcommit },
46    }, 
47    repo.Id).Result;    
48    Console.WriteLine("project {0}, repo {1}", project.Name, repo.Name);
49    Console.WriteLine("push {0} updated {1} to {2}",
50        push.PushId, push.RefUpdates.First().Name, push.Commits.First().CommitId);}
51else
52{    //build cannot be done (migrated) when either:
53    //1. there is no cake file (rather critical for this to work)
54    //2. there is already an azure pipeline build yml file in the root of the repo
55    Console.WriteLine($"The builds of repo {repo.Name} cannot be migrated because:");
56    if (!tree.TreeEntries.Any(relpath => relpath.RelativePath.EndsWith(cakefilesuffix)))
57    {
58        Console.WriteLine($"\tThere exists no cake file in the repo {repo.Name}");
59        builddefcanbecreated = false;
60    }
61    if (tree.TreeEntries.Any(relpath => relpath.RelativePath.EndsWith(pipelinesSuffix)))
62    {
63        Console.WriteLine($"\tThere already exists a yml file ending on {pipelinesSuffix} in the repo {repo.Name}");
64    }
65}

Setting up the build pipeline (and linking everything together)

Next up is the creation of the build pipeline, combined with the linking of the created azure-pipelines.yml file:

 1
 2
 3if (builddefcanbecreated)
 4{
 5    //3. generate build based on yml file
 6    var buildsForProject = buildclient.GetDefinitionsAsync(project.Id).Result;
 7    
 8    if (buildsForProject.Any(x => x.Name.Equals(buildname)))
 9    {
10        Console.WriteLine($"This build already exists, doing nothing!");
11        var builddef = buildsForProject.FirstOrDefault(x => x.Name.Equals(buildname));
12        builddef.Queue = new AgentPoolQueue()
13        {
14            Name = "YourAgentPool",//does nothing when set in the yaml file or with multi stage pipelines!
15        };
16    }
17    else
18    {
19        Console.WriteLine($"creating new build def!");
20        var repoToLink = new BuildRepository()
21        {
22            Id = repo.Id.ToString(),
23            DefaultBranch = WithoutRefsPrefix(repo.DefaultBranch),
24            Type = RepositoryTypes.TfsGit, // important: can also be git (and then service connection is required)
25            CheckoutSubmodules = true,
26            Clean = "true"
27        };
28        repoToLink.Properties.Add("reportBuildStatus", "true"); // shows build status on repolevel, which is nice
29        var builddefinition = new BuildDefinition()
30        {
31            Name = buildname,
32            Project = new TeamProjectReference()
33            {
34                Id = project.Id
35            },
36            BadgeEnabled = true,
37            Type = DefinitionType.Build,
38            Repository = repoToLink,
39            Process = new YamlProcess()
40            {
41                YamlFilename = pipelinefile,
42            },
43            Queue = new AgentPoolQueue()
44            {
45                Name = "YourAgentPool",
46            },
47            QueueStatus = DefinitionQueueStatus.Enabled,
48        };
49
50        //What follows is optional: it allows you to set the build variables that I talked about earlier
51
52        if (!string.IsNullOrEmpty(caketarget))
53        {
54            builddefinition.Variables.Add("caketarget", new BuildDefinitionVariable() { AllowOverride = true, IsSecret = false, Value = caketarget });
55        }
56        if (!string.IsNullOrEmpty(cakeverbosity))
57        {
58            builddefinition.Variables.Add("cakeverbosity", new BuildDefinitionVariable() { AllowOverride = true, IsSecret = false, Value = cakeverbosity });
59        }
60        
61        if (!string.IsNullOrEmpty(buildconfig))
62        {
63            builddefinition.Variables.Add("buildconfig", new BuildDefinitionVariable() { AllowOverride = true, IsSecret = false, Value = buildconfig });
64        }
65        if (!string.IsNullOrEmpty(cakefile))
66        {
67            builddefinition.Variables.Add("cakefile", new BuildDefinitionVariable() { AllowOverride = true, IsSecret = false, Value = cakefile });
68        }
69        //finally (!) send everything to Azure DevOps to create the build pipeline
70        var resultingbuilddefinition = buildclient.CreateDefinitionAsync(builddefinition, project.Id).Result;
71        if (resultingbuilddefinition != null)
72        {
73            Console.WriteLine($"successfully created build definition for {resultingbuilddefinition.Name}");
74        }
75    }
76}

Conclusion

And that's it! Your pipeline has been created! You can do a lot more (such as setting up CI on specific branches), but that is something that I leave for you to discover (where is the fun in the job otherwise 😉). All you need to do now is to put this in a big for loop, define your azure-pipelines.yml file and get started!

I know that it is a lot of code (and that it might have been better put somewhere else), but in the spirit of a good story, I decided otherwise! Also, the code is pretty self explaining and wherever needed, I added some comments to clarify stuff that I deemed important to emphasize. But, If you need info, please do not hesitate to contact me!

One of the following posts will be focused (more in detail) what I did to get this all working better together with Cake

Have fun!

Posts in this Series

comments powered by Disqus