Running Cake in Azure Pipelines and accounting for optional build-process-variables

If you want to learn about this, then I really encourage you to take some time and read those posts!

In this post, I will bring those 2 posts together and describe you my process of finding the (for me) ideal azure-pipelines.yml file that is being used to define (and run) my Azure Pipelines. I will also explain you how you can set up multiple azure pipelines with only one azure-pipelines.yml file and change the behavior of the pipeline based on optional pipeline variables.

Why?

I hear you asking: Why would you want this? Well, the answer is easy: In Cake , you can define arguments (with default values) and these arguments can be (optionally) overridden when starting a cake build. This is useful and allows you (for example) to define a build with CI functionality, that only builds and runs the tests, and another "FullBuild" that also packages and pushes your artefact to an artefact repo. All you need to do is to pass "an override" to this default to the task that initiates your cake build.

This is only one of the many scenario's, but not the one that I want to cover in this post as this is something that you can do already today. What is harder, is setting up an azure-pipelines.yml file that takes into account the notion that variables do not necessarily "have to be defined" on build level! The idea of the (current) cake extension is that you can set up a build with variables, but you cannot make them "optional" and that is what I wanted/needed...(especially in the context of a org wide used template)

This is something that I ran into during the build migration that I was preparing: The goal was to migrate all the Cake builds from TeamCity to Azure DevOps and keep as much logic as possible in Cake and to keep the azure-pipelines.yml file as simple as possible.

Previously, the only logic that I had set up in TeamCity, was exactly what I'm writing about in this post and in that context, I was able to solve it with a TeamCity build template that accounted for "optional parameters".

Now, In Azure Pipelines, I tried to come up with something similar and I soon started writing PowerShell tasks that deal with these "optional variables". The issue with this code, was that it had to be "replicated" for each variable that I wanted to support and that was a bit of an issue to me as the azure-pipelines.yml became hard to read and to maintain.

What?

It is therefore that I started thinking and during one evening, I created an azure pipelines extension that helps me to account for those "optional" variables and that builds the "argument string" that is being passed to the (in our case) PowerShell task. I used it for a while (in a private context) but after this "private preview", I am now happy to announce that I'm making it publicly available for everyone to use!

You can find it here in the Visual studio MarketPlace ! The documentation can also be found on this page, but it's really easy to use!

To give you an idea, I will give you a little preview:

With this extension, all you need to do is:

1. Define
   1. Which arguments you want to support (in the context of cake, this can be the target, verbosity, Deploy(yes/no)?, ... )
   2. The name of the variable that will keep the output. This variable will grow as it passes through the ScriptArgumentParser tasks in your pipeline. Below, I called it: parsedargumentlist (yes, you can have multiple if you want/need to)
2. Add these in your azure-pipelines.yml file
3. Set up the PowerShell task and pass the parsedargumentlist variable to the arguments input
4. Set up one of more Azure Pipelines based on the same azure-pipelines.yml
5. Define one or more of these variables on pipeline level so that these builds actually do something different (otherwise, it would be really stupid)
6. That's it, you can start running your build(s)!

Such a task would look like this:

1- task: ScriptArgumentParser@0
2        inputs:
3          VariableName: caketarget
4          VariableNameInTool: Target
5          VariableBinder: '=' # Optional -> this allows you to adopt the (to be) result to the needs of your tool/platform
6          ParsedArgumentListName: parsedArgumentList

You can have as many of those as you want and this really will help you to have a flexible solution.

Using the result of the previous step(s) in the PowerShell task can then be done like this:

1 - task: PowerShell@2
2        inputs:
3          filePath: buildscript.ps1
4          arguments: $(parsedArgumentList) # -> what you have been building
5          workingDirectory: $(workdir) #-> optional

There is only one gotcha at the moment where defining no variables (at all) will most likely fail your build. I solved this issue and also documented it on the page of the extension (under the "Known issues" section)

Combined, your YAML file, might look like this:

(omitting all the stuff that is not needed here!)

 1- task: ScriptArgumentParser@0
 2        inputs:
 3          VariableName: caketarget
 4          VariableNameInTool: Target
 5          VariableBinder: '=' # Optional -> this allows you to adopt the (to be) result to the needs of your tool/platform
 6          ParsedArgumentListName: parsedArgumentList
 7- task: ScriptArgumentParser@0
 8        inputs:
 9          VariableName: cakeverbosity
10          VariableNameInTool: Verbosity
11          VariableBinder: '=' # Optional -> this allows you to adopt the (to be) result to the needs of your tool/platform
12          ParsedArgumentListName: parsedArgumentList
13- task: ScriptArgumentParser@0
14        inputs:
15          VariableName: deply
16          VariableNameInTool: deploy
17          VariableBinder: '=' # Optional -> this allows you to adopt the (to be) result to the needs of your tool/platform
18          ParsedArgumentListName: parsedArgumentList  
19 - task: PowerShell@2
20        inputs:
21          filePath: build.ps1 # -> this one calls cake
22          arguments: $(parsedArgumentList) # -> what you have been building
23          workingDirectory: $(workdir) #-> optional                    

How do you do this at scale?

As I talked to you about migrating a huge amount of builds, you might already suspect that I put this logic in a central place that is easy to manage. For this, I used a concept of Azure Pipelines that is called templates . Typically, these templates reside in the same (git) repo, but you can also add a reference to another (git) repo. In this context, these other repo's are referenced as resources, which can be referenced. (This is described here )

The main idea is that I now have a template, where I have defined my process to collect the optional pipelines variables that I support and then subsequently call the cake script with the parsed argument list.

All the "end user needs to do" is set up a very small azure-pipelines.yml file that includes this resource and calls the template. Allowing them to work efficiently and without having to understand Cake and Azure Pipelines!

This really is something!

Conclusion

Bringing all these things together (Azure Pipelines, Cake and my new extension), really leverages what you can do with Cake in the context of Azure DevOps and this without you wanting to pull out your hair!

I really hope that this will help you too!