Running Cake in Azure Pipelines and accounting for optional build-process-variables
Last week I wrote about two related topics that can be seen as a basis for this post: (1) Cake in the enterprise which is about building your enterprise applications with Cake (at scale) and what you need to do to make it manageable and (2) Migrating your builds to Azure DevOps pipelines which is about migrating hundreds of builds from TeamCity to Azure DevOps.
If you want to learn about this, then I really encourage you to take some time and read those posts!
In this post, I will bring those 2 posts together and describe you my process of finding the (for me) ideal
azure-pipelines.yml file that is being used to define (and run) my Azure Pipelines. I will also explain you how you can set up multiple azure pipelines with only one
azure-pipelines.yml file and change the behavior of the pipeline based on optional pipeline variables.
I hear you asking: Why would you want this? Well, the answer is easy: In Cake , you can define arguments (with default values) and these arguments can be (optionally) overridden when starting a cake build. This is useful and allows you (for example) to define a build with CI functionality, that only builds and runs the tests, and another "FullBuild" that also packages and pushes your artefact to an artefact repo. All you need to do is to pass "an override" to this default to the task that initiates your cake build.
This is only one of the many scenario's, but not the one that I want to cover in this post as this is something that you can do already today. What is harder, is setting up an
azure-pipelines.yml file that takes into account the notion that variables do not necessarily "have to be defined" on build level! The idea of the (current) cake extension
is that you can set up a build with variables, but you cannot make them "optional" and that is what I wanted/needed...(especially in the context of a org wide used template)
This is something that I ran into during the build migration that I was preparing: The goal was to migrate all the Cake builds from TeamCity to Azure DevOps and keep as much logic as possible in Cake and to keep the
azure-pipelines.yml file as simple as possible.
Previously, the only logic that I had set up in TeamCity, was exactly what I'm writing about in this post and in that context, I was able to solve it with a TeamCity build template that accounted for "optional parameters".
Now, In Azure Pipelines, I tried to come up with something similar and I soon started writing PowerShell tasks that deal with these "optional variables". The issue with this code, was that it had to be "replicated" for each variable that I wanted to support and that was a bit of an issue to me as the
azure-pipelines.yml became hard to read and to maintain.
It is therefore that I started thinking and during one evening, I created an azure pipelines extension that helps me to account for those "optional" variables and that builds the "argument string" that is being passed to the (in our case) PowerShell task. I used it for a while (in a private context) but after this "private preview", I am now happy to announce that I'm making it publicly available for everyone to use!
You can find it here in the Visual studio MarketPlace ! The documentation can also be found on this page, but it's really easy to use!
To give you an idea, I will give you a little preview:
With this extension, all you need to do is:
- Which arguments you want to support (in the context of cake, this can be the target, verbosity, Deploy(yes/no)?, ... )
- The name of the variable that will keep the output. This variable will grow as it passes through the
ScriptArgumentParsertasks in your pipeline. Below, I called it:
parsedargumentlist(yes, you can have multiple if you want/need to)
- Add these in your
- Set up the PowerShell task and pass the
parsedargumentlistvariable to the
- Set up one of more Azure Pipelines based on the same
- Define one or more of these variables on pipeline level so that these builds actually do something different (otherwise, it would be really stupid)
- That's it, you can start running your build(s)!
Such a task would look like this:
1- task: ScriptArgumentParser@0 2 inputs: 3 VariableName: caketarget 4 VariableNameInTool: Target 5 VariableBinder: '=' # Optional -> this allows you to adopt the (to be) result to the needs of your tool/platform 6 ParsedArgumentListName: parsedArgumentList
You can have as many of those as you want and this really will help you to have a flexible solution.
Using the result of the previous step(s) in the PowerShell task can then be done like this:
1 - task: PowerShell@2 2 inputs: 3 filePath: buildscript.ps1 4 arguments: $(parsedArgumentList) # -> what you have been building 5 workingDirectory: $(workdir) #-> optional 6
There is only one gotcha at the moment where defining no variables (at all) will most likely fail your build. I solved this issue and also documented it on the page of the extension (under the "Known issues" section)
Combined, your YAML file, might look like this:
(omitting all the stuff that is not needed here!)
1- task: ScriptArgumentParser@0 2 inputs: 3 VariableName: caketarget 4 VariableNameInTool: Target 5 VariableBinder: '=' # Optional -> this allows you to adopt the (to be) result to the needs of your tool/platform 6 ParsedArgumentListName: parsedArgumentList 7- task: ScriptArgumentParser@0 8 inputs: 9 VariableName: cakeverbosity 10 VariableNameInTool: Verbosity 11 VariableBinder: '=' # Optional -> this allows you to adopt the (to be) result to the needs of your tool/platform 12 ParsedArgumentListName: parsedArgumentList 13- task: ScriptArgumentParser@0 14 inputs: 15 VariableName: deply 16 VariableNameInTool: deploy 17 VariableBinder: '=' # Optional -> this allows you to adopt the (to be) result to the needs of your tool/platform 18 ParsedArgumentListName: parsedArgumentList 19 - task: PowerShell@2 20 inputs: 21 filePath: build.ps1 # -> this one calls cake 22 arguments: $(parsedArgumentList) # -> what you have been building 23 workingDirectory: $(workdir) #-> optional 24
How do you do this at scale?
As I talked to you about migrating a huge amount of builds, you might already suspect that I put this logic in a central place that is easy to manage. For this, I used a concept of Azure Pipelines that is called templates . Typically, these templates reside in the same (git) repo, but you can also add a reference to another (git) repo. In this context, these other repo's are referenced as resources, which can be referenced. (This is described here )
The main idea is that I now have a template, where I have defined my process to collect the optional pipelines variables that I support and then subsequently call the cake script with the parsed argument list.
All the "end user needs to do" is set up a very small
azure-pipelines.yml file that includes this resource and calls the template. Allowing them to work efficiently and without having to understand Cake and Azure Pipelines!
This really is something!
Bringing all these things together (Azure Pipelines, Cake and my new extension), really leverages what you can do with Cake in the context of Azure DevOps and this without you wanting to pull out your hair!
I really hope that this will help you too!
Posts in this Series
- Considerations when setting up a build (migration) strategy
- Running a build in Azure DevOps with Cake and Azure Artifacts
- Running Cake in Azure Pipelines and accounting for optional build-process-variables
- Automatically migrating your builds to Azure DevOps
- Extending build.cake with your (enterprise) functionality
- Dealing with error TF401019 when using submodules in Azure Pipelines
- Migrating your builds to TeamCity