Learn how to use tSQLt to professionally unit test T-SQL code. To enroll in the free self-paced course visit here

Azure ARM template function internals

  • Posted on: 18 December 2017
  • By: Ed Elliott

I have been working with azure arm functions for quite a while and they are exceptionally useful but also quite a pain to work with, visualizing what it is that the functions will resolve to is often hard.

If you haven't used an ARM template, ARM templates are JSON documents which are effectively a whole load of properties made up of a name and a value. The value can either just be a value or it can include arm template functions. If we look at this example:

"resources": [
"type": "Microsoft.Storage/storageAccounts",
"name": "[variables('storageAccountName')]",
"apiVersion": "2016-01-01",
"location": "[resourceGroup().location]",
"sku": {
"name": "Standard_LRS"
"kind": "Storage",
"properties": {}

The value for type, apiVersion, sku name and kind are standard values, standard hardcoded values. The value for name and location are determined at runtime - you can tell the difference because the functions are wrapped in [ and ]. In this case "[variables('storageAccountName')] just grabs a value variable. Variables are defined by a json object called "Variables". Location's [resourceGroup().location] is a little more complicated. It calls a function "resourceGroup()" which returns an object and then the value of location is used.

I originally thought that Microsoft must be using something like node or something that parsed the functions using javascript as the language was very similar to javascript, or so I thought. When I wrote my vs code extension to parse these functions so I could test them locally rather go through a full publish cycle that takes forever in azure; That is exactly what I did, implemented all the functions like 'resourceGroup' and 'variables' as javascript functions and then called 'eval' to execute them.

When I was doing this, two things really stood out as not being quite right, the first was the 'greater' function - what this does is:

"Checks whether the first value is greater than the second value."

The function takes two parameters which could be an int or a string. Now the int version is fairly simple - 100 is bigger than a 1. The string version not quite as simple, you see normally when you compare strings the language follows the ASCII table so an upper case 'A' is less than a lower case 'a' as 'A' maps to number 65 and 'a' maps to 97. In my javascript version, I use "param1 > param2" which works exactly like that but the example on the docs page works the other way around:

'A' is greater than 'a'

So that was the first thing, if they just used javascript to eval the code then they would need to have implemented a custom version of > and I couldn't see why you would do that.

The second thing that was odd is that what I did was implement every function such as greater and equals and when I called eval, those functions were called great but 'if' is a reserved statement in javascript so you can't just call a function 'if' (even if it is a method on a class, I tried) because it is reserved. I had to use a regex to convert all calls to 'if' to 'if_'. This worked great for me but felt a little hacky and I wasn't sure if that is what Microsoft would have done.

So, I implemented my javascript version of the functions and forgot about it for a week or so but I couldn't help think that the greater function was odd (I have only implemented the wrong version so far so I know I need to change it at some point). Anyway, after thinking about this off and on for a little while I woke up one morning and realised that I can probably find out exactly how these were implemented if I used Azure Stack. Azure Stack is a version of Azure that runs on-premise.

So, I grabbed a copy of the Azure Stack SDK and started the download, I thought it would be an installer, install some stuff and then grab the files but it turned out to be a little bit harder.

The setup for Azure Stack is basically:

- Download VHDX of Windows 2016 with a Setup folder
- Run a script that:
- Validates you meet the minimum requirements
- Creates some virtual machines
- Creates a storage cluster
- Does some other stuff

I initially grabbed started the machine and started the installation and it failed pretty quickly because I didn't have a butt load of ram on my laptop and I wasn't running on bare metal and I didn't have enough disk space. I found that these checks were pester tests so I deleted them and the install continued merrily along (albeit slowly, you need hours to install Azure Stack). I also had to enable nested virtualization on the Azure Stack VM on my laptop using:

Set-VMProcessor -VMName stacky -ExposeVirtualizationExtensions $true

Eventually, I got the install running and after a while, I noticed that there was a vhdx being attached to one of the new virtual machines that was called "CommonNugets". Being a nosy sort of a fellow I stopped the install and the Virtual Machine, mounted the original VHDX that had been created by the AzureStack installer and had a poke around until I found this nuget file:


This nuget file contained, drum roll please....


Then I grabbed uniextract to extract the contents and I was a little shocked to not see any javascript files but instead .net dll's. This one in particular:


I have tried to search for some reference to EDM but it seems it is Microsoft internal as I cannot find any reference to it at all.

So, inside this dll, it shows that they use Antlr.Net to parse the code values in the text. I was particularly pleased that we both worked out if a value was a standard value or a template function, in the same way, basically doing:

(expression.First() == '[')) && (expression.Last() == ']'));

Now, 'greater' how is it implemented by Microsoft?

First, they check if either the first or second parameters are strings if they are then if one is a string and one is not a string then they throw an exception.

Assuming that both are strings then they use the .net string.Compare to return an int, if string.Compare returns 0 or greater then the first value is greater than the second.

"string.Compare" is implemented by "System.Globalization.CompareInfo" because as well as the two strings they also pass in "StringComparison.InvariantCulture".

If we look at we can see that "FILE" is greater than "file" when using "StringComparison.InvariantCulture" so what we have isn't a straight map to the ASCII table but the sort order that windows uses.

Once I knew they had their own parser then the whole if / if_ regex thing became clear. I guess now I can fix greater at some point - I won't rush as calling greater on a pair of strings just seems a bit odd to me and I probably won't use it but at least now I can sleep without wondering how 'greater()' was implemented.


My Azure ARM template development workflow

  • Posted on: 13 December 2017
  • By: Ed Elliott

Writing infrastructure as code is pretty nice and I enjoy the declarative approach of defining what you want and letting the tooling take care of the how. I guess this is because of my background as a SQL developer.

What I don't like so much is the development experience and if you don't get this part right it can be a right pain.

What are the challenges?

The first real challenge is writing the correct json for each thing you want to create. For example, to create a virtual machine you need to know what the api version is that you should use and which properties are needed and which properties are required vs optional properties. Once you have that for a virtual machine you need to know the same details for a nic, nsg, storage account for vm diagnostics and any vm extensions you might want to include (and what properties they have).

Getting all this information together can be quite painful, there are four approaches really:

  • 1. "Automation Script" in the portal
  • 2. Viewing JSON schema docs
  • 3. Examples
  • 4. Visual Studio Deployment project

The first seems the easiest which is you deploy something then in the portal you hit the "Automation script" button which generates the arm templates and a script you can use to deploy it with.

There are two problems with this, firstly not everything is supported and so if you are deploying cutting-edge resources this won't work for you. The second issue is the way the script generator names all the resources and creates parameters for everything called something like resource_name_name_parameter if you have a fair few resources then this gets really confusing, really quickly.

The generated script is also for everything in the resource group, even if you only try to generate it for a single resource. Finally, the generated script also includes properties like status which aren't part of a deploy and just obscure the actual bits you need. If you do use the automation script option it must only really be used as a guide to get you started or to check something.

The second option is to look at the arm JSON schema's and work out what properties are supported with api version, you can find them:

The third option using examples from either the Microsoft docs site or github is good but often samples do one very specific thing and you need to spend a while tweaking them. The samples are also all different and there is not a common way to name things so merging multiple together is often a pain.

The last option is the visual studio deployment project which has a wizard to add resources, this is often quite out of date and the fix suggested by Microsoft (see comments is to just edit the json manually.

So, there really isn’t one ideal way to create arm templates the way I work is a combination of 1-3, I don’t really use visual studio because I have migrated to use vscode.

Once you have the resource versions and properties you then have two more problems, the first is that you end up with a big blob of json and navigating resources and doing things like copying and pasting and moving things about is a little risky. Visual studio has a “JSON Viewer” that is a tree viewer and lets you navigate different objects like parameters and resources but not the properties of resources so if you have a large resource you can still only navigate to the beginning of it.

The second problem is that ARM templates have their own JavaScript like language where you can have things like:


If you use this sort of thing to name your objects you don’t really know if they are right until you deploy, these things can get really complicated. Both visual studio and vscode have add-ins that provide intelligence and red squiggles to show if these are right or wrong but won’t let you see if you have used the right parameter or got the final concatenation correct, the amount of times I have deployed something to see the name isn’t quite correct so I have had to redeploy is literally infuriating!

What is the solution?

The first thing is getting the correct resource versions and properties and to be honest I haven't found a better way than using a combination of the approaches. I will look at bits already deployed and use the automation script to see what it can look like and also use existing samples and the azure docs such as:

If I get really stuck then I will use the template schemas.

For the tooling I was using visual studio but I was having a problem because I was using octopus deploy variables which would do a find and replace with parameters in the parameters file and this is fine for strings and numbers but when passing arrays the format I need in the parameters file means the file is invalid json and when I copied and pasted something into that file Visual Studio tried to help the invalid format and ended up messing it up so I would have to undo the changes and be very careful about changing that file.

Because of this problem with the parameters file I started using vscode and the Microsoft Azure arm extension which gave me some bugs like intelligence etc and I could edit the parameters file easily but it didn't have the JSON outliner that Visual Studio had so I back to having a blob of JSON and editing it became a pain again.

This led me to write a similar JSON outliner for vs code but because I wanted to get a better idea of what the code would look like I also evaluate any properties which have values that are code values like:


Then the treeview also shows the code value and the evaluated like:


I also added the facility to run a template script like the one above to see what it would look like when deployed, typing in lots of if(equals((((((((( etc into a string and not knowing what it will actually do is scary, so this helps me.

To get this to work there were a couple of things I have to do, firstly we need a parameters file that we can find so I look for a file called parameters.json if you have a different file for your parameters or you have something like I have invalid JSON then you can create a dummy parameters file and point to it using the .vscode/armed.json file so when we evaluate an expression we have something we can use for the parameter value.

I also haven't implemented everything, if you use "reference()" to do something like looking up a property of a deployed resource then I don't evaluate it - I don't want to make the api call as part of an extensions which would be very slow - it is something I am thinking about implementing using something similar to the parameters file but will see if I need it or not.

There are also a couple of things like the uniqueString, I can't find the type of hash that is used so can't guarantee that it will be the same and the gt() function on strings behaves the opposite way to the way I expect it to. If you call gt on a start 'a' is normally greater than 'A' but in arm it isn't. I've implemented it the way I think it should be and using gt() with two strings is odd anyway so I wouldn't use it.

The other oddity is that I use eval to run the code so you could if you wanted to also put valid javascript in and it would work but wouldn't actually work if you deployed it, I don't know why anyone would do that though.

So I use a mixture of ways to get the resources and my armed extension to navigate templates, I display resources and properties in the window so you can navigate either resources easier. I also use the evaluate tool to see whether my code not just compiles but looks correct.

ARMED - Azure ARM VS Code extension

  • Posted on: 6 December 2017
  • By: Ed Elliott

I have been working with ARM templates for a little while and have found it really annoying to test functions inside the templates, you see if you have an ARM template it is a JSON document made up of lots of key/values like:

{"name": "a_name"}

but it is slightly more complicated because there is a set of functions in a javascript like language you can use in the values so you might have:

{"name": "[concat('a_', 'name')]"}

which effectively does the same thing, now because ARM templates make lots of use of these template functions it is hard to write them and see if they are correct before deploying so I wrote an extension for vs code ( which does two things (at the moment), the first is an outliner similar to the visual studio (fat version) and more excitingly, it includes a tool to test these functions so when you have something complicated like:

"name": "[concat(parameters('servicesNames')[copyIndex()], 'pip')]",

and hoping you have the right parameters, you can use the vs code add-in to see what it is actually going to be before you send it off to Azure.

If you want to see it in action, check this:

Any issues shout or raise an issue

Site Search with Duck Duck Go