When building an Azure Pipeline in Azure DevOps (AzDo), it’s commonplace to build infrastructure as part of a pipeline. If you’re building infrastructure in Azure, Microsoft provides an infrastructure-as-code approach called Azure Resource Management (ARM) templates. When invoking a deployment from an ARM template within an Azure pipeline though can sometimes prove troublesome, especially when dealing with ARM output variables.
In this article, you’re to learn one of the most troublesome (personal opinion) aspects of using ARM templates in AzDo pipelines – managing ARM output variables.
Understanding ARM Output and Templates
When you invoke an ARM deployment via a template, ARM gives you the ability to send output back to the process that invoked the deployment called outputs. Outputs allow you to send values back out of the deployment just like a PowerShell function, for example.
ARM output is not unique to running deployments in Azure Pipelines. Regardless of how you invoke ARM deployments, you can always output values.
If you need to capture the value generated via an ARM template deployment, you can do so using the outputs section of an ARM template. For example, suppose you have an ARM template that creates an Azure SQL Server and database. You are invoking this ARM template in a larger automation workflow like an Azure pipeline. As part of that pipeline, you need to know what the full-qualified domain name (FQDN) of the SQL Server created and the database name.
If you include an outputs section at the bottom of the template, you can see how to return both the SQL Server’s FQDN and database name as seen in the following code snippet.
The following code snippet assumes a parameter of
sqlServerNameandsqlDbNamehave been defined in the ARM template.
"outputs": {
"sqlServerName": {
"value": "[reference(resourceId('Microsoft.Sql/servers', parameters('sqlServerName'))).fullyQualifiedDomainName]",
"type": "string"
},
"databaseName": {
"value": "[parameters('sqlDbName')]",
"type": "string"
}
}Below you can see the entire ARM template the outputs section below came from. The outputs section is all the way to the bottom.
{
"$schema": "<https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#>",
"contentVersion": "1.0.0.0",
"parameters": {
"sqlServerName": {
"type": "string",
"metadata": {
"description": "Enter a globally unique hostname for your SQL server"
}
},
"sqlDbName": {
"type": "string"
},
"adminUserName": {
"type": "string",
"metadata": {
"description": "Enter a username for SQL admin"
}
},
"adminPassword": {
"type": "securestring",
"metadata": {
"description": "Enter a password for SQL admin"
}
}
},
"resources": [
{
"name": "[parameters('sqlServerName')]",
"type": "Microsoft.Sql/servers",
"apiVersion": "2015-05-01-preview",
"location": "[resourceGroup().location]",
"tags": {
"displayName": "[parameters('sqlServerName')]"
},
"properties": {
"administratorLogin": "[parameters('adminUserName')]",
"administratorLoginPassword": "[parameters('adminPassword')]"
},
"resources": [
{
"type": "firewallRules",
"apiVersion": "2015-05-01-preview",
"dependsOn": [
"[resourceId('Microsoft.Sql/servers', parameters('sqlServerName'))]"
],
"location": "[resourceGroup().location]",
"name": "AllowAllWindowsAzureIps",
"properties": {
"startIpAddress": "0.0.0.0",
"endIpAddress": "0.0.0.0"
}
}
]
},
{
"name": "[concat(parameters('sqlServerName'), '/',parameters('sqlDbName'))]",
"type": "Microsoft.Sql/servers/databases",
"apiVersion": "2014-04-01",
"location": "[resourceGroup().location]",
"dependsOn": [
"[resourceId('Microsoft.Sql/servers', parameters('sqlServerName'))]"
],
"properties": {
"collation": "SQL_Latin1_General_CP1_CI_AS",
"edition": "Basic",
"maxSizeBytes": "1073741824",
"requestedServiceObjectiveName": "Basic"
}
}
],
"outputs": {
"sqlServerName": {
"value": "[reference(resourceId('Microsoft.Sql/servers', parameters('sqlServerName'))).fullyQualifiedDomainName]",
"type": "string"
},
"databaseName": {
"value": "[parameters('sqlDbName')]",
"type": "string"
}
}
}However you invoke the ARM deployment using the template above, ARM will return the FQDN of the SQL Server and the database name.
Invoking ARM Templates in Azure Pipelines
If you’re building a YAML-based Azure pipeline, one popular way to invoke ARM template deployments is to use the Azure Resource Group Deployment task. Using this task, you can define the ARM template, resource group to deploy to, the path to the template and so on.
In the following code snippet, you can see an example of an Azure Resource Group Deployment task defined in an Azure pipeline.
- task: AzureResourceManagerTemplateDeployment@3
name: "Compute"
inputs:
azureResourceManagerConnection: "ARM"
subscriptionId: "xxxxxx"
action: "Create Or Update Resource Group"
resourceGroupName: xxxxxxxx
location: "xxxxxx"
templateLocation: "Linked artifact"
csmFile: "$(System.DefaultWorkingDirectory)/template.json"
deploymentMode: "Incremental"
deploymentOutputs: database_armCapturing ARM Output for the Azure Pipeline
Notice one attribute in the code snippet above, in particular – deploymentOutputs. By default, even if the ARM template defines outputs, the Azure pipeline will have no idea of them. The pipeline will simply discard the outputs. Unless you use the deploymentOutputs attribute.
The deploymentOutputs attribute tells the pipeline to assign all of the values returned via the outputs section in the ARM template and assign them to a pipeline variable in JSON format. In the above example, that pipeline variable is called database_arm.
The value
database_armis not required. You are free to use whatever variable name you wish.
If the task shown below were used to deploy the database ARM template mentioned earlier, the database_arm pipeline variable would contain a JSON string that looks like the code snippet below.
{
"sqlServerName": {
"value" : "[some sql server name].database.windows.net",
"type": "string"
},
"databaseName": {
"value" : "[some sql db name]",
"type": "string"
}
}At this point, the Azure Pipeline would now know about an output variable called database_arm.
Parsing ARM Output Variables in a Pipeline
Creating a pipeline variable containing the ARM output is great but what are you going to do with them? You don’t just create a pipeline variable for nothing!
Let’s say you need both the FQDN of the SQL Server and the database name to pass to an Azure SQL Database Deployment task later on in the pipeline. You need to provide both of those values to the ServerName and DatabaseName attributes of the task.
We still only have a single pipeline variable called database_arm that consists of a JSON string. As-is, there’s no way to parse the values of sqlServerName and databaseName in that JSON string to the actual values you need.
You’d like to have two separate pipeline variables called databaseName and sqlServerName that each only contains the expected string value of [some sql db name] and [some sql server name] respectively. You need to parse the JSON and create two new pipeline variables.
When done, you’d love to have an Azure SQL Database Deployment task that looks like the below example. Notice absolutely no database_arm variable to speak of.
- task: SqlAzureDacpacDeployment@1
inputs:
azureSubscription: 'ARM'
AuthenticationType: 'server'
DatabaseName: "$(databaseName)" ## From database ARM deployment
ServerName: "$(sqlServerName)" ## From database ARM deployment
SqlUsername: "$(SqlAdminUsername)" ## From keyvault
SqlPassword: "$(SqlAdminPassword)" ## From keyvault
deployType: 'SqlTask'
SqlFile: '$(System.DefaultWorkingDirectory)/file.sql'
IpDetectionMethod: 'AutoDetect'So how do you get from a database_arm pipeline variable full of loose JSON text to two pipeline variables called databaseName and sqlServerName? PowerShell!
Using PowerShell to Create Pipeline Variables
We have a mission: create two pipeline variables called databaseName and sqlServerName from a pipeline variable called database_arm that looks like the below code snippet.
{
"sqlServerName": {
"value" : "[some sql server name].database.windows.net",
"type": "string"
},
"databaseName": {
"value" : "[some sql db name]",
"type": "string"
}
}To succeed in this mission, first, forget about Azure Pipelines all together. You must now create some PowerShell code to “convert” these variables.
Defining Pipeline Variables
Before you should write any code, you first need to know how to define pipeline variables with PowerShell. Doing so required returning one of two strings via a PowerShell task.
To create output (and non-output) variables, return specifically formatted string as shown below. The below example sets a pipeline variable called foo to a value bar.
"##vso[task.setvariable variable=foo;]bar"If you need the pipeline variable to be an output variable, you’d add isOutput=true to the string like below.
"##vso[task.setvariable variable=foo;isOutput=true]bar"Wrap up either one or both of those strings in some PowerShell and you’d get:
## Creates a standard pipeline variable called foo and sets the value to bar
Write-Output "##vso[task.setvariable variable=foo;]bar"
## Creates an output variable called foo and sets the value to bar
Write-Output "##vso[task.setvariable variable=foo;isOutput=true]bar"Converting an ARM Output Variable to Specific Values
You know by now that you can create an outputs section in an ARM template, deploy that template in an Azure pipeline and use the deploymentOutputs attribute on the pipeline task to create a pipeline variable containing a JSON string with the values of the outputs ARM template. section. You’ve also learned how to use PowerShell to create pipeline variables.
Let’s now bring everything together and automate this whole process as much as possible!
You’ve seen above that the database_arm pipeline variable created via the ARM template deployment looks like the following code snippet.
{
"sqlServerName": {
"value" : "[some sql server name].database.windows.net",
"type": "string"
},
"databaseName": {
"value" : "[some sql db name]",
"type": "string"
}
}How would you grab the values from the JSON with PowerShell if this string weren’t in a pipeline at all?
Let’s say that you’ve copied and pasted the above JSON snippet into a variable called $armOutput in your local PowerShell session.
$armOutput = @'
{
"sqlServerName": {
"value" : "[some sql server name].database.windows.net",
"type": "string"
},
"databaseName": {
"value" : "[some sql db name]",
"type": "string"
}
}
'@Using the handy ConvertFrom-Json cmdlet, you can easily convert the above JSON string into a PowerShell object.
$armOutputObj = $armOutput | convertfrom-jsonOnce you have the JSON string in a PowerShell object, you can then inspect all properties and values using the PSObject.Properties property that’s on every PowerShell object as shown below.
$armOutputObj.PSObject.Properties | ForEach-Object {
$keyname = $_.Name
$value = $_.Value.value
Write-Host "The value of [$keyName] is [$value]"
}The above code snippet will yield this beautiful sight below.
You can now reference each value in $armOutput as separate values. You’re so close!
Since you’ll be performing the actions above in a pipeline and need to make those values pipeline variables, use your knowledge of creating pipeline variables with PowerShell.
$armOutputObj.PSObject.Properties | ForEach-Object {
$keyname = $_.Name
$value = $_.Value.value
## Creates a standard pipeline variable
Write-Output "##vso[task.setvariable variable=$keyName;]$value"
## Creates an output variable
Write-Output "##vso[task.setvariable variable=$keyName;isOutput=true]$value"
}Now take the code above and add it to a PowerShell pipeline task like below. If you were following along in this example, you should now have two pipeline variables created called databaseName and sqlServerName!
- pwsh: |
## database_arm is already a pipeline variable returned via the ARM deployment
## task's deploymentOutputs attribute
$armOutputObj = $env:database_arm | convertfrom-json
$armOutputObj.PSObject.Properties | ForEach-Object {
$keyname = $_.Name
$value = $_.Value.value
## Creates a standard pipeline variable
Write-Output "##vso[task.setvariable variable=$keyName;]$value"
## Creates an output variable
Write-Output "##vso[task.setvariable variable=$keyName;isOutput=true]$value"
}Better yet, create a PowerShell script in the source repo of the Azure DevOps project like below to get fancy. The script below accounts for other types of deployment outputs like arrays, secure strings and is configurable to create output variables or not.
[CmdletBinding()]
param (
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[string]$ArmOutputString,
[Parameter()]
[ValidateNotNullOrEmpty()]
[switch]$MakeOutput
)
Write-Output "Retrieved input: $ArmOutputString"
$armOutputObj = $ArmOutputString | ConvertFrom-Json
$armOutputObj.PSObject.Properties | ForEach-Object {
$type = ($_.value.type).ToLower()
$keyname = $_.Name
$vsoAttribs = @("task.setvariable variable=$keyName")
if ($type -eq "array") {
$value = $_.Value.value.name -join ',' ## All array variables will come out as comma-separated strings
} elseif ($type -eq "securestring") {
$vsoAttribs += 'isSecret=true'
} elseif ($type -ne "string") {
throw "Type '$type' is not supported for '$keyname'"
} else {
$value = $_.Value.value
}
if ($MakeOutput.IsPresent) {
$vsoAttribs += 'isOutput=true'
}
$attribString = $vsoAttribs -join ';'
$var = "##vso[$attribString]$value"
Write-Output -InputObject $var
}Perhaps you’ve saved the above code to a file called parse_arm_deployment_output.ps1 in the project’s source repo. At that point, you can just reference the script to automatically create the ARM output pipeline variables to your choosing.
- pwsh: $(System.DefaultWorkingDirectory)/parse_arm_deployment_output.ps1 -ArmOutputString '$(database_arm)' -MakeOutput -ErrorAction StopOnce the above PowerShell script runs, you can accomplish the goal of referencing the value of both variables as a pipeline variable in future tasks in the pipeline.
- task: SqlAzureDacpacDeployment@1
inputs: azureSubscription: 'ARM'
AuthenticationType: 'server'
DatabaseName: "$(databaseName)" ## From database ARM deployment
ServerName: "$(sqlServerName)" ## From database ARM deployment
SqlUsername: "$(SqlAdminUsername)" ## From keyvault
SqlPassword: "$(SqlAdminPassword)" ## From keyvault
deployType: 'SqlTask'
SqlFile: '$(System.DefaultWorkingDirectory)/file.sql'
IpDetectionMethod: 'AutoDetect'Conclusion
Managing ARM templates, deployments and ARM output variables can be challenging in an Azure pipeline. Once you come up with a solution proposed in this article though, you no longer have to worry about the ins and outs of how to do it. Be sure to grab the PowerShell script referenced in this article to save yourself tons of time in the future!
