Azure Plugin Breakdowns - pt 3 - DB Component

May 20, 2020 · 7 min read

Today we'll be showcasing the db hamlet component through the lens of the Azure Resource Manager (ARM) templates. Generated by the Azure provider engine plugin for hamlet, this example serves to illustrate not just how the Azure plugin implements this common component but more broadly how hamlet is able to adapt to each provider without becoming overly generic in function. Making use of some of the more advanced provider-specific template features here, we'll also cover how hamlet is used to perform utility actions before and after a deployment, filling some of the orchestration gaps typical of Infrastructure-as-Code.

As is the nature of hamlet, the examples provided below will change over time. This is to be expected, however the topics covered here will still be relevant.

ARM Template and Parameters File

Diving right in, here are the Azure Resource Manager files we're going to be looking at.

// template.json
{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "hamletio-db-secret": {
      "type": "securestring"
    }
  },
  "variables": {},
  "resources": [
    {
      "name": "postgresserver-db-hamletio1234567890",
      "type": "Microsoft.DBforPostgreSQL/servers",
      "apiVersion": "2017-12-01",
      "properties": {
        "createMode": "Default",
        "version": "11",
        "sslEnforcement": "Disabled",
        "storageProfile": {
          "backupRetentionDays": 35,
          "storageMB": 20480,
          "storageAutogrow": "Disabled"
        },
        "administratorLogin": "cheekyadminname",
        "administratorLoginPassword": "[parameters('hamletio-db-secret')]"
      },
      "location": "australiasoutheast",
      "sku": {
        "name": "GP_Gen5_2"
      }
    },
    {
      "name": "postgresserver-db-hamletio1234567890/hamletdb123",
      "type": "Microsoft.DBforPostgreSQL/servers/databases",
      "apiVersion": "2017-12-01",
      "properties": {},
      "dependsOn": [
        "[resourceId('Microsoft.DBforPostgreSQL/servers', 'postgresserver-db-hamletio1234567890')]"
      ]
    },
    {
      "name": "postgresserver-db-hamletio1234567890/postgresvnetrule-db-hamletio",
      "type": "Microsoft.DBforPostgreSQL/servers/virtualNetworkRules",
      "apiVersion": "2017-12-01",
      "properties": {
	    "virtualNetworkSubnetId": "<subnet-id>"
      },
      "dependsOn": [
        "[resourceId('Microsoft.DBforPostgreSQL/servers', 'postgresserver-db-hamletio1234567890')]"
      ]
    }
  ],
  "outputs": {
    "postgresserverXdbXnoteXpropertiesXfullyQualifiedDomainName": {
      "type": "string",
      "value": "[reference(resourceId('Microsoft.DBforPostgreSQL/servers', 'postgresserver-db-hamletio1234567890'), '2017-12-01', 'Full').properties.fullyQualifiedDomainName]"
    }
  }
}

// parameters.json
{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "hamletio-db-secret": {
      "reference": {
        "keyVault": {
          "id": "/subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/hamletio-rg/providers/Microsoft.KeyVault/vaults/hamletio-vault"
        },
        "secretName": "hamletio-db-secret"
      }
    }
  }
}

Using a Parameter File

Unlike with the “s3” component which was all included within the template file, some of the “db” component has been extracted out of the file and placed into a Parameter file. The values in this file are passed to the template at runtime, allowing the user to set different values every time, or for a template author to develop the template and expose only a few settings to users. Typically you would do this for an ARM template wherever a value might be unique to the person running it or where it may be desirable to keep the value secret - that’s what we’re doing here, but with an extra step.

When setting up a database via template you need to tell it what the administrator credentials are going to be for the database. Even though these templates are going to be in a private code repository it isn’t ideal to leave this information just laying around. So what we do instead is:

In the template, set the administrators password as the value of a parameter: [parameters('hamletio-db-secret')]
At the top of the template, we set the type of the parameter to securestring. ARM know’s never to expose the value of this type anywhere, including logs.
In the parameters file, rather than hardcode a value for the password (which would be a string), we instead set the value to a specifically formatted reference object with a single reference property. ARM knows that this is a KeyVault reference, and when provided the name of the vault to look up and the name of the secret to retrieve, will retrieve the secret from the key vault at deployment time and pass it to the template as a securestring type.

Prologue & Epilogue Scripts

Another difference with the “db” component in hamlet is that it makes use of additional shell scripts - which hamlet also generates as necessary - to perform actions that are typically outside of the scope of what Azure Resource Manager does. As a “desired state” templating language where you defined the desired state of your infrastructure, ARM does not include capabilities of typical utility tasks that you would perform during administration or even deployment such as file download/upload, file encoding/decrypting or in the DB’s case - password generation. That secret that we’ve referenced in the Parameters section wasn’t uploaded manually, nor can it be defined in ARM templates. Instead, hamlet makes use of a Prologue script to ensure a complex password is created and uploaded into KeyVault whilst still ensuring that the password is not exposed.

The following script uses a lot of hamlet-specific functions, but you should be able to follow along with the steps its undertaking (below):

if [[ ! $(az_check_secret "hamletio-vault" "hamletio-db-secret") = *SecretNotFound* ]]; then

  info "Generating Master Password... "

  master_password=""

  while ! [[ "${master_password}" =~ [[:alpha:]] && "${master_password}" =~ [[:digit:]] ]]; do
    master_password="$(generateComplexString "20" )"
  done

  info "Uploading Master Password to Keyvault... "

  az_add_secret "hamletio-vault" "hamletio-db-secret" "${master_password}" || return $?

  create_pseudo_stack "DB Master Secret" "${CF_DIR}/$(fileBase "${BASH_SOURCE}")-secret-pseudo-stack.json" "postgresdbXdbXhamletioXsecret" "hamletio-db-secret" || return $?
fi

First we only perform the action if the secret doesn’t already exist in KeyVault
We generate a random, alpha-numeric string of 20 characters in length, ensuring that it has both numbers and characters (a mandatory requirement for databases in Azure).
Then we upload the password a secret to KeyVault, with a name that corresponds with the secret used by the Parameters.json file.

Pseudo Outputs

A typical ARM template might define outputs that can be consumed by another template - however with the inclusion of Prologue/Epilogue utility scripts, hamlet needs to be able to define the outputs of those utilities to pass their outcomes on to other resources. hamlet accomplishes this with “pseudo” outputs. You can see this on the final line of the Prologue script above. hamlet creates a Pseudo Output - mimicking the same structure of a template deployment’s output structure - and creates a new output for the secret it created (seen below). This structure is not native to ARM but instead of native to hamlet. The next time hamlet compiles all the outputs that it has access to and reference, it will be able to use this Pseudo Output to discover the name of the secret in KeyVault that contains the database administration password. Anything that needs to access this password - such as a component that needs to construct a connection string to write to the database, will first lookup this output.

// prologue-secret-pseudo-stack.json
{
    "Stacks": [
        {
            "Comment": "DB Master Secret",
            "Outputs": [
                {
                    "OutputKey": "postgresdbXdbXhamletioXsecret",
                    "OutputValue": "hamletio-db-secret"
                }
            ]
        }
    ]
}

Putting it all together

So to put all the above into context, here’s the order of operations for the template generation/deployment for this “db” component.

hamlet Create

`hamlet create template ...`

Template generation is run, and outputs these 3 primary files: template.json, parameters.json and prologue.sh

hamlet Deployment

`hamlet manage deployment ...`

hamlet checks for any prologue.sh scripts. Finding one, it executes it.
- Checking the Azure KeyVault for a secret called "hamletio-db-secret” it finds nothing, so generates a complex password and securely stores it in KeyVault under this name.
- The prologue scripts final task is to output a new file - prologue-secret-pseudo-stack.json - this will allow other hamlet components to find the database secret that needs to be referenced.
hamlet looks for a template.json file. Finding one it also checks for a parameters.json which is also present in this case. hamlet creates a new ARM deployment using both files to create the database.
- During deployment the Parameters file performs they KeyVault lookup and retrieves the value of our new secret. It passes the value to the template as a securestring-type, which in turn sets the database administrator password.
hamlet then saves the usual ARM outputs in a new file - stack.json