In a loose continuation of my previous post on using Terraform with Azure Stack Hub, I describe a technique for those deployingLinux VM's in an envirionment where an Enterprise CA has been used to sign the endpoint SSL certs.

Problem statement

Normally, when using a trusted thrid party Certificate Authority to sign the TLS endpoint certs, the root/intermediate/signing Cert Authority public certificate is usually already available in the CA truststore, so you don't have to add them manually. This means you should be able to access TLS enpoints (https sites) without errors being thrown.

As Azure Stack Hub is typically deployed in corporate environments, many use an internal Enterprise CA or self-signed CA to create the mandatory certificates for the public endpoints. The devices accessing services hosted on ASH should have the internal enterprise root CA public cert in the local trusted cert store, so there will be no problems from the client side.

The problem, however is if you want to deploy Marketplace VM's (e.g. you've downloaded Marketplace items), they won't have your signing root CA in the truststore. This is an issue for automation as typically the install script is uploaded to a Storage Account, which the Azure Linux VM Agent obtains and then runs. If the storage account endpoint TLS certificate is untrusted, an error is thrown and you can't run your script :(

Importing the root CA into the truststore

If you're building VM's, there are two options to ensure that the internal root CA is baked into the OS at provisioning time:

create a custom image and publish it to the marketplace. Import the root CA and use the generalized VHD as the base OS disk.
run a command at buildtime to import the root CA

The first option is quite involved, so I prefer the second option :) Thankfully, the command can be distilled to a one-liner:

Debian based systems:

sudo cp /var/lib/waagent/Certificates.pem /usr/local/share/ca-certificates/ash-ca-trust.crt && sudo update-ca-certificates

Fedora based systems:

sudo cp /var/lib/waagent/Certificates.pem /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust

The Azure Linux VM Agent has a copy of the rootCA in the waagent directory, hence making the one-liner possible.

Using Terraform

So, we want to provision a VM, and then run a script once the VM is up and running to configure it. We need to import the root CA before we can get download the script and run it. It's all fairly straightforward, but we do have one consideration to make.

Using the azurestack_virtual_machine_extension resource, we need to define the publisher and type. Typically this would be:

publisher            = "Microsoft.Azure.Extensions"
  type                 = "CustomScript"
  type_handler_version = "2.0"

This would allow us to run a command (we don't have to use a script!):

settings = <<SETTINGS
    {
        "commandToExecute": "sudo cp /var/lib/waagent/Certificates.pem /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust"
    }
SETTINGS

In theory, we can deploy another vm extension which has a dependency on the CA import resource being completed.

That is correct, but we are restricted to deploying only one CustomScript extension type per VM, otherwise when running terraform plan it will fail telling us so.

We need to find an alternative type which will achieve the same objective.

Here's where we can use the CustomScriptForLinux. It's essentially the same as the CustomScript type, but it allows us to get around the restriction.

Here's how it would look:

locals {
  vm_name              = "example-machine"
  storage_account_name = "assets"
 
}
resource "azurestack_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurestack_virtual_network" "example" {
  name                = "example-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurestack_resource_group.example.location
  resource_group_name = azurestack_resource_group.example.name
}

resource "azurestack_subnet" "example" {
  name                 = "internal"
  resource_group_name  = azurestack_resource_group.example.name
  virtual_network_name = azurestack_virtual_network.example.name
  address_prefix     = ["10.0.2.0/24"]
}

resource "azurestack_network_interface" "example" {
  name                = "example-nic"
  location            = azurestack_resource_group.example.location
  resource_group_name = azurestack_resource_group.example.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurestack_subnet.example.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "tls_private_key" "ssh_key" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

# Storage account to store the custom script
resource "azurestack_storage_account" "vm_sa" {
  name                     = local.storage_account_name
  resource_group_name      = azurestack_resource_group.example.name
  location                 = azurestack_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}
# the container to store the custom script
resource "azurestack_storage_container" "assets" {
  name                  = "assets"
  storage_account_name  = azurestack_storage_account.vm_sa.name
  container_access_type = "private"
}
# upload the script to the storage account (located in same dir as the main.tf)
resource "azurestack_storage_blob" "host_vm_install" {
  name                   = "install_host_vm.sh"
  storage_account_name   = azurestack_storage_account.vm_sa.name
  storage_container_name = azurestack_storage_container.assets.name
  type                   = "Block"
  source                 = "install_host_vm.sh"
}
# Create the VM
resource "azurestack_virtual_machine" "example" {
  name                  = "example-machine"
  resource_group_name   = azurestack_resource_group.example.name
  location              = azurestack_resource_group.example.location
  vm_size               = "Standard_F2"
  network_interface_ids = [
    azurestack_network_interface.example.id,
  ]
  
  os_profile {
    computer_name  = local.vm_name
    admin_username = "adminuser"
  }

  os_profile_linux_config {
    disable_password_authentication = true
    ssh_keys {
      path     = "/home/adminuser/.ssh/authorized_keys"
      key_data = tls_private_key.pk.public_key_openssh
    }
  }

  storage_image_reference {
    publisher         = "Canonical"
    offer             = "0001-com-ubuntu-server-jammy"
    sku               = "22_04-lts"
    version           = "latest"
  }
  storage_os_disk {
    name              = "${local.vm_name}-osdisk"
    create_option     = "FromImage"
    caching           = "ReadWrite"
    managed_disk_type = "Standard_LRS"
    os_type           = "Linux"
    disk_size_gb      = 60
  }
}

# import the CA certificate to truststore
resource "azurestack_virtual_machine_extension" "import_ca_bundle" {
  name                 = "import_ca_bundle"
  virtual_machine_id   = azurestack_virtual_machine.vm.id
  publisher            = "Microsoft.Azure.Extensions"
  type                 = "CustomScriptForLinux"
  type_handler_version = "2.0"
  depends_on = [
    azurestack_virtual_machine.vm
  ]
  protected_settings = <<PROTECTED_SETTINGS
    {
        "commandToExecute": "sudo /var/lib/waagent/Certificates.pem /usr/local/share/ca-certificates/ash-ca-trust.crt && sudo update-ca-certificates"
    }
PROTECTED_SETTINGS
}

# install the custom script using different extension type
resource "azurestack_virtual_machine_extension" "install_vm_config" {
  name                 = "install_vm_config"
  virtual_machine_id   = azurestack_virtual_machine.vm.id
  publisher            = "Microsoft.Azure.Extensions"
  type                 = "CustomScript"
  type_handler_version = "2.0"
  depends_on = [
    azurestack_virtual_machine_extension.import_ca_bundle
  ]
  settings = <<SETTINGS
    {
        "fileUris": "${azurestack_storage_blob.host_vm_install.id}"
    }
SETTINGS
  protected_settings = <<PROTECTED_SETTINGS
    {
        "storageAccountName": "${azurestack_storage_account.vm_sa.name}",
        "storageAccountKey": "${azurestack_storage_account.vm_sa.primary_access_key}"
        "commandToExecute": "bash install.sh"
    }
PROTECTED_SETTINGS
}

Thanks for reading and I hope it's given some inspiration!