Associate a WAF policy with an existing Application Gateway - using Azure CLI

I recently had to associate a WAF policy that I had created to an existing Application Gateway that has another WAF policy assigned. The official documentation shows this is possible, and gives an example using PowerShell. I wanted to do the same, but with Azure CLI. I couldn’t find any examples when searching the Web, so here’s what I put together, for anyone else needing to do the same (examples are using Linux):

1. Obtain the Id of the WAF Policy you want to assign to the App Gateway

   Update the configuration on the Application Gateway:

export WAF_POL_ID=$(az network application-gateway waf-policy show -g <WAF Resource Group> --name <WAF policy name> --query id -o tsv)

2. Update the configuration on the Application Gateway:

az network application-gateway update --resource-group <App Gateway RG Name> --name <App Gateway Name> --set firewallPolicy.id=$WAF_POL_ID

That’s it!

Here’s an example code snippet you can use in your scripts:

export APPGW_RG="<app gw rg>"
export WAF_RG="<waf policy rg>"
export WAF_POL_NAME="<waf policy name>"

export APPGW_NAME=$(az network application-gateway list -g ${APPGW_RG} --query [].name -o tsv)

export WAF_POL_ID=$(az network application-gateway waf-policy show -g ${WAF_RG}  --name ${WAF_POL_NAME}  --query id -o tsv)

# update the firewall policy assigned to the WAF
az network application-gateway update  --resource-group $APPGW_RG --name $APPGW_NAME --set firewallPolicy.id=$WAF_POL_ID

Terraform and WSL2 issue

Here’s a quick note on an issue that I encountered today (plus it seems, many other people).

I went to run a Terraform workflow on my system via WSL2, but I cam across a number of problems.

First, was that I couldn’t obtain the State that was stored in an Azure Storage account container. Previously, I used the following config:

backend "azurerm" {
    resource_group_name  = ""
    storage_account_name = ""
    container_name       = "terraform-backend"
    key                  = ""
 }

At runtime, I would specify the values like the example below.

export TF_CLI_ARGS_init="-backend-config=\"storage_account_name=${TERRAFORM_STATE_CONTAINER_NAME}\" -backend-config=\"resource_group_name=${RESOURCE_GROUP_NAME}\" -backend-config=\"access_key=${STG_KEY}\""

However, today, that didn’t work as it just stalled trying to connect to the storage container.

I thought it was something wrong with my credentials, so for troubleshooting purposes, I added the storage account key to see if that made a difference

backend "azurerm" {
    resource_group_name  = ""
    storage_account_name = ""
    container_name       = "terraform-backend"
    key                  = ""
    access_key           = ""
}

I added the primary storage key and lo and behold, this time, it worked.

Strange, as I hadn’t updated the terraform cli or providers.

The next problem I saw was that when I tried to run

terraform plan

it would not complete, seemingly freezing. To troubleshoot this, I ran

export TF_LOG="TRACE"

before running the plan to tell me what was happening in the background.

This in turn produces a verbose output, but something that did catch my was this:

View fullsize

Strange. I know I have internet connectivity and I could certainly connect to Azure using az cli, so I did some Goole-fu and found the following: https://github.com/microsoft/WSL/issues/8022

It was exactly the same problem I had encountered.

Applying the fix https://github.com/microsoft/WSL/issues/5420#issuecomment-646479747 worked for me and persisted beyond a reboot.

(run the code below in your WSL2 instance)

sudo rm /etc/resolv.conf
sudo bash -c 'echo "nameserver 8.8.8.8" > /etc/resolv.conf'
sudo bash -c 'echo "[network]" > /etc/wsl.conf'
sudo bash -c 'echo "generateResolvConf = false" >> /etc/wsl.conf'
sudo chattr +i /etc/resolv.conf

It appears to have occurred in the latest Windows update and affects WSL2. It only appears to affect Go / Terraform as far as I can tell.

Hopefully this will help anyone having a similar issue until the Go provider is fixed.

Fixing Azure Firewall Monitor Workbook

TLDR; Here’s a version of The Azure Firewall Workbook that I fixed: https://github.com/dmc-tech/az-workbooks :)

For a client project, I had to deploy an Azure Firewall and I want to ease the monitoring burden, so I deployed the Azure Monitor workbook as per the article here.

The article has a link to a Workbook that can be deployed to your Azure subscription, and is a great resource giving you plenty of insight into what activity has been taking place on the firewall, via a Log Analytics Workspace configured as part of the diagnostics settings for the resource.

However, I did notice that some of the queries didn’t work as expected and produced some interesting results for the Application rule log statistics.

Below is an example:

View fullsize

If you check out the Action column, you can see that it has quite a lot of information, where I would expect to see ‘Allow’ or ‘Deny’.

View fullsize

I also noticed that some of the other panes did not return any results (such as above), when I expected to see data, so I dug a little deeper, having not really had experience of editing Workbooks.

First of all, I had to check the underlying query, so had to go into ‘edit’ mode.

View fullsize

Once in edit mode, I selected one of the panels that was affected by the faulty query (anything concerning ‘Allow’ for Application Log. Click on the ‘Edit’ button.

View fullsize

We’re concerned with checking the logic and parsing the log, so that the Action is correctly represented, plus the Policy and Rule Collection are populated.

View fullsize

To help triage. I opened the query in the Logs view.

View fullsize

I’ve highlighted where the issues were. First, the logic was incorrect, so the query above was matched, and that did not parse the msg_s field correctly. Second, the parse missed out the ‘space’ for Policy and Rule Collection Group, so would capture incorrectly.

Here’s how the query should look:

View fullsize

Add and msg_s !has “Rule Collection Group as indicated; remove the highlighted and msgs_s !has “Rule Collection , and add spaces as indicated to the parse statement correctly attributes the values to the parameter.

You can see in the query results that the Allow entries no longer have the additional Policy:… text added.

Now that we’ve identify the issue, we need to update the Workbook.

Go back to the workbook end edit the query, putting the identified fixes in place.

Remember to click ‘Done Editing’ when you’re finished.

View fullsize

Here’s a snippet of the query:

(
materializedData
| where msg_s !has "Web Category:" and  msg_s !has ". Url" and msg_s !has  "TLS extension was missing" and msg_s !has "No rule matched" and msg_s !has "Rule Collection Group"
| parse msg_s with Protocol " request from " SourceIP ":" SourcePort " to " FQDN ":" DestinationPort ". Action: " Action ". Rule Collection: " RuleCollection ". Rule: " Rule
),
(
materializedData
| where msg_s !has "Web Category:" and  msg_s !has ". Url" and msg_s !has " Reason: "
| where msg_s has "Rule Collection Group"
| parse msg_s with Protocol " request from " SourceIP ":" SourcePort " to " FQDN ":" DestinationPort ". Action: " Action ". Policy: " Policy ". Rule Collection Group: " RuleCollectionGroup ". Rule Collection: " RuleCollection ". Rule: " Rule
)

Great, we’ve fixed one panel, unfortunately there are more. I’ve shown the process I used to fix the queries, so you can go on and find the the other panels with the same issues and fix yourself, or just go ahead and import a fixed version of the workbook that I uploaded :)

https://github.com/dmc-tech/az-workbooks

Configuring Azure Application Gateway for accessing Kibana

Here’s a quick post on how to configure Azure Application Gateway for any instance of Kibana that is being protected will work.

Background:

I’m working with a private OpenShift cluster deployed to Azure (not ARO, it was deployed via IPI), that I want to publish to the public, but protected via the App GW WAF.

Once the cluster had been deployed and is published via the App Gateway, when trying to access Kibana, an internal 500 error was displayed. If accessing directly from within the virtual network, it worked fine, so I know it’s definitely the App Gateway causing the issue.

Looking at the Kibana logs, I saw the following:

Although I obfuscated my Public IP address, you’ll notice that the port is appended, could this be the problem? (of course the answer is yes, IP address:port isn’t a valid IP address!)

The error message kind of gives the clue:

"message": "Cannot resolve address x.x.x.x:50735: [security_exception] Cannot resolve address x.x.x.x:50735"

I needed to figure out how to rewrite the request header so it would work

(I won’t talk about how I setup backend address pools, http settings, frontend ports, listeners and probes, as that will be part of future in-depth post on how to do it, but I will describe a particular rewrite rule required so that Kibana works.)

X-Forwarded-For Rewrite Rule

The offending header is X-Forwarded-For . This is added by the Application Gateway which includes the IP + port. Microsoft describe this here.

From the portal, open up your Application Gateway and open up Rewrites

Add a Rewrite set

Give your rule a name that is something meaningful, or just go with the defaults.

Click on the Click to configure this action link (1) and enter the settings below. Once configured, don’t forget to Update the rewrite set.

Once the rewrite rule was in place, Kibana opened as expected.

Azure Bastion undocumented requirement gotcha

Just a quick post to highlight an undocumented requirement for Azure Bastion that I came across when deploying a Landing Zone.

I’m creating a new landing zone for a client and we’re using Azure Bastion for secure access to IaaS VM’s. I decided to create the resource in a separate resource group than the Virtual Network as it was uncertain whether this was going to be required long term or not. There’s nothing in the current documentation that indicates that it isn’t possible, so I tried to deploy.

After a few minutes, it failed:

View fullsize

Here’s the less than helpful error:

View fullsize

No matter what I tried (Portal, Terraform, Azure CLI), the same occurred.

Upon speaking to Azure Support, this is a known issue and the mitigation is to deploy the Bastion host within the same Resource Group as the Virtual Network that it is trying to connect to.

I’ve experienced the same when deploying API Management in Azure, but at least the errors from ARM are meaningful and pointed me in the right direction.

Hopefully if you come across the same, and the problem isn’t resolved, this will help you out.