Azure to Azure Stack site-to-site IPSec VPN tunnel failure... after 8 hours

We had a need to create a site-to-site VPN tunnel for a POC from Azure Stack to Azure.  It seemed pretty straight forward.  Spoiler alert, obviously I'm writing this because it wasn't. The tunnel was created okay, but each morning it would no longer allow traffic to travel across it.  The tunnel would show connected in Azure and in Azure Stack but traffic just wouldn't flow; ping, SSH, RDP, DNS and AD all wouldn't work.  After some tinkering we found we would have to change the connection's sharedkey value to something random, save it, then change it back to the correct key.  This only worked from the Azure Stack side of the connection, to re-initiate successfully and allow traffic to flow again (or recreate the connection from scratch).  It would work for another 8 hours and then fail to pass traffic again.

My suspicion was the re-keying, as this would explain why it worked at first and would fail the next day (everyday, for the last 5 days).  I tried using VPN diagnostics on the Azure side, as they don't currently support VPN diagnostics on Azure Stack (we are on update 1805).  After reviewing the IKE log there were some errors, but it was hard to find something to tell me what was going wrong, more specifically something I could do to fix it.  Below is the IKE log file I collected through the VPN diagnostics from Azure.

I logged a case with Microsoft support.  The first support person did their best.  While Microsoft can identify the endpoints they are connecting to, from Azure, they do not have permission to dig any deeper and look into the contents of our subscriptions hosted on Azure Stack.  I was asked to change the local VPN gateways from specific subnets to be the entire vnet address space.  While it worked initially, again it failed after 8 hours.

The support engineer collected some network traffic and other logs and forwarded the case to an Azure Stack support engineer. Once the call was assigned they asked me to connect to the privileged endpoint (PEP) and we proceeded with breaking the glass to Azure Stack to troubleshoot.  The engineer gave me a few PowerShell commands to run to investigate what was going on.

#First find out which of the VPN gateways is active. icm Azs-gwy01,Azs-gwy02 { get-vpns2sinterface }

#Check Quick Mode Key Exchange icm Azs-gwy01 { get-netIpsecQuickModeSA } 

#Check Main Mode Key Exchange icm Azs-gwy01 { get-netIpsecMainModeSA }

The Microsoft engineer had a hunch of exactly what he was looking for and was on point. The commands showed that the Quick mode key exchange had failed to complete the refresh, yet the Main Mode had succeeded.  This explained why the tunnel was up but no traffic could flow across it.

We rebooted the active VPN gateway so the tunnels would fail-over to the second gateway.  Logging was on by default so we just had to wait for the next timeout to occur.  When it did I was given the task of collecting and uploading the logs from the PEP.

These logs are a series of ETL files that need to be processed by Microsoft to make sense of them.  Fortunately it turned up the following log entries.

As commented above, the root cause was that the PFS and CipherType setting were incorrect on the Azure VPN gateway.  I was given a few PowerShell commands to run against the Azure Subscription to reconfigure the IPSec policy for the connection on the Azure side to match the policy of the VPN gateway and connection on Azure Stack.

$RG1 = 'RESOURCE GROUP NAME' $CONN = 'CONNECTION NAME' $GWYCONN = Get-AzureRmVirtualNetworkGatewayConnection -Name $CONN -ResourceGroupName $RG1 $newpolicy  = New-AzureRmIpsecPolicy -IkeEncryption AES256 -IkeIntegrity SHA256 -DhGroup DHGroup2 -IpsecEncryption GCMAES256 -IpsecIntegrity GCMAES256 -PfsGroup PFS2048 -SALifeTimeSeconds 27000 -SADataSizeKilobytes 33553408 Set-AzureRmVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $GWYCONN -IpsecPolicies $newpolicy 

Almost there.  When I tried to run the command, the basic Sku doesn't allow for custom IPSec policies.  Once I changed the Sku from basic to standard the command worked and the tunnel has been up and stable.

While this is any easy fix that anyone can run against their Azure subscription without opening a support ticket, this does incur a cost difference.  Hopefully in the future these policies will match out-of-the-box between Azure and Azure Stack so every consumer can use the basic VPN Sku to connect Azure Stack to Azure over a secure tunnel.

Reporting on Resource Group Tags in Azure

You might have seen either of Mike's blog posts on resource groups or resource tagging or just be looking to generate a report on resource group tags in Azure, if so, you're in the right place. Yesterday we were taking a look at our subscription and looking to clean up some resources.  We needed a report to review Azure resource groups and their tags.  While this relatively easy to do with PowerShell, getting a report that you can share easily was a little harder. I thought I would take some time and write a PowerShell script to generate a report utilizing ReportHTML powershell module.

Resource Group Tag Report Generated with ReportHTML

Just like most things in IT there were a few bumps in the road.  Mainly that tag names are in a hashtable and that they are case sensitive.  I wrote some code to auto-discover key names and it will prefix the key name with a number so you can find all case versions of a tag and correct them if needed. This report also includes a hyperlink to take you directly to the resource in Azure.

Once you know the tag names you want to report on you can specify them as an array and pass that in as a parameter. If you specify the Tag Names array the first two tag names will be used to generate some pie charts as shown above. EG -KeyNames=@("Owner","Solution").  By default, the report is generated in your temp directory. You can use the -ReportOutputPath param to specify an output path.  There is also a parameter for your logo URL.  It must be a small image -YouLogoHereURLString

You can view and install this report from the PowerShell Gallery here using the following Install-Script -Name run-ReportAzureResourceGroupTags

Or here is the code.

[powershell]

Param ( [parameter(Mandatory=$false,ValueFromPipeline = $true)] [Array]$KeyNames, [parameter(Mandatory=$false)] [String]$ReportOutputPath, [parameter(Mandatory=$false)] [String]$YouLogoHereURLString )

[switch]$AutoKeyName =$false $m = Get-Module -List ReportHTML if(!$m) {"Can't locate module ReportHTML. Use Install-module ReportHTML";break} else {import-module reporthtml}

if ([string]::IsNullOrEmpty($(Get-AzureRmContext).Account)) {Login-AzureRmAccount}

$RGs = Get-AzureRmResourceGroup if ($KeyNames.count -eq 0) { [switch]$AutoKeyName =$true $KeyNames = (($rgs.Tags.keys) | select -Unique) }

$SubscriptionRGs = @() foreach ($RG in $RGs) {

$myRG = [PSCustomObject]@{ ResourceGroupName = $RG.ResourceGroupName Location = $RG.Location Link = ("URL01" + "https" + "://" + "portal.azure.com/#resource" + $RG.ResourceId + "URL02" + ($RG.ResourceId.Split('/') | select -last 1) + "URL03" ) }

$i=0 foreach ($KeyName in $KeyNames) { if ($AutoKeyName) { $myRG | Add-Member -MemberType NoteProperty -Name ([string]$i + "_" + $keyname) -Value $rg.Tags.($KeyName) $i++ } else { $myRG | Add-Member -MemberType NoteProperty -Name ($keyname) -Value $rg.Tags.($KeyName) } } $SubscriptionRGs += $myRG }

$rpt = @() if ($YouLogoHereURLString -ne $null) { $rpt += Get-HTMLOpenPage -TitleText "Azure Resource Groups" -LeftLogoString $YouLogoHereURLString -RightLogoString ("https" + "://" + "azurefieldnotesblog.blob.core.windows.net/wp-ontent/2017/02/ReportHTML.png") } else { $rpt += Get-HTMLOpenPage -TitleText "Azure Resource Groups" }

if (!$AutoKeyName) { $Pie1 = $SubscriptionRGs| group $KeyNames[0] $Pie2 = $SubscriptionRGs| group $KeyNames[1]

$Pie1Object = Get-HTMLPieChartObject -ColorScheme Random $Pie2Object = Get-HTMLPieChartObject -ColorScheme Generated

$rpt += Get-HTMLContentOpen -HeaderText "Pie Charts" $rpt += Get-HTMLColumnOpen -ColumnNumber 1 -ColumnCount 2 $rpt += Get-HTMLPieChart -ChartObject $Pie1Object -DataSet $Pie1 $rpt += Get-HTMLColumnClose $rpt += Get-HTMLColumnOpen -ColumnNumber 2 -ColumnCount 2 $rpt += Get-HTMLPieChart -ChartObject $Pie2Object -DataSet $Pie2 $rpt += Get-HTMLColumnClose $rpt += Get-HTMLContentclose }

$rpt += Get-HTMLContentOpen -HeaderText "Complete List" $rpt += Get-HTMLContentdatatable -ArrayOfObjects ( $SubscriptionRGs) $rpt += Get-HTMLContentClose

$rpt += Get-HTMLClosePage

if ($ReportOutputPath -ne $null) { Save-HTMLReport -ShowReport -ReportContent $rpt -ReportName ResourceGroupTags } else { Save-HTMLReport -ShowReport -ReportContent $rpt -ReportName ResourceGroupTags -ReportPath $ReportOutputPath } [/powershell]

There is a lot more that can be done with this code so please feel free to share your ideas and code below for others. If you want to add your own logos or edit the style of the report, check out the help file here or run Get-htmlReportHelp with the module installed.  I hope you find this helpful

Enjoy

Using Multiple Azure Identities Simultaneously

Many Azure end users and developers have to deal with the challenges of holding multiple Microsoft and/or Azure Active Directory identities.  At a minimum, you might be like me and have an MSDN account as well as a 1 or more corporate accounts.  There may also be situations where you have development or test tenants and those use separate logins as well.  Another use case is when doing testing and having different users with different roles (i.e. Admin users, basic user, user with no access, etc.) In these situations, it can be painful (or at least annoying) to switch contexts when using those identities on the web since web browsers can only log you into one identity at a time when using sites such as portal.azure.com.  Have you ever gone to the Azure portal only to realize you last logged in with a different account and then you need to logout and back in with different credentials?  This is a common situation for me, and although it only takes 10 seconds or so to login with different credentials, the frequency this happens makes it quite a hassle.

One solution is to use different browsers for different identities (i.e. one login in Firefox, one login in Chrome).  This may work for 2 or 3 different identities, but it's not ideal since every browser will behave differently and may have different conventions.

The solution I use, which I will detail below is to utilize named profiles within Chrome, which allows for logging into as many identities as needed all at the same time.  No more logout/login hassle!

Step-by-step Guide

Here are the steps to add additional profiles to Chrome:

1. Within Chrome, click your named profile and select Manage people
2. Click Add Person on the dialog
3. Type a name for the profile, select an identifying icon if desired, check or uncheck creating a desktop shortcut and then save
4. Repeat for as many profiles you wish to utilize.  For example, I have my default which uses corporate production login, a secondary corporate development login as well as my MSDN/Microsoft login
5. Now when you click on your profile, you have the option of opening a new window for each profile and each window maintains it's own set of cookies, browser history, etc.
6. Here is an example of all 3 of my profiles being logged into the Azure portal all at the same time

Hope you find this useful!

Avanade Announces New Microsoft Azure Stack Solution

Great news! We can now talk about the new Azure Stack offering that many of us have been working on for many months now. Here is the official press release: http://www.prnewswire.com/news-releases/avanade-announces-new-microsoft-azure-stack-solution-300469372.html