Rotating Event Hubs RP External Certificate on Azure Stack Hub

I have been testing the Event Hubs public preview release for Azure Stack Hub, looking at the install process and what kind of actions an Operator would need to do to keep things running. One of the important ones for me, are rotating secrets / certificates. If your certificates expire, you won’t be able to access the RP, hence the importance.

If you check the current documentation for rotating secrets , it is the generic instructions for rotating external certificates for the Azure Stack Hub Stamp. I expect that this will be corrected in the near future, but until then , how do you do go about it for the Public Preview?

Firstly, you need the latest version of the Azure Stack Hub PowerShell modules:

• AzureRM compatible module

• Az compatible module

Then you need to connect to your Azure Stack Hub Admin environment. Use the steps detailed in the following article : https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-powershell-configure-admin?view=azs-2002

(Remember if using the Az Module to rename the Commands per https://docs.microsoft.com/en-us/azure-stack/operator/powershell-install-az-module?view=azs-2002#7-use-the-az-module)

Copy the Event Hubs pfx file to a local directory and run the following script (The example is using theAz module)t:

$ProductId = 'microsoft.eventhub'
$productVersion = (Get-AzsProductDeployment -ProductId microsoft.eventhub).properties.deployment.version
$PackageId = ('{0}.{1}' -f $ProductId, $productVersion)
$packageSecret = ((Get-AzsProductSecret -PackageId $PackageId).value.name).split('/')[2]
$certPath = 'C:\AzsCerts\EventHubs\cert.pfx'

$pfxPassword = (ConvertTo-SecureString '<pfxPassword>' -AsPlainText -Force)
Set-AzsProductSecret -PackageId $PackageId -SecretName $packageSecret -PfxFileName $certPath  -PfxPassword $pfxPassword -Force -Verbose

Invoke-AzsProductRotateSecretsAction -ProductId $ProductId

Modify the $certPath variable and <pfxPassword> to match what you have set and then run the script.

The process will take quite a long time to complete. Whilst the operation is taking place, you will receive the status of the command.

View fullsize

If you choose to stop the CmdLet/script, the process will continue in the background. You can check the status at anytime by running the following:

(Get-AzsProductDeployment -ProductId microsoft.eventhub).properties

You should see something like this when the process is still running:

View fullsize

… and when successfully finished:

View fullsize

Hope that helps until the official documentation is released!

Article updated 16 July 2020 with an updated method to obtain the secret name, provided by @kongou_ae - Thanks!

ARM Limits on Azure Stack Hub

I came across an interesting error when I was doing some maintenance work on an Azure Stack Hub stamp. I was updating App Service to version 2020 Q2 and also installing the Event Hubs Public Release preview. I got the following error message:

View fullsize

‘Number of read requests for subscription ‘‘ exceeded the limit of ‘15000’ for time interval ‘01:00:00’. Please try again after ‘‘ seconds. (Code: SubscriptionRequestsThrottled)’

I’d certainly not come across this before and was intrigued to see if I could find out any more information about this. I checked the Azure Stack Hub documentation and couldn’t find any reference to this limit, nor could I find anyone else who had come across this before.

What I did find related to Azure, so I thought it should be relevant. Details of the limits applied can be found in the article Throttling Resource Manager requests.

It turns out that Azure Stack Hub also applies limits to requests via ARM, although the limits are slightly different. Whereas for Azure the Read limit for a subscription is set to 12000, for Azure Stack Hub, it is set to 15000, per the error message I received.

I don’t know what the write / delete limits are, but assume they are similar to the ones set for Azure, give or take a thousand :)

So, how can you check how many read operations are remaining before hitting the limit?

The Throttling Resource Manager requests article details the methods you can use, by inspecting headers, or by using a script that was referenced in the doc:  Check Resource Manager Limits for a Subscription.

This script doesn’t quite work for Azure Stack Hub, also, it doesn’t work if you’re the the AZ PowerShell Module, so I made some modifications and made them available here.

There are two versions, one for AzureRM and the other for Az PowerShell modules

Get-AzArmLimits.ps1

Get-AzureRMArmLimits.ps1

You need to be logged in to the Azure environment you want to check, but it will work for Azure and Azure Stack Hub

Here are some examples of the output:

View fullsize

Using AzureRM module; Azure

View fullsize

Using AzureRM module; Azure Stack Hub

View fullsize

Using Az module; Azure Stack Hub

Azure Stack Hub: When you can't view/add permissions in a tenant subscription from the portal

I have noticed on a few occasions that for a tenant subscription hosted in an Azure Stack region that I am either unable to view the IAM permissions, or add a user/service principal/group for the subscription. I have needed to do this to assign a service principal as a contributor to a subscription so that I can deploy a Kubernetes cluster via AKSE.

Typically for viewing the permissions, clicking on the Refresh button does the trick. More problematic is adding permissions via the portal. Doing so renders a screen like below:

View fullsize

As highlighted, the animated ‘dots’ show the blade constantly trying to retrieve the data but can’t. It is actually a known issue and is highlighted in the release notes.

The remediation it offers is to use PowerShell to verify the permissions, and gives a link to the Get-AzureRmRoleAssignment CmdLet. Not helpful if you want to set permissions, so here’s a step-by-step description of what you’ll need to do.

Pre-Reqs:

• Your user account is the owner of the subscriptions when they were set up.

• You have the correct PowerShell for Azure Stack Hub installed

For the example shown, I am using Azure AD identities, I have more than one tenant subscription assigned to my user account and I am adding a service principal. At the end of the post, I will show the commands for adding a group or a user .

Step-By-Step

From PowerShell, connect to the tenant subscription you want to change the permissions on. The documentation is for connecting as an operator, so here’s how to do it as a user:

$Region = '<yourRegion>'
$FQDN = '<yourFQDN>'
$AADTenantName = '<yourAADTenantName>'
$envName = "AzureStack$region" 

 # Register an Azure Resource Manager environment that targets your Azure Stack instance. Get your Azure Resource Manager endpoint value from your service provider.
Add-AzureRMEnvironment -Name $envName -ArmEndpoint "https://management.$Region.$FQDN" `
    -AzureKeyVaultDnsSuffix vault.$Region.$FQDN `
    -AzureKeyVaultServiceEndpointResourceId https://vault.$Region.$FQDN

# Set your tenant name.
$AuthEndpoint = (Get-AzureRmEnvironment -Name $envName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]

# After signing in to your environment, Azure Stack cmdlets
# can be easily targeted at your Azure Stack instance.
Add-AzureRmAccount -EnvironmentName $envName -TenantId $TenantId

As I have more than one subscription, let’s get a list so we can select the correct one:

Get-AzureRmSubscription | ft Name

I got the following output:

View fullsize

• I want to modify the AKSTest subscription:

#Set the context
$ctx = set-azurermcontext -Subscription AKSTest
$ctx

Running that command, I get :

View fullsize

Cool; I’ve got the correct subscription now, so let’s list the permissions assigned to it:

Get-AzureRmRoleAssignment

View fullsize

OK, so those are the standard permissions assigned when the subscription is created. Let’s add a service principal.

First, let’s get the service principal object

Get-AzureRmADServicePrincipal | ? {$_.DisplayName -like '*something*'} |ft DisplayName

I want to assign AKSEngineDemo

$spn = Get-AzureRmADServicePrincipal -SearchString AKSEngineDemo

View fullsize

Lets assign the service principal to the tenant subscription:

New-AzureRmRoleAssignment -ObjectId $spn.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"

View fullsize

When I take a look in the portal now (after doing a refresh :) ), I see the following:

View fullsize

Here’s the commands for adding an Azure AD User:

#Find the User object you want to add
Get-AzureRmADUser | ? {$_.DisplayName -like '*something*'} |ft DisplayName

#Assign the object using the displayname of the user
$ADUser = Get-AzureRmADUser -SearchString <UserName>
New-AzureRmRoleAssignment -ObjectId $ADUser.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"

Finally, how to add an Azure AD Group:

#Find the Group object you want to add
Get-AzureRmADGroup | ? {$_.DisplayName -like '*something*'} |ft DisplayName

#Assign the object using the displayname of the user
$ADGroup = Get-AzureRmADGroup -SearchString <GroupName>
New-AzureRmRoleAssignment -ObjectId $ADGroup.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"

Adding external load balancer IP address to an existing Kubernetes Service to update <pending>

This is one of those little things. I’m sure the veteran Kube admins out there are saying “well of course” but sometimes trying to sort through forums of Q&A on the internet is problematic even with the help of your favorite search engine. I thought I’d add one more page that might turn up and help someone else.

So you have created a self-managed K8s cluster and have you load balancer setup for your service and you have a bunch of <pending> when you look at your services. You can simply patch your service.

kubectl patch svc <svc-name> -n <namespace> -p '{"spec": {"type": "LoadBalancer", "externalIPs":["x.x.x.x"]}}