I recently deployed new Azure Stack integrated system, and despite a few of the usual issues I was expecting (network integration!!!), everything went well up until the point of me needing to syndicate items in to the Marketplace via the admin portal. https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-download-azure-marketplace-item?view=azs-1908#connected-scenario
I could download some of the smaller items successfully, such as the VM extensions, but those that were larger, failed.
Initially, I thought it was a transient network issue, so deleted the failed items from the Marketplace and re-attempted the download, but I had the same problem re-occurred.
Because the admin portal only gives 3 states for Marketplace item (Downloading, Downloaded, or Failed), I wanted to try and determine where the problem lay before calling Support.
To do this, I used the Azure Stack Tools, more specifically: Export-AzSOfflineMarketplaceItem CMDLet. https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-download-azure-marketplace-item?view=azs-1908#disconnected-or-a-partially-connected-scenario. By running in PowerShell, I felt I had more chance in figuring out what was going on by using Verbose logging.
To get more verbose information, I used the azcopy option. When I first started investigating the problem, the version of the tools required AZCopy v7.3/8.1. This required installation via an MSI. However, earlier last week, a new version of the tools was released which uses AZCopy v10 https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10. I much prefer Microsoft’s approach to new releases of this tool as it is a single file, does not require installation, and therefore does not require admin rights.
Here’s a little wrapper script based on that in the documentation to download the Azure Stack tools. It also retrieves azcopy v10 and places it into the tools directory:
# Change directory to the root directory.
cd \
# Download the tools archive.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
invoke-webrequest `
https://github.com/Azure/AzureStack-Tools/archive/master.zip `
-OutFile master.zip
# Expand the downloaded files.
expand-archive master.zip `
-DestinationPath . `
-Force
# Change to the tools directory.
cd AzureStack-Tools-master
# Download azcopy v10
invoke-webrequest `
https://aka.ms/downloadazcopy-v10-windows `
-OutFile azcopy.zip
# Expand the downloaded files.
expand-archive azcopy.zip `
-DestinationPath . `
-Force
Here’s a script to download marketplace items from Public Azure and then upload it to your stamp:
$Region = "local" # For ASDK, this is local, change to match your region for integrated deployments
$FQDN = "azurestack.external" # For ASDK, this is azurestack.external
$AzSEnvironmentName = "AzureStackadmin" # Change this if you want more than one Azure Stack registraion on your system
$RegistrationRG = 'AzureStack' # Get this from the AdminPortal / Dashboard / Region / Properties / REGISTRATION RESOURCE GROUP if unsure
$RegistrationUserName = '<user>@<your tenant>.onmicrosoft.com' # User with rights to the Registraion Resource Group
$OperatorUserName = '<operator user name>@<your tenant>' # Username of Operator that is contributor/owner or Default Subscriber Subscription
$OperatorAADTenantName = '<Operator tenant name>' # the AAD tenant Name e.g. <mytenant>.onmicrosoft.com
$mktPlcFolder = "D:\Mkt" # Directory to store downloaded content. Must exist before running
$azcopypath = "C:\AzureStack-Tools-master\azcopy_windows_amd64_10.2.1\azcopy.exe"
# Register an Azure Resource Manager environment that targets your Azure Stack instance. Get your Azure Resource Manager endpoint value from your service provider.
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "https://adminmanagement.$Region.$FQDN" `
-AzureKeyVaultDnsSuffix adminvault.$Region.$FQDN `
-AzureKeyVaultServiceEndpointResourceId https://adminvault.$Region.$FQDN
# Set your tenant name
$AuthEndpoint = (Get-AzureRmEnvironment -Name $AzSEnvironmentName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($OperatorAADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]
#Get the credentials for an identity with permission to the subscription that the stamp is registered to. sed to download Marketplace item from azure
$RegistrationCreds = get-credential $RegistrationUserName -Message "Enter Azure Subscription Credentials"
# Get the credentials for an identity that has contributor/owner rights to the Default Provider Subscription. used to upload Marketplace item to the Stamp
$operatorCreds = Get-Credential -Message "Enter the azure stack operator credential:" -UserName $OperatorUserName
# first, connect to Public Azure...
Add-AzureRmAccount -Credential $RegistrationCreds -Environment AzureCloud
Get-AzureRmSubscription | Select-AzureRmSubscription
cd C:\AzureStack-Tools-master
Import-Module .\Syndication\AzureStack.MarketplaceSyndication.psm1
# Download the item. You will be prompted to choose from an Out-Grid window...
Export-AzSOfflineMarketplaceItem -Destination $mktPlcFolder -resourceGroup $RegistrationRG -AzCopyDownloadThreads 16 -azCopyPath $azcopypath
# Once the download has finished, swich to the Stack admin environment and upload the Marketplace item.
Add-AzureRmAccount -EnvironmentName $AzSEnvironmentName -TenantId $TenantId -Credential $operatorCreds
Import-AzSOfflineMarketplaceItem -origin $mktPlcFolder -AzsCredential $operatorCreds
When you run the script, you’ll be prompted for credentials and then which items you want to download. You can choose multiple items (CTRL and select), but I advise selecting 1 item at a time, as you will be prompted to accept the legal terms and conditions, as well as selecting the download method (azcopy) per choice. You may miss the prompts for subsequent items if the first download takes a while.
Once the selection has been made, you’ll see the following:
Select ‘y’ for both questions, and the download should start.
For the environment I was operating in, downloading via the internet took a while, as there were QoS rules applied.
After a while, I saw the following error:
OK, so I seemed to have the same problem via the portal and with the PowerShell tools. As I was using AZCopy for the download, there are logs, so that was the first port of call for me. The logs are stored in the following directory:
%USERPROFILE%\.azcopy
So I navigated there and opened the latest log file. I found the following towards the end:
I’ve highlighted the key entry that pointed me towards the problem:
‘…the MD5 hash of the data, as we received it, did not match the expected value, as found in the Blob/File Service. This means there is a data integrity error OR another tool has failed to keep the stored hash up to date.’
OK, so I thought this could have been a problem with some inline firewall or web proxy, but then I could open the smaller items, such as the icons associated with the marketplace item, or the manifest json files.
To prove if it was an issue with the environment I was operating in or not, I decided to spin up a Windows Server 2016 VM in Azure and attach a 200GB data disk and run through the same process as above. Thankfully, the downloads were a lot quicker, as would be expected given I was using the Azure Network fabric, but I found that the download failed again, and I saw the same error regarding the MD5 hash. Weird!
I decided to see if there was a way I could circumvent the MD5 hash check to see if I could at least complete the download and get something into the Marketplace so I could test if the item worked or not. This capability is not native within the Export-AzSOfflineMarketplaceItem, but there is a parameter within azcopy to do this: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-files#download-files. If I could add --check-md5=NoCheck or --check-md5=LogOnly to the azcopy command within Export-AzSOfflineMarketplaceItem, I could at least test it.
This is actually quite simple to do. by editing C:\AzureStack-Tools-master\Syndication\AzureStack.MarketplaceSyndication.psm1 (replace C:\AzureStack-Tools-master to match path where you have the tools), modify lines 591 & 593 (as of the current version at time of writing) to read:
& $azCopyPath copy $Source $tmpDestination --recursive --check-md5=LogOnly
It should look like this:
If you already had the AzureStack.MarketplaceSyndication.psm1 loaded, simply close your PowerShell session.
Once I made the changes, I retried the process again, and this time, SUCCESS!
The Marketplace images downloaded and I was able to import them to my stamp, with no issue. I was able to deploy VM’s using these images with no problem.
I’m not sure if there’s an underlying problem with azcopy and the Marketplace content, but at least I managed to figure out a workaround that doesn’t appear to have any detrimental effects, so hopefully it can help out someone else who might have a similar problem.
