When we need to deploy an application to Azure from VSTS (Visual Studio Team Services), we use the Azure tasks prepared by Microsoft. These tasks require a contributor account in Azure AD to make changes to your subscription. As this account is not a regular user account but an application account we call it a Service Principal. A very basic build pipeline might look as follows:

The “Azure App Service Deploy” task is an example of a task that will use a Service Principal account to update your App Service in Azure. VSTS makes it easy to create the Service Principal account; it also automatically assigns a contributor role in your subscription to this newly created account. When you want to have full control over your Azure AD you may manually create an App Registration (another name for the Service Principal) in the portal and give it the required rights. You will also need a key to authenticate the service in Azure:

In the next step, you create a new Azure Resource Manager Service Endpoint, providing all the collected information:

Once you add the Service Endpoint, you won’t be able to modify or retrieve its credentials. This might be problematic if you don’t have access to the Azure AD, and you need to create a Service Endpoint for a different subscription but using the same Service Principal account. In this post, I will present you a way to get hold of the Service Principal credentials using the build pipeline only. To make the things harder, we will use the Hosted Agent – one provided by Microsoft, with no access through RDP.

A Memory Dump in the Build Pipeline

The first place we usually check when looking for process secret data is the process memory. Here it will be no different. As we don’t have direct access to the target system (and the process), we will use the build pipeline to download procdump, collect a dump, and publish it as a built artifact. We also need to have an Azure task in our pipeline (otherwise VSTS won’t use any Service Principal credentials). The Azure Powershell task fulfills all our requirements. It runs in a context of a Service Principal and executes provided PowerShell code. Our PowerShell script is concise:

Invoke-WebRequest -UseBasicParsing ` -Uri https://live.sysinternals.com/procdump.exe ` -OutFile $env:BUILD_SOURCESDIRECTORY\procdump.exe & "$env:BUILD_SOURCESDIRECTORY\procdump.exe" -accepteula ` -o -ma $pid $env:BUILD_SOURCESDIRECTORY\ps.dmp

The BUILD_SOURCESDIRECTORY is a special environment variable which VSTS sets to the path of the directory containing application source code. Our build pipeline in VSTS might look as follows:

Below, you may see a VSTS log of the PowerShell task execution. The highlighted line is the one when VSTS creates the Azure context (using the Add-AzureRMAccount cmdlet).

##[section]Starting: Azure PowerShell script: InlineScript 2017-09-21T22:11:20.8230853Z ============================================================================== Task : Azure PowerShell Description : Run a PowerShell script within an Azure environment Version : 2.0.2 Author : Microsoft Corporation Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=613749) ============================================================================== ##[command]Import-Module -Name C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ResourceManager\AzureResourceManager\AzureRM.Profile\AzureRM.Profile.psd1 -Global ##[command]Import-Module -Name C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\Storage\Azure.Storage\Azure.Storage.psd1 -Global ##[command]Add-AzureRMAccount -ServicePrincipal -Tenant ******** -Credential System.Management.Automation.PSCredential -EnvironmentName AzureCloud ##[command]Select-AzureRMSubscription -SubscriptionId <stripped> -TenantId ******** ##[command]& 'd:\a\_temp\f3289bb3-6ee3-4f5f-bb32-b5bf080fd932.ps1' ProcDump v9.0 - Sysinternals process dump utility Copyright (C) 2009-2017 Mark Russinovich and Andrew Richards Sysinternals - www.sysinternals.com [22:11:37] Dump 1 initiated: d:\a\1\s\ps.dmp [22:11:37] Dump 1 writing: Estimated dump file size is 380 MB. [22:11:48] Dump 1 complete: 381 MB written in 11.4 seconds [22:11:49] Dump count reached. ##[section]Finishing: Azure PowerShell script: InlineScript

Looking for Secrets in the Build Process Memory

After a successfull build, the process memory dump is available as a build artifact. We may now download it and open in WinDbg. Both Login-AzureRMAccount and Add-AzureRMAccount use the PSAzureContext class from the Microsoft.Azure.Commands.Profile assembly. Let’s find it in the memory:

0:000> .loadby sos clr 0:000> !DumpHeap -type Microsoft.Azure.Commands.Profile.Models.PSAzureContext Address MT Size 000000f166132b80 00007ff7e2100038 56 000000f166270010 00007ff7e2100038 56 Statistics: MT Count TotalSize Class Name 00007ff7e2100038 2 112 Microsoft.Azure.Commands.Profile.Models.PSAzureContext Total 2 objects Fragmented blocks larger than 0.5 MB: Addr Size Followed by 000000f1667592b8 19.6MB 000000f167afb408 System.Byte[] 0:000> !mdt 000000f166132b80 000000f166132b80 (Microsoft.Azure.Commands.Profile.Models.PSAzureContext) <Account>k__BackingField:000000f166132bb8 (Microsoft.Azure.Commands.Profile.Models.PSAzureRmAccount) <Environment>k__BackingField:000000f166132fb8 (Microsoft.Azure.Commands.Profile.Models.PSAzureEnvironment) <Subscription>k__BackingField:000000f166133060 (Microsoft.Azure.Commands.Profile.Models.PSAzureSubscription) <Tenant>k__BackingField:000000f166133108 (Microsoft.Azure.Commands.Profile.Models.PSAzureTenant) <TokenCache>k__BackingField:000000f1661324a0 (System.Byte[], Elements: 1597)

The TokenCache property seems interesting:

0:000> db 000000f1661324a0+0x10 L100 000000f1`661324b0 02 00 00 00 01 00 00 00-99 01 68 74 74 70 73 3a ..........https: 000000f1`661324c0 2f 2f 6c 6f 67 69 6e 2e-6d 69 63 72 6f 73 6f 66 //login.microsof 000000f1`661324d0 74 6f 6e 6c 69 6e 65 2e-63 6f 6d 2f 34 38 30 34 tonline.com/4804 ... 000000f1`66132500 2f 3a 3a 3a 68 74 74 70-73 3a 2f 2f 6d 61 6e 61 /:::https://mana 000000f1`66132510 67 65 6d 65 6e 74 2e 63-6f 72 65 2e 77 69 6e 64 gement.core.wind 000000f1`66132520 6f 77 73 2e 6e 65 74 2f-3a 3a 3a 34 63 39 31 66 ows.net/:::4c91f 000000f1`66132530 63 63 30 2d 39 35 34 32-2d 34 38 62 61 2d 38 63 cc0-9542-48ba-8c 000000f1`66132540 64 61 2d 31 38 38 33 32-34 63 38 65 62 37 66 3a da-188324c8eb7f: 000000f1`66132550 3a 3a 31 98 0b 7b 22 41-63 63 65 73 73 54 6f 6b ::1..{"AccessTok 000000f1`66132560 65 6e 22 3a 22 65 79 4a-30 65 58 41 69 4f 69 4a en":"eyJ0eXAiOiJ 000000f1`66132570 4b 56 31 51 69 4c 43 4a-68 62 47 63 69 4f 69 4a KV1QiLCJhbGciOiJ 000000f1`66132580 53 55 7a 49 31 4e 69 49-73 49 6e 67 31 64 43 49 SUzI1NiIsIng1dCI 000000f1`66132590 36 49 6b 68 49 51 6e 6c-4c 56 53 30 77 52 48 46 6IkhIQnlLVS0wRHF 000000f1`661325a0 42 63 55 31 61 61 44 5a-61 52 6c 42 6b 4d 6c 5a BcU1aaDZaRlBkMlZ

The ASCII column shows that we found the Bearer token used to authenticate requests to the Azure API. But it is not the Service Principal password. After examining few more classes and finding nothing interesting it is time to try a brute force method. I know that Azure generates the Service Principal password in the form of base64 text usually longer than 40 characters. Let’s dump all managed strings longer than 40 characters from the .NET heap (using an excellent !strings command from the SOSEX extension):

0:000> .load sosex 0:000> !strings -n:40 Address Gen Length Value --------------------------------------- 000000f166130328 0 56 Switch.System.Runtime.Serialization.DoNotUseTimeZoneInfo 000000f1661309f0 0 1432 {"AccessToken":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkhIQnlLVS0wRHFBcU1aaDZaRlBkMlZXYU90ZyIsImtpZCI6IkhIQnlLVS0wRHFBcU1a... 000000f166133c08 0 40 AddAzureRMAccountCommand end processing. 000000f166133db0 0 54 10:11:32 PM - AddAzureRMAccountCommand end processing. 000000f166133e68 0 40 AddAzureRMAccountCommand end processing. 000000f166134010 0 54 10:11:32 PM - AddAzureRMAccountCommand end processing. ... 000000f174b13fb8 LOH 52296 <?xml version="1.0" encoding="utf-8" ?> <Configuration> <ViewDefinitions> <View> <Name>Microsoft.Azure.Commands.... --------------------------------------- 7965 matching strings

7965 lines are too many to analyze manually. Therefore, I will copy the output of the !strings command to a file and will scan it for valid base64 strings using PowerShell:

Get-Content .\ps.dmp.txt | ` % { $_ -split " " } | ` ? { $_ -and ($_.Length -gt 40) } | ` % { try { ` $null = [Convert]::FromBase64String($_); ` Write-Host $_ ` } catch { } }

This command produces much less text, and it is relatively easy to spot the interesting lines (highlighted):

ABvh6MRNCdsJqHJKL3C6/xlwUa8LvDeAXuJW9eAtA3U= AzureRmSqlServerActiveDirectoryAdministrator AzureRmSqlServerBackupLongTermRetentionVault AzureRmSqlServerActiveDirectoryAdministrator AzureRmSqlServerBackupLongTermRetentionVault AzureRmSqlServerActiveDirectoryAdministrator AzureRmSqlServerActiveDirectoryAdministrator AzureRmSqlServerBackupLongTermRetentionVault AzureRmSqlDatabaseExecuteIndexRecommendation AzureRmSqlDatabaseExecuteIndexRecommendation PropagateExceptionsToEnclosingStatementBlock NormalizeRelativePathUnauthorizedAccessError ParameterArgumentValidationErrorEmptyArrayNotAllowed 4ESBcfx2mEo/o3U7QGxoTP2r5HzX+nv8you9ChT7JpI= AlreadyExistingUserSpecifiedPropertyNoExpand AzureKeyVaultCertificateAdministratorDetails AzureKeyVaultCertificateAdministratorDetails ...

We could stop here and test the strings against Azure API to check which of them works. However, I would stay in the WinDbg window a bit longer and extract some more information about the highlighted strings. Their addresses in memory are as follows:

000000f166240168 0 44 ABvh6MRNCdsJqHJKL3C6/xlwUa8LvDeAXuJW9eAtA3U= 000000f165415e38 2 44 4ESBcfx2mEo/o3U7QGxoTP2r5HzX+nv8you9ChT7JpI=

And their GC roots:

0:000> !GCRoot 000000f166240168 Found 0 unique roots (run '!GCRoot -all' to see all roots). 0:000> !GCRoot 000000f165415e38 Finalizer Queue: 000000f16541c130 -> 000000f16541c130 System.Management.Automation.CommandProcessor -> 000000f1653f9b88 System.Management.Automation.SessionStateScope -> 000000f1653f98d8 System.Management.Automation.MutableTuple`32[[System.Object, mscorlib],[System.Object[], mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Management.Automation.PSScriptCmdlet, System.Management.Automation],[System.Management.Automation.PSBoundParametersDictionary, System.Management.Automation],[System.Management.Automation.InvocationInfo, System.Management.Automation],[System.String, mscorlib],[System.String, mscorlib],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.SwitchParameter, System.Management.Automation],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.ConfirmImpact, System.Management.Automation],[System.String, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation]] -> 000000f1654192e0 System.Management.Automation.PSObject -> 000000f165419410 System.Management.Automation.PSMemberInfoInternalCollection`1[[System.Management.Automation.PSMemberInfo, System.Management.Automation]] -> 000000f165419430 System.Collections.Specialized.OrderedDictionary -> 000000f1654194c0 System.Collections.ArrayList -> 000000f165419508 System.Object[] -> 000000f165419818 System.Collections.DictionaryEntry -> 000000f165419728 System.Management.Automation.PSNoteProperty -> 000000f165415fb0 System.Management.Automation.PSObject -> 000000f1654166c0 System.Management.Automation.PSMemberInfoInternalCollection`1[[System.Management.Automation.PSMemberInfo, System.Management.Automation]] -> 000000f1654166e0 System.Collections.Specialized.OrderedDictionary -> 000000f1654167d0 System.Collections.ArrayList -> 000000f165416818 System.Object[] -> 000000f1654167f8 System.Collections.DictionaryEntry -> 000000f165416690 System.Management.Automation.PSNoteProperty -> 000000f165416088 System.Management.Automation.PSObject -> 000000f1654161a8 System.Management.Automation.PSMemberInfoInternalCollection`1[[System.Management.Automation.PSMemberInfo, System.Management.Automation]] -> 000000f1654161c8 System.Collections.Specialized.OrderedDictionary -> 000000f165416258 System.Collections.ArrayList -> 000000f1654162a0 System.Object[] -> 000000f1654163c8 System.Collections.DictionaryEntry -> 000000f165416398 System.Management.Automation.PSNoteProperty -> 000000f165415e38 System.String

As you can see the GC root to our second string exists only on the finalizer queue, which means that if only the memory pressure on the machine was higher, our precious password would be gone. It is just something you need to keep in mind when playing with the process memory.