Each build and release definition in TFS has a set of custom variables assigned to it. Those variables are later used as parameters to PowerShell/batch scripts, configuration file transformations, or other tasks being part of the build/release pipeline. Accessing them from a task resembles accessing process environment variables. Because of TFS detailed logging, it is quite common that values saved in variables end up in the build log in a plain text form. That is one of the reasons why Microsoft implemented secret variables.

The screenshot below presents a TFS build configuration panel, with a sample secret variable amiprotected set (notice the highlighted padlock icon on the right side of the text box):

Once the secret variable is saved, it is no longer possible to read its value from the web panel (when you click on the padlock, the text box will be cleared).

And this is how the output log looks like if we pass the secret variable to a PowerShell script and print it:

Let’s now have a look where and how the secret variables are stored.

How secret variables are stored

I started the search by examining the TFS database server. There are separate databases for each team collection. Each database groups tables into schemas with easily understandable names, such as Build, Release, or TfsWorkItemTracking. There are also few tables in the default dbo schema. Our secret variable was a part of a build definition, thus checking the [Build].[tbl_Definition] seemed like a valid approach. Here is the screenshot of the first few columns of this table:

The Variables column lists all the variables in JSON format. Unfortunately, I didn’t learn much about our secret variable (except that it is secret :)). So, I started looking for a table named “tbl_Variables” or “tbl_Secrets”, but with no success. As a last resort, I switched to the tool I always use when I don’t know what is going on: procmon. I recorded the trace while saving the secret variable value and found some interesting events in the trace:

I picked one of them and checked its call stack:

Then I needed a little trick to decode managed frames in procmon trace. The last frame pointed to the TeamFoundationStrongBoxService.AddStream method. The name of the class is quite peculiar and made me look again at the database. I quickly found the interesting tables: tbl_StrongBoxDrawer,

tbl_StrongBoxItem,

and tbl_SigningKey.

As you can see the row in the tbl_StrongBoxDrawer table references the build definition and groups the secret variables. The tbl_StrongBoxItem table holds all the values the secret variable had (including the current one) in an encrypted form. Finally, the row in the tbl_SigningKey table stores some private key, which is probably used during the encryption process.

It was time to fire up dnspy and start a more meticulous analysis. The AddStream method contains logic for adding sensitive data to the TFS database. Firstly, the secret value is encrypted using AES (with auto-generated key and a random initialization vector) and saved in a byte array. Secondly, the AES key is encrypted using the signing key and saved in another byte array. Finally, those two tables and the initialization vector are saved as the encrypted value of the secret variable. In our case, the value looked as follows:

The signing key is saved in a form of a RSA CSP blob; in our case:

After retrieving the data from the database, you may use this sample code to decrypt it:

byte[] aesKey; using (RSACryptoServiceProvider rsaCryptoServiceProvider = new RSACryptoServiceProvider(2048, new CspParameters { Flags = CspProviderFlags.UseMachineKeyStore })) { var cspBlob = "...hex string..."; rsaCryptoServiceProvider.ImportCspBlob(Hex.FromHexString(cspBlob)); var encryptedAesKey = "...hex string..."; aesKey = rsaCryptoServiceProvider.Decrypt(Hex.FromHexString(encryptedAesKey), true); } var iv = Hex.FromHexString("...hex string..."); var cipher = Hex.FromHexString("...hex string..."); using (var aes = Aes.Create()) { var transform = aes.CreateDecryptor(aesKey, iv); using (var encryptedStream = new MemoryStream(cipher)) { using (var cryptoStream = new CryptoStream(encryptedStream, transform, CryptoStreamMode.Read)) { using (var decryptedStream = new MemoryStream()) { cryptoStream.CopyTo(decryptedStream); Console.WriteLine(Hex.PrettyPrint(decryptedStream.ToArray())); } } } }

As you can see the tbl_SigningKey table contains really sensitive data and you should make sure that only authorized users can access it. I am a bit surprised that no DPAPI or other encryption mechanism is used here.

When secret variables are not so secret

In the previous paragraph, we decrypted the variables using the data stored in the database, but there is a much easier way. If you are allowed to modify the build/release pipeline, you may simply create a PowerShell task such as the one below:

As there is a space between the first letter and the rest of the string, the output won’t be masked and you will see the password in the log in plain text:

Thus, you may assume that people who can modify the build/release pipeline or who can edit the content of the script used in the pipeline are able to read the secret variables.

Extracting secret variables from the TFS memory

Finally, let’s have a look at the memory of the TFS server process (w3wp.exe) to find out what we can reveal from it. I created a full memory dump of the process and opened it in WinDbg (with SOSEX and SOS loaded). We may start by listing all the instances of the StrongBoxItemInfo:

0:000> !mx *.StrongBoxItemInfo AppDomain 0000000001eb8b50 (/LM/W3SVC/2/ROOT/tfs-1-131555514123625000) --------------------------------------------------------- module: Microsoft.TeamFoundation.Framework.Server class: Microsoft.TeamFoundation.Framework.Server.StrongBoxItemInfo module: Microsoft.TeamFoundation.Client class: Microsoft.TeamFoundation.Framework.Client.StrongBoxItemInfo … 0:000> .foreach (addr {!DumpHeap -short -mt 000007fe92560f80}) { !mdt addr } ... 0000000203419dc8 (Microsoft.TeamFoundation.Framework.Server.StrongBoxItemInfo) <drawerid>k__BackingField:(System.Guid) {c2808c4d-0c0d-43e5-b4b2-e743f5121cdd} VALTYPE (MT=000007feef467350, ADDR=0000000203419df8) <itemkind>k__BackingField:0x00 (String) (Microsoft.TeamFoundation.Framework.Server.StrongBoxItemKind) <lookupkey>k__BackingField:000000020341a1f8 (System.String) Length=24, String="9/variables/amiprotected" <signingkeyid>k__BackingField:(System.Guid) {c2808c4d-0c0d-43e5-b4b2-e743f5121cdd} VALTYPE (MT=000007feef467350, ADDR=0000000203419e08) <expirationdate>k__BackingField:(System.Nullable`1[[System.DateTime, mscorlib]]) VALTYPE (MT=000007feef4c13b8, ADDR=0000000203419e18) <credentialname>k__BackingField:NULL (System.String) <encryptedcontent>k__BackingField:000000020341a3f0 (System.Byte[], Elements: 312) <value>k__BackingField:NULL (System.String) <fileid>k__BackingField:0xffffffff (System.Int32) …

The LookupKey field contains the name of the variable. To combine it with a build/release definition we would need to find an instance of the StrongBoxDrawerName with Id equal to c2808c4d-0c0d-43e5-b4b2-e743f5121cdd. But our main point is to decrypt the value of the EncryptedContent field. For that we need to have the Signing Key (id: c2808c4d-0c0d-43e5-b4b2-e743f5121cdd). TFS keeps the recently used Signing Keys in the cache, so if we are lucky, we may still find them there. It is a bit longer process – we start by listing instances of the TeamFoundationSigningService+SigningServiceCache class:

0:000> !mx *.TeamFoundationSigningService+SigningServiceCache module: Microsoft.TeamFoundation.Framework.Server class: Microsoft.TeamFoundation.Framework.Server.TeamFoundationSigningService+SigningServiceCache 0:000> !DumpHeap -mt 000007fe932d99e0 Address MT Size ... 0000000203645ce0 000007fe932d99e0 96

And then go down the object tree to our precious private key:

Microsoft.TeamFoundation.Framework.Server.TeamFoundationSigningService+SigningServiceCache |-m_cacheData (00000002036460a0) |-m_cache (0000000203646368) |-m_dictionary (0000000203646398) 0:000> !mdt 0000000203646398 -e:2 -start:0 -count:2 … [1] (…) VALTYPE (MT=000007fe969d7160, ADDR=0000000100d6d510) key:(System.Guid) {c2808c4d-0c0d-43e5-b4b2-e743f5121cdd} VALTYPE (MT=000007feefdc7350, ADDR=0000000100d6d520) value:000000020390d440 |-Value (000000020390d410) |-<Value>k__BackingField (000000020390d3f0) |-m_Item1 (000000020390d2b0) 0:000> !mdt 000000020390d2b0 000000020390d2b0 (Microsoft.TeamFoundation.Framework.Server.SigningServiceKey) <keydata>k__BackingField:000000020390ce00 (System.Byte[], Elements: 1172) <keytype>k__BackingField:0x0 (RSAStored) (Microsoft.TeamFoundation.Framework.Server.SigningKeyType)

We could have started by searching instances of the SigningServiceKey class, but then we would need to print the GC Roots for each of them to check if its Id is the one we are looking for. I just find the top-down approach easier (still quite tedious though).

Conclusion

I hope you find this post informative. Based on what we have analyzed, we can say that it is possible to store sensitive data in TFS secret variables securely , but one needs to make sure that:

only authorized users can modify build/release pipeline (including the content of the tasks which use the secret variables)

only authorized users can access the tbl_SigningKey table in the TFS database

Alternatively, a carefully configured TFS agent may perform all the sensitive tasks, using a Key Vault or some local DPAPI encrypted blob as a secret store.