A few months ago, I started my first EU full-time job as a cyber security engineer. This job was at one of Bitcoin's leading mining pools - BTC.com in Amsterdam, Netherlands. BTC.com holds billions of euros in cryptographic assets (cryptocurrency). These assets are typically stored as private cryptographic keys, which in simpler terms can be likened to a list of secret magic words (yes like "open sesame") that unlock a cave full of actual cash.

So how do we keep these secret words from prying eyes? Bear in mind that one day, whenever the need arises, you will need to use this secret phrase (more technically known as a private key stored as a BIP39 mnemonic phrase) to obtain the actual money or at least make a transaction for a purchase like buying pizza on Takeaway.com. The risk of keeping private keys increases exponentially with the frequency of its use. Therefore, just like the Ali Baba cave, any time you mutter the words "open sesame", you draw closer and closer to the day when some passerby (hacker) finally overhears you.





HOT WALLET / COLD WALLET PROJECT

For regular users, several techniques may be employed which range from using complex m-of-n hardware wallets, writing the secret on a sheet of paper and hiding in a safe, to simply converting the secret into a QR-code and tattooing it on a covered part of your body (keeping it truly private). Although these methods may reduce the risks to some extent, the underlying designs and / or protocols for unlocking cryptographic assets, using them, and locking them again may be subject to breaches from side channel attacks, highly intelligent malware, vulnerable software, or even accidentally getting your secret bitcoin paper eaten by your dog.

However, for large organisations like banks, credit card companies, and mining companies like BTC.com, more rigorous strategies need to be enforced. One typical high level strategy is to split all cryptographic assets into 2 (or more) parts based on the frequency of use. The less frequently used assets are kept out of reach of advanced technology - the goal here is to keep these keys away from any processor or block of code that is smart enough to be tricked into embedding a malicious program, therefore sadly, very few environments. Private keys saved like this are called cold storage schemes or wallets (any object that holds a private key). They can be likened to a savings account that is not frequently touched. The other part of the cryptographic assets which are more frequently used are also typically protected by a device called a Hardware Security Module (HSM). This is technically called a hot wallet, and can be likened to a current account.

I had just graduated out of a double master degree program in Cyber Security & Privacy, and a specialised degree in Advanced Cryptography and my first project was to develop a cold storage strategy and a design for operation of a hot wallet using an HSM provided by the renowned German HSM vendor Utimaco IS GmbH.





WHAT IS AN HSM?

An HSM is a device that performs the following functions:

Stores sensitive data in a secure environment Performs sensitive operations like encryptions, decryptions, digital signatures, and hashing at very fast rates in a secure environment, Serves as a secure source of entropy (True Random Number Generation) Deploys countermeasures against physical tampering, side channel attacks, and malicious software.

If we were to strip away all the fancy terms, we would immediately realise that most, if not all of us, actually use HSMs in our everyday lives. Most debit cards and credit cards are functional HSMs. Anytime you swipe a credit card at a supermarket, you are actually performing sensitive operations with sensitive private keys stored securely on your card. However, there are levels of security when it comes to HSMs. One of the most accredited standards is the Federal Information Processing Standard (FIPS) standard, and I was privileged to work on the latest Utimaco PCIe CSe10 and Se-series Gen2 which, at the time of writing, are rated FIPS 140-2 level 3 and FIPS 140-2 level 2 respectively.





THE ATTACK PROCESS

These devices truly gave me mixed feelings. On one hand, their speed of processing, secure environment, and True entropy made them any cryptographer's favourite toy. On the other hand, their insane defensive countermeasures against all manner of attacks made them any cryptanalyst's worst nightmare. For example, on the slightest suspicion of physical tampering, change in temperature or pressure, or any form of side-channel attack, the HSM's touchy feely sensors automatically trigger a panic alarm which immediately wipes the key RAM by flooding it multiple times with 0's and 1's.

Therefore, after designing my hot wallet HSM solution and validating it with my colleagues, I decided to test the solution by launching a series of penetrative tests (attacks) in order to compromise either the confidentiality, availability, or integrity of keys used. To my surprise, not a single attack yielded any success. All the standard cryptographic operations (like RSA, DSA, SHA, AES, etc) deployed left little or no attack surfaces. The Utimaco HSM was a complete monster. Yhupp, it was laughing at me. I could hear it in my sleep. "I will not let it win", I said to myself.

Next I began to attack the operational modes and how different users interact with keys protected by the HSM using all the supported APIs (Microsoft CSP/CNG/, JCE and PKCS#11). When I got to PKCS#11, a standard API called Cryptoki for communicating with tokens (how PKCS#11 calls an HSM), I discovered a fundamental contradiction between the original PKCS#11 documentation and the HSM library implementation. "Got you!!".

Utimaco HSMs provide 2 ways to store private keys: Internal storage and External storage. In internal storage mode, all keys in the HSM are encrypted with an internal master key (IMK) which never leaves the HSM. Back to the basics, this provides confidentiality (no unauthorised person can view the keys), integrity (keys cannot be tampered with), and availability (physical and side channel attacks to the HSM may cause a panic which wipes all keys on the device, however, there is still availability because the setup process strongly advises a backup).

If a user requires that an HSM stores more keys than its physical storage provisions, Utimaco provides the flexibility of securely storing keys on a device external to the HSM such as the server's storage device. These keys are of course encrypted as a blob with another key called the Master backup key (MBK) in the HSM. When it becomes necessary to perform operations with the keys in external storage, the HSM decrypts them on demand. It should be noted that in terms of availability, internal key storage is strictly more secure than the external key storage blob, however, both internal and external key storage have very high integrity and confidentiality guarantees.





ATTACK DETAILS

It turns out that when keys are put in external storage, the PKCS#11 Security Officer (SO) is able to view properties of the cryptographic keys in selected slot. This directly contradicts section 4.4 of the PKCS#11 Cryptographic Interface Base Specification, which states that:

“When the CKA_PRIVATE attribute is CK_TRUE, a user may not access the object until the user has been authenticated to the token.”

More precisely, it flies in the face of section 2.4 of the PKCS#11 User Guide which states that

“This version of Cryptoki recognises two token user types. One type is a Security Officer (SO). The other type is the normal user. Only the normal user is allowed access to private objects on the token, and that access is granted only after the normal user has been authenticated.”

The segregation of responsibilities or separation of powers used in PKCS#11 enforces the security of keys by reducing the attack surface to only users, thus, providing accountability and traceability.

After finding this bug, the next question I asked myself, as would any rational adversary, was "How do I turn this technical glitch into a full-fledged economically beneficial attack?" More importantly, if someone hacked into my system, how could they benefit from this? Since the externally stored keys are in a hostile environment, I attempted to manipulate the key store in order to existentially forge a key pair that could hopefully be imported into the HSM by a regular USER and then be used as a Key Encryption Key (KEK) / Wrapping Key to extract all keys in external storage, including ones marked sensitive and non-extractable. This technique is known as key conjuring. I figured if I conjured some keys from thin air and combined it with the regular class of PKCS#11 attacks, it may yield something more economically beneficial.

Fortunately / Unfortunately, Utimaco gave no leg room to such attacks. The attack surfaces were tactfully closed. Thus, revealing the actual details of the stored keys or manipulating them were at best impossible, and at worst, a job for a future date. Despite the PKCS#11 vulnerability, all keys in external storage were still protected from attacks on Confidentiality and Integrity. That leaves the last friend in the triumvirate, Availability.





ATTACK RESULTS

My final attempt was to create a regular PKCS#11 slot, create some key pairs, log in as the Security Officer, and find a way to delete all keys in external storage. "Bam!!! Got you again!". All the keys were deleted from the HSMs external storage. In that moment, it dawned on me: If there was a bank protecting credit card details with any Utimaco HSM or a cryptocurrency company protecting billions of euros worth of private keys in any Utimaco HSM, the dominant attack if applicable, would be as follows:

Attacker makes a copy of the encrypted external key storage blob since it's already in a hostile environment Attacker runs a script to delete all the externally stored keys. At this point any keys not physically backed up will be lost to the victim organisation,

However, once the victim organisation provided some money (ransom), the encrypted external storage blob would be returned by the attacker. I thought to myself, "I will call this attack the reverse ransomware attack". This attack is especially efficient because it can be executed remotely for LAN type HSMs and even possible with PCIe type HSMs since Utimaco provides a way to administer such HSMs over SSH using a pin-pad daemon.

However, I always compared cryptography and cryptanalysis to "dark magic" from Harry Potter. You can wear a white hat, be Dumbledore and experiment with dark magic all you want, however, one must always exercise self-control to avoid putting on the black hat. It was at this point, that in the spirit of responsible disclosure, I sent a documentation of the attack to Utimaco, detailing the vulnerability, and recommending an immediate patch to be rolled out with the 2 images below as proof of breach.

The amazing Utimaco tech team responded immediately and worked tirelessly, keeping in touch at every turn, to determine the extent of the bug and created a patch in no time. In addition they provided our team at BTC.com with exam vouchers to get certified as CryptoServer HSM Engineers - an exam which I immediately took, passed, and was happy to add to my certifications. They also provided us with some free accessories for the company (BTC.com). Most importantly I received my first Common Vulnerabilities and Exposure (CVE) ID from The MITRE Corporation. For cryptography and security enthusiasts, I have uploaded more technical details of this vulnerability to the CVE with ID CVE-2018-19589.





CONCLUSION

In the light of the above, it is crystal clear that just using an HSM does not automatically guarantee the security of all cryptographic keys. With some HSMs, it is not immediately apparent that backup processes do not include external key storage, since they are already outside the HSM and are considered backed up. For example, in this particular attack, if the victim keeps regular backups of all keys even in external storage, even without a patch, they reduce the risk of loss considerably when attacked. We must not substitute good security practices (like Role-Based Access Control (RBAC), fault tolerant designing, etc) for advanced technology.

More so, as a rule of thumb, you should deploy your defensive systems such that the cost of attack is at least equal to the value of the protected keys or assets. It is fundamental, but has to be said because no technology provides perfect security. Without controlling the attackers economic incentives, most security strategies are transient. There are always several avenues of attack to consider, especially, with the human factor that interacts with the device (Key Management). This is especially so when you have all your eggs in one basket (HSM).

Furthermore, many organisations also fall into the security by obscurity trap - e.g. "Utimaco does not support Multi-Sig wallets. If we develop a custom version and keep the design secret, the hackers will never understand how it works". This is a dangerous cat and mouse game, especially if the "soon to be victim" company starts developing their own crypto against the accepted standards. This kind of development ideology, a.k.a crypto-voodoo, usually never ends well. The solution is simple: If you want to keep your design a secret, at least build from standard building blocks. In the same vein, if you want to develop new crypto at least get second eyes. Ideally, one should do both.

Finally, if you are reading this article, and

you are a member of a large company that is using a Utimaco HSM in external storage mode with the PKCS#11 R2 package,

you have not patched for the latest vulnerability from December 2018,

you have not made a physical copy of your external key storage

I would like extend my sincerest condolences to you. Patch quickly before the Voldemorts of the Cyber security world get to you. Details on this CVE and how to patch it can be found here.

Special thanks to Jeroen Van Kessel, Thomas Kerin, and all the colleagues at BTC.com who urged me on in these exciting times.