The Basics

Merkle trees are widespread in distributed systems — including blockchains. To understand what they are, we need to understand the basic motivation behind them. In environments of limited trust between network computers, as is the case in blockchains, a main concern is verifying that data has not been corrupted or tampered with between or during transmissions over a network. One way to confirm received data is identical to the original is by the use of cryptographic hash functions.

Cryptographic Hash Functions

Cryptographic hash functions allow us to compute a short digest, called a hash code, from data files of any size. A hash function receives data of arbitrary length as its input, and outputs a hash code. Inputs may be very large files such as heavy media files, or tiny data entries, such as a password. It does not matter what size the file or data was, a hash functions output is always the same size.

A hash code can be thought of as a unique fingerprint of the input data. By comparing two fingerprints we can detect if both hash codes belong to the same input data.

Let’s look at an example using a widely used hash function called MD5.

MD5 ("Imagine there's no heaven") = 683fb6357275ccf96686850d01f81bb8 MD5 ("Imagine there’s no heaven") = 505a36bc6e1aff57092a4541af547a37 MD5 (yellowpaper_ver_e94ebda.pdf) = b5312a1b11d3f9d0d9c426e84235e3b8

Above are MD5 hash codes for three different inputs. The first two phrases are almost identical. If you look closely, you’ll notice in the first phrase a single quote instead of the apostrophe found in the second phrase. This small difference in input, practically unnoticeable to the human eye, translates into a dramatic difference in the resulting hash code. This is an important characteristic of hash functions. The third example is a hash code of the Ethereum yellow paper document in pdf format. It is included here simply to show how size and format of our input data does not affect the hash code form or size.

In fact, we cannot say much about the input just by looking at its hash code. Given just a hash code its practically impossible to arrive back at any input yielding exactly the same hash code. This is why cryptographic hash functions are said to be one-way functions: it’s easy to find the hash code for any data, but very hard to find any data for a given hash code if the original input is not available. This characteristic makes it impossible to fake an input with a hash code similar to some other input. What this means is that identical hash codes usually mean identical inputs.

Let’s look at an example of how these properties may be used. When a large file needs to be distributed to many clients, it is sometimes distributed through multiple servers for quick or parallel transmission. Often, these servers are not known to the client and cannot be authenticated. However, if a trusted hash code of the real data can be obtained from a trusted source, any version of the file, regardless of where it was obtained, can be corroborated by reapplying the hash function and checking to see if both hash codes match.

figure 1

This method works well when two requirements are satisfied:

There is a secure channel over which we can communicate hash codes The data is consumed entirely or not at all. Clients typically download the full data first, and trust it only after its hash code is computed and compared with the expected hash code

Coming back to thin clients and ledger state in blockchains, we satisfy the first requirement by means of a consensus algorithm. For each block, a hash code digest of the ledger state is included in the corresponding block header. A consensus protocol lets any client validate block headers to obtain trusted hash codes for the current ledger state. But thin clients don’t normally maintain a state DB for the entire state ledger.

Which leads us to a difficulty satisfying the second requirement. When a thin client accesses ledger state it usually requests a specific account balance, for instance. It has no use for the entire blockchain state, nor can it spare the resources to track and maintain a full state DB. Rather, at any point in time it typically is interested in a very small subset of ledger state. As we’ve seen, a single hash code can be validated only against the entire data. So how can a thin client corroborate any claim about a particular accounts balance received over the network? This is the starting point of our Merkle tree story.

Suppose we tried to overcome this problem by dividing the ledger into smaller blocks and computing a separate hash code for every such segment in advance. Without a Merkle Tree, we would soon run into the problem of having to clog our block headers with too many hash codes. Since we don’t know which chunk will be needed later on, we would attach all of our hash codes to each block header. This, however, defeats the whole point of sending lightweight block headers in the first place: We would now have too many hash codes in each block header. If we wanted to go all the way to the finest granularity — a multitude of hash codes may be required similar in size to the full state database. So we would end up in the same place, having to send and store large quantities of data we don’t really plan on using. Merkle trees solve this problem while requiring only a single hash code for each revision of our ledger state.

Merkle Trees

Merkle trees are an extension of the previous idea. They require us to send only one hash code for each revision of our ledger state via a trusted channel, as in the naive method in figure 1. But, unlike the previous method, in order to verify any claim about a single entry in our state dataset we only need to send the client a message with a space requirement of O(logN), where N is the number of entries in our ledger, instead of the entire data as in with the naive approach. We will see how this is done in detail first for a non-blockchain use case.

Basic Merkle Tree

In a merkle tree, the master hash code is called the Merkle Root. Computing the merkle root is a bit more complicated than just providing the entire state to the hash function in just one go. It has multiple stages.

Let’s assume we have a way to split the data into fine-grained segments, and that each segment is addressed by a number. We could then order all our segments in a row, and compute each one’s hash code. To start with a simple example, let’s say our data volume is V, and we break it down to four segments V1..V4.

If we took the hash codes h(V1)..h(V4) for each segment and grouped each pair of adjacent hash codes we could regard every pair of hash codes in this row as a new data segment and compute a single hash code from their union. The result will be a second row of hash codes, half the length of the first row. This process of stacking rows of hash codes one on top of the other can continue until we get a pyramid of hash codes. At the top of this pyramid we have the root hash, the hash of all hashes, marked H7 in the example below. This simple example shows a merkle tree with four leaf nodes. Clearly, if any of our original data elements were different, the Root hash, H7, would also change as a result. The important thing here is that all our fine-grained data segments (V1..V4) determine the root hash together.

To stress the difference between the two methods for computing a hash code, let’s look at how each hash code is computed (see figure 2):

Simple hash for the data set V1..V4 = h(V1,V2,V3,V4) Merkle tree hash for the data set V1..V4 = h(h(h(V1),h(V2)),h(h(V3),h(V4)))

To obtain the root hash in a merkle tree we apply our hash function repeatedly over our data segments and their hash codes. But besides the merkle root, along the way we also compute many more intermediate hash codes. These intermediate hash codes will help us later on, but you may already intuitively appreciate that every one of these intermediate codes, can succinctly represent an entire subtree of values when we come to calculate the top hash. For instance, if we knew H5 and H6 we could easily compute H7 without having to know anything else about any of the leaf values.

figure 2. Mekrle tree for the dataset V = (V1, V2, V3, V4). Each node is labeled with a Hash code H1..H7

As before, we can assume all clients had previously obtained H7 via a secure channel before making any state queries to server machines. We further assume that all server machines have access to the full state database including the merkle tree structure above it.

When a client asks an untrusted server for the value if V2, for instance, the latter will now be able to attach a short proof to its reply which will enable clients to verify the query result against the root node’s hash value, H7. Unlike the case with naive hash code for a large data file, the client will have to follow a series of validations carefully in order to arrive at H7.

To understand how this works, let’s look at the branch directly above V2:

figure 3. Nodes in a merkle tree above one leaf

When we lay out the merkle tree branch like this, it becomes clear how we can arrive at H7 by applying a series of hash functions. At each stage, the input to the hash function is composed from the previous stage hash function output and another hash code which must be provided as part of the proof. In this case, H1 and H6 must be provided in order for the proof to be properly validated.

Let’s recall that to prove that the input V2 is really part of our data, we must present it as input to a hash function. We do this here indirectly by composing several invocations of the same hash function. H2 is just the hash code of V2, this can be validated easily.

To compute H5 we must add a new piece of information, H1. But H1 is already known to the server and can be included in the proof. We can now concatenate H1 and H2 to arrive at H5. H6 is included in the proof same as H1, so we can now compute H7. This proves that V2 was part of the original computation which led us to H7 in the first place. And, it proves V2 is part of our ledger data. We have successfully presented V2 as the base of a successive applications of our hash function leading us back to H7. This is almost everything we need to prove the value of V2 against the main hash code H7. The only thing that’s missing now is to prove that the value we were presented with is really V2 and only V2.

When evaluating proofs the client must make sure the series of computations implied in the proof matches the requested data subset, V2 in our example. Otherwise, the client might be tricked by a malicious server into believing V4 or V3 are V2, for example. In a merkle tree, it’s not enough to arrive back at the merkle Root. The server can present a valid chain of computations leading from any leaf value V1..V4 back to the root hash. But only one such chain of computations is applicable to each distinct leaf or data segment.

Since the location of V2 in the tree is determined by it’s index (2), the client can inspect the order of concatenations used at each step of the proof to determine the location of the data segment each proof applies to. What this means is that a client must have intimate knowledge of the structure of a merkle tree in order to assess proofs. In our very simple example, the tree is a full binary tree and all the leaves are sorted by their indexes. This is why we can easily verify that the proof really addresses V2 and no other fragment of the data. Only for V2 the correct order of concatenation of inputs and composition of hash functions look like this:

Merkle Root = H7 = h(H5,H6)=h(h(H1,h(V2)),H6)

This is just another representation of the same proof. Once again, in each step of the computation we apply the hash function on top of the supposed value of V2 to arrive ultimately at H7. But for each position of a leaf node, we must apply a different formula. Here are the correct formulas for each segment V1..V4 in the example above:

H7 = h(h(h(V1),?),?) = h(h(?,h(V2)),?) = h(?,h(h(V3),?))= h(?,h(?,h(V4)))

When preparing the proof, our untrusted server will simply pull out the branch of merkle tree nodes and bundle them together to present as proof for the response to the query. As depicted in figure 4, it’s enough to just collect the missing hash codes (marked by question marks above) and attach them as a list, because the client already knows the correct formula for each data segment. But as we will see in my next post, it’s useful to present the branch in it’s entirety for more complex tree structures.

figure 4

Notice that only the merkle root and not the proofs need to be transmitted over a secure channel in advance. By successfully computing H7 using the matching formula for each data segment the client in fact verifies the value was indeed part of the original dataset and it’s position or address in the original set.

If anything goes wrong in the verification of the proof against H7, it means that either the segment address, value, or proof has been tampered with.

Radix Tries

Understanding the data structure of Radix Tries is important for the following discussion. But if you are already familiar with Tries you may prefer to skip this section.

A Radix tree is a compact Prefix tree. It is a space-efficient representation of a set of words. Words in the set are represented in a tree structure where lookup is very quick with time complexity of O(n), where n is the length of an average word.

A Radix trie is also efficient in looking up a set of words with a common prefix, and in finding the next available letters of valid keys in a set.

Lookup time is not determined by the number of words stored in the tree, rather, by the length of each word. The following illustration demonstrates the mechanism

figure 5. Simplistic illustration of a Radix trie with the words red, read, ready, readability.

The example above illustrates how words are represented in a trie. A trie is a word play on the words Tree and Retrieval. And it usually serves for retrieval of values from an index. It is also very useful for iterating through possible endings to any given prefix. This is why it is sometimes also called a Prefix tree. The structure above contains the words:

read (value: 1)

readability (value: 2)

ready (value: 3)

red (value: 4)

Notice the character unit “re” is not included in my list. This is because some nodes simply represent a branching in the tree, without there being a real word at the point of divergence. In reality, each node usually also holds some flag saying if this there is a word ending at the current branch. In this example, “re” is not considered a word, so the node’s data will indicate that it is different than the other nodes in the example.