Introducing replication was also an opportunity for file owners to take greater control over their data: we could allow them to choose the number of copies they wanted to store. We called this quantity a file’s baseline redundancy.

At this point, we had a multi-peer network that encrypts file contents with private key encryption before upload and implements simple file replication redundancy scheme so that a file could still be retrievable if one of its host nodes disconnects.

File Sharding

What if a file owner wanted to upload a large file, say, 100GB? Each host node would have to store all 100GB. A 100GB file is a substantial storage and bandwidth burden for hosts. It’s possible that no host on the network even has that much free storage space. Further, hosts are probably casual users without top-tier bandwidth capabilities. Asking them to download an entire 100GB file is not only a storage burden: it’s also a bandwidth burden.

A file owner uploads two copies of its 100GB file to the network.

It would be far better if the network could distribute the bandwidth and storage burdens of storing large files across multiple nodes on the network. We accomplished this with a method called file sharding.

File sharding allowed us to prevent an entire file from being stored on a single host by splitting that file into pieces and sending each piece to a different host on the network.

A file owner uploads two copies of a 100GB file to the network, but each copy is split into three pieces. Each piece is then sent to a different host.

At this stage in development, we had a multi-peer network in which file owners’ files were encrypted, sharded into pieces, and then distributed to different hosts on the network. Copies of each shard would be distributed to different hosts to satisfy a pre-set level of baseline redundancy.

Keeping track of shards

With the introduction of file sharding, we introduced a one-to-many relationship between an owner’s file and the shard-files distributed across the network.

As far as host nodes on the network were concerned, each shard was a distinct file. There was nothing inherent in the shards themselves that preserve the relationship between a file and each of its shards. Further, there was nothing indicating the order in which these shards should be combined to reconstruct the encrypted file successfully.

We solved this problem by generating a manifest file for each file uploaded to the network. The purpose of the manifest file was to keep track of each shard and its duplicates, as well as the order in which these shards need to combine to reconstruct the original file’s contents.

To accommodate all the shards, we modified the upload process so that a shards directory stored all shard copies (as shown in the demo above), which are then concurrently uploaded to different hosts across the network. If a file is sharded into 5 pieces and 3 copies of each shard were made, then the shards directory stores15 pieces which would be sent off to 15 different hosts (assuming there are at least 15 different hosts on the network).

With this new approach, a file owner was essentially “trading” their uploaded file for an (ideally) smaller manifest file

Each “chunk” in the manifest file represents a distinct shard. Each distinct shard is uniquely by the hash of its contents. Each distinct shard is paired with an array of IDs, which are the IDs of the shard’s copies.

Sharding Method Tradeoffs

Managing the size of the manifest file was a fascinating challenge to explore. Since the manifest file stores a record of shards, increasing the shard count increases the size of the manifest. However, recall that the purpose of sharding was to reduce the storage and bandwidth burden of large files on hosts.

Keeping the shard count constant increases the shard size as a file grows. However, it holds the size of the manifest file constant, benefitting the file owner. It also causes each shard of k constant shards to increase as the size of the uploaded file increases.

Conversely, keeping the shard size constant increases the shard count as a file grows. However, it holds the size of each shard constant, benefitting the file host. It also causes the manifest file to grow larger for larger uploaded files.The question is:

Which method is more space efficient?

For each sharding method, one party’s storage burden grows linearly with file size, while the other party’s storage burden remains constant.

Shard-by-k-size

In the shard-by-k-size method, the storage cost of the file owner, or the size of the resulting manifest file, is represented by the following function:

manifestSize = 20 bytes * (fileSize/shardSizeConstant) * duplication Constant

In plain english, this is stating that the size of the manifest is roughly equivalent to the amount of space of a single shard ID (20 bytes) times the number of shards multiplied by the number of duplicates of each shard. The only non-constant value is fileSize , indicating a linear relationship between file size and manifest size.

Shard-by-k-quantity

In the shard-by-k-quantity method, the storage cost of the file host, or the size of each shard, is represented by the following function:

shardSize = sizeOfFile / numOfShardsConstant

In plain english, this is stating that the size of each shard is equivalent to the size of the file itself divided by the pre-set quantity of resulting shards. If the number of resulting shards was set to 10, for example, then each shard would be 1/10th the size of the file. It is again clear that the shardSize grows linearly in relation to the size of the file.

For each sharding method, one party’s storage burden increases linearly with the size of the file, while the other party’s storage burden remains constant

Neither method is optimal

It turns out that a pure shard-by-k-size and a pure shard-by-k-quantity method both present significant drawbacks.

A shard-by-k-quantity approach is sure to fail. If the constant quantity is too low, then large files will produce shards that are unreasonably large for file hosts. We could set the constant quantity really high, but that would produce issues for small files.

Sharding-by-k-size presents similar problems. We could choose a constant shard size that could work for many files, but not for large files, where the risk would be producing so many shards that the manifest file actually grows to an unreasonable size.

For example, perhaps our duplication constant was 500, meaning that for each shard, the manifest stored 500 * 20 bytes, or 10KB. Perhaps our constant shard size was 500KB (a common shard size for files uploaded in BitTorrent). Imagine a file owner was uploading a 1GB file:

A 1GB file sharded into 500KB pieces, which would result in 2000 shards.

At 10KB per shard in the manifest, that ends up producing a manifest of about 20MB in size.

If we doubled the shard size from 500KB to 1MB, we could halve the size of the manifest (saving the file owner 10MB of storage) whilst only costing each file host 0.5MB of extra storage.

The best sharding method is, therefore, one that chooses values dynamically to minimize the storage burdens on each party within certain limits (for example, a shard probably shouldn’t climb to 10GB in size in an attempt to reduce the size of the manifest file. If it came down to it, a larger manifest file is better than a large shard since the manifest doesn’t need to be transmitted or downloaded). Both BitTorrent and Storj shard files differently based on the size of the file. (See here and here)

Proofs of Retrievability

At this point, our system looked like this: we had a multi-peer network, where uploading entailed encrypting, sharding, replicating, and dispersing. The encryption kept the file content private. The sharding kept the storage and bandwidth burden low. The replication increased the file’s availability.

However, replication only solved part of the availability problem. The likelihood that hosts have gone offline, lost or corrupted a data owner’s file increases with time. Ideally, Layr would offer a way for file owner’s to maintain their files’ baseline redundancy (which is the number of copies they have chosen to upload to the network).

An owner needs to detect when a duplicate of any shards of their file(s) have been corrupted or lost (either because the host is offline or because the host deleted the file) to respond to decreases in a file’s baseline redundancy.

A naive way to test whether a shard is retrievable is merely to attempt to retrieve each shard. If the host is offline, the test automatically fails. Otherwise, if the host returns the shard, the tester (file owner, in our case) can make sure the shard’s contents are intact. This method did not suit our needs for a couple of reasons. First, it produces a lot of communication overhead. Second, the owner would need to keep a local, authoritative copy of each shard for comparison, which violates our requirement that a cloud storage system should allow users to delete their local copies of uploaded files.

Naive way to test data integrity by downloading the file and checking its contents.

A second method we explored was requesting the hash of the file’s contents instead of the contents itself. This minimizes the communication overhead and requires the file owner to only store the hash of each shard(something they already store in the manifest file). Unfortunately, this method proved to be insufficient as well: A host could easily cheat by storing the hash of the shard’s content rather than the shard’s content itself.

Improved audit: Ask for the hash of the file’s content and compare against the hash already contained in the manifest file.

The third method we considered was hash-based as well but precluded the host from generating an answer ahead of time. Instead of asking the host for the hash of the content itself, we asked for the hash of the content + a randomly generated salt. Since the salt would not be revealed until the file owner performed the audit, the host would need the shard’s contents at the time of the audit as well if they were to pass.

Because the owner should be able to delete their file and its shards after the upload process, these challenge salts + the hash of the shard content and salt needed to be pre-generated during the upload process. That also meant that an upper bound was placed on the number of audits an owner could perform. Despite these limitations and others, this approach provided a strong level of security, which is why we chose it.

For each shard copy, its ID is associated with a set of challenges and then a hash of the challenge + the shard data. When a file owner wants to audit a shard, it sends the challenge over to the host who then has to return the hash of the challenge + shard data. Since the host doesn’t see the challenge until the time of audit, and since the shard data is required alongside the challenge to generate the correct hash, they are forced to possess the intact shard data at the time of audit in order to pass.

Patching files

The benefit of proofs of retrievability was that they allowed file owners to detect when the redundancy of their file(s) decreased. However, this knowledge is far more valuable to a data owner if they can do something about it. In lieu of this, we implemented an operation called patching. If an audit revealed that some copy or copies of a particular shard were unretrievable, then the patch operation would identify a copy that was still retrievable, download that copy, generate a new unique ID for that copy, and send that copy to a new host. It would then update the manifest file, replacing the unretrievable copy with the newly uploaded copy.

Incentivizing participation and cooperation

At this point in our development process, our system had come a long way. We had a redundancy scheme, file sharding, manifest files to track shards, proofs of retrievability, and file patching. There was still a burning question: Why would people participate in the first place? Sure, file owners had a strong incentive to join the network, but what about potential hosts who had free storage but didn’t necessarily have anything to store? A network composed entirely of peers who only wanted to upload their files wouldn’t be a useful network.

We researched different ways to incentivize behaviors into a system, and ultimately settled on financial incentive. Because we were building a decentralized system, it followed that we ought to use a decentralized payment system, which implied the use of some form of cryptocurrency. We opted for Stellar because of their robust JavaScript SDK, their testnet, and low transaction fees.

Naive Payments

We thought of the most straightforward incentive scheme we could: pay-by-upload. In a pay-by-upload scheme, file owners would pay hosts for storing their files during the upload process. Choosing a simple incentive scheme allowed us to hone in on the technical aspects of implementing transactions in a no-trust environment, despite the fact that a simple incentive scheme was probably not economically robust (and, in fact, a pay-by-upload scheme has important drawbacks. See Limitations and Future Steps for details).

The first implementation strategy we thought of was: pay-before-upload. In a pay-before-upload strategy, the file owner would first identify the host to which they plan to send their file. Then, they would pay the host. They would then send the file to the host after the payment went through.

The problem with this approach was that the file owner couldn’t trust that their data would make it to the host after the payment went through. Because payment and file-transfer were separate operations, the owner could not be sure that both succeeded: they ran the risk of paying for space that their file(s) never actually occupied. Similar problems arose when we explored a pay-after-upload implementation strategy: the problem remained that the file owner cannot be sure that the host has their file intact at the time of payment.

The owner pays the host then sends the data to the host. The payment goes through but the data transfer does not, causing the host to lose money.

Since a p2p network is a distributed system composed of peers who all possess a local state and can only update each other’s state through passing messages, it made sense that these methods did not provide atomicity between acknowledging receipt of data and payment for that data. Both actions were taking the form of distinct messages. A message is not guaranteed to reach its destination within a particular time frame, nor is it guaranteed to reach its destination at all. We needed a way to “combine” these messages so that one could not succeed without the other succeeding as well.

Introducing Smart Contracts

We accomplished this with smart contracts.

Smart contracts provided us with a means to perform transactions only when all parties involved have satisfied certain conditions.

Smart contracts batch transactions together and apply constraints to these transactions. If one transaction fails, the whole batch fails. If one party wants to pay another party only if certain conditions are met, smart contracts allow us to accomplish this.

Smart contracts batch transactions together and apply constraints to these transactions. If one transaction fails, the whole batch fails. If one party wants to pay another party only if certain conditions are met, smart contracts allow us to accomplish this.

Imagine I wanted to hire you to do some work on my house. I didn’t want to pay you ahead of time because I didn’t trust you to run off with the money. You didn’t want to work without getting paid first because you didn’t trust me to pay you once you were finished. So we decide to involve an attorney. You and I decide on a set of specifications the renovations must meet with the attorney present. I give the amount I want to pay you to the attorney, who holds onto it so that I can’t take it while you’re working. Once you finish your work, you submit proof that the work was completed to our agreed-upon specifications. If your work does indeed meet our specifications, then the attorney gives you the money I deposited earlier on. Otherwise, the money is refunded back to me.

Smart contracts can replace the attorney in the above scenario.

How Stellar Smart Contracts (SCCs) Work

Before I go into how we used smart contracts to accomplish the above goal, it is worth taking a small detour to cover pre-requisite knowledge about how smart contracts work in Stellar. From the Stellar developer’s guide:

A Stellar Smart Contract (SSC) is expressed as compositions of transactions that are connected and executed using various constraints.

Further, transactions in the Stellar network are“commands that modify the ledger state,” somewhat akin to SQL queries in a database. Transactions can be composed of multiple operations which are actually responsible for manipulating the ledger’s state. Transactions execute operations sequentially as one batched, atomic event. If one operation fails, the entire transaction fails.

All operations have a particular threshold and all signatures, a particular weight. A transaction will automatically fail if the sum of the weights of the signatures of the transaction do not meet the threshold of any operation within the transaction.

Stellar accounts can also create more accounts with the create_account operation. This operation can be combined with other operations to set a variety of constraints on the newly created account. In essence, by combining Stellar operations, developers can create complex relationships between users, as well as independent places to hold funds and only release those funds under certain conditions, to certain parties. Importantly (for Layr) Stellar transactions can use a combination of operations to create an automated escrow contract between file hosts and file owners..

Paying with smart contracts

In our scenario, we wanted the funds from the file owner to be released to the host only if they could prove they had the owner’s data.

The way we accomplished this was with the following workflow:

File owner creates and deposits funds into a smart contract File owner hashes the encrypted file data File owner adds this hash as a hashX signer to the contract File owner transmits file and contract ID to the host The host attempts to unlock the funds in the contract by submitting a transaction and signing it with the file data it received Stellar then hashes the file data and checks to make sure the hash matches the hashX signer If the hashes match, then the transaction goes through. Otherwise, it is halted.

Graphical representation of smart-contract payments with HashX signers.

An important note here is that although the hash(fileData) is added as the signer, someone cannot sign with the hash(fileData) : they must submit the fileData itself. This is important because it ensures that the host cannot pre-generate the hash(fileData) , delete the fileData and then sign off on the transaction: they must sign off on the transaction with the fileData itself, thereby guaranteeing atomicity between proof of data possession and payment.

Limitations and future steps

Smart contracts

Our smart-contract payments with hashX signers took care of the basic batching of proof of data possession and payment, but there were potential weaknesses to that approach.

If the hashX signer has enough weight to sign a payment transaction, then the weight of the hashX signer meets medium threshold. Anyone with the pre-image of the hash can therefore sign off on anyone transaction with a medium threshold, including further payments. The pre-image of the hash is made public once a transaction signed with the pre-image is submitted to the Stellar network (read about HashX signers here).

In regards to problem 1, the most destructive action a signer with a medium threshold can perform is create a payment. Adding other signers or merging an account require signers with greater weight. Since the account is only funded with enough money to pay for the file data, there is no risk of the funder (file owner) losing extra money. If we were to use a different incentive scheme, such as pay-by-passed-audit, we would opt for keeping the same escrow account between the file owner and each host, funding it for each audit. We would also update the hashX signer to match the hash of the fileData + challenge salt for each subsequent audit. In this case, we would need to prevent the hashX signer from withdrawing more than they were allowed to withdraw. What we would do to prevent this is:

Set the weight of the hashX signer to a weight < the low threshold Create a pre-authorized transaction that only authorizes withdrawal of funds to cover the passed audit and add this pre-authorized transaction as as a signer. Set the weight of the preAuth transaction such that hashX weight + preAuth weight = medium threshold.

This would prevent the host from withdrawing more than they were supposed to.

Problem 2 is a non-issue for us because the smart contracts’ sole purpose is to facilitate a single payment between the host and owner during the upload process. It’s not a privacy issue either since the pre-image (or file data) is encrypted.

However, if the owner needed to issue multiple payments to the host over the course of service — e.g., if the owner was paying the host when the host passed file audits — then this could become an issue because the hashX signature is publicly visible.

The use of pre-authorized transactions, combined with reducing the weight of the HashX signer, mitigates this risk as well.

A more sophisticated incentive scheme

A pay-by-upload incentive scheme is very simple which allowed us to focus on the nuances and technical challenges of payments in a p2p environment. But designing economically sound (rather than just technically sound) incentive schemes is an important challenge for any decentralized product.

Although a pay-by-upload scheme incentivizes participation, it doesn’t incentivize some of the most important behaviors for file hosts: hosts could easily delete the files immediately after getting paid without any repercussions. In fact, they are incentivized to do so because they would make the same amount of money whilst also preserving their storage space.

An effective incentive scheme should promote the following attributes on the part of hosts:

Host participation (which pay-by-upload successfully incentivizes) Uptime (being available on the network) File preservation (keeping the files safe from loss or corruption)

Both uptime incentive and file preservation can be encouraged with a pay-by-passed-audit incentive scheme. In such a scheme, hosts are only paid when they successfully pass audits (i.e., when the files they are storing are both present and unmodified). Since a file owner initiates audits at their discretion, hosts are incentivized to remain online as often as possible (remember that an audit automatically fails if the target host is offline).

Erasure coding

Erasure coding is a class of redundancy schemes that produce the same levels of file availability as replication without requiring as much storage. We plan to explore erasure coding in greater depth moving forward before deciding whether its benefits are worth its potential drawbacks.

In our simple replication model, a shard copy for each distinct shard is necessary to reconstruct the original file:

Simple replication: A copy of each distinct shard is required to reconstruct the original file.

That means that the availability of the file is equivalent to the availability of the least available set of shard copies. All copies of shards 2, 3, and 4 could be online, but if the copies of shard 1 are unavailable, then the file as a whole is also unavailable.

Erasure coding removes this limitation by creating something called parity shards. Parity shards are like “wildcards” that can fill in for other missing shards. If the file is split into 4 shards and then 8 parity shards are created, that provides a total of 12 shards. However, any 4 out of the 12 total shards can be used to construct the original file. As long as any 4 shards are available, the file is also available.

Reconstructing the file with any 4 out of 12 total shards.

Proofs of retrievability

We are currently exploring more sophisticated proof of retrievability methods that:

A. Provide an unbounded number of audits

B. Do not require the host to process the entire file to respond to the audit

C. Do not require the owner to store extra information

D. Still provide high levels of security

Conclusion

I hope you’ve enjoyed reading about our experience building Layr. As experimental software, there are many challenges and improvements ahead. I also hope you’ve developed a clear mental model about some of the core challenges involved with building a decentralized storage system and I urge you to give it a shot! It’s a wonderful learning experience.

Once again, we are all open for full-time opportunities at the time of this article’s writing. If you are interested in having one of us joining your team, please reach out!

Please also feel free to contact any one of us if you have further questions about decentralized cloud storage in general, or the process of building Layr specifically.