Development

GitHub metrics:

Developer activity (from Coinlib.io):

After a lot of hard work, the Storj team announced that Pioneer Beta for the V3 network has arrived. Now that V3 is here, the development team is also ready to launch the beta for Tardigrade, decentralized cloud object storage. So, what changes are coming with the V3 network? V3 is more scalable, performant, secure, decentralized, and economical than previous offerings. It was specifically built with the enterprise in mind, focusing on performance, availability, and reliability similar to major providers, at half the price.

Tardigrade is built on the Storj network and is backed by enterprise-grade SLAs comparable to traditional cloud storage providers, which means users can rest assured their data is securely stored. Because Tardigrade uses the Minio S3 gateway, it can be implemented by changing just a few lines of code, which allows potential users to try out the Storj network and the Tardigrade platform without having to integrate APIs through its developer library.

Here are some of the key features of the Pioneer beta release that Tardigrade enterprise users will enjoy:

Scalability — Tardigrade has been designed to scale to support exabytes of data.

— Tardigrade has been designed to scale to support exabytes of data. Performance — Tardigrade will deliver performance comparable to Amazon S3 and other centralized cloud storage providers. Even while in beta, Tardigrade has delivered upload and download speeds that are on par with or better than S3. For example, a 10 MB file can be uploaded to the network with a median time of 2.15 seconds and downloaded with a median time of 1.69 seconds. Even more impressively, even at the 95th percentile, those times change by less than a quarter of a second.

— Tardigrade will deliver performance comparable to Amazon S3 and other centralized cloud storage providers. Even while in beta, Tardigrade has delivered upload and download speeds that are on par with or better than S3. For example, a 10 MB file can be uploaded to the network with a median time of 2.15 seconds and downloaded with a median time of 1.69 seconds. Even more impressively, even at the 95th percentile, those times change by less than a quarter of a second. Durability — Since the final alpha network wipe, Tardigrade has delivered 100% durability (i.e. no file loss).

— Since the final alpha network wipe, Tardigrade has delivered 100% durability (i.e. no file loss). Retrievability — The ability to download a file on the first attempt is over 99.93%.

— The ability to download a file on the first attempt is over 99.93%. Affordability — At half the price of centralized cloud storage providers, Tardigrade can save users millions on their cloud storage bills.

— At half the price of centralized cloud storage providers, Tardigrade can save users millions on their cloud storage bills. Easy-to-use — Designed specifically for users to be intuitive and simple to use with built-in Amazon S3 compatibility.

— Designed specifically for users to be intuitive and simple to use with built-in Amazon S3 compatibility. Resilient — Decentralized storage is architected to be inherently more reliable and resilient than centralized offerings.

— Decentralized storage is architected to be inherently more reliable and resilient than centralized offerings. Private — Default client-side encryption and file sharding spread over a decentralized network ensures that only people with encryption keys can access data.

Tardigrade users can incorporate the Minio S3 gateway and start storing data on the network within just a few minutes of setting it up. The platform can also be integrated into applications using the CLI and libuplink developer library in Golang, C, and Android to take full advantage of the network’s capabilities. These libraries provide similar ease-of-development commands and bindings that are consistent with common languages, making integration intuitive.

The Storj team launched the first beta of the V3 Storj software, and the team simultaneously launched the beta for the Tardigrade Cloud Storage Service, which is (they expect) the first of many decentralized storage services built by many organizations using Storj software.

This post discusses the very important qualification gates, the team has established for each of the major milestones: Pioneer 1 (beta 1), Pioneer 2 (beta 2), and Voyager (production). It also discusses where Storj stands in terms of meeting those gates.

Background

The team’s goal with Tardigrade is ambitious: they aim to deliver the first truly enterprise-grade, decentralized cloud storage service. This means they not only have to deliver on the security, scale, and privacy promise of decentralization, but must also deliver enterprise-class durability, availability, performance, and support. This also has to measure up or be comparable to the well-established cloud service providers — and they also aim to do it at a fraction of their price. The tech team also needs to provide the proper economic support and incentives to its large and growing network of storage node operations, open source partners, and demand side partners.

This isn’t an easy task. Introducing any new storage technology is very difficult and cloud storage has an even bigger uphill battle, especially given the well established and generally well-regarded cloud services offered by some of the largest companies in the world. Hitting these lofty goals on a decentralized network, with enterprise-grade characteristics is much more difficult than building a centralized cloud service. In fact, it really hasn’t ever been done before.

Storj’s V2 network (launched in 2017) achieved scale rapidly. Storj reached 150 PB of capacity, over 100,000 globally distributed nodes, and steady usage faster than expected. However, Storj didn’t deliver durability and performance comparable to the centralized players. And the expansion factor of eight in V2 (i.e., consumed 8 GB of total storage for every 1 GB stored) made it difficult to make the network work economically. To ensure that the V3 network is sustainably enterprise-grade, the team took a different approach. Storj is S3 compliant, has an expansion factor of 2.7, and has the promise of far better durability, performance, and availability. Storj has also proceeded much more deliberately, with an emphasis not only on code and on network growth, but also on network quality, stability, and performance. One of the important elements in achieving this has been the adoption of qualification gates.

Qualification Gates

If you listened to its last town hall, you’ve learned that Storj established a formal series of qualification gates for the two Tardigrade betas and this has informed going into production. In general, you only get one chance to win customers over in the cloud storage space. So, to help ensure that Storj is truly enterprise-grade, it is holding to these gates. The gates are specific, objective criteria that Storj must meet in order to officially move into a particular phase.

A subset of those gates you’ll find outlined in the table below. In the first column, Storj lists the gates. In the second column, it lists where Storj is at now. In the third through fifth columns, you’ll see the gates for entering Pioneer 1 (beta 1), Pioneer 2 (beta 2), and finally Voyager (production). Any item in green has been met.

A few notes of explanation about the gates, other notes and explanations can be found here:

File Durability

This is perhaps the most important gate. The goal is that no file is ever lost. After doing the last major alpha release 2.5 months ago, Storj hasn’t lost a single file or segment (i.e. durability of 100%). The production goal is 99.9999999% (Nine, nines — meaning you’re far more likely to win the lottery or be struck by lightning than lose a file on Tardigrade).

Storj measures durability not only by randomly downloading large numbers of files, but also by looking at segment durability. While the random test is a sample, the segment durability looks at the entire population of segments.

As a reminder, each file is broken up into one or more segments of 64 MB each. Each segment is (using Reed Solomon) divided into 80 pieces, of which any 29 can be used to reconstitute the segment. Each part of the 80 pieces is located on a different storage node in the network. A segment would be lost if the number of available pieces dropped below 29. However, long before that point, we would initiate a repair process by reconstituting all 80 pieces. In theory, we should never lose a file, since nodes have low correlation risk, operating on different hardware, different power supplies, different networks, with different operators, etc.

As you can see from the graph below, Storj constantly measures the number of pieces available for segments in the network. It reports the median number of pieces for each segment, the distribution of pieces, and the segment with the lowest number of pieces. This means the “least healthy” segment has never dipped below 57 pieces, and the median is holding steady at around 75. As is the case with many of the metrics, durability should only improve as we add more nodes to the network and a broader distribution of nodes.

Figure 2: Segment Health Chart (Segments must have a minimum of 29 pieces to be reconstituted).

Other notes and explanations can be found here.

Recent development accomplishments:

Enhanced the storage nodes to automatically delete expired or rejected orders to save space on their disks.

Enhanced the performance on Satellites by grouping some database calls into single transactions.

Finished a few design docs for features which is going to be working on in the near future. If you want to learn more about those please visit the forum.

Refactored some parts of the Satellite user GUI to be more performant and simpler in order to help Storj implement future features.

What the team is focused on next: