S3 and Lambda Implementation Update

The Oyster development team has been hard at work prototyping new solutions for integrating AWS S3 and Lambda into the Oyster Storage platform, and today’s update will provide more information regarding some of the designs the development team is working on for these integrations.

For those readers who have not followed our development updates up until this point, the integration of AWS S3 and Lambda into Oyster Storage is to further increase upload speeds, as well as the overall efficiency of the system. For current file uploads, the data chunks are sent from the client and stored on a broker node, which then triggers a Lambda function that attaches the data to the Tangle. This is cumbersome on the system, as it requires a lot of bandwidth to move data from the client to the broker, and from the broker to the Lambda function. Furthermore, these actions are resource intensive, as it requires the broker node’s IO resources and disk space before the file is pushed to AWS Lambda. In layman’s terms, the current design has the broker node set as the middleman that stores chunks until they are ready to be uploaded to the Tangle. This design is not particularly scalable, as a broker node’s address may run out of disk space as chunks from other uploaded files begin to pile up in the queue.

The Solution

The Oyster development team will use AWS S3 to store the data chunks, as opposed to having them stored on the broker nodes. Not only will this free up a significant portion of CPU resources, it will also allow multiple clients to upload to S3 concurrently without BadgerDB or the broker nodes’ hard drive locking up, ultimately increasing the overall efficiency and scalability of the Oyster platform.

A basic diagram of this solution. Subject to change.

There are a few additional benefits to using S3 to store the data chunks instead of the Broker node hard drives. For example, uploading to S3 will be much faster, which will allow Oyster Storage to serve uploaded files from S3 even if they are still waiting to be uploaded to the Tangle. This benefit will create a much more pleasant user experience resulting in a drastic decrease in the wait time required for users to access, or share their files with their Oyster Handle. Another benefit to using S3 is that S3 can be used as a backup for the uploaded data in case Oyster’s private Tangle becomes corrupt, adding additional redundancy to the Oyster platform.

Change in Download Architecture

Another design that the development team is currently prototyping is a change to Oyster’s download logic that would allow a user to download a file directly from the broker node, or AWS S3. This change in download logic addresses a significant bottleneck for the Oyster platform involving the time required to perform the PoW necessary to attach data to the Tangle.

Note: This change in download logic would be implemented as a “wrapper”, and will be an optional implementation that will not be built directly into the Oyster platform.

Here is a basic outline of how this change would work. The client will start a download as it usually would by making queries to the Tangle for the data chunks required to re-assemble the file being requested. For this example, only 2/3 of the requested file has been uploaded to the Tangle (i.e., the middle third of the file’s chunks have not yet been attached to the Tangle). When the client reaches the first address of the middle-set of chunks that is currently not available on the Tangle, the client could resolve this issue in one of two ways. The first solution involves placing an endpoint on the broker node where the client could query for the remaining data chunks. The second solution involves using the AWS S3 API to retrieve the remaining data chunks from the S3 Bucket. Either of these solutions is made possible by storing information in the metadata chunk about who the client should ask for the missing data chunks (i.e., the broker node or the S3 Bucket).

This change in the download design would mean that as soon as the client uploads its chunks, the user would not have to wait for the chunks to be attached for the user to access their file. From a user perspective, the file would be made immediately downloadable, even if the file itself takes longer to fully attach to the Tangle.

Ultimately this change in download logic would require several other implementations in other parts of the Oyster Storage system to maintain security, and this current design is in its infancy and subject to change. However, the Oyster development team wants to remain transparent with the plans they are currently working on by giving the community a glimpse into the new design architecture that has been iterated on over the last two weeks.

As always, we encourage our community to post any questions or comments regarding these development updates to our Reddit or Telegram channel.