You Could Have Invented Skynet

Skynet represents a huge step forward for the Sia network. It both greatly simplifies the average user’s access to data, and provides decentralized file sharing via short links. At its heart, Skynet is a natural extension of the existing Sia platform. It consists of three main components:

A way to upload files to Sia hosts, wrapped in special metadata. A way to download wrapped files and retrieve the original data. A link format that uniquely identifies the file.

In this post, I’ll show you how to use the us library to implement each of these components in just a few dozen lines of code. When we’re done, you’ll have a program that can upload files and give you a link, or take a link and give you a file – just like a Skynet portal. The only prerequisite is a muse server with some contracts. (If you need such a server, ping me on reddit or Discord and I’ll give you one for free.)

Part 1: Architecture

Normally, files on Sia are encrypted and uploaded to a subset of the hosts on the network, and each file’s metadata – specifying its hosts, sectors, erasure-encoding settings, and decryption key – is stored on the local computer. If you want to download the file later, you need this metadata.

This approach works fine, but it doesn’t meet Skynet’s requirements. We want Skynet users to be able to share files using short links. If we kept all of the usual metadata, Skynet links would be hundreds of characters long! By contrast, links on Skynet today are just 46 characters. How is this possible?

The trick is that the link does not encode any information about which hosts the file is stored on. Instead, it encodes the Merkle root of a 4 MiB sector, and some positional information that denotes which part of the sector contains the file. In order to download the file, we simply ask each host whether they have this sector, and then use the first host that replies in the affirmative.

(Note that this strategy only works for files that are smaller than 4 MiB. For larger files, we need another indirection: instead of the 4 MiB sector storing the file data directly, it stores a list of other sectors that store the data. For the purposes of this article, we’ll focus on the simpler case of single-sector files.)

Part 2: Uploading

There are three steps involved in uploading a Skynet file. First, we need to construct the sector that contains the file data, prefixed by a header. We then upload this sector to a set of hosts. Finally, we encode a link using the Merkle root of the sector.

The Skynet sector header consists of two parts: a fixed-size “layout” section (encoded in binary), and a variable-size “metadata” section (encoded in JSON):

type linkfileLayout struct { Version uint8 Filesize uint64 MetadataSize uint64 FanoutSize uint64 FanoutDataPieces uint8 FanoutParityPieces uint8 CipherType [ 8 ] byte CipherKey [ 64 ] byte } type linkfileMetadata struct { Filename string Mode os.FileMode }

For our purposes, we only care about unencrypted files that fit in a single sector, so we can ignore the Fanout and Cipher fields. Let’s write a simple function that constructs a sector from a file:

func encodeLinkFile(name string , data [] byte ) *[renterhost.SectorSize] byte { metadata, _ := json.Marshal(linkfileMetadata{ Filename: name, Mode: 0666 , }) layout := linkfileLayout{ Version: 1 , FileSize: uint64 ( len (data)), MetadataSize: uint64 ( len (metadata)), } var sector [renterhost.SectorSize] byte buf := bytes.NewBuffer(sector[: 0 ]) binary.Write(buf, binary.LittleEndian, layout) buf.Write(metadata) buf.Write(data) return §or }

Once we have a sector, uploading it is easy. We’ll request the active contracts from our muse server, open a Session for each contract, and use the Write RPC to upload the data to each host. And since this is Go, we’ll of course want to upload in parallel:

func uploadSector(mc *muse.Client, sector *[renterhost.SectorSize] byte ) error { currentHeight, err := mc.SHARD().ChainHeight() if err != nil { return err } contracts, err := mc.Contracts( "default" ) if err != nil { return err } results := make ( chan *renterutil.HostError) for _, c := range contracts { go func (c muse.Contract) { s, err := proto.NewSession(c.HostAddress, c.HostKey, c.ID, c.RenterKey, currentHeight) if err == nil { _, err = s.Append(sector) s.Close() } results <- &renterutil.HostError{HostKey: c.HostKey, Err: err} }(c) } var errs renterutil.HostErrorSet for range contracts { if err := <-results; err.Err != nil { errs = append (errs, err) } } if len (errs) == len (contracts) { return errs } return nil }

Great – we constructed a Skynet sector and uploaded it to some hosts. Now we just need to encode it into a link. Skynet links use a fancy algorithm for cramming a version, offset, and length into 16 bits. (We’ll omit the implementation here, since it’s of lesser importance; if you were inventing Skynet from scratch, you could use a simpler approach at the cost of slightly longer links.)

func encodeLink(data [] byte , sector *[renterhost.SectorSize] byte ) string { var linkdata [ 34 ] byte binary.LittleEndian.PutUint16(linkdata[: 2 ], computeLinkBits( uint32 ( len (data)))) root := merkle.SectorRoot(sector) copy (linkdata[ 2 :], root[:]) return "sia://" + base64.RawURLEncoding.EncodeToString(linkdata[:]) }

We now have all the pieces we need to pin a file, making it accessible from any Skynet portal. Let’s try it out on this blog post:

func main() { mc := muse.NewClient( "<my muse server URL>" ) data, _ := ioutil.ReadFile( "post.html" ) sector := encodeLinkFile( "post.html" , data) if err := uploadSector(mc, sector); err != nil { log.Fatal(err) } log.Println(encodeLink(data, sector)) }

Neat! We can confirm that this worked by accessing the link via a Skynet portal.

Part 3: Downloading

Downloading is just the inverse of uploading. The main difference is that we don’t know which hosts the file was stored on, so we need to try all of them. That means our muse server needs to have contracts with every host. (There’s a reason not everyone runs a portal!) For our purposes, we’ll assume that our muse server has at least one contract with a host storing the file.

We’ll proceed in the opposite direction of uploading: first we’ll parse the link, then we’ll download the sector, then we’ll extract the data. As before, we’ll omit the fancy logic of decodeLinkBits :

func decodeLink(link string ) (req renterhost.RPCReadRequestSection, err error) { link = strings.TrimPrefix(link, "sia://" ) data, err := base64.RawURLEncoding.DecodeString(link) if err != nil { return } else if len (data) != 34 { err = errors.New( "invalid Link" ) return } req.Length, err = decodeLinkBits(binary.LittleEndian.Uint16(data[: 2 ])) copy (req.MerkleRoot[:], data[ 2 :]) return }

Now we can call the Read RPC with this request object to retrieve the header and file data. But if the data could be stored on any host, how do we decide which host to try first? Well, for now, we’ll just do the simplest thing that works, which is to try all of them, one at a time:

func downloadSector(mc *muse.Client, req renterhost.RPCReadRequestSection) ([] byte , error) { contracts, err := mc.Contracts( "default" ) if err != nil { return nil , err } var errs renterutil.HostErrorSet for _, c := range contracts { s, err := proto.NewSession(c.HostAddress, c.HostKey, c.ID, c.RenterKey, 0 ) if err == nil { var buf bytes.Buffer err = s.Read(&buf, []renterhost.RPCReadRequestSection{req}) s.Close() if err == nil { return buf.Bytes(), nil } } errs = append (errs, &renterutil.HostError{HostKey: c.HostKey, Err: err}) } return nil , errs }

(A real Skynet portal queries multiple hosts in parallel. Implementing this is left as an exercise for the reader.)

Okay, we now have the raw sector data. We just need to parse it and extract the actual file:

func decodeLinkFile(sector [] byte ) (md linkfileMetadata, data [] byte , err error) { buf := bytes.NewBuffer(sector) var lfl linkfileLayout if err = binary.Read(buf, binary.LittleEndian, &lfl); err != nil { return } if lfl.FanoutSize > 0 { err = errors.New( "can only handle single-sector files" ) return } if err = json.Unmarshal(buf.Next( int (lfl.MetadataSize)), &md); err != nil { return } data = buf.Next( int (lfl.Filesize)) return }

And that’s it! We have all the pieces we need to download the file we previously uploaded. Let’s try it:

func main() { mc := muse.NewClient( "<my muse server URL>" ) req, err := decodeLink( "sia://KABaBLrJzYkQ5GYWvVpbw5zRhUnQL_Onv2JU4Y8-arC8Vw" ) if err != nil { log.Fatal(err) } sector, err := downloadSector(mc, req) if err != nil { log.Fatal(err) } meta, data, err := decodeLinkFile(sector) if err != nil { log.Fatal(err) } ioutil.WriteFile(meta.Filename, data, meta.Mode) }

It works! Even though our downloading strategy is extremely naive, it actually performs quite well if you use the same muse server for both uploading and downloading. You can view the full source code here.

Now that we have the basic functionality working, we could add fun features, like serving an HTTP API (like Skynet portals do), or adding an LRU cache to reduce latency and cost. At any rate, I hope I’ve demystified Skynet a little. You could have invented it!