One Year of us

Today marks one year since I introduced us , my “alternative interface to Sia.” I figured I’d mark the occasion with a little retrospective: what progress was made on us in 2019, and what’s next for 2020?

Let’s start from the beginning. In the two weeks after my initial announcement, I published three blog posts. How to use Sia Irresponsibly came first, demonstrating the power and flexibility of us by using it to upload files without encryption or redundancy. In hindsight, I regret writing this post. It seems to have made a strong first impression, and not a good one – many people (even some Nebulous employees) came away thinking that us was not capable of encryption or redundancy at all! What was intended to be a cute demo ended up backfiring on me quite severely.

My next post, How to use Sia Responsibly, showed how us allows developers to add custom encryption and redundancy to their data. This post did not attract as much interest as the first, and the primary topic of community discussion was “please change the name.” It’s true, us is a somewhat awkward name, but no one has suggested a better one. Personally, I like that us can be used to form larger words ( user , walrus , muse , etc.), just as us -the-library can be used to build larger pieces of software.

My third post, Files as Capabilities in us, departed from the previous two by including no code samples and instead focusing on the project’s design philosophy. It received even less attention, except for one commenter who insisted that the us metafile format would not scale to hundreds of thousands of files. Not really the point of the post, but ok. ¯\_(ツ)_/¯

My intention with this series of posts was to excite and inspire developers by showing them what was possible with the new tools I had developed. Needless to say, reading the comments on these posts left me feeling a bit deflated. It took me a long time to realize what was, in retrospect, obvious: most of my audience is invisible! According to the 90-9-1 rule, for every 100 people who read a post, maybe 10 will upvote (or downvote), and maybe 1 will leave a comment. In other words, I was only seeing the tip of the iceberg.

To my relief, the project's reception among the "silent majority" was more favorable than I initially thought. I repeatedly observed the community members I respected the most – the people creating apps and services on top of Sia – speaking highly of the us project. And in private chats, nearly every developer I spoke to told me that they were watching the project closely and were considering building something with it. It was only a matter of time before one of them took the plunge, and in February, Salva Herrera (aka @hakkane) became the first to do so. Salva and I collaborated to develop a continuous host benchmarking tool that measures the transfer speed of every host on Sia on a regular basis. us was well-suited to the task: the tool only required ~200 lines of code and took just a few hours to write. It now serves as the backbone of the SiaStats Host Monitor.

In May, I added FUSE support to user , enabling it to mount a virtual Sia filesystem that can be used with any other program. user is a CLI program that leverages the us libraries to upload and download files, much like the siad renter. Although I had released user alongside us back in January, it had not seen much interest, so I was hoping to attract more attention to it by leap-frogging the capabilities of siad . Unfortunately, FUSE didn’t move the needle much. Despite its attractive features and performance, user simply does not offer a good experience for the average Sia fan. It requires you to take an active role in selecting hosts, renewing contracts, and repairing files – responsibilities that siad handles automatically. As long as this continues to be the case, user will be restricted to a fairly small audience of enthusiasts.

The next month, I collaborated with another community member: Michal Šefl, aka @Danger, who wanted to know if us could be used to build a Sia mobile app. I had some inkling of how to do it, so down the rabbit hole I went. When I emerged, I had implemented something pretty neat: cross-language bindings for us . Michal used these bindings to call us code from C#, releasing a demo in which he uploads and downloads a file on an iPhone, connecting directly to Sia hosts. His blog post was well-received, as many in the community have long desired a mobile app for Sia. I followed up Michal’s post two weeks later with one of my own, Talking to Sia Hosts from…Ruby??, showing off the new cross-language bindings. Reception was positive here as well, although no one took me up on my offer to help them port us to additional languages.

In August, there was some internal discussion at Nebulous regarding “garbage collection.” Simply put, Sia software has never really supported deleting data that you no longer want to store. Historically, this hasn’t been a problem, because storage is so cheap. But in the long-term, we need a way to delete. This seemed like a good test of us , so I took at stab at it. Within a day I had a working implementation, albeit one that is unlikely to scale to massive amounts of storage. This is in line with my previous experiences implementing features like HTTP streaming, FUSE, etc. with us – I can achieve “minimum viability” extremely quickly, and iterate from there. This helps me allocate my effort more effectively: If it turns out that people don’t care about garbage collection, then I won’t have wasted much time on it; conversely, if people use the feature a lot and start hitting scaling limits, I can justify spending more time optimizing it.

Later that month, us turned an important corner by winning its first big “customer:” StoreWise. The StoreWise developers reached out to me in August to explore building their v2 product architecture on us . This architecture will allow StoreWise’s users to connect directly to Sia hosts, bringing them the latency and throughput benefits of a decentralized network while also massively reducing the company’s bandwidth costs. I’m tremendously pleased that StoreWise has chosen to bet on us – in fact, I had them in mind as a potential partner long before I announced the project. Now it’s up to me to ensure that their bet pays off! If the integration is a success, it will bring real legitimacy to the us project, and will likely inspire other developers to pursue integrations of their own.

With the year half over, I switched my focus from storing data to storing siacoins. In September I released walrus and narwal , two new wallet projects built from us libraries. walrus features first-class support for the Ledger Nano S hardware wallet and a developer-friendly API, while narwal offers a “lite wallet” version of walrus that lowers the barrier of entry to the Sia ecosystem. Specifically, narwal makes it possible to form and renew Sia file contracts without having to run a full node or waiting hours to sync the blockchain. My accompanying blog post, A Lightweight Wallet for Sia, attempted to clarify the distinction between walrus and narwal , with...dubious success. The post was received well on reddit, and two community members offered to run narwal nodes to reduce centralization. But real usage did not materialize: I watched the logs on my narwal server, and only a handful of people tried it out. I’m hoping that this will improve with the introduction of new wallet frontends on desktop and mobile, currently expected for release in 2020.

On a whim – probably inspired by Satoshi’s Treasure – I created a series of three “treasure hunts” for the Sia community in October, each requiring the use of various us tools. The clue for the first hunt was simply a user “metafile” encoded in base64; inspecting the metafile revealed that the file was stored on a single host. After forming a contract with the host, anyone could download the file, thus winning the “treasure” of this meme. The clue for the second hunt was again a metafile, this time split into its (rot13’d) index and single (base64) shard. Downloading it (using the same process as before) yielded a file named “Rick.And.Morty.S04E01.mkv”, which actually contained this video. For the third challenge, I upped the stakes a bit, offering 100,000 siacoins instead of silly memes. I also increased the difficulty: this time, the clue consisted of nothing but an Ed25519 public key. To my surprise (and slight dismay), this hunt was solved in just over an hour! I was very impressed by all the participants in these hunts. Given that us has mostly "flown under the radar," I expected that substantial hand-holding would be required, but no – the community hardly had any more trouble solving my challenges than I would have myself! Overall, the treasure hunts were both fun and gratifying (and perhaps drew more attention to the project), so I plan to do more of them in 2020.

I capped off the year by creating muse , the first Sia contract server. muse integrates with other us tools like walrus , shard , and user , making it possible to upload and download files with nothing more than a user binary and a muse server URL. This is something that I’ve wanted to do for ages – I remember discussing it in a Nebulous board meeting years ago, and since then I’ve brought it up whenever someone asks about apps they could build with us . Just stick a paywall in front of a muse server, and you’ve got a viable business that sells contracts for fiat! I teased some muse functionality back in November, but sadly I wasn't able to get it to a release-ready state before the end of the year. It will be released sometime in January, with an accompanying blog post describing the important role it will play in the us ecosystem.

Takeaways

My most important takeaway this year was the "90-9-1" rule from above: most of the people using my software don’t open pull requests, or ask for new features, or report bugs, or even hit the“Star” button on GitHub. And yet, in the past two weeks alone, the various us repos have been cloned over 1000 times! Who are all these people? What feedback could they give me? It pains me to know that some of them must have lost interest due to a bug that I could have easily fixed – if only they had reported it. Even developers that I know are using us seem hesitant to open issues or ping me with questions, as if they’re afraid they’ll bother me. So if you’re reading this now, and you want to report a bug/request a feature/ask a question, please open an issue, email me, tag me on Discord, whatever!

Regardless, the “invisibility” problem means that I have re-evaluate my approach to releasing code. Previously, I assumed that I could release code as soon as it was minimally useful, and my users would let me know what to improve. This might work for a project that receives multiple bug reports and PRs per week, but it’s not going to fly for us . Instead, I need to put extra care into polishing my code before release – I need to make a good first impression! Otherwise, people won't stick around long enough to help me make things better.

A more personal takeaway for me is that I am much more productive when working on projects that I have complete ownership of. Over the years, siad has grown from an amateur two-man effort into a true flagship product supported by a full complement of engineers. That's great, but I found that the more sophisticated siad became, the more “burnt out” I felt when working on it. It was nice to discover that I have not burnt out on programming in general; hacking on us feels much like the early days of hacking on siad . Accordingly, I am taking a step back from siad development and refocusing on other ways that I can contribute to the vision of Sia – including us – without burning out. So far, so good. :)

What comes next?

I have a few ideas for 2020, but on the whole it’s rather open-ended. My top priority will be working with StoreWise (and other “customers,” if they reach out to me) to ensure that us has the features and performance they need. After all, if Sia is going to succeed, it will be because of projects like StoreWise – projects that bring decentralized storage to the masses.

I expect to put a lot of time into polishing muse , as it’s still quite rough around the edges. I am very eager to see a site that sells contracts for currencies other than siacoin, so if I can find someone willing to operate such a site, muse will evolve as necessary to meet their needs. It occurred to me that siad could implement the muse API; that way, users could rely on siad to pick hosts and maintain contracts, two areas where us is currently lacking. I’m also interested in collaborating with a community member to develop a graphical frontend for muse , as the CLI frontend is not very user-friendly.

On that note, more work is needed on user to make it appeal to a wider audience. Specifically, I need to quit waffling and add some form of automatic migration (i.e. file repair). Historically, I’ve been uneasy about doing this; I don’t like making decisions on the user's behalf, but maybe it’s acceptable as long as its opt-in. On the other hand, I expect that user will eventually be made obsolete by more professional apps, so I’m not certain how much energy I ought to devote to it.

As for walrus , I hope to see one or two graphical clients released in 2020. A mobile app has been in active development for the past few months, which I’ve contributed to in the form of additional C# bindings. I have also explored writing a desktop UI for walrus with first-class support for hardware wallets. I don’t have the design sensibilities (or JavaScript chops) to make this a reality by myself, but with a committed frontend developer, I think we could put something together very quickly.

Lastly, I’m interested in devising some more treasure hunts. The hunts I organized in October went really well, and I’m sure I can come up with new ones. In particular, I’d like to design a treasure hunt that is more collaborative, so that participants are incentivized to share their knowledge rather than hoarding it for themselves to gain an advantage.