Networked DHT Implementation is Ready for Integration!

Holochain Dev Pulse 07

Summary

With our improved, faster automated build/test pipeline, we’ve been able to fully land more features in Holochain. The Rust version is quickly approaching feature parity with the Go prototype, and we’re getting very close to fully networked nodes!

Highlights

1.Node-to-Node Messaging: Feature for Private Communication between Agents

2. Validation of Entries against Source by DHT Nodes

3. Networked DHT Implementation: Ready to Integrate

4. Development Container: You can Now Persist Your Chain to Disk

5. Massive Speed-ups in Build Pipeline: Happier, more productive core developers

6. We want your feedback! Please take 3.5 minutes to take our Dev Pulse Survey

weekly Holochain development pulse

Details

1. Node-to-Node Messaging: Feature for Private Communication between Agents

Everyone knows it can get slow to send emails back and forth when you and a colleague are trying to get something done. Sometimes you need to pick up the phone and don’t need a record of the conversation. hApps can feel the same way. Most node-to-node communication doesn’t need to be recorded for all eternity. Instead, one node just needs to get another’s attention so they can reach a private agreement on something.

If you’ve built a hApp on the Go prototype, you’ll be familiar with the `send()` function and `receive()` callback. These are now implemented on Holochain-Rust and will land in an official release soon. (Note: Examples of these functions haven’t made it into our app-spec yet. You can find the latest API references here).

One node sends a message to another node, using the same underlying system as Holochain uses to gossip between nodes. If that node is online, the ‘receive’ callback will be triggered, and the result of that function call will be returned over the network. The sender will wait a maximum of 60 seconds for a response, or return a timeout error. All the normal API functions are callable from the receive callback, enabling you to perform complex functions. However, the passed and returned value, must be a string, so if you want to pass complex data, you will need to convert it to and from a serialized format.

node-to-node messaging

2. Validation of Entries against Source by DHT Nodes

When your peers receive your data for validation, they make sure it doesn’t break the app’s validation rules. They may need to know your whole history to do this. Before a node validates your data, it asks your node for a ‘validation package’; this package might consist of your entire source chain, only the headers, or just the current entry with its header. This all happens behind the scenes. As a developer, you only need to decide what the validation package should contain, and decide what to do with the package.

This is one more step towards full validation capability. As one of the building blocks that allows us to create distributed networks with similar integrity guarantees as blockchain, but at a fraction of the computing cost, it is a necessary component in the Holochain architecture.

3. Networked DHT Implementation: Ready to Integrate

You may know from previous Pulses that we’ve been working hard on networking — after all, a distributed networking framework isn’t all that useful without networking! First to land was local ‘mock networking,’ enabling developers to run multi-agent tests. This was followed by preparations in the core to make everything network aware.

As we were building the various components in parallel, we found that things were changing in networking code as fast as they were changing in core. Any experiments or updates would then have to go through a lengthy merge process. Therefore, in order to rapidly create prototypes of the networking layer, we decoupled the core Holochain runtime from specific assumptions about how to communicate with peers — find neighbors, do gossip, send/get DHT data, etc.

Today, our first MVP networking prototype is built and ready to be connected to core. In order to establish fully production-level networking, there will still be major updates following the MVP release, but we’re not there yet. We will revise, and soon, your hApps will be able to create real networks over the real Internet!

Our aim is to get a working MVP into developers’ hands as quickly as possible.

“First make it work, then make it right, and finally, make it fast.” Stephen Johnson and Brian Kernighan

4. Development Container: You can Now Persist Chains to Disk

Running hApps have been able to store their source chains to disk for a while now, but until recently, hApps in development mode have kept everything in memory. This works for most testing scenarios, but can be frustrating for end-user testers because most apps only become useful after a history has been built. Now `hc run` (the replacement for `hcdev`) allows you to specify whether to persist chains to disk.

5. Massive Speed-ups in Build Pipeline: Happier, more productive core developers

Any development project of sufficient size and importance needs to be tested regularly for errors. Doing this manually becomes painful. Instead, you need to compile your code and let your tests run, which means you’re out of commission for a while. Most projects turn to ‘continuous integration’ services pretty quickly, in which dedicated servers compile and test the ‘official’ version of the code whenever anyone publishes changes.

Ironically, this productivity-improving measure has been slowing us down. Important changes were taking nearly an hour to test, each getting backed up in the queue, waiting for the previous changes to pass the tests. We reported some speedups in Dev Pulse 04, but we’re happy to say that we’ve now managed to speed up our whole pipeline, up to 5× improvements in some places! This has mostly come through the effective use of caching for unchanged components.

Why does this matter?

A faster build pipeline can process more jobs. More processed jobs means more features integrated, and happier developers. Happier developers means more elegant and clean code.

6. We want your feedback!

Please take 3.5 minutes to take our Dev Pulse Survey, and help us name a crucial piece of Holochain!

Networking DHT ready for implementation

What’s Coming Up

0.0.3 Developer Preview Rust Release

Connect with the developer community