A peak under the hood

Firewall… what firewall?

Ensuring that Hosts could live anywhere has been a key focus for the engineering team.

The obstacles faced by a server running on a home network traditionally make it very difficult to create a reliable entry point into a networked service. For a start, most ISPs use an IP pool, meaning that your home IP address regularly changes without warning. Even if with a fixed IP, you still need to configure port forwarding on the router, and manually assign a single port per networked machine.

Remote Procedure Calls (RPC) is a protocol that allows for the distribution of procedures across a network of machines. We’ve chosen to make use of GRPC — a universal RPC framework — to distribute traffic from a single Gateway to multiple Hosts within the network. Because GRPC uses standards-based HTTP/2 as transport, it easily traverses proxies and firewalls, enabling it to be setup in a home environment without the need for router/firewall configuration — a key requirement for the Host component in the network.

Distributed configuration coordination

Keeping an eye on resources is an essential requirement for any distributed infrastructure. To manage hardware, the network needs realtime information on current capacity, granular system load information and application integrity.

Apache Zookeeper is an extremely simple, reliable application for handling the distributed synchronisation and coordination of persistent and ephemeral configuration data. Each Stargate in the DADI network Hosts a Zookeeper server, creating persistent configuration directories for Gateways and Hosts to interact.

When Gateways or Hosts connect to a Stargate, they create their own unique ephemeral node. And each node replicates it in appropriate directories according to status, for instance if a Gateway needs more Hosts, it replicates itself into ‘Awaiting Hosts’. Any new or idle Hosts in the network watch this directory for changes and connect to any new candidates that join the directory.

Adding apps into the mix

For the first iteration of the network we decided to deploy apps to DADI Host with Docker. In the next phase we’ll be moving the containers onto an encrypted VMM.

When a Host launches, the Host system app checks for running Docker processes, identifying those that are running DADI services, such as CDN, Web, API and Publish.

The Host polls the Gateway request queue for requests pertaining to the application running on the Host machine.

App distribution

We’re working on methods to handle distributing apps within the network. Each consumer app will have unique system and bandwidth requirements, which should be configurable to allow for growth/contraction over time. Our approach to this remains work in progress.

Want to get involved?

We’re really happy with the performance we’ve achieved so far on the testnet. So happy that we’d like to bring some of the community into the next phase of network testing.

To this end we will opening up applications to be part of the testnet. We will be looking for individuals from around the world to join the network — successful applicants will be provided a low-powered computer with DADI Host preinstalled and capable of running DADI CDN and DADI Web.

We’ll be providing more details regarding the community involvement project in a future post. Keep an eye on our usual social channels — plus the news feed in your account page at dadi.cloud — to be the first to hear about it.