Smart Nodes

Following up on the previous development update, more work has been done on the smart node service scripts. Rocket Pool isn’t only made of smart contracts, but also a network of nodes that can listen to the main smart contracts and receive instructions from those contracts. These scripts also allow the smart nodes to check in with the main contract on a regular basis to report on their server load which helps with load balancing users who stake with Rocket Pool + more.

These smart node scripts run as daemons (background processes) on an Ubuntu server running a full Ethereum node. The smart node services are designed to be client agnostic, so we can add even more redundancy to the network by not relying on just one node client. Be it a Parity node or Geth node, they can run along side the node client and listen for instructions from the main Rocket Pool smart contracts + allow an extra level of monitoring not seen on any Ethereum node to date.

One of the most important aspects of Rocket Pool will be node up time. Casper will punish validating nodes that are not online when they should be, this punishment is referred to as an “inactivity leak” which slowly drains the deposit of any validator at approximately twice the rate at which the validator would earn interest, although the exact formula is still being worked on.

Liveliness and Monitoring

Node client software can sometimes be unpredictable, from crashes to frequent updates, it’s important to make sure liveliness of the node is maintained, so if it does go down for any reason, it is quickly started again. The same applies for our smart node services.

To ensure any client or service that Rocket Pool relies on maintains a very high level of liveliness, both are managed using PM2, an advanced production process manager for Node.js. If either the client or services crash, they are restarted automatically and notifications of the crash are reported directly to Rocket Pool regardless if the services immediately resume operating correctly and that’s just the start…

Using Keymetrics we have designed custom monitoring probes that are run by our smart node processes. These probes allow a whole new level of monitoring on Ethereum nodes.

An early preview of just a single smart nodes monitoring dashboard.

A smart probe allows us to monitor every smart node on the Rocket Pool network from a single dashboard, regardless of where that node is hosted. If it’s on AWS, Rackspace or a laptop in grannies basement, doesn’t matter.

At a glance we can see if each node is currently synced with the blockchain, what block the node is currently processing, the highest block that node can see, how many peers the node has, if that node is connected to the Ethereum network, the last checkin that node did with the Rocket Pool smart contracts, the time of that checkin and the nodes current sever load.

Logs of the smart node processes can be viewed in real time.

With our smart probes also comes active alerts. There’s many issues that can effect a node in various ways, from the common crashes to more irregular issues such as the node losing sync with the blockchain due to peers going offline or that node having network issues. With these active alerts, if a node even loses sync with the blockchain for just a tiny 15 seconds, an alert is sent to Rocket Pool for investigation. Similar alerts will be sent for nodes who have a small amount of peers, nodes using too much memory/cpu and a lot more.