Hi all,

We’ve had some significant downtime this week at BrainBlocks, and we wanted to post a quick write-up to let you all know the state of things. We’re back online, and in good shape; but here’s what happened.

On monday, our team encountered an issue with our Nano node that led to it crashing and falling out of sync. We immediately paused all payment processing until the problem could be resolved. Once all payments were paused, we worked to identify what was causing the issue and we found that our node’s data files had become corrupt, and blocks were unable to be processed properly by the Nano node. After successfully re-bootstrapping our Nano node and getting back into sync with the Nano network, we decided to resume processing payments.

Upon resuming the processing, we encountered an issue with how our BrainBlocks core system talks to the Nano node. A small number of payments were becoming stuck in our system and were not being sent or refunded to their destinations, due to the Nano node’s wallet system becoming unresponsive under high volume. After we identified that payments were becoming stuck, we decided to pause all of the processing of payments again until the problem could be resolved. We spent the next day or so working closely with the core Nano team, to figure out a more efficient way to write blocks to the network, and update our processing system to avoid relying on wallets.

With our processing system refactored, we cycled over all transactions from the past four days to make sure that all funds arrived at their correct destinations. Now that all of those the payments have been processed, we’ve re-opened our server back to the public to begin processing new transactions.

We appreciate the support from the Nano team, as well as the community’s patience and understanding. For anyone working on your own Nano service, we’d strongly recommend sticking to raw block generation and publishing if you’re using RPC, rather than relying on some of the higher-level RPC operations for sending and receiving funds — particularly anything involving wallet creation and deletion. So far, we’ve found this to be a lot more of a stable approach, which allows for far better idempotency, statelessness, and distributability.

We’ve also started doing payment processing using dedicated GPUs, so once you’ve checked out on a merchant site, your payments should be processed and sent to the right place even faster.

Finally, in other news, we’ve had a quick refresh on our logo — here’s the new one:

Our latest BrainBlocks platform has been pushed to our production site, and is faster and more stable than ever. So hope you enjoy, and good luck to all of the BrainBlockers out there!