However, it also raises many problems, for example, the user experience or challenges your business model in multiple ways. In the case of UX, it requires significant knowledge about the used technology by the application, e.g. how to use the Metamask wallet to interact with an application. In the case of the business model, it asks questions how to store sensitive or user data for the regulatory purposes for example in the KYC pipeline.

I understand that Web3 is the goal, but at the current state, performance, and adoption of all DLTs, developers are encouraged to adopt well-known solutions with blockchain augmentations, the so-called Web 2.5. In this post, I want to share how we at Neufund approached creating a backend operating together with the Ethereum network.

Universe and contract interfaces

Two initial problems that we have encountered knew which of our Smart Contracts can we trust and when the contract was active — used be the platform in the given time. We introduced a special contract — the Universe that solves this problem by being the registry of trusted Neufund Smart Contracts. The primary objective of the Universe is being the root of trust by gathering all currently active Neufund contract addresses and tracking the history of the implementation changes. Also, thanks to the way how the Universe is implemented, the operator of the Neufund platform can upgrade used smart contracts.

We achieved this by creating contract interfaces and assigning a unique identifier for each of those interfaces. The Universe stores the mapping of those interfaces into current public Ethereum addresses. The first collection stores contracts that can have only one implementation in a single point in time. The second collection stores contracts that implement the same interface with information whether it is active at this moment. In addition to that, both mappings provide a handful of current address getters of all singletons.

The Universe also logs the changes of each interface change using two events: LogSetSingleton for single implementation smart contracts and LogSetCollectionInterface for a change in the collection of contracts that implements the same interface.

Microservices enhanced to work with Ethereum

The fundamental property of all our microservices is the fact that we have assigned the Ethereum private key and public address, so each microservice is attached to its wallet. We use this service’s property to reuse existing Ethereum cryptography for multiple purposes: to sign and execute transactions with Smart Contracts, to authenticate microservices using eth_sign during communication over message queues or APIs using JWT, to verify all externally used URLs generated services and to encrypt secrets that can only be decrypted by the chosen receiver microservice.

We also decided to apply the following constraints over our microservices:

Each service can read information from non-payable and constant functions of smart contracts using the Universe.

Only one service can transact with a Smart Contract at the same moment.

Services never look for the Ethereum events by itself.

Ethereum Nodes Infrastructure

Another important thing is our Ethereum nodes infrastructure. Having your own trusted Ethereum nodes is extremely important. It requires additional operational costs, but it pays off by being sure that our system interacts with the network correctly and we can trust the data we receive from it. For each microservice, we assign one node as the transactional node that given service will communicate with. Also, each microservice has a list of validation nodes to perform a verification of the transactional node. Those verifications are executed before any reading from or writing to a smart contract to be sure that the transactional node is in sync with the validation nodes hence with the Ethereum network.

Reading the data from Neufund’s Smart Contracts

To read the data from our Smart Contracts, we decided to create a Python class that provides access to the Universe. This class creates clients to the smart contract using an interface or smart contract ABI with their current address from the Universe.

By using this class, the microservice is sure that it is connected to the proper version of the Ethereum Network and uses the ABI of the deployed contract.

Also, the returned connected client is an instance of the web3.py ConciseContract that will be in the future reimplemented as ContractCaller by the web3.py contributors.

Transact with Neufund Smart Contracts

When a microservice wants to perform a transaction on one of our smart contracts it has to get the contract client from Neufund Contracts Connector and switch it to the web3.py Contract using the to_full_contract function. Then the service builds the transaction and signs it using its private key. Before we send the transaction to the Ethereum network, we check the state of the node, to which service is connected — if it is in sync with blockchain, if there is no temporary fork and if all validation nodes are in the same state. We also check if the microservice account has enough Ether to perform the transaction. The last requirement is that there is no pending transaction in the pool for the currently used smart contract. If that is the case, we wait to be sure that all of our transactions are executed in the correct order, and a chance of failing transaction is near zero. Next, we send the signed transaction to the assigned transactional Ethereum node to be mined. When we get the transaction receipt from the transactional node, we wait for a few blocks to avoid a temporal Ethereum fork. Then we ask all of our validation nodes if the transaction is propagated in the exact block. If all steps are correct, we can go to performing the next transaction. Unfortunately, those microservices cannot be scaled horizontally, because of the way how the Ethereum transactions work and our need to be sure that each transaction was executed and propagated into the network. In case of an unexpected microservice shutdown, we created a solution to resume operations from the point of a crash with the correct state. Hence we cannot block our system into trying to start a new operation when there is still a transaction in the pending pool of the network. In unexpected situations, we added a way to invalidate the transaction by signing a new empty transaction with the same nonce. This mechanism should not be used in daily operations nor by an automatic process.

Handling Ethereum events

Getting Ethereum event logs is unreliable in the current Ethereum nodes implementations. Not all nodes are configured to save historical events. Or a version of an ethereum node has a bug which makes the platform miss essential data. In addition to that creating many log filters in Ethereum nodes can decrease their performance. Another performance issue can be the receiving of data from too many blocks at the same time.

We decided to have a single point of truth in our system. One microservice that is responsible for getting Ethereum events from the network and decodes them using current ABIs and publishes them into our platform queue system that is based on RabbitMQ. Then all other services interested in any Ethereum event can handle it on their own. Those services can trust that any received message is valid thanks to the fact that all published messages are signed by the event publisher service private key.

The event publisher uses the Universe deployment block number as a point in time when we start getting events. It builds the structure of Neufund’s smart contracts that were active from that point using two Universe events: LogSetSingleton and LogSetCollectionInterface. Thanks to that structure the publisher only takes events from smart contracts that were authorised by the Universe. Of course, the change can happen inside a block, and the publisher should not treat events before that change and should stop using the events for an interface for contracts.

The publisher is also responsible for sending all events in the correct order based on three properties: block number, transaction index and log index of the published event. Thanks to RabbitMQ’s message order guarantees we know that all events will be received in the same order as published. Microservices interested in a given event create handlers to react accordingly. All events should be treated as idempotent, so getting the same event multiple times cannot change the state of the microservice.

Conclusion

By implementing all of those principles within our backend source code, we established a very efficient and reliable way to communicate with the Ethereum network. Our services are always connected to the trusted and active Smart Contract, immediately respond to the events from the Ethereum network, and still are in the correct state to execute a new transaction. From the developer perspective, it is easy to create a new service. Shortly, we are going to open source our solution alongside our platform solution to develop microservices in Python.