As you might be aware, we completed the IPLV2 project, which introduced the practical use of KIN tokens to Kik users. In the earlier post, we have presented our new mobile SDKs that streamline working with KIN from mobile applications. In this post, we show what we have done on the server-side and present our ERC20 Python SDK. The purpose of this SDK is to significantly lower the barrier for integration of ERC20 token wallets in Python, and especially in server applications.

When we started the project, we were not sure whether yet another Ethereum SDK for Python was really needed. But after playing with web3.py and other packages, we found that they are still too low-level and do not have the abstractions we want, especially relating to ERC20 tokens. Below are some of the hurdles a developer can face when working with web3.js-style SDKs.

API-related

Basic understanding of internal workings of Ethereum network is required

Good understanding of JSON-RPC API is required, but most of the API is actually not needed for the application developer

Basic understanding of cryptography is required

A lot of boilerplate code for communication, number conversions, encoding, and cryptography is needed

Ethereum-related

Due to the decentralised nature of the network, different nodes usually have a different state, so your network view will depend on the node you work with

Transaction nonce sequentiality is one of the big issues. If your nonce is incorrect, some weird things can happen (Read more on this here)

sequentiality is one of the big issues. If your nonce is incorrect, some weird things can happen (Read more on this here) Node software from different vendors (and even from different versions) differs in the way it supports JSON-RPC API and the protocol implementation issues it has

For example, here is what a developer needs to know in order to send tokens to a recipient:

Know what is Ethereum Contract ABI and how to work with it Understand how to build an Ethereum transaction object Know what to put in transaction data field Know how to calculate transaction nonce Understand what is transaction gas and how to calculate it Know how to sign transactions Know how to RLP encode transactions Know how to send the encoded transactions and handle the resulting errors

As you can see, there is a lot of code that can ideally be done in one succinct method call:

send_tokens(to, amount)

Consequently, after a transaction is sent, it needs to be monitored, which involves another set of JSON-RPC API functions that relates to statuses, filters and logs. And even the seemingly simple parts of the API still need some amount of decoding and data conversions.

High-level SDK

Clearly, there was a need to develop a higher-level SDK, that hides most of the blockchain-related details and eases application development.

Such an SDK has been developed and is now available in this repository:

https://github.com/kinfoundation/erc20token-sdk-python

Internally, the SDK uses the great web3.py package, but the complicated implementation details are now abstracted from the user.

Installation

The SDK is available as a pypi package and can be easily installed using pip:

$ pip install erc20token

Initialisation

To initialize the SDK, you need to provide the following parameters:

JSON-RPC API endpoint URI of your Ethereum node (for example, http://mainnet.infura.io)

The address of your token contract

The ABI of your token contract as json

(optional) either your private key, or a keyfile+password

The SDK has two initialization modes. In the generic mode, the SDK is initialized without a private key:

import erc20token

import json



# Init SDK without a private key (for generic blockchain queries)

token_sdk = erc20token.SDK(provider_endpoint_uri='http://localhost:8545', contract_address='0x04f72aa40046c5fb3b143aaba3ab64d1a82410a7',

contract_abi=json.loads(contract_abi))

In this mode, you will NOT be able to use the following functions:

get_address, get_ether_balance, get_token_balance, send_ether, send_tokens

In a wallet mode, the SDK initialization includes a private key:

# Init SDK with a private key

token_sdk = erc20token.SDK(provider_endpoint_uri='http://localhost:8545', private_key='a60baaa34ed125af0570a3df7d4cd3e80dd5dc5070680573f8de0ecfc1957575,contract_address='0x04f72aa40046c5fb3b143aaba3ab64d1a82410a7', contract_abi=json.loads(contract_abi))

In this mode, all SDK functions are available.

Note that using a clear-text private key is not a sound security practice. As an alternative, the SDK lets you use a password-protected keystore file. The popular MyEtherWallet will create this file for you, or you can create one using a function provided by the SDK:

erc20token.create_keyfile('a60baaa34ed125af0570a3df7d4cd3e80dd5dc5070680573f8de0ecfc1957575', 'my password', 'keyfile.json')

Then, initialise the SDK using this file:

# Init SDK with the keyfile

token_sdk = erc20token.SDK(provider_endpoint_uri='http://localhost:8545',

keyfile='keyfile.json', password='my password', contract_address='0x04f72aa40046c5fb3b143aaba3ab64d1a82410a7',

contract_abi=json.loads(contract_abi))

However, this approach is still problematic, as the password has to be hardcoded in the configuration file. In the production environment, we recommend to use a third-party key management service such as KMS by Amazon in order to keep your private key or password secure.

Getting Account Balance

Even if you intend to work only with your tokens, you still need your wallet to support Ether, because network gas is still paid with Ether. Therefore, the wallet has two sets of account-related functions, one for your tokens, another one for Ether.

Here is how you get token and Ether balances of an account you initialize with a private key:

# Get Ether balance of my account

eth_balance = token_sdk.get_ether_balance()



# Get token balance of my account

token_balance = token_sdk.get_token_balance()

And here is how you get the account balances of another account:

# Get Ether balance of some address

eth_balance = token_sdk.get_address_ether_balance('address')



# Get token balance of some address

token_balance = token_sdk.get_address_token_balance('address')

The balances are returned in Ether and tokens, so numbers like 0.0000000000000123 are not uncommon.

Important Note: The SDK currently only supports tokens with 18 decimals, which is the most common number of decimals for tokens. When using tokens with a different number of decimals, you will need to make your own conversions.

Sending Coin

To transfer your tokens or Ether, you need to make just one simple function call:

# Send Ether from my account to some address. The amount is in Ether.

tx_id = token_sdk.send_ether('address', 10)



# Send tokens from my account to some address. The amount is in tokens.

tx_id = token_sdk.send_tokens('address', 10)

Due to the specifics of the Ethereum protocol, there are differences in the error response when sending tokens and Ether. If you do not have enough Ether, send_ether and send_tokens will raise an exception. If you do not have enough tokens, however, send_tokens will still finish successfully. The faulty token transaction will end up as FAILED on the blockchain, consuming all the supplied gas.

Getting Transaction Status

As a result of sending Ether or tokens, you receive a transaction id, or transaction hash. If you just want to check the status of this transaction, use the following function:

# Get transaction status

tx_status = token_sdk.get_transaction_status(tx_id)

# Returns one of:

# erc20token.TransactionStatus.UNKNOWN = 0

# erc20token.TransactionStatus.PENDING = 1

# erc20token.TransactionStatus.SUCCESS = 2

# erc20token.TransactionStatus.FAIL = 3

Because transaction statuses are sequential numbers, you can use a simple pattern to wait until transaction reaches a certain status:

for wait in range(0, 5000):

tx_status = token_sdk.get_transaction_status(tx_id)

if tx_status > erc20token.TransactionStatus.UNKNOWN:

break

sleep(0.001)

Unfortunately, a transaction status is not fully deterministic and deserves some explanation:

A transaction is initially submitted to the node you are connected to. The node dry-runs the transaction to verify it and rejects it if something is wrong (wrong signature, bad data, not enough Ether and more). If the transaction looks good, it enters the transaction pool on the node and is also sent via a p2p protocol (ÐΞVp2p) to other nodes, where it is verified again and included in their transaction pool. Whenever the transaction is included in the pool, its status is set to PENDING and it becomes a candidate for inclusion in the next block.

As Ethereum nodes race to mine their blocks, there is no guarantee that the transaction will be mined in the next block. It can stay in the PENDING state for some time, until one of the nodes that included the transaction in their candidate block succeeds to mine it. When that happens, the transaction status becomes SUCCESS or FAIL. At this time, however, there is no absolute guarantee that the transaction will have this status forever. There is a small chance that the blockchain has forked and the mined block is on the “shorter fork”. In this case, the block will be disposed, the transaction rebroadcasted to the network and added into some next block. In this unlikely event, your transaction becomes pending after already being mined! Therefore, if you want to be more certain in the outcome of your transaction, you need to consider the number of block confirmations.

Getting Transaction Data

A transaction object contains a lot of information, some of which can be very useful. We have just mentioned that you might need to know the number of block confirmations for the transaction. As another example, consider a case when someone claims to have sent you tokens and gives you a transaction id to prove it. You will need to verify that this transaction was indeed sent from him to you, and contains the amount of tokens you expect. The following function lets you obtain transaction details:

# Get transaction details

tx_data = token_sdk.get_transaction_data(tx_id)

# Returns an erc20token.TransactionData object containing the following fields:

# from_address - the address this transaction was sent from

# to_address - the address this transaction was sent to. For token transactions, this is the decoded recipient address.

# ether_amount - the amount of transferred Ether. 0 for token transactions.

# token_amount - the amount of transferred tokens. 0 for Ether transactions.

# status - the transaction status, see above.

# num_confirmations - the number of confirmations for this transaction:

# -1 if transaction is not found

# 0 if transaction is pending

# >0 if transaction is confirmed

Monitoring Transactions

You can monitor your transactions by polling the blockchain using the functions get_transaction_status and get_transaction_data . But the SDK eases this task by providing a callback-based transaction monitoring infrastructure that uses JSON-RPC API filters.

In order to set up transaction monitoring on your account, first define a callback function that will be called each time the transaction changes state. Then, call the function monitor_token_transactions or monitor_ether_transactions , giving it your callback function and the filters to monitor with (currently only from_address and to_address are possible). When the transaction changes state, your callback will be run. Below is a full example of how to monitor all token transactions sent from your wallet.

# Setup monitoring callback

tx_statuses = {}

def mycallback(tx_id, status, from_address, to_address, amount):

tx_statuses[tx_id] = status



# Monitor token transactions from me

token_sdk.monitor_token_transactions(mycallback, from_address=token_sdk.get_address())



# Send tokens

tx_id = token_sdk.send_tokens('to address', 10)



# In a short while, the transaction enters the pending queue

for wait in range(0, 5000):

if tx_statuses[tx_id] > erc20token.TransactionStatus.UNKNOWN:

break

sleep(0.001)

assert tx_statuses[tx_id] >= erc20token.TransactionStatus.PENDING



# Wait until transaction is confirmed

for wait in range(0, 90):

if tx_statuses[tx_id] > erc20token.TransactionStatus.PENDING:

break

sleep(1)

assert tx_statuses[tx_id] == erc20token.TransactionStatus.SUCCESS

SDK Limitations

Though the SDK aims to simplify the use of ERC20 tokens, it does not completely resolve all the issues inherent to the distributed character of the Ethereum network.

Transaction Nonce

One of the major issues one encounters when working with Ethereum transactions is how to correctly determine transaction nonce. In reality, you can have several application servers each communicating with its own JSON-RPC API instance,. If you work with the same account on all the servers (for example, you have one wallet from which you pay your users), all the SDKs will calculate the nonce at the same time, which results in high probability of nonce collision. So how do you make sure the nonces are sequential and do not collide?

A proper solution would involve some sort of a centralized counter with a locking mechanism. The SDK does not provide such an encompassing solution yet, it only supports one application server working with one JSON-RPC API instance. In the scope of one application server, the locking mechanism ensures that the nonce will be calculated correctly.

What if you have several application servers? Because the nonce is account-related, one of the solutions to handle nonce collision is to shard your accounts, such that each application server handles its own set of accounts.

What if you are running several JSON-RPC API nodes behind a load balancer? In this case, you should enable connection stickiness on the load balancer: stickiness ensures that requests from your application server will always reach the same node. Moreover, because sending a transaction to one node will not be immediately visible on another node, stickiness ensures consistent transaction-state when polling on nodes.

Transaction Monitoring

monitor_xxx_transactions functions use the filter subset of JSON-RPC API. Most public nodes (for example, http://mainnet.infura.io) will have this API subset disabled in order to prevent abuse. You can overcome this by running your own node, or by polling implementation that uses the stateless functions get_transaction_status and get_transaction_data instead.

ERC20 Limitations

The SDK currently supports a subset of ERC20 Token Standard, namely totalSupply , transfer and balanceOf functions. Additional functionality will be added as needed. Your PRs are welcome!

As was already mentioned, the SDK assumes that your token has 18 decimals.

Please do not forget to check the current limitations before you start developing with the SDK!

Conclusion

During our experiments, we found out that building a scalable server application that interacts with the Ethereum network is not a simple task. Our SDK tries to overcome many hurdles, but is still subject to limitations, some of which we hope to resolve in the future. We appreciate your feedback and suggestions to help us get further along in this challenging path.

Check our repos, ask questions and open GitHub Issues. Good luck!

Links: