The Bitcoin network has been seeing spiralling fees and slow confirmations due to congestion in the mempool.

Zebpay struggled to keep up with customers transfers and customers complained about the unreliability in fees and confirmations. The average fees that we were paying initially was 100 bits , and we had increased the average fees to 200 bits to ensure better confirmations. We found that during peak clogged times, even though we were charging 500 bits from our customers, we were still paying additional fees from our pocket to ensure that transactions got confirmed.

The Problem

Since Zebpay uses a hosted wallet, our fees cannot be compared to customer wallets such as Mycelium or Core. Those wallets can construct the exact utxo when users initiate a transaction. For Zebpay, lots of users can initiate a transaction at the same time and it is tough to predict the utxo set. The unreliability in mempool makes it difficult to predict the actual fees that customers would have to pay at the time they initiated during the transaction. We also use a multi-sig wallet for increasing security of funds. Multi-sig inputs increase the size of transactions as it requires more signatures and hence more data.

Most of the funds are moved to a multi-sig cold storage

We store most of our coins in a multisig hardware wallets for security reasons and maintain a hot wallet with a minimum possible amount. That’s the reason users see their transactions initiated from a different bitcoin address than the one where they received coins.

Inputs are expensive:

In a bitcoin transaction, if you have more inputs it requires more bytes and hence more fees to get confirmed. So essentially number of inputs (unspents) always determine costs.

For timely delivery, transactions require a confirmed spendable input to send as an output to a bitcoin address. We have to ensure that there are confirmed inputs in our hot wallet which can be used to process outgoing transactions.

If we do not have sufficient inputs, we cannot construct a transaction. So we either have to wait for inputs to get confirmed from previous change or get new inputs into the hot wallet from the storage wallets.

The problem with Fixed fees:

Since we use a hot wallet, we take user’s requests to process withdrawal first and then process it afterwards in secure environment. In the meantime both mempool and unspents in our hot wallet may have changed making previous calculations obsolete.

Companies using hosted wallets sometimes pay out of their pocket to keep a common interface and charge fees rationally.

Cost of Stuck Transactions:

The cost of a stuck transaction is much higher for a hosted wallet as customer expects that the transaction goes through with the fees he had confirmed but because of the unreliability in the mempool, if the mempool clogs up, Zebpay will have to bear the additional fees to ensure confirmation.

The cost of customer support in case of stuck transaction is higher than the actual fees of transaction.

Other Bitcoin exchanges/businesses that use hosted wallets like us will try to pay higher fees competing with everyone to make sure their transaction goes through. This competition during a clogged mempool makes prediction of fees even more complicated adding to the uncertainty of block confirmations.

The cost of transaction not getting confirmed also results in a bad user experience and everyone wants to avoid that leading to a situation where everyone overpays. That resulted in paying fees out of our pocket which became a significant expense. Due to these set of problems, the Zebpay engineering team received a mandate to focus on decreasing fees and increasing the confirmation time.

Implemented Solutions:

The Zebpay engineering team took upon this challenge and analysed past data and mempool confirmations to come up with different solutions.

Optimization of input availability

Outputs are created from different inputs

We don’t know what our users will request as outgoing transaction amount at a particular point of time. So whenever we process outgoing transaction we try to create an input that is greater than the total amount being transferred out. This is tricky and as we may be spending huge amounts in fees due to combining small value unspents to process bigger amounts.

We performed an analysis of past data of user’s transfers to predict the average output. As per our data analysis, we noticed that for higher amounts, we needed more unspents. For example if you want to send 0.1 BTC there are high chances that we may find unspent which we can satisfy this. But when you send 10 BTC there are high chances that we will need to use more than one, sometimes huge number of, inputs to satisfy this amount. That’s why we shifted to amount based fees. We analyse existing fees we are paying, divide them in slabs and pass fees pro-rata basis to all users.

Slab-wise fees

Zebpay’s slab based fees structure

We changed the setup of BTC fee to a slab-based fees. India is a country where users can transact in very small amounts of BTC and it becomes tough for users to pay fee that is almost equivalent to the total amount they are transferring. So we implemented slab wise feee where people would pay a fee that is dependent on the transaction amount. In this way, people transacting lower amounts would pay lower fee and people transferring higher amounts would pay higher fee which would subsidise transfers of lower amounts. For smaller transactions, we use smaller number of UTXOs which result in smaller transaction size.

Batch transactions

We batch all transactions every 5 minutes and create one transaction that would take 1 input from our hot wallet and the outputs would contain all the addresses that users wanted to transfer to. This ensured that we could pay a high fees for one transaction that combined all transactions during the 5 minute period. We also keep checking mempool congestion and during times when the mempool is not clogged, we consolidate all utxos of our wallets to better manage our wallet with lesser utxos.

Segwit implementation

We started using Segwit on our hot wallet as it gave us two advantages — transaction would get priority in mempool, and segwit fees are at a discount compared to non-segwit transactions.

We received tremendous help from Jameson Lopp and Andreas M. Antonopoulos who helped us through the whole implementation.

Today, Zebpay has a world class solution where we ensure that users transactions get confirmed within 2 blocks at a low fee.