Building Lightning into Bitrefill

Justin Camarena

https://www.bitrefill.com/

What is Bitrefill? Originally prepaid phone credits seller in 107+ countries, recently expanded into gift cards.

The Path to Lightning Integration

Hired to remove Bitrefill’s dependence on third party APIs, essentially build btcpayserver before it existed.

Scaling Issues

Merchants were hit hard due to fee spikes and backlogs. Bitrefill was affected due to many low value products in some countries. Bitrefill started using SegWit very early, and started accepting altcoins because some users wanted to use them. Started integrating Lightning.

Actually Integrating Lightning

Original backend was btcd + LND. Justin was a tester for LND for over a year before. Nodejs backend + LND’s gRPC interface.

Application backend calls out to bitcoin backend api server when generating a lightning order or deposit invoice. Memo uses backend app’s UUID. Application backend keeps state of orders being paid or refunded. Expiration is handled on the lightning end which is great to avoid payments after expiration. Much better UX.

SubscribeInvoice on the bitcoin backend service, sending relevant notifications to the application backend, resends notifications using listinvoices. Data passed is memo, amount, settled, and amount paid in satoshis.

Deploying Lightning

Originally testnet only. This was before beta and shortly after segwit activated, working in 2017.

Challenges integrating Lightning

Unstable, bugs, money loss on testnet, wallet issues out of state (ghost UTXOs with LND), Interoperability issues (channels force closing due to protocol disagreements or fees). Sometimes eclair, clightning, and LND worked together, sometimes not.

Lightning Nodes

Merchants should be routing nodes, not everyone agrees on this. Bitrefill is a routing node, already running 24/7, suited to route. Non-routing merchants right now are at mercy of routers for UX.

Liquidity Lightning

Incoming capacity is not a problem, if anything Bitrefill has more than enough incoming capacity. Issues offloading funds due to exchanges that don’t use lightning.

Avoiding Dust UTXO

Minimum incoming channel 0.01 btc has helped avoid small channels that could more likely close with small local balances. Can not actually avoid small utxos.

Integrating Lightning Payments

Focus on receiving pmts, and refunding on-chain initially. Stealing funds via routing fees with refunds. Receiving is easy, sending is hard!

Fee Siphoning Attack

Attacker pays for service that fails and requires a refund. Attacker has originating node and routing node, attacker sets up imbalanced channels and gouge the routing fee on the refund. This can be automated and scam the merchant.

Mitigations

Rate limit refunds to nodes but that may cause problems for those using a shared custodial node. Fee limits do not solve this issue only limit fund loss to a slower rate. Must pass on routing fees to the user, refund x satoshis less and use that for routing fees. Before any service/exchange integrate sending lightning pmts, implement fee limiting, need best practices to develop.

Other Work

Automatically restarting LND after crashes with supervisord. Migrated node to bitcoind. Lightning deposits and withdrawals.

There’s hope!

Even with all the issues, they can be fixed or mitigated. Many of them have already been fixed or are being worked on right now. Lightning will only get better with recent improvement proposals, maturity of documentation, and new developers being onboarded.