Posted by AllCrypt Staff on March 11, 2014

For dramatic flare, we’ll tell this story out of order.

March 10th. Monday morning. I’m dying. I’m up at what my body says is 5:40am (for those of you who do not live in the US – “springing forward” for daylight savings is a nightmare for a few days as your body adjusts) to get my son to school and see my phone lit up with AllCrypt.com alerts. Oh god. What happened?

RPC ERROR. User XXX withdrawal of .62736153 BTC failed. Insufficient funds.

What? OHMYGOD. WHAT? No! NO!? NO! We’ve been open a week and we’re hacked!? What the what? We have security. A LOT of it. We didn’t leave any holes anywhere! We read poloniex.com’s bitcointalk.org post and made sure we couldn’t be hit by the same thing! Everything is encrypted! My phone goes apeshit if our security software detects anything abnormal! Why didn’t I hear the alerts! Why didn’t the failsafes work!?

I leap out of bed, and bolt downstairs and practically tear the screen off my laptop opening it up. Thank god for SSD boot drives, I’m in and hitting the admin console in seconds. RPC command. Bitcoind getbalance.

4.83000428

Before everyone goes “Oh my god!” – that’s all we had. Remember, we’ve only been open a week! We haven’t even really advertised yet

Ok… so… it’s all there? I run accounting, and yeah, it’s all there…

So I check the internal accounting of the bitcoind wallet. Run a listaccounts and look for the user’s account.

Balance: .62736152

His actual balance is 1 satoshi less than the database says it is…?

Now, let’s backtrack 3 days…

On Friday evening after a user reported having a -0.00000001 Held for Orders balance, we did some looking and noticed a minor bug – minor enough that it could wait until Monday to fix. You see, when trades are executed, a buy or sell order might only partially fill, leaving a portion unsold. And when an order only partially sells, you still need to take the fees and charity percentages out of that partial trade. Which leaves a possible rounding issue. If you always round down you could execute a big order in lots of small chunks and completely avoid some fees, especially our charity donations which are a mere .05%. If you round up, well, sometimes you overcharge the buyer by a few satoshi. Oops.

We did some testing and saw that about 15 accounts were short by anywhere from 1 to 7 satoshi each. No big deal, we’ll adjust the numbers Monday, fix the code, and refund the overcharges from our system account. Did I say we lost some BTC? It was .00000043 BTC. Maybe we were being sensationalist there…? I guess if we’re going to “lose” BTC it’s good that it was only 2.5% of a penny.

Back to Monday morning. It dawns on me the error we found Friday. I test something out. bitcoind move [system account] [users account] .00000001

3 seconds later his withdrawal processes.

Now the revelation: See, bitcoind, and all other crypto wallets by the fact they are all derivative, have a feature that not many people are aware of. Accounts. bitcoin-qt users in a graphical environment never see or have a use for that feature.

It’s mostly an accounting feature. You can use commands like bitcoind newaddress and add an account name after it, and do things like bitcoind listaccounts and get a listing of all accounts with their balance. You can move funds back and forth using the move command – it doesn’t really do anything other than just keep a visual indicator of each “account’s” balance. During alpha testing, when we were building the trade daemon, we had a bug where it looped in the wrong spot and transferred coins from one user to the other like 14 times before we caught it. The database had verification turned on and stopped the multiple transfers, but the bitcoind accounts had no such restriction. One account had 24 bitcoin, the other had -22.

But, hey, it’s a useful little tool – every time we move coins in the database, we move them on the wallet as well. We thought of it as backup accounting. We could make sure the database matched the accounts in the wallet. Can’t hurt.

We found in previous bitcoind uses on Linux that accounts are a great tool for tracking incoming BTC. One of our team runs a website that accepts BTC, and the backend software generates a new account for each transaction, named with the order number. The backend software can query the wallet and see if the order was paid. When he moves the funds from the wallet to his personal wallet, all the account balances stay the same. Kinda cool that you have that tracking to see which orders were paid for also in the wallet. We even used it for the AllCrypt.com voting – each coin has an account called “Voting for: XXX” and as donations came in, we’d check for donations to coins and add them as votes. When it came time to empty the wallet to disburse to charity, all the coin balances stayed intact, which helped keep the votes in place.

EXCEPT.

Except for one small thing that even we didn’t know until this happened Monday morning

bitcoind sendtoaddress [receiving BTC address] .62736153 would have sent .62736153 BTC to that user’s personal wallet.

We don’t do that.

bitcoind sendfrom [users account] [receiving BTC address] .62736153 sends the .62736153 AND reduces the account’s balance – it’s less bookkeeping.

And it FAILS if the account balance is not sufficient. Our wallet had over 4.8 BTC in it, but this users account had 1 satoshi less than the requested withdrawal.

And it failed. Over and over. Spamming my phone with non-critical error messages (thus my confusion as to where the alarms were. If I had not been panicking over seeing INSUFFICIENT FUNDS over and over, I’d have read the entire error message.).

And NOT withdrawing the BTC.

We tested this on Monday. A lot. If the internal bitcoind user account does not have the necessary funds, it will NOT withdraw, regardless of how much BTC is in the wallet as a whole.

If poloniex.com used user accounts – the 12.3% BTC they lost would not have been lost. The first withdrawal would have processed. Their daemon would then try to process the extra erroneously added withdrawals as well, and the bitcoind server would refuse to send the funds, because “User: asshole_hacker” would have an insufficient balance.

The other exchanges that get hacked, that have their withdrawal system tricked into pulling coins from the cold wallet into the hot to withdraw – it couldn’t have happened if user accounts were used.

No, it won’t stop all hacks. If they get into your user database and log in and start withdrawing – this won’t stop that. If they hack into the server itself and get RPC or command line access to your wallets, and they have the passwords or they are unencrypted (why are your wallets unencrypted? Why is the decryption key stored in a file on the server?) it won’t stop that.

But for a good many ways to break your exchange and trick your daemons into draining your wallets… it can stop that.

Please: All exchanges that are not using user accounts because they seem redundant, or unnecessary, add this extra layer of security. Even if someone breaks your database, breaks your daemons… you still have bitcoind itself refusing the withdrawal.