thecomputerscientist



Offline



Activity: 48

Merit: 0







NewbieActivity: 48Merit: 0 The MtGox Debacle Explained February 09, 2014, 06:56:19 PM #1



TL;DR version:



1) The withdrawal problems at MtGox are technical.

2) It is likely that a hacker exploit has taken place.

3) Any damage is likely to be limited.

4) Other exchanges need a heads up and could also be vulnerable.

5) MtGox is going through all erroneous transactions and will update all balances. This is the reason why BTC withdrawals are frozen.

6) Countermeasures need to be taken (for all exchanges)



BTC Withdrawal Problems

------------------------

A couple of weeks ago (around January 26-28) I noticed that things at MtGox were not the way it supposed to be. Normally, withdrawing BTC is an instant process. This time my withdrawals went stuck. MtGox provides an API so that transactions that didn't get through were available for public scrutiny:

I took my stuck transactions which were available in raw format and try to rebroadcast them manually. (MtGox no longer publishes the raw format; they are now redacted for a very good reason.) To my surprise it complained that some of the transaction inputs were already spent. Furthermore, this happened to many of my friends as well. I investigated their transactions as well and tried to rebroadcast them manually, but without luck due to complaints of double spending.

My immediate (now wrong) conclusion was that MtGox F-d up big time and couldn't handle a simple concurrency problem. If several people are withdrawing BTC at the same time it is important to ensure that this is counted as an atomic operation so that coins from the wallet pool are not double spent. It turns out that it was much more interesting than I've first anticipated. Another (wrong) conspiracy theory of mine that MtGox did this intentionally to cover up the fact that they were running low on BTC as they use "fractional reserve bitcoin".



Exchanges and Custom Wallet Software

--------------------------------------

Most exchanges have completely custom bitcoin software. Either they are heavily modified source code of the official client, or everything is written from scratch. To my best knowledge MtGox has written their client completely from scratch. Some people critize them for that, but the standard client is not scalable to an exchange with a million of customers. You must modify the original source so at least the wallet part is going through a more suitable database, and also the built-in security only works for a single customer.

The cons with writing your own custom bitcoin client are of course that you would from time to time become out of sync with the official client. It turns out that this is very problematic.



Erroneous Transactions and Fatal Consequences

----------------------------------------------

Suppose there's something that is inconsistent with MtGox client software with the rest of the bitcoin network. What would be the outcome of that? MtGox would broadcast the transaction to the bitcoin network and miners would reject it, so the transaction becomes stuck. After a couple of days, MtGox gives up because it can't get the transaction published in the blockchain so it returns the balance to the customer. This turns out to be VERY dangerous. BTC should not be returned to a customer without proper investigation.

You may ask why? A hacker can exploit the erroneous transactions broadcasted by MtGox by modifying them manually (so they become consistent with the official bitcoin software) and then rebroadcast them manually her/him-self. If this happens, then the stuck transaction (at MtGox) gets actually through and at the same time the balance is returned to the customer's account. Therefore, the customer has doubled her/his BTC withdrawal attempt. If you repeat this process a couple of times then you can empty MtGox BTC vault without having to hack into their computers.

So what about all those erroneous transactions with "double spending", surely this has nothing to do with the erroneous transactions mentioned recently? At the time the hacker broadcasts the modified (correct) transaction based on MtGox erroneous one, the transaction gets through, but MtGox still thinks the coins are still unspent. After all, it is only MtGox that has the private keys, so it is impossible (in general) that someone else can spend them. Therefore, MtGox still thinks those coins are unspent and trying to reuse them as fresh coins for other transactions. This explains why we had so many transactions that tried to double spend coins.



What is MtGox Doing Now?

-------------------------

First, the hackers that tried to modify the erroneous transactions and rebroadcast them manually are likely identified (MtGox surely knows the name of every customer). Their accounts will likely to be frozen.

Second, MtGox has an accounting mess to clean up. There are many transactions registered as unsuccessful at MtGox that need to be checked whether they actually went through or not. Then MtGox needs to update all the BTC balances. This will likely take a couple of days and this is the main reason why all BTC withdrawals are blocked at this time. Once this is done MtGox will open for BTC withdrawals again.



Lessons Learned and Countermeasures

-------------------------------------

What happened at MtGox can happen at other exchanges as well. So how do we prevent these disasters from happening again in future? I have some proposals,

1) Try to stay close to the official bitcoin client and merge in new changes as soon as possible. Stay updated.

2) Bitcoin Foundation could setup some public servers that always run the latest official version of the bitcoin client. Exchanges should then be able to verify that the transaction is legitimate to the latest bitcoin client before broadcasting them.

3) At an exchange, when a transaction becomes stuck for whatever reason, always check if some other transaction with the same inputs and outputs has already been accepted by the network before returning the customers' balance.

Currently, there's too much Fear Uncertainty and Doubt.TL;DR version:1) The withdrawal problems at MtGox are technical.2) It is likely that a hacker exploit has taken place.3) Any damage is likely to be limited.4) Other exchanges need a heads up and could also be vulnerable.5) MtGox is going through all erroneous transactions and will update all balances. This is the reason why BTC withdrawals are frozen.6) Countermeasures need to be taken (for all exchanges)BTC Withdrawal Problems------------------------A couple of weeks ago (around January 26-28) I noticed that things at MtGox were not the way it supposed to be. Normally, withdrawing BTC is an instant process. This time my withdrawals went stuck. MtGox provides an API so that transactions that didn't get through were available for public scrutiny: https://data.mtgox.com/api/0/bitcoin_tx.php I took my stuck transactions which were available in raw format and try to rebroadcast them manually. (MtGox no longer publishes the raw format; they are now redacted for a very good reason.) To my surprise it complained that some of the transaction inputs were already spent. Furthermore, this happened to many of my friends as well. I investigated their transactions as well and tried to rebroadcast them manually, but without luck due to complaints of double spending.My immediate (now wrong) conclusion was that MtGox F-d up big time and couldn't handle a simple concurrency problem. If several people are withdrawing BTC at the same time it is important to ensure that this is counted as an atomic operation so that coins from the wallet pool are not double spent. It turns out that it was much more interesting than I've first anticipated. Another (wrong) conspiracy theory of mine that MtGox did this intentionally to cover up the fact that they were running low on BTC as they use "fractional reserve bitcoin".Exchanges and Custom Wallet Software--------------------------------------Most exchanges have completely custom bitcoin software. Either they are heavily modified source code of the official client, or everything is written from scratch. To my best knowledge MtGox has written their client completely from scratch. Some people critize them for that, but the standard client is not scalable to an exchange with a million of customers. You must modify the original source so at least the wallet part is going through a more suitable database, and also the built-in security only works for a single customer.The cons with writing your own custom bitcoin client are of course that you would from time to time become out of sync with the official client. It turns out that this is very problematic.Erroneous Transactions and Fatal Consequences----------------------------------------------Suppose there's something that is inconsistent with MtGox client software with the rest of the bitcoin network. What would be the outcome of that? MtGox would broadcast the transaction to the bitcoin network and miners would reject it, so the transaction becomes stuck. After a couple of days, MtGox gives up because it can't get the transaction published in the blockchain so it returns the balance to the customer. This turns out to be VERY dangerous. BTC should not be returned to a customer without proper investigation.You may ask why? A hacker can exploit the erroneous transactions broadcasted by MtGox by modifying them manually (so they become consistent with the official bitcoin software) and then rebroadcast them manually her/him-self. If this happens, then the stuck transaction (at MtGox) gets actually through and at the same time the balance is returned to the customer's account. Therefore, the customer has doubled her/his BTC withdrawal attempt. If you repeat this process a couple of times then you can empty MtGox BTC vault without having to hack into their computers.So what about all those erroneous transactions with "double spending", surely this has nothing to do with the erroneous transactions mentioned recently? At the time the hacker broadcasts the modified (correct) transaction based on MtGox erroneous one, the transaction gets through, but MtGox still thinks the coins are still unspent. After all, it is only MtGox that has the private keys, so it is impossible (in general) that someone else can spend them. Therefore, MtGox still thinks those coins are unspent and trying to reuse them as fresh coins for other transactions. This explains why we had so many transactions that tried to double spend coins.What is MtGox Doing Now?-------------------------First, the hackers that tried to modify the erroneous transactions and rebroadcast them manually are likely identified (MtGox surely knows the name of every customer). Their accounts will likely to be frozen.Second, MtGox has an accounting mess to clean up. There are many transactions registered as unsuccessful at MtGox that need to be checked whether they actually went through or not. Then MtGox needs to update all the BTC balances. This will likely take a couple of days and this is the main reason why all BTC withdrawals are blocked at this time. Once this is done MtGox will open for BTC withdrawals again.Lessons Learned and Countermeasures-------------------------------------What happened at MtGox can happen at other exchanges as well. So how do we prevent these disasters from happening again in future? I have some proposals,1) Try to stay close to the official bitcoin client and merge in new changes as soon as possible. Stay updated.2) Bitcoin Foundation could setup some public servers that always run the latest official version of the bitcoin client. Exchanges should then be able to verify that the transaction is legitimate to the latest bitcoin client before broadcasting them.3) At an exchange, when a transaction becomes stuck for whatever reason, always check if some other transaction with the same inputs and outputs has already been accepted by the network before returning the customers' balance.

steadymobbin



Offline



Activity: 33

Merit: 0







NewbieActivity: 33Merit: 0 Re: The MtGox Debacle Explained February 09, 2014, 07:29:31 PM #3 Quote from: thecomputerscientist on February 09, 2014, 06:56:19 PM



TL;DR version:



1) The withdrawal problems at MtGox are technical.

2) It is likely that a hacker exploit has taken place.

3) Any damage is likely to be limited.

4) Other exchanges need a heads up and could also be vulnerable.

5) MtGox is going through all erroneous transactions and will update all balances. This is the reason why BTC withdrawals are frozen.

6) Countermeasures need to be taken (for all exchanges)



BTC Withdrawal Problems

------------------------

A couple of weeks ago (around January 26-28) I noticed that things at MtGox were not the way it supposed to be. Normally, withdrawing BTC is an instant process. This time my withdrawals went stuck. MtGox provides an API so that transactions that didn't get through were available for public scrutiny:

I took my stuck transactions which were available in raw format and try to rebroadcast them manually. (MtGox no longer publishes the raw format; they are now redacted for a very good reason.) To my surprise it complained that some of the transaction inputs were already spent. Furthermore, this happened to many of my friends as well. I investigated their transactions as well and tried to rebroadcast them manually, but without luck due to complaints of double spending.

My immediate (now wrong) conclusion was that MtGox F-d up big time and couldn't handle a simple concurrency problem. If several people are withdrawing BTC at the same time it is important to ensure that this is counted as an atomic operation so that coins from the wallet pool are not double spent. It turns out that it was much more interesting than I've first anticipated. Another (wrong) conspiracy theory of mine that MtGox did this intentionally to cover up the fact that they were running low on BTC as they use "fractional reserve bitcoin".



Exchanges and Custom Wallet Software

--------------------------------------

Most exchanges have completely custom bitcoin software. Either they are heavily modified source code of the official client, or everything is written from scratch. To my best knowledge MtGox has written their client completely from scratch. Some people critize them for that, but the standard client is not scalable to an exchange with a million of customers. You must modify the original source so at least the wallet part is going through a more suitable database, and also the built-in security only works for a single customer.

The cons with writing your own custom bitcoin client are of course that you would from time to time become out of sync with the official client. It turns out that this is very problematic.



Erroneous Transactions and Fatal Consequences

----------------------------------------------

Suppose there's something that is inconsistent with MtGox client software with the rest of the bitcoin network. What would be the outcome of that? MtGox would broadcast the transaction to the bitcoin network and miners would reject it, so the transaction becomes stuck. After a couple of days, MtGox gives up because it can't get the transaction published in the blockchain so it returns the balance to the customer. This turns out to be VERY dangerous. BTC should not be returned to a customer without proper investigation.

You may ask why? A hacker can exploit the erroneous transactions broadcasted by MtGox by modifying them manually (so they become consistent with the official bitcoin software) and then rebroadcast them manually her/him-self. If this happens, then the stuck transaction (at MtGox) gets actually through and at the same time the balance is returned to the customer's account. Therefore, the customer has doubled her/his BTC withdrawal attempt. If you repeat this process a couple of times then you can empty MtGox BTC vault without having to hack into their computers.

So what about all those erroneous transactions with "double spending", surely this has nothing to do with the erroneous transactions mentioned recently? At the time the hacker broadcasts the modified (correct) transaction based on MtGox erroneous one, the transaction gets through, but MtGox still thinks the coins are still unspent. After all, it is only MtGox that has the private keys, so it is impossible (in general) that someone else can spend them. Therefore, MtGox still thinks those coins are unspent and trying to reuse them as fresh coins for other transactions. This explains why we had so many transactions that tried to double spend coins.



What is MtGox Doing Now?

-------------------------

First, the hackers that tried to modify the erroneous transactions and rebroadcast them manually are likely identified (MtGox surely knows the name of every customer). Their accounts will likely to be frozen.

Second, MtGox has an accounting mess to clean up. There are many transactions registered as unsuccessful at MtGox that need to be checked whether they actually went through or not. Then MtGox needs to update all the BTC balances. This will likely take a couple of days and this is the main reason why all BTC withdrawals are blocked at this time. Once this is done MtGox will open for BTC withdrawals again.



Lessons Learned and Countermeasures

-------------------------------------

What happened at MtGox can happen at other exchanges as well. So how do we prevent these disasters from happening again in future? I have some proposals,

1) Try to stay close to the official bitcoin client and merge in new changes as soon as possible. Stay updated.

2) Bitcoin Foundation could setup some public servers that always run the latest official version of the bitcoin client. Exchanges should then be able to verify that the transaction is legitimate to the latest bitcoin client before broadcasting them.

3) At an exchange, when a transaction becomes stuck for whatever reason, always check if some other transaction with the same inputs and outputs has already been accepted by the network before returning the customers' balance.



Currently, there's too much Fear Uncertainty and Doubt.TL;DR version:1) The withdrawal problems at MtGox are technical.2) It is likely that a hacker exploit has taken place.3) Any damage is likely to be limited.4) Other exchanges need a heads up and could also be vulnerable.5) MtGox is going through all erroneous transactions and will update all balances. This is the reason why BTC withdrawals are frozen.6) Countermeasures need to be taken (for all exchanges)BTC Withdrawal Problems------------------------A couple of weeks ago (around January 26-28) I noticed that things at MtGox were not the way it supposed to be. Normally, withdrawing BTC is an instant process. This time my withdrawals went stuck. MtGox provides an API so that transactions that didn't get through were available for public scrutiny: https://data.mtgox.com/api/0/bitcoin_tx.php I took my stuck transactions which were available in raw format and try to rebroadcast them manually. (MtGox no longer publishes the raw format; they are now redacted for a very good reason.) To my surprise it complained that some of the transaction inputs were already spent. Furthermore, this happened to many of my friends as well. I investigated their transactions as well and tried to rebroadcast them manually, but without luck due to complaints of double spending.My immediate (now wrong) conclusion was that MtGox F-d up big time and couldn't handle a simple concurrency problem. If several people are withdrawing BTC at the same time it is important to ensure that this is counted as an atomic operation so that coins from the wallet pool are not double spent. It turns out that it was much more interesting than I've first anticipated. Another (wrong) conspiracy theory of mine that MtGox did this intentionally to cover up the fact that they were running low on BTC as they use "fractional reserve bitcoin".Exchanges and Custom Wallet Software--------------------------------------Most exchanges have completely custom bitcoin software. Either they are heavily modified source code of the official client, or everything is written from scratch. To my best knowledge MtGox has written their client completely from scratch. Some people critize them for that, but the standard client is not scalable to an exchange with a million of customers. You must modify the original source so at least the wallet part is going through a more suitable database, and also the built-in security only works for a single customer.The cons with writing your own custom bitcoin client are of course that you would from time to time become out of sync with the official client. It turns out that this is very problematic.Erroneous Transactions and Fatal Consequences----------------------------------------------Suppose there's something that is inconsistent with MtGox client software with the rest of the bitcoin network. What would be the outcome of that? MtGox would broadcast the transaction to the bitcoin network and miners would reject it, so the transaction becomes stuck. After a couple of days, MtGox gives up because it can't get the transaction published in the blockchain so it returns the balance to the customer. This turns out to be VERY dangerous. BTC should not be returned to a customer without proper investigation.You may ask why? A hacker can exploit the erroneous transactions broadcasted by MtGox by modifying them manually (so they become consistent with the official bitcoin software) and then rebroadcast them manually her/him-self. If this happens, then the stuck transaction (at MtGox) gets actually through and at the same time the balance is returned to the customer's account. Therefore, the customer has doubled her/his BTC withdrawal attempt. If you repeat this process a couple of times then you can empty MtGox BTC vault without having to hack into their computers.So what about all those erroneous transactions with "double spending", surely this has nothing to do with the erroneous transactions mentioned recently? At the time the hacker broadcasts the modified (correct) transaction based on MtGox erroneous one, the transaction gets through, but MtGox still thinks the coins are still unspent. After all, it is only MtGox that has the private keys, so it is impossible (in general) that someone else can spend them. Therefore, MtGox still thinks those coins are unspent and trying to reuse them as fresh coins for other transactions. This explains why we had so many transactions that tried to double spend coins.What is MtGox Doing Now?-------------------------First, the hackers that tried to modify the erroneous transactions and rebroadcast them manually are likely identified (MtGox surely knows the name of every customer). Their accounts will likely to be frozen.Second, MtGox has an accounting mess to clean up. There are many transactions registered as unsuccessful at MtGox that need to be checked whether they actually went through or not. Then MtGox needs to update all the BTC balances. This will likely take a couple of days and this is the main reason why all BTC withdrawals are blocked at this time. Once this is done MtGox will open for BTC withdrawals again.Lessons Learned and Countermeasures-------------------------------------What happened at MtGox can happen at other exchanges as well. So how do we prevent these disasters from happening again in future? I have some proposals,1) Try to stay close to the official bitcoin client and merge in new changes as soon as possible. Stay updated.2) Bitcoin Foundation could setup some public servers that always run the latest official version of the bitcoin client. Exchanges should then be able to verify that the transaction is legitimate to the latest bitcoin client before broadcasting them.3) At an exchange, when a transaction becomes stuck for whatever reason, always check if some other transaction with the same inputs and outputs has already been accepted by the network before returning the customers' balance.

Hey thanks for the update! How do you have access to this info? Hey thanks for the update! How do you have access to this info?

notig



Offline



Activity: 294

Merit: 250







Sr. MemberActivity: 294Merit: 250 Re: The MtGox Debacle Explained February 09, 2014, 07:36:59 PM #4 Quote from: smoothie on February 09, 2014, 07:01:38 PM You failed to mention the possibility that they are lying about it being a technical problem and that they do not have all the funds they claim to (I.e. Solvency).



Don't be naive to think that this is not a possibility.



Ponzi operators work in the exact same manner. They try to buy more time with vague excuses.



So the proper way to run a ponzi scheme is to broadcast failed transactions for all the world to see? Thanks that is good to know. The brilliance of people here is staggering. So the proper way to run a ponzi scheme is to broadcast failed transactions for all the world to see? Thanks that is good to know. The brilliance of people here is staggering.

steadymobbin



Offline



Activity: 33

Merit: 0







NewbieActivity: 33Merit: 0 Re: The MtGox Debacle Explained February 09, 2014, 07:56:44 PM #6 Quote from: thecomputerscientist on February 09, 2014, 07:39:31 PM Thanks for your support. Unfortunately, I cannot reveal my sources.



Understood. Is it safe to say that for someone with some BTC in Gox, they should be somewhat comforted in the fact that the problem is identified and being worked on and they will receive their coins back in the near future? Understood. Is it safe to say that for someone with some BTC in Gox, they should be somewhat comforted in the fact that the problem is identified and being worked on and they will receive their coins back in the near future?

HairyMaclairy



Offline



Activity: 1218

Merit: 1962





Degenerate bull hatter & Bitcoin monotheist







LegendaryActivity: 1218Merit: 1962Degenerate bull hatter & Bitcoin monotheist Re: The MtGox Debacle Explained February 09, 2014, 09:49:22 PM #9 Have you considered that Gox could be turned into a Ponzi scheme by the technique described above.

infoman



Offline



Activity: 42

Merit: 0







NewbieActivity: 42Merit: 0 Re: The MtGox Debacle Explained February 09, 2014, 09:57:28 PM #10 Hi OP, I was told there wasa software could hack into gox account and "create" BTCs on the fly using fractions of micro transactions other users are doing.

there was a video of it online some time ago.



is there any connection or that is just hoax?



thecomputerscientist



Offline



Activity: 48

Merit: 0







NewbieActivity: 48Merit: 0 Re: The MtGox Debacle Explained February 09, 2014, 09:59:06 PM #11 Quote from: HairyMaclairy on February 09, 2014, 09:49:22 PM Have you considered that Gox could be turned into a Ponzi scheme by the technique described above.



It's not impossible of course, but I believe that the hacker attacks have caused limited damage. The main reason is that MtGox only allows 100 BTC withdrawals per day. Each unsuccessful withdrawal takes a week (before the balance is returned). This started two weeks ago and MtGox got warned early enough and took countermeasures before everything got out of control. This is the reason why they frozen all BTC withdrawals. I believe they'll be able to clean up everything, but it will take some time. It's certainly a very interesting situation. There are multiple different problems that magnifies the situation. Again, other exchanges could suffer from the same problem. As far as I understand, all the exchanges are currently being contacted and warned about the current situation.

It's not impossible of course, but I believe that the hacker attacks have caused limited damage. The main reason is that MtGox only allows 100 BTC withdrawals per day. Each unsuccessful withdrawal takes a week (before the balance is returned). This started two weeks ago and MtGox got warned early enough and took countermeasures before everything got out of control. This is the reason why they frozen all BTC withdrawals. I believe they'll be able to clean up everything, but it will take some time. It's certainly a very interesting situation. There are multiple different problems that magnifies the situation. Again, other exchanges could suffer from the same problem. As far as I understand, all the exchanges are currently being contacted and warned about the current situation.

thecomputerscientist



Offline



Activity: 48

Merit: 0







NewbieActivity: 48Merit: 0 Re: The MtGox Debacle Explained February 09, 2014, 10:22:55 PM #15 Quote from: aahzmundus on February 09, 2014, 10:19:00 PM How could the transaction be changed without needing to be re-signed? I lack the technical knowledge to understand how what you describe is possible.



You don't touch the actual signature, but there are meta-data around it. In a recent version of the official bitcoin client the format of that meta-data has been tightened so the transaction data provided by MtGox is now being rejected by the latest official version. A hacker can then take the rejected raw txdata provided from MtGox, patch it and rebroadcast it. It will get through, but MtGox still thinks it is invalid and returns balance.

You don't touch the actual signature, but there are meta-data around it. In a recent version of the official bitcoin client the format of that meta-data has been tightened so the transaction data provided by MtGox is now being rejected by the latest official version. A hacker can then take the rejected raw txdata provided from MtGox, patch it and rebroadcast it. It will get through, but MtGox still thinks it is invalid and returns balance.