got stuck debugging things so it didnt crash, even while syncing BTC and BTCD at the same time.



harder was getting it to transition from historical syncing to realtime mode and it appears to be doing that now. pretty cool to see it get the latest block for both BTC and BTCD as they come in.



I also added support for getting the readonly files from a different directory and if not there, get it from the current location. This way, once you get a validated read only dataset, that takes precedence and then you can keep updating it with normal files. At some point, a way to create a single compressed volume is probably useful, but for now it shouldnt be so important.



Since my VPS has 32GB of RAM, without compressing the data to 20GB, it doesnt fit and so starts swapping and becomes 100x slower. When it is in RAM, it can reconstruct the entire account balances in 2 minutes, but I made it so even the address balances and utxo can go into the readonly section.



Now, if you are paying attention, you will be saying "wait! you cant put the address balances or utxo into the read only section."



Well, turns out it is probably the most efficient thing to do!



After struggling for a while with how to efficiently deal with the realtime updates of the new blocks coming in, the problem is that each vin can spend any prior unspent vout, from any block/bundle. If I could use a r/w memory map, then maybe having the full dataset in a r/w file would be possible, but chrome OS only supports readonly memory maps, plus once you allow r/w memory map, then any stray memory write could corrupt the dataset. So for both safety and portability, r/w memory map is not good.



I am seeing around 10 million vins per recent bundle, so even with 100 bytes per vin would fit in a GB of RAM and on average half that. Now this is big, but not too big and smaller than the lifetime balance/spent state. If space was really important, each vin could be encoded into 6 bytes or so and combined with a bitmap would be very small, but searching for things would be slow and I dont like slow when it could be 1000x faster by using a bit of RAM.



What I did was make an in memory hash table with all the information that would be in the bundle boundary balance/utxo files. However, only the blocks from the current bundles will put things into this in RAM hashtable. This means that regardless of how large the blockchain gets, all except the current bundle's utxo even has to be in RAM. And from prior posts, we know that data in the readonly section is much lower cost as it can be distributed via torrents.



So now needing to be able to do up to date listunspent is the gating issue. I figured out a way to make things much simpler than the prior 100*n + 10*m + 1 to 9. I will make it so as each block comes in, the ramchain data structure can be incrementally updated and by not saving to disk, this data can be accessed as is. It will actually be the identical data structures as all the readonly datasets, so very little changes are needed to support it. Just need to make sure there are no assumptions that all bundles are full.



[read only bundles] + [partial realtime bundle] + [utxo RAM tables]



With the above three datasets, it will allow for parallel searching address balances as of any block, in addition to all the normal block explorer and of course all the blockchain level RPC. For those who are worried about data consistency and atomic updates, I have a single thread handling RPC requests and updating the utxo RAM tables and I will have that thread do the updating of the realtime bundle. This way, when things are out of sync, there is nothing responding to RPC. Atomicity by avoidance of non-atomic timing, which doesnt even require setting locks.



I have a contractor working on generating a snapshot of all address balances via a modified bitcoind, seems it takes 5+ hours to just scan the DB after it is already synced. I guess I shouldnt feel so bad it now takes an hour to fully sync iguana.



After I get the incremental updating debugged, then I will be able to compare against independently calculated balances to validate the integrity of the data.



The only issue I can see left would be the downtime as a bundle boundary is reached. It will take a minute or so to create the new bundle and then update the balance files. Maybe having a bit of downtime is ok, but of course people will complain, so I will need to be able to continue with more than a full bundle worth of "partial" utxo data, but maybe I can just create bundles as they can be created and until things are restarted, it just keeps adding to the RAM utxo data. Then on restart, I would need to check for the non-readonly data to get the latest balance/utxo data.



OK, I think that works.



James