5 builds later...



it was a bit tricky to find the right condition to allow safe resolving of the vin txids. Initially such are just stored as full txid (yikes! no compression), but as soon as all the previous bundles are there, then all vins should be able to be resolved. At least I have confirmed if a txid cant be found, it complains a lot



With everything happening in parallel, it is easy to end up thinking a bundle is ready before it is totally ready. So I had to combine the "1st" field along with just waiting 3 seconds after a bundle is written out. That gives time for avoiding race conditions, I am seeing some cases where writing the data out to file is not completed, even though it is memory mapped. When dealing with multiple threads, it does take time to get a fully synchronized state across all threads, especially when dealing with buffered files.



I just realized with 80% of vins resolved within the bundle, it could well be worth doing the verify of those during bundle creation. Since I dont need any extra fields to denote an external reference, it wont create chances for inconsistent data.



1st.52 N[202] Q.-319 h.402586 r.128003 c.0:128000:0 s.225966 d.64 E.64:243166 M.113998 L.402584 est.64 28.0MB 0:07:58 445.930 peers.127/1024 Q.(0 0)



CPU usage is now around 600%, so a lot more load and not much extra capacity so not sure if doing sig validation will totally mess up the processing speed. With so much data avoiding another pass through would save a lot of time, but if not enough CPU it wont help.



To give an idea of how fast iguana is, even using the serial search through all the bundles, it is doing a search for hundreds of millions of vin txids on-the-fly, during heavy system load.



utxo 10177 spendinds.[49] errs.0 [14.22%] emitted.1836 10.758kb of 10177 | GENERATED UTXO for ht.98000 duration 0 seconds



above is a typical one, I benchmark it but all of the early bundles show 0 seconds as I am doing seconds level timings. I guess I should have used milliseconds. But with 1836 vins fully processed (including writing out the file), means 500 microseconds per search, but it could be a lot less as it all finished the same second it started



utxo 196742 spendinds.[80] errs.0 [21.85%] emitted.78865 462.100kb of 196742 | GENERATED UTXO for ht.160000 duration 10 seconds



got one which took 10 seconds, so it looks like about 120 microseconds per vin, from around the 40% mark. So even from the latest blocks it would be about 250 microseconds per search.



then there is:



utxo 829521 spendinds.[91] errs.0 [20.21%] emitted.80203 469.939kb of 829521 | GENERATED UTXO for ht.182000 duration 1 seconds



So maybe it is heavy system load that caused the slow performance, and the real time is closer to 20 microseconds?



1st.108 N[202] Q.-644 h.402589 r.222002 c.0:222000:0 s.337799 d.111 E.111:355686 M.314000 L.402584 est.64 28.1MB 0:20:04 187.693 peers.127/1024 Q.(0 0)



I checked the compressibilty of the utxo vectors and it compressed by over 90%, so the total compressed size would a very small 30MB, which starts to be small enough to fit in the CPU L* cache! But all that is needed is a bitmap for spent status. Depending on the distribution, using the same sparsehash table method might be best. I think 90% of unspents are spent, so that makes a bitmap pretty sparse and maybe it can be run length encoded, but that makes looking it up a bit more complicated. unless it is put into a compressed filesystem



utxo 1417963 spendinds.[109] errs.0 [14.56%] emitted.154623 905.994kb of 1417963 | GENERATED UTXO for ht.218000 duration 5 seconds



utxo 1525706 spendinds.[111] errs.0 [14.19%] emitted.170555 999.346kb of 1525706 | GENERATED UTXO for ht.222000 duration 161 seconds



clearly overall system activity has big effect on the time for searching txid. following was during 200% CPU load:



utxo 1495166 spendinds.[116] errs.0 [14.81%] emitted.256541 1503.170kb of 1495166 | GENERATED UTXO for ht.232000 duration 2 seconds. so it seems 10 microseconds per search is about right.



1st.118 N[202] Q.-1214 h.402592 r.254002 c.0:254000:0 s.396135 d.127 E.127:416280 M.402591 L.402592 est.64 28.1MB 0:33:39 116.434 peers.128/1024 Q.(0 0)



it seems that there isnt much CPU left, so I will put the sig validation in the background