Pieter Wuille





Offline



Activity: 1064

Merit: 1039







LegendaryActivity: 1064Merit: 1039 Experimental pre-0.8 builds for testing December 09, 2012, 12:18:32 AM

Last edit: December 09, 2012, 01:39:54 AM by Pieter Wuille #1



I've uploaded a new set of builds of bitcoind/Bitcoin-Qt to my



Disclaimer: don't use these for any serious stuff - I am not responsible for destroyed wallets or exploded CPU's.



Testing of these is very welcome.



As the most invasive change is in the block/transaction validation system, this is the most interesting part to test. I am very interested in reports about speed, CPU usage, memory usage, ... during initial block synchronization, in particular on Windows systems. Obviously, crashes or other unexpected behaviour is even more important.



As these builds use the new 0.8 database code ("ultraprune", as I called it before), they will not use the existing block files/database from 0.7.2 or earlier. There are several ways to import the data: Put bootstrap.dat in your data directory.

Start the program with options -loadblock=path/to/old/blk0001.dat -loadblock=path/to/old/blk0002.dat.

If you already have a full 0.8 database (perhaps from a next-test build), you can use the new -reindex option to rebuild your index from scratch, redoing the block validation process, reusing the block data you already have on disk.

option to rebuild your index from scratch, redoing the block validation process, reusing the block data you already have on disk. Obviously, you can also just let it synchronize from network, but as the current block-fetch logic is somewhat of a hack, you may get unlucky, and hit a slow peer, or have downloads that stop for long periods of time. It is advised to use -connect=IP to connect to a single fast peer (you can use my server, 178.18.90.41, for this purpose if needed) for that.

As these builds use parallel script verification (only enabled after block 193000, the last checkpoint), you may wish to control how many threads are used (don't use more than the number of CPU cores you have). By default, it will try to autodetect, but you can use the -par=N option to select the number of threads. One interesting thing you can test is comparing -par=1 with higher numbers, and check the speedup (in particular of the blocks after 193000).



Other options which may be useful to play with: -dbcache=N tweaks the amount of cache used. N is a number in megabytes (default is 25), but higher numbers may result in significant speedups.

tweaks the amount of cache used. N is a number in megabytes (default is 25), but higher numbers may result in significant speedups. -logtimestamps adds timestamps to the output in the debug.log file, which may be very useful to compare speed.

adds timestamps to the output in the debug.log file, which may be very useful to compare speed. -benchmark produces detailed timing information during block validation (not very accurate on Windows, unfortunately). Hello all,I've uploaded a new set of builds of bitcoind/Bitcoin-Qt to my builds page . The new "turbo" build is based on current master on git, plus some more experimental patches (Hal Finney's optimized secp256k1 code , and parallel script verification). The source code for these can be found here (.tgz archive here ).Disclaimer: don't use these for any serious stuff - I am not responsible for destroyed wallets or exploded CPU's.Testing of these is very welcome.As the most invasive change is in the block/transaction validation system, this is the most interesting part to test. I am very interested in reports about speed, CPU usage, memory usage, ... during initial block synchronization, in particular on Windows systems. Obviously, crashes or other unexpected behaviour is even more important.As these builds use the new 0.8 database code ("ultraprune", as I called it before), they will not use the existing block files/database from 0.7.2 or earlier. There are several ways to import the data:As these builds use parallel script verification (only enabled after block 193000, the last checkpoint), you may wish to control how many threads are used (don't use more than the number of CPU cores you have). By default, it will try to autodetect, but you can use theoption to select the number of threads. One interesting thing you can test is comparingwith higher numbers, and check the speedup (in particular of the blocks after 193000).Other options which may be useful to play with: I do Bitcoin stuff.

justusranvier



Offline



Activity: 1400

Merit: 1006









LegendaryActivity: 1400Merit: 1006 Re: Experimental pre-0.8 builds for testing December 09, 2012, 02:47:10 AM #2



Quote time bitcoin-qt -par=8 -connect=192.168.XXX.XXX



real 48m58.429s

user 114m35.786s

sys 2m33.817s



Again with a larger cache.



Quote time bitcoin-qt -par=8 -connect=192.168.XXX.XXX -dbcache=4096



real 42m10.028s

user 109m33.141s

sys 1m9.843s

This is v0.7.1-244-g3ccb06f-beta starting from a completely bare directory, downloading the blockchain from a peer on the same LAN to remove internet connection speed as a factor.Again with a larger cache.

dserrano5



Offline



Activity: 1960

Merit: 1022









LegendaryActivity: 1960Merit: 1022 Re: Experimental pre-0.8 builds for testing December 09, 2012, 02:22:27 PM #5



Code: nice bitcoind -logtimestamps -loadblock=dot-bitcoin/blk0001.dat -loadblock=dot-bitcoin/blk0002.dat -loadblock=dot-bitcoin/blk0003.dat -dbcache=50 -benchmark &

Took 1 hour 57 to reach height 211494 when at that moment the latest block was 211518. As soon as 211519 was found some minutes later the remaining blocks were processed/downloaded/whatever.



4 CPU cores. RAM usage reached 420 Mb RSS, after synchronization it ended up at 375 Mb. Restarting the daemon leaves it at 130 Mb. I've never used -dbcache until now, if that matters. Usual RAM usage until now was 80-90 Mb just after starting, then going up slowly as days passed (like in, 130 Mb after ~10 days). Starting from an empty ~/.bitcoin directory (having only bootstrap.dat and a bitcoin.conf with an rpc password) and running this:Took 1 hour 57 to reach height 211494 when at that moment the latest block was 211518. As soon as 211519 was found some minutes later the remaining blocks were processed/downloaded/whatever.4 CPU cores. RAM usage reached 420 Mb RSS, after synchronization it ended up at 375 Mb. Restarting the daemon leaves it at 130 Mb. I've never used -dbcache until now, if that matters. Usual RAM usage until now was 80-90 Mb just after starting, then going up slowly as days passed (like in, 130 Mb after ~10 days).

goodbc



Offline



Activity: 99

Merit: 10









MemberActivity: 99Merit: 10 Re: Experimental pre-0.8 builds for testing December 10, 2012, 04:51:41 AM #9 I downloaded the bitcoin-0.7.1-244-g3ccb06f-win32-setup.exe, installed on my test laptop (Dual core 2GHz Intel, 2GB DDR, with XP) and, without any other extra parameters, it took ~6:40 hours to process 193395 blocks of transaction history. The cpu use was so far ~ 25%.



After this I had to restart the client (the download apparently stopped) and the cpu usage increased to ~75% (sometimes 97-99%). Later it dropped to ~65% while download speed was ~ 250 KB/s.



A new restart was needed after 9:43 hours. The download bar at the bottom of the window disappeared, hopefully this won't scare a regular user.



Finally, the block chain fully synchronized after ~13:00 hours...





I have to admit I expected a way shorter time than that.



Disclaimer: your mileage may vary.

Check your IQ! Send any amount to this address:

1GoodBTCiGyd1J1LkDhCThfTHG8n9WJnNn

Pieter Wuille





Offline



Activity: 1064

Merit: 1039







LegendaryActivity: 1064Merit: 1039 Re: Experimental pre-0.8 builds for testing December 11, 2012, 12:01:48 AM #12 Quote from: Mushoz on December 10, 2012, 09:25:35 PM Do you happen to know when the way the blockchain gets downloaded gets changed? Perhaps even downloading in parallel from multiple peers?



When someone implements an improvement. I plan to have a look at this myself soon, but no promises.



Quote

Edit: And is there an ETA for the 0.8 release? I'm so looking forward to all these great changes to hit to main release

It still needs a lot of testing, and several things are missing before we can have a new stable release. When someone implements an improvement. I plan to have a look at this myself soon, but no promises.It still needs a lot of testing, and several things are missing before we can have a new stable release. I do Bitcoin stuff.

jgarzik





Offline



Activity: 1596

Merit: 1008







LegendaryActivity: 1596Merit: 1008 Re: Experimental pre-0.8 builds for testing December 12, 2012, 03:43:49 AM #14 Quote from: Mushoz on December 11, 2012, 03:47:37 PM

Sounds great, keep up the good work

+1, Pieter has been doing the (majority?) of the Satoshi client heavy lifting. He deserves lots of kudos and free beer.



Quote

And no need to rush

Actually, we do! We need to stabilize 0.8 as quickly as is reasonably possible, because people are beginning to avoid 0.7.x due to slowness, choosing instead less secure but faster bitcoin clients.



Any and all testing, on any and all platforms, is greatly appreciated.



Even posting "it works, my platform is [Ubuntu version X | Fedora version Y | Windows version Z]..." is helpful.



+1, Pieter has been doing the (majority?) of the Satoshi client heavy lifting. He deserves lots of kudos and free beer.Actually, we do! We need to stabilize 0.8 as quickly as is reasonably possible, because people are beginning to avoid 0.7.x due to slowness, choosing instead less secure but faster bitcoin clients.Any and all testing, on any and all platforms, is greatly appreciated.Even posting "it works, my platform is [Ubuntu version X | Fedora version Y | Windows version Z]..." is helpful. Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj

paybitcoin



Offline



Activity: 85

Merit: 10





1h79nc







MemberActivity: 85Merit: 101h79nc Re: Experimental pre-0.8 builds for testing December 12, 2012, 07:05:14 AM #15 Hello,



Testing with bitcoin-0.7.1-244-g3ccb06f-win32-setup.exe and a bootstrap.dat from the torrent. All defaults, except using my server as the peer.

CPU is Win7 x64 Athlon II X4 2.8GHz (4 cores), 4GB RAM, HDD.



bootstrap.dat finished in 53 minutes at 25% CPU, ~120 MB RAM max. With no connection so its not downloading stuff in the background.

The rest of the blocks took 79 minutes. Download was constant at 350-450KBps (about 30% of my total DSL bandwidth.) 70-80% CPU the whole time, ~163 MB RAM max.



blocks dir is now 4.53 GB.



Went pretty smoothly. Much, much better than the previous days-long sync.

kwukduck



Offline



Activity: 1941

Merit: 1000







LegendaryActivity: 1941Merit: 1000 Re: Experimental pre-0.8 builds for testing December 12, 2012, 07:16:52 AM #16 Clean setup in win7_x64

Still syncing the blockchain (over 15 hrs now)

Barely any CPU usage, but using 1.3GB memory. Also seems to have a lot of disk activity making the system a bit slow. 14b8PdeWLqK3yi3PrNHMmCvSmvDEKEBh3E

m0mchil



Offline



Activity: 171

Merit: 101







Full MemberActivity: 171Merit: 101 Re: Experimental pre-0.8 builds for testing December 12, 2012, 02:14:25 PM #17



bitcoin-qt built with gitian-builder from sipa/turbo rev 3ccb06f5ae538260817b6ef474f93029be41a64d





Ubuntu 12.04 64 bit, SSD A-DATA SX910 256 GB



64 bit, no -par (default), default cache



first 193000 blocks from bootstrap.dat - 11m:40s CPU time



real 9m50.565s

user 10m47.800s

sys 0m56.548s





64 bit, no -par (default), default cache



loading from blk000x.dat, up to arround block 211100



-loadblock=blk0001.dat -loadblock=blk0002.dat -loadblock=blk0003.dat



1 GB peak memory, ~80% average CPU usage, total CPU time 3h:16m



real 41m30.004s

user 196m43.378s

sys 2m47.834s



debug.log.ubuntu.12.04.x64.7z









Windows 7 Professional x64, two Seagate 512 GB HDD in RAID 0



-loadblock=blk0001.dat -loadblock=blk0002.dat -loadblock=blk0003.dat



default cache, with '-par 8', initially it seemed to run in single thread mode - first 193 000 blocks loaded significantly slower than on Ubuntu (same hardware), reason perhaps is slower storage.



After block 193000 the average CPU utilization rose to ~75%. Looking at threads, there were 16 threads executing same code of which 8 were actually active.



92 minutes real time, 6h:05m CPU time, 72% average CPU utilization, 235 MB peak memory working set



I then tried with '-par=4' and setting CPU affinity to physical cores only. Now there were initially 12 threads at transaction crunching, with again only one actively processing pre-SatoshiDice blocks.



debug.log.win32.7z intel core i7 860 (4 physical, 8 logical cores)bitcoin-qt built with gitian-builder from sipa/turbo rev 3ccb06f5ae538260817b6ef474f93029be41a64dUbuntu 12.04 64 bit, SSD A-DATA SX910 256 GB64 bit, no -par (default), default cachefirst 193000 blocks from bootstrap.dat - 11m:40s CPU timereal 9m50.565suser 10m47.800ssys 0m56.548s64 bit, no -par (default), default cacheloading from blk000x.dat, up to arround block 211100-loadblock=blk0001.dat -loadblock=blk0002.dat -loadblock=blk0003.dat1 GB peak memory, ~80% average CPU usage, total CPU time 3h:16mreal 41m30.004suser 196m43.378ssys 2m47.834sWindows 7 Professional x64, two Seagate 512 GB HDD in RAID 0-loadblock=blk0001.dat -loadblock=blk0002.dat -loadblock=blk0003.datdefault cache, with '-par 8', initially it seemed to run in single thread mode - first 193 000 blocks loaded significantly slower than on Ubuntu (same hardware), reason perhaps is slower storage.After block 193000 the average CPU utilization rose to ~75%. Looking at threads, there were 16 threads executing same code of which 8 were actually active.92 minutes real time, 6h:05m CPU time, 72% average CPU utilization, 235 MB peak memory working setI then tried with '-par=4' and setting CPU affinity to physical cores only. Now there were initially 12 threads at transaction crunching, with again only one actively processing pre-SatoshiDice blocks. POCLBM

slothbag



Offline



Activity: 369

Merit: 250









Sr. MemberActivity: 369Merit: 250 Re: Experimental pre-0.8 builds for testing December 13, 2012, 09:55:52 AM #18 I upgraded to the latest Jenkins build of bitcoin-qt ( pre-0.8 ) on windows. I tried for a week and a half running -loadblock and -reindex, stopping restarting etc.. it was horribly slow, and going no where.



So I eventually downloaded the SIPA turbo build linked in the OP, deleted my existing blockchain, downloaded the bootstrap and let it run from scratch with an empty wallet..



Took about 6 hours to load in the bootstrap file while CPU sat at 30% and disk I/O not that high, so not sure what the bottleneck was there. then after block 190000 CPU jumped to 100% and it took about 10 hours to get back in sync to current block. Then I added my old wallet and rescanned that which went for maybe 10-20mins..



So significantly faster than the first time I sync'd 18 months ago.. but still quite frustrating.



I'm running a Atom CPU based nettop with windows 7, hence the slower than average speeds.

