Rogue Star



Offline



Activity: 89

Merit: 10







MemberActivity: 89Merit: 10 Re: Experimental pre-0.8 builds for testing December 22, 2012, 08:02:39 AM #23 Here's my results from the 15th. I was syncing solely from a local peer and did not use block loading



Configuration

-------

Version: bitcoin-0.7.1-283-gf6aadb1-win32 (aka "turbo" build)

Flags: -connect=x.x.x.x -keypool=0

Hardware: VM w/full hw acceleration

vOS: Windows 7 64-bit latest

vCPU: AMD A8-3850 2 cores @ 2.9 GHz

vRAM: 4GB @ 1333 MHz

vDisk: 7200 rpm

vNetwork: 1 gigabit link



performance syncing up to block 193K

-----------

Duration: 56m23s

CPU Utilization: ~50%

CPU threads: ~32

Network Utilization: ~1% over gigabit link



RAM usage:

100MB @ ~130K blocks

110MB @ ~148K blocks

120MB @ ~157K blocks

130MB @ ~176K blocks

140MB @ ~190K blocks



performance for full sync up to network (212433)

-----------

Duration: 5h 15m total

CPU Utilization: up to 100%

Network Utilization: ~0.25% over gigabit link you can donate to me for whatever reason at: 18xbnjDDXxgcvRzv5k2vmrKQHWDjYsBDCf

finway



Offline



Activity: 714

Merit: 500







Hero MemberActivity: 714Merit: 500 Re: Experimental pre-0.8 builds for testing December 25, 2012, 10:55:19 AM #24

After a crush, i can't start bitcoin-qt, so i installed the newest build, still can't.



Windows7 32bit AMD CPU 4GB ram.



debug.txt



Code: Bitcoin version v0.7.1-283-gf6aadb1-beta (2012-12-13 11:11:02 +0100)

Using OpenSSL version OpenSSL 1.0.1c 10 May 2012

Startup time: 12/25/12 10:52:08

Default data directory C:\Users\Administrator\AppData\Roaming\Bitcoin

Used data directory C:\Users\Administrator\AppData\Roaming\Bitcoin

Using 2 threads for script verification

dbenv.open LogDir=C:\Users\Administrator\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\Administrator\AppData\Roaming\Bitcoin\db.log

Bound to [::]:8333

Bound to 0.0.0.0:8333

Loading block index...

Opening LevelDB in C:\Users\Administrator\AppData\Roaming\Bitcoin\blktree

Opened LevelDB successfully

Opening LevelDB in C:\Users\Administrator\AppData\Roaming\Bitcoin\coins

Opened LevelDB successfully

LoadBlockIndex(): last block file = 33

LoadBlockIndex(): last block file: CBlockFileInfo(blocks=246, size=35583097, heights=212710..212955, time=2012-12-18..2012-12-20)

block index 3982ms

Loading wallet...

nFileVersion = 79900

wallet 329ms

InvalidChainFound: invalid block=000000000000030ced7d02b925d6a8709a4d1559afe535d863cbc56d0e569894 height=212955 work=672233111659878960356 date=12/20/12 23:05:13

Error report:After a crush, i can't start bitcoin-qt, so i installed the newest build, still can't.Windows7 32bit AMD CPU 4GB ram.debug.txt Tips welcome: 1ALatCsyZ2fh4gLQNij1RuTeU3cuywE9YD

smtp



Offline



Activity: 70

Merit: 10







MemberActivity: 70Merit: 10 Re: Experimental pre-0.8 builds for testing December 27, 2012, 10:08:19 PM #25



If you think this message is the wrong place, place shift it to the correct topic/thread:



Please have a look at my posting at



And if you think it is appropriate, you are free to move it to this (or a better) board/thread.



BTW: Thank you for taking me out off the newbie-posting-restriction jail.



Regards

smtp HiIf you think this message is the wrong place, place shift it to the correct topic/thread:Please have a look at my posting at https://bitcointalk.org/index.php?topic=15911.msg1422360#msg1422360 regarding "blkindex.dat" and "reindexing" real time speed of the bitcoin client.And if you think it is appropriate, you are free to move it to this (or a better) board/thread.BTW: Thank you for taking me out off the newbie-posting-restriction jail.Regardssmtp

Pieter Wuille





Offline



Activity: 1064

Merit: 1038







LegendaryActivity: 1064Merit: 1038 Re: Experimental pre-0.8 builds for testing December 27, 2012, 11:29:38 PM #26 Quote from: smtp on December 27, 2012, 10:08:19 PM



And if you think it is appropriate, you are free to move it to this (or a better) board/thread.



BTW: Thank you for taking me out off the newbie-posting-restriction jail.

Please have a look at my posting at https://bitcointalk.org/index.php?topic=15911.msg1422360#msg1422360 regarding "blkindex.dat" and "reindexing" real time speed of the bitcoin client.And if you think it is appropriate, you are free to move it to this (or a better) board/thread.BTW: Thank you for taking me out off the newbie-posting-restriction jail.

Hello there,



I can't take you out of the jail, but I can explain what we're doing in 0.8.



The old blkindex.dat file is a (very non-efficient, 12 bytes per txout) index that stores the disk position of all transactions in the blk00*.dat files, and for each transaction output whether and where it got spent. During block validation, this means that for every transaction that needs to be validated, we need to look up all input's previous transactions in the index, verify all inputs aren't spent yet, then read those transactions from disk, and then evaluate scripts, and finally update the index with spends and new transaction outputs created. This means the working set (the amount of data that needs to be readily available) is most of the block index, and part of the block data itself too (especially the recent/last part needs frequent access). This means fast access to gigabytes, and you're right to point out that this is slow. Too slow, because it is wasteful in two ways: 1) we need access to full transactions (even the parts that are already spent, or aren't necessary for validation of inputs) and 2) the index to search through contains all transactionsm even those that are already completely spent.



For 0.8, this is changed completely. There is no transaction index at all anymore. Instead, we keep - in addition to the block files - a database (LevelDB, not BDB) that contains just all unspent outputs (specifically, for every not-fully-spent transaction: its version, its height in the chain, and for all its unspent outputs: the value and the script) using an efficient encoding (so not pointers into the block files, the actual data itself). This database is right now (including all overhead and indexing structures) 146 MiB (126 MiB of which is actual data). This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory.



There is some overhead in maintaining a database, and encoding and decoding the txout data, but on my laptop, if I disable signature verification and increase the db cache size to 200 MB, reindexing to block 213769 (including all block and transaction validation, just not ECDSA verification) takes around 10 minutes (mostly at 100% CPU usage on one core). Using hash tables instead of trees for the cache, and increasing its size would further reduce that number. That's not the next limiting factor though. The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined).



I do plan to make some improvements to the in-memory unspent-output-database-cache still, so that if made large enough, one shouldn't ever wait on disk, but I don't expect huge improvements there anymore. The most important changes for speeding up are reworking the block download mechanism, and parallellizing signature verification (which is already implemented, but not merged in mainline).

Hello there,I can't take you out of the jail, but I can explain what we're doing in 0.8.The old blkindex.dat file is a (very non-efficient, 12 bytes per txout) index that stores the disk position of all transactions in the blk00*.dat files, and for each transaction output whether and where it got spent. During block validation, this means that for every transaction that needs to be validated, we need to look up all input's previous transactions in the index, verify all inputs aren't spent yet, then read those transactions from disk, and then evaluate scripts, and finally update the index with spends and new transaction outputs created. This means the working set (the amount of data that needs to be readily available) is most of the block index, and part of the block data itself too (especially the recent/last part needs frequent access). This means fast access to gigabytes, and you're right to point out that this is slow. Too slow, because it is wasteful in two ways: 1) we need access to full transactions (even the parts that are already spent, or aren't necessary for validation of inputs) and 2) the index to search through contains all transactionsm even those that are already completely spent.For 0.8, this is changed completely. There is no transaction index at all anymore. Instead, we keep - in addition to the block files - a database (LevelDB, not BDB) that contains just all unspent outputs (specifically, for every not-fully-spent transaction: its version, its height in the chain, and for all its unspent outputs: the value and the script) using an efficient encoding (so not pointers into the block files, the actual data itself). This database is right now (including all overhead and indexing structures) 146 MiB (126 MiB of which is actual data). This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory.There is some overhead in maintaining a database, and encoding and decoding the txout data, but on my laptop, if I disable signature verification and increase the db cache size to 200 MB, reindexing to block 213769 (including all block and transaction validation, just not ECDSA verification) takes around 10 minutes (mostly at 100% CPU usage on one core). Using hash tables instead of trees for the cache, and increasing its size would further reduce that number. That's not the next limiting factor though. The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined).I do plan to make some improvements to the in-memory unspent-output-database-cache still, so that if made large enough, one shouldn't ever wait on disk, but I don't expect huge improvements there anymore. The most important changes for speeding up are reworking the block download mechanism, and parallellizing signature verification (which is already implemented, but not merged in mainline). I do Bitcoin stuff.

notme



Offline



Activity: 1904

Merit: 1001







LegendaryActivity: 1904Merit: 1001 Re: Experimental pre-0.8 builds for testing December 28, 2012, 12:01:29 PM #27 Trying to compile the turbo build on linux I get these errors:

/usr/bin/ld: cannot find -lshlwapi

/usr/bin/ld: cannot find -ldbghelp



It seems to be complaining about windows only libraries being missing. https://www.bitcoin.org/bitcoin.pdf

While no idea is perfect, some ideas are useful.

12jh3odyAAaR2XedPKZNCR4X4sebuotQzN While no idea is perfect, some ideas are useful.12jh3odyAAaR2XedPKZNCR4X4sebuotQzN

smtp



Offline



Activity: 70

Merit: 10







MemberActivity: 70Merit: 10 Re: Experimental pre-0.8 builds for testing December 28, 2012, 09:39:35 PM #29 Quote from: Pieter Wuille on December 27, 2012, 11:29:38 PM I can't take you out of the jail, but I can explain what we're doing in 0.8.

The first was no longer necessary (else I could not habe posted the previous notice in THIS topic/board) and

thx for the later.



So, reindexing and the "-loadblock" option in 0.7.1 not only made a new blkindex.dat file but furthermore do a fully validation (including the expensive signature check). I think there is no secret option in 0.7.1 or 0.7.2 to disable the signature verification when using "-loadblock", isn't it?



Else a new bitcoin-client user could simple load a valid blockchain from a URL he trusted and simply load it to get this "blkindex.dat" to save MUCH real time -- this was my aim. For history: I deleted the only 6 orphan blocks form my personal blk00*.dat and thus wanted/needed a new blkindex.dat.



Quote from: Pieter Wuille For 0.8, this is changed completely. There is no transaction index ..... This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory.

This sounds well - I like this idea. I just checked: nearly exactly a 7.5-th of all deposits/txout-channels is not redeemed currently. And I believe this fraction will increase in the future further more. To check/validate a NEW block/transaction, the bitcoin-client gets only this data in RAM is sufficient, but sadly not for my blockparser.

Moreover the idea to keep the whole out-scripts in memory is very good because there is only a very small number of effectively different response-scripts - they mainly differ only in different RMD-160-bit hashes for their public key. So, I guess in > 99% a table-no and this 20 byte hash is sufficient data to store the script. I also looked at all the input- and output-scripts and there are only very few effectively different ones -- so your "encoding and decoding the txout data" should be done quick and you can keep the coding-table small -- as long as the number of different out-scripts stay small (as in the past).



Quote from: Pieter Wuille The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined). The last point can only be critical, if CPU-usage tends to 100% not the waiting time of the bitcoin client. The only checking my blockparser doesn't do is the merkeltree-hash of each block (I was still too lazy to do the coding) and this OP_CHECKSIG (resp. OP_CHECKMULTISIG), but I had read and understood its detailed description on



To be frank - I should add that I had hoped 0.8 would be released as a Christmas present and after this, once more as a New Years present, but my hope faded when I read more on the bitcoin forum. Thus I posted my proposal in the hope it could help to improve the 0.8 version.



Thank you for your explanation. Now we can only continue to hope that 0.8 with the new data-structure will be released as reasonable quick as possible.



smtp

The first was no longer necessary (else I could not habe posted the previous notice in THIS topic/board) andthx for the later.So, reindexing and the "-loadblock" option in 0.7.1 not only made a new blkindex.dat file but furthermore do a fully validation (including the expensive signature check). I think there is no secret option in 0.7.1 or 0.7.2 to disable the signature verification when using "-loadblock", isn't it?Else a new bitcoin-client user could simple load a valid blockchain from a URL he trusted and simply load it to get this "blkindex.dat" to save MUCH real time -- this was my aim. For history: I deleted the only 6 orphan blocks form my personal blk00*.dat and thus wanted/needed a new blkindex.dat.This sounds well - I like this idea. I just checked: nearly exactly a 7.5-th of all deposits/txout-channels is not redeemed currently. And I believe this fraction will increase in the future further more. To check/validate a NEW block/transaction, the bitcoin-client gets only this data in RAM is sufficient, but sadly not for my blockparser.Moreover the idea to keep the whole out-scripts in memory is very good because there is only a very small number of effectively different response-scripts - they mainly differ only in different RMD-160-bit hashes for their public key. So, I guess in > 99% a table-no and this 20 byte hash is sufficient data to store the script. I also looked at all the input- and output-scripts and there are only very few effectively different ones -- so your "encoding and decoding the txout data" should be done quick and you can keep the coding-table small -- as long as the number of different out-scripts stay small (as in the past).The last point can only be critical, if CPU-usage tends to 100% not the waiting time of the bitcoin client. The only checking my blockparser doesn't do is the merkeltree-hash of each block (I was still too lazy to do the coding) and this OP_CHECKSIG (resp. OP_CHECKMULTISIG), but I had read and understood its detailed description on https://en.bitcoin.it/wiki/OP_CHECKSIG . Thus I can't judge how cpu-intensive this OP_CHECKSIG evaluation really is.To be frank - I should add that I had hoped 0.8 would be released as a Christmas present and after this, once more as a New Years present, but my hope faded when I read more on the bitcoin forum. Thus I posted my proposal in the hope it could help to improve the 0.8 version.Thank you for your explanation. Now we can only continue to hope that 0.8 with the new data-structure will be released as reasonable quick as possible.smtp

niko



Offline



Activity: 756

Merit: 500





There is more to Bitcoin than bitcoins.







Hero MemberActivity: 756Merit: 500There is more to Bitcoin than bitcoins. Re: Experimental pre-0.8 builds for testing December 30, 2012, 05:13:52 PM

Last edit: December 30, 2012, 05:39:38 PM by niko #31



Windows XP SP3

Pentium M, 1.6 GHz

760 MB RAM

Disk space available initially: ~35 GB



Installed bitcoin-0.7.1-315-gfd95a8a-win32-setup.exe



bitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmark



Started up fine, connected through LAN to the specified peer, started downloading. Typical CPU load (essentially no competing processes) was 20%-65%, rarely over 70%. Peak memory usage ~180 MB.



It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this?



I cleaned up some old files to make additional 7GB of space available, and started again as above. It continued downloading, then progress bar disappeared. The animated "sync" icon in the corner was still active, and downloading and verification was still going on fine. The CPU usage appeared somewhat lower then earlier, 20%-40% typically. Three hours into it, ran out of disk space again! The "coins" folder now grew to >37GB.



Cleaned up more GB, restarted the client, and two hours later and few hundreds of blocks before the end it ran out of disk space again! "Coins" folder now 40.1 GB, 21,734 files.



Cleaned up more space, but bitcoin-qt now crashes immediately after start. Rebooted, still the same. Windows crash report:

Code: <?xml version="1.0" encoding="UTF-16"?>

<DATABASE>

<EXE NAME="bitcoin-qt.exe" FILTER="GRABMI_FILTER_PRIVACY">

<MATCHING_FILE NAME="bitcoin-qt.exe" SIZE="20755456" CHECKSUM="0x604C3D23" BIN_FILE_VERSION="0.7.99.0" BIN_PRODUCT_VERSION="0.7.99.0" PRODUCT_VERSION="0.7.99.0" FILE_DESCRIPTION="Bitcoin-Qt (OSS GUI client for Bitcoin)" COMPANY_NAME="Bitcoin" PRODUCT_NAME="Bitcoin-Qt" FILE_VERSION="0.7.99.0" ORIGINAL_FILENAME="bitcoin-qt.exe" INTERNAL_NAME="bitcoin-qt" LEGAL_COPYRIGHT="2009-2012 The Bitcoin developers" VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x40004" VERFILETYPE="0x1" MODULE_TYPE="WIN32" PE_CHECKSUM="0x13D233D" LINKER_VERSION="0x10000" UPTO_BIN_FILE_VERSION="0.7.99.0" UPTO_BIN_PRODUCT_VERSION="0.7.99.0" LINK_DATE="01/30/2011 00:00:00" UPTO_LINK_DATE="01/30/2011 00:00:00" VER_LANGUAGE="Language Neutral [0x0]" />

<MATCHING_FILE NAME="uninstall.exe" SIZE="366985" CHECKSUM="0x7224CC4B" BIN_FILE_VERSION="0.7.99.0" BIN_PRODUCT_VERSION="0.7.99.0" PRODUCT_VERSION="0.7.99" FILE_DESCRIPTION="" COMPANY_NAME="Bitcoin project" PRODUCT_NAME="Bitcoin" FILE_VERSION="0.7.99" LEGAL_COPYRIGHT="" VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x4" VERFILETYPE="0x1" MODULE_TYPE="WIN32" PE_CHECKSUM="0x1B641" LINKER_VERSION="0x60000" UPTO_BIN_FILE_VERSION="0.7.99.0" UPTO_BIN_PRODUCT_VERSION="0.7.99.0" LINK_DATE="02/19/2012 15:01:49" UPTO_LINK_DATE="02/19/2012 15:01:49" VER_LANGUAGE="Language Neutral [0x0]" />

<MATCHING_FILE NAME="daemon\bitcoind.exe" SIZE="6174208" CHECKSUM="0x2B4948A" MODULE_TYPE="WIN32" PE_CHECKSUM="0x5ECEB6" LINKER_VERSION="0x10000" LINK_DATE="01/30/2011 00:00:00" UPTO_LINK_DATE="01/30/2011 00:00:00" />

</EXE>

<EXE NAME="kernel32.dll" FILTER="GRABMI_FILTER_THISFILEONLY">

<MATCHING_FILE NAME="kernel32.dll" SIZE="990208" CHECKSUM="0xCC2C4544" BIN_FILE_VERSION="5.1.2600.6293" BIN_PRODUCT_VERSION="5.1.2600.6293" PRODUCT_VERSION="5.1.2600.6293" FILE_DESCRIPTION="Windows NT BASE API Client DLL" COMPANY_NAME="Microsoft Corporation" PRODUCT_NAME="Microsoft® Windows® Operating System" FILE_VERSION="5.1.2600.6293 (xpsp_sp3_gdr.121001-1622)" ORIGINAL_FILENAME="kernel32" INTERNAL_NAME="kernel32" LEGAL_COPYRIGHT="© Microsoft Corporation. All rights reserved." VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x40004" VERFILETYPE="0x2" MODULE_TYPE="WIN32" PE_CHECKSUM="0xFBCBC" LINKER_VERSION="0x50001" UPTO_BIN_FILE_VERSION="5.1.2600.6293" UPTO_BIN_PRODUCT_VERSION="5.1.2600.6293" LINK_DATE="10/03/2012 04:58:13" UPTO_LINK_DATE="10/03/2012 04:58:13" VER_LANGUAGE="English (United States) [0x409]" />

</EXE>

</DATABASE>



Debug.log:



All in all, almost complete download via LAN and verification took ~10 hours. I decided to try out the low extreme, so I dug out a fossil and fired it up:Windows XP SP3Pentium M, 1.6 GHz760 MB RAMDisk space available initially: ~35 GBInstalled bitcoin-0.7.1-315-gfd95a8a-win32-setup.exebitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmarkStarted up fine, connected through LAN to the specified peer, started downloading. Typical CPU load (essentially no competing processes) was 20%-65%, rarely over 70%. Peak memory usage ~180 MB.It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this?I cleaned up some old files to make additional 7GB of space available, and started again as above. It continued downloading, then progress bar disappeared. The animated "sync" icon in the corner was still active, and downloading and verification was still going on fine. The CPU usage appeared somewhat lower then earlier, 20%-40% typically. Three hours into it, ran out of disk space again! The "coins" folder now grew to >37GB.Cleaned up more GB, restarted the client, and two hours later and few hundreds of blocks before the end it ran out of disk space again! "Coins" folder now 40.1 GB, 21,734 files.Cleaned up more space, but bitcoin-qt now crashes immediately after start. Rebooted, still the same. Windows crash report:Debug.log: http://www.filehosting.org/file/details/407213/debug.log All in all, almost complete download via LAN and verification took ~10 hours. They're there , in their room.

Your mining rig is on fire, yet you're very calm.

Pieter Wuille





Offline



Activity: 1064

Merit: 1038







LegendaryActivity: 1064Merit: 1038 Re: Experimental pre-0.8 builds for testing December 31, 2012, 12:53:44 AM #32 Quote from: niko on December 30, 2012, 05:13:52 PM It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this?



This is very unexpected and wrong. The size of the coins directory shouldn't exceed 150 MB or so (unless you increase -dbcache, in which case it may grow to something like 150 + dbcache/2 MB). Can you tell me which files (extensions) are the bulk of the data? This is very unexpected and wrong. The size of the coins directory shouldn't exceed 150 MB or so (unless you increase -dbcache, in which case it may grow to something like 150 + dbcache/2 MB). Can you tell me which files (extensions) are the bulk of the data? I do Bitcoin stuff.