jgarzik



Offline



Activity: 1596

Merit: 1008







LegendaryActivity: 1596Merit: 1008 [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client November 27, 2012, 08:25:01 PM

Last edit: November 30, 2012, 08:16:09 AM by jgarzik #1



I'd like to announce another bitcoin implementation, which is really two useful pieces in one:



libccoin - a bitcoin library, written in C

picocoin - A lightweight, C-based



libccoin supports all key network datastructures (block, transaction, etc.), script parsing and validation, transaction and block validation, a "headers-only" or full block database, and many other features essential to any bitcoin client. libccoin passes all key encoding, script and transaction tests available in the Satoshi reference bitcoin client.



picocoin is much more under construction. When complete, it will be a very low resource, command line / JSON-driven bitcoin wallet. Advanced security features already implemented include required wallet encryption,



Status: Alpha quality, developer release. Passes reference client base58/script/transaction tests, but is still a developer-only preview.



Feature list:

Liberal Mit/X11 free software license

A full-feature bitcoin support library. The library will not be limited to "what picocoin needs."

Supports all core data structures and network messages

Full script implementation

Passes hundreds of available reference client tests

Supports multiple block chains: main or testnet3

Very low resource usage (cpu, disk, and memory)

Small codebase (both source code and compiled object)

Fast. Loads all block headers in under 2 seconds.

Supports advanced thin-client features such as bloom filtering, an upcoming proposal that will reduce client bandwidth usage.

Works on big endian machines, as well as little endian machines

Multi-platform: Linux, Mac OSX, BSD are working. Windows support in progress.

Library works on Windows. picocoin will work on Windows with some modifications, but be a bit less secure than other platforms due to lack of fork.

Improved security: fork-based process separation firewall between networking and wallet code -- your wallet is never directly exposed to the network.

directly exposed to the network. Follows the philosophy of "do, not hype." This library is already far more secure and capable than other libraries hyped as the "future of bitcoin" by their authors.

Code contributions are welcome (see github URL above).

Comments are welcome.

Donations are welcome too (see signature).



Source code URL: https://github.com/jgarzik/picocoin/ I'd like to announce another bitcoin implementation, which is really two useful pieces in one:- a bitcoin library, written in C- A lightweight, C-based SPV bitcoin wallet clientlibccoin supports all key network datastructures (block, transaction, etc.), script parsing and validation, transaction and block validation, a "headers-only" or full block database, and many other features essential to any bitcoin client. libccoin passes all key encoding, script and transaction tests available in the Satoshi reference bitcoin client.picocoin is much more under construction. When complete, it will be a very low resource, command line / JSON-driven bitcoin wallet. Advanced security featuresinclude required wallet encryption, fork -based process separation of P2P networking and wallet (and chroot/SELinux jailing coming soon), something that the reference Satoshi client does not even support.Status:Feature list:Code contributions are welcome (see github URL above).Comments are welcome.Donations are welcome too (see signature). Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj

2112



Offline



Activity: 2128

Merit: 1038









LegendaryActivity: 2128Merit: 1038 Re: [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client November 27, 2012, 09:22:49 PM #5 Quote from: jgarzik on November 27, 2012, 08:25:01 PM Very low resource usage (cpu, disk, and memory)

Small codebase (both source code and compiled object)

Multi-platform: Linux supported currently. Should work on OSX/BSD/Windows with minimal modifications. Comments are welcome.

Comments are welcome.



It will be nearly impossible to convert your code later unless you very early make a commitment to supporting the event-driven programming like using lwIP-style network interface in addition to socket-style networking.



The same architectural choice needs to be made early to accomodate various lightweight flash-storage access methods instead of Unix-style open/lseek/read/write/close. Same for fork/wait/pthreads versus any RTOS-style explicit synchronization primitives.



One possible advantage of such choices would be that instead of relying purely on donations you may be able to obtain funds via cooperation with future manufacturers of the future Bitcoin devices that will greatly benefit from the "very low resource usage". Jeff, I have one comment that has value only very early in the design of your architecture. Since you've made a choice of C as a implementation language I suggest that you add one more target to the list of supported platforms: standalone a.k.a. bare metal.It will be nearly impossible to convert your code later unless you very early make a commitment to supporting the event-driven programming like using lwIP-style network interface in addition to socket-style networking.The same architectural choice needs to be made early to accomodate various lightweight flash-storage access methods instead of Unix-style open/lseek/read/write/close. Same for fork/wait/pthreads versus any RTOS-style explicit synchronization primitives.One possible advantage of such choices would be that instead of relying purely on donations you may be able to obtain funds via cooperation with future manufacturers of the future Bitcoin devices that will greatly benefit from the "very low resource usage".

Long-term mining prognosis: Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0 Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0

jgarzik



Offline



Activity: 1596

Merit: 1008







LegendaryActivity: 1596Merit: 1008 Re: [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client November 27, 2012, 09:26:57 PM #6 Quote from: 2112 on November 27, 2012, 09:22:49 PM Jeff, I have one comment that has value only very early in the design of your architecture. Since you've made a choice of C as a implementation language I suggest that you add one more target to the list of supported platforms: standalone a.k.a. bare metal.



Yes, embedded usage is another target.



Each module in libccoin is carefully designed to minimize internal dependencies. The core data structures, address generation, script execution and transaction validation are wholly independent of any filesystem or network design.



These modules may be used on a non-POSIX flash filesystem, with zero network support, today.



picocoin (the client) requires certain network, filesystem and process features, but libccoin (the library) does not. This is by design.



Yes, embedded usage is another target.Each module in libccoin is carefully designed to minimize internal dependencies. The core data structures, address generation, script execution and transaction validation are wholly independent of any filesystem or network design.These modules may be used on a non-POSIX flash filesystem, with zero network support, today.picocoin (the client) requires certain network, filesystem and process features, but libccoin (the library) does not. This is by design. Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj

2112



Offline



Activity: 2128

Merit: 1038









LegendaryActivity: 2128Merit: 1038 Re: [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client November 27, 2012, 10:45:58 PM #10 Quote from: jgarzik on November 27, 2012, 09:26:57 PM Yes, embedded usage is another target.



Each module in libccoin is carefully designed to minimize internal dependencies. The core data structures, address generation, script execution and transaction validation are wholly independent of any filesystem or network design.



These modules may be used on a non-POSIX flash filesystem, with zero network support, today.



picocoin (the client) requires certain network, filesystem and process features, but libccoin (the library) does not. This is by design.

But even the definitions in your "ccoin" include files are deeply incompatible with any standalone environment, even if it uses GCC toolchain. Major rework would be required even now, when there's just couple of hundred lines of code.



The big-endian compatibility is already super-confusing. Istead of using uint32_t in bu256_t use an "union le_int32" to at least flag the mixing of little endian uint32_t with guint32_t.



From my extensive commercial experience of porting from Linux/Posix/Intel-only code to anything standalone or lightweight OS will unfortunatly go through the "first deny then rewrite from almost scratch" stages.



If I may suggest two things:



1) from the start build also on Mac Leopard or Snow Leopard to verify the endian-neutrality with "-arch ppc" with Rosetta on Intel CPU. Edit: Never mind, I forgot that you have access to the Linux on POWER and z/Arch through your association with RedHat.

2) obtain any embedded development/simulation environment. If you have any Xilinx Spartan6 boards (with memory) laying around you could use them to test using MicroBlaze CPU with Xilinx EDK/SDK.



Thank you for yor time.

But even the definitions in your "ccoin" include files are deeply incompatible with any standalone environment, even if it uses GCC toolchain. Major rework would be required even now, when there's just couple of hundred lines of code.The big-endian compatibility is already super-confusing. Istead of using uint32_t in bu256_t use an "union le_int32" to at least flag the mixing of little endian uint32_t with guint32_t.From my extensive commercial experience of porting from Linux/Posix/Intel-only code to anything standalone or lightweight OS will unfortunatly go through the "first deny then rewrite from almost scratch" stages.If I may suggest two things:1) from the start build also on Mac Leopard or Snow Leopard to verify the endian-neutrality with "-arch ppc" with Rosetta on Intel CPU. Edit: Never mind, I forgot that you have access to the Linux on POWER and z/Arch through your association with RedHat.2) obtain any embedded development/simulation environment. If you have any Xilinx Spartan6 boards (with memory) laying around you could use them to test using MicroBlaze CPU with Xilinx EDK/SDK.Thank you for yor time.

Long-term mining prognosis: Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0 Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0

jgarzik



Offline



Activity: 1596

Merit: 1008







LegendaryActivity: 1596Merit: 1008 Re: [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client November 27, 2012, 11:34:42 PM #11 Quote from: 2112 on November 27, 2012, 10:45:58 PM But even the definitions in your "ccoin" include files are deeply incompatible with any standalone environment, even if it uses GCC toolchain. Major rework would be required even now, when there's just couple of hundred lines of code.



This seems quite an exaggeration, with no supporting evidence of what needs a major rework. I do embedded programming for the day job, so I don't see this. The Linux kernel is as bare metal as it gets, and must work in all environments, all platforms.



The main chore of embedded use is simply the dependencies, most notably OpenSSL, and to a lesser extent GLib. Once you have that, or its replacement, embedded use is straightforward.



Quote The big-endian compatibility is already super-confusing. Istead of using uint32_t in bu256_t use an "union le_int32" to at least flag the mixing of little endian uint32_t with guint32_t.



Using "union" decreases the performance and quality of code output for zero gain. It is trivial enough to use



Quote 1) from the start build also on Mac Leopard or Snow Leopard to verify the endian-neutrality with "-arch ppc" with Rosetta on Intel CPU. Edit: Never mind, I forgot that you have access to the Linux on POWER and z/Arch through your association with RedHat.



The code has long since been verified to work on big endian.



This seems quite an exaggeration, with no supporting evidence of what needs a major rework. I do embedded programming for the day job, so I don't see this. The Linux kernel is as bare metal as it gets, and must work in all environments, all platforms.The main chore of embedded use is simply the dependencies, most notably OpenSSL, and to a lesser extent GLib. Once you have that, or its replacement, embedded use is straightforward.Using "union" decreases the performance and quality of code output for zero gain. It is trivial enough to use sparse with a type specifier to guarantee endian purity, like the Linux kernel does, if that is desired ("__le32" etc.)The code has long since been verified to work on big endian. Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj

2112



Offline



Activity: 2128

Merit: 1038









LegendaryActivity: 2128Merit: 1038 Re: [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client November 28, 2012, 01:49:32 AM #14 Quote from: jgarzik on November 27, 2012, 11:34:42 PM

This seems quite an exaggeration, with no supporting evidence of what needs a major rework. I do embedded programming for the day job, so I don't see this. The Linux kernel is as bare metal as it gets, and must work in all environments, all platforms.



For a die-hard Linux developer like you I think the minimum definition of bare-metal would be uClinux. Full Linux doesn't count anymore for a true embedded development.



Quote from: jgarzik on November 27, 2012, 11:34:42 PM The main chore of embedded use is simply the dependencies, most notably OpenSSL, and to a lesser extent GLib. Once you have that, or its replacement, embedded use is straightforward.

Both GLib and OpenSSL are heavy duty harvester combines. Removing the dependencies on them will at least halve the resource usage.



For a RedHat-ter at least make the code work with newlib instead of glib. The required knowledge should be no more than the internal phone call away for you. This includes a whole sermon on why you don't want to build a dependency on them.



Quote from: jgarzik on November 27, 2012, 11:34:42 PM



The code has long since been verified to work on big endian.

Using "union" decreases the performance and quality of code output for zero gain. It is trivial enough to use sparse with a type specifier to guarantee endian purity, like the Linux kernel does, if that is desired ("__le32" etc.)The code has long since been verified to work on big endian.



The performance loss for passing unions around may have been true for GCC 2.96. I'm not normally working with Linux kernel and/or GCC.



But I have a great, positive experience with code of the following style(C++ used for brevity, ... expands to long #ifdef trees using platform- and compiler-specific intrinsics)

Code: union be_int {

unsigned char b[4];

int i;

inline operator =(int a) { ... }

inline operator int() { ... }

};

The explicit strong-typing of endianness blocks the vast majority of possible endian errors and allows to produce meaningfull, endian-bug-free contributions from people who are exclusively using little-endian hardware. In my (rather extensive) experience it actually increases the performance because programmers can visualize the data structures better and can spend more time designing the algorithms instead of debugging trivial omissions or transpositions.



All in all, I apologise for jumping in into the battle between you and MatthewLM. Regretfully I can't easily build neither of yours code, because I'm almost exclusively working on the closed-source platforms like Windows, MacOS, etc. and standalone environments not relying on GCC/GLIB/autogen. But it is a shame that another C implementation of Bitcoin is useless for the developers of Bitcoin hardware wallets.

For starters, you'll have heck of a time to remove your dependency on glib.h from GTK. But there's more. You could actually try to compile your code from within the Linux kernel tree to see for yourself.For a die-hard Linux developer like you I think the minimum definition of bare-metal would be uClinux. Full Linux doesn't count anymore for a true embedded development.Both GLib and OpenSSL are heavy duty harvester combines. Removing the dependencies on them will at least halve the resource usage.For a RedHat-ter at least make the code work with newlib instead of glib. The required knowledge should be no more than the internal phone call away for you. This includes a whole sermon on why you don't want to build a dependency on them.As I've noted: I've forgot that you are one of the very few Bitcoin developers with regular access to big-endian hardware. I'm thinking that you will continue to be the only one developer who can test picocoin on the big-endian architecture. The mojibake-endian problem will continue to badly affect the future contributors/users of your library if you aren't going to use your own medicine (that is "sparse" and "__le32")The performance loss for passing unions around may have been true for GCC 2.96. I'm not normally working with Linux kernel and/or GCC.But I have a great, positive experience with code of the following style(C++ used for brevity, ... expands to long #ifdef trees using platform- and compiler-specific intrinsics)The explicit strong-typing of endianness blocks the vast majority of possible endian errors and allows to produce meaningfull, endian-bug-free contributions from people who are exclusively using little-endian hardware. In my (rather extensive) experience it actually increases the performance because programmers can visualize the data structures better and can spend more time designing the algorithms instead of debugging trivial omissions or transpositions.All in all, I apologise for jumping in into the battle between you and MatthewLM. Regretfully I can't easily build neither of yours code, because I'm almost exclusively working on the closed-source platforms like Windows, MacOS, etc. and standalone environments not relying on GCC/GLIB/autogen. But it is a shame that another C implementation of Bitcoin is useless for the developers of Bitcoin hardware wallets.

Long-term mining prognosis: Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0 Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0

jgarzik



Offline



Activity: 1596

Merit: 1008







LegendaryActivity: 1596Merit: 1008 Re: [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client November 28, 2012, 02:54:09 AM #15 Quote from: BkkCoins on November 28, 2012, 01:22:27 AM So am I understanding right? I could run picocoin and it would be lightweight like Electrum but also support the Bitcoin network by interacting on the network like a full satoshi client. I've been using Electrum mostly to get away from having to use around 5GB disk space and for quick install/startup but I'd really prefer to be supporting Bitcoin more by being another validating node in it's network.



picocoin is only a lightweight client. It is not a "full node" and probably never will be.



The included library, libccoin, does provide all the tools and gadgetry needed to build your own "full node" bitcoin client that takes 5GB disk space, if you have the programming chops to do that.



picocoin is only a lightweight client. It is not a "full node" and probably never will be.The included library, libccoin, does provide all the tools and gadgetry needed to build your own "full node" bitcoin client that takes 5GB disk space, if you have the programming chops to do that. Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj

jgarzik



Offline



Activity: 1596

Merit: 1008







LegendaryActivity: 1596Merit: 1008 Re: [ANNOUNCE] picocoin and libccoin -- C-based bitcoin library and client November 28, 2012, 03:11:49 AM #16 Quote from: 2112 on November 28, 2012, 01:49:32 AM For starters, you'll have heck of a time to remove your dependency on glib.h from GTK.



You mean "remove your dependency on glib.h from libccoin", I presume? GTK has no relevance to this, besides being the administrative umbrella project for GLib.



Removing the GLib dependency is straightforward for any developer. GLib is used for a few ADTs like variable length strings, hash tables and arrays.



Quote Both GLib and OpenSSL are heavy duty harvester combines. Removing the dependencies on them will at least halve the resource usage.



For a RedHat-ter at least make the code work with newlib instead of glib. The required knowledge should be no more than the internal phone call away for you. This includes a whole sermon on why you don't want to build a dependency on them.



Not sure you understand the problem space... First, newlib is wholly different from GLib, and does not provide the ADTs that GLib provides. Second, replacing GLib and OpenSSL would not change resource usage much at all. ADTs, Big Integer support, SHA hashing and ECDSA are required regardless of lib chosen. They would simply be replaced with... code that did the same thing under name other than "OpenSSL" or "GLib."



Quote The performance loss for passing unions around may have been true for GCC 2.96. I'm not normally working with Linux kernel and/or GCC.



It continues to be true for all known compilers I've seen or heard of, when compared with a simple, native machine integer as used by the current implementation.



Quote All in all, I apologise for jumping in into the battle between you and MatthewLM. Regretfully I can't easily build neither of yours code, because I'm almost exclusively working on the closed-source platforms like Windows, MacOS, etc. and standalone environments not relying on GCC/GLIB/autogen.



It will soon be building on MacOSX and Windows. Will post when this happens (volunteers willing to post building outputs are helpful). As stated, it is a developer-only release right now, so there are plenty of rough edges.



Quote But it is a shame that another C implementation of Bitcoin is useless for the developers of Bitcoin hardware wallets.



I respectfully disagree with that assessment. Anyone looking at the code may see precisely what dependencies need replacing, for an embedded environment. If anyone actually developing hardware or embedded bitcoin solutions wishes to contact me, I would be happy to illustrate how it would work.



You mean "remove your dependency on glib.h from libccoin", I presume? GTK has no relevance to this, besides being the administrative umbrella project for GLib.Removing the GLib dependency is straightforward for any developer. GLib is used for a few ADTs like variable length strings, hash tables and arrays.Not sure you understand the problem space... First, newlib is wholly different from GLib, and does not provide the ADTs that GLib provides. Second, replacing GLib and OpenSSL would not change resource usage much at all. ADTs, Big Integer support, SHA hashing and ECDSA are required regardless of lib chosen. They would simply be replaced with... code that did the same thing under name other than "OpenSSL" or "GLib."It continues to be true for all known compilers I've seen or heard of, when compared with a simple, native machine integer as used by the current implementation.It will soon be building on MacOSX and Windows. Will post when this happens (volunteers willing to post building outputs are helpful). As stated, it is a developer-only release right now, so there are plenty of rough edges.I respectfully disagree with that assessment. Anyone looking at the code may see precisely what dependencies need replacing, for an embedded environment. If anyone actually developing hardware or embedded bitcoin solutions wishes to contact me, I would be happy to illustrate how it would work. Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj