jgarzik





Offline



Activity: 1596

Merit: 1007







LegendaryActivity: 1596Merit: 1007 python-bitcoinlib: Comprehensive bitcoin library for python March 08, 2013, 03:13:49 AM #1

Repository: git://github.com/jgarzik/python-bitcoinlib.git



The python library for



Features:

Easy object interface to all bitcoin core data structures: block, transaction, addresses, ...

Full transaction script engine

Fully verifies main and testnet block chains (via pynode)

ECDSA verification (OpenSSL wrapper)

Object interface to all known network messages

Binary encoding/decoding (serialization) for full bitcoin protocol interoperability

Passes many of the tests shipped with the bitcoin reference client (bitcoind/Bitcoin-Qt)

Like pynode, this library is currently a developer-only release, not recommended for highly secure production sites.



Pull requests, comments, questions and donations always welcome.





GitHub URL: https://github.com/jgarzik/python-bitcoinlib Repository: git://github.com/jgarzik/python-bitcoinlib.gitThe python library for pynode has matured sufficiently to have a home of its own. The python-bitcoinlib project attempts to present a lightweight, modular, a la carte interface to bitcoin data structures and network protocols.Features:Like pynode, this library is currently a developer-only release, not recommended for highly secure production sites.Pull requests, comments, questions and donations always welcome. Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj

jgarzik





Offline



Activity: 1596

Merit: 1007







LegendaryActivity: 1596Merit: 1007 Re: python-bitcoinlib: Comprehensive bitcoin library for python September 03, 2013, 05:30:51 PM #4



was also merged into this library, so that RPC support does not require a separate library.



Thanks to Peter Todd for both updates.



Merged an important bug fix, ensuring bug-for-bug compatibility with reference implementation's SignatureHash() https://github.com/jgarzik/python-bitcoinrpc/ was also merged into this library, so that RPC support does not require a separate library.Thanks to Peter Todd for both updates. Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj

Peter Todd





Offline



Activity: 1106

Merit: 1045







LegendaryActivity: 1106Merit: 1045 Re: python-bitcoinlib: Comprehensive bitcoin library for python September 19, 2013, 05:25:05 PM

Last edit: September 19, 2013, 08:18:25 PM by retep #5



Right now it's mainly the core transaction and scripting classes that I've modified; you can now write code like:





proxy = bitcoin.rpx.Proxy()



unspent = proxy.listunspent()



(some steps left out deliberately because I haven't tested this code!)



change_out = CTxOut(FIXME, proxy.getnewaddress().to_scriptPubKey())

msg_out = CTxOut(0, CScript([OP_RETURN, 'Hello World!']))



tx = CTransaction(vins, [change_out, msg_out])



r = proxy.signrawtransaction(tx)



if r['complete']:

print(proxy.sendrawtransaction(r['tx'])

else:

print("Didn't work!")





Jeff Garzik has said that if I modify pynode to be compatible with my changes he will likely merge them into bitcoinlib proper. FWIW I'm working on making python-bitcoinlib more pythonic: https://github.com/petertodd/python-bitcoinlib/tree/pythonize Right now it's mainly the core transaction and scripting classes that I've modified; you can now write code like:Jeff Garzik has said that if I modify pynode to be compatible with my changes he will likely merge them into bitcoinlib proper. BTC: 1FCYd7j4CThTMzts78rh6iQJLBRGPW9fWv PGP: 7FAB114267E4FA04

Peter Todd





Offline



Activity: 1106

Merit: 1045







LegendaryActivity: 1106Merit: 1045 Re: python-bitcoinlib: Comprehensive bitcoin library for python September 19, 2013, 05:49:02 PM #7 Quote from: knowitnothing on September 19, 2013, 05:40:32 PM

Hey there retep. Regarding the RPC, have you considered making their calls async ? I have rolled my own implementation that does that using the futures package, it might make everything a bit harder for the conventional user but there are situations where you simply cannot use blocking RPC. I will publish it later somewhere.

Patches welcome, though I'll warn you I don't actually know if bitcoind itself has the performance to make async RPC worthwhile.



I'm working on classes to do things like manage the blockchain information (eg. the set of all know blocks and block headers) which may lead to a model where it makes more sense to just do it all in bitcoinlib based on a internal database of blockchain information.



FWIW my personal motivation for working on all this stuff is to make it easier to prototype and experiment at a fairly low level for Bitcoin developers. For instance one of the first things I used python-bicoinlib for was to test some DoS attacks against the Bloom filter code and patches defeating those attacks. (that's why I wrote the CBloomFilter class, but haven't written the corresponding CMerkleBlock class needed to make it useful to SPV clients) Patches welcome, though I'll warn you I don't actually know if bitcoind itself has the performance to make async RPC worthwhile.I'm working on classes to do things like manage the blockchain information (eg. the set of all know blocks and block headers) which may lead to a model where it makes more sense to just do it all in bitcoinlib based on a internal database of blockchain information.FWIW my personal motivation for working on all this stuff is to make it easier to prototype and experiment at a fairly low level for Bitcoin developers. For instance one of the first things I used python-bicoinlib for was to test some DoS attacks against the Bloom filter code and patches defeating those attacks. (that's why I wrote the CBloomFilter class, but haven't written the corresponding CMerkleBlock class needed to make it useful to SPV clients) BTC: 1FCYd7j4CThTMzts78rh6iQJLBRGPW9fWv PGP: 7FAB114267E4FA04

knowitnothing



Offline



Activity: 294

Merit: 250







Sr. MemberActivity: 294Merit: 250 Re: python-bitcoinlib: Comprehensive bitcoin library for python September 19, 2013, 06:33:46 PM #8 Quote from: retep on September 19, 2013, 05:49:02 PM Quote from: knowitnothing on September 19, 2013, 05:40:32 PM

Hey there retep. Regarding the RPC, have you considered making their calls async ? I have rolled my own implementation that does that using the futures package, it might make everything a bit harder for the conventional user but there are situations where you simply cannot use blocking RPC. I will publish it later somewhere.

Patches welcome, though I'll warn you I don't actually know if bitcoind itself has the performance to make async RPC worthwhile.



I'm working on classes to do things like manage the blockchain information (eg. the set of all know blocks and block headers) which may lead to a model where it makes more sense to just do it all in bitcoinlib based on a internal database of blockchain information.



FWIW my personal motivation for working on all this stuff is to make it easier to prototype and experiment at a fairly low level for Bitcoin developers. For instance one of the first things I used python-bicoinlib for was to test some DoS attacks against the Bloom filter code and patches defeating those attacks. (that's why I wrote the CBloomFilter class, but haven't written the corresponding CMerkleBlock class needed to make it useful to SPV clients)

Patches welcome, though I'll warn you I don't actually know if bitcoind itself has the performance to make async RPC worthwhile.I'm working on classes to do things like manage the blockchain information (eg. the set of all know blocks and block headers) which may lead to a model where it makes more sense to just do it all in bitcoinlib based on a internal database of blockchain information.FWIW my personal motivation for working on all this stuff is to make it easier to prototype and experiment at a fairly low level for Bitcoin developers. For instance one of the first things I used python-bicoinlib for was to test some DoS attacks against the Bloom filter code and patches defeating those attacks. (that's why I wrote the CBloomFilter class, but haven't written the corresponding CMerkleBlock class needed to make it useful to SPV clients)

That is great, I appreciate the library.



Now, regarding the earlier comment, here is some code that does async RPC:



The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it. That is great, I appreciate the library.Now, regarding the earlier comment, here is some code that does async RPC: https://gist.github.com/anonymous/6627745 The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it.

Peter Todd





Offline



Activity: 1106

Merit: 1045







LegendaryActivity: 1106Merit: 1045 Re: python-bitcoinlib: Comprehensive bitcoin library for python September 19, 2013, 07:51:44 PM #9 Quote from: knowitnothing on September 19, 2013, 06:33:46 PM



Now, regarding the earlier comment, here is some code that does async RPC:



The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it.

That is great, I appreciate the library.Now, regarding the earlier comment, here is some code that does async RPC: https://gist.github.com/anonymous/6627745 The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it.

Your code uses the futures library - one of my goals with what I'm doing with python-bitcoinlib is to eventually be able to either port the performance critical parts to Cython, or even better, turn the "consensus critical" parts of the reference node codebase into a separate library that python-bitcoinlib could in turn use; possibly both approaches will be used.



With that in mind I'd rather that bitcoinlib have as few external dependencies as possible, in particular ones that tie code to a particular way of operating. Similarly with the pynode code I'm working on removing dependencies on the Twisted library so that an implementation of a node can use whatever threading model is desired. (possibly none at all)



FWIW Maaku also has a





EDIT: A good approach for your use-case might be to make a async layer library for the python-bitcoinlib library. Basically you would take the bitcoin.rpc.Proxy class and make a sub-class of it that maintained a thread pool and distributed work as required to those threads. If you need to modify the underlying JSON-RPC code as part of this effort I can work with you to make it possible for the jsonrpc module that the Proxy class uses to be replaced too. Your code uses the futures library - one of my goals with what I'm doing with python-bitcoinlib is to eventually be able to either port the performance critical parts to Cython, or even better, turn the "consensus critical" parts of the reference node codebase into a separate library that python-bitcoinlib could in turn use; possibly both approaches will be used.With that in mind I'd rather that bitcoinlib have as few external dependencies as possible, in particular ones that tie code to a particular way of operating. Similarly with the pynode code I'm working on removing dependencies on the Twisted library so that an implementation of a node can use whatever threading model is desired. (possibly none at all)FWIW Maaku also has a fork of python-bitcoinlib that he is working on and has used for his UTXO ledger prototyping, but he has taken the exact opposite approach and embraced using features that can only be done in interpreted Python.EDIT: A good approach for your use-case might be to make a async layer library for the python-bitcoinlib library. Basically you would take the bitcoin.rpc.Proxy class and make a sub-class of it that maintained a thread pool and distributed work as required to those threads. If you need to modify the underlying JSON-RPC code as part of this effort I can work with you to make it possible for the jsonrpc module that the Proxy class uses to be replaced too. BTC: 1FCYd7j4CThTMzts78rh6iQJLBRGPW9fWv PGP: 7FAB114267E4FA04

jgarzik





Offline



Activity: 1596

Merit: 1007







LegendaryActivity: 1596Merit: 1007 Re: python-bitcoinlib: Comprehensive bitcoin library for python September 19, 2013, 08:05:55 PM #10 Would prefer that the direction of python-bitcoinlib head in the Cython-ish direction, while continuing general pythonization.



Pull requests welcome, including the pythonize branch once its mature.



Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.

Visit bloq.com / metronome.io

Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj

knowitnothing



Offline



Activity: 294

Merit: 250







Sr. MemberActivity: 294Merit: 250 Re: python-bitcoinlib: Comprehensive bitcoin library for python September 19, 2013, 08:10:15 PM #11 Quote from: retep on September 19, 2013, 07:51:44 PM Quote from: knowitnothing on September 19, 2013, 06:33:46 PM



Now, regarding the earlier comment, here is some code that does async RPC:



The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it.

That is great, I appreciate the library.Now, regarding the earlier comment, here is some code that does async RPC: https://gist.github.com/anonymous/6627745 The motivation is not to send many commands at once to saturate (or some other better term) bitcoind, but just to not block the call. When combined with other services, I simply cannot make a blocking call and that is my motivation for doing it like that. Note that in this code, the important pieces (for these non-blocking calls) are the lines 74, 79, and the function generic_cb for handling the results from the async call. This is not a patch but maybe you (or someone else) could consider it (modified or not) for inclusion in the current code or another branch of it.

Your code uses the futures library - one of my goals with what I'm doing with python-bitcoinlib is to eventually be able to either port the performance critical parts to Cython, or even better, turn the "consensus critical" parts of the reference node codebase into a separate library that python-bitcoinlib could in turn use; possibly both approaches will be used.



...



EDIT: A good approach for your use-case might be to make a async layer library for the python-bitcoinlib library. Basically you would take the bitcoin.rpc.Proxy class and make a sub-class of it that maintained a thread pool and distributed work as required to those threads. If you need to modify the underlying JSON-RPC code as part of this effort I can work with you to make it possible for the jsonrpc module that the Proxy class uses to be replaced too.

Your code uses the futures library - one of my goals with what I'm doing with python-bitcoinlib is to eventually be able to either port the performance critical parts to Cython, or even better, turn the "consensus critical" parts of the reference node codebase into a separate library that python-bitcoinlib could in turn use; possibly both approaches will be used....EDIT: A good approach for your use-case might be to make a async layer library for the python-bitcoinlib library. Basically you would take the bitcoin.rpc.Proxy class and make a sub-class of it that maintained a thread pool and distributed work as required to those threads. If you need to modify the underlying JSON-RPC code as part of this effort I can work with you to make it possible for the jsonrpc module that the Proxy class uses to be replaced too.

Well, it does use the futures package and it was explicitly mentioned in my first reply. It is using futures because (it is easy to use and) it is available on Python 3.2 and newer, so it is not an external dependency there. It also doesn't involve any other large frameworks, like Twisted you mentioned.



Also, I'm surprised to hear there are critical parts requiring Cython here. What kind of usage do you have in mind that actually has a benefit with Cython ? Why not PyPy then ? I would think that the RPC calls alone are a much greater bottleneck than anything else, and the other relatively heavy code are already coded in C.



Your EDIT sounds reasonable, I will take a look into that later and post some code whenever I find some time. Thanks for looking at it. Well, it does use the futures package and it was explicitly mentioned in my first reply. It is using futures because (it is easy to use and) it is available on Python 3.2 and newer, so it is not an external dependency there. It also doesn't involve any other large frameworks, like Twisted you mentioned.Also, I'm surprised to hear there are critical parts requiring Cython here. What kind of usage do you have in mind that actually has a benefit with Cython ? Why not PyPy then ? I would think that the RPC calls alone are a much greater bottleneck than anything else, and the other relatively heavy code are already coded in C.Your EDIT sounds reasonable, I will take a look into that later and post some code whenever I find some time. Thanks for looking at it.