up2pp://

The “http://” of Peer to Peer networking needs to exist

With decentralization in the mind of Bitcoin inspired hackers and entrepreneurs worldwide, by creating a general purpose universal peer to peer protocol there is a great opportunity to:

Eliminate the risk of failure for not being able to implement peer to peer networking correctly.

Save them the time and hardship of building their own p2p protocols and solving the same issues over and over.

Enable millions of developers to innovate by easily creating decentralized applications, even on the web browser.

After having worked with Gnutella, creating our own message oriented peer to peer file sharing protocol, hacking a full featured BitTorrent client, and learning about the design and implementations of Kademlia based DHT details, you tend to see lots of similarities to the challenges developers had to face along the way, and you can see how they went about it in their solutions. You can also see that stable and proven implementations took years to get right.

Then after being on several projects where people without p2p experience start proposing reinventing the wheel and wasting precious and tremendous time solving these problems and not the ones related to the problem they were trying to face in the first place, it seems like it is time that we all have the chance to ride on something that works, a well accepted p2p standard protocol and implementation.

Imagine how hard it would’ve been to have launched your website if all HTTP servers were closed source and you had to implement the way your page gets downloaded to your visitors with your own protocol…

It must be easy to use, and easy to extend.

It must provide with a minimum set of functionality for anybody to implement peer to peer networking functionality, basically allowing any node to send a message to any other node reliably.

HTTP servers (aka Web Servers) are often used for a central node to have separate conversations with its client nodes, but clients are not able to send HTTP requests to each other, they’re deaf to each other, unless they relay messages through the HTTP servers they connect to, this is usually called a Server-Client network architecture, what we want to provide is a Client-Client architecture.

All peer to peer networks must solve the NAT traversal issue, on this network all nodes that can accept incoming connections and meet a minimum set of bandwidth and uptime requirements are promoted by other peers to act as proxy peers, thus allowing the routing of messages between two known nodes that are connected behind restrictive firewalls. Proxy peers use a portion of their available bandwidth to pass messages along.

The Universal Peer to Peer Protocol or UP2PP

With this protocol, any client should be able to specify a Message and send it to a Recipient or Recipients. Messages are sent via TCP or UDP, under the hood the implementation provides message relaying transparently, and the relaying algorithm implementation can be changed in order to allow for experimentation and future optimization.

The protocol provides with various common messages to avoid developers from reinventing the chicken, and to allow them to focus on the parsing of their own messages.

An Encryption Channel or Encrypted Message implementation would be ideally provided.

A Group of Recipients should be an expected feature, for delivery of 1 to N, it’d be nice if such implementation were somehow scalable, anybody could implement a twitter like network if there were enough bandwidth and processing power available on the network.

A MailMessage implementation that would allow for the implementation of email like message delivery, where a message could persist on the network for a plausible period of time on which the message can be received, say 6 months.

Most peer to peer services require a way of communicating peers reliably, currently everyone is scrambling for different ways to do this, and finding ways to do this anonymously is hard. Having this messaging capability would enable for tremendous innovation.

My proposed implementation would include the following commands:

sendMessage(peer : Peer,

. . . . . . . . . . . . port : int,

. . . . . . . . . . . . messageData : byte [<= 1000],

. . . . . . . . . . . . nonce : long,

. . . . . . . . . . . . minZerosOnNoncedHash : int)

sendMailMessage(recipientId : byte[160],

. . . . . . . . . . . . . . . . recipientPublicKey : byte[<=512],

. . . . . . . . . . . . . . . . publicKeyLength : int,

. . . . . . . . . . . . . . . . encryptionAlgorithm : byte, //DSA, RSA, ECDSA

. . . . . . . . . . . . . . . . messageData : byte[<= 4096],

. . . . . . . . . . . . . . . . nonce : long,

. . . . . . . . . . . . . . . . minZerosOnNoncedHash : int, //system wide difficulty

. . . . . . . . . . . . . . . . redeliverForNDays : int,

. . . . . . . . . . . . . . . . redeliveryIntervalInHours : int)

Encrypts message data, stores it locally for delivery, and passes encrypted message to nearest peers, all holders of this message keep trying to redeliver for N days, every redeliveryIntervalInSeconds, where this number can be forced to be a minimum number of hours network wide. The nonce and minZerosOnNoncedHash are there to avoid spam before sending the message to the first set of peers. With the use of a random nonce, the hash(nonce ++ messageData) should have a minimum of minZerosOnNoncedHash zeroes on the left hand side of the hash in order to be delivered, otherwise other nodes will refuse to try to store and redeliver the message to the recipient.

onMailMessageReceived(messageHashId : byte[160],

. . . . . . . . . . . . . . . . . . . . . . . . signedMessageHashId : byte[])

Every node keeping a copy of this message, will have indexed the message by the sha1hash of the encrypted messageData payload. They will also store the public key wich was used to encrypt the message. They will be able to verify the signature of the messageHashId with the public key, when verified, they can delete the message from their local storage and stop any attempt of delivery of this message. Probably makes sense to put the messageHashId on a list to not deliver in case a buggy or malicious client tries to keep delivering the same message (spam)

Thinking that storage keeps growing and getting cheaper at an exponential rate, we’ll easily be able to save a few hundred megabytes of p2p unread messages on every node. Earlier implementations would impose certain limitations on the message size to make the network realistic in today’s hardware and networking capabilities. Say, 15 day re-delivery, Limited Message size of 4kb so you could store and retry sending 25,000 messages with just 100Mb of disk space, attachments would be links to p2p content, like torrent info hashes)

All clients in the network connect to the libtorrent DHT, and the procol allows for ways to interact with this DHT in a generic manner, here’s a few commands it would provide to interact with it:

(this would basically make a up2pp client a libtorrent dht client)

getPeers (associatedHash : byte[40], . . . . . . . . . callback(peers : [ Peer ]))

Returns a list of peers announced under a given hash.

announcePeer( associatedHash : byte[40],

. . . . . . . . . . . . . . . port : int,

. . . . . . . . . . . . . . . callback())

Announces a peer and a UDP/TCP port to connect to, in relationship to the given hash.

putItem(randomTransactionId : itemData : byte[<=1000])

Should we have Paid Message Delivery/Storage?

It should be for debate allowing for paid messaging, some service might want to prioritize message delivery, and up2pp clients could become a market that competes over the delivery of such messages.

Basic messages would forcibly be free, messages that would break established limits would be paid for. Say you want to put more data than allowed in a MailMessage or on a DHTPut, a sub-network of clients willing to take Bitcoin for providing such services would be there. This should be a subject of good debate, as we wouldn’t want the commercial aspect of the network to make the basic free p2p services to be throttled down in favor of the paid stuff.

Implementations

I envision a cross platform implementation shared library in C or C++, which can then be used, initially as the core of a browser plugin which provides all the p2p network services to the browser.

Ideally the plugin would extend things like url schema parsing to include upnp:// urls, and a javascript api that allows the browser to interact with the universal p2p network. It would also enhance the browser settings to let the user control how much resources to allocate (storage space, bandwidth throttling), or to toggle p2p connectivity.

In a perfect world implementations for Firefox, Chrome and Opera would be merged into the browser code itself, thus making every Firefox, Chrome and Opera instance out there support the protocol out of the box, thus forming the largest known p2p network.

Web servers could also be part of the network as they are online most of the time, have much more bandwidth capabilities and don’t have NAT issues to deal with. I envision the integration with webservices through configurable modules/extensions that depend on our shared library. Think nginx, apache, lighttpd modules and their respective configuration directives.