Bitsocket 2.0

Welcome to Bitsocket 2.0. It solves everything.

1. Scalable Architecture

To address the scalability challenge discussed above, Bitsocket 2.0 was rewritten from scratch, with a completely different paradigm of event processing. Let’s compare with Bitsocket 1.0 to see what has changed:

Bitsocket 1.0

In Bitsocket 1.0, every event was processed individually for every filter, and this resulted in the unmanageable explosion of in-memory event processing tasks with a complexity of O(m*n) . And as m grows exponentially, the event processor would fail to catch up to the speed of incoming transactions, leading to huge delays in event delivery. Sometimes, the job queue would grow larger than the Planaria database itself. Just to review, here’s what it looked like (Every event processing task takes place in-memory):

Bitsocket 2.0

Bitsocket 2.0 removes this problem by unbundling incoming events from outgoing push notifications. Instead of Bitsocket directly interfacing with the Bitcoin network, it interfaces with Planaria which functions as a realtime event database:

For example, if there were 1,000,000 new transactions per second, this would have been enough to kill (or slow down) the old Bitsocket 1.0 engine with 1000 clients, because that would mean 1,000,000 * 1,000 = 1,000,000,000 event processing tasks must be processed PER SECOND.

How about Bitsocket 2.0? Because the indexing and event processing are unbundled, each module (Planaria and Bitsocket) only needs to focus on its own tasks. Using the same example, now Bitsocket only needs to process 1000 tasks per second. Bitsocket scales just fine in linear relationship along with the client session size: O(n) .

Also, this goes both ways. The new unbundled architecture not only makes the Bitsocket side efficient, but also takes a huge load off of the Planaria side because now Planaria only needs to focus on writing O(m) . With Bitsocket 1.0, the occasional stress tests would not only slow down the socket notifications but also slow down the entire planaria database response time for ordinary queries. This is no longer the case since these two modules are now separate. Best of both worlds!

2. Never Miss an Event

Bitsocket 2.0 makes use of a powerful built-in feature of SSE (Server Sent Events) called Last-Event-ID which lets each client remember the last “checkpoint” of the stream and automatically continue where it left off when it comes back from offline.

This, along with the new persistent event database architecture of Bitsocket, means events are no longer ephemeral, and therefore you will NEVER miss an event!

Here’s how it works:

In the following example, a client makes an SSE subscription request to the /s/eyJ2IjozLCJxIjp7ImZpbmQiOnt9LCJwcm9qZWN0Ijp7InR4LmgiOjEsInRpbWVzdGFtcCI6MSwib3V0LnMxIjoxfX19 endpoint with a Last-Event-ID of 5e167232dad37a1127611446 , to which the server can send customized notifications:

=> Request

GET /s/eyJ2IjozLCJxIjp7ImZpbmQiOnt9LCJwcm9qZWN0Ijp7InR4LmgiOjEsInRpbWVzdGFtcCI6MSwib3V0LnMxIjoxfX19 HTTP/1.1

Host: txo.bitsocket.network

Accept: text/event-stream

Last-Event-ID: 5e167232dad37a1127611446

Everytime the Bitsocket server sends a new notification event, it updates the Last-Event-ID on the client side. Using this property, a Bitsocket 2.0 client always keeps track of the last timestamp of the batch of events it has received. And this timestamp updates every time the client receives a new notification, as seen below:

The id field represents the Last-Event-ID header, which is always a unique incrementing value. Note that the first request doesn’t have an id and shows undefined . This is because it only gets the last event id timestamp once it starts getting events.

On the server side, Bitsocket looks at these ID values and ensures that the notification returns the entire batch of events which:

match the query filter and have happened since the ID timestamp

Basically it takes the original query and creates a conjunction query by combining it with “greater than the last timestamp” condition, and then streams it in realtime.

For example, if your client went offline for 10 minutes and just came back online, it would resume the connection with the last event id it remembers. Then Bitsocket would notice the checkpoint and start streaming all events whose ID timestamp is greater than the ID value sent by your client.

And just like this, Bitsocket 2.0 can guarantee a complete delivery of all relevant events for your applications, and you will never miss an event.

3. 100% Consistent with Planaria

The most important innovation with Bitsocket 2.0 is that Bitsocket is now powered by a persistent event database (Previously all events were ephemerally processed, filtered, and emitted without a persistent event database).

This means the Bitsocket and Planaria database are exactly the same. There is no way an event may be triggered but doesn’t exist in the corresponding Planaria db, and there is no way that a transaction exists in a Planaria db but a corresponding event is never triggered, because now the event comes straight from the database.

With Bitsocket 2.0, the same Bitquery can be used to both:

Query Planaria Subscribe to Bitsocket events

And they would return 100% identical results over time. They are 100% identical because they both use the Planaria database.

4. More Powerful Queries

Because Bitsocket 1.0 filters worked by emulating MongoDB queries, a lot of more complex queries either didn’t work, or didn’t work accurately enough to be reliable.

Since Bitsocket 2.0 directly makes use of the event database powered by a true MongoDB backend, now you can filter anything a typical Bitquery can filter.

One such operation is project . This was not supported in Bitsocket 1.0. But with Bitsocket, you can easily listen to a subset of each transaction object tree:

{

"v": 3,

"q": {

"find": { "out.tape.cell.s": "19dbzMDDg4jZ4pvYzLb291nT8uCqDa61zH" },

"project": { "out.tape.cell.s": 1 }

}

}

Additionally, the r part of the query language has greatly improved. Instead of putting everything in memory, now everything is streaming based. The JQ engine that powers the r.f is now powered by StreamJQ. So the events are literally streamed live from MongoDB to JQ to the HTTP response directly to the client.

{

"v": 3,

"q": {

"find": { "out.tape.cell.s": "19dbzMDDg4jZ4pvYzLb291nT8uCqDa61zH" },

"project": { "out.tape.cell.s": 1 }

},

"r": {

"f": "[.[] | .out[0].tape[1].cell[2].s | fromjson]"

}

}

5. Efficient and Fast Delivery

In Bitsocket 1.0, every atomic event was a single item since all events were individually processed and emitted as they came in from the Bitcoin peer network. Basically, every new transaction created a new event processing task which was pushed to the global job queue. This resulted in the job queue overload. Eventually the job queue will never be able to catch up with the speed of incoming transactions whenever the transaction volume increases.

With Bitsocket 2.0, we have no such inefficiency because there is a buffer-the planaria event database.

The events come from the Planaria event database which is constantly indexing the entire Bitcoin transaction event universe in realtime. Because of this buffer (event database), all events can be batch processed and emitted in chunks. For example, if there were 10,000 relevant event items within the last 1 second, instead of having to create 10,000 separate atomic tasks which process and emit 10,000 individual events, now Bitsocket only needs to run a single task which makes a single roundtrip to the database and triggers a single event made up of 10,000 items. It would be 10,000 times more efficient and faster.

In the following example, you will notice the events come in in chunks: Each atomic unit of events is an array of multiple transaction items:

6. Reliable Scalable Infrastructure

The most important focus of Bitsocket 2.0 is reliability. As mentioned above, until recently, Planaria has suffered from the lack of reliability guarantee, because I didn’t have enough time and resources to build new tools as well as maintain existing ones.

But today, we are ready to move forward with more reliable infrastructure. The launch of Bitsocket 2.0 beta is the first step to this goal.

Bitsocket 2.0 no longer runs from a single server. We have implemented a scalable architecture which runs as a cluster of replicated and load balanced containers which can dynamically meet the usage demand. Of course this is the first release so it is not yet perfect, but we are starting with the right architecture which is fundamentally more scalable, reliable, and maintainable than before.

Also we are dedicated to making sure Bitsocket delivers realtime events lightning fast not just to a certain local region, but globally around the world.

Please try out Bitsocket wherever you are located and let us know (on Twitter or Atlantis slack) if it’s not fast enough. We will make sure to guarantee speedy delivery around the world.