Highway rider / aiohttp + Tarantool vs. aiohttp + asyncpg / Python 3.5 vs. Python 3.6

As the first benchmark, I’ll use the aiohttp web framework, one of the first asynchronous (asyncio library in particular) web servers for Python. Pure Python code, no cheats, no tricks.

Additionally, I’m going to test the performance of different versions of Python (3.5.3. and 3.6.1).

Tarantool

Let’s see what Tarantool has to offer: https://tarantool.org.

There are MessagePack tuples, Lua application server, memory allocation mechanism and write-ahead log inside Tarantool. All data is stored in memory and processed incredibly fast.

MessagePack is faster and smaller than JSON. It stores messages in bytecode, and you can’t read them like JSON messages.

When you create a new record in Tarantool, it allocates slots of memory for this tuple and provides access to it. All tuples (they might be called rows) are stored in spaces (they might be called tables), and tuples are divided into fields. Spaces have primary or secondary keys. Each field can contain simple data, like integer or text, or compound data, such as a tuple or an address link to another table. Every tuple can be extended with new data, but mandatory data must be entered into fields that have primary or secondary keys set on them. Keys support the TREE, RTREE, BITSET and HASH indexing algorithms.

Each operation (with data in a database) is a built-in Lua function. For example, to run a join query you have to implement you own Lua function that joins the tables. Within this function, you need to iterate over one record in a space and add data from records of another space, and then return a consolidated dataset.

Configuring Tarantool is simple. In fact, the configuration file is a long Lua script, where you declare spaces, fields, indexes, create users and can implement custom data transformation functions. Moreover, you can run a web server and all other features provided by Lua.

Lua’s performance can be compared to that of Java. Lua is much faster than Python, but a little slower than Java. In my case, I needed to take into consideration that Tarantool calculates data faster than the main application with Python would.

Snapshotting is a mechanism that provides on-disk storage for in-memory data. Tarantool loads snapshots at startup and then additionally loads data from the WAL for unsaved transactions. You can safely stop and run the Tarantool server without losing your data.

Replication (master-slave, master-master) and sharding mode allow building very sophisticated architectures. The built-in Lua language makes it possible to handle all database architecture-specific aspects within Tarantool, so you don’t need to overload your application code.

As for stability, Tarantool is stable alright. I haven’t noticed any problems or bugs. If you reserve 128 MB of memory and use it all up, you will get a “could not allocate memory” message on insert, but you’ll still be able to connect to the DB and read data and, in some cases, upsert records. There’re no crashes or lags. But you should keep in mind that this is all about memory space. You can’t run three 256 MB Tarantool instances at the same time on a server with 512 MB of RAM. Well, actually, you can do this, though the more data is stored in these instances, the more memory they need, and at some point the OS will start killing them to free some memory.

My buzzword list for Tarantool

aiohttp-tarantool-multi — aiohttp web framework creates a new Tarantool connection on each page request. The scenario is the same as the PHP example mentioned above.

— aiohttp web framework creates a new Tarantool connection on each page request. The scenario is the same as the PHP example mentioned above. aiohttp-tarantool-one — aiohttp web framework creates one Tarantool connection and uses it all the time. The script auto-reconnects to Tarantool if the database goes offline for a while or if it loses connection (bonus feature provided by the aiotarantool library).

— aiohttp web framework creates one Tarantool connection and uses it all the time. The script auto-reconnects to Tarantool if the database goes offline for a while or if it loses connection (bonus feature provided by the library). aiohttp-tarantool-one-uvloop — same as aiohttp-tarantool-one, but using uvloop instead of a standard asyncio loop.

Tests results / Tarantool + aiohttp / asyncio vs. uvloop / Python 3.5 vs. Python 3.6

Brief conclusions

I have ~1000 RPS on the Index page and ~1500 RPS on the Good page with Tarantool.

page and ~1500 RPS on the page with Tarantool. The cost of connecting to Tarantool is very high — ~110 RPS on the Index and Good pages with connections to Tarantool on every request.

and pages with connections to Tarantool on every request. About and Ugly show the same RPS as PostgreSQL (see below), because only aiohttp is used and no DB requests are made.

PostgreSQL

PostgreSQL has a new JSONb type and allows working with it as with native JSON storage. I recreated the data architecture with this JSONb feature — and this was painful.

Another way was to use PostgreSQL’s array feature, but when I found this feature, it was too late to rewrite the whole code. Below is an example of how to work with tuples in Python as part of JSONb data in Postgres.

Upsert with PostgreSQL and JSONb was hard to code and to test. You can find an example in the source code. The statement looks something like this:

INSERT INTO sessions (data)

VALUES (

'[

"dcddbc803b387a2fd3a35087ea613420",

1491829389,

0,

"192.168.99.1",

"browser info"

]')

on conflict (((data ->> 0)::uuid))

do update set data =

jsonb_set(

jsonb_set(

jsonb_set(

jsonb_set(SESSIONS.data, '{1}','1491829389'),

'{2}',

(select (((data->>2)::int+1)::text)::jsonb from sessions

where data->>0 = 'dcddbc803b387a2fd3a35087ea613420')),

'{3}', '"192.168.99.1"'),

'{4}', '"new browser info"')

For a database driver, I’m using asyncpg , currently the fastest asynchronous Python driver.

Problems

The first time around, I received a poor result: about 64 RPS on Index and 100 RPS on Good .

After rewriting the select statement, taking one sticker from a table with 16k records and rechecking the code to remove the fetch function (replaced it with execute) on delete, the update statement produced the following result: 190 RPS on Index and… 110 RPS on Good … Well, that was strange.

By searching, investigating and giving prepared statements a try, I achieved 650 RPS on Good on my dev machine. But I couldn’t reproduce it on the t2.nano instance. The code would crash and produce an error: “cannot perform operation: another operation is in progress”. My best guess is that prepared statements with asyncpg can’t work on a single-CPU server, but this issue is beyond the scope of the article.

The next try was to find the root cause of the 100 RPS problem and run explain for the SQL statement. As a result, I found out that indexing wasn’t done in the right order. I fixed it as follows:

CREATE INDEX stickers_data_indx_rating_1 ON stickers

USING BTREE(((data ->> 1)::int));

This index fixed the performance issue, see the results below.

Tests results / PostgreSQL + aiohttp / asyncio vs. uvloop / Python 3.5 vs. Python 3.6

Best PostgreSQL results vs. best Tarantool results with aiohttp

Conclusion

In my example, Tarantool proved ~7 times faster in the DB-intensive scenario and ~3 time faster in the rarely intensive scenario than PostgreSQL.

In all scenarios, uvloop was faster than a standard asyncio loop and Python 3.6 was faster than Python 3.5.