The Promise

We’ve already established that querying data, doing computation and almost everything is faster in memory than on disk, so it’s no different for a database. We could store all our data in-memory and access it with SQL. MemSQL and Apache Ignite both offer this capability. The appeal of doing this should be obvious. Take MySQL or Postgres and shove it all in memory, distribute it with the same APIs, it sounds like a good time and it is a good time.

The Expense

In-Memory Databases require tons of memory, since you are trying to 1-to-1 put data from disk to memory. Additionally, they need memory to just operate, and they tend to need A LOT of it. Operationally, both MemSQL and Apache Ignite don’t have any glaring “gotchas” and are fairly beginner friendly. In my opinion, these Databases are better used for distinct use cases, like an analytics layer or real-time needs and not for application layer.

Notable Projects: MemSQL, Apache Ignite IMDB

Conclusions

Distributed in-memory data processing is here to stay and we should see more projects come up that take advantage of it. Overall, the tooling is very good and relatively mature. If costs are a concern, there are often better options than going with these solutions and even if costs are not a concern there are some considerations with using memory centric solutions in each of these domains. Consider each carefully, experiment and test the ergonomics!