I am currently looking for a better way to save and retrieve and save data for my favorite side project. I did some research and I thought it might be nice to share what I found out so far.

Requirements

The app is meant to run as pwa as well as electron desktop app and uses rxjs to a wide degree. At the moment there is no backend and if there ever will be one it will probably be optional. As the main focus is the electron app I fortunately don't have to care too much about browser support, though all the libs seem to have support for all modern browsers.

These are the requirements I came up with:

Essential

Super speed: it should feel as fast as a native desktop app for about 2000 different entities at minimum and up to 10mb of data stored and quick to load combined data and subsets of data

Relatively easy to implement without too much overhead

Future proof: should be under active development and supported by a big community

No UI Blocking should occur even when working with larger datasets

Framework agnostic

Bonus

Working typescript support (at least no errors)

Readable indexddb model => the data should be browseable via chrome dev tools

Ideally should provide the ability to get a dataset as an observable or it should be easy to implement such thing

Replication to a backend server: though not yet implemented, it might be nice to have the option

Bonus bonus: simple file storage replication to google drive, owncloud or dropbox

The competitors

As a starting point I used this list. Please keep in mind that I am by no means an expert for any of the libraries used here, so chances are that I missed something important. If so please let me know in the comments.

These are my picks:

WatermelonDB

Description: Reactive & asynchronous database for powerful React and React Native apps

advantages

Sync capabilities build in

Quite fast thanks to LokiJS

Includes a query language similar to mongodb

Schema validation

Database interactions run inside a worker so no risk of ui blocking

Write sqllite to disk (which might be nice to use with electron)

drawbacks

Typescript Errors

Strongly focused on react

Some issue with fast consecutive writes (but that might have been me using it wrong)

IndexedDB is just stored as one big string

RxDB

Description: A realtime Database for JavaScript Applications

advantages

Sync (thanks to PouchDB probably the most advanced)

Nice Syntax

Wide framework support and examples

Schema validation via JSON Schema

drawbacks

Very slow for larger datasets

Render blocking occurs when interacting with larger datasets (this might be my fault again though)

Data is stored separately but is not easy browsable because of all the PuchDB stuff

Dexie

Description: A Minimalistic Wrapper for IndexedDB

advantages

Schema Validation

Relatively fast

Clean Database usage for IndexedDB

Framework agnostic

drawbacks

Not clear to me yet if syncing is easy to implement

No real observable interface (yes there is dexie observable but, but that is just for watching change events rather then getting updated data)

LokiJS

Description: javascript embeddable / in-memory database

advantages

Fast

Includes a query language similar to mongodb

Framework agnostic

Less overhead than RxDB and WatermelonDB

drawbacks

No sync capabilities built in

? No observable data interface ?

remotestorage

Description: An open protocol for per-user storage on the Web

advantages

Schema validation

Own your data: Sync to google drive / dropbox and more

drawbacks

Out of date dependencies

No active development in the last couple of months

Smaller community than the rest

Native IndexedDB

advantages

no overhead

relatively fast

framework agnostic

typescript should work

drawbacks

completely custom sync required

No observable interface

probably lots of code to write

not all browsers support all the features

Speed Test

This might be outdated but apart from my own non systematic tests I used this tool to compare the performance.

Premature Conclusion

So far I've only experimented with WatermelonDB, RxDB and a little bit with native IndexedDB. Here are the branches for WatermelonDB and RxDB in case you want to check for yourself:

https://github.com/johannesjo/super-productivity/tree/feat/watermelonDB

https://github.com/johannesjo/super-productivity/tree/feat/rxdb

WatermelonDB put me off because of its strong focus on React and it not working well with typescript atm. It shares one of the "problems" with LokiJS: The data in stored in IndexedDB is just a big string. This might not be a real problem, but it feels wrong to go about persistence like that. I didn't come much further than playing around a little with LokiJS, because I felt that it might be basically WatermelonDB with less issues, but also less of the stuff I need and I was still hoping for something better.

RxDB looked pretty promising and I assume that it really shines when you are using a server. But the bad performance was just not an option for an app that aims at desktop app like performance.

Using native IndexedDB should work fine for the most part. The main disadvantage is that I would have to write a lot of code and I don't want to re-invent the wheel (just quite yet).

Next step is Dexie! The community might not be as big as the ones of it's competitors and the lack of a prebuild performant way to use collections as observables is a little bit disappointing. I am also unsure about the sync stuff, as I didn't find too many good examples and it seemed to as if it was more of a gimmick than something people are using in production, but I hope I am wrong! So far it has been the only library offering a clean IndexedDB to browse while at the same time offering some structure making sure I am not the only going about saving data this way.

I'll update the article once I know more!

Did I get something wrong? Probably! This why I'm very curious to hear your thoughts and experiences on the matter. Please share them in the comments!