Migrate all of my big datasets from Firebase to Hasura

Steps as follows:

Create the Postgres schema in Hasura

Put the database and the Hasura server in production

Reconstruct all of the data

Create the script to migrate the data from Firebase to Hasura

Update the front end and the backend (queries and authentication)

Create the Postgres schema in Hasura

My Firebase schema was pretty easy to reconstruct in Hasura. To create the schema in Hasura it’s pretty slick with the interface

Reconstruct all of the data

My Firebase data was normalized, but I deconstructed some data for performance optimization.

You know, things like this:

I reconstructed all of my data to have good schema with a nodeJS script.

Put the database and the Hasura server in production

We are a small team and I want every service to be hosted in the cloud and fully managed (backup, scaling in 1 click…).

Hasura provides a docker image; so it was pretty easy to deploy it on the Google Cloud Platform. (Thanks to the Google Cloud credits provided by Google Cloud).

Create the script to migrate the data from Firebase to Hasura

To migrate all of my data, I wrote a script that does graphQL mutations directly to my Production server.

It wasn’t an easy task because of the size of the dataset. Firstly, I download the JSON file from the RTDB of Firebase, then normalize + flatten the data and there is NO WAY to do it with firestore, I had to query data with their API (time consuming, because we have to worry how to not overload Firestore)!

I use intensively the library underscore.js to migrate data.

The pick and the chunk functions were usefull:

Pick to normalized objects, because in Firebase, sometimes, it’s not normalized and your graphQL update will not work because it’s SQL and not no-SQL 😈

Chunk to create small bulks update because one full update wasn’t working (data was lost).

Update the frontend and the backend (queries and authentication)

- Queries

Querying with my frontend and the backend (just have to import fetch on nodeJS) was nearly similar because, it’s using apollo and it’s the same structure. Things that I haven’t with Firebase before because it was a different library for the frontend and the backend.

The only difference was that there is less boilerplate in Angular thanks to Apollo-angular (https://github.com/apollographql/apollo-angular).

After the update of all the queries, the frontend was responsive (not freezing anymore) and has a major speed improvement, but data was still heavy on the network.

There is nothing on the Hasura doc about compressing the response from Hasura, I had to search in the issues and I found that they are using Caddy to manage the server: https://github.com/hasura/graphql-engine/issues/1253.

Basically, compressing the response was just adding one line of code to the caddy file. The response was now compressed with GZIP 🔥 .

- Authentication

Authentication is like with Firebase, we can define ACL for permissions. When querying Hasura from the front, we pass in the header of the request the Firebase auth token, then Hasura send it to the authentication server. The authentication server decode the token received and send it back to Hasura with the custom claims (group permissions + user id).

All of our ACL are based on the user Id or the custom claims. We passed in the custom claims the role of the user.

🎉 Finally

I deployed all this migrations one night, it takes me like 10 minutes. I lost 10 minutes of data, but that was ok and not impacting our business. 👌