Elasticsearch is fantastic for indexing and filtering data. But hey, you have your data on a Cloud Firestore in production. How do you copy all data from Firestore to ElasticSearch? And that too, while making sure that nothing goes wrong. Don’t worry, this is indeed possible. We at appbase.io have made an awesome CLI tool called ABC which will allow you to do this with a single command.

abc import --src_type=firestore --sac_path=<path_to_service_account_credentials> <elasticsearch_uri>

THAT’s it. Seriously, this is all you need to sync Firestore database to an Elasticsearch index.

The Steps

The first step is to install ABC if you have not done so already. So go to the GitHub releases page for ABC and download the most recent version. It’s a single no-dependancy binary so put it anywhere you like. We recommended putting it inside a PATH directory so that it can be accessed from anywhere in the terminal.

Ensure that ABC is working by running the following command.

$ abc version

Version: 0.7.0

Variant: oss

Go version: go1.11.1

OS: linux

Arch: amd64

Now let’s take a Firestore database and we are going to sync it to an Elasticsearch index hosted on Appbase.io. To get started, you will need to set up a Firebase project if you already don’t have one and then we will be making use of Firebase Admin SDKs to sync data from the Firestore instance.

If you don’t already have a Firebase project, add one in the Firebase console. The Add project dialog also gives you the option to add Firebase to an existing Google Cloud Platform project.

Create a new Firebase project

Navigate to the Service Accounts tab in your project’s settings page.

tab in your project’s settings page. Click the Generate New Private Key button at the bottom of the Firebase Admin SDK section of the Service Accounts tab.

Firebase project settings

After you generated the private key, a JSON file containing your service account’s credentials will be downloaded. Save the file to the desired location. We will be using this file with ABC. Note: This file is only generated once. If you lose or accidentally publish the key publically, you can follow the above instructions to create a new JSON key for the service account.

Next, let’s add some data to the Firestore project we just created. Click on the Firestore project icon on the console to navigate to the project dashboard. Expand the Develop section on the project navigation bar and click on Database and create a new database in test mode for now. Now, let’s create a new collection in the database by clicking on Add collection.

We will be creating users collection, so appropriately the collection id would be users. Click on Next to proceed.

Create a new collection

As shown above, we will need to populate the collection with some users. Leave the Document ID as is. Now insert id (Type=number, Value=1), name (Type=string, Value=foobar), email (Type=string, Value=foobar@made.up) and bio (Type string, Value=comic) fields for our users collections. Click on Save to add our first user to the collection. You can similarly add in more users by clicking on Add document button. After we are done populating our source Firestore database, we will create the sink Elasticsearch index.

Create your own app at https://dashboard.appbase.io

We then go to Appbase.io and create a new app called abc_firestore_test. The complete URL to this index looks like this:

So now we have both the source and the sink ready. It’s time for some ABC magic. Using this source and sink, we can build the import command. It will be as follows:

abc import --src_type=firestore --sac_path=<path_to_service_accounts_credentials.json> "https://USER:PASS@scalr.api.appbase.io/abc_firestore_test"

Once you run this command, you should see that the command will finish in some time with no errors. Now if you visit the appbase.io dashboard, you can see that the data has been transferred to the target ElasticSearch index.

Appbase.io dashboard

Voila. Everything works. The data has been transferred to ElasticSearch and that too without doing anything at all. Next, we will see how to transform and manipulate data as it goes from source to the sink.

Transforming Data before indexing into Elasticsearch

There are times when you don’t need the data to go as it is from source to the sink. You might like to change the target type name (example — users to accounts ) or you might like to remove certain fields (e.g. bio ) or create new fields. For all this, we have the transforms feature in ABC. It can be used by the transform_file parameter.

abc import --src_type=firestore --sac_path=<path_to_service_accounts_credentials.json> --transform_file="transform_file.js" "https://USER:PASS@scalr.api.appbase.io/abc_firestore_test"

The transform_file parameter takes the path to the transform file. That file contains the JavaScript transforms that should be applied to the pipeline. Let’s take the contents of transform_file.js as follows:

t.Source("source", source, "/.*/")

.Transform(omit({"fields":["bio"]}))

.Save("sink", sink, "/.*/")

In the above transform, you can see that we are going to omit the bio field from the data transfer. Now when we run the new import command, we should have the following result.

As you can see, the bio field was omitted when data reached the sink. More documentation on transform file can be found on GitHub. It supports lots of inbuilt functions like omit and even supports running custom JavaScript code as a transform. It’s a good idea to explore its documentation.

Summary

This article is part of the abc article series.

If you found this interesting, consider giving this article a 👏🏻 so that others readers can find it on Medium too. 🤓

Further Reading

ABC’s README is a good place to start if you want to learn more about the application. You can also have a look at the Firestore adaptor docs. Furthermore, you may ⭐️ the repo on GitHub and watch it to stay tuned for updates.CLI for indexing from Firestore to Elasticsearch