This is part 3 of how and why our system works (part one: The stack discovery, part two: The API dissection) and I’ll try to explain how we built our commercial API service.

We’ve been providing our free API service for over 3 years now, this makes us one of the oldest and most experienced API developers in the crypto space. We managed to keep everything running and with 99.99% availability even through the crazy 2017 boom and then the slow 2018 crash. It wasn’t easy, we handled over 50,000 calls a second in Dec 2017 / Jan 2018 and our APIs were still chugging along. Some of us lost a few nights to keep it all together but we’d be lying if we said we did not enjoy it.

From 2 servers and 100 calls a second to 25+ servers and 50,000 calls a second, it was a once in a lifetime crisis management, scaling and coding experience.

We are now at around 30M calls an hour and since advertising revenue from the site started slowly diminishing (lead mostly by the price crash) we started looking at other sources of revenue to support our growing business. At the same time, since our previous API license only supported non-commercial use cases, we started getting a lot of requests for a commercial offering. While working on scaling and improving the API, we allowed over 3,000 businesses to use our data commercially for free.

Our streaming service is handling an average of 25,000 connected clients and it’s the next service we are looking to document properly. The only documentation available now is on GitHub, thanks to Quynh Tran-Thanh and Jordan’s work.

API keys are easy, right?

The journey was a long one. We deployed a new user management system (June 2018), with better password management, 2-factor auth (it was fun implementing the RFC) and a lot more security features as well as much easier and better scalability. Not to brag too much, but it’s cool, it uses a Redis Cluster, a Postgres Cluster and an ElasticSearch one. All data is replicated on 3-node clusters, with the Postgres cluster being the source of truth and used for reporting. The Redis Cluster is used on the data access layer and ElasticSearch for some reporting (Kibana) and for username searches.

When you register or update your account info, we create entries in Postgresql, Redis, and ElasticSearch. Your interactions with the website generally happen though Redis and ElasticSearch. We managed to reduce our “user management” costs by 83% for close to 1 million users. Our TTFB (Time To First Byte) reduced from 750ms to under 100ms.

We needed to do the user migration so we can have more control over how our/your user data is stored and over the security and authentication aspect. After this was done, we were ready to issue API keys.

Now that we had all the process in place to issue keys, it should have been straightforward to just get a few keys for your account.

Permissions are easy, right?

The first problem we ran into was permissions: should API keys have the same permissions as auth keys? Of course not, you want a lot more granular control over your API keys. After a few weeks of changes, we finally had the code to be able to issue permissions and attach them to your keys. It did not take long, as we knew this would be a requirement when we were developing our new user management system. Mapping all the permissions and thinking about all the possible scenarios was not a quick task. Our “permissions matrix” looks something like this:

We had the keys, the users and the permissions so we just needed to put it all together.

Rate limiting is easy, right?

Rate limiting in a load balanced environment at 50,000 requests a second is not as easy as it sounds. If you ever meet us — just ask Jordan. Our previous limits were designed around sticky sessions by IP so if you hit API01, you would keep getting load balanced to API01. It worked really well and fast as each server only had to care about its own hits but it was really hard to monitor and account for. Storing historical data was difficult and inefficient and if the server was restarted, the rate limits would be lost.

We decided to centralize it and use a Redis Cluster to handle the rate-limiting load. We wanted proper accounting since it was something we would charge for and we also wanted granularity (per second, per minute, per hour, per day, per month). The diagram itself was easy:

Trying to count at scale is not as easy as the diagram, our 3x2 cluster (3 masters with 1 slave each) is handling around 250,000 requests per second on average. We are running at around 50% capacity, when we get to 80% we’ll add more servers to the cluster and rebalance.

You can read more about how we implemented Rate Limits in our Rate Limits with Redis-Cluster and NodeJs blog post.

With all of this in place — and it was a struggle to get it working efficiently and well, we could now build a dashboard that we’d be proud of. We’re using our API with our keys for internal scripts as well. I love the ‘eat your own dog food approach’, it keeps us honest on the customer side as we go through the same experience. This is what part of our internal scripts dashboard looks like:

2 of our scripts: The Top List Volume is used to generate the homepage and coin list order and the 7 Days Chart is used to generate the spark charts on the same pages (thanks Jordan)

All of the above took around 3 months with a small development team. James mostly focused the long and frustrating process of migrating the API code to our internal framework and on managing the team.

The next and last part was payments.

Payments are easy, right?

Generally speaking, payment providers handled most of the logic, but we still had to do quite a bit of process design. All in all, it took some effort to get it working properly end-to-end with recurring payments, correct tax reporting and an interface we liked. Joe did most of the dev/backend work and the Paul/Jerome combo (UI & UX) did an amazing job on process flows and interface.

What’s available?

The API is available at a personal and commercial level, in 4 different tiers. The personal plan is available for free and gives individuals 100,000 calls/month to explore the capabilities of the API for non-commercial research and personal projects.

The commercial plans offer three different tiers, suitable for developers, businesses, institutional and enterprise organisations looking for a fully-licensed service. The plans start from 250,000 calls/month, covering full historical data and more than 20 endpoints.

Some highlights

The free cryptocurrency data plan enables developers to explore the capabilities of the API for non-commercial research and personal projects . This can also serve as sandbox testing for businesses before committing to a commercial subscription.

. This can also serve as sandbox testing for businesses before committing to a commercial subscription. The API keys page tracks usage and helps you pick the plan you need.

and helps you pick the plan you need. All tiers can access full historical data, key to trading strategy backtesting and quantitative research and analysis.

The commercial tiers include data redistribution rights , enabling third parties greater flexibility in using CryptoCompare’s data for their crypto investment products .

, enabling third parties greater flexibility in using CryptoCompare’s data for their . All tiers benefit from dedicated support .

. Businesses can request custom endpoint access tailored to their market data needs as well as fully separated API servers with unlimited use .

access tailored to their market data needs as well as . Enterprise organisations can benefit from bespoke solutions with custom data, service level agreements (SLA) and data caching / local storage.

How will this affect existing users?

We will continue to support our existing public API, with IP rate limits. However, its support and limits will be reviewed and reduced over time as we slowly get people to switch to using API keys. Open source and free products will always have a generous IP rate-limited service as well — but the limits will be lower than using a free key. As such we recommend our public API users to switch to a free or commercial API plan for a more secure and scalable service.

How do we deploy?

We use a mix of Ansible, Nodejs, Nginx and the forever NPM module to make sure we can deploy with no downtime. To read more about our deployment process have a look at our Deploying Node.js at Scale blog post.

What’s coming next?

The data products team is currently working on adding extended minute level history, tick data, blockchain data, and social data. A full minute and second history rather than just the last 7 days should be available soon as we have all the historical trade data and we are slowly rebuilding on top of it.

The data integrations team is working on making level 1, level 2 and level 3 order book streaming and historical data available through the API. We have it all available internally, it’s just an exercise in scaling and making it available externally.

Our streaming service will benefit from the same level of care as streaming order book data and streaming historical data will be added. This is all coupled with a new documentation section, API keys access, increased resilience, and stability as well as a ton of other features and message types. This is the only documentation we have now for our streaming service but there is a lot more coming soon.

We’re all really excited to continue improving our APIs

This is part 3 of how and why our system works (part one: The stack discovery, part two: The API dissection)