Enigma Developer Update, October 16th, 2017:

Optimizing for bandwidth and usability

One of the great features of Catalyst is that it provides historical pricing data for all currency pairs traded on a given exchange, so that you can backtest your trading strategies with ease. We recently announced the provisioning of minute-resolution data and outlined the scope of the data volumes Catalyst processes in the backend: 2.5 years of trading data on the Poloniex exchange translate to 221 million recorded trades that take up 16GB of disk space, which in the end Catalyst ships to you in ~340MB.

When we forked Catalyst from Zipline several months ago, we inherited a serious limitation in how all this pricing data is bundled and handled. Originally, the data bundles were monolithic, which meant that every time you wanted to update your historical pricing data, you had to download again all the data for all prices for all coins since the beginning of trading. This is a very inefficient use of bandwidth — since all you need is an incremental update — and of disk space on the side of the user, where old bundles pile up after continued use.

We’re pleased to be on track for our next release of Catalyst coming this week, which will address this and many other issues. Last week we discussed naming standardization and market coverage for Catalyst. One of the additional features that we will be introducing in our upcoming release this week is providing backtesting data on a per exchange basis, instead of one universal set. With the original monolithic data bundles, this addition would have made things even worse. Thus it became a priority for the team to address this limitation.

We are finishing the processing of historical pricing data for the Bitfinex exchange, which started trading on March of 2013 and thus involves 4.5 years of trading data, although most of the trading pairs have only been added in the past year. Still it is another 1.5 GB of pricing data, that Catalyst brings down to a mere 50MB for shipping.

Moreover, we are also adding the functionality to choose what currency pairs you want to download historical data for, and over which time intervals, so that if you are anxious to get started testing your trading strategies on the first run, you don’t have to wait for a lengthy download. All data downloads are treated incrementally, so the next time you need the same or a different pricing data set, Catalyst will figure out what you are missing, and only download what you need. Pretty sweet :)

Stay tuned for more news about our upcoming release later this week, and get ready to fine tune those algorithmic trading strategies and put them to work live on the exchanges!