ARK Utilities is a separate library providing common functions for working with data (sorting, filtering, …) and performing common tasks. All functions in ARK Utilities will make use of our custom designed benchmark framework to ensure it is the best candidate in each respective area.

ARK Utilities significantly increases speed and performance across the entire ARK Core, and this post will go into technical details while also showing actual test percentage increases.

ARK Core has come a long way over the past year. With the new codebase securely in place and a proper foundation set for our future development efforts, we can now focus on optimizing some of its functionality for better overall performance. As such, we have created a new dedicated library named ARK Utilities with the aim to provide a wide selection of commonly used algorithms tailored to performance.

ARK Utilities is available at: https://github.com/ArkEcosystem/utils

By introducing this new layer of dependency, we are able to replace many third party dependencies in Core with @arkecosystem/utils as those will now be part of the ARK Utilities library. Additionally, we can now provide a common API. This allows us to swap out or make changes to the implementations as we identify performance issues and other shortcomings, without affecting the API.

ARK Utilities is written in TypeScript and exposes commonly used functions while hiding the concrete implementation. It will fully be utilized with our upcoming Core v2.1 update, and will gradually get additional utilities as we introduce a series of optimized functions. This will also help other (non-ARK) projects to make use of this library as it will be independent from our Core code.

We would like to invite any and all projects who want to help improve the overall scope of commonly used crypto functions and libraries to suggest improvements, open GitHub issues, or provide pull-requests if they have ideas for how we can improve the library. We will be laying the first bricks with our upcoming release and we hope others in the greater blockchain community will help us continue to build ARK Utilities into a powerful resource for the entire industry.

What are some of these functions?

A lot of our code is using the popular lodash library. As many of you may already know, lodash is not exactly known for having the best performance, so we started by replacing a few functions with alternatives which proved to be faster in our benchmarks. Here are a few examples which will be part of the new ARK Utilities:

ARK Utilities leverages a few algorithms from the Fast.JS library which provides higher performance versions of many common JavaScript functions (map, filter, …).

For sorting we switched from Lodash’s sortBy to Fast Sort. As we expose the same API as before, we only have to change the import statement to benefit from it.

While we are still benchmarking and measuring, we can say that so far everything is faster than lodash .

That sounds great in theory, but give us some actual numbers!

Concrete numbers can better demonstrate the actual level of improvement so here are some of the performance results we have seen. These numbers show the increase in several different functions and were taken from a real working environment, and data from the actual ARK Network. We would like to note that these improvements become even more evident with larger data sets and the % increase in performance in some instances goes up by as much as 12x.

Utils.camelCase vs lodash.camelCase ✓ utils x 20,166,554 ops/sec ±2.98% (78 runs sampled) ✓ lodash x 402,498 ops/sec ±1.33% (88 runs sampled) Result: utils is 4910.35% faster than lodash. Utils.orderBy vs lodash.orderBy (10 items) ✓ utils x 1,842,682 ops/sec ±1.73% (81 runs sampled) ✓ lodash x 760,114 ops/sec ±1.73% (82 runs sampled) Result: utils is 142.42% faster than lodash. Utils.orderBy vs lodash.orderBy (100 items) ✓ utils x 410,181 ops/sec ±1.23% (87 runs sampled) ✓ lodash x 151,037 ops/sec ±1.58% (85 runs sampled) Result: utils is 171.58% faster than lodash. Utils.orderBy vs lodash.orderBy (1000 items) ✓ utils x 52,281 ops/sec ±0.68% (93 runs sampled) ✓ lodash x 17,441 ops/sec ±1.55% (89 runs sampled) Result: utils is 199.76% faster than lodash. Utils.orderBy vs lodash.orderBy (10000 items) ✓ utils x 5,282 ops/sec ±0.48% (91 runs sampled) ✓ lodash x 1,530 ops/sec ±2.43% (83 runs sampled) Result: utils is 245.26% faster than lodash. Utils.orderBy vs lodash.orderBy (100000 items) ✓ utils x 538 ops/sec ±0.80% (89 runs sampled) ✓ lodash x 80.41 ops/sec ±1.65% (64 runs sampled) Result: utils is 569.48% faster than lodash. Utils.sortBy vs lodash.sortBy (10 items) ✓ utils x 1,921,146 ops/sec ±1.33% (87 runs sampled) ✓ lodash x 723,453 ops/sec ±2.61% (78 runs sampled) Result: utils is 165.55% faster than lodash. Utils.sortBy vs lodash.sortBy (100 items) ✓ utils x 325,372 ops/sec ±0.52% (92 runs sampled) ✓ lodash x 153,786 ops/sec ±1.49% (87 runs sampled) Result: utils is 111.57% faster than lodash. Utils.sortBy vs lodash.sortBy (1000 items) ✓ utils x 34,702 ops/sec ±0.92% (91 runs sampled) ✓ lodash x 16,023 ops/sec ±1.37% (86 runs sampled) Result: utils is 116.58% faster than lodash. Utils.sortBy vs lodash.sortBy (10000 items) ✓ utils x 3,562 ops/sec ±0.46% (93 runs sampled) ✓ lodash x 1,529 ops/sec ±2.14% (83 runs sampled) Result: utils is 132.94% faster than lodash. Utils.sortBy vs lodash.sortBy (100000 items) ✓ utils x 363 ops/sec ±0.47% (88 runs sampled) ✓ lodash x 71.07 ops/sec ±2.94% (58 runs sampled) Result: utils is 411.23% faster than lodash. Native.map vs Utils.map vs lodash.map (10 items) ✓ native x 23,282,872 ops/sec ±1.55% (80 runs sampled) ✓ utils x 41,298,044 ops/sec ±1.95% (81 runs sampled) ✓ lodash x 6,841,156 ops/sec ±1.17% (87 runs sampled) Result: utils is 77.38% faster than native. Native.map vs Utils.map vs lodash.map (100 items) ✓ native x 2,411,309 ops/sec ±1.09% (89 runs sampled) ✓ utils x 5,051,863 ops/sec ±4.03% (84 runs sampled) ✓ lodash x 751,676 ops/sec ±1.61% (87 runs sampled) Result: utils is 109.51% faster than native. Native.map vs Utils.map vs lodash.map (1000 items) ✓ native x 245,745 ops/sec ±3.92% (77 runs sampled) ✓ utils x 458,229 ops/sec ±2.00% (85 runs sampled) ✓ lodash x 80,584 ops/sec ±1.19% (90 runs sampled) Result: utils is 86.47% faster than native. Native.map vs Utils.map vs lodash.map (10000 items) ✓ native x 28,034 ops/sec ±1.44% (87 runs sampled) ✓ utils x 45,515 ops/sec ±1.35% (86 runs sampled) ✓ lodash x 8,094 ops/sec ±0.95% (90 runs sampled) Result: utils is 62.36% faster than native.

Here is another example to show this increase at scale. We increased the sample number to higher numbers and tested sorting 100,000 wallets in descending order. In this case, the utilities function is 639.44% faster than lodash. As we continue to grow, the performance gains will only get more noticeable.

Utils.sortByDesc vs lodash.sortByDesc (100000 wallets)

✓ utils x 533 ops/sec ±0.48% (87 runs sampled)

✓ lodash x 72.12 ops/sec ±2.40% (59 runs sampled) Result: utils is 639.44% faster than lodash.

How do I use it?

ARK Utilities is available at : https://github.com/ArkEcosystem/utils

Install as a dependency:

yarn add @arkecosystem/utils

And import a function from @arkecosystem/utils :