ARK Core continues to evolve! For the 2.4 upgrade we focused on Peer 2 Peer (P2P) layer switching from API to WebSockets. P2P is a communications layer where interconnected nodes (“peers”) share resources amongst each other without the use of a centralized administrative system. It is how nodes that run Core interact with each other on the ARK Public Network. The switch to use WebSockets brings better performance, higher resistance to some types of potential attacks and real-time exchanging of information. This is only one of the changes in this release. This post will cover the most significant changes in today’s upgrade.

In addition to Core v2.4 upgrade, there is a new Desktop Wallet update available as well, v2.4.1. Please update your wallet to properly interact with the Core v2.4 update. There will also be larger Desktop Wallet update coming soon, so please stay tuned.

Download newest Desktop Wallet v2.4.1 that supports Core v2.4 at: https://wallet.ark.io (SHA-256 hashes at the end of this blog post)

What’s New in Core v2.4?

Since the biggest change was the switch to WebSockets protocol, lets go over the reasoning and benefits behind this first. Before the switch, P2P communication was done via API (HTTP requests). Whenever nodes made a request (blocks, transactions, wallets, …) a connection was opened, data was transferred, and then after completion the connection was closed. This continuous opening and closing creates a lot of overhead, and for ARK’s Core that relies on constant responses or real time interactions, this isn’t the most viable long-term approach.

WebSockets is a very flexible protocol for transferring data via P2P. Unlike API, the socket that is connected to the server stays “open” for communication. That means data can be “pushed” to the other node in real-time on demand, without creating additional requests.

We chose Socketcluster, which is a pure WebSocket framework solution, to help us with P2P WebSockets implementation. Some of the features it supports:

Scales linearly as you add more CPU cores and workers.

Scales horizontally across multiple machines.

Resilient on both the client and backend — Process crashes, lost connections and other failures are handled seamlessly.

Supports custom codecs for compressing messages during transmission.

Supports both pub/sub channels and direct client-server communication.

Authorization via middleware functions (access control for emit, publish in, publish out, subscribe and handshake).

Client sockets automatically reconnect (by default) if they lose the connection.

Designed to work alongside any database/datastore.

Another new addition (which was scheduled to be implemented in a later version) is initial implementation of parallel block downloading and batch block database inserts when syncing. Before v2.4, while syncing, nodes could only download chunks of blocks from a single other node of the ARK network. With the introduction of parallel block downloading, we have increased this number to 25 concurrent connections to different nodes throughout ARK network. This means instead of only using a single node, it now downloads blocks simultaneously from 25 different peers. This not only speeds up the syncing process, but it also levels out the network traffic by spreading it across multiple nodes.

The transaction pool has been improved to handle faulty data more accurately and to make it even stricter for bad or malicious data to enter the pool, making it more efficient and safe.

The JSON-RPC we offer, formerly known as Core-JSON-RPC , has received a rework to turn it into a real RPC that is easier to use and maintain. The biggest change is that it now offers programmatic use to make integration into ARK Core easier while simultaneously allowing it to run as a standalone application detached from a relay.

Various improvements have also been added to ARK Deployer, making bridgechain deployment even easier and more user friendly. We have implemented Explorer as a Core plugin and added a new Wallet API that caters to the needs of developers and ease of integration in end user products.

We have also implemented replay command to verify blockchain data from block 0, making it easier to do verification of all data in the database the same way it is done during sync, but without storing the data again (replay blocks in-memory from the local database).

What’s Next for Core?

While we just released v2.4, our v2.5 upgrade is almost complete as well and will go live for testing on the public development network in the upcoming days. This won’t be as tremendous of an upgrade as v2.3 to v2.4 was, but will focus on deprecating API v1 support. You can read more on this here — if you are still using API v1 for anything, make sure to update to API v2 as soon as possible. Numerous improvements will also be made on making the code leaner, removing obsolete (legacy) code, and making it even more developer friendly.

Our next major change will be v2.6, for which we can be certain will be THE BIGGEST and most VALUABLE upgrade of the ARK network — AIP 11 (new transaction types), AIP 18 (multisignatures) and full AIP 29 support (easy custom transaction type integration) are just some of the major improvements coming to the network, but since we don’t want to spoil everything in this blog post we’ll cover this in one of our upcoming blog posts. Stay tuned!

All Changes From v2.3 to v2.4

For node operators of the current ARK Core, in order to update, please follow these migration instructions (v2.3 to v2.4) before you do the update:

https://docs.ark.io/releases/v2.4/migrating_2.3_2.4.html

Changes In Numbers

11 different developers contributing to the Core.

contributing to the Core. 234 new commits to the Core.

to the Core. 1,049 files changed in the Core.

in the Core. 47,980 code additions to the Core.

to the Core. 37,603 code deletions in the Core.

As we move towards 2.6 and the completion of AIP11, AIP18 and AIP29, there will be various breaking changes. The main concern of most developers will be breaking changes to @arkecosystem/crypto , so go through the commits listed below and make sure you adjust everything in your application that is affected by a change.

Breaking Changes

Implement WebSockets with SocketCluster (#2273)

Switch transaction expiration from seconds to chain height (#2461)

Implement Block.fromHex, Block.fromBytes and Block.fromData methods in @arkecosystem/crypto (#2377)

(#2377) Implement BlockFactory in @arkecosystem/crypto (#2429)

(#2429) Implement TransactionFactory in @arkecosystem/crypto (#2437)

(#2437) Purge invalid transactions after a milestone change (#2499)

Make transaction amount required (#2574)

Always use crypto identities to work with keys and addresses (#2443)

Enforce BigNumber for satoshi based values (#2391)

Move in-memory wallet logic to core-state (#2489)

Replace bignumify with Utils.BigNumber.make (#2416)

Replace Joi with AJV for internal validation (#2426)

Bind plugin options before registering the plugin (#2375)

Extend the node.js event emitter (#2440)

Move the wallet interfaces to @arkecosystem/core-state (#2515)

(#2515) Simplify the transaction pool by not using insertion order (#2495)

Drop no longer needed pagination from @arkecosystem/core-webhooks (#2424)

(#2424) Extract transaction signing and utils out of the model in @arkecosystem/crypto (#2514)

(#2514) Extract transaction verification out of the model in @arkecosystem/crypto (#2506)

(#2506) Make all Slots.* methods static in @arkecosystem/crypto (#2473)

methods static in (#2473) Move interfaces, types and models in @arkecosystem/crypto (#2379)

(#2379) Move mainnet exceptions to config in @arkecosystem/crypto (#2529)

(#2529) Remove extraneous Client class in @arkecosystem/crypto (#2417)

(#2417) Split the Crypto class into Hash and Transaction in @arkecosystem/crypto (#2444)

class into and in (#2444) Replaced @arkecosystem/core-json-rpc with @arkecosystem/core-exchange-json-rpc (Use @arkecosystem/core-exchange-json-rpc programmatically) (#2643)

with (Use programmatically) (#2643) Remove unused methods to get/set/reset height from Slots (#2467)

Added

Implement in @arkecosystem/core-state to manage the state of in-memory data (#2479)

to manage the state of in-memory data (#2479) Implement a blockchain replay command (#2526)

Save blocks in batches during sync (#2500)

Implement v2/node/fees endpoint (#2393)

Allow setting a vendor field for transactions created via @arkecosystem/core-json-rpc (#2425)

(#2425) Limit the number of accepted peers per subnet (#2507)

Parallel block download (#2433)

In-memory storage for last N blocks and transactions (#2492)

Require the user to choose a snapshot if the blocks flag is missing in @arkecosystem/core commands (#2522)

commands (#2522) Integrate end-to-end tests (#2468)

Initial Implementation of core-wallet-api (#2544)

(#2544) Accept block height to list block transactions in core-api (#2567)

(#2567) Functional test matchers for core-jest-matchers (#2562)

(#2562) Don’t trust headers and verify config and plugin connectivity of peers (#2559, #2553, #2552)

Proxy API calls to core-api until fully developed (#2558)

Add database configuration command to CLI (#2557, #2563)

Add command to generate network configuration CLI (#2582)

Initial implementation of core-explorer (#2604)

Fixed

Insert the genesis block as soon as the database is ready (#2376)

Use public API to auto-configure @arkecosystem/core-tester-cli (#2517)

(#2517) Parse only the last line of pm2 stdout to avoid parsing faulty input (#2484)

Delete bad rounds after unclean shutdown (#2581)

Divide blocks into smaller chunks for batch processing (#2586)

Remove forged transactions from pool before discarding block (#2555)

Various sync issues with devnet and mainnet (#2565)

Do not suspend peer for AppNotReady (#2590)

(#2590) Allow use of old and new block IDs via core-json-rpc (#2593)

(#2593) Assign calculated delegate ranks from temp wallets to prevent wrong ranks on boot (#2611)

Camelize block keys before bignum transformation for snapshots (#2615)

Deserialize transactions before they leave the pool to remove bad ones (#2622)

Require all properties in schema and handle 404 resources for core-webhooks (#2634)

(#2634) Check if transactions can still be applied before forging (#2635)

Off by one error in transaction confirmations via API (#2645)

Set the correct channel if core was directly installed with @next (#2646)

(#2646) Invalid orderBy causes Internal Server Error via API ([#2653)

via API ([#2653) Avoid trying to INSERT duplicates in rounds via core-snapshots (#2651)

(#2651) Handle failing optional plugins gracefully (#2657)

Correctly purge invalid transactions from disk on start (#2665)

Don’t append duplicate rounds rows to a snapshot (#2662)

Use temporary wallets for transaction validation (#2666)

Correctly display second signature if available via core-api (#2670)

(#2670) Missing block confirmations on v2 API endpoints (#2674)

Delay transaction purge on start until after StateBuilder finished (#2685)

Check claimed state of peer (#2686)

Ignore overheight blocks and keep forging (#2687)

Changed

Replace Joi with AJV for internal validation (#2426)

Remove height difference ban for peers (#2360)

Move mainnet exceptions to config in @arkecosystem/crypto (#2529)

(#2529) Invalidate blocks with expired transactions (#2528)

Transaction type agnostic wallet bootstrap to support AIP29 (#2539)

Return all schema errors in hapi-ajv (#2571)

Clean up SocketCluster shutdown and logging (#2560)

Remove timeout banning (#2597)

Use dayjs as it now has official UTC support (#2592)

Require a minimum of 0 as pubKeyHash (#2628)

Expire transactions that don’t have an expiration (#2672)

Removed

Remove peer caching (#2606)

Remove peer banning (#2612)

Remove coldstart period (#2619)

Remove whitelist access log (#2655)

Special Thanks

Thanks to all who have helped test the newest Core on the Development Network, prior to moving it to Public Network, by providing valuable feedback, reporting issues, and helping to resolve issues by opening pull-requests.

Especially delegate Alessio, who reported a few security vulnerabilities in testing phase of newest Core (prior to Core v2.4 going on Mainnet), making it even more secure and robust —a big thank you.

For those running your own servers with ARK Core, if you experience any errors or issues, please open a Github issue report for easier tracking and resolution: https://github.com/ArkEcosystem/core/issues.

Don’t forget we also have a development program for all who want to help improve ARK and earn some ARK: https://bounty.ark.io.

Hashes for 2.4.1 of Desktop Wallets (SHA256)