Figure 6. Ordering Service Process

Myth 7

Channels are free.

Continuing from myth 6, in many real-world scenarios, channels can multiply quadratically.

When creating the Inter-Bank Payment System (IBPS) for Singapore’s central bank, the Monetary Authority of Singapore, our network needed to support a little over 200 banks privately communicating with each other through bi-lateral transfer channels to ensure no information would appear in other organization’s ledgers. This meant that each bank had approximately 200 channels to "talk" with other banks, and each of those banks similarly had ~200 connections.

In graph theory, this is called a complete graph; in networking, it is a fully connected mesh topology. The number of connections is proportional to n2, and thus even in little Singapore, this results in over 20,000 channels.

In the United States, with nearly 7,000 banks, you would need over 20 million channels! The ordering service wasn’t designed to handle so many channels and becomes quite slow. For example, in a test network with 10

banks, you would have 45 bi-lateral channels. Creating these 45 channels takes 90 seconds—2 seconds per channel. For 20k channels, a rough time estimate would be 11 hours. For the USA with 20M channels, a rough estimate would be 1.5 years!

If you want to ensure complete channel isolation, you would need to operate 20k Kafka clusters! This just isn’t practical, and a better solution is required—private collections.

Myth 8

Private collection solves the channel explosion problem.

Given the shortcomings of channels, it should come as no surprise that the Fabric team started searching for a better solution shortly after the release of 1.0. The idea we settled on was originally called side channels, but became productized as private collections. This enables organizations to establish private databases that a subset of organizations are allowed to see. The transaction data from these is never sent through an external ordering service. So, although the ordering service is no longer a trust or performance problem, instead a peer-to-peer (P2P) network must be established between all organizations’ peers. These P2P private collections are defined in a JSON file, specifying the organization(s) that can see the contents.

This file will need to specify 20k pair-wise combinations of banks in the Singapore IBPS example. This change places more burden on the network administrators to appropriately define the routes and private networks, perhaps VPNs, to create a secure foundation for communication. So, although burdensome, this all sounds good.

Figure 7, below, shows an example of this JSON document. As new banks are added, removed, or merge, this document would need to be updated to match the new configuration.

Figure 7. Fabric Private Collection Policy Definition

Myth 9

Data privacy is easy.

Continuing with the theme of data privacy, even with channels or private collections, the act of slicing up your overall asset pool into hundreds of bi-lateral partitions introduces its own challenges.

Returning to the IBPS example, if you divide your bank’s overall liquidity one-hundred fold to avoid other network participants having a total view of your liquidity, it is now more difficult for you to access it all at any one time. Simple acts like computing your total liquidity require cross-channel or collection queries. When performing a transfer, it is now much more likely that insufficient liquidity will be available in a given channel or collection.

To resolve this, money must be transferred from another private collection in a way that ensures no funds are created/destroyed in the process while preserving privacy. And, unfortunately, there are other caveats. To deal with ordering and read-write set MVCC consistency, the public transaction used to record the private transaction occurring must include the read-set keys. Therefore, if you are in a situation where even revealing that there has been activity on a particular key is unacceptable, you will need to salt+hash the key to avoid revealing this to others. The salt used for a particular key will either need to be stored off chain or as a value in another row that enables lookups. Only by blinding the key can we prevent other organizations from becoming aware of activity. To deal with all of this requires that you hire smart programmers familiar with cryptography and distributed systems, specifically multi-party algorithms or homomorphic encryption, to develop custom solutions.

Myth 10

Fabric is easy to administer.

Fabric, in contrast with other blockchain systems, has many components that an administrator must become familiar with to operate—peer, CouchDB, ordering service (Kafka, zookeeper, raft), memberships services provider, docker. It also requires specialized routing and firewall rules to establish secure communication links between organization peers, MSPs, and the ordering service. Additional administrators will need to be involved with management of a multi-organization certificate authority (CA).

To resolve this, IT network engineers must work as part of a consortium to determine the appropriate way to expose their peers to other organizations to receive transaction endorsement proposal/simulation requests while minimizing an attacker's ability to gain access to sensitive information stored in the simulating peer’s database.

One way to simplify this effort is to co-locate all organization’s peers within a single data center or provider, such as AWS, Azure, or IBM. Of course, being in a single availability zone (AZ) is a risk unto itself. When dealing with mission-critical systems accessing core records, you need a team of highly-skilled network engineers and systems administrators onboard to plan appropriately.

Figure 8. Typical Fabric Components

Myth 11

Clusters don’t need backups.

Fires. Earthquakes. Tornados. Lightning. Floods. Hurricanes. Disasters happen. If all Fabric peers are in the same data center/locale and suffer a direct hit, only a remote backup will be sufficient for recovery.

Crypto ransomware is ever present and a blockchain system is a prime target. If all online volumes are encrypted, having an offline backup prior to the attack provides the option to reject paying the attacker’s ransom demands without complete data loss.

Even in less disastrous scenarios, recovery by replaying the block log from the genesis block can be painfully slow. Private collections complicate this further, as it is insufficient to perform a backup at a single peer. Each organization may have a unique set of private data not fully available at any other peer. For example, if private collections exist A+B, B+C, and A+C, then each organization should be performing independent backups, although not necessarily at the same time.

Sadly, the Fabric team does not agree that backups are a necessity.

There are two strategies for backup: offline and online. If you can take a peer offline temporarily, this is the easiest option. If you are using private collections, remember that all organizations with unique access to a private collection will need to perform a backup.

To perform an online backup, we must have a consistent version of all blocks, indexes, and logs. A no-code solution is to use LVM and volume snapshots. A better solution is to pause Fabric processing commits and to use a native backup tool for the database to ensure a consistent restore point.

Further Details: http://www.bchainledger.com/2019/02/hyperledger-fabric-ledger-backupand.html

Myth 12

Fabric needs few network and hardware resources to operate.

All blockchain systems are, in essence, unbounded, append-only logical logs. Storing all data from the genesis block onward and maintaining it has a non-trivial cost.

One way to mitigate this cost in a production deployment is to use hierarchical storage to reduce the cost of storing old blocks. To increase the value of storing old blocks, the history API should be enabled so that users can access records that aid in fraud detection, KYC, AML, and similar efforts.

The ordering service is a significant source of network congestion. All transactions must flow into it one time and are replicated out to "N" organizations. This problem grows quadratically with the number of organizations. If N is the number of organizations and T is the number of transactions per second per organization, the total transactions per second being handled by the ordering service is given by the formula:

This problem is somewhat mitigated by organizations using gossip to replicate between peers within an organization.

The long term solution to this problem is to have a gossip-based transaction-distribution mechanism to eliminate the need for the ordering service to directly talk with all organizations.

Conclusion & Final Thoughts

Many of the issues discussed herein have analogs in other blockchain systems. As enterprises look to blockchain, they need to think as deeply about organizational issues as they do technical ones. Adopting blockchain is not simply about running a parallel database mirroring existing OLTP systems—it is about developing new models for business made possible through enhanced trust and information sharing. Technologically, blockchain makes this

possible through a common view of information, non-repudiation of transactions, and access

to immutable historical information.

Leaders are encouraged to consider issues such as:

Will users transact as themselves, or will transactions be signed by the organization they are acting on behalf of? Will you organically grow, hire, or contract expertise in Fabric to deploy smart contracts that are performant and secure? How are you going to design effective smart contracts that create true counter-parties that have a vested interest in holding others responsible? Looking to the future of the business, how should systems be architected to deal with the possibility of systems integration due to mergers and divestment? When acting as part of a consortium, what is the governance model that will be used for smart contract development? How will core systems be exposed to other organizations as required for peer endorsement?

In answering these questions, leaders should recognize that experience matters, both with blockchain and with the domain they operate in. The journey to blockchain adoption is a marathon, and they should start to build in-house experience today, especially for DevOps and networking.

Adopting Fabric or any blockchain system as an integral part of a business requires thoughtful analysis to maximize the ROI. Existing efforts at digitalization and a concrete understanding of existing business processes are critical to success.

When Fabric 1.0 was being designed, it was not clear what the emerging business model requirements would be. The hope was to be sufficiently flexible to provide a good starting platform given broad configurability and modularity. As the most enterprise-ready of the first wave of blockchain products, there are few situations for which Fabric cannot be adapted. Successful adoption requires significant expertise in network security and routing, cryptography, game theory, databases, containerization, and distributed computation. Just as Rome was not built in a day, nor will the future of business in blockchain.

How to eat an elephant? Grab an ear and get chewing!

References