Happy New Year — Development Update — By Shift Team in Newsletter on January 2nd, 2019:

Completed Tasks:

- Add endpoint for calcLockedBytes()

An endpoint is the API address used by frontend elements (in our case the client wallet) to request certain information from the backend. Adding this endpoint, used by the client software to establish the number of bytes locked by an account, is essential in the creation and verification of valid UNLOCK, PIN and UNPIN transactions at the client level.

- Create a verifyPin endpoint

This function is used by Phoenix to confirm that a PIN transaction has been successfully recorded on the blockchain. It requires two parameters, the asset’s size in bytes (asset_size) and the asset’s hash (asset_hash), the alphanumeric string used to point to the file in its entirety.

- Write cluster statistics to mem_tables by block interval

In order to process a LOCK transaction, the network needs to have the information required to complete the pricing formula embedded within that transaction. This data must therefore be gathered from the cluster at regular intervals. The team has set up the system so that this occurs every ten blocks (nine times per delegate round), at which time the blockchain nodes will call the /TotalStorage endpoint of Phoenix. The value for total_storage (in bytes) will then be stored in a memory table (mem_cluster_stats) present in all blockchain nodes so that the data is always up-to-date.

- Take the mode average of cluster statistics

In order to avoid storing raw data on the blockchain every block (which could be liable to inaccuracies), the mode average of the last nine snapshots for total_storage from the mem_cluster_stats table will be taken. These nine values for total_storage will be sorted from small to large, and, subsequently, the fifth (middlemost) number of that queue will be taken. Use of the mode average is beneficial as it negates the necessity of recording outliers within the cluster statistics entries.

Remaining Tasks:

- Add cluster snapshot columns to the blocks table and persist them every time a block is forged

Every time a block is forged, the mode average of the last nine snapshots for total_storage is recorded in a new column of the blocks table. These values will eventually be processed during LOCK verification and used as the Y-value (the total supply of the platform’s storage capacity expressed in bytes) of the pricing formula, in order to determine the amount of SHIFT tokens required to acquire any given amount of storage capacity. This block processing is thus essential in allowing the ‘price’ to be determined dynamically.

- Establish an acceptable tolerance mechanism for cluster statistics during block forging

The team intend establishing a mechanism where, when the 100 witnesses (forging delegates) verify the block broadcast by the forger, they ensure that the mode averaged value for total_storage is in a specific degree of tolerance with their own value (10% configurable). If this is not so, they will automatically reject the block that contains a value for total_storage that doesn’t fall within this acceptable level. This is in order to establish consensus between forging delegates and to prevent the chain from forking.

Forging delegates will obtain the cluster statistics from different storage peers (the process is randomized and thus decentralized). This means that even though the raw data is mode averaged, it could still happen that the mode averaged value for total_storage in the same block differs among the forging delegates. Once the forger has broadcast the block to its 100 witnesses (forging delegates), an unnecessary fork is prevented by introducing the tolerance/margin for an acceptable level wherein the values for total_storage may differ.

- Add an asset_size verification process for Phoenix

This is the only remaining task of the blockchain integration that needs to be included specifically for Phoenix. The asset_size and asset_hash of a file will be included as data entries within a PIN transaction. When a user uploads a file, a hash will be returned. The hash and the size of the file will be included within the PIN transaction, that will be sent to the forging delegates for verification of the signature of the transaction. After verification, the PIN transaction will be stored in the blockchain and the file, as well as the PIN transaction, will be distributed to cluster participants. Storage peers then verify the asset_size using the /verifyPin endpoint, prior to actually executing the PIN. This verification process allows the storage peers to use the data contained in the PIN transaction, in addition to the properties of the actual file, to read and compare the file size directly. Using the /verifyPin call they will be able to do this by looking at the blockchain for a matching asset_hash and asset_size. In the case that there is not a match, the storage peers will refuse the PIN.

The inclusion of an asset_size verification process means that, if a user were to manipulate a PIN transaction in an attempt to use less of their locked storage capacity, by, for example, uploading a file (asset_hash A) of 100 MB (asset_size A) but changing the asset_size to 1 MB (asset_size B), while the manipulated transaction would be accepted by the blockchain (assuming the user possesses enough storage locked for the 3 MB necessary to meet the x3 replication factor and the SHIFT necessary to pay the small fee), when the PIN transaction is sent to the storage peers along with the actual file, the mismatch would become evident. The use of the /verifyPin endpoint by the storage peers would allow them to look at the blockchain for the matching pair of asset_hash A with an asset_size A of 100 MB, but because the manipulator’s actions mean that this pair does not exist at the blockchain level (there would only be the pair of asset_hash A and an asset_size B of 1 MB), they will ignore the PIN request.

From Official Ryver chat:

Ralf S (Shift President and Lead Developer) on December 20th 2018:

“I’m sorry if the team seems a bit quiet lately. Obviously we are working full time on the storage core.

To be specific working on the writing of network stats to blocks. With consensus, which is why it’s not an easy task.

A new progress report will explain what we’re doing in more detail.”

To be specific working on the writing of network stats to blocks. With consensus, which is why it’s not an easy task. A new progress report will explain what we’re doing in more detail.” “Still making progress on the backend every day. But I admit it’s an awful lot of work. We’ve received various request in DM to show something to the public. But it’s mainly code work we’re doing now, no eye candy.

But I guess I can share a bit of what we’re working on right now. I will paste some screens in our Discord channel.

Shift (SHIFT) event: December 31, 2018: