Summary: From 5229 to 1276 Characters (24.4%)

Updated 03-25-2020: No Longer Supported

Celody takes advantage of the data feature of IOTA by publishing music stream files into the Tangle. If you're not aware, every IOTA transaction contains a signatureMessageFragment field that can contain 2187 Trytes (See Transaction Structure). So you have this powerful storage object that you don't want to waste. You want to fit as much information into the field as possible. This is just good coding practice. Plus, it makes you a good citizen by decreasing the size that node operators have to locally store and process. In this blog post, I'm going to describe how Celody compresses data for the Tangle. If you have a better approach, please contribute on github.

Note: In practice what matters is minimizing the number of transactions, not just minimizing the total message length. However, by minimizing the message length you minimize the number of transactions. So this post just focuses on total message length.

Starting Stream File: 5229 characters

A basic Celody stream file looks like the following block of text. Don't worry about what the text means. It's an example of a medium size file. Just know that this is what we want to put into the Tangle. At 5229 characters, this definitely will not fit in 1 transaction. How can we make it smaller?

[{ "streamName": "Neon Strings", "artist": "Artist Merge", "createDate": "05-20-2019", "tokens": [], "tokenAddresses": [], "tempo": 85, "playSpeed": 0.5, "beatsPerBar": 4, "bars": 8, "baseUnit": 128, "backgroundLevel": 0.79, "dynamics": 0.2, "density": 0.7, "variety": 0.3, "spread": 0.3, "spice": 0.1, "instruments": { "0": [ { "instrument": "drum", "type": ["primary","7"], "color": "rgb(106,90,205)", "fileSource": ["https://celody.com/sounds/drum/kick/k9d.mp3"], "basePhrase": [0,0,0,0,0,0,0,0], "baseStart": [0.001,0.125,0.25,0.375,0.5,0.625,0.75,0.875], "volumeStart": [0.5,0.6,0.5,0.4,0.5,0.5,0.6,0.6], "fxFla": [0,0,0,0,0,0,0,0], "fxLow": [0,0,0,0,0,0,0,0], "fxAtt": [0.4,0.4,0.4,0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5], "fxPan": [0.65,0.65,0.65,0.65,0.65,0.65,0.65,0.65] } ], "1": [ { "instrument": "violin", "type": ["primary","8"], "color": "rgb(238,130,238)", "fileSource": ["https://celody.com/sounds/violin/g3.mp3"], "basePhrase": [0,0,0,0,0,0], "baseStart": [0.001,0.25,0.375,0.5,0.75,0.875], "volumeStart": [0.3,0.2,0.3,0.2,0.2,0.2], "fxFla": [0,0,0,0,0,0], "fxLow": [0.4,0.4,0.4,0.4,0.4,0.4], "fxAtt": [0.4,0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.5,0.5], "fxPan": [0.4,0.4,0.4,0.4,0.4,0.4] } ], "2": [ { "instrument": "cello", "type": ["primary","9"], "color": "rgb(75,0,130)", "fileSource": ["https://celody.com/sounds/cello/g3.mp3","https://celody.com/sounds/cello/d4.mp3","https://celody.com/sounds/synth/flangednoise/g0.mp3"], "basePhrase": [0,1,2,1,1,2,1,1,1], "baseStart": [0.25,0.375,0.375,0.4375,0.5,0.5,0.5625,0.625,0.8125], "volumeStart": [0.3,0.3,0.4,0.3,0.4,0.3,0.3,0.4,0.3], "fxFla": [0,0,0,0,0,0,0,0,0], "fxLow": [0,0,0,0,0,0,0,0,0], "fxAtt": [0.4,0.4,0.4,0.4,0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.4,0.4,0.5,0.5,0.5], "fxPan": [0.33,0.33,0.33,0.33,0.33,0.33,0.33,0.33,0.33] } ], "3": [ { "instrument": "bell", "type": ["primary","5"], "color": "rgb(250,255,0)", "fileSource": ["https://celody.com/sounds/bell/a1.mp3","https://celody.com/sounds/bell/a2.mp3","https://celody.com/sounds/bell/e3.mp3"], "basePhrase": [0,0,1,2,1], "baseStart": [0.125,0.25,0.625,0.75,0.875], "volumeStart": [0.5,0.6,0.5,0.5,0.6], "fxFla": [0,0,0,0,0], "fxLow": [0.4,0.4,0.4,0.4,0.4], "fxAtt": [0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.5], "fxPan": [0.4,0.4,0.4,0.4,0.4] } ], "4": [ { "instrument": "modularbass", "type": ["primary","6"], "color": "rgb(0,140,255)", "fileSource": ["https://celody.com/sounds/synth/modularbass/a1.mp3","https://celody.com/sounds/synth/modularbass/a3.mp3","https://celody.com/sounds/synth/modularbass/d2e2.mp3","https://celody.com/sounds/cello/a3.mp3"], "basePhrase": [0,1,3,2,1,0], "baseStart": [0.125,0.25,0.5,0.625,0.75,0.875], "volumeStart": [0.5,0.6,0.4,0.6,0.6,0.6], "fxFla": [0,0,0,0,0,0], "fxLow": [0,0,0,0,0,0], "fxAtt": [0.4,0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.5,0.5], "fxPan": [0.33,0.33,0.33,0.33,0.33,0.33] } ], "5": [ { "instrument": "bell", "type": ["secondary"], "color": "rgb(255,255,0)", "fileSource": ["https://celody.com/sounds/bell/a3.mp3","https://celody.com/sounds/bell/d3.mp3"], "basePhrase": [0,1,0,1,0], "baseStart": [0.001,0.25,0.5,0.625,0.875], "volumeStart": [0.5,0.6,0.5,0.5,0.6], "fxFla": [0.2,0.2,0.2,0.2,0.2], "fxLow": [0,0,0,0,0], "fxAtt": [0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.5], "fxPan": [0.4,0.4,0.4,0.4,0.4] } ], "6": [ { "instrument": "modularbass", "type": ["secondary"], "color": "rgb(0,140,255)", "fileSource": ["https://celody.com/sounds/synth/modularbass/a1.mp3","https://celody.com/sounds/synth/modularbass/a2.mp3","https://celody.com/sounds/synth/modularbass/d2e2.mp3","https://celody.com/sounds/cello/a3.mp3"], "basePhrase": [0,1,3,2,1,0], "baseStart": [0.125,0.25,0.5,0.625,0.75,0.875], "volumeStart": [0.5,0.6,0.4,0.6,0.6,0.6], "fxFla": [0,0,0,0,0,0], "fxLow": [0,0,0,0,0,0], "fxAtt": [0.4,0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.5,0.5], "fxPan": [0.33,0.33,0.33,0.33,0.33,0.33] } ], "7": [ { "instrument": "drum", "type": ["secondary"], "color": "rgb(106,90,205)", "fileSource": ["https://celody.com/sounds/drum/kick/k9d.mp3"], "basePhrase": [0,0,0,0,0,0,0,0], "baseStart": [0.001,0.125,0.25,0.375,0.5,0.625,0.75,0.875], "volumeStart": [0.5,0.6,0.5,0.4,0.5,0.5,0.6,0.6], "fxFla": [0,0,0,0,0,0,0,0], "fxLow": [0,0,0,0,0,0,0,0], "fxAtt": [0.4,0.4,0.4,0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5], "fxPan": [0.65,0.65,0.65,0.65,0.65,0.65,0.65,0.65] } ], "8": [ { "instrument": "violin", "type": ["secondary"], "color": "rgb(238,130,238)", "fileSource": ["https://celody.com/sounds/violin/c4.mp3","https://celody.com/sounds/synth/flangednoise/g0.mp3"], "basePhrase": [0,0,1,0,0,0], "baseStart": [0.001,0.25,0.4375,0.5,0.75,0.875], "volumeStart": [0.2,0.2,0.2,0.3,0.2,0.2], "fxFla": [0.2,0.2,0.2,0.2,0.2,0.2], "fxLow": [0,0,0,0,0,0], "fxAtt": [0.4,0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.5,0.5], "fxPan": [0.4,0.4,0.4,0.4,0.4,0.4] } ], "9": [ { "instrument": "cello", "type": ["secondary"], "color": "rgb(75,0,130)", "fileSource": ["https://celody.com/sounds/cello/g3.mp3","https://celody.com/sounds/cello/d4.mp3","https://celody.com/sounds/synth/flangednoise/g0.mp3"], "basePhrase": [0,1,2,1,1,2,1,1,1], "baseStart": [0.25,0.375,0.375,0.4375,0.5,0.5,0.5625,0.625,0.8125], "volumeStart": [0.3,0.3,0.2,0.3,0.4,0.3,0.3,0.4,0.3], "fxFla": [0,0,0,0,0,0,0,0,0], "fxLow": [0,0,0,0,0,0,0,0,0], "fxAtt": [0.4,0.4,0.4,0.4,0.4,0.4,0.4,0.4,0.4], "fxRel": [0.5,0.5,0.5,0.5,0.4,0.4,0.5,0.5,0.5], "fxPan": [0.33,0.33,0.33,0.33,0.33,0.33,0.33,0.33,0.33] } ] } }]

Basic Substitution: 3430 Characters

Because a Celody stream file has a known structure, we can just substitute out long words for a 2 character code. So long repeated strings like "https://celody.com/sounds/" can be shortened to just something like "=8". Celody first performs a bunch of regular expression (regex) substitutions to make this conversion.

jsonString = jsonString.replace(/=a/g, '"streamName":') jsonString = jsonString.replace(/=b/g, '"artist":'); jsonString = jsonString.replace(/=c/g, '"createDate":'); jsonString = jsonString.replace(/=d/g, '"forkedFromStreamId":'); jsonString = jsonString.replace(/=e/g, '"mergedFromStreamId1":'); jsonString = jsonString.replace(/=f/g, '"mergedFromStreamId2":'); ....

Pako Compression: 641 Characters

So we're down to 3430 from substitution alone. Let's get smaller. We can use the Pako Library (see Here) to perform a zlib compression on the substituted text. This converts the text into 641 symbols that looks like the following:

xíÍ0Ç_eÅ©\06...¦ôhîx¢eÇ3{%Ý¤úê»ËeóÜ+

IOTA and non-ASCII characters

At 641 characters in total, we are now quite small. However, we have a problem. IOTA does not support non-ASCII characters. So the symbols above will cause an error when using the iota.js library. To get around this limitation, we perform the next step.

LZ Base 64 Compression: 1276 Characters

We can use the LZ String library (see Here) to compress the Pako symbols into base64. This about doubles the size, but now we are good to use the iota.js library. We can then convert the LZ Base 64 compressed text (shown below) into IOTA Trytes. The trytes get uploaded to the Tangle.

B4HItwaQswcQ2QBgAgHGD6BTAowSoKkA6CAbAGcDFASQPAC+AAQG8BLAxAFMCBAIO80gA2AQgCawxdgG6A7ggAcADAANAdIAtAbYDTAKQDVAEYDCAMYB4APoB+AP4DzAEwBHAZgDyAGoD0ATQC6ALu0ALzEAKJ4AO0AaE4A2gDXAMoAzgA0AFsAxwEA5x4Aky4AGQI+xbQABAoWAHbctWYA9kgA/QD3ADMA7wAJAIQAegAGANgANgCX/RbUAIgA/AAd3OMAoWCVQgA9JsQAggA/AIAAOAYA/sNwAEQAewAQGgBGGmGHwABtSgCvAIwAPgAzwBkvDEADlAGACABPv6kfoiABAAGgAHQAX4AywArABPYAA3AB0ADfAAoAEMACOGjDMEAuWEOOAA0QAXRhYWrtAA35AAKwCADwABfRAA/pAAYgCLABOoSlajKALYyvzcSI3AAwHgQABYABfjADsAB8AIeos26hQGEHAAAaCmR2ytFNyFppAEKAFlk4D+gCcCwAd3YzQA4GkAQwAj8BaGB6MUAADPC0J9MAagAZikAJuVLQAXGGAOiAFcfAAVADqICtNJBYqQAE6ACjAZgABXxfIAwQBckC4nwAOTkADmHjOBHL0ciALEAcQRnQAXoMAGIAMl3tTsCYAWHJJnYEktepwtOjc1oAF8AKIApKeAI0cprMACOACJAAyAAiAAFIEAEEACUAPlbswTSwQAVVkAC9QIALVAchIoANJ6AAXQAFi4ADkKAAFQLCScwKMwhFQmEaxoE0aFzAAZQoaH7gA3qMkG4gAqCIHLAKwPzAJimFJBxgxwJEOBAgAmE4ABOtCTrBL4AOCEVgNIALS+gAwxA4ycHy1ZrsUNx1gAYDcGAAA5Qh2b5+GBIJLGpOlbioQEADvjPucacBcAACAD1AAp9AAKsGAA5loDhmgUABr3DYgAJA2IJ1gAoAAyE0wxqQAFdiSQAJmGho0FqlFxGHHMACw2zFGRAAHAAShpoaeeAAM1dpO/oAKy4oMeEAJK7hx/puMadYZXWvWnl4OAAHWUD8DjKfFeDMB2dZoM6ABQGWCig0QAFdsl2ACnfwjvixogbiky6qiABqYEUgArQASgUpATQApSub5JAAlCOgzbDoCqwTFpBwDF0SYmaeAATjhVsnc+YGGSGUAOuHGEaDFbiT6nm+cYrsEUDhowCx8gCACRgNtQA8UlfLAIKHJoQAbsEHEcoMHEAC9soVxHhsAABFGAoC4yK8W+AQwNBT53ZII5aBgQI6cpgW5scQA

Final Length: Recap

We have compressed the data payload from 5229 charactes to 1276. 1276 is about 24.4% of the starting payload. Now when we extract the compressed text from the Tangle, we just reverse the process to inflate back to the original. We then play the stream file using the Celody Core module.