RawMessage is a type in Golang’s JSON package that represents a raw encoded JSON object — a slice of bytes. You can use it to delay JSON decoding.

When consuming from Kafka, throughput is a major key to success. — DJ Khaled

Throughput is key and decoding JSON can be a killer, especially when working with complex objects where you don’t know the types of every field.

var m map[string]interface{}

json.Unmarshal(bytes, &m) // slowwwwwwwww

We often need only 1 or 2 fields from the JSON, but with the code above you pay the cost of the decoding the whole object.

RawMessage saves your butt.

Let’s illustrate an example where we have a Kafka consumer that is consuming JSON objects and writing them to ElasticSearch.

We need to decode the JSON to know the ID and type of the object. The JSON is encoded and produced to Kafka like so:

// JSON msg.

{

“id”: “numero uno”,

“type”: “transaction”,

// … a whole bunch of dynamic fields

“amount”: “1000”,

“currency”: “usd”,

// … etc.

}

The ID and type are all that we need, but we’re forced to decode into a map[string]interface{} since we don’t know what the other fields are.

var m map[string]interface{}

json.Unmarshal(bytes, &m)

es.Index(index, m[“type”].(string), m[“id”].(string), "", "", nil, bytes, false)

But if we produce JSON like so:

{

“id”: “numero uno”,

“type”: “transaction”,

// data is the JSON msg from above

“data”: {

“id”: “numero uno”,

“type”: “transaction”,

// … a whole bunch of dynamic fields

“amount”: “1000”,

“currency”: “usd”,

// … etc.

}

}

And we make a Message type to decode into:

type Message struct {

ID string `json:”id”`

Type string `json:”type”`

Data json.RawMessage `json:”data”`

}



var m Message

json.Unmarshal(bytes, &m)

es.Index(index, m.Type, m.ID, "", "", nil, m.Data, false)

Then we’re only decoding the data that we need.

Throughput is high and we fly.