Avro Converter and Spring MVC integration for efficient serialization of Apache Avro binaries between different Message-Driven components. Converting Avro object from/to JSON and XML documents.

Prerequisite

Recently a newest version of Avro Converter project was released. It provides few extremely valuable features for working with Apache Avro, such as:

Converting Avro object from and to JSON and XML documents

Providing out of box integration with Spring Boot/Spring MVC

Helping to reuse Apache Avro objects from existing Message-Driven, Big Data and whatever systems for defining RESTful interface instead of DTOs.

Apache Avro is versatile and quite powerful data serializing system, it describes an object using language neutral schema and unlike other data formats or serializing systems it requires to have a reference to the object schema for every read and write operation. This is a real power of Apache Avro and it helps to serialize and work with object binaries efficiently.

As an addition, Apache Avro in collaboration with Confluent Schema Registry supporting schema evolution. This business case is so well implemented, so, I’d rate it as “#1 Feature”.

Having this in mind it is easy to define use-cases for Apache Avro:

Decoupling application components by introducing Message-Driven architecture with Apache Kafka (we will focus on this scenario in details)

Enables language interoperability

Supported within Apache Software Foundation projects (Hadoop/MapReduce, Hive/AvroSerDe, Kafka/Confluent Schema Registry etc)

Designing Message-Driven Architecture

Further in this article we’re going to make an overview of Avro Converter and Apache Avro while designing a Message-Driven system with Spring Boot/Spring MVC as REST API which operates JSON and XML documents received from client applications like: AngularJS, REST Client or any third-party service and Apache Kafka as Pub/Sub distributed streaming platform.

At this point you might’ve thought already: “Why do we need to have a deal with non-Avro documents?” — the answer on this is quite pragmatic:

In those cases when we are able to sacrifice Avro’s compression and schema evolution features for readability and compatibility with client services — we have to do it.

Finally, after some analysis we can make a design overview of our toy-service with Message-Driven architecture:

Message-Driven Architecture of “Avro Converter Demo” Project

Source code of a sample-project, detailed instruction of installation and other related information you can find by this link

Defining Avro Schema. The first step which we are going to do in our toy example is a design of an object which is going to be transferred between system components.

Avro Converter Demo. Avro Schema

Fortunately, there is Maven plugin avro-maven-plugin responsible for POJO generation based on Avro Schema files, it will automatically create EmailData class on the project build stage. We will need it to complete our next steps.

Describing REST Endpoints with Serialization of Apache Avro Object via Spring MVC. For this step the target looks extremely simple: we need to define REST interface using Spring Boot/Spring MVC and set in use newly generated Apache Avro EmailData object for @ReqestBody and return type as well.

Avro Converter Demo. Bad REST Endpoint

Facing an Issue. Finally, after running this basic application and triggering a PUT request with a CURL command we’re able to define a problem:

[org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON: Not a map: {...}

(through reference chain: com.vladkrava.avroconverterdemo.domain.EmailData["schema"]->org.apache.avro.Schema$RecordSchema["valueType"])]

In few simple words standard MappingJackson2HttpMessageConverter says: “Sorry. I can’t serrialize your this object just because I don't know anything about it”.

That’s where Avro Converter comes into play.

Avro Converter

We figured out already that default for Spring Boot/Spring MVC message converter can’t handle Apache Avro objects, that’s why we will introduce Avro Converter to this project.

In order to use Avro Converter Java library on your project you will need corresponding Maven dependency:

<dependency>

<groupId>com.vladkrava</groupId>

<artifactId>avro-converter</artifactId>

<version>1.0.1</version>

</dependency>

Manual Operations. To perform manual convertions we have to get in use one of the following classes responsible for convertion of XML and JSON documents to Apache Avro objects and vice-versa: XmlToAvroConverter , JsonToAvroConverter , AvroToXmlConverter , AvroToJsonConverter :

Avro Converter. Manual Operations

Declare HTTP Message Converters (Spring MVC only). Integration with Spring MVC looks even easier, there are only 2 classes which we have to declare as message converters, so they could handle application/avro-json and application/avro-xml content types:

Avro Converter. Defining Message Converters

In the last step of our journey we will specify to already designed REST endpoints the content type which they are going to handle. Avro Converter has predefined variables for this purpose, however you can use your own:

Avro Converter. Defining Message Converter for Specific REST Endpoint

Well, now you’re set to design of your own much more serious distributed Message-Driven systems and business scenarios with the help of Avro Converter, Apache Kafka, Apache Avro and Spring Boot.

Take it easy!