gRPC is a powerful framework providing fast efficient communication between services. What gRPC is and how to use it is beyond the scope of this article. If you are interested in getting started with gRPC I suggest you read the gRPC docs located here.

A central component of gRPC is the code generation for client/server applications. This code is generated by the protoc compiler from the .proto files. While the generated code follows several good practices such as immutable objects, and builder patterns there are other decisions that you may or may not like about the generated code.

On the surface gRPC seemed like a perfect solution for us. But we ran into several problems, not with gRPC or Protobuf directly but with choices that were made specifically within the generated code.

We were trying to migrate an existing Java application to micro-services using gRPC. Unfortunately, our existing code made judicious use of null in our domain objects. (Whether or not this was a good idea is irrelevant. It’s what we had and we couldn’t afford the time to refactor it.) On the surface, this doesn’t seem like a problem until we started working with the code that is generated by gRPC.

Here is a quick example. Lets say I have an existing Java domain object Employee:

@Value

public class Employee {

String name;

BigDecimal salary;

BigDecimal bonus;

}

In our existing model if an employee didn’t have a bonus we would leave the field null.

A proto definition for this domain object could look like:

import "google/type/money.proto";



message Employee {

string name = 1;

google.type.Money salary = 2;

google.type.Money bonus = 3;

}

We now have a protobuf model for our domain object. Let’s start using it. A pattern we adopted, as our transition phase, was to build proxy interfaces and transformers between the original domain objects and our new proto objects. This allowed us to transition exiting code to using protobuf with little to no refactoring.

Example:

public class EmployeeTransformer {



public Employee toEmployee(EmployeeProto employeeProto) {

return Employee.newBuilder()

.name(employeeProto.getName())

.salary(toBigDecimal(employeeProto.getSalary()))

.bonus(toBigDecimal(employeeProto.getBonus()))

.build();

}



public EmployeeProto toEmployeeProto(Employee employee) {

return EmployeeProto.newBuilder()

.setSalary(toMoney(employee.getSalary()))

.setBonus(toMoney(employee.getBonus()))

.build();

} ...

}

Looks ok, right?

As good developers we should write a few test cases:

def "test toEmployeeProro"(EmployeeProto input, Employee expected) {

given:

EmployeeTransformer transformer = new EmployeeTransformer()



expect:

transformer.toEmployeeProto(expected) == input



where:

input | expected

createEmployeeProto("bob", 45, 100) | createEmployee("bob", 45, 100)

createEmployeeProto("bob", 45, 0) | createEmployee("bob", 45, 0)

createEmployeeProto("bob", 45, null) | createEmployee("bob", 45, null)

}

Let run this spec, all is well right? Wrong!

Condition failed with Exception: transformer.toEmployeeProto(expected) == input

| | |

| | Employee(name=bob, salary=45, bonus=null)

| java.lang.NullPointerException

com.efenglu.jprotoc.example.EmployeeTransformer@7c83dc97

Hmm, that’s weird.

Let’s see, when the bonus was null we got a NPE because we called set with a null.

This is because:

Protobuf does not allow nulls

Well it does and it doesn’t. Protobuf does not allow you to set a field to null. It will throw an NPE on calling set with a null.

Well, what about our other spec:

def "test toEmployee"(EmployeeProto input, Employee expected) {

given:

EmployeeTransformer transformer = new EmployeeTransformer()



expect:

transformer.toEmployee(input) == expected



where:

input | expected

createEmployeeProto("bob", 45, 100) | createEmployee("bob", 45, 100)

createEmployeeProto("bob", 45, 0) | createEmployee("bob", 45, 0)

createEmployeeProto("bob", 45, null) | createEmployee("bob", 45, null)

}

How did this one fair:

Condition not satisfied: transformer.toEmployee(input) == expected

| | | | |

| | | | Employee(name=bob, salary=45, bonus=null)

| | | false

| | name: "bob"

| | salary {

| | currency_code: "USD"

| | units: 45

| | }

| Employee(name=bob, salary=45, bonus=0)

com.efenglu.jprotoc.example.EmployeeTransformer@198b6731

We were expecting a null value but got back 0. Why?

Protobuf does not allow nulls

Protobuf will return a default object when a field has NOT been set. Ergo, the value is 0 instead of null.

Again this article is not meant to discuss the merits of the use of nulls/non nulls etc. Lets move beyond and figure out how to deal with the cards we’ve been delt.

We have a domain model that is expecting nulls We have a transport framework that does not allow nulls

On the surface this seems to be an intractable problem. But not entirely. While protobuf doesn’t directly support the concept on null it does provide a ‘has’ concept. The has boolean is meant to tell you if a value has been set. We can use this to transport our concept of null.

Let’s modify our code to use has and avoid the NPE’s.

public class EmployeeTransformerHas {



public Employee toEmployee(EmployeeProto employeeProto) {

final Employee.EmployeeBuilder builder = Employee.newBuilder()

.name(employeeProto.getName());

if (employeeProto.hasSalary()) {

builder.salary(toBigDecimal(employeeProto.getSalary()));

}

if (employeeProto.hasBonus()) {

builder.bonus(toBigDecimal(employeeProto.getBonus()));

}

return builder.build();

}



public EmployeeProto toEmployeeProto(Employee employee) {

final EmployeeProto.Builder builder = EmployeeProto.newBuilder()

.setName(employee.getName());

if (employee.getSalary() != null) {

builder.setSalary(toMoney(employee.getSalary()));

}

if (employee.getBonus() != null) {

builder.setBonus(toMoney(employee.getBonus()));

}

return builder.build();

}



...

}

While our tests now pass, the transformer code is quite ugly.

Also, while the transformer provides help for existing domain objects we would really prefer if our clients spoke protobuf natively. Once our programmers start utilizing the code they will have to write all of these checks by hand. It would be nice if we could somehow support null on set and null on get.

Enter the protoc plugin.

Protoc Plugin Extension Point

Protoc has an extension point mechanism to allow you to insert code along side generated code. This will allow us to add helper/utility methods that have the behavior we want.

First some bad news.

Without directly editing the original C code for the protoc compiler you can’t change any existing generated code. So you can’t directly change the set method to accept null, or the get method to return null.

What you can do is add new methods to the generated classes. Like adding a setOrClear method and a nullGet method.

For our project we decided to add a setOrClear and an optionalGet method.

setOrClear

Provides a setOrClear method to the generated builder which accepts a null value.

If the value provided is null the builder will call clear on the associated field.

If the value provided is non-null the builder will call set on the associated field with the provided value. This avoids the NPE that is normally thrown when set is called with a null value.

Example:

public Builder setOrClearBonus(com.google.type.Money value) {

if (value != null) {

return setBonus(value);

} else {

clearBonus();

}

return this;

}

optionalGet

Adds a method optional{Field} that will wrap all non primitivefields of a message type in a java.util.Optional object.

When has{Field} returns false the optional is empty

When has{Field} returns true the optional contains the value of calling get{Field}.

Example:

public java.util.Optional<com.google.type.Money> optionalBonus() {

if (hasBonus()) {

return java.util.Optional.of(getBonus());

} else {

return java.util.Optional.empty();

}

}

With the combination of setOrClear and optional we felt this was the best of both worlds. We can set nulls and still provide non null return types.

But…

Some other bad news. This approach requires developers to call the correct methods. set still throws NPE, and get still returns a default value. This isn’t ideal but could also be useful for those used to the expected Protobuf behavior.

Also these new methods ONLY apply to proto’s that you have compiled. So if you’re reusing someone elses prepackaged proto definitions you won’t get these new methods.

Creating the Plugin

How do we write this magical plugin. You could write it in C and use some of the existing protoc library code. However, we are a Java shop and maintaining C code wasn’t really an option.

Thankfully, protoc doesn’t actually care what your plugin is written in. All protoc does is invoke the plugin executable and pass in a protobuf string describing the protobuf we want to generate.

Very meta, also very cool.

Salesforce helpfully provides a maven plugin to assist our effort. This allows us to write our plugin in Java as a regular maven artifact.

A link to the full codebase is found at the end of the article but lets focus on a couple pieces first.

Step 1: Setup maven module

Nothing really special here. A regular maven jar module is all we need. Just make sure we depend on Salesforce jprotoc

<dependency>

<groupId>com.salesforce.servicelibs</groupId>

<artifactId>jprotoc</artifactId>

<version>0.9.0</version>

</dependency>

Step 2: Create your extension point main class

As we mentioned earlier protoc treats all extensions as executable programs we therefore need to create a Java class with a standard main entry point:

public class JProtoc extends com.salesforce.jprotoc.Generator {



public static void main(String[] args) {

com.salesforce.jprotoc.ProtocPlugin.generate(new JProtoc());

}

...

}

Simple enough so far.

Next override the super generate method:

generate

We are returning a stream of File objects. These represent files that are to be generated by protoc

We are filtering it by checking if the proto to generate is in the list of files to generate. You will be passed all the proto files, even imported ones, so be sure to filter these out. (Unless you want to generate code for them)

We then call our other method to generate our code in handleProtoFile

handleProtoFile

Nothing too fancy here. Just be aware that the java package could come from two different locations. Either implicitly equal to the proto package or explicitly with the proto option.

handleMessageType

Now we’re getting into the real meat.

First build the filename of the class we are going to be adding code to.

This isn’t really correct here since I’m assuming you are using the multiple file option. Without this all proto classes and inner classes and should be delt with differently.

We then call other sub-methods to build the strings for the code we are inserting. It’s nothing really special, just inserting values into a mustache template. Checkout the full code for details.

The real kicker here is how we generate the file object.

Notice how we set the fileName, the content we want to insert and the insertion point.

The insertion point is a special string in a comment that shows up in the protoc generated code. There are several different insertion points. The ones we are using are the builder_scope and the class_scope insertion points.

Example:

// @@protoc_insertion_point(class_scope:com.efenglu.protobuf.EmployeeProto)

builder_scope

Inserts code at the end of the inner builder class. This is where we want to add our setOrClear methods.

class_scope

Inserts code at the end of the java class for the message type. This is where we want to add our optionalGet methods.

Other insertion points exist. Open the generated proto files and search for protoc_insertion_point to see what’s available.

Step 3: Use the Plugin

Using the new plugin is quite easy.

Configure the maven project to build the proto files using the Maven Proto Buffers Plugin:

Here is how we hook in our new plugin:

os-maven-plugin: Helps us in downloading the correct protoc binary for our build system

protobuf-maven-plugin: Hooks in the protoc binary and our custom protoc plugin

Notice the protocPlugin element. Here we’ve hooked in our protoc plugin. We specify the maven artifact for our plugin and the executable class within that artifact. The protobuf-maven-plugin has excellent docs exampling the usage in greater detail.

With this we have our new custom methods.

With great power…

You can basically add whatever you want into your generated proto code using this approach. But there are severals things to consider before doing so.

This is only added to your code

This is not what someone would expect

Could you do it some other way?

I hope this has been helpful. There are several other challanges to using the protoc plugin framework, such as adding comments from the proto files, dealing with oneof, maps and repeated types, fields name collisions. But this should get you started.

For the complete source code referenced in this article check the Github repo: