Archie and the gRPC API

Archie, one of the system’s backend services written in Go, is responsible for keeping track (in-memory) of the water sessions, henceforth known as a Splash. But before I get there, I want to describe my project’s API service, which is based on gRPC.

To refresh your memory, I’ll quickly say that gRPC is an RPC (remote procedure call) framework. To put it simply, it is a communication protocol for executing a procedure (in this case, functions) from one system to another. What I really like about gRPC is that it’s defined using Protocol Buffers (also known as Protobuf), a “language-neutral, platform-neutral, extensible mechanism for serializing structured data.” What I mean here is that once the service and objects are defined in Protobuf, you can use one of the many code generators to convert it to one of the many languages it supports.

The whole thing is a bit tricky to grasp, but hopefully, the code will clarify things:

syntax = "proto3";



package api;



import "google/protobuf/timestamp.proto";



service DrinkWater {

rpc LogSplash(Splash) returns (LogSplashResponse);

rpc WaterConsumed(Since) returns (WaterConsumedSince);

}



message LogSplashResponse {

bool ok = 1;

string error = 2;

}



message Splash {

google.protobuf.Timestamp ts = 1;

int32 amount = 2;

}



message Since {

google.protobuf.Timestamp ts = 1;

}



message WaterConsumedSince {

int32 amount = 1;

}

This is the project’s Protobuf file. In the first three lines, I’m merely setting the syntax, the package name, and importing an external time-related library I like to use. Then, I’m defining the gRPC service, named DrinkWater . The service, as you can see there, is composed of two methods: LogSplash and WaterConsumed . The former, LogSplash , which we saw before, takes as a parameter a Splash , and returns a LogSplashResponse , while the latter, WaterConsumed , takes a Since object, and returns a WaterConsumedSince (I’ll get the details soon). But what are all these water and splashy-splashy thingies?

Right after the service, you’ll find the definition of these messages. The first of them, Splash, contains two fields: a Timestamp object (that thing I imported) and an integer (water in ml.). Then, there’s LogSplashResponse , which consists of a boolean that will be false if an error occurred during LogSplash , and a string explaining the error (if there’s one). Following this, is the Since message, which encapsulates a timestamp. Then, lastly, is WaterConsumedSince , made of a single integer.

While these five structures define my complete service, they aren’t actual source code I could use. So, my next step was to generate Golang and Python code.

In my experiment and my opinion, generating the Go code is much easier than the Python. With only one command $ go generate protoc -I=. — go_out=plugins=grpc:. endpoint.proto (executed from the directory where the Proto file is) you’ll be able to generate the code. Nonetheless, my experience with Python wasn’t the best due to some issues related to the path where the generated code resides. For those of you who are curious about how I created the Python code, this is the command I ran (from the Water Retriever’s root directory)

$ python3 -m grpc_tools.protoc -I.:${PROJ}api/v1/ \

— python_out=app/api/v1/ \

— grpc_python_out=app/api/v1/ ${PROJ}api/v1/endpoint.proto

But the story doesn’t end there. If you run Water Retriever as it is, it won’t find the generated code. So, I had to go to one of the many __init__.py files and append the current directory to the PYTHONPATH (I hated this part).

OK! Now we have our generated code. But I won’t show that here, no — that’s boring!

Instead, I’m going to present what the functions are actually doing. You might have noticed that the Protobuf definitions are just that, definitions — or the skeleton of the functions. Inside, there’s no functionality whatsoever. The functionality has to be described by you. The following snippet shows this:

package endpoint



import (

"context"

"log"



"cloud.google.com/go/bigquery"

"github.com/golang/protobuf/ptypes"

pb "github.com/juandes/teamaqua/api/v1"

)



// Service implement the gRPC endpoints

type Service struct {

splashes []*pb.Splash

uploader *bigquery.Uploader

}



// NewService creates a new Service

func NewService(u *bigquery.Uploader) *Service {

return &Service{

splashes: []*pb.Splash{},

uploader: u,

}

}



func (s *Service) LogSplash(ctx context.Context, in *pb.Splash) (*pb.LogSplashResponse, error) {

log.Println(in)



s.splashes = append(s.splashes, in)

err := s.uploader.Put(ctx, in)

if err != nil {

// TODO: Handle error better and don't just exit :/

log.Fatalf("Error uploading to BQ: %v", err)

}



return &pb.LogSplashResponse{

Ok: true,

}, nil

}



func (s *Service) WaterConsumed(ctx context.Context, in *pb.Since) (*pb.WaterConsumedSince, error) {

var waterConsumed int32

since, err := ptypes.Timestamp(in.Ts)

if err != nil {

log.Fatalf("Error converting ptypes.Timestamp to time: %v", err)

}



for _, splash := range s.splashes {

splashTime, err := ptypes.Timestamp(splash.Ts)

if err != nil {

log.Fatalf("Error converting ptypes.Timestamp to time: %v", err)

}



if splashTime.After(since) {

waterConsumed += splash.Amount

}

}



return &pb.WaterConsumedSince{

Amount: waterConsumed,

}, nil

}

This piece of code is my service structure and it implements the gRPC method we previously defined. However, besides implementing functions, my service structure is responsible for keeping the list of Splash . Moreover, this struct also contains the BigQuery uploader, the object whose function is writing rows in BigQuery . To keep it simple, I won’t show the main function of the service, where I initialize service and the uploader. If you wish to see it, check out the file here.

The first function, NewService , simply creates the new service object. Then, there’s LogSplash , a method of service. As we saw before, LogSplash takes a Splash (and a context I won’t explain), and returns a LogSplashResponse , and an error (this is something particular to Golang, feel free to ignore it).

The content of the function is pretty straightforward. First, it prints the Splash, then it appends it to the list and writes to BigQuery . In the end, it returns a LogSplashResponse where “ok” is true. The second method of the struct is WaterConsumed , and its function is to sum all the amount of water consumed after the given timestamp. That’s the end of Archie.