Down To Business: Blog Service API

For the remainder of this blog post we will be building a simple CRUD API for a blog service. Since MongoDB just released their official Go driver, we’ll be using MongoDB. We’ll create a server and a small CLI application for the client using Cobra. Let’s get our hands dirty !

Cobra does not yet support go-modules for generating subcommands, thus for this tutorial we’ll be using good ol’ $GOPATH :) .

This is what your folder structure should look like:

$GOPATH/grpc-mongo-crud/

- server

- client

- proto

$GOPATH/grpc-mongo-crud/ - server - client - proto Alternatively you could just generate the subcommand files in your $GOPATH and copy-paste the code into your own repository and use relative paths instead (this is the route I took).

You can find the code repository on my github.

1. Creating a Proto file

After setting up a new working directory in your $GOPATH, create a new file in the proto folder, let’s call it blog.proto. For simplicity we’ll create seperate messages for each request/response altough some will overlap.

First a little bit of boilerplate that expresses we use protbufs v3, our package is called blog, and our generate go package will be called blogpb (short for blog protocol buffer).

We’ll need 5 services, each service takes in a request and returns a response:

CreateBlog (unary)

ReadBlog (unary)

UpdateBlog (unary)

DeleteBlog (unary)

ListBlog (server streaming)

CreateBlog, ReadBlog and UpdateBlog will return a Blog item in their responses, so we’ll need a Blog Message Type. Messages can be nested, so we can use our Blog Message inside Request/Response messages.

CreateBlog Request / Response

Both the request and response will consist out of a Blog message, the main difference being that in the request, the ID will be blank as that is created by MongoDB upon creation of the document inside the database. In the response the ID will be filled in.

UpdateBlog Request / Response

Very similar to CreateBlog, but the Blog message in our Request will already contain the id to find and update that specific item in our database. The response will return the updated document.

ReadBlog Request / Response

We’ll keep this straightforward and make Blogs searchable by Id only. It is possible though to use id OR title in a single request message using oneof .

DeleteBlog Request / Response

The request object is identical to ReadBlog, we find the blog by id and remove it (oneof can be used here too). On success we return true otherwise an error is returned.

ListBlogs Request/Response

This one is special, ListBlogs will list all blogs in our database. All our previous services were unary , which means they adhere to the classic request/response mechanism. It works fine when the data is small, but for large data streaming is more optimal. ListBlogs will use server-streaming; for one request message the server will send back multiple blog messages.

Protocol Compiler: Protoc

Now that we have created our .proto definitions it is time to generate our Client and Server stubs in Go. We do this using the protocol buffer compiler combined with the protoc-gen-go plugin.

You can check whether the installation was successful:

$ protoc — version

libprotoc 3.7.1

If it is you can compile your proto file to Go stubs, this will result in a new file in the /proto folder: blog.pb.go

protoc proto/blog.proto --go_out=plugins=grpc:.

I suggest you go through the newly generated file, you’ll see types and interfaces for the server and client, methods to create a new server and client, types to instantiate (requests, responses and blog) and getters for the fields of those structs.

You will also see this line at the top of the file, please take it to heart:

// Code generated by protoc-gen-go. DO NOT EDIT.

2. Server Implementation

Starting / Stopping the server

First things first, we’ll need some boilerplate to create a gRPC server and mongoDB connection, give some feedback to the console and handle server shutdowns by the user through a shutdown hook.

We’ll set all of this up within our main() function, feel free to do some code splitting and cleanup by yourself.

Add filename and linenumber flags when we use the log package Start our listener Start a new gRPC server with no service registered yet. Create a blog service and register it to our new gRPC server. Connect to MongoDB. Accept incoming connections on our listener Handle user initiated server shutdown (CTRL+C)

Now let’s start creating our main() function.

Now let’s start a new gRPC server and register our Blog service. To achieve this we’ll use the grpc package as well as the blogpb package created by compiling our .proto file. Our blogpb package has a method called RegisterBlogServiceServer that takes in a pointer to a fresh gRPC server and a BlogServiceServer struct that adheres to the BlogServiceServer interface from our generated blog.pb.go file.

Great job ! Now well create our mongodb connection, so make sure you have MongoDB installed as well as the official mongodb-go-driver we’ll be using. If you’ve ever worked with a mongodb driver in another language, you’ll see that it’s quite similar.

Last but not least we’ll serve the listener to accept incoming connections and handle proper server shutdown using a goroutine and channel to block the main channel until the user presses CTRL+C.

And we’re done! You can now start the server running go run server/main.go , make sure however to run sudo service mongod start beforehand, or db.Ping() will return an error. You can then close the server by pressing CTRL+C.

If all went well you should see something like:

** Note: You’ll likely need to add empty methods from the next section to not get compile errors, since BlogServiceServer does not adhere to it’s interface (in blog.pb.go) yet.

Creating our Proto Service Handlers

Now that we have our server ready, it’s time to add methods for the services we defined in our .proto file. The methods will take our BlogServiceServer as receiver so it will comply with its interface.

In the function bodies we’ll generally use the following workflow:

Protbuf Message (Request) → Regular Go Struct → Convert to BSON + Mongo Action → Protobuf Message (Response)

What’s currently missing is the regular go struct to represent a blog, which can be converted to BSON. So let’s create this type above the main function and add some tags, so that the bson package will use the meta information to assign keys.

Now let’s make our CreateBlog method. It takes in a context and a request, in this case CreateBlogReq. First we’ll extract the Blog message from our request message and convert it to a regular go struct. InsertOne() will convert it into BSON for us, using the meta tags we provided. The result from InsertOne() returns the Object ID of the new document inside our collection, which we can then add to our Blog protobuf message to be wrapped in a CreateBlogResponse.

Next we’ll do ReadBlog(), the ReadBlogReq will take in the Object ID of the blog and retrieve that document from our collection. The MongoDB FindOne() methods takes in a context and a filter, which is a BSON document for which to filter by its keys. We’ll simply have to decode the found document to a regular go struct so we can convert it back to protobuffer messages.

Now that we can create and read a blog, we should also be able to delete one. For DeleteBlog we’ll use the mongodb DeleteOne() method, which takes in an Object ID of the document to remove. If the removal is succesful we’ll return a boolean else an error.

For the last of our CRUD operations, we’ll need method to update a blog. For simplicity’s sake we are going to entirely replace the existing document inside the collection with the document encoded from our request. In a real-world scenario we’d do nil checks on the arguments so that only filled out fields are actually updated (using the $set keyword from MongoDB). The second quirk is that Mongo’s FindOneAndUpdate() returns the old document, so we’ll have to explicitly tell it to return the new document instead.

The last method we’ll implement, ListBlogs, is slightly different. Our previous operations were unary CRUD operations, which means they use a classical 1 response for 1 request mechanism.

If we want to list all blogs however, the data payload might grow too large to return it in one response, therefore we’ll implement server streaming through gRPC. For one request to list the blogs, the server will keep streaming blog messages until all blogs have been sent back to the client.

MongoDB is really helpful with this through it’s cursor feature. A cursor in MongoDB is a pointer to the results of a query rather than all of the results in memory.

A pointer to the result set of a query. Clients can iterate through a cursor to retrieve results.

So what we’ll do is iterate over the cursor using cursor.Next() and send each blog as message to the client, until the cursor yields no more results at which point we’ll close the server side stream. It’s more straightforward when you see the code:

Well done, that’s it for our entire server implementation. You’re on the path to becoming a supergopher ! Next we’ll start working on our CLI client application using Cobra.

3. Client Implementation

Cobra

As mentioned earlier, we’ll be making a CLI application for our client using Cobra. It comes with a handy generator to generate boilerplate for us.

The cobra generator doesn’t work with go modules yet, so we’ll generate the command files in our $GOPATH and then copy paste them into our /client folder where we are creating our project (if you started in your $GOPATH from the start, you can run the generator from the /client folder).

First things first, install cobra. This will create an executable in $GOPATH/bin that we can run from anywhere in the terminal (if you have setup Go correctly).

$ go get github.com/spf13/cobra/cobra

Once Cobra is successfully installed, we can initiate a new project.

$ cobra init

This will generate a main.go file and a /cmd folder. The main.go file doesn’t do anything aside from calling execute on the root command, located in /cmd/root.go .

Two things in /cmd/root.go are important to us, the rootCmd struct (a cobra.Command type), which contains the command name and descriptions and the init() function , where we’ll initiate the client. Since sub-commands are bound to the root command, Cobra will always initiate a client before executing the sub-command.

Give your root command a name and some descriptions:

And initialize the client:

I’m not kidding if I say that was the hardest part of our client, pretty straightforward right? Feel free to play around with Cobra by adding flags or commands, you also have “help” flag out of the box. In our finished application it will look something like this when called:

Create Blog Command

Now that our root command is set up, let’s create our first sub-command so that we can actually add blogs to our database.

$ cobra add create

This will create a new file, /cmd/create.go with roughly the same layout as root.go . There’s a struct of type cobra.Command and an init() function.

In the init() function we’ll define our flags. Every blog entry needs an author, title and content so let’s make those flags required as well. To add a flag we first need the Flagset for that command, with StringP(name, shorthand, defaultVal, description) we can add a flag that has a shorthand (eg. ‘a’ for ‘author’). At the end of the init() function we bind our sub-command to the root.

To run a method associated with the create command, our cobra.Command struct will take an additional property RunE. This allows us to execute a method that returns an error, because remember we’re returning internal gRPC errors. The Flagset associated with the command (command.Flags() ) allows us to easily retrieve the arguments given to the flags.

Out of these flags we can then create our request, CreateBlogReq and call the CreateBlog method on the client. Under the hood this will send the RPC request on the wire and return a response.

Gist doesn’t like proper alignment it seems…

Read Blog Command

That’s pretty easy right? Let’s continue with something even easier and more straightforward, especially now that you’re getting the hang of it! A command to read a blog; this will take in a single flag, the blog Object Id.

Similar to the init() function for our create command, we’ll create our flag for the id and bind the sub-command to our root.

And in our command struct we’ll set the appropriate name and description. Our runE method will be similar as well. We’ll retrieve the id from the flagset, create our request (ReadBlogReq) out of it and invoke the RPC call.

Update Blog Command

Updating a blog is nearly identical on the client as creating a blog, with the addition of the id flag, because we need to tell mongoDB which document to update of course. Let’s call cobra add update and edit the generated file.

Delete Blog Command

Similar to the read command, this command takes a single flag, the blog id and removes it from the database. The code is nearly identical apart from the invoked client method.

List Blogs Command

This one will be slightly different, because if you remember it’s a RPC method that uses server side streaming. Calling ListBlogs will return a grpc client stream. If you wish to know more about this I suggest you check out the interface for it in the gRPC package. But essentially to list all blogs we’ll loop until we reach the end of the stream.

4. Finishing Up

It’s been a long road, but we’re at the finish line. Now you can actually start playing with the CLI application you made. If you’ve been working from your $GOPATH you can go install the CLI app and use the root command directly from the terminal. Otherwise you can call commands like so:

go run client/main.go create -a “Nico Vergauwen” -t “Learning Go” -c “The quick brown fox jumps over the lazy dog”

And that’s it , you’re well on your way to become a king at Go and gRPC ! Thank you for reading, I hope you enjoyed it!