GoSocial is Medium’s service (written in Go and backed by a Neo4j database) which manages all of our social data. For more on GoSocial, check out this post.

Many of the design decisions in GoSocial were influenced by our experiences and practices in other parts of the Medium system. But there were a few places where we made design decisions that were unique to Neo4j or Go, and one of the most interesting ones is how we run queries in Neo4j concurrently.

Let’s say we want our application server to be able to get a list of all the posts recommended or published by my friends in the past day. This is a common problem; for example, we see it on our homepage.

To do this, we can have the application server hit a route on GoSocial, something like

/queries?

queryName=friends-activity&

userId=<userId>&

startTimestamp=<yesterday>

where the queryName tells us what we want, the userId tells us who for, and the startTimestamp tells us from when.

For this particular query, friends-activity, we then get two relevant cypher queries to run in Neo4j:

FRIENDS_PUBLISHED_POSTS = “MATCH (u:USER {userId})-[f:FOLLOWED]->(b:USER)-[r:PUBLISHED]->(c:POST) “ +

“WHERE r.created_at > yesterday” +

“RETURN b.user_id, r.created_at, c.post_id”

This one basically says “Get all the paths in our database that match this pattern (where our user follows a friend who has published a post) and then return that friend’s user id, the post id for the post that was published, and the time it was published.

Or, visually:

We also create a second query for getting posts recommended by friends:

FRIENDS_RECOMMENDED_POSTS = “MATCH (u:USER {userId} )-[f:FOLLOWED]->(b:USER)-[r:RECOMMENDED]->(c:POST) “ +

“WHERE r.created_at > yesterday “ +

“RETURN b.user_id, r.created_at, c.post_id”

Or, in English, “Get all the paths in our database that match this pattern, where our user follows a friend who has recommended a post. And then in these cases, return that friend’s user id, the recommended post’s post id, and the time it was recommended.”

Visually this is similar to the first one:

Given these two queries, we construct QueryRequests for each of them. A QueryRequest is defined like this:

type QueryRequest struct {

Name string

Query *neoism.CypherQuery

}

The meat of a QueryRequest is the Query itself. The Query is a neoism.CypherQuery, which includes, among other things, a string for the cypher query that Neo4j will run. For these QueryRequests we can just use the two queries that we just constructed above.

So we get something like this:

queryReqPublished := QueryRequest{

Name: “friends-published-posts”,

Query: &neoism.CypherQuery{

Statement: FRIENDS_PUBLISHED_POST,

…

},

} queryReqRecommended := QueryRequest{

Name: “friends-recommended-posts”,

Query: &neoism.CypherQuery{

Statement: FRIENDS_RECOMMENDED_POST,

…

},

}

So now we have these two QueryRequest objects, and we then call on our trusty concurrent-query-runner:

func RunConcurrentQueries(neo4jClient Neo4jClient, queries []QueryRequest, handler (func([]QueryResult) (interface{}, error)) (interface{}, error)

RunConcurrentQueries does exactly what you think it does: Given multiple Neo4j queries, it kicks off a new goroutine for each and runs them all concurrently. It takes three parameters:

A neo4jClient, which is our more-testable abstraction over a Neoism Neo4j client. A slice of QueryRequests, as constructed above. A handler function that takes a slice of QueryResults and returns an interface{} or an error; the idea here is that the handler can take the results of each of the individual Neo4j queries and then combine them however it sees fit. In our example we will want to sort the results by time, so we can use a handler that will sort QueryResults by time.

And it returns an interface{} or an error; in fact, it will return whatever the handler function returns.

So we can pass our QueryRequests, along with a client and a handler to sort the results, into RunConcurrentQueries, and get a nice array of sorted QueryResults back.

queries := []QueryRequest{queryReqPublished, queryReqRecommended}

handler := magicSortingHandler() // amazing how my post compiles

results, err := RunConcurrentQueries(client, queries, handler)

And then, after a quick bit of cleanup and shaping, GoSocial can return the results back to the application server, which knows how to turn something like this into something like your homepage:

Boom!

By the way, if you liked this post, please recommend it (so it, too, will get to be returned when we run queryReqRecommended, and your friends will see it on their homepages).