During the seven-week Insight Data Engineering Fellows Program recent grads and experienced software engineers learn the latest open source technologies by building a data platform to handle large, real-time datasets. Akshay Wadia (now a Data Platform Engineer at the start-up Stitch Fix) discusses his project of building a Spark GraphX library for handling dynamic graphs.

Shortest distances and paths have many uses in real world graph applications. For instance, in a social network like the Facebook graph, shortest distance can serve as a measure of relevance. When you search for another user on Facebook, users with smaller shortest distances are more relevant than users farther away. The shortest distance algorithm is also used as a subroutine for many other graph problems. For computing on large scale graphs, this problem can easily be cast into an iterative map-reduce job. Indeed, Spark’s graph library, GraphX , ships with such an implementation.

An important characteristic of real world graphs is that they are dynamic, in that their structure changes over time. For instance, the Facebook graph changes every time a user adds or removes a friend. The question is, how do we update the shortest distances when the graph changes. A trivial solution is to run the shortest distance algorithm again on the entire graph, recalculating the distances for every Facebook member. However, this seems wasteful, as it is possible that a particular update only changes the shortest distances of a few vertices. For example, in the figure below, the addition of the dotted edge might change the shortest distance from u to v, but will not affect p1 or p2.

Of course, the size of the subgraph affected by a particular update is strongly dependent on the structure of the graph. However, in the cases where the affected subgraph is small, we can identify that subgraph, and only recompute over that?

The goal of my project was to write such an algorithm for Spark.

The Algorithm

I implemented an algorithm called GraphInc from a recent paper1 that converts any graph analytics algorithm into a new algorithm that has the same functionality as the original, but is incremental. Here, “incremental” means that when the structure of the graph changes, the algorithm (implicitly) identifies the subgraph which is affected by the change, and then re-computes over that subgraph, rather than the entire graph. Depending on the graph structure and the nature of the update, this subgraph may be much smaller. Essentially, GraphInc utilizes the classic space-vs-time tradeoff known as memoization. In the first round, it computes over the entire graph and stores parts of the computation for later use. In subsequent rounds, after updates to the graph, it uses the stored information to compute only the affected subgraph. For this project, I implemented a restricted version of this algorithm that is specific to computing shortest distances.

Shortest Distance with Map-Reduce

First, let’s review how shortest distances are computed in the map-reduce framework. Recall that the input is a directed, weighted graph, along with a source vertex s. The task is to compute the shortest distance from s to each vertex in the graph. The map-reduce version of this problem is essentially the Bellman-Ford algorithm2 on a distributed cluster, which iteratively updates the shortest distance from the source vertex s to each vertex in the graph. The ‘map’ operation maps an edge to a message containing an update for the shortest distance to the vertex that the edge is directed towards, while the ‘reduce’ operation collects all the messages sent to each node and selects the shortest distance for each node.

More precisely, let (u,v,w) be any edge, where u is the source vertex, v is the destination vertex, and w is the weight of the edge. With each vertex v, we will let v.dist denote the current shortest distance estimate for v. The ‘map’ operation checks if v’s shortest distance estimate can be improved by using the edge from u. That is, it checks if v.dist > u.dist + wand sends the message (v.dist, u.dist + w) to the destination vertex if it does. The ‘reduce’ phase collects all update messages from the incoming edges and updates the current estimatev.dist with the minimum distance, Min(v.dist, {u.disti + wi }). It can be shown that after O(n)map-reduce iterations, where n is the number of vertices, v.dist will be the shortest distance for each vertex in the graph (see figure below).

Figure 2. In (a), all vertices are initialized to have v.dist set to infinity, except the source vertex which is set to 0. (b) In the map phase, the edges where v.dist > u.dist + w (shown in green) send a message to the respective vertices. The reduce phase updates these vertices with the minimum distance from the incoming messages. In (c), the next map-reduce iteration sends new messages (red), which account for the updated distance estimates. In (d), more messages (orange) propagate through the graph, completing the algorithm in just three map-reduce iterations.3

Incremental Shortest Distance

Now let’s turn to the incremental algorithm. Assuming we have run the above algorithm once, v.dist contains the correct shortest distance for every vertex v. If we change the weight of one edge in the graph, how do we decide if a particular edge, say (u,v,w), should produce a message or not? Since v.dist depends on u.dist and w, it follows that only those edges where either u.dist or w changes should be considered for producing messages.

Suppose that (u,v,w) is such an edge and produces a message, let’s see how the reduce step will work. If u.dist + w is smaller than v.dist, then we can simply update v.dist to the new value. However, if the new u.dist + w is greater, and the current shortest path to v goes through u, then we are in trouble. We cannot simply increase the value of v.dist, as there might be other edges with destination v, which did not participate in the update, but now have shorter paths going through them. Fortunately, this problem is easily solved by storing the weights of the shortest paths through all the incoming vertices to v, rather than only the current shortest distance. Then, after every message, v.dist is set to the minimum of the incoming message, and all the stored messages from the previous computation (see the figure below).

Figure 3. In (a), the shortest path to v goes through u2. In (b), the weight of edge (u2, v) increases to 8. Note that only the edge (u2, v) produces a new message, as nothing changes for the edge (u1, v). Therefore, in the reduce step, we do not have sufficient information to update v.dist, as now the shortest path goes through u1. In (c), we store lengths of all incoming paths (instead of just the minimum length) in v. Thus, in (d), after the update, we have enough information in the reduce step to update to the correct value.

Implementation

In this section, we will give a brief overview of the implementation in Scala. To store the shortest distances through every incident vertex, we use a Scala Map, which mapsVertexIDs to weights, which are of type double. We also need to store the current estimate of the shortest distance, and bundle it with the previous Map into a case class, as follows:

type MsgDigest= Map[VertexId,Double]

case class Memo(dist : Double, msgs : MsgDigest)

GraphX provides a very general implementation of graphs, and it is very easy to associate any arbitrary data structure to a vertex, for eg., by using the mapVertices(), as follows:

/* Initialize vertices and convert Graph[Int,Double] to Graph[Memo,Double]

*/

def initAttr(gr : Graph[Int,Double]) : Graph[Memo,Double] = {

return gr.mapVertices{ (vid,vattr) =>

val newDist = if (vid == 0) 0 else Double.MaxValue

new Memo(newDist,Map[VertexId,Double]())}

}

This function attaches Memo to each vertex, and additionally initializes the shortest distance estimate for each vertex (0 for the source, and Double.MaxValue for every other vertex).

Once the graph has been initialized, we have to implement the actual iterated map-reduce job. Fortunately, GraphX makes this very easy too! It provides a Pregel object that takes care of the iteration control flow — you only need to provide it with the map and reduce functions. As discussed before, the reduce function simply computes the minimum, while the map function (called sendMessage) is as follows:

def sendMessage(edge: EdgeTriplet[Memo,Double]) :

Iterator[(VertexId, MsgDigest)] = {

if (edge.srcAttr.dist == Double.MaxValue)

return Iterator.empty

else {

val potentialMsg = edge.srcAttr.dist + edge.attr

if (edge.dstAttr.msgs.contains(edge.srcId) &&

(edge.dstAttr.msgs(edge.srcId) == potentialMsg))

return Iterator.empty

else

return Iterator((edge.dstId, Map(edge.srcId -> potentialMsg)))

}

}

We first check if the source vertex has a valid shortest distance estimate or not. If it doesn’t, then we output an empty message. If it does, then we compute the potential message for updating the destination. Then we check if we should produce this message, by comparing the potential message to the previously stored message computed in earlier stage. If it is the same, then we again generate an empty message. Otherwise, we generate a message with the updated estimate.

For more information about the implementation, the code for this project is available at:github.com/akshaywadia/graphMod.

Summary

In general, repetitive computation often presents an opportunity to decrease amortized complexity by pre-computation. The GraphInc algorithm and the restricted version that I implemented can be seen as instantiations of this principle. By storing all incoming messages to vertices, we can save computation in later rounds by determining if a particular vertex needs to participate or not. Whether or not this trade-off — increased space required to store state versus savings accrued in subsequent rounds — makes sense for a particular application is strongly dependent on the graph structure and the kind of updates.

Some real-world graphs have a few extremely large degree nodes, and it may be too costly to store all incoming messages. Graph connectivity is another property that plays a crucial role in determining the usefulness of this strategy. For example, for a completely connected graph, the incremental algorithm might not result in any savings, as even a single edge update might affect all the shortest paths. Thus, a careful examination of the problem is required to determine if this optimization is worthwhile.