Posts

PageRank and MapReduce

Data mining and machine learning

About

Part of what makes Google such an amazing engine of innovation is their internal technology stack: a set of powerful proprietary technologies that makes it easy for Google developers to generate and process enormous quantities of data. According to a senior Microsoft developer who moved to Google, Googlers work and think at a higher level of abstraction than do developers at many other companies, including Microsoft: “Google uses Bayesian filtering the way Microsoft uses the if statement” (Credit: Joel Spolsky). This series of posts describes some of the technologies that make this high level of abstraction possible. The technologies I’ll describe include:

The Google File System: a simple way of accessing enormous amounts of data spread across a large cluster of machines. Acts as a “virtual file system” that makes the cluster look to developers more like a single machine, and eliminates the need to think about details like what happens when a machine fails.

a simple way of accessing enormous amounts of data spread across a large cluster of machines. Acts as a “virtual file system” that makes the cluster look to developers more like a single machine, and eliminates the need to think about details like what happens when a machine fails. Bigtable: Building on the Google File System, Bigtable provides a simple database model which can be run across large clusters. This allows developers to ignore many of the underlying details of the cluster, and concentrate on getting things done.

Building on the Google File System, Bigtable provides a simple database model which can be run across large clusters. This allows developers to ignore many of the underlying details of the cluster, and concentrate on getting things done. MapReduce: A powerful programming model for processing and generating very large data sets on large clusters, MapReduce makes it easy to automatically parallelize many programming tasks. It is used internally by Google developers to process many petabytes of data every day, enabling developers to code and run simple but large parallel jobs in minutes, instead of days.

Together, these technologies make it easy to run large parallel jobs on very big data sets. Running on top of this technology stack are many powerful data mining and machine learning algorithms. I’ll describe at least two of these:

Web Crawling

PageRank

and may describe more, including statistical approaches to spell checking and machine translation, and recommendation algorithms.

In addition to understanding how these technologies work, we’ll also:

Develop toy implementations of some of the technologies.

Investigate some related open source technologies, such as Hadoop, CouchDB, Nutch, Lucene, and others.

I’ve never worked at Google. The posts are based on the published literature, especially some key technical publications released by Google describing the Google File System, Bigtable, MapReduce, and PageRank. I’m sure there are some significant differences between the material I will describe, and the current internal state-of-the-art. Still, we should be able to build up a pretty good picture of a data-intensive web company, even if it’s not accurate about Google’s particulars.

Exercises and problems: The posts will contain exercises for you to do to master the material, and also some more challenging and often open-ended problems. Note that while I’ve worked on many of the problems, I by no means have full solutions.

FriendFeed room: There is a FriendFeed room for the series.

Accompanying lectures: I’m basing a lecture series in Waterloo, Ontario, on the posts. Here’s some details about the lectures.

Audience: The audience will include both theoretical physicists and developers; the lectures should be accessible to both sets of people.

Time: Lectures will be held every Tuesday, 7:00-8:30pm. The exception is the first lecture, which will be Thursday, December 4, 2008, 7:00-8:30pm.

Location: AideRSS have kindly offered us a room for the course, at 505-180 King Street South, Waterloo – it’s opposite the Brick Brewery, and near the Hospital. To get into the building, you need someone from AideRSS to let you in. If no one from AideRSS is by the entrance, call (519) 498 1476.

Background reading

The lectures are based primarily on the following sources:

Related resources include:

First of five lectures on MapReduce. Lecture course on scalable systems at the University of Washington, covering many similar topics. Videos of the lectures are included.

Many more resources will be posted in the FriendFeed room for the course.