As a culture, we don’t have a lot of warm fuzzy feelings when we contemplate a future in which artificially intelligent machines are a reality. Our fiction seems to regard them as either entirely evil, or in the case of Douglas Adams’ Genuine People Personalities, pathologically dysfunctional, at best.

Many readers might then be alarmed to learn that a branch of AI called machine learning is inserting itself into their lives at an astonishing rate using big data concepts. Machine learning (ML) is a field that intersects computer science and mathematics. The goal of ML is designing software systems that can make use of data without being explicitly programmed with information about the structure or contents of that data.

These complex algorithms create increased data demands that require massive throughput. Processing this much data at the speeds it takes to simulate AI was impossible just a few years ago. Recent inventions like powerful flash memory, innovative software, faster CPUs and DRAM have accelerated datacenter infrastructures to make machine learning possible.

ML runs counterintuitive to the general understanding that we have about how computer programs work. Even non-programmers have a general understanding of executable code as a series of explicit instructions. How then, is it possible for a machine to “learn” from input data? Math, obviously.

I Was Told There Would Be No Math

Mathematics have a tendency to scare some people, which may explain our wariness of any kind of intelligent machinery. After all, isn’t it the cold, compassionless logic that we fear about our future robot overlords? ML is certainly loaded with a lot of complex math, but it is possible to develop an intuitive understanding of how it all works with some simple examples. Maybe a little understanding can make it all a little less horrifying.

Imagine that you’re a real estate agent in a posh neighborhood and you have a new 5000 square foot house to list. How do you determine the listing price on your house? There have been three recent sales in the neighborhood: a 4000 sq ft house for $4M, a 4500 sq ft house for $4.5M and a 6000 sq ft house for $6M. With nice gigantic round numbers like this, it’s pretty clear that a reasonable price for the 5000 sq ft house is $5M. If you worked that out in your head, mathematically speaking, you just performed an algorithm called linear regression, and then performed a predictive analysis by using the known feature (5000 square feet) to ascertain the target information ($5M). Aren’t you clever?

Stated with a little bit of jargon, you have used data to develop a hypothesis about the data that can be used to make predictions. The hypothesis in this case is a line-plot. As you might remember from high school geometry, this graph is just another way of representing a function; so what you have really drawn is a hypothesis function.

Teaching Algorithms to Computers

Linear regression is the algorithm that allows computers to analyze data in the same way. Once the system is given a set of input data to “train” on, linear regression will output a hypothesis function that can be used to make predictions.

But what if the data points aren’t so neatly aligned? Imagine the data is represented by the plot below:

Just by looking at that data, you could draw a line that represents the overall trend and make the same prediction as in the previous example. However, houses are not priced on square footage alone, and linear regression is not limited to only two dimensions of data. In fact, the input data can have many more dimensions than humans can visualize. Features like the age of the house, number of bedrooms and bathrooms, neighborhood median income, distance to the nearest pub and the quality of the local school systems can all influence the price of the house. It should now be clearer how powerful this kind of capability is: Given data with some discernible pattern, machines can learn the pattern and make predictions about new things that fit the pattern.

Real World Machine Learning

This example illustrates just one capability in the ML field. This is an example of supervised learning. In supervised learning, an algorithm is given a set of training data. This is a dataset with known answers for the features that will be used to make predictions after the training phase is complete. One of the best-known examples of supervised learning is optical character recognition (OCR) and handwriting recognition. In both of those cases, the algorithm is trained with a large set of images of characters that need to be recognized, and it is told what the correct answers are. Then, when the algorithm is fed a new image that it hasn’t seen before, it is able to determine what that character is, given the characters that it has seen before. This kind of recognition is often done with a technology called Neural Networks due to their conceptual similarity to biological nerve structures.

The other class of ML algorithm is unsupervised learning. While not as commonly used, unsupervised algorithms can allow machines to sift through data to find patterns. This is especially useful if there is a pool of data, and it is known that much of the data is interrelated, but there is not a deterministic way to group the data. For example, given a set of medical records it is possible for an unsupervised algorithm to find related cases. This can be very useful in research. It can also allow a music application to group similar songs together for automated playlist creation.

Data Scientists Powering Machine Learning

The comforting aspect of ML, for those of us that fear machines becoming self-aware, is that implementing these systems still requires a lot of work from teams of programmers, mathematicians, and domain experts. In fact, ML has helped spawn an entire new profession: The data scientist. It is the data scientist’s responsibility to understand the available data and apply algorithms to accomplish tasks, like make movie recommendations, suggest products we might like, diagnose disease, rate credit risks and even predict elections.

So snuggle up to your mathematically-selected significant other and enjoy all the robot apocalypse movies the Hollywood data scientists have calculated will excite and entertain you. Now that you get the math, they aren’t so scary anyway. Except for HAL. That guy gives me the willies.

Matt Kennedy is a Big Data Solutions Architect for Fusion-io.