At this point I feel mentally exhausted knowing I’ve spent 5 hours writing what is at most a third of my entire article. So here’s what you need to know: from 1980–1987 another boom occured — just like that, it was summer time again!

Japan launched its aggressively (just shy of $1 billion USD) funded Fifth-Generation Computer Systems project, a public-private partnership that focused on advancing computer architecture to, as a result, enable innovation in AI algorithms of high space and time complexity. Many other countries such in the Europe and the U.S. followed in hopes to simulate the economic success Japan experienced throughout the years.

Following the failures from the previous AI winter, many researchers decided to focus on centralizing knowledge/information so that these intelligent agents could become informed in their reasoning. Again, remember, the mainstream Internet and online search succeeded this era by over 10 years. And as a result, the new direction for AI was centralized around knowledge: expert systems. An expert system is a series/combination of logical rules layered on top of each other to determine some output based on an input. These rules were usually devised by domain experts. A common use case for expert systems were to determine the possible illnesses based on a collection of symptoms the user inputs.

A diagram representation of expert systems. It really is surprising how this was ever considered AI! — http://www.igcseict.info/theory/7_2/expert/files/stacks_image_5738.png

Expert systems, however, were really the epitome of “dumb” AI — each expert system was effectively a database that had to be coded by hand and was not self-adaptable to any extent. It was time consuming and expensive to develop, validate, and maintain — not to mention the impracticality of requiring a standalone computer to just run a single program. By the late 1980s, this growth cycle had too run its tiring course.

It was clear that expert systems were a dead-end — how could these… databases ever be further developed to create machines which could think on their own? Fortunately, in 1982, physicist John Hopfield devised a form of neural network (denoted as the “Hopfield net”) that pushed the field forward. Hopfield nets could learn and process information in an entirely new way — and they contributed to modern day Artificial Neural Networks. Their outputs could be rewired into their inputs so they are a form of Recurrent Neural Networks (RNNs). At the same time, David Rumelhart evangelized the “backpropagation” algorithm discovered by Paul Werbos two years prior. The backpropagation algorithm exploited differential Calculus to identify how much each weight in a neural network contributes to the overall error. With the ability to pinpoint the accuracy of each individual weight, modifying the entire system to move in a more optimal direction could be immediately clear.

A Hopfield net with four nodes. Does it look familiar? —https://upload.wikimedia.org/wikipedia/commons/9/95/Hopfield-net.png

Sidenote: The backpropagation algorithm is awesome, and I’m really excited to delve into the Mathematics behind it in a couple of days!

Connectionism and algorithms modelled after the brain experienced a surge in popularity. They experienced commercial success in the 1990s where they were used for programs like optical character recognition and speech recognition.

If all these terms (ANNs, RNNs, Hopfield nets, backpropagation, etc.) seem similar to you, it’s because these inventions are still widely prominent in AI today, classed under the umbrella of “Machine Learning”. Deep Learning (which, as previously mentioned, is just extremely deep layered neural networks) is perhaps the AI-craze of the 21st century.

History Repeats Itself, Part 2

So it should be surprising to find out that AI experienced a second winter from 1987–1993. Advances in Connectionism are still appreciated today, but it was time for expert systems to retire. Ultimately, The Fifth-Generation Computer Systems (alongside the Western counterparts) failed to meet its objectives (which was actually primarily centralized around NLP). As history repeated itself a second time, skepticism in AI was perhaps at its global maximum. Artificial Intelligence, the field where “ultimate failure almost always followed ultimate success”, became shunned by investors and academics.

It is important to note, however, that the field continued to make advances despite heavy criticism. That is why, when the winter inevitably thawed in 1993, it has remained in a state of California-esque summer until today, February 2016 and beyond.

Sidenote: At the end of the 1980s, Nouvelle AI — the idea that Artificial Intelligence should possess a physical body, achieved through robotics — became increasingly popular amongst researchers. I am unsure the extent to which this has developed in the present day, but I would love to hear from anyone who is knowledgeable on this topic.

II. The Present

A New Approach

And so, from 1993 onwards, the field has thrived in overcoming G.O.F.A.I. (Good Old Fashioned Artificial Intelligence) — or the “symbolic Artificial Intelligence” that we have described previously. Examples of where the field has experienced great success include neural networks, statistical modelling, probabilistic modelling, clustering, and genetic algorithms (those which can evolve a set of candidate solutions based on a digital implementation of biological natural selection). These alternative, more Mathematical paradigms rekindled optimism and excitement in the early 1990s. Century old Artificial Intelligence goals were finally achieved. There definitely seemed to be a greater focus on creating models for data through systems using geometric model related approaches. The different ways of achieving these optimal models include convex optimization, evolutionary algorithms, pattern/clustering recognition, etc.

Modern day deep Neural Networks… that’s a lot of computations! — http://www.rsipvision.com/wp-content/uploads/2015/04/Slide5.png

These algorithms tie together a plethora of different disciplines, most notably including Mathematics, Economics, and Psychology to make it a much more “scientific” discipline. Other practices such as NLP are also being worked on by many, but experience less innovation (perhaps because perfect natural language is considered to be an “AI-Complete” problem — that is a problem whose solution is essentially the equivalent of a human-level general intelligence / Strong AI). Companies like Google, Baidu, and Facebook have opened Artificial Intelligence research centers just for the purpose of pushing the field forward — they open source and publish (presumably) all of their findings. Artificial Intelligence is now recognized to be one of the “hottest” fields of Computer Science to be in during college and in career.

Overcoming Limiting Factors

In this way, there is great emphasis on machines that can “learn” from past experiences and data it is provided (and, on that note, there is a great volume of information due to the Internet). This training, especially in high dimensions, is very computationally and memory expensive but, due to advancements in hardware, we are at a point where it no longer matters. We have Moore’s Law to thank for this.

Moore’s Law: an exponential curve—http://www.extremetech.com/wp-content/uploads/2015/04/MooresLaw2.png

Moore’s Law is the observation that the number of transistors (and hence computational power) that can fit into a dense, integrated circuit has doubled approximately every two years. The observation was made by Gordon Moore, the co-founder of Intel. We can postulate that Computational Power = a ⋅ 2^(t/2) where a is the initial computational power and t is the number of years after the first existing transistor circuit. Compared to the mid 1970s, we in 2016 have at best a greater computational power of 2²⁰!

Thus, availability of data/information and limited computer power was no longer an issue. On top of that, heuristic-based search that exploited prior knowledge and the target domain solved the combinatorial explosion that search algorithms faced (this solution became more prevalent as search spaces increased eg. with mapping software). In addition, with research ongoing not only in academia but also at top Silicon Valley & tech companies, Computer Vision (eg. with Google’s self driving cars) experienced rapid development, especially through exploitation of the clustering, segmentation, and neural network algorithms that had been unlocked as the other limiting factors diminished.

A video segmentation algorithm — https://www.mpi-inf.mpg.de/fileadmin/_processed_/csm_must-links_4f7afcb337.jpg

Tangible Academia Achievements

It’s remarkably difficult to name and describe examples of breakthrough Artificial Intelligence projects in the last 20 years because… there’s so damn much! In this decade, there appears to be a new revolution every week. But, as mentioned before, advancements in game AI is a decent, general indicator on the progress of AI, and so I will discuss the new inventions with regards to this scope.

In 1997, Deep Blue became the first AI to beat the world champion (Garry Kasparov) in a game of Chess. This is a classic example of “Weak AI” — that is AI that matches or exceeds human level intelligence but only in one (or one in the potential millions) domain. By contrast, Strong AI matches or exceeds human intelligence in general (in majority of domains). In 2005, a Stanford robot won the DARPA Grand Challenge by driving autonomously for 131 miles on a desert track without any prior rehearsal on said track. Just a couple years later, CMU won a similar challenge by successfully driving an autonomous car through an urban city environment without any crashes or accidents. These efforts inspired companies like Google, Tesla, and perhaps Apple to develop commercial self-driving cars. In 2011, IBM challenged two Jeapordy! (a popular trivia/quiz game show) champions to its universal AI system Watson and successfully won by a significant margin.

Such defeats (or ties) against top contenders extended to games such as Checkers, Backgammon, Othello, Crosswords, Scrabble, Bridge, Poker, FreeCell, etc. In March of this year (2016), Google’s Deep Learning system will challenge the world’s #1 champion in a game of Go (which, as mentioned previously, is remarkably complex). These feats were not due to abrupt revolutions or constant paradigm shifts, but rather involved taking some existing algorithm eg. Neural Networks and applying tedious application of engineering skill by iteratively improving on both the efficiency and accuracy through new techniques (like Dropout, Convolutions, or Simulated Annealing) or ruthless feature tuning. Many of these new applications are published and digested/built-on by the AI community. This is Google’s paper on using Convolutional Neural Networks to power their Go AI.

Sidenote: I know very little about Convolutional Neural Networks but I think now’s the time to delve in deep :) Thanks Lenny Khazan for introducing me to Stanford’s course on the topic.