Marvel/YouTube

Mark Zuckerberg has ended 2016 having completed his personal challenge to build a Jarvis-style AI to run his home.

He announced at the start of the year that he wanted to build a simple AI that could control his home, including his lights, temperature, appliances, music and security. He also wanted it to "learn his tastes and patterns, learn new words and concepts, and even entertain Max" (his daughter.)


And now he has published a blog post explaining how he did it.

Zuckerberg's Jarvis uses several artificial intelligence techniques, including natural language processing, speech recognition, face recognition, and reinforcement learning, written in Python, PHP and Objective C.

Read next The world’s cities face a congested future. Integrated, smart technology can save them The world’s cities face a congested future. Integrated, smart technology can save them

"Before I could build any AI, I first needed to write code to connect these systems, which all speak different languages and protocols," the Facebook founder explained. "I had to reverse engineer APIs for some of these to even get to the point where I could issue a command from my computer to turn the lights on or get a song to play.

Machine learning versus AI: what's the difference? Machine Learning Machine learning versus AI: what's the difference?


"Further, most appliances aren't even connected to the internet yet. It's possible to control some of these using internet-connected power switches that let you turn the power on and off remotely. But often that isn't enough."

For example, he said it's hard to find a toaster that will let you push the bread down while it's switched off so he had to find a 1950s toaster and rig it up with a connected switch. Elsewhere, he had to modify a food dispenser to feed his dog Beast and a t-shirt cannon to deliver his iconic grey shirts.

"For assistants like Jarvis to be able to control everything in homes for more people, we need more devices to be connected and the industry needs to develop common APIs and standards for the devices to talk to each other," Zuckerberg continued.

Read next The rise of AI is forcing Google and Microsoft to become chipmakers The rise of AI is forcing Google and Microsoft to become chipmakers

From there, he taught Jarvis to respond to text before enabling voice recognition. This began with simple keywords such as "bedroom", "lights", and "on" and evolved to be more specific to each room. He then moved onto vision and facial recognition and later, mobile.


Facebook

"I programmed Jarvis on my computer, but in order to be useful I wanted to be able to communicate with it from anywhere I happened to be. That meant the communication had to happen through my phone, not a device placed in my home," he said.

He began by building a Messenger bot to communicate with Jarvis "because it was so much easier than building a separate app". He now texts the Jarvis bot and it can translate audio clips into commands. In the middle of the day, if someone arrives at his home, Jarvis also texts him an image to tell him who's there, or it can text him when he needs to go do something.

"I have always been optimistic about AI bots, but my experience with Jarvis has made me even more optimistic that we'll all communicate with bots like Jarvis in the future."

Read next When neural networks name planets, they call them Tina When neural networks name planets, they call them Tina

Facebook

Zuckerberg said the most difficult part for him to work on was face recognition "because most people look relatively similar compared to telling apart two random objects." He used Facebook's research, and the work it has done on facial recognition from photos, to help improve Jarvis.

He paired this software with cameras which he installed on his front door showing multiple angles. From this, he built a simple server that continuously watches the cameras and runs a two-step process: first, it runs face detection to see if any person has come into view, and second, if it finds a face, then it runs face recognition to identify who the person is.

Once it identifies the person, it checks a list to confirm Zuckerberg is expecting that person, from his calendar for example, and will let them in and tell him that they've arrived.

"This type of visual AI system is useful for a number of things, including knowing when Max is awake so it can start playing music or a Mandarin lesson, or solving the context problem of knowing which room in the house we're in so the AI can correctly respond to context-free requests like "turn the lights on" without providing a location," Zuckerberg continued.


Zuckerberg has built the first version of the Jarvis app for iOS and plans to build an Android version soon too.

"In the longer term, I'd like to explore teaching Jarvis how to learn new skills itself rather than me having to teach it how to perform specific tasks. If I spent another year on this challenge, I'd focus more on learning how learning works," Zuckerberg continued.

"Finally, over time it would be interesting to find ways to make this available to the world. I considered open sourcing my code, but it's currently too tightly tied to my own home, appliances and network configuration. If I ever build a layer that abstracts more home automation functionality, I may release that. Or, of course, that could be a great foundation to build a new product."