As Mark Zuckerberg rings in 2017, he can ask his personal assistant Jarvis to summon Auld Lang Syne through his living room speaker, or allow guests into his party, using facial recognition and a smart lock. His project to make himself a virtual personal assistant—so much like Tony Stark’s in Iron Man that it carries the same name—was one of the Facebook founder’s yearly goals for self-fulfillment. It also happens to be a great way to show off Facebook Messenger.
“In some ways, this challenge was easier than I expected,” Zuckerberg writes in a Facebook post, comparing it to his other goal of running 365 miles in 2016. He says that building the virtual assistant, which “uses several artificial intelligence techniques, including natural language processing, speech recognition, face recognition, and reinforcement learning, written in Python, PHP and Objective C,” took less time than the running.
(Zuck runs a sub-6 minute mile, so you do the math. Actually, I did the math, he would have spent less than 36.5 hours making Jarvis, if he runs a consistent 6-minute pace.)
Jarvis is meant to control Zuckerberg’s home: it can automatically unlock the front door for guests, control the lights and home security systems, manage the temperature, and activate a toaster from the 1950s that’s been upgraded with an internet-connected power switch. The bot is controlled two ways: by a Messenger bot and an iOS app he built for voice recognition. (When Zuckerberg showed me the system, he issued voice commands through the Messenger app.)
Some of the tasks that Jarvis handles, like facial recognition, seem to be clever custom systems. Zuckerberg uses multiple cameras to identify specific faces and then checks the name against a pre-approved list of visitors. If you’re on the list, you’re in! (Hopefully he set specific times that it would work.) Others are simple developer tools connected to Facebook Messenger, like turning on and off lights.
Coming out of the personal challenge, Zuckerberg seems more bullish on the idea of chatbots (of course!), but also has some fresh thoughts about how they should work for users. He found himself not only wanting to speak commands, but also be able to type them out naturally. He says that while speech works well for things that will affect a group of people, it can be otherwise uncomfortable.
“If I’m doing something that relates to [to a group], like playing music for all of us, then speaking feels fine, but most of the time text feels more appropriate,” he writes. “Similarly, when Jarvis communicates with me, I’d much rather receive that over text message than voice. This suggests that future AI products cannot be solely focused on voice and will need a private messaging interface as well.”
Zuckerberg probably won’t release the code to Jarvis, he writes, because it’s too closely tied to his home security system. But he briefly ponders giving away software for a similar system focused on home automation. Could we see a Facebook AI-powered speaker to rival Amazon and Google?
“Of course,” he writes, “That could be a great foundation to build a new product.”