Cordon bleu without tears (Image: Yu Suzuki et al. at Kyoto Sangyo University)

CELEBRITY chef apps, online how-to videos and recipe-sharing websites have all joined traditional cookbooks as guides for the amateur epicurean. But wouldn’t it be nice if your kitchen could help you prepare a meal?

Computer scientist Yu Suzuki and colleagues at Kyoto Sangyo University in Japan kitted out a kitchen with ceiling-mounted cameras and projectors that overlay cooking instructions on the ingredients. This lets you concentrate on slicing and dicing without having to look up at a book or a screen. “Cooks can easily and visually understand how to prepare an ingredient for a recipe even if they have no cooking experience,” says Suzuki.

Suppose you want to fillet a fish. Lay it down on a chopping board and the cameras will detect its outline and orientation so the projectors can overlay a virtual knife on the fish with a line indicating where to cut. Speech bubbles even appear to sprout from the fish’s mouth, guiding you through each step (see picture).


If that is not enough, the kitchen also comes equipped with a small robot assistant named Phyno that sits on the countertop. When its cameras detect the chef has stopped touching the ingredients, Phyno asks whether that particular step in the recipe is complete. Users can answer “yes” to move on to the next step or “no” to have the robot repeat the instructions.

“A small robot assistant called Phyno sits on the countertop and guides you through the recipe”

There are some limitations, however. “Currently we have to develop a system based on a manual analysis of real cooking processes,” says Suzuki, so for now the system can only help you prepare fish and slice onions. “In the future, we will automate the analysis process.” He will present the work at the Asia Pacific Conference on Computer Human Interaction in Matsue, Japan, later this month.

“It’s a great idea,” says Thomas Ploetz at Newcastle University, UK, who has built a smart kitchen himself (“Mastering the art of French by cooking“).

Meanwhile, Jinna Lei at the University of Washington has also installed cameras in the kitchen to watch over novice chefs. Lei and colleagues used Kinect-like depth-sensing cameras capable of recording both the shape and appearance of kitchen objects, allowing them to track cooking actions, such as whether a particular ingredient has been added to a bowl.

The system uses both object and action-recognition to keep track of what the cook is doing. Each object, such as a bowl or apple, has a number of actions related to it. For example, bowls are generally used for mixing, while apples can’t be mixed but can be chopped. Tracking is about 80 per cent accurate and Lei is investigating ways to improve this, such as adding a thermal camera to better identify the user’s hands by their body heat. She will present the work at the UbiComp conference in Pittsburgh, Pennsylvania, next month.

Eventually, Lei hopes the system will be able to prompt chefs when they make a mistake. “For example, if the system detects sugar pouring into a bowl containing eggs, and the recipe does not call for sugar, it could log the aberration,” says Lei. She tested it on a cake-baking video containing seven objects and 17 actions and it identified the start and end points of actions to within a third of a second in over 90 per cent of cases.

Ploetz suggests that Lei’s research could complement Suzuki’s by helping to automate recognition of new ingredients, while adding his own work to the mix could make your kitchen even smarter. “All of them together would be a nice combination.”

Mastering the art of French by cooking Smart kitchens could help you learn another language as you hone your cookery skills. Thomas Ploetz and colleagues at Newcastle University, UK, put accelerometers in a variety of kitchen equipment as a way to record users as they perform culinary tasks. A computer gives cooking instructions in French and can sense if users have picked up the right equipment to carry out the recipe. If not, the system will rephrase again in French before finally giving an English translation. Ploetz also has plans to create equipment that can score your cooking skills, based on how well you execute each task. “The next step would be gamification,” he says. “You could compete with Jamie Oliver.”