Robocook: Culinary bots replicate recipes after watching videos

07 Jun 2023 --- Robotic chefs could bring health and economic benefits if the technology is widely implemented. Unlike humans – who can learn recipes by watching someone else cook or a cooking video – programming a robot to create various dishes is costly and time-consuming. However, new research shows that video content could be a rich data source for automated food production, enabling more accessible and cheaper robot chefs. 

Researchers programmed a robotic chef to recreate eight simple salads from a cookbook. The robot could identify which recipe was prepared in a cooking video and make it. The system also learned a ninth recipe from only watching a video.

New age kitchen
The robot determined which recipe was being prepared by identifying the ingredients and actions of the human chef. 

Out of sixteen videos watched, the robot recognized the correct recipe 93% of the time, even though it only detected 83% of the human chef’s actions. The robot could also see that making a double portion or human error was a variation, not a new recipe. 

“We wanted to see whether we could train a robot chef to learn in the same incremental way humans can – by identifying the ingredients and how they go together in the dish,” says lead author Grzegorz Sochacki from the University of Cambridge, the UK.

“It’s amazing how much nuance the robot was able to detect. These recipes aren’t complex – they’re essentially chopped fruits and vegetables. Still, it was effective at recognizing, for example, that two chopped apples and two chopped carrots are the same recipe as three chopped apples and three chopped carrots.” 

Programming through video demonstrations 
The research team developed eight salad recipes and filmed themselves making these. They trained their robot chef through a publicly available neural network programmed to identify various objects, such as the fruits and vegetables used in the recipes – broccoli, carrot, apple, banana and orange. 

The robot recognized the correct recipe from its cookbook 93% of the time after watching sixteen videos.The robot could identify objects and features by analyzing each video frame, such as ingredients and cooking equipment. It detected the demonstrator’s arms, hands and face. It converted the recipes and videos to vectors and used mathematics to determine the similarity between a demonstration and the vector. 

If the human demonstrator held a knife in one hand and a carrot in the other, the robot chef inferred that the carrot would be chopped up. 

However, the demonstrator needed to hold up the carrot so the robot could see the whole vegetable to identify it correctly. 

“Our robot isn’t interested in the sorts of food videos that go viral on social media – they’re simply too hard to follow,” explains Sochacki. “But as these robot chefs get better and faster at identifying ingredients in food videos, they might be able to use sites like YouTube to learn a whole range of recipes.”

Future of robotics
Food accelerator programs believe innovative start-ups, such as those piloting artificial intelligence and robotics, are crucial to developing sustainable food systems.

The study’s authors expect that cooking automation in homes and hospitality sectors offers benefits by reducing time spent on cooking and thus increasing access to high-quality meals. Experiments in robotic chefs vary from dishwasher packing and burger-flipping robots to building automated restaurants. 

However, the widespread implementation of cooking robots is limited by costs. For example, according to a BBC article, the minimum price for the first robotic chef developed by Moley Robotics was set at GBP£150,000 (US$ 186,570) in 2021. 

The robotic kitchen, introduced in 2015, was developed by a team of 100 engineers over six years. Moley Robotics Kitchen can prepare 5,000 types of meals with robotic arms. The group recorded a chef’s cooking skills in the robot’s system, whose robotic arms imitate every move. 

The article, published in IEEE, concludes that more developments are needed to make the setup practical for kitchens. The system should be put into one machine-learning model instead of the pipeline system used in the experiment. Moreover, robotic control needs to be improved. 

By Jolanda van Hal