RecipeQA is a dataset for multimodal comprehension of cooking recipes. It consists of over 36K question-answer pairs automatically generated from approximately 20K unique recipes with step-by-step instructions and images. Each question in RecipeQA involves multiple modalities such as titles, descriptions or images, and working towards an answer requires (i) joint understanding of images and text, (ii) capturing the temporal flow of events, and (iii) making sense of procedural knowledge.

RecipeQA Paper

EMNLP 2018 Talk Slides

To better know about RecipeQA, please read our comprehensive datasheet documenting and describing the details about its creation, strengths and limitations.

RecipeQA Datasheet

What makes RecipeQA different?

RecipeQA is meant to facilitate research on comprehending procedural knowledge in a multimodal setting where cooking recipes are used as testbed. It differs from existing reading comprehension datasets in the following ways: (1) it leverages data from real natural language found online, (2) the multimodal aspects of the questions makes the benchmark less gameable, preventing questions from easily answerable through shallow signals, and (3) it involves a large number of images which are taken by ordinary people in unconstrained environments.

Blog post

Getting Started

Browse the examples in RecipeQA:

Explore RecipeQA

Download a copy of the RecipeQA dataset in json format:

Download Training Set (150 MB)

Download Validation Set (18 MB)

Download Test Set (18 MB)

Download Images (2.3 GB)

Download the recipes used in the questions:

Download Validation Set Recipes (400 MB)

Evaluation

To evaluate your models, we provide an evaluation script that will be used for the official evaluation, along with a sample prediction file. To run the evaluation, use:

python evaluate.py {path-to-prediction-file} {path-to-validation-set} {path-to-output-file}

Evaluation Script

Sample Prediction File

(on Validation Set)

Submission

Once you are satisfied with your model performance on the validation set, you can submit it to get the official score on the test set. To preserve the integrity of the test results, we do not release the test set to the public. Follow this tutorial on how to submit your model for an official evaluation:

Submission Tutorial