bAbi





So with a deep interest in the subject in general and why my code was getting answers wrong in particular, I present a few minor nits to pick with the dataset (and a big one with task 2).



While most of these issues are rare, because this is synthetic data, errors are more serious, as they suggest problems with the underlying model.



Version of bAbi used: 1.2.



The authors:

fb.ai/babi

Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, Alexander M. Rush, "Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks", arXiv:1502.05698 [cs.AI].

Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus, "End-To-End Memory Networks", arXiv:1503.08895 [cs.NE]. Others:

Stephen Merity (Smerity) has previously written about duplication issues in the bAbi data, both heavily repeated training examples and identical questions in train and test, while describing his RNN solution.

Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Richard Socher, "Ask Me Anything: Dynamic Memory Networks for Natural Language Processing", arXiv:1506.07285 [cs.CL].

Baolin Peng, Zhengdong Lu, Hang Li, Kam-Fai Wong, "Towards Neural Network-Based Reasoning", arXiv:1508.05508 [cs.AI]. I thought it would be clever to solve bAbi, with the intention of creating my own dataset.So with a deep interest in the subject in general and why my code was getting answers wrong in particular, I present a few minor nits to pick with the dataset (and a big one with task 2).While most of these issues are rare, because this is synthetic data, errors are more serious, as they suggest problems with the underlying model.Version of bAbi used: 1.2.The authors:Others:

Task 2 Two supporting facts



1 Mary journeyed to the bathroom.

2 Sandra went to the garden.

3 Daniel went back to the garden.

4 Daniel went to the office .

5 Sandra grabbed the milk there .

6 Sandra put down the milk there .

7 Where is the milk? garden 6 2

Lots of coreference errors. I haven't checked each one, but I think about a quarter of the test answers are wrong in this manner.

Task 3 Three supporting facts



1 John moved to the bathroom.

2 Sandra took the apple there.

3 Daniel journeyed to the hallway.

4 John journeyed to the hallway.

5 John went to the kitchen.

6 Sandra moved to the bedroom.

7 Sandra went back to the garden.

8 Sandra grabbed the football.

9 John travelled to the office .

10 John grabbed the milk .

11 Daniel went back to the office.

12 Sandra dropped the apple.

13 Where was the apple before the garden? bedroom 12 7 6

14 Mary moved to the garden.

15 Mary got the apple.

16 Mary moved to the office.

17 Daniel went back to the bedroom.

18 Mary went back to the kitchen.

19 Mary went to the hallway.

20 Where was the apple before the hallway? kitchen 15 19 18

21 Mary went back to the garden.

22 Mary discarded the apple.

23 Where was the apple before the hallway? kitchen 22 19 18

24 Mary went to the bedroom.

25 Sandra got the apple.

26 John moved to the kitchen .

27 Daniel went to the hallway.

28 Mary journeyed to the bathroom.

29 John moved to the garden .

30 John travelled to the kitchen .

31 John moved to the office .

32 Mary journeyed to the garden.

33 Daniel went to the kitchen.

34 John left the milk there .

35 Sandra discarded the apple.

36 Where was the milk before the office? kitchen 34 31 30

37 Sandra moved to the hallway.

38 Mary went to the bedroom.

39 Where was the milk before the kitchen ? garden 34 30 29

Multiple correct answers.

Task 5 Three arg relations



1 Mary journeyed to the bathroom.

2 Mary moved to the hallway.

3 Mary went to the kitchen.

4 Bill went back to the bedroom.

5 Bill grabbed the apple there.

6 Fred went back to the garden.

7 Mary went to the garden.

8 Fred took the milk there.

9 Jeff moved to the hallway.

10 Bill dropped the apple there.

11 Fred handed the milk to Mary.

12 Mary handed the milk to Fred.

13 Who gave the milk? Mary 12

14 Fred went back to the bedroom.

15 Fred passed the milk to Bill.

16 Who did Fred give the milk to? Bill 15

17 Fred took the apple there.

18 Fred gave the apple to Bill.

19 Who gave the apple? Fred 18

20 Jeff went to the kitchen.

21 Bill dropped the milk.

22 What did Fred give to Bill? milk 15

Multiple correct answers. Surely a human would do badly here.

Task 8 Lists, sets



1 John went to the hallway.

2 Daniel travelled to the hallway.

3 Mary went to the bathroom.

4 Daniel got the football there.

5 What is Daniel carrying? football 4

6 Mary journeyed to the office.

7 Daniel grabbed the apple there.

8 What is Daniel carrying? football,apple 4 7

9 Mary journeyed to the kitchen.

10 Daniel went to the bedroom.

11 What is Daniel carrying? football,apple 4 7

12 Daniel dropped the football there.

13 Daniel grabbed the football there.

14 What is Daniel carrying? football , apple 4 12 13 7

Items not listed in time or alphabetical order, but some other way.