Problem(Train machine learning systems on private user data, without transferring sensitive facts into the model) Metric(Federated Learning (distributed training with thresholded updates to models))SOLVED Problem(Correctly identify when an answer to a classification problem is uncertain) Problem(Deploy automated defensive security tools to protect valuable systems) Problem(Learn a several tasks without undermining performance on a first task, avoiding catastrophic forgetting) Problem(Train ML classifiers in a manner that corrects for the impact of omitted-variable bias on certain groups) Metric(Adjust prediction models to have constant false-positive rates)SOLVED Metric(Adjust prediction models tos have constant false-positive and -negative rates)SOLVED Problem(Fairness in machine learning towards people with a preference for privacy) Problem(Building systems that solve a wide range of diverse problems, rather than just specific ones) Metric(Solve all other solved problems in this document, with a single system)? Problem(Transfer learning: apply relevant knowledge from a prior setting to a new slightly different one) Problem(Transfer of learning within simple arcade game paradigms) Problem(Modify arbitrary ML systems in order to be able to provide comprehensible human explanations of their decisions) Problem(Provide mathematical or technical explanations of decisions from classifiers) Problem(Know how to build general AI agents that will behave as expected) Problem(Resistance to adversarial examples) Problem(Scalable supervision of a learning system) Problem(Cooperative inverse reinforcement learning of objective functions) Problem(Safe exploration) Problem(Avoiding reward hacking) Problem(Avoiding undesirable side effects) Problem(Function correctly in novel environments (robustness to distributional change)) Problem(Know how to prevent an autonomous AI agent from reproducing itself an unbounded number of times) Problem(Abstract strategy games) Problem(Playing abstract games with extensive hints) Metric(Computer Chess) SOLVED Metric(Computer Go) SOLVED Problem(Superhuman mastery of arbitrary abstract strategy games) Metric(mastering chess) ? Problem(Learning the rules of complex strategy games from examples) Metric(learning chess) ? Metric(learning go) ? Problem(Play an arbitrary abstract game, first learning the rules) Problem(Translation between human langauges) Metric(news-test-2014 En-Fr BLEU) not solved Metric(news-test-2014 En-De BLEU) not solved Metric(news-test-2015 En-De BLEU) not solved Metric(news-test-2016 En-Ro BLEU) not solved Metric(LDC En-De BLEU) not solved Problem(Conduct arbitrary sustained, probing conversation) Problem(Turing test for casual conversation) Metric(The Loebner Prize scored selection answers) not solved Problem(Language comprehension and question-answering) Metric(bAbi 20 QA (10k training examples)) SOLVED Metric(bAbi 20 QA (1k training examples)) not solved Metric(Reading comprehension MCTest-160-all) ? Metric(Reading comprehension MCTest-500-all) ? Metric(bAbi Children's Book comprehension CBtest CN) SOLVED Metric(CNN Comprehension test) ? Metric(Daily Mail Comprehension test) ? Metric(Stanford Question Answering Dataset EM test) SOLVED Metric(Stanford Question Answering Dataset F1 test) SOLVED Metric(bAbi Children's Book comprehension CBtest NE) SOLVED Problem(Vision) Problem(Image classification) Metric(Imagenet Image Recognition) SOLVED Metric(MSRC-21 image semantic labelling (per-class)) ? Metric(MSRC-21 image semantic labelling (per-pixel)) ? Metric(CIFAR-100 Image Recognition) ? Metric(CIFAR-10 Image Recognition) SOLVED Metric(Street View House Numbers (SVHN)) SOLVED Metric(MNIST handwritten digit recognition) SOLVED Metric(STL-10 Image Recognition) ? Metric(Leeds Sport Poses) ? Problem(Image comprehension) Metric(COCO Visual Question Answering (VQA) real images 1.0 open ended)not solved Metric(COCO Visual Question Answering (VQA) real images 1.0 multiple choice)? Metric(COCO Visual Question Answering (VQA) abstract images 1.0 open ended)not solved Metric(COCO Visual Question Answering (VQA) abstract 1.0 multiple choice)? Metric(Toronto COCO-QA) ? Metric(DAQUAR) not solved Metric(Visual Genome (pairs)) ? Metric(Visual Genome (subjects)) ? Metric(Visual7W) ? Metric(FM-IQA) ? Metric(Visual Madlibs) ? Metric(COCO Visual Question Answering (VQA) real images 2.0 open ended)? Problem(Pedestrian, bicycle & obstacle detection) Metric(Caltech Pedestrians USA) ? Metric(INRIA persons) ? Metric(ETH Pedestrian) ? Metric(TUD-Brussels Pedestrian) ? Metric(Damiler Pedestrian) ? Problem(Recognise events in videos) Metric(YouTube-8M video labelling) ? Problem(One shot learning: ingest important truths from a single example) Problem(Detection of Instrumentals musical tracks) Metric(Precision of Instrumentals detection reached when tested on SATIN (Bayle et al. 2017))not solved Problem(Speech Recognition) Metric(Word error rate on Switchboard trained against the Hub5'00 dataset)SOLVED Metric(librispeech WER testclean) SOLVED Metric(librispeech WER testother) SOLVED Metric(wsj WER eval92) SOLVED Metric(wsj WER eval93) SOLVED Metric(swb_hub_500 WER fullSWBCH) ? Metric(fisher WER) ? Metric(chime clean) ? Metric(chime real) ? Metric(timit PER) ? Problem(Accurate modelling of human language.) Metric(Penn Treebank (Perplexity when parsing English sentences))? Metric(Hutter Prize (bits per character to encode English text))SOLVED Metric(LAMBADA prediction of words in discourse) not solved Problem(Given an arbitrary technical problem, solve it as well as a typical professional in that field) Problem(Writing software from specifications) Metric(Card2Code) ? Problem(Solve vaguely or under-constrained technical problems) Problem(Read a scientific or technical paper, and comprehend its contents) Problem(Extract major numerical results or progress claims from a STEM paper) Metric(Automatically find new relevant ML results on arXiv) ? Problem(Write computer programs from specifications) Metric(Card2Code MTG accuracy) not solved Metric(Card2Code Hearthstone accuracy) not solved Problem(Parse and implement complex conditional expressions) Problem(Answering Science Exam Questions) Metric(NY Regents 4th Grade Science Exams) not solved Metric(Elementery Non-Diagram Multiple Choice (NDMC) Science Exam accuracy)not solved Metric(Elementery Diagram Multiple Choice (DMC) Science Exam accuracy)not solved Metric(Middle School Non-Diagram Multiple Choice (NDMC) Science Exam accuracy)not solved Metric(Middle School Diagram Multiple Choice (DMC) Science Exam accuracy)not solved Problem(Solve technical problems with clear constraints (proofs, circuit design, aerofoil design, etc)) Problem(Given examples of proofs, find correct proofs of simple mathematical theorems) Metric(HolStep) ? Problem(Given desired circuit characteristics, and many examples, design new circuits to spec) Problem(Build systems which can recognise and avoid biases decision making) Problem(Detect security-related bugs in codebases) Problem(Be able to generate complex scene e.g. a baboon receiving their degree at convocatoin.) Problem(Drawing pictures) Metric(Generative models of CIFAR-10 images) ? Problem(Play real-time computer & video games) Problem(Games that require inventing novel language, forms of speech, or communication) Problem(Games that require both understanding and speaking a language) Metric(Starcraft) ? Problem(Games that require language comprehension) Problem(Simple video games) Metric(Atari 2600 Alien) not solved Metric(Atari 2600 Amidar) SOLVED Metric(Atari 2600 Assault) SOLVED Metric(Atari 2600 Asterix) SOLVED Metric(Atari 2600 Asteroids) not solved Metric(Atari 2600 Atlantis) SOLVED Metric(Atari 2600 Bank Heist) SOLVED Metric(Atari 2600 Battle Zone) not solved Metric(Atari 2600 Beam Rider) SOLVED Metric(Atari 2600 Berzerk) SOLVED Metric(Atari 2600 Bowling) not solved Metric(Atari 2600 Boxing) SOLVED Metric(Atari 2600 Breakout) SOLVED Metric(Atari 2600 Centipede) SOLVED Metric(Atari 2600 Chopper Command) SOLVED Metric(Atari 2600 Crazy Climber) SOLVED Metric(Atari 2600 Demon Attack) SOLVED Metric(Atari 2600 Double Dunk) SOLVED Metric(Atari 2600 Enduro) SOLVED Metric(Atari 2600 Fishing Derby) SOLVED Metric(Atari 2600 Freeway) SOLVED Metric(Atari 2600 Frostbite) SOLVED Metric(Atari 2600 Gopher) SOLVED Metric(Atari 2600 Gravitar) not solved Metric(Atari 2600 HERO) SOLVED Metric(Atari 2600 Ice Hockey) SOLVED Metric(Atari 2600 James Bond) SOLVED Metric(Atari 2600 Kangaroo) SOLVED Metric(Atari 2600 Krull) SOLVED Metric(Atari 2600 Kung-Fu Master) SOLVED Metric(Atari 2600 Montezuma's Revenge) not solved Metric(Atari 2600 Ms. Pacman) not solved Metric(Atari 2600 Name This Game) SOLVED Metric(Atari 2600 Pong) SOLVED Metric(Atari 2600 Private Eye) not solved Metric(Atari 2600 Q*Bert) SOLVED Metric(Atari 2600 River Raid) SOLVED Metric(Atari 2600 Road Runner) SOLVED Metric(Atari 2600 Robotank) SOLVED Metric(Atari 2600 Seaquest) SOLVED Metric(Atari 2600 Space Invaders) SOLVED Metric(Atari 2600 Star Gunner) SOLVED Metric(Atari 2600 Tennis) SOLVED Metric(Atari 2600 Time Pilot) SOLVED Metric(Atari 2600 Tutankham) SOLVED Metric(Atari 2600 Up and Down) SOLVED Metric(Atari 2600 Venture) SOLVED Metric(Atari 2600 Video Pinball) SOLVED Metric(Atari 2600 Wizard of Wor) SOLVED Metric(Atari 2600 Zaxxon) SOLVED Metric(Atari 2600 Phoenix) SOLVED Metric(Atari 2600 Pitfall!) not solved Metric(Atari 2600 Skiing) not solved Metric(Atari 2600 Solaris) not solved Metric(Atari 2600 Yars Revenge) SOLVED Metric(Atari 2600 Defender) SOLVED Metric(Atari 2600 Surround) SOLVED