The MIT RELATE (Research in Learning, Assessing and Tutoring Effectively) group has released a study comparing learning gains in the MITx Mechanics Review physics course with students in residential classrooms. According to the study, MOOC students learned a bit more than students in a traditional university course, but less than students taught with an interactive engagement pedagogy (as would typically be exemplified by a blended learning pedagogy integrating technology with in-person instruction). The study looked at learning gains across a diverse range of students, and found that learning occurs at all levels, including the extremes. Students with minimal background in mathematics and physics learned approximately as well as normal students, who in turn learned about as well as high school physics teachers filling gaps in their knowledge.

I’m personally very excited by these findings. EdX was founded on the belief that technology can improve learning, based on decades of experience at MIT, Harvard, and others schools with projects such as TEAL, peer instruction, CyberTutor, and many others. These technology-enabled projects allowed professors to efficiently apply a range of evidence-based techniques in classrooms, leading to substantially improved student outcomes.

When I helped to create the original MITx platform (which evolved into the edX platform), one of my main goals was to enable techniques and best practices from education research — such as mastery learning, rapid feedback, active learning, constructive learning, and some uses of gamification — to be applied easily across a range of settings in which it was previously economically infeasible. For example, in a residential classroom, a single teacher may be responsible for both course design and managing dozens of students. In such a setting, it is often impossible for students to receive the kind of continuous, rapid feedback we see in 1:1 tutoring or intelligent tutoring systems, which we know leads to large improvements in learning.

With our first course, MIT’s residential and online Circuits and Electronics, we saw anecdotal evidence that technology did, in fact, allow us to apply such evidence-based techniques at a scale of dozens of students. This appeared to lead to better student learning. As edX grew, we saw increasingly strong evidence that effective use of active learning enabled by edX in residential settings consistently leads to improved student outcomes, ranging from modest to very significant.

However, how much learning occurs in MOOCs with thousands of students remained an open question. While there is no significant difference in learning between traditional residential and traditional online courses, MOOCs are a far cry from traditional online courses (which have substantial student-teacher interaction, albeit at a distance). MOOCs are fundamentally different. Individual students receive no direct instructor support. However, technology, analytics, and economies of scale allow us to design courses around sophisticated evidence-based techniques in ways previously impossible in most settings. To date, evaluation of MOOCs has been limited. While we had plenty of informal evidencethat MOOCs worked, we did not know how the benefits and costs of MOOCs balanced.

For these reasons, I was delighted to read the RELATE study. RELATE is a physics education research group at MIT, renowned for their work in educational psychometrics. Their study is perhaps the most rigorous I have seen in the field. It controls for most of the issues seen in evaluations of MOOCs:

Their research is calibrated with respect to a standardized measurement instrument, the Mechanics Baseline Test. It is phrased in terms of normalized learning gains, which controls for prior subject matter knowledge, and has published comparative results for 60 different classes. The results are robust across differences in background across a range of student subpopulations, which helps mitigate the effects of sample bias. They do a sniff test to confirm that results are not grossly affected by student dropout. They apply IRT, a technique which allows measurement of weekly performance of a range of students, including some who do not complete the posttest.

The study gives rigorous evidence that learning gains remain consistent for demographics ranging from students with no physics background to physics teachers, and furthermore, fall between the learning gains seen in a traditional physics classrooms, and those in a physics classroom based on an interactive engagement pedagogy.

It is worth pointing out that, as with most social sciences research, individual research results should be considered evidence, not proof. While the RELATE paper is one of the most rigorous I have seen, it is often difficult or impossible to deliver conclusive proof in the social sciences. For example, as the paper points out, pre/post-tests are taken under very different conditions residentially than in a MOOC. There are many similar confounding factors. In addition, the results are also specific to one specific course — MITx’s Mechanics ReView — which was created by one of the top physics education groups in the nation. While this research presents strong evidence that well-designed MOOCs can be effective learning tools, it gives little information about the effectiveness of a typical MOOC.

References: Colvin, K., Champaign, J., Liu, A., Zhou, Q., Fredericks, C., & Pritchard, D. (2014). Learning in an introductory physics MOOC: All cohorts learn equally, including an on-campus class. The International Review Of Research In Open And Distance Learning, 15(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1902/3058

Figure 1: Normalized learning gains are essentially defined in terms of what portion of the things a student does not know going into the course, the student learns during the course. The slope of the red line represents the normalized learning gains for all of the students in 8.MReVx on non-force-related items. The points show the learning gains for a range of student demographics taking the 8.MReVx MOOC analyzed in the RELATE study.