Is Technology-Enhanced Learning Effective? Recent Research and the "No Significant Difference" Hypothesis
Instructors approaching technology-enhanced learning (TEL) for the first time frequently want to know whether digital technology is educationally effective. Their question is not whether students like digital technology, or whether students are engaged by it, but instead whether it enhances student learning outcomes.
The four articles listed below attempt to answer this question by analyzing the results of, in total, 305 studies that compare the learning outcomes of students in traditional, face-to-face sections of a course with those of students in technology-enhanced sections of the same course.
Literature Review
A Significant Difference?
The research of Thomas Russell (see http://www.nosignificantdifference.org/) is well-known among those who study or participate in TEL. Russell has amassed hundreds of comparison studies of distance education, dating from the early 1900s. The vast majority of these studies, according to Russell, show no significant difference in learning outcomes between comparable face-to-face and distance-delivered sections of the same course. This finding is disappointing for those who hope that digital technology has the potential to improve education.
Initially, the meta-analyses reviewed here appear to confirm Russell’s conclusion. If all studies of TEL courses considered in these analyses are taken together, the overall conclusion is one of no (or very small) significant difference between the learning achieved by students in the different sections.
A closer look at the data reveals that this is not the end of the story, however. Both Bernard and Zhao note that studies of technology-enhanced courses show great variability in the student learning outcomes they report:
“While the average effect of DE [distance education] was near zero, there was a tremendous range of effect sizes…in achievement outcomes” (Bernard, 406).
“There exists tremendous variation in the outcomes of distance education and face-to-face education…In fact, contrary to Russell’s claim, it is rarely the case that the individual studies…reported no significant difference between distance and face-to-face instruction. However, the difference disappears when the studies are considered as a whole” (Zhao, 1839).
This large degree of variation shows that TEL has the potential to outperform average face-to-face classes in terms of student learning outcomes, and it raises the question of what features of technology-enhanced courses account for their relative success or failure.
Bernard’s rigorous meta-analysis of 232 comparison studies conducted from 1985-2002 provides a partial answer to this question. He distinguishes between synchronous DE (courses in which the distance and face-to-face sections are simultaneous, for instance via instructional television, or ITV), and asynchronous DE (for example, internet-based courses). Bernard then notes that ADE on average produced better student learning outcomes than face-to-face, while SDE typically produced worse outcomes (Bernard, 408). The effects in both cases were relatively small, but they were statistically significant.
Hiltz (2001) confirms Bernard’s findings. Her rather less rigorous analysis examines nineteen studies of courses that included both face-to-face sections and ADE sections which used “asynchronous learning networks”. Hiltz found that when learning outcomes were measured subjectively, via student perceptions of various sorts, there was overall no significant difference between the face-to-face and ADE sections, but when they were measured objectively, by means of course grades, final exam grades, etc., the preponderance of studies found greater learning in the ADE sections (Hiltz 2001, 4).
What Makes TEL Courses Effective?
Bernard used multiple regression analysis to determine that in the studies he examined, pedagogical features of the courses accounted for a greater percentage of the variance in learning outcomes than did features of the media used. In particular, the following features of DE courses improved learning outcomes (Bernard, 411 ff):
- the “use of systematic instructional design,”
- “active learning … that includes … some degree of collaboration among students,” and
- “opportunities for communication, both face-to-face and through mediation.”
Zhao’s systematic meta-analysis confirms these findings, particularly the emphasis on collaboration and on combining face-to-face and mediated interaction. Zhao examined 51 carefully selected comparison studies conducted from 1966-2002 and determined that the best DE courses in terms of learning outcomes had the following characteristics (Zhao, 1860 ff):
- instructor involvement in the delivery of the course; the presence of a “live” instructor (as opposed to a course consisting only of self-paced, self-guided materials) was particularly important;
- the presence of both synchronous and asynchronous interactions among students and between students and instructors;
- the inclusion of a face-to-face component in the course (the “hybrid” model); and
- undergraduate- rather than graduate-level content.
Finally, Zhao reports that “studies prior to 1998 found distance education to be less effective than face-to-face education, whereas those post-1998 found the opposite, could be an indication that distance programs are getting better – with more powerful delivery media and more sophisticated support systems.” (Zhao, 1864-5)
Hiltz’s (2000) review of three comparative studies of asynchronous learning networks provides further confirmation of the importance of interaction and collaboration in technology-enhanced courses:
… when students are actively involved in collaborative (group) learning on-line, the outcomes can be as good as or better than those for traditional classes, but when individuals are simply receiving posted material and sending back individual work, the results are poorer than in traditional classrooms (Hiltz 2000, 1).
The Upshot
The importance of the research reviewed here for instructors considering using educational technology in their courses can be summarized as follows:
- Internet-based technology-enhanced courses yield student learning outcomes that are, on average, slightly but significantly better than learning outcomes in comparable face-to-face courses.
- TEL courses may be improving over time: student learning outcomes from studies conducted after 1998 were better than those from studies conducted prior to 1998.
- There is great variability among the outcomes of TEL courses: some greatly outperform, and some greatly underperform, their face-to-face counterparts. The research reviewed here identifies ways in which instructors can ensure that their TEL courses are among the more successful ones, such as the following.
Use a careful planning process to build active, collaborative learning activities into the course. These activities should involve a significant amount of both student-student and student-instructor interaction.
Include a face-to-face component in the course if possible, and in any case try to increase teacher presence in the online components of the course.
Readings
Bernard, Robert M., Philip C. Abrami, Yiping Lou, Evgueni Borokhovski, Anne Wade, Lori Wozney, Peter Andrew Wallet, Manon Fiset, and Binru Huang. “How Does Distance Education Compare With Classroom Instruction? A Meta-Analysis of the Empirical Literature.” Review of Educational Research 74, no. 3 (2004).
Hiltz, Starr Roxanne, Yi Zhang, and Murray Turoff. "Studies of Effectiveness of Learning Networks." 2001. http://www.alnresearch.org/HTML/aln_study.htm.
Hiltz, Starr Roxanne, Nancy Coppola, Naomi Rotter, Murray Turoff, and Raquel Benbunan-Fich. "Measuring the Importance of Collaborative Learning for the Effectiveness of ALN: A Multi-Measure, Multi-Method Approach." Journal of Asynchronous Learning Networks 4, no. 2 (2000). Available after registering at: http://www.alnresearch.org.
Zhao, Yong, Jing Lei, Bo Yan, Chun Lai, and Hueyshan Sophia Tan. “What Makes the Difference? A Practical Analysis of Research on the Effectiveness of Distance Education.” Teachers College Record 107, issue 8 (2005). http://www.blackwell-synergy.com/doi/abs/10.1111/j.1467-9620.2005.00544.x.
