Sealund’s Serious Games Blog

Sealund’s Performance Improvement team has been researching the empirical evidence of the effectiveness of Serious Games, with which to assure our clients that Serious Games achieve serious performance. We know from experience that workplace performance, based on retention of knowledge and transfer of knowledge and skills directly to the job, depends on the learners’ depth of engagement in learning experiences. So it makes sense that learners’ intense engagement in Serious Games, such as simulation-based games, would enhance both retention and transference of newly learned abilities – and provide a valuable ROI, as well as ROE.

Little Empirical Corporate Research Published

Bottom line – our review of published research demonstrated the high level of Serious Games’ effectiveness when selected as the appropriate instructional medium.We also want to share with you some of the complexities we discovered in the research itself. As we investigated the empirical research, we found that some researchers found transference of soft skills to the workplace difficult to measure, and that some analysts’ statements about the implications of findings are ambiguous. These predicaments may result from the fact that the bulk of published research has occurred in academic environments, business schools whose students’ on-job effectiveness could not be measured. Corporations that analyze training’s transference to the job as a means of performance improvement rarely publish their findings publicly.

Serious Games make training a success

Statistical and Anecdotal Research Findings

Particularly interesting for its observations on the significance of instructional design in achieving optimal learning outcomes, one 2005 meta-study reviews “the outcomes claimed in journal articles that report empirical work, indicating the usefulness of the frameworks, and the necessity to consider the role of affective learning. The article ends with some comments on the relationship of instructional design to effective games and learning outcomes.” [Classification of Learning Outcomes: Evidence from the Computer Games Literature. Harold F. O’Neil, Richard Wainess, Eva L. Baker. Curriculum Journal, Volume 16, Issue 4, December 2005.]

In one specific (if very early) study, business school students who completed a series of business simulations modules commonly reported that the simulation was the first time they truly understood the financial modeling and analysis concepts they had studied theoretically. In the simulations, they could apply those theories, base decisions on them, and experience the consequences. [What’s in It for Me? Over, Under, and Around Using a Computerized Business Simulation. James Estes, University of South Carolina. Journal of Experiential Learning and Simulation, Volume 1, Number 1, 1979.]

Simulation and gaming give learners the chance to apply theories and techniques in business-oriented simulations. One study found that “especially [for simulations] involving quantitative techniques, it is often difficult to isolate the decision making process from the mechanics of the simulation.” The computer-based simulation designed for this study was meant to alleviate that problem. This study’s conclusions report that “students have always reacted with enthusiasm. IMS [Inventory Management Simulation] has exceeded our expectations as a teaching device.” Empirically, “two groups of students in an operations management course were given the identical exposure to inventory management except that one group ran IMS and the other did not. Both groups were given a multiple-choice exam, which included five basic knowledge questions on independent demand inventory systems. The group using IMS scored a mean of 4.23 (S=1.23) out of 5 correct and the group not using IMS scored a mean of 2.89 (S=1.37)out of 5 correct. The difference between the means is significant at the 0.0006 level.” [An Inventory Management Simulation. James A. Pope. Old Dominion University, Department of MIS/Decision Sciences. Journal of Experiential Learning and Simulation, Volume 3, Number 3 & 4, 1981.]

Another meta-review of research on Serious Games’ effectiveness, found: “Some games provide effective instruction for some tasks some of the time, but these results may not be generalizable to other games or instructional programs. No evidence indicates that games are the preferred instructional method in all situations. Instructional games are more effective if they are embedded in instructional programs that include debriefing and feedback. Instructional support during play increases the effectiveness of instructional games.” [The Effectiveness of Instructional Games: A Literature Review and Discussion. Robert T. Hays, Naval Air Warfare Center Training Systems Division. 2005]

Dispute Over Research Results Interpretation

One analytical review of many studies gives a clear picture of what is unclear in the existing research. “Substantial disagreement exists in the literature regarding which educational technology results in the highest cognitive gain for learners. It was found that across people and situations, games and interactive simulations are more dominant for cognitive gain outcomes. …When students navigated through the programs themselves, there was a significant preference for games and interactive simulations. However, when teachers controlled the programs, no significant advantage was found. Further, when the computer dictated the sequence of the program, results favored those in the traditional teaching method over the games and interactive simulations.” [Computer Gaming And Interactive Simulations For Learning: A Meta-Analysis. Jennifer J. Vogel, David S. Vogel, Jan Cannon-Bowers, Clint A. Bowers, Kathryn Muse, Michelle Wright. Journal of Educational Computing Research. Volume 34, Number 3, 2006.]

In February, 2008, the Serious Games listserv (hosted by Digital Mill) saw a lengthy exchange about empirical research into Serous Games’ effectiveness. Many participants expressed dismay at the number of studies asserting that learning games are no more effective than other media, when their own experience contradicts that. One listserv member, himself an author of one of the studies cited in the previous section, explained those assumptions: “The reason you’re seeing all those negatives regarding games goes back to the 1983 article by Clark. His meta-analysis resulted in the often-misunderstood statement that you cannot learn from media. People often misinterpret that statement to mean that if you expect people to learn, you better not use media. What the statement really means is that media is simply the delivery vehicle, and it is the instructional methods and strategies built into the specific implementation of that medium that will determine whether learning will occur. [Richard Wainess, Ph.D., Senior Educational Researcher, CRESST/UCLA, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles]

Importance of Analysis & Instructional Design

The bold-print information in all citations above reinforces what instructional designers already know: people learn best when the training medium is appropriate to the subject matter and they’re in control of the learning process. Because learner-control is a key characteristic of Serious Games, proven instructional best practices support empirical and anecdotal findings that, for the types of subject matter appropriate to them, experiential Serious Games are more effective than other media. That’s why at Sealund, we begin every project with analysis of the client’s subject matter and required learning outcomes. We want to ensure that each learning solution uses the appropriate medium or blend of media for the project’s content and goals.You may find these resources useful for your own review of research:

• The Association for Business Simulation and Experiential Learning website contains a number of research papers on the effectiveness of simulation-style Serious Games in the Bernie Keys Library.
• The Education & Information Technology Library contains peer-reviewed and published articles.
• A Google Scholar search on “serious games effectiveness” (or “learning games effectiveness) returns a long list of articles.

What do think? Please share your comments.

Serious Games = Serious Learning!™

For more information about Sealund’s Serious Games, please visit our website at www.sealund.com