Serious Games – What’s In It for My
Organization? By Barbara Sealund
You
get more direct knowledge and skill transfer to the job.
Trainers
and front-line supervisors alike have long observed that emotion encodes
experience in memory. The competition inherent in games provides that emotional
boost to learning and its transfer to the job. Any athlete or coach will tell
you that repetition plus the emotion of competition makes practice most
effective.
In
the business world, game-based simulations of realistic situations allow for
“safe failures” so the participants can learn from trial-and-error, one
preferred learning mode for learners who exhibit leadership talent with a
can-do attitude. The gaming experience rewards decision-making and reasonable
risk-taking, can add coaching along the way, and provides diverse experience in
thinking skills themselves – not just getting the one right answer.
These
benefits have been observed and reported anecdotally for decades and explained
in part by studies of brain function. (Ruth Palumbo Weiss, Emotion and Learning – implications of new neurological
research for training techniques, on BNet.)
Empirical
research proves it.
Now,
ground-breaking empirical studies conducted upon business and economics
students by the Department of Defense have proven the job-transfer benefits of
game-enhanced learning (the experimental group) over other modes of learning
alone (the control group).
“Three
research studies were conducted at a national university to examine the
difference in academic achievement among students who did and did not use video
games in learning. Three different video games were added to approximately half
the classes of freshmen Introduction to Business and Technology courses, 3rd
year Economics courses, and 3rd year Management courses. Identical testing
situations were used in all courses while data collected included game use, test
scores, gender, ethnicity, and age. ANOVA, chi-squared, and t tests were used
to test game use effectiveness. Students in classes using the game scored
significantly higher means than classes that did not. There were no significant
differences between genders, yet both genders scored significantly higher with
game play. There were no significant differences between ethnicities, yet all
ethnic groups scored significantly higher with game play. Students 40 years and
under scored significantly higher with game play, while students 41 and older
did not. These studies add definitive research in the area of game-based
learning. The DOD now has studies proving the efficacy of digital game-based
learning and how it can improve learning.” (See Richard Blunt, Ph.D., “Does
Game-Based Learning Work? Results from Three Recent Studies.” The published
version of this paper was presented at the I/ITSEC conference in November,
2007. An abstract is available at http://digiplay.info/node/3143.)
Those
means-based results reflect simple post-test numbers. Your organization’s
performance measurements can provide even more meaningful metrics related
directly to employees’ performance criteria and objectives. To discover the bang
serious games give for your buck, start using them and measuring the
performance results.
Serious Games Effectiveness Research By
Barbara Sealund
Sealund’s
Performance Improvement team has been researching the empirical evidence of the
effectiveness of Serious Games, with which to assure our clients that Serious
Games achieve serious performance. We know from experience that workplace
performance, based on retention of knowledge and transfer of knowledge and
skills directly to the job, depends on the learners’ depth of engagement in
learning experiences. So it makes sense that learners’ intense engagement in
Serious Games, such as simulation-based games, would enhance both retention and
transference of newly learned abilities – and provide a valuable ROI, as well
as ROE.
Little
Empirical Corporate Research Published
Bottom
line – our review of published research demonstrated the high level of Serious
Games’ effectiveness when selected as the appropriate instructional medium. We
also want to share with you some of the complexities we discovered in the
research itself. As we investigated the empirical research, we found that some
researchers found transference of soft skills to the workplace difficult to
measure, and that some analysts’ statements about the implications of findings
are ambiguous. These predicaments may result from the fact that the bulk of
published research has occurred in academic environments, business schools
whose students’ on-job effectiveness could not be measured. Corporations that analyze
training’s transference to the job as a means of performance improvement rarely
publish their findings publicly.
Statistical
and Anecdotal Research Findings
Particularly
interesting for its observations on the significance of instructional design in
achieving optimal learning outcomes, one 2005 meta-study reviews “the outcomes
claimed in journal articles that report empirical work, indicating the
usefulness of the frameworks, and the necessity to consider the role of
affective learning. The article ends with some comments on the relationship
of instructional design to effective games and learning outcomes.” [Classification
of Learning Outcomes: Evidence from the Computer Games Literature. Harold
F. O’Neil, Richard Wainess, Eva L. Baker. Curriculum Journal, Volume 16, Issue
4, December 2005.]
In
one specific (if very early) study, business school students who completed a
series of business simulations modules commonly reported that the simulation
was the first time they truly understood the financial modeling and analysis
concepts they had studied theoretically. In the simulations, they could
apply those theories, base decisions on them, and experience the consequences.
[What’s in It for Me? Over, Under, and Around Using a Computerized Business
Simulation. James Estes, University of South Carolina. Journal of
Experiential Learning and Simulation, Volume 1, Number 1, 1979.]
Simulation
and gaming give learners the chance to apply theories and techniques in
business-oriented simulations. One study found that “especially [for
simulations] involving quantitative techniques, it is often difficult to
isolate the decision making process from the mechanics of the simulation.” The
computer-based simulation designed for this study was meant to alleviate that
problem. This study’s conclusions report that “students have always reacted
with enthusiasm. IMS [Inventory Management Simulation] has exceeded our
expectations as a teaching device.” Empirically, “two groups of students in an
operations management course were given the identical exposure to inventory
management except that one group ran IMS and the other did not. Both groups
were given a multiple-choice exam, which included five basic knowledge
questions on independent demand inventory systems. The group using IMS scored
a mean of 4.23 (S=1.23) out of 5 correct and the group not using IMS scored a
mean of 2.89 (S=1.37) ©out of 5 correct. The difference between the means is
significant at the 0.0006 level.” [An Inventory Management Simulation.
James A. Pope. Old Dominion University, Department of MIS/Decision Sciences.
Journal of Experiential Learning and Simulation, Volume 3, Number 3 & 4,
1981.]
Another
meta-review of research on Serious Games’ effectiveness, found: “Some games
provide effective instruction for some tasks some of the time, but these
results may not be generalizable to other games or instructional programs. No
evidence indicates that games are the preferred instructional method in all
situations. Instructional games are more effective if they are embedded
in instructional programs that include debriefing and feedback.
Instructional support during play increases the effectiveness of instructional
games.” [The Effectiveness of Instructional Games: A Literature Review and
Discussion. Robert T. Hays, Naval Air Warfare Center Training Systems Division.
2005]
Dispute
Over Research Results Interpretation
One
analytical review of many studies gives a clear picture of what is unclear in
the existing research. “Substantial disagreement exists in the literature regarding
which educational technology results in the highest cognitive gain for
learners. It was found that across people and situations, games and
interactive simulations are more dominant for cognitive gain outcomes. …When
students navigated through the programs themselves, there was a significant
preference for games and interactive simulations. However, when teachers
controlled the programs, no significant advantage was found. Further, when the
computer dictated the sequence of the program, results favored those in the
traditional teaching method over the games and interactive simulations.” [Computer
Gaming And Interactive Simulations For Learning: A Meta-Analysis. Jennifer
J. Vogel, David S. Vogel, Jan Cannon-Bowers, Clint A. Bowers, Kathryn Muse,
Michelle Wright. Journal of Educational Computing Research. Volume 34, Number
3, 2006.]
In
February, 2008, the Serious Games listserv (hosted by Digital Mill)
saw a lengthy exchange about empirical research into Serous Games’
effectiveness. Many participants expressed dismay at the number of studies
asserting that learning games are no more effective than other media, when
their own experience contradicts that. One listserv member, himself an author
of one of the studies cited in the previous section, explained those
assumptions: “The reason you’re seeing all those negatives regarding games goes
back to the 1983 article by Clark. His meta-analysis resulted in the
often-misunderstood statement that you cannot learn from media. People often
misinterpret that statement to mean that if you expect people to learn, you
better not use media. What the statement really means is that media is
simply the delivery vehicle, and it is the instructional methods and strategies
built into the specific implementation of that medium that will determine
whether learning will occur. [Richard Wainess, Ph.D., Senior Educational
Researcher, CRESST/UCLA, National Center for Research on Evaluation, Standards,
and Student Testing, University of California, Los Angeles]
Importance
of Analysis & Instructional Design
The
bold-print information in all citations above reinforces what instructional
designers already know: people learn best when the training medium is
appropriate to the subject matter and they’re in control of the learning
process. Because learner-control is a key characteristic of Serious Games,
proven instructional best practices support empirical and anecdotal findings
that, for the types of subject matter appropriate to them, experiential Serious
Games are more effective than other media. That’s why at Sealund, we begin
every project with analysis of the client’s subject matter and required
learning outcomes. We want to ensure that each learning solution uses the
appropriate medium or blend of media for the project’s content and goals.
You
may find these resources useful for your own review of research:
o
Dubbels,
Brock R. (2008) Video games, reading, and transmedial comprehension. In
R. E. Ferdig (Ed.), Reference. Information Science Handbook
of research on effective electronic gaming in education.
Job Aid: Steps for Successful Deployment of
Serious Games
at Your Organization
8.
Build upon the foundation by incorporating
new Serious Games.
© Sealund & Associates Corporation