Research that Examines the Effectiveness of Simulations

There is plenty of anecdotal evidence that simulations are effective teaching tools. For example, in How to Assess Performance, Learning, & Perceptions in Organizations, Swanson and Holton discuss how learning activities that resemble real business circumstances foster better transfer of learning. Peter Senge, author of The Fifth Discipline, believes that human beings learn best from experience, particularly when feedback from actions is rapid and unambiguous.

In their book Experiential Management Development: From Learning to Practice, Hoberman and Mailick provide a number of benefits of business simulations including:

• highly motivated and involved students
• improved ability to connect learning to real-world situations
• freedom to experiment with new behaviors in a risk-free environment
• opportunity for immediate feedback from decisions
• enhanced ability to teach teamwork and leadership

Experts who cite anecdotal evidence generally assert that students learn more effectively because students find simulations engaging. Students expend more effort when using simulations and more persistently pursue simulation goals because

• simulations are enjoyable to play, interesting, and build confidence (that is, they are fun)
• games involve iteratively playing through analysis-decision-result cycles that provide instant and accurate assessment of performance throughout the exercise

However, much of this evidence relies on post-simulation student evaluations and not on actual performance or learning data.

What the Research Shows

Fortunately, there have been some carefully designed research studies that examine the effectiveness of simulations and test these anecdotal assertions. In their paper, Developing Managerial Effectiveness: Assessing and Comparing the Impact of Development Programs Using a Management Simulation or a Management Game, John Kenworthy and Annie Wong tested 100 students using one of the following learning tools:

• a business game
• a business simulation
• a case study

For their experiment, Kenworthy and Wong distinguished between a game and a simulation by level of realism. For example, the simulation used data from real businesses and the game did not. The students who used the case study were part of the control group. By testing students after they ran a simulation, game, or used a case study, the researchers could compare the relative effectiveness of each approach.

Kenworthy and Wong also collected demographic data about students such as age, gender, and work experience to see if there any connections between the students’ backgrounds and their ability to learn from a simulation, game, or case study.

Their research found that:

• simulations and games are more effective at transferring learning to students than case studies
• younger managers who have used computer games since early childhood enjoy simulations and games more than case studies; they also learned more from simulations and games
• senior managers over forty years old prefer simulations that use real industry data over games that used fabricated data
• students with non-convergent learning styles enjoy simulations more than those with convergent learning styles

Other studies such as An Assessment of the Effectiveness of Simulation as an Instructional System in Food Service, a study of food service industry managers, also concluded that simulations improve learning. And like Kenworthy and Wong, this study found that younger and less experienced managers learned the most from simulation exercises.

Effectiveness of Simulations as a Performance Evaluation Tool

Often simulations are used as an exercise for students to apply what they have learned during a lecture. Researchers Anderson and Lawton in their study, Is Simulation Performance Related to Application?, found a strong relationship between the number of concepts students used in a marketing class and their performance in a simulation.

They concluded that

• simulations are effective at getting students to apply concepts that they have learned through lectures or reading
• simulations can be used for testing because they can evaluate student comprehension of key concepts taught throughout a course

Other studies have found that, measured by their grade on a final exam, students who played simulations during a course performed significantly better than those who did not.

Other Important Findings

Not surprisingly, research into the training effectiveness of simulations and games has found that the material and instruction provided with a simulation influences its effectiveness as a training tool. For example:

• Several studies have found that skilled instruction is critical to the success of a simulation especially during the early stages when teams are formed and during the debrief immediately following a simulation exercise to ensure that summary insights are acquired from the experience.
• Other studies have found that simulations need the support of ancillary material such as online help, student manuals, tooltips, and other activities for learning to be effective.