Forio's Forum

What makes for a successful simulation?  Turns out, the key ingredients are the same that they were 10 years.  Simulations work when they are:

• Fun
• Accessible
• Clear
• Educational

From one of the earliest posts on Forio’s Forum, read more about The Four Key Attributes of Successful Simulations:

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Most of our first-time clients have had limited experience using business simulations. The term ‘simulation’ gets applied to a lot of different applications. So, to clarify things, I’ll describe a couple successful simulations that I’ve worked on. I think it’s easier to understand how to do things right by looking at success stories. Then I’ll distill the attributes from these and other successful simulations I’ve worked on to the four key attributes of successful training simulations.

Case Example 1: Competitive Strategy and Change Management

A few years ago I worked with the CEO of a large capital equipment manufacturing company. The company was losing market share, mostly to new, international competitors. The CEO saw new opportunities emerging in foreign markets, but customers in these markets needed simpler and less expensive products than the company made. He realized the company needed to change the way it had done business for decades. To overcome their challenges and build market share, senior management developed a new and radically different vision for the company.

To communicate this new vision, the CEO organized a three-day offsite meeting for his top 100 managers. In addition to the usual presentations and seminars, managers ran a competitive strategy simulation during the three days. It was called a ‘corporate wargame’ because teams of managers would be simulating their real company and competing against each other. From the CEO’s perspective, the objective was to build an intuition and deep understanding among his management team of how the new vision and strategy could be implemented.

The 100 managers were divided into 10 competitors, 10 people per competitor. All teams competed in one simulated market. That meant that if one team gained market share, it would be at the expense of the other teams. The simulation matched the company’s actual market size and demand for different products and prices in regions around the world.

Schedule Constraints

This CEO had a lot of material that he wanted to cover in three days. Also he wanted this retreat to be upbeat and fun. Some of the participants hadn’t seen each other in years, so the group needed time for socializing and group outings.

In order to meet other scheduling constraints, the CEO decided that the simulation would be run before breakfast, at lunch, and then after dinner for each of the three days. The problem with this schedule was that people typically used this time to relax. We were worried that the managers would rebel or disengage from realistic play.

Competition and Fun

Fortunately, the teams became competitive very quickly. Usually these managers would meet for a beer at the hotel bar after dinner. At this retreat, they took their drinks into the conference rooms we had set up for each team and strategized how to beat their competitors. Because we had intentionally made the teams as diverse as possible, the players spent more time interacting with their colleagues in other countries.

The teams got so engaged that the CEO had to personally go from room to room around midnight on the second day to get them to stop playing. He was worried that they would stay up all night and wouldn’t be alert for the next day’s presentations.

As the simulation progressed, the teams got creative. Even though it wasn’t initially included in the simulation, two teams merged to make a larger and stronger competitor. Other teams discussed alliances and tried to gather competitive information. At lunch each day, we announced performance and distributed a newspaper with stories featuring interviews team members along with performance statistics and stock prices.

Competitive Strategy Simulation Summary

The simulation produced a vivid experience that managers remembered and discussed years later. Gold, Silver, and Bronze medals were awarded to the top three teams. As an indication of the level of player engagement, one of the members of the winning team was seen wearing his gold medal at breakfast the next morning as the buses prepared to drive everyone to the airport.

Because the simulation was competitive, participants were highly engaged and the managers learned the rationales and thinking behind the new company vision and strategy. Importantly, they built their intuition on how to best implement these new strategies. The simulation turned out to be a cost-effective way to communicate a lot of information in a short period of time. After the event was over, the managers were able take copies of the simulation with them use at their home offices to educate their own teams.

Case Example 2: AgLand

A few years ago, a crisis was developing for US farmers. A federal farm subsidy called the Conservation Reserve Program (CRP) was ending. Farmers, conservationists, and local citizens had different reasons to be concerned. In the short-run, the termination of the CRP would have a direct financial effect on farmers and, in the long-run, the economic vitality of the local communities, soil erosion, and water quality would all be affected.

Don Seville and I worked with Steven Taff from the University of Minnesota and Peter Buessler from the Minnesota Department of Natural Resources to create AgLand, a simulation that lets users test policies and see the impact on farms and the local environment.

What happened in the simulation

The simulation lets farmers and policy makers change roles and experience differing incentives and pressures. Because it takes many years to see the effect of environmental changes, the simulation runs through twenty years in an afternoon. That way, players could see the short- and long-term impacts of their decisions.

For the game, people who are farmers in real-life became policy makers. Before running the simulation, they would likely have espoused that they were free market advocates. Yet, in the debriefing, most participants came to realize how much their decisions had been driven by a goal of equalizing income. Most players ended up creating more government programs and attempted lots of quick fixes. They discovered that the policy tools they had available to them were blunt instruments.

People who played the role of farmers discovered that a system of government incentive programs could actually create a disincentive to adopting conservation practices. To quote one player, “We found ourselves farming the government or markets instead of the land.” In the debriefing, farmers complained they couldn’t achieve their goals because the government or market prices wouldn’t support them.

Typically the room became noisy as players moved about airing their opinions and making decisions. Facilitators encouraged conversation after each round of play, and periodically stop the game to discuss what’s going on.

Resource Constraints

AgLand is played in community centers and classrooms throughout the Midwest. Unlike most well-equipped corporate training centers, there’s no way that each participant will have a computer available. Often, they’re lucky to get one computer for twenty participants.

To get around this constraint, we focused the play of the simulation around a game board. The computer and printer are off to the side and, for each round of the simulation, players make decisions on the board by moving game pieces.

The game board is a map: on the map are four farms, a small town, a river with surrounding wetlands, and wildlife. An elected governing body known as the Policy Council is responsible for keeping citizens satisfied by influencing the management decisions of the farmers through policies.

The information from the game board is fed into the computer. The computer simulates one year forward, and the results are put back on the game board.

AgLand Summary

Although AgLand is a simulation, it is not a human-versus-machine game. Nor is it solvable. AgLand mimics the ever-changing dynamics of the real world. Its main objective is to help participants develop strategies for dealing with complex and ambiguous issues and then think about the consequences of their choices. They might then reconsider assumptions they may have about the driving forces behind their local economy and environment.

The Four Key Attributes of Successful Simulations

While AgLand and the competitive strategy ‘wargame’ are very different simulations, they share some similar attributes. These attributes are similar to other successful simulations. First, let’s review the main differences between these two examples.

• The audiences were vastly different. AgLand involved people with few shared goals or objectives while all the managers for the competitive strategy simulation worked for the same company.
• There were different constraints on the simulation. AgLand had technical resource constraints while the competitive strategy simulation had schedule constraints.
• The simulation run times were different. AgLand was over in a few hours while the competitive strategy simulation lasted a few days.

So what do these two cases have in common? It turns out that these two examples share four attributes that are part of all successful training simulations.

It’s useful to consider attributes that aren’t included on this list. One attribute that comes up frequently that isn’t on the list is realism. This doesn’t mean that realism, especially for some audiences, isn’t important. It just means that realism is only useful in how it affects these four attributes.

Importance of Simulation Realism

Realism can make a simulation more fun, clear, and educational. A realistic simulation can be more fun because it feels familiar and relevant. Realism also makes it clear to users what it is they are supposed to do. The simulation behaves in a way that is plausible and reasonable. Finally a realistic simulation can be more educational because it’s easier for participants to draw a correspondence between the simulation results and the results of their actual organizations.

But realism isn’t always beneficial. Realism can also be boring. It’s sometimes useful to tune-up the drama in a simulation in order to engage the player. Drama makes simulations more interesting in the same way novels involving extreme characters or plots can be more interesting.

F.A.C.E. Value

Imagine ranking a simulation with a 0 to 10 score for each of these four attributes. If the simulation fails for any one of the four attributes, then the whole simulation is a failure. Also a mediocre score on any of the four attributes results in a mediocre simulation.

You have to rank each attribute from the perspective of your audience. What is fun for Midwestern policy makers may not be fun for managers at a capital equipment manufacturer.

You can calculate a F.A.C.E. score by ranking each attribute on a scale of 0 to 10 and using the following formula:

All four attributes must be present in order for a simulation to be successful. Moreover, in order to create a really successful simulation, all four attributes should be as close to a perfect 10 as possible.

For example, a training simulation you are evaluating might get a score of:

The F.A.C.E. Score for this simulation would be (7 x 10 x 8 x 10) / 1000 = 5.6. It’s a good simulation, but you could nearly double its success by making it more engaging and doing additional usability testing.

So what can you do to make your simulation more successful? I’ll discuss each of these four attributes in detail in future articles, but for now, here’s a short list of features that can make simulations fun, accessible, clear, and educational.

Successful Simulations

Calculating a score based on four attributes is a useful check, but it’s a bottom-up approach. Ultimately, simulation success is determined by your users; so there’s also a simple top-down approach to measuring success.

The simplest measure of success for a simulation is: people enthusiastically use the simulation without being required to. They tell their friends about it because they think their friends will also enjoy it and benefit from it.

Successful simulations don’t require a lot of advocacy and persuasion to motivate potential users because people are internally motivated to play them. The use of the simulation spreads via word-of-mouth throughout your organization. Colleagues compete with each other and proudly discuss their successes with others. Eventually, a successful simulation can become ingrained as part of a shared corporate culture and history. If the simulation is designed from the beginning with success in mind, then the terminology, lessons, and experiences it provides become deeply ingrained in the corporate culture. The simulation will have had a deep impact on affecting the future success of the organization.

Few ideas, it seems, stand the test of time.  What was once ‘cutting edge’ can within a few years seem stale and dated.  Or worse, it can be just plain wrong.  So going back to the early days of Forio’s Forum is a bit scary.  But here’s an article that still has relevance.  The references are a bit dated, but the message is as valid as ever.  [Originally published in May, 2001]

++++++++++++++++++++++++++++++++++++++

I was happy to learn that the Iridium network was relaunched under the new company Iridium Satellite. In the 1980s and through 1990s Iridium was a big idea. The scale and risk of Iridium made it an exciting venture: 66 satellites at a cost of $5 billion. Just a few years later, Iridium Satellite paid only $25 million for the entire network—discounted more than 99% from the original investment.

To me the failure of the original Iridium is interesting because it was technically successful, but a commercial failure. Not enough people subscribed to the $3,000 phones and the expensive service. What seemed like a good idea in 1985, performed poorly 15 years later.

In general, I don’t like criticisms of failed businesses, especially when those businesses are attempting to do something new and innovative. Looking backward, reasons for failure are often obvious, but at the time decisions are made, it’s difficult to know how things will turn out.

Avoiding risk when making decisions that are difficult to retract (as opposed to reviewing risk factors after the fact) is why business prototyping is so important. I first saw the word prototyping used to describe business decision-making in Michael Schrage’s excellent book, Serious Play. Prototyping, a word usually reserved for industrial design, just means trying a few versions of something before making a decision. Prototyping occurs all the time in businesses, often with spreadsheets and sometimes through simulation.

It’s almost always a good idea to prototype decisions if you’re attempting something new. And this applies not only to business; it also applies to personal decisions. Prototypes don’t have to be a computer simulation. It just has to give you a feel for what life will be like after you’ve made your decision.

The realism of the prototype you create should vary according the complexity and consequences of the decision you’re making. The reason for this is that what you learn from a prototype changes significantly as that prototype realism changes.

Successful Personal Prototypes

I recently observed a demonstration of how prototype realism alters how and what you learn. A friend of mine moved from California to Virginia. Changing states meant he had to get a new license plate for his car and he thought it would be fun to get a vanity plate, with his own custom combination of letters and numbers for the license.

Choosing a vanity license plate is a little like getting a tattoo—once you’ve made your decision you’re committed. Changing or eliminating the thing is going to be difficult and expensive. OK, I admit that removing the tattoo is more difficult, but it’s still an example of a decision that you are far better off making the right decision the first time.

Occasionally I would get emails asking my opinion about particular choices he was considering. The original list consisted of the following names:

WONTO
7-GRAIN
INQUIRY
I ASKWHY
IMNAKED
HUGTREE

Now, most people would have just picked a name at this point. But my friend realized that whatever he chose would be associated with his car for a long time. So he took it a step further, to get a better understanding of what is car would look like with the new plate.

Without violating the law, you can’t create a fake license and drive around for a while to see how you like it. But fortunately Virginia has a web site where you can check the availability of words you are considering for your plate and prototype your license plate. For example, 7-GRAIN (one of the names he considered) would look like this:

Now that is a lot better than just looking at a name on a list. For example, you can see that the name won’t be exactly centered. It’s shifted a little over to the left. This is one of the many little things that might affect his decision.

While the license plate prototyping offered at the Virginia web site is helpful, it’s actually possible to take it one step further. My friend got a digitized photo of the back of his car and pasted the new license images on the back. The result looked like this:

Looking at an image of his car with the plate makes it a lot more obvious what the final result will look like. Compared with the original list of names, it’s much more of a simulated experience of what the future will be like with these new plates. Although picking a license plate ultimately isn’t a critical decision, still spending thirty minutes on a photo editing program possibly saved hours of regret later. And it illustrates why the more realistic your prototyping medium is, the more likely you are to make a good decision. For my friend, increasing the realism (by using photos instead of words) didn’t so much build his understanding as it did build his intuition.

Prototyping Suggestions

Doing some kind of prototyping in business, even if your prototype is just a simple spreadsheet, is far better than doing nothing at all. Unless your decision is trivial, spending a few hours or even a few minutes in front of Excel could save you money and problems in the future.

Depending on how important and retractable your decision is, you can vary the sophistication of your prototype. For example, for a simple investment decision with limited consequences, an Excel spreadsheet is probably a good choice. Spreadsheets produce results that are like my friend’s original written list of license plate names. Spreadsheets prototypes give you the dry facts without much realism as to what life will be like after you’ve made the decision, but a decent spreadsheet can often be pulled together in a few hours. So there’s a tradeoff between effort and results.

For decisions with big consequences it’s usually worth the effort of producing a realistic prototype of your decision on the computer. Not only will it help you understand the consequences of the decision, but you can also show your prototype to colleagues. Because a good prototype is a simulated experience, a prototype can communicate alternatives in ways that are hard for words and even Powerpoint presentations. This is why change management training simulations are a great way to explain a new strategy to help others understand its implications.

Of course, the ultimate prototype is reality. Businesses managers, reporters, and other satellite communications companies learned a lot from Iridium’s daring experiment. But the experience was far too costly for Iridium’s investors.

Research that Examines the Effectiveness of Simulations

There is plenty of anecdotal evidence that simulations are effective teaching tools. For example, in How to Assess Performance, Learning, & Perceptions in Organizations, Swanson and Holton discuss how learning activities that resemble real business circumstances foster better transfer of learning. Peter Senge, author of The Fifth Discipline believes that human beings learn best from experience, particularly when feedback from actions is rapid and unambiguous.

In their book Experiential Management Development: From Learning to Practice, Hoberman and Mailick provide a number of benefits of business simulations including:

• highly motivated and involved students
• improved ability to connect learning to real-world situations
• freedom to experiment with new behaviors in a risk-free environment
• opportunity for immediate feedback from decisions
• enhanced ability to teach teamwork and leadership

Experts who cite anecdotal evidence generally assert that students learn more effectively because students find simulations engaging. Students expend more effort when using simulations and more persistently pursue simulation goals because

• simulations are enjoyable to play, interesting, and build confidence (that is, they are fun)
• games involve iteratively playing through analysis-decision-result cycles that provide instant and accurate assessment of performance throughout the exercise

However much of this evidence relies on post-simulation student evaluations and not on actual performance or learning data.

What the Research Shows

Fortunately, there have been some carefully designed research studies that examine the effectiveness of simulations and test these anecdotal assertions. In their paper, Developing Managerial Effectiveness: Assessing and Comparing the Impact of Development Programs Using a Management Simulation or a Management Game, John Kenworthy and Annie Wong tested 100 students using one of the following learning tools:

• a business game
• a business simulation
• a case study

For their experiment, Kenworthy and Wong distinguished between a game and a simulation by level of realism. For example, the simulation used data from real businesses and the game did not. The students who used the case study were part of the control group. By testing students after they ran a simulation, game, or used a case study, the researchers could compare the relative effectiveness of each approach.

Kenworthy and Wong also collected demographic data about students such as age, gender, and work experience to see if there any connections between the students’ backgrounds and their ability to learn from a simulation, game, or case study.

Their research found that:

• simulations and games are more effective at transferring learning to students than case studies
• younger managers who have used computer games since early childhood enjoy simulations and games more than case studies; they also learned more from simulations and games
• Senior managers over forty years old prefer simulations that use real industry data over games that used fabricated data
• Students with non-convergent learning styles enjoy simulations more than those with convergent learning styles

Other studies such as An Assessment of the Effectiveness of Simulation as an Instructional System in Food Service a study of food service industry managers, also concluded that simulations improve learning. And like Kenworthy and Wong, this study found that younger and less experienced managers learned the most from simulation exercises.

Effectiveness of Simulations as a Performance Evaluation Tool

Often simulations are used as an exercise for students to apply what they have learned during a lecture. Researchers Anderson and Lawton in their study, Is Simulation Performance Related to Application? found a strong relationship between the number of concepts students used in a marketing class and their performance in a simulation.

They concluded that

• simulations are effective at getting students to apply concepts that they have learned through lectures or reading
• simulations can be used for testing because they can evaluate student comprehension of key concepts taught throughout a course

Other studies have found that, measured by their grade on a final exam, students who played simulations during a course performed significantly better than those who did not.

Other Important Findings

Not surprisingly, research into the training effectiveness of simulations and games has found that the material and instruction provided with a simulation influences its effectiveness as a training tool. For example:

• Several studies have found that skilled instruction is critical to the success of a simulation especially during the early stages when teams are formed and during the debrief immediately following a simulation exercise to ensure that summary insights are acquired from the experience.
• Other studies have found that simulations need the support of ancillary material such as online help, student manuals, tooltips, and other activities for learning to be effective.

Do you know of other research that measures the effectiveness of simulations and games? Post it in the comments below.

The basic concept behind supply chain management is simple: customers order products from you; you keep track of what you’re selling, and you order enough raw materials from your suppliers to meet your customers’ demand. So why is it that, in a recent article, the Economist claimed that, “Managing a supply chain is becoming a bit like rocket science?”

The problem turns out to be one of coordination. Suppliers, manufacturers, sales people, and customers have their own, often incomplete, understanding of what real demand is. Each group has control over only a part of the supply chain, but each group can influence the entire chain by ordering too much or too little. Further, each group is influenced by decisions that others are making.

This lack of coordination coupled with the ability to influence while being influenced by others leads to what Stanford’s Hau Lee refers to as the Bullwhip Effect. Decisions made by groups along the supply actually worsen shortages and overstocks.

The bullwhip effect is illustrated by a story Prof. Lee tells about how Volvo found itself with extra inventories of green cars. To get them off the dealers’ lots, Volvo’s sales department offered special deals, so demand for green cars increased. Production, unaware of the promotion, saw the increase in sales and ramped up production of green cars.

Cisco faced a similar problem last year that resulted in a $2.2 billion inventory write-down. Only a few months before the write-down, Cisco wasn’t able to get its products to customers quickly enough. Quoting a supplier to Cisco interviewed in CIO Magazine, “People see a shortage and intuitively they forecast higher. Salespeople don’t want to be caught without supply, so they make sure they have supply by forecasting more sales than they expect. Procurement needs 100 of a part, but they know if they ask for 100, they’ll get 80. So they ask for 120 to get 100.”

Delays Wreak Havoc

But coordination isn’t just about communication. Even in supply chains where communication is perfect, manufacturing and procurement delays can wreak havoc. That’s because while customers are asking for increased orders, backlogs are building, and it is oh-so-easy to confuse backlogged orders with increases in demand.

Thousands have felt the frustration of supply chain management in a simulation developed at MIT’s Sloan School of Management called the beer game. The simulation is run as a board game in teams playing the roles of retailers, wholesalers, distributors, and brewers of beer. As the backlog for orders increase, players order too much inventory, forcing their teammates into severe backlogs further down the supply chain. The game can be emotionally intense. John Sterman, Director of MIT’s System Dynamics Group writes, “During the game emotions run high. Many players report feelings of frustration and helplessness. Many blame their teammates for their problems; occasionally heated arguments break out.”

The Near Beer Game Simulation

You can try a version of the beer game called the Near Beer Game. It’s called the Near Beer Game because, although it’s not identical to the original beer game, it teaches many of the same lessons. It also teaches one extra lesson not in the original game: even with perfect information, even when there are no breakdowns in communication, you’ll still feel the bullwhip effect due to procurement and manufacturing delays.

Here’s how the Near Beer Game works: at the beginning of the simulation your supply chain is in perfect equilibrium. Customers are ordering ten cases of beer each week, you have ten cases in inventory, ten cases are brewing, and ten cases worth of raw materials are arriving from your vendors. In week two, demand increases to fifteen cases per week and remains at fifteen cases for the remainder of the simulation. The game ends when you manage to get your supply chain back in equilibrium for fifteen cases of beer.

Sounds easy right? Try it out and see how many weeks it takes you. See if you can bring the supply chain back into equilibrium without the bullwhip oscillations of stock-outs followed by over-supply.

How to Reduce the Bullwhip Effect

One way to reduce the bullwhip effect is through better information, either in the form of improved communication along the supply chain or (presumably) better forecasts. Because managers realize that end-user demand is more predictable than the demand experienced by factories, they attempt to ignore signals being sent through the supply chain and instead focus on the end-user demand. This approach ignores day-to-day fluctuations in favor of running level.

Another solution is to reduce or eliminate the delays along the supply chain. In both real supply chains and simulations of supply chains, cutting order-to-delivery time by half can cut supply chain fluctuations by 80%. In addition to savings from reduced inventory carry costs, operating costs also decline because less capacity is needed to handle extreme demand fluctuations.

In addition to cycle time reductions throughout the supply chain, Hau Lee, V. Padmanabhan, and Seungjin Whang recommend the following actions to reduce the supply chain management bullwhip effect:

1. Focus on end-user demand through point-of-sale (POS) data collection, electronic data interchange (EDI), and vendor-managed inventories (VMI) to reduce distortions in downstream communication.
2. Work with vendors to create smaller order increments and reduce order batching. Order batching exacerbates demand fluctuations.
3. Maintain stable prices for products. Price fluctuations encourage customers to over-purchase when prices are low and cut back on orders when prices are high, leading to large demand fluctuations.
4. Allocate demand among customers based on past orders, not present orders to reduce hoarding behavior when shortages occur.

Additional reading:

• “The Bullwhip Effect in Supply Chains,” Wikipedia
• The Beer Game, Prof. John Sterman, MIT Sloan School of Management
• “Chain Reaction: Managing a Supply Chain is Becoming a Bit Like Rocket Science,” The Economist
• “What Went Wrong at Cisco,” CIO Magazine