I'm Serious.net

Assessment and Evaluation

There seems to be a growing impression among the serious game design community that Assessment and Evaluation are interchangeable terms. They are not.

Assessment applies to learners/players. Before or as a learner is introduced to an intervention (in our case, a serious game), s/he should be given an opportunity to determine whether s/he is really "ready" to move ahead. By "ready," we mean that s/he has the skills and/or knowledge necessary to proceed successfully. Designing the activities that offer this pre-assessment ("pre-" because it occurs prior to beginning the game) is an important part of the serious game designer's job; the activities themselves should, in form and format, reflect the game itself, and should offer appropriate feedback and remediation to the player who needs additional instruction or assistance.

At the conclusion of the game experience, learners/players are given another opportunity to demonstrate skills and/or knowledge, this time focusing on the intended outcome of the game. We call this "post-assessment" for obvious reasons.

Evaluation, on the other hand, looks at the overall effectiveness of the intervention. During the evaluation phase, we ask ourselves questions like: Does the game achieve its goals? Are learners learning what they're supposed to? Are they motivated by their experience? Are the actual costs of development in line with projections? Was the project completed on time?

As with assessment, evaluation is not just a post-project implementation activity. Evaluation is a critical alpha and beta activity as well, allowing designers to tweak and improve game design and other program implementation elements prior to full deployment.

In the most stark terms possible, then, assessment measures the progress and achievement of learners (them). Evaluation measures the effectiveness of the design and its implementation - in other words, us.

I'm serious,
Anne

Google’s got game

Have you played Google Image Labeler yet? It's Google's clever answer to improving image search.

Here's how it works, in Google's words:

You'll be randomly paired with a partner who's online and using the feature. Over a two-minute period, you and your partner will be shown the same set of images and asked to provide as many labels as possible to describe each image you see. When your label matches your partner's label, you'll earn points depending on how specific your label is. You'll be shown more images until time runs out. After time expires, you can explore the images you've seen and the websites where those images were found. And we'll show you the points you've earned throughout the session.

Why do I like it? It's fun, fast, competitive and compelling. It's easy (although it did take me a few tries to "get it"). I'm interacting with random people from around the world. Even while I'm competing, I'm collaborating. I'm learning something - in this case, I'm refining how I think about tags. What's not to like?

I'm serious,
Anne

Interaction Strategies can aid Transference

I'm a fretter.

Near the top of my "List of Things to Fret Over" is the issue of proper integration of a SG into the rest of the players' learning AND performance regimen. Has the purpose and seriousness of the game been conveyed to the players? Have players been given the opportunity to develop the requisite skills and knowledge to play the game successfully? And, most anxiety-producing of all, are players going to know how to transfer what they've learned in the game back into their real lives?? It's this question of transference, after all, where the rubber hits the road. This is what you were actually promising your client/sponsor when you convinced them to fund the project at the start.

Rather than continuing to toss and turn, I'm re-reading The Handbook of Blended Learning. Like Jay Cross, author of one of the book's forwards, I have dismissed the concept of blended learning as superfluous. I mean, duh. But in fact, blending the SG into the rest of the learning program is precisely what I'm fussing over. I'm glad I picked this book up again.

I'm particularly glad to come back to Ellen Wagner's, "On Designing Interaction Experiences for Blended Learning." Interactions are "reciprocal events that require at least two objects and two actions," says Wagner. "[They] occur when the objects and events mutually influence one another." (We'll come back to this another time to discuss the implications this has on persistence.)

As leader of a design team in the 90s (now she's Sr. Director, eLearning Solutions at Adobe Systems), Wagner came to view interaction "less as a theoretical construct and more as a variable..., a strategy for achieving specific learning or performance outcomes." She has since honed her list of interaction experiences into:

- Interaction for participation
- Interaction for communication
- Interaction for feedback
- Interaction for elaboration
- Interaction for learner control and self-regulation
- Interaction for motivation
- Interaction for negotiation
- Interaction for team building
- Interaction for discovery
- Interaction for exploration
- Interaction for clarification

Wagner, of course, elaborates on each of these. But, what I like about this list (to which I would humbly add "Interaction for social norming through the use of social rating systems") is that it helps us organize our thinking about what kinds of interactions support which kinds of outcomes.

And THAT helps us be not just better SG designers, but helps us grapple more easily with what's required to find the holy grail of transference. After all, when the focus of my SG is team-building among compliance officers and your SG is intended to introduce new hires to project finance, the interaction strategies we use to assist our players/learners in the transference process are likely to be quite different.

There. Maybe I can sleep tonight.

I'm serious,
Anne

What do South Koreans know that we don’t?

Two items from the New York Times caught my eye.

This past weekend, Blizzard's 2007 Worldwide Invitational "Starcraft" tournament was held in Seoul, with thousands of fans cramming into the Olympic Gymnastics Arena. Seth Schiesel writes about this event and the gaming craze that grips South Korea in today's "To the Glee of South Korean Fans, a Game's Sequel is Announced." E-sports (what South Koreans call competitive gaming), he notes, has not just millions of fans, but millions of participants. Just imagine the enthusiasm in the arena yesterday when Blizzard announced its sequel to Starcraft.

"Why Work Is Looking More Like a Video Game" doesn't offer any new information to us in the field, but gives a nice glimpse into how a number of companies are using games to their advantage. Author Michael Fitzgerald points out, by the examples he uses, the power and importance of social networking to a serious game's success. It's the collaboration that results from the networking that facilitates learning transference back to the job.

So we have e-sports mania in South Korea contrasted with just a dawning awareness among the world's business leaders of the power of games for learning, for innovation, for team-building, for on-boarding... The good news is that with attention from the likes of the New York Times, serious games are getting a serious look by more and more decision-makers.

I'm serious,
Anne

Wii, Haptic Computing, and SGs

Today's San Francisco Chronicle included an article entitled, "Wii needs more games." Game developers misread the market, says author Ryan Kim, when this latest generation of game consoles was released, initially leaping on the PS3 and XBox bandwagons. Now, Wii sales are outpacing its competitors', and developers are starting to leap its way. In addition to a backlog in console demand, there is a shortage of titles that really show off Wii's aerobic potential.

At the same time, people like Ovidiu Ciobanu are pushing the envelope of dental education with smart, sophisticated uses of interactive technology and haptic computing. Technology is a few years out for convenient and cost-effective haptic devices, but what if Dr. Ciobanu had a $250 Wii in his lab in the meantime?

And if we broaden our minds a bit, aren't there scores of ways serious game designers could incorporate motion into our game mechanics? Certainly in the accomplishment of manual skills. But what would the impact be on learner-learner interactions if players could simulate touching each other? In conflict? On a first meeting? At a family gathering? Might this have an impact on learner-generated content?

I don't know the answers to any of these questions, but I sure would like to. I'll let you know as I find out. And please tell me of your experiences.

I'm serious,
Anne

The Power of Virtual Communities

Charles Babcock, Information Week, recently wrote a piece, "What's the Greatest Web Software Ever Written?". Great article. After nominating World of Warcraft to #9 (of 12 - and it's heady company), he explains:

"...[M]uch has been made of a three-dimensional virtual world known as Second Life. For my money, more significant examples of 3-D virtual reality can be found in massively multiplayer online games, which offer playful, real-time activity where the actions of one participant affect another. The possibilities in terms of training people to complete complicated team-oriented tasks seem obvious. The godfather of this genre is Blizzard Entertainment's World of Warcraft. Introduced in 2004, World of Warcraft's 8.5 million rabid fans--3.5 million in China alone--put Second Life's 6 million registered residents in the shade."

Serious Definitions redux

Heard from a reader this morning (this is also on the thread from yesterday's post) with an interesting perspective and a revision to the definition I put on the table. This reader's concern is that I over-emphasize the online and multi-player aspects, and ignore other genres that influence our work. So this is what (s)he proposes instead:

Serious Games are modelled on many video game genres including MMOG, RPG, RTS, TBS, FPS, ARG, simulators and others. Entertainment is not their sole purpose and this differentiates them from conventional video games. Serious Games have the serious purpose to change, in a pre-defined way, the beliefs, skills and/or behaviours of the individuals and cohort groups who play the game. Serious games aim to provide a deep level of user interaction that conventional elearning and marketing fails to do.

Other thoughts?

Rant: Serious Definitions

The eLearning Guild issued a research report in February entitled, Immersive Learning Simulations. The first ten pages of the report are devoted to defining the term "serious game," and then rejecting the term itself in favor of one the authors hope will be more acceptable to the eLearning Guild membership: "immersive learning simulation." And hence the report title.

What definition did they arrive at?

"An optimized blend of simulation, game element, and pedagogy that leads to the student being motivated by, and immersed into, the purpose and goals of a learning interaction. Serious Games use meaningful contextualization and optimized experience to successfully integrate the addictive nature of well-designed games with serious learning goals."

If they want to call that Immersive Learning Simulation, they have my vote. Because that definition misses the essence of the power of serious games. Where's the inter-player interaction? Where's the cohort-building? Where's the strengthening/altering of belief systems? Where's the communication and community? In short, where's the multi-player component? Nevermind learner-generated content... (Plus, I'm not fond of the implication that I, a player and a designer, engage in and/or promote addictive behavior.)

I'm no happier with Wikipedia's effort:

"Serious games (SGs) or persuasive games are computer and video games used as persuasion technology or educational technology. They can be similar to educational games, but are often intended for an audience outside of primary or secondary education. Serious games can be of any genre and many of them can be considered a kind of edutainment.
"A serious game may be a simulation which has the look and feel of a game, but corresponds to non-game events or processes, including business operations and military operations. The games are intended to provide an engaging, self-reinforcing context in which to motivate and educate the players. Other purposes for such games include marketing and advertisement."

Again, really misses the essence, I fear. And there are a LOT of other definitions floating about, each remiss in its special way.

I have an opinion about this, believe it or not. I offer my own definition:

Modeled on MMOGs, Serious Games have the serious purpose to change, in a pre-defined way, the beliefs, skills and/or behaviors of the individuals and cohort groups who play the game.

Thoughts? Responses?

I'm serious,
Anne

Workgroup Tools

We all have challenges keeping our teams on track with reviews and changes. Making sure client sign-offs are clean and efficient is particularly important in the serious game design/development process, given the number of "moving parts".

If your team doesn't have a good handle on doing these things well, check out April's issue of HOW Magazine. The technology review looks at five workgroup collaboration/project management tools: Adobe Acrobat 8 Professional, Basecamp, Proof-it-Online, Robohead, and Workgroups 2006.

Cobbler’s Children Get Shoes!

Accessibility is, or ought to be, a very important topic for serious game designers. Accessibility issues can be left in the dust, though, in the excitement of developing game play.

To help us all address accessibility in our products, the Human-Computer Interaction Lab in Heraklion, Crete have put together a game for us (! how often do we get to design games for ourselves?), called Game Over! Actually, it's a series of game episodes, each one highlighting a different accessibility issue through non-example in game play. It's clever, and effective - and it's in English.

Many thanks to Mindy Jackson at enspire learning for bringing this to our attention!

I'm serious,
Anne