Proceedings of the 7th European Conference on Games Based Learning ECGBL 2013 Volume 2 by ACPIL

Proceedings of the 7th European Conference on Games-Based Learning Instituto Superior de Engenharia do Porto, Portugal 3-4 October 2013

Volume Two

Edited by Dr Carlos Vaz de Carvalho and Dr Paula Escudeiro Instituto Superior de Engenharia do Porto Portugal A conference managed by ACPI, UK www.academic-conferences.org

The Proceedings of The 7th European Conference on Games Based Learning Hosted by Instituto Superior de Engenharia do Porto (ISEP) Portugal Volume Two 3-4 October 2013 Edited by Dr Paula Escudeiro Programme Chair and Carlos Vaz de Carvalho Conference Chair

Copyright The Authors, 2013. All Rights Reserved. No reproduction, copy or transmission may be made without written permission from the individual authors. Papers have been double-blind peer reviewed before final submission to the conference. Initially, paper abstracts were read and selected by the conference panel for submission as possible papers for the conference. Many thanks to the reviewers who helped ensure the quality of the full papers. These Conference Proceedings have been submitted to Thomson ISI for indexing. Please note that the process of indexing can take up to a year to complete. Further copies of this book and previous yearâ&#x20AC;&#x2122;s proceedings can be purchased from http://academic-bookshop.com E-Book ISBN: 978-1-909507-65-4 E-Book ISSN: 2049-100X Book version ISBN: 978-1-909507-63-0 Book Version ISSN: 2049-0992 CD Version ISBN: 978-1-909507-66-1 CD Version ISSN: 2049-1018 The Electronic version of the Proceedings is available to download at ISSUU.com. You will need to sign up to become an ISSUU user (no cost involved) and follow the link to http://issuu.com Published by Academic Conferences and Publishing International Limited Reading UK 44-118-972-4148 www.academic-publishing.org

Contents Paper Title

Author(s)

Page No.

Preface

vii

Committee

viii

Biographys

Volume One Teachers’ Beliefs About Game Based Learning: A Comparative Study of Pedagogy, Curriculum and Practice in Italy, Turkey and the UK

Yasemin Allsop, Ebru Yeniman Yildirim and Marina Screpanti

Using Gamification to Animate a Virtual Community

António Andrade and Carlos Vaz de Carvalho

SIMaging the CITY: The Educational use of Simulation Video Games for Disadvantaged Youth

Massimiliano Andreoletti and Gianna Cappello

“The Chest That Longs to be Moved”: A Serious Game for the Greek Muslim Minority Children

Alexandra Androussou, Evangelia Kourti and Nelly Askouni

Transformational Play; Using 3D Game-Based Narratives to Immerse Students in Literacy Learning

Anna Arici and Sasha Barab

Approaches to Collaborative Game-Making for Fostering 21st Century Skills

Susan Bermingham, Nathalie Charlier, Francesca Dagnino, James Duggan, Jeffrey Earp, Kristian Kiili, Evelien Luts, Lien van der Stock and Nicola Whitton

Best Practices for Deploying Digital Games for Personal Empowerment and Social Inclusion

Lizzy Bleumers, Ilse Mariën, Jan Van Looy, James Stewart, Dana Schurmans and Anissa All

Investigating the Relationship Between School Performance and the Abilities to Play Mind Games

Rosa Maria Bottino, Michela Ott and Mauro Tavella

Experience With Digital Game-Based Embodied Learning: The Road to Create a Framework for Physically Interactive Digital Games

Carsten Busch, Florian Conrad, Robert Meyer and Martin Steinicke

Toward Improvement of Serious Game Reliability

Thibault Carron, Fabrice Kordon, Jean-Marc Labat, Isabelle Mounier and Amel Yessad

The Effects of Gamification on Student Attendance and Team Performance in a Third-Year Undergraduate Game Production Module

Hope Caton and Darrel Greenhill

Game-Based Learning in Health Sciences Education

Nathalie Charlier, Evelien Luts and Lien Van Der Stock

Specification and Design of a Generalized Assessment Engine for GBL Applications

Yaëlle Chaudy, Thomas Connolly and Thomas Hainey

105

Safer Internet: Enhancing Good Practices on the Internet Through Games Based Learning for Greek Elementary School Students

Vasiliki Choleva, Loukas Koutsikos Simeon Zourelidis, Vlassios Filis, Dimitris Metafas and Charalampos Patrikakis

115

Using Game Mechanics to Measure What Students Learn

Jill Denner, Linda Werner, Shannon Campe and Eloy Ortiz

123

Combining Game Based Learning With Content and Language Integrated Learning Approaches: A Case Study Utilizing QR Codes and Google Earth in a Geography-Based Game

Kyriaki Dourda, Tharrenos Bratitsis, Eleni Griva and Penelope Papadopoulou

130

Paper Title

Author(s)

Page No.

The Design and Evaluation of a Multiplayer Serious Game for Pharmacy Students

Maciej Dudzinski , Darrel Greenhill , Reem Kayyali , Shereen Nabhani , Nada Philip , Hope Caton , Sonya Ishtiaq and Francis Gatsinzi

140

Cheating and Creativity in Pervasive Games in Learning Contexts

Stine Ejsing-Duun, Thorkild Hanghøj and Helle Skovbjerg Karoff

149

Supporting Teachers in the Process of Adoption of Game Based Learning Pedagogy

Valérie Emin-Martinez and Muriel Ney

156

Cognitive Walkthrough for Learning Through Game Mechanics

David Farrell and David Moffat

163

Global Math: Development of Online Platform for Mathematical Thinking Games

Toru Fujimoto, Keiichi Nishimura, Kaoru Takahashi, Masahiro Yachi, Kiyoshi Takahashi and Yuhei Yamauchi

172

What Can Play Theory Tell us About Computer Games for Young Children?

Georgy Gerkushenko and Svetlana Sokolova

179

Role Game Playing as a Platform for Creative and Collaborative Learning

Lisa Gjedde

190

Development and Evaluation of a Generic E-CLIL Web2.0 Games Engine

Thomas Hainey and Thomas Connolly

198

Designing Games to Disseminate Research Findings

Claire Hamshire, Rachel Forsyth and Nicola Whitton

208

Facilitating Teacher Students’ Innovation Competence through Problem-Based Game Design Processes

Thorkild Hanghøj and Sia Hovmand Sørensen

216

Deploying Serious Games for Management in Higher Education: Lessons Learned and Good Practices

Jannicke Baalsrud Hauge, Francesco Bellotti, Rob Nadolski, Michael Kickmeier-Rust, Riccardo Berta and Maira Carvalho

225

Neuroeducational Research in the Design and use of Games-Based Teaching

Wayne Holmes, Paul Howard-Jones, Erico Tanimoto, Carol Jones, Skevi Demetriou, Owen Morgan, Philip Perkins and Neil Davies

235

Playing and Learning: An iPad Game Development Case Study

Jennifer Jenson and Rachel Muehrer

244

An Overview of Game Console Motion Sensor Technologies Exploited for Education

Marina Kandroudi and Tharrenos Bratitsis

252

Picking the Right Interface for Engaging Physical Activity Into Game Based Learning

Helle Skovbjerg Karoff, Gunver Majgaard, Lars Elbæk and Mona Have Sørensen

261

Playing and Gaming – Studied in an Informal Learning Setting

Helle Skovbjerg Karoff, Stine Ejsing-Duun and Thorkild Hanghøj

268

Game Based Learning in Mathematics: Teachers' Support by a Flexible Tool

Aikaterini Katmada, Apostolos Mavridis, Thrasyvoulos Tsiatsos

275

Learning Analytics with Games Based Learning

Harri Ketamo

284

Gamification and Intelligent Feedback Mechanisms for a Division Learning Tool

Michael Kickmeier-Rust and Dietrich Albert

290

Developing Games for Health Impact: Case Brains vs Zombies

Kristian Kiili, Manuel Ninaus, Mikko Koskela, M Tuomi and Antero Lindstedt

297

Meleon - a Casual Mobile Game Supporting Immersion and Reflection in Learning

Luise Klein

305

Paper Title

Author(s)

Page No.

The Literature Race - NFC Based Mixed Reality Game

Antti Koivisto, Harri Ketamo, Eero Hammais and Juho Salli

314

Bringing Game Achievements and Community Achievements Together

Johannes Konert, Nico Gerwien, Stefan Göbel and Ralf Steinmetz

319

Modeling the Player, Learner and Personality: Independency of the Models of Bartle, Kolb and NEO-FFI (Big5) and the Implications for Game Based Learning

Johannes Konert, Stefan Göbel and Ralf Steinmetz

329

Raising Awareness on Archaeology: A Multiplayer Game-Based Approach With Mixed Reality

Mathieu Loiseau, Élise Lavoué, Jean-Charles Marty and Sébastien George

336

Scientific Discovery Games for Authentic Science Education

Rikke Magnussen, Sidse Damgaard Hansen, Tilo Planke, Jacob Friis Sherson

344

Creating Games in the Classroom – From Native Gamers to Reflective Designers

Gunver Majgaard

352

A Holistic Framework for the Development of an Educational Game Aiming to Teach Computer Programming

Christos Malliarakis, Maya Satratzemi and Stelios Xinogalos

359

Examining Early Childhood Education Students’ Attitudes Toward Educational Computer Games in Kindergarten

Dionissios Manessis

369

Integrating Non-Virtual Electronic Activities in Game-Based Learning Environments

Jean-Charles Marty, Thibault Carron, Stéphane Talbot, Gregory Houzet and Philippe Pernelle

378

From « Haute-Couture » to « Ready-to-Wear »: Typology of Serious Games Implementation Strategies in Higher Education

Hélène Michel

386

Motivation and Manipulation: A Gamification Approach to Influencing Undergraduate Attitudes in Computing

Nicholas Mitchell, Nicky Danino and Lesley May

394

Sit Down to Table and Confess who you are! Design of an Epistemic Game for Nutritional Education at Secondary School

Réjane Monod-Ansaldi, Eric Sanchez, Daniel Devallois, Thomas Abad, Pierre Bénech, Anne Brondex, Isabelle Mazzella, Sandrine Miranda, Claudie Richet and Céline Recurt

401

Learning in Context Through Games: Towards a new Typology

Alex Moseley

409

Let the Students Construct Their own fun And Knowledge - Learning to Program by Building Computer Games

Peter Mozelius, Olga Shabalina, Christos Malliarakis, , Florica Tomos, Chris Miller and David Turner

418

Towards Understanding the Instructional Value of Real-Time Continuous Feedback From the use of Simulation Games

Mathews Nkhoma, Jaime Calbeto, Narumon Sriratanaviriyakul, Thu Yein Win, Quyen Ha Tran and Thanh Kim Cao

427

Learning Math as you Play: Comparing Arithmetic Performance Enhancement Induced by Game Play and Paper Exercises

Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy

434

Serious Game Adaptive Learning Systems

Chinedu Obikwelu and Janet Read

442

Combatting Social Isolation and Cognitive Decline: Play a Physical or Digital Game?

Daire Ó Broin and Ross Palmer

450

Volume Two

iii

Paper Title

Author(s)

Page No.

Sports Games’ Role for Learning Health Knowledge

Kelly O’Hara, Dulce Esteves, Rui Brás, Ricardo Rodrigues, Paulo Pinheiro and Marco Rodrigues

458

A Multi-Agent Architecture for Collaborative Serious Game Applied to Crisis Management Training: Improving Adaptability of non Played Characters

M’hammed Ali Oulhaci, Erwan Tranvouez, Sébastien Fournier and Bernard Espinasse

465

Nuclear Mayhem – a Pervasive Game Designed to Support Learning

Trygve Pløhn

475

StartUp_EU: Using Game-Based Learning and Web 2.0 Technologies to Teach Entrepreneurship to Secondary Education Students

Aristidis Protopsaltis, Thomas Hainey, Spiros Borosis, Thomas Connolly, Jesus Copado and Sonia Hezner

484

Measuring Effects of Reflection on Learning: A Physiological Study

Wen Qi, Dominique Verpoorten and Wim Westera

495

Evaluation of Adaptive Serious Games using Playtraces and Aggregated Play Data

Christian Reuter, Florian Mehm, Stefan Göbel and Ralf Steinmetz

504

Learning Effectiveness of Management Simulation Game Manahra

Petr Smutný, Jakub Procházka and Martin Vaculík

512

Using the Master Copy - Adding Educational Content to Commercial Video Games

Heinrich Söbke, Thomas Bröker and Oliver Kornadt

521

An Application of Adaptive Games-Based Learning Based on Learning Style to Teach SQL

Mario Soflano, Thomas Connolly and Thomas Hainey

531

Can Moral Sensitivity be Enhanced by Game Play?

Gunilla Svingby

539

Digital Educational Games: Adopting Pedagogical Agent to Infer Leaner‘s Motivation and Emotional State

Ogar Ofut Tumenayu and Olga Shabalina

546

Adapting the Complexity Level of a Serious Game to the Proficiency of Players

Herre van Oostendorp, Erik van der Spek and Jeroen Linssen

553

Designing Casual Serious Games in Science

Ayelet Weizman

561

Designing a Collaborative Serious Game for Team Building Using Minecraft

Viktor Wendel, Michael Gutjahr, Philipp Battenberg, Roman Ness, Sebastian Fahnenschreiber, Stefan Göbel and Ralf Steinmetz

569

Application of the Principles of Gamification to Facilitate Acquisition of Self-Management Skills in Young People With Long-Term Medical Conditions

Andrew Wilson and Janet McDonagh

579

Development of an Implementation Framework for Games-Based Construction Learning Using Scratch in Primary Education

Amanda Wilson, Thomas Hainey and Thomas Connolly

587

Game Literacy Revisited: Developing Critical Play in Schools

Rafael Marques de Albuquerque and Shaaron Ainsworth

599

A Systematic Literature Review of Methodology Used to Measure Effectiveness in Digital GameBased Learning

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy

607

Investigating Collaborative Games to Teach Mathematics-Based Problem Solving in the Classroom

Reem Al-Washmi, Gail Hopkins and Peter Blanchfield

617

PHD papers

Paper Title

Author(s)

Page No.

Training Flexible and Adaptive Arithmetic Problem Solving Skills Through Exploration With Numbers: The Development of NumberNavigation Game

Boglárka Brezovszky, Erno Lehtinen, Jake McMullen, Gabriela Rodriguez and Koen Veermans

626

Trials to Assess Team-Based Mixed-reality (TBMR) Games in HE

John Denholm and Sara de Freitas

635

Understanding ‘Game-Ness’ Within the ® SCRABBLE Family of English Word Games

Paridhi Gupta

645

Interactive Story as a Motivator Element in an Educational Video Game

José Rafael López-Arcos, Francisco Luis Gutiérrez Vela, Natalia Padilla-Zea and Patricia Paderewski

656

A Domain Ontology of Game Theory Applied to Game Based Learning

Yemna Mejbri, Maha Khemaja and Rafik Braham

666

Puzzle-Based Games as a Metaphor for Designing in Situ Learning Activities

Javier Melero, Patricia Santos, Davinia HernándezLeo and Josep Blat

674

Supporting and Facilitating Collaborative Learning in Serious Games

Kimmo Oksanen and Raija Hämäläinen

683

Playing for the Future - Examining Gameplay, Narrative and fun in Games-Based Training

Mark O’Rourke

691

Towards Game Based Learning Design Process Based on Semantic Service Oriented Architecture (SSOA)

Kaouther Raies, Maha Khemaja and Rafik Brahamm

698

Using Games for Learning, From the Students’ Perspectives

Aishah Abdul Razak and Thomas Connolly

706

Incidental Learning in a World of Warcraft Guild, a Case Study

Gabriela Rodríguez

714

In Search for the Right Measure: Assessing Types of Developed Knowledge While Using a Gamified Web Toolkit

Martin Ruskov, Paul Ekblom and Angela Sasse

722

The Influence of Digital Games on Learning Reading: A Closer Look

Mas Idayu Md Sabri, Peter Blanchfield and Gail Hopkins

730

The Mediatization of Digital Games for Learning – a Dual Rub-Off Effect

Helga Sigurdardottir and Robin Munkvold

740

Efficacy of Reward Allotment on Children’s Motivation and Learning

Zhenhua Xu, Earl Woodruff and Bodong Chen

748

Applying Ideas From Intelligent Tutoring Systems for Teaching Programming in Game Based Learning

Matej Zapušek and Jože Rugelj

756

Cultivating Preschoolers Creativity Using Guided Interaction With Problem Solving Computer Games

Georgios Fessakis and Dimitrios Lappas

763

Haptic Physics Simulation

Luciano Santos and Carlos Vaz de Carvalho

771

Evaluating the Embedding of Games Based Learning in a Computing Subject at University

Emilia Todorova and David Moffat

776

Masters

WIP Papers

Paper Title

Author(s)

Page No.

A Design Approach for Implementing 3D Educational Collaborative Virtual Environments on Virtual World Platforms

Rosa Reis, Benjamin Fonsecaand Paula Escudeiro

785

EMOTE: Embodied-Perceptive Tutors for Empathy-Based Learning in a Game Environment

Sofia Serholt, Wolmet Barendregt, Tiago Ribeiro, Ginevra Castellano, Ana Paiva, Arvid Kappas, Ruth Aylett and Fernando Nabais

790

Exploring Learning Effects During Virtual Sports Through Biomechanical Analysis (a Work in Progress)

Pooya Soltani and Jo達o Paulo Vilas-Boas

793

siLang: Culturally Oriented Language Skill Development in Line With Workplace Needs

Hariklia Tsalapatas, Olivier Heidmann, Rene Alimisi and Elias Houstis

797

Developing Ethical Decision Making Skill of Novice Volunteers in Natural Disaster Response

Didin Wahyudin, Shinobu Hasegawa and Tina Dahlan

800

Preface These proceedings represent the work of researchers participating in the 7th European Conference on GamesBased Learning, which is being organised and hosted this year by the Instituto Superior de Engenharia do Porto (ISEP), Porto, Portugal. The Conference Chair is Dr Carlos Vaz de Carvalho and the Programme Chair is Dr Paula Escudeiro, both from the Instituto Superior de Engenharia do Porto, Porto, Portugal. The conference will be opened with a keynote from Dr. Baltasar Fernández-Manjón, from Complutense University of Madrid, Spain, on the topic of Learning Analytics Applied to Serious Games. The opening keynote address on the second day is by Paulo Gomes, Game Director and Producer at BIGMOON STUDIOS. The Conference is a valuable platform for individuals to present their research findings, display their work in progress and discuss conceptual advances in many different areas and specialties within Games-Based Learning. It also offers the opportunity for like minded individuals to meet, discuss and share knowledge. ECGBL continues to evolve and develop, and the wide range of papers and topics will ensure an interesting twoday confercence. In addition to the main streams of the conference, there are mini tracks focusing on the areas of Multi-User Virtual Environments, Content and Assessment Integration, User Profiling and Barriers and Opportunities for the introduction of GBL in Educational Settings. In addition to the presentations of research the conference this year has introduced a competition to provide an opportunity for educational game designers and creators to participate in the conference and demonstrate their game design and development skills in an international competition. This competition has been sponsored by SEGAN – Serious Games Network. With an initial submission of more than 50 games, 24 finalists will present their games at the conference. Prizes will be awarded to the three games judged to demonstrate the best quality and originality of game play itself and the positioning and articulation of the game’s contribution to the educational domain. With an initial submission of 179 abstracts, after the double blind peer review process, there are 71 research papers, 18 PhD research papers, 3 Masters research papers and 5 work-in-progress paperspublished in these Conference Proceedings. These papers represent research more than 30 countries, including Algeria, Australia, Austria, Belgium, Brazil, Canada, Czech Republic, Denmark, Finland, France, Germany, Greece, Hong Kong, India, Ireland, Israel, Italy, Japan, The Netherlands Norway, Portugal, Russia, Slovenia, Spain, Sweden, Switzerland, Tunisia, UK, USA and Vietnam. We hope that you have an enjoyable conference. Dr Paula Escudeiro Programme Chair and Carlos Vaz de Carvalho Conference Chair October 2013

vii

Conference Committee ECGBL Conference Director Professor Thomas M Connolly, University of the West of Scotland, UK Conference Executive: Dr Carlos Vaz de Carvalho, Instituto Superior de Engenharia do Porto, Portugal Dr Paula Escudeiro, Instituto Superior de Engenharia do Porto, Portugal Dulce Mota, Instituto Superior de Engenharia do Porto (ISEP), Porto, Portugal Isabel Azevedo, Instituto Superior de Engenharia do Porto (ISEP), Porto, Portugal António Castro, Instituto Superior de Engenharia do Porto (ISEP), Porto, Portugal Ana Barata, Instituto Superior de Engenharia do Porto (ISEP), Porto, Portugal Bertil Marques, Instituto Superior de Engenharia do Porto (ISEP), Porto, Portugal Rosa Reis, Instituto Superior de Engenharia do Porto (ISEP), Porto, Portugal Mini Track Chairs Prof. Dr Wilfried Admiraal, Leiden University, The Netherlands Dr Kristian Kiili, Tampere University of Technology, Finland Prof Konstantinos Kalemis, National and Kapodistrian University of Athens, Dr Thomas Hainey, University of the West of Scotland, UK Dr Jordi Sánchez-Navarro, Open University of Catalonia, Barcelona, Spain Dr Daniel Aranda, Open University of Catalonia, Barcelona, Spain Dr Stefan Göbel, Technical University of Darmstadt, Germany Viktor Wendel, Technical University of Darmstadt, Germany Johannes Konert, Technical University of Darmstadt, Germany Committee Members The 2013 conference programme committee consists of key people in the games based learning community, both from the UK and overseas. The following people have confirmed their participation: Dr Wilfried Admiraal (Leiden University, Leiden, The Netherlands); Dr. Minoo Alemi (Sharif University of Technology, Iran); Anissa All (iMinds-MICT-Ghent University, Belgi); Daniel Aranda (Universitat Oberta de Catalunya, Spain); Nikolaos Avouris (University of Patras, Greece); Isabel Azevedo ( Instituto Superior de Engenharia do Porto (ISEP), Portugal); Ana Barata ( Instituto Superior de Engenharia do Porto (ISEP), Portugal);Dr. Wolmet Barendregt (Gothenburg University, department of applied IT, Sweden); Francesco Bellotti (University of Genoa, Italy,); Mary Bendixen-Noe (Ohio State University, USA);Dr Tobias Bevc (University of Augsburg, Germany);Dr Bani Bhattacharya (IIT Kharagpur, India); Dr Peter Blanchfield (School of Computer Science, University of Nottingham, UK); Natasha Boskic (The University of British Columbia, Canada);Dr. Rosa Maria Bottino (Istituto Tecnologie Didattiche - Consiglio nazionale Ricerche, Italy); Hadya Boufera (University Of Mascara, Algeria); Philip Bourke (LIT-Tipperary, Ireland);Dr Liz Boyle (University of the West of Scotland, UK); Dr Tharrenos Bratitsis (University of Western Macedonia, Greece);Prof Anthony Brooks (Aalborg University, Denmark);Prof David Brown (Nottingham Trent University, UK);Prof. Dr.-Ing. Carsten Busch (University of Applied Sciences HTW-Berlin, Germany,); Dr George Caridakis (University of the Aegean / NTUA, Greece,); Dr Thibault Carron (Université de Savoie, Chambéry, France); Rommert Casimir (Tilburg University, The Netherlands); António Castro ( Instituto Superior de Engenharia do Porto (ISEP), Portugal);Dr Erik Champion (Massey University, New Zealand);Prof Maiga Chang (Athabasca University, Canada); Dimitris Charalambis ( University of Athens, Greece);Dr Darryl Charles (University of Ulster, UK); Nathalie Charlier (Catholic University of Leuven, Belgium);Dr Yam San Chee (Nanyang Technological University, Singapore);Dr. Ming-Puu Chen (National Taiwan Normal University, Taiwan,); Satyadhyan Chickerur (M S Ramaiah Institute of Technology, India);Professor Thomas Connolly (University of West of Scotland, UK); Tamer Darwish (Brunel University, UK); Ioannis Darzentas (University of Aegean, Greece);Dr Sara De Freitas (Birkbeck College University of london, UK);Dr Sophia Delidaki (Hellenic American Educational Foundation, Greece,);Dr Ioannis Deliyannis (Ionian University, Greece,);Dr. Muhammet Demirbilek (Suleyman Demirel University, Turkey);Dr David Edgar (Glasgow Caledonian University, UK); Patrick Felicia (Waterford Institute of Technology, Ireland); Georgios Fesakis (University of the Aegean, Greece);Dr. Brynjar Foss (University of Stavanger, Norway);Dr Christos Gatzidis (Bournemouth University, UK); Dr Sebastien George (INSA Lyon, France); Panagiotis Georgiadis (University of Athens, Greece); Andreas Giannakoulopoulos (Ionian University, Greece);Dr Stefan Goebel (Technical University Darmstadt, Germany); Pedro Pablo Gomez-Martin (Universidad Complutense, Madrid, Spain); Cleo Gougoulis (Peloponnesian Folklore Foundation, Greece);Dr Dimitris Gouscos (University of Athens, Greece); Maria Grigoriadou ( University of Athens, Greece);Dr David Guralnick (Kaleidoscope Learning, New York, USA);Dr Thomas Hainey (University of the West of Scotland, UK); Paul Hollins (The University of Bolton, United Kingdom);Dr Birgitte Holm Sorensen (Aalborg University, Copenhagen, Denmark);Professor Rozhan Idrus (Universiti Sains Malaysia, Malaysia);Dr Jose Ignacio Imaz (University of the Basque Country, UPV-EHU, Spain); Jeffrey Jacobson (Carnegie Museum viii

of Natural History, Pittsburgh, Pennsylvania, USA); Ruben Jans (Limburg Catholic University College, Belgium); Runa Jesmin (Global Heart Forum, UK);Dr Larry Jones Esan (London Academy Business School, UK); Alexandros Kakouris (University of Athens, Greece); Fragiskos Kalavassis (University of the Aegean, Greece);Prof Konstantinos Kalemis (National Centre of Local Government and Administration, Greece); Dr Michail Kalogiannakis (University of Crete, Faculty of Education, Crete); Dr Anastasios Karakostas (Aristotle University of Thessaloniki, Greece); Dr Elisabeth Katzlinger-Felhofer (Johannes Kepler University, Linz, Austria);Dr Harri Ketamo (Satakunta University of Applied Sciences, Finland);Dr Kristian Kiili (Tampere University of Technology, Pori, Finland); Evangelia Kourti (University of Athens, Greece); Rolf Kretschmann (University of stuttgart, Germany);Dr Timo Lainema (University of Turku, Finland);Prof Miguel Leitao (ISEP, Portugal);Dr. Ximena Lopez Campino (Initium, Italy,); Carrie Lui (James Cook University, Australia);Dr Hamish Macleod (University of Edinburgh, UK);Dr. Rikke Magnussen (Danish school of education, Aarhus university/Steno Health Promotion Center, Denmark); Emanuela Marchetti (Aalborg University Esbjerg, Denmark,); Bertil Marques (Instituto Superior de Engenharia do Porto (ISEP), Portugal);Dr Jean-Charles Marty (LIRIS lab, Lyon, France); Stephanos Mavromoustakos (European University Cyprus, Cyprus); Florian Mehm (Technische Universität Darmstadt, Germany); Michail Meimaris (University of Athens, Greece); Bente Meyer (The Danish University of Education, Denmark); Prof Florence Michau (Grenoble Institute of Technology, France);Dr Christine Michel (INSA-Lyon, France);Dr Jonathan Moizer (University of Plymouth, UK); Assoc Prof Begona Montero-Feta Universitat Politecnica de Valencia Dr Adam Moore (Trinity College, Ireland); Alexander Moseley (University of Leicester, UK); Dulce Mota ( Instituto Superior de Engenharia do Porto (ISEP), Portugal); Constantinos Mourlas (University of Athens, Greece); Peter Mozelius (Stockholm University, Department of Computer and Systems Sciences, Sweden); Karen Neville (University College Cork, Ireland);Dr Piotr Nowakowski (John Paul II Catholic University of Lublin, Poland); Kimmo Oksanen (Finnish Institute for Educational Research, University of Jyväskylä, Finland);Dr John O'Mullane (University College Cork, Ireland);Dr. Michela Ott (Institute Educational Technology, Italy); Dimitra Panagouli (Hellenic American Educational Foundation, Greece); George Papakonstantinou (University of Thessaly, Greece); Agis Papantoniou (Multimedia Laboratory of the School of Electrical and Computer Engineering (ECE) of the National Technical University of Athens (NTUA). , Greece);Dr Marina Papastergiou (University of Thessaly, Greece); Paul Peachey (University of Glamorgan, Treforest, UK); Gilbert Peffer (CIMNE, Spain);Dr Neil Peirce (Trinity College Dublin, Ireland);Dr Eva Petersson Brooks (Aalborg University Esbjerg, Denmark); Elias Pimenidis (University of East London, UK);Professor Selwyn Piramuthu (University of Florida, Gainesville, USA);Prof. Dr. Maja Pivec (FH JOANNEUM University of Applied Sciences, Austria); Angeliki Poylymenakou (Athens University of Economics & Business, Greece); Dr Aristidis Protopsaltis (Institut für Lern-Innovation Friedrich-Alexander-Universität, Germany); Rosa Reis ( Instituto Superior de Engenharia do Porto (ISEP), Portugal);Prof Dr Bernd Remmele (WHL Graduate School of Business and Economics Lahr, Germany); Vyzantinos Repantis (Psychico College, Hellenic-American Educational Foundation, Greece,); Simos Retalis (University of Piraeus, Greece);Dr Pauline Rooney (Dublin Institute of Technology, Ireland);Dr Eleni Rossiou (University of Macedonia,Thessaloniki, Greece);Dr Maria Roussou (makebelieve design & consulting, Greece);Dr Florin Salajan (North Dakota State University , Canada); Jordi Sanchez Navarro (Universitat Oberta de Catalunya, Spain); Manthos Santorineos (School of Fine Arts in Athens, Greece);Dr Olga Shabalina (Volograd State Technical University, Russia); Samir Shah (Penn State University, USA);Dr Markus Siepermann (Technische Universität Dortmund, Germany); Helga Sigurdardottir (Nord Trøndelag University College and the Norwegian University of Science and Technology, Norway);Dr Gavin Sim (University of Central Lancashire, England);Dr. JulieAnn Sime (Lancaster University, UK);Dr Chrysanthi Skoumpourdi (University of the Aegean, Greece,);Prof Venustiano Soancatl (Universidad del Istmo, Mexico); Elsebeth Sorensen (University of Aarhus, Denmark);Dr Mark Stansfield (University of West of Scotland, UK); Martin Steinicke (University of Applied Sciences HTW Berlin, Germany);Dr. Arnab Sylvester (Coventry University, UK);Dr Sabin Tabirca (University College Cork, Ireland, ); Uday Trivedi (R.C. Technical Institute, India);Dr. Thrasyvoulos Tsiatsos (Aristotle University of Thessaloniki, Greece,);Dr Chuang Tsung-Yen (National University of Tainan, Taiwan); Richard Tunstall (University of Leeds, UK);Dr Andrea Valente (Aalborg University Esbjerg, Denmark);Dr Peter Van Rosmalen (CELSTEC / Open University of the Netherlands, The Netherlands);Dr Linda Van Ryneveld (Tshwane University of Technology, Pretoria, South Africa);Dr Carlos Vaz de Carvalho ( Instituto Superior de Engenharia do Porto (ISEP), Portugal);Dr Ayelet Weizman (Snunit center for the advancement of web-based learning, the Hebrew University, Israel); Viktor Wendel (Technical University Darmstadt, Germany); Nicola Whitton (Manchester Metropolitan University, UK); Dorothy Williams (Robert Gordon University , UK); Andrew Wilson (Birmingham City University, United Kingdom); Amanda Wilson (University of the West of Scotland, Scotland);Dr. Aljona Zorina (ESCP Europe, France)

Biographies Conference Director Professor Thomas M Connolly is the original instigator of this conference in 2007, Thomas Connolly is a Professor in the School of Computing at the University of the West of Scotland, having managed the Department of Computing and Information Systems for several years. Thomas worked for over 15 years in industry as a Manager and Technical Director in international software houses before entering academia. His specialisms are games-based learning, online learning and database systems. He has developed three fully online MSc programmes and developed and leads the undergraduate BSc Computer Games Technology programme. He is co-author of the highly successful academic textbooks Database Systems (now in its 4th edition) and Database Solutions (in its 2nd edition). He is a reviewer for several international journals and has been on the committee for various international conferences. He is a member of CPHC (Council of Professors and Heads of Computing) and member of the Higher Education Academy.

Conference Chair Dr Carlos Vaz de Carvalho has a PhD in Information Systems and Technology. He is a Professor at the Engineering School of the Porto Polytechnic (ISEP) and the Director of the R&D group GILT (Graphics, Interaction and Learning Technologies). He was e-Learning Director (2001-2005) of ISEP and Director of the Distance Learning Unit of the Porto Polytechnic (1997-2000). He has published over 100 references on the subject including several books.

Programme Chair

Dr Paula Escudeiro is a Professor at IPP-ISEP with vast experience in project supervision and evaluation, accumulated for the past 21 years. She has a PhD on Informatics/Information Systems on Education and prior experience on software industry related to the development of educational software. She is the director of the Multimedia Laboratory at ISEP and belongs to the Graphics, Interaction and Learning Technologies research center (GILT).

Keynote Speakers Dr Baltasar Fernández-Manjón is a full professor in the Facultad de Informatica at the Complutense University of Madrid (2001-2006) and the Vice Dean of Research and Foreign Relationships at the Computer Science School at UCM (2006-2010). He is IEEE Senior Member and in 2010-2011 he has been Visiting Associate Professor at Harvard University and Visiting Scientist at LCS-MGH. He received a Bachelor in Physics (major in Computer Science) and a PhD in Physics from the UCM. He is member of the IFIP Working Group 3.3 "Research on the Educational uses of Communication and Information Tecnlogies" and of the Spanish Technical Committee for E-learning Standarization (AENOR CTN71/SC36 "Tecnologías de la información para el aprendizaje"). His main research interests are e-learning technologies, educational uses of serious games, application of educational standards, and user-modelling. Dr Paulo J. Gomes is the Game Director and Producer at BIGMOON STUDIOS and he produced games such ‘WRC3’, ‘MotoGP13’, ‘Jagged Alliance: Back in Action’, ‘Trapped Dead Lockdown’ and many others. Has more than 20 years of experience in software development and project management. He’s credited in more them 15 videogames published worldwide on PS3, Xbox360, Wii, PC, Linux, Mac and Mobile. Paulo has a PhD in Computer Science, MBA and he is a Multimedia Professor at Portucalense University.

Mini Track Chairs Prof. Dr Wilfried Admiraal is a full professor of Educational Sciences and chair of the research program Teaching and Teacher Learning of Leiden University Graduate School of Teaching. His research interest combines Educational Sciences, Social Psychology and technology. He published journal articles on mobile game-based learning and game creation by students. Dr Daniel Aranda is a Senior Lecturer in the Department of Information and Communication Studies at the Open University of Catalonia. He is researching on how young people use digital technologies. He works in the research group SPIDER (Smarter People through Interactive Digital Entertainment Resources), Communication & New Media (at the Internet Interdisciplinary Institute / IN3) and eCo (research and innovation in e-learning, information and communication), at the eLearn Center (UOC).

Dr Stefan Göbel holds a PhD in computer science from TUD and has long-term experience in Graphic Information Systems, Interactive Digital Storytelling, Edutainment applications and Serious Games. After five years work as researcher at Fraunhofer Institute for Computer Graphics, from 2002 to 2008 he was heading the Digital Storytelling group at the Computer Graphics Center in Darmstadt. In late 2008 he moved to TUD and is heading the prospering Serious Gaming group at the Multimedia Communications Lab. Dr. Göbel is author of numerous papers and member of different program committees such as ACM Multimedia, ICME, Edutainment, Foundations on Digital Games, Serious Games Conference and serves as jury member of the Serious Games Award. Dr. Thomas Hainey is a Lecturer in Computer Games Technology and Serious Games Researcher at the University of the West of Scotland. He teachers an honours level course in serious games and is primarily interested in the empirical evaluation of games-based learning applications and how to integrate assessment into games-based learning applications. He has a number of publications in this area. Dr Kristian Kiili works as a Senior Research Fellow and an Adjunct Professor at the Tampere University of Technology in which he heads the Advanced MultimediaCenter research laboratory. His current research focuses on game-based learning, user generated game content, game design, and educational exertion games. He is presently involved in two European initiatives: the Game and Learning Alliance (GALA NoE) and Making Games in Collaboration for Learning (MAGICAL). Results received from his research has been published in several scientific publications as well as applied in commercial products Johannes Konert finished his diploma in Computer Science and accompanying studies in cultural studies at the Karlsruhe Institute of Technology (KIT) with a thesis proposing a web-based knowledge management system for the integration of workflow and learnflow. After three years working on the foundation and development of the online social network friendcafe as CEO and senior developer he joined the research group at Multimedia Communication Lab (KOM) at Technische Universität Darmstadt to focus on Serious Games and Social Networks. He became a Ph.D. student of the DFG Research Training Group “Feedbackbased Quality Management in E-learning”. In his research he focusses on the development of solutions to use Social Media concepts for knowledge transfer between peers in Serious Games. Prof Konstantinos Kalemis is an Instructor at the National Centre for Public Administration and Local Government (E.K.D.D.A.) in Adult Education and Lifelong Learning and assigned at the Dept of Primary Education in National and Kapodistrian University of Athens. He has authored a large number of scientific articles, studies and papers in Educational Congress and Seminars. His interests focus on the introduction of New Technologies as an alternative teaching process and the design of new curriculum plans for the open and d-Learning. His research interests also include the education of immigrant ethnic minorities focusing on the gifted and talented students and aim to advance the theory and technology of natural language and knowledge processing, especially semantic analysis that bridges the gap between language and knowledge, by the novel use of both machine learning and inference methods. Dr Jordi Sánchez-Navarro is a Senior Lecturer in the Department of Information and Communication Studies at the Open University of Catalonia (Universitat Oberta de Catalunya).His research revolves around innovation in entertainment and how this interacts with the new practices of cultural consumption in the contemporary media landscape, and how they impact on education. Dr Thrasyvoulos Tsiatsos is currently Assistant Professor in the Department of Informatics of Aristotle University of Thessaloniki. He obtained his Diploma, his Master's Degree and his PhD from the Computer Engineering and Informatics Department of Patras University (Greece). His research interests include Networked Virtual Learning Environments, Computer Uses in Education, Evaluation methods of Internet Learning Environments and Open and Distance Education using Multimedia and Internet Technologies. He has published more than 120 papers in Journals and in well-known refereed conferences and he is coauthor in 3 books. He has been a PC member and referee in various international journals and conferences and participated in more than 20 R&D projects. Also he is member Technical Chamber of Greece. Viktor Wendel received his degree in Computer Science from the Julius-Maximilians-University of Würzburg in 2009. Since November 2009, he is working as a research assistant at the MultimediaCommunications-Lab at the Technical University of Darmstadt. Research topics are Game Mastering in Multiplayer Serious Games, and Collaborative Learning. Further, he is an editor for ACM SIGMM Records.

Biographies of Presenting Authors Aishah Abdul Razak received her MSc. in Information Technology from Multimedia University, Malaysia. She is now pursuing her PhD in the School of Computing at the University of the West of Scotland. Her research interest is in games-based learning for primary school children. Wilfried Admiraal is full professor of Educational Sciences and chair of the research program Teaching and Teacher Learning of Leiden University Graduate School of Teaching. His research interest combines Educational Sciences, Social Psychology and technology. He published journal articles on mobile game-based learning and game creation by students Rafael Marques de Albuquerque graduated as Bachelor (2008) and Master (2011) in Graphic Design in the Federal University of Santa Catarina (Brazil) and is currently carrying out his PhD in Education in the Learning Sciences Research Institute of the University of Nottingham (UK). His research interests are digital games, learning and school, especially game literacy. Rene Alimisi has a rich engineering background and thorough knowledge of the field of Information Communication Technology in Education. She holds a master with distinction in ICT in Education (IOE, University of London). She has more than 4 years experience within the area of Lifelong Learning European Projects and teaching experience in well- known institutions in Greece and UK. Anissa All works as a junior researcher at IBBT-MICT (Ghent University) since July 2011. Since January 2013, Anissa is working on a PhD through a IWT grant (Flemish agency for Innovation by Science and Technology). In this PhD research, she will develop a standard procedure to measure effectiveness of serious games aimed at cognitive learning outcomes. Yasemin Allsop has been working as an ICT coordinator in primary Schools across London for almost 10 years. MPhil/PhD student at Goldsmiths, University of London. Focus is on children’s learning and cognitive development through digital game making activities. Interested in the role of teachers when teaching with digital games and game design. Reem Al-Washmi is a second-year PhD student at the University of Nottingham, in the School of Computer Science. Her PhD is entitled “Collaborative games-based learning to support problem solving in UK KS2 Mathematics”. Her long term interests include the study of problems that children face in mathematics and the use of technology to overcome these. António Andrade has a Degree in Communication Design from the Faculty of Arts of the University of Porto. Currently he is working on his Computer Science MSc at the School of Engineering of the Porto Polytechnic (ISEP), researching on Virtual Communities of Practice. Massimiliano Andreoletti Professor of Educational gaming and animation at the Catholic University of the Sacred Heart of Milan. Researcher and author of several essays and articles in the Media Education and Educational Technologies (internet, video games, mobile, tablet/pad, cloud). He’s still a videogamer and father of a child of five years. Alexandra Androussou is a Associate Professor in Teaching Methodology at National and Kapodistrian University of Athens, Greece. Her research and writing focus on teaching practices, teacher education and education of minority groups. She also deals with the development of educational materials for children in electronic and conventional form and she produces educational materials for teachers such as the website www.kleidiakaiantikleidia.net Anna Arici is a Learning Scientist with the Center for Games & Impact, at Arizona State University, where she designs and researches game-based learning environments for educational and social challenges, change, and sustainable impact. Additionally, as director of Quest2Teach, she creates game-infused learning curricula and gamification systems for pre-service teachers to prepare and support highly effective educators. Jannicke Baalsrud Hauge is research scientist at Bremer Institut für Produktion und Logistik(BIBA). She is teaching decision making in SC at the University of Bremen and Jacobs University. Main interest: Serious games, TEL, use of ICT in logistics. Responsible for many WPs in EU and national projects on ICT applications, logistics and TEL. Authored 100+ papers. Sasha Barab is a Professor in the Teachers College at Arizona State University where he co-founded and serves as the Executive Director of the Center for Games & Impact. Dr. Barab is an internationally recognized learning scientist who has researched, designed, and published extensively on the challenges and opportunities of using games for impact.

xii

Lizzy Bleumers graduated as a Master in Psychology and obtained a postgraduate degree in user-centered design. She now conducts user research at iMinds-SMIT (Vrije Universiteit Brussel) within the area of play, learning and participation. She was recently involved in a European project, which explored how digital games can be part of empowerment and inclusion initiatives. Dr. Rosa Maria Bottino Director of the Institute for Educational Technology of the Italian National Research Centre (ITDCNR). Author of more than 100 scientific publications both in national and international journals, books and conference proceedings. Dr. Bottino participated in both national and European projects and Networks of Exellence in Technology Enhanced Learning, including GALA NoE on serious games Cyril Brom is an assistant professor at Faculty of Mathematics and Physics of Charles University in Prague. His research interest is in serious games, modelling behaviour and episodic memory of virtual human-like characters, in level of detail AI, and in computational biology. Hope Caton lectures Game Creation Processes at Kingston University, London, where she also leads inKUbator, a multidisciplinary games development studio Hope founded in 2010 with Dr Darrel Greenhill. In addition to investigating the effects of introducing gamification in the classroom, Hope’s areas of research include developing serious games for health and education. Nathalie Charlier is an assistant professor at the Faculty of Pharmaceutical Sciences and co-ordinator of the Teacher Training in Health Sciences Education at the KU Leuven, Belgium. She obtained a BSc and MSc in Pharmaceutical Sciences and her PhD in Medical Sciences. Her current research interests are (i) game-based learning in health science education and (ii) the use of new technologies in education. Yaëlle Chaudy researcher at the University of the West of Scotland. She obtained an MSc in computing from INSA Lyon (Institut National des Sciences Appliquées) and a bachelor in French as a Second Language from Stendhal University in Grenoble. Interested in both computing and education, she is now studying the integration of assessment in GBL applications. John Denholm In final stages of PhD at Serious Games Institute, Coventry, researching into value of educational games. MSc from Imperial College, London and held several senior positions in major UK companies, mainly corporate planning and development of strategic business models. He has lectured on Business, Project Management and Finance courses at Birmingham City and Coventry Universities and supervises Master’s students at Warwick and Manchester Universities. Jill Denner, PhD, is a Senior Research Scientist at Education, Training, Research, a non-profit organization in California where she studies how students learn while creating computer games. She has written numerous peer-reviewed articles, and coedited: “Beyond Barbie and Mortal Kombat: New Perspectives on Gender and Gaming,” published by MIT Press in 2008. Kyriaki Dourda is a post-graduate student at the Early Childhood Education Department, at the University of Western Macedonia, Greece. She has graduated from the School of English Language and Literature at the Aristotle University of Thessaloniki. Her research interests include: Learning and Teaching Modern languages, GBL, CLIL, Language learning strategies. Ronald Dyer has held senior positions in the area of technology strategy, transformation and change management for performance improvement in the US & Trinidad & Tobago. He is a final year candidate for the Doctorate in Business Administration at Grenoble Ecole de Management, France, focused on research on serious game. Stine Ejsing-Duun is interested in the relation between technology, perception and cognition. Her ambition is to describe how technologies allow us to transcend ourselves. Her research has been connected to games, play and playful processes in various areas. Her present studies are within the fields of learning and art. Lars Elbæk (Ph.D.) is associate professor at University of Southern Denmark, and director of the research group ‘PE Pedagogy and Sports Psychology’. He has worked with interaction design and use of IT in children and youth sport and physical education through a number of years. Lars Elbæk is currently working partly at the Play and Learning – Kids n’ Tweens Lifestyle EU founded project. www.kidsntweens.dk Valérie Emin, PhD, is a researcher at the Institut Français de l'Éducation, member of S2HEP Laboratory. She coordinates since 2008 a research project on pedagogical scenario design in science and technique discipline. Her current research topics are "Pedagogical scenarios design" and "Game based learning". She's an associate member of GALA european network of excellence.

xiii

David Farrell is a lecturer in game design at Glasgow Caledonian University with interests in player centric design and Serious Games. He was the lead designer and developer of the e-Bug Serious Games and his research focuses on improving the integration of good game design practice with well grounded pedagogy. Toru Fujimoto is an assistant professor at the Center for Research and Development of Higher Education, The University of Tokyo. He completed his graduate study at the Pennsylvania State University (Ph.D. in Instructional Systems). His research focus is on the design of learning environments using digital games and social media. Georgy Gerkushenko conducts research in the area of social informatics and e-learning since 2000. He gained PhD degree on creating electronic educational resources in 2004 and MBA degree on “Chief Information Officer” in 2009. Currently he is a senior lecturer of CAD department at Technical University in Volgograd, Russia. Main research interests are personal learning environment, education social networking and electronic educational resources. Dr. Lisa Gjedde is Professor with special responsibilities at Aalborg University in Copenhagen at the Dept. for Learning and Philosophy, director of the Research Center for Creative and Immersive Learning Environments: reCreate. Dr. Stefan Göbel holds a PhD in computer science from TUD and has long-term experience in Graphic Information Systems, Interactive Digital Storytelling, Edutainment applications and Serious Games. Since 2008 he is heading the Serious Gaming group at the Multimedia Communications Lab of Technische Universität Darmstadt. Paridhi Gupta is a PhD research student in the department of School of Design at the Hong Kong Polytechnic University (Hong Kong). Her research investigation focuses on interactive play and games within English Language subject classrooms in Hong Kong local Primary schools. She has a Master’s Degree in visual communication from IDC, IIT Mumbai. Dr. Thomas Hainey is a Lecturer in Computer Games Technology and Serious Games Researcher at the University of the West of Scotland. He teachers an honours level course in serious games and is primarily interested in the empirical evaluation of games-based learning applications and how to integrate assessment into games-based learning applications. He has a number of publications in this area. Claire Hamshire has worked at Manchester Metropolitan University (MMU) since 2003; initially as a Senior Lecturer in Physiotherapy and from 2008 as a Senior Learning and Teaching Fellow. This role combines faculty teaching with a cross institutional contribution to technology and games-based innovation. She was awarded a Higher Education Academy National Teaching Fellowship in 2012 Thorkild Hanghøj is an Associate Professor at the ResearchLab: IT, Learning and Design (ILD), Aalborg University, Copenhagen. He holds a PhD on the playful knowledge in educational gaming. Research areas include: game-based teaching, games and Mother Tongue Education, and problem-based game design. Shinobu Hasegawa received his B.S., M.S., and Ph.D. degrees in systems science from Osaka University in 1998, 2000, and 2002. He is now an associate professor in Center for Graduate Education Initiative, Japan Advanced Institute of Science and Technology. His research areas include support for Web-based learning, game-based learning, language learning, and community based learning. Wayne Holmes is currently Head of Education for the games-based learning company zondle. Previously, he was a teacher and the Head of Research for an education charity. He has an MA in Philosophy, an MSc (Oxon) in Education, and is completing his DPhil (doctorate) in Education (researching games-based learning) at the University of Oxford. Robyn Hromek is a practicing educational psychologist working in Australian schools and an Honorary Associate of the The University of Sydney, Australia. She has created a set of 15 board games to teach social and emotional skills to children and young people and has spoken at numerous international conferences on games-based learning. Jennifer Jenson is Professor and Director for the Institute of Research on Learning Technologies at York University, Canada. She has published on games, game design, gender and game play and digital games in education Helle Skovbjerg Karoff (PhD) is Assistant Professor at Aalborg University/Copenhagen and a member of ILD: IT and Learning Design. Helle´s main research field is play and interaction with technology, especially questions of the dynamics of play, for example through danger, movement and sociality. Harri Ketamo, PhD founder/chief scientist, Eedu ltd. and Adjunct Professor, Tampere University of Technology. Specialized in Complex Adaptive Systems, Cognitive Psychology of Learning, Neural Computing and Educational Technology. Was Direcxiv

tor of Education, Satakunta University of Applied Sciences and CEO & founder GameMiner ltd, company focusing on Data Mining/game AI. Published international/peer-reviewed research articles; presentations on studies in international forums. Several awards and nominations related to R&D activities. Michael Kickmeier-Rust holds a PhD in cognitive psychology and he is an experienced project manager and software developer. His research and development activities focus primarily on technology-enhanced learning, in particular intelligent, adaptive educational systems and human-computer interaction. Since 2010 Michael is with the Knowledge Management Institute at Graz University of Technology. Kristian Kiili works as an adjunct professor at Tampere University of Technology. His research focuses on game based learning, exergaming, and game design issues. Results received from his studies has been published in several scientific publications as well as applied in commercial e-learning products. Luise Klein obtained a MSc. degree in Digital Media from the University of Applied Sciences Bremerhaven. Her interests are in enabling learning with and about media and technology, especially in informal playful settings. She also develops her game-based learning and mobile learning applications. Antti Koivisto is a Ph.D. student at the Tampere University of Technology in Pori, Finland. He currently works at Satakunta University of Applied Sciences as a researcher and at Eedu Ltd as a game developer. His research interests are exergames. He is currently researching how games suit for elderly and mentally disabled people. Johannes Konert finished his diploma in Computer Science at the Karlsruhe Institute of Technology (KIT). After three years work on the foundation and development of the online social network friendcafe, in June 2010 he joined the research group at Multimedia Communication Lab (KOM) at Technische Universität Darmstadt to focus on Serious Games and Social Networks. Evangelia Kourti is an associate Professor of Social Psychology specializing in communication at the National and Kapodistrian University of Athens, Greece. Her research interests cover the scientific fields of communication, media and children and the psychology of cyberspace. Loukas Koutsikos holds a Master Degree (MSc) in "ICT for Education", from the National Kapodistrian University of Athens. He holds a Bsc of Electrical Engineering Educator from the Higher School of Pedagogical and Technological Education. He works in Secondary Vocational Education and has participated in various programs dealing with the implementation of Educational Technology. Dimitrios Lappas has graduated from the Hellenic Military Academy in 2005. He also has a Bachelor's Degree from Pre-school Education and Educational Design Department of the University of the Aegean. He is currently a postgraduate student in “Models of Designing and Planning Educational Units”, Master's and PhD Degree program at the University of the Aegean. José Rafael López-Arcos Member of the GEDES research group in the Department of Computer Languages and Systems at the University of Granada. His research focuses on the integration of storytelling into educational video games. Rikke Magnussen associate professor at ResearchLab: ICT and Design for Learning, Aalborg University Copenhagen. Main research interest is how game-based technology can open for innovation in science education. Part of numerous national and international learning game development and research projects for over ten years and published extensively on subject of game's potentials in science education. Gunver Majgaard (PhD) is Associate Professor at The Maersk Mc-Kinney Moller Institute, University of Southern Denmark. Her research interests are design of digital educational tools; human computer interaction; participatory design processes; learning processes; didactical design; program and curriculum development. She has developed the engineering program "Learning and Experience Technology". Christos Malliarakis is a teacher of Computer Science in Mandoulides Schools, a large private primary/secondary school in Thessaloniki, Greece. He holds a BSc and an MSc in Informatics from the Computer Science Department of the Aristotle University of Thessaloniki and he is undergoing his PhD research in Game Based Learning on Computer Programming since May 2011. Dionissios M. Manessis holds a M.Sc. in ICT for education, from the University of Athens, Greece. He is now a Ph.D. student at the department of Early Childhood Education of the University of Athens. His research interests include the use of digital games in Early Childhood Education and students’ attitudes towards Statistics. xv

Jean-Charles Marty associate professor at LIRIS lab in Lyon (France). Research interests are in observation of collaborative activities, through traces of these activities. Research results are applied to Technology Enhanced Learning, and in particular to learning game environments. Participates to several projects in this field (Learning Adventure, Learning Games Factory, Serious Lab for Innovation, Pegase). Organized an international school on Game-Based Learning in June 2011. Apostolos Mavridis is a PhD candidate on the subject of “Game Based Learning” in the Department of Informatics, Aristotle University of Thessaloniki, Greece. He holds a BSc in Computer Science and an MSc in ICTE (Information and Communication Technology in Education). Mas Idayu Md Sabri is a PhD student at the University of Nottingham. She is currently on study leave from her employment as a lecturer at the University of Malaya, Malaysia. She obtained her BComp Science from University of Malaya, majoring in Software Engineering and obtained her MSc Multimedia Technology from the University of Bath. Her research interests are multimedia technology, edutainment, and interactive learning. Emna Mejbri She obtained the master degree in computer sciences, from the University of Kairouan, Tunisia in 2011. Currently, she is a phD student at the National School of Computer Sciences of Mannouba, Tunisia. Her main research interest is in the area of Learning and Games development. Javier Melero received both his Engineering degree in Computer Science (2008) and Master in Information, Communication and Audiovisual Media Technologies (2009) from the Universitat Pompeu Fabra (UPF), Spain. Since 2006, he has been involved in European and National projects in the field of TEL. His main research focus is about designing technology-supported puzzle-based games. Hélène Michel is a Senior Professor in Grenoble Ecole de Management (France). With a specialization in Innovation Management, she started working on Serious Games in 2003. Her research focuses on the strategic approach of serious games and on their performance’s evaluation. Alex Moseley is an Educational Designer and University Teaching Fellow at the University of Leicester, with long experience of course design and development in higher education. His research areas are in games-based learning, student engagement and effective research skills, and he designs games for education and museum contexts. Peter Mozelius has since 1999 been employed as a teacher for the Stockholm University and the Royal Institute of Technology at the Department of Computer and Systems Sciences (DSV) in Kista, Sweden. He is currently working as an IT-Pedagogue and researcher. His research interests are in the fields of ICT4D, game-based learning and software engineering. Robin Munkvold has been teaching software design at Nord-Trondelag University College (Norway) since 1999. The last five years he has been Program Director within the field of Digital Games and Media Technology. He has published several papers on subjects regarding online learning and ICT as a tool for supporting different pedagogical approaches. Rob Nadolski is assistant professor in TEL at CELSTEC at the Open University of the Netherlands. Main interests: competencebased education, serious games, especially enhancing learner support facilities by exploiting newest technologies. Involved in design and project management of e-learning applications for acquiring complex cognitive skills and research on such applications. Has participated in European and national projects. Elena Núñez Castellar received her degree in Psychology in 2006. In 2011 she obtained her degree of Doctor in Psychology from the Ghent University. During her PhD she got broad experience with research methods in cognitive neuroscience, namely EEG (electro-encephalography) and reaction times research. Since 2012 she joined the research group for Media & ICT (iMinds-MICT). Chinedu Obikwelu is a PhD research student with the ChiCI research group, University of Central Lancashire, he is currently researching the scaffolding mechanism in serious games with emphasis on adaptive scaffolding. He has worked in both the educational and IT sector as a teacher and an IT Support Officer respectively. Daire Ó Broin holds a Ph.D. in Computer Science from TCD, which focussed on approaches to developing the conditions of flow. He has been a lecturer at IT Carlow since 2008, where he teaches on the Computer Games Development programme . His research interests include increasing engagement and intrinsic motivation in games and learning. Kelly O’Hara (Ph.D. Sport Science) is professor at Beira Interior University, Portugal, and a researcher at Sports Sciences, Health Sciences and Human Development Centre. Her research interests are develop game based-learning environments by

xvi

integrating, sports and health, and tennis training. She has published several papers, book chapters, and she is reviewer in international journals. Kimmo Oksanen (Lic.Ed.) is doctoral student at the Finnish Institute for Educational Research (FIER), University of Jyväskylä. He is working on his doctoral thesis about supporting and evaluating collaborative learning in a game context. His research interests include game experience, game design and collaborative learning. Mark O'Rourke is an Educational Advisor with the Curriculum Innovation Unit at Victoria University, Melbourne, Australia. He has worked as a VET researcher, Multimedia Lecturer, Program Manager, Head of School, and Chair Academic Board. Mark's research activity focuses on games-based learning and he is a Fulbright Professional Scholar having undertaken research at USC in LA. M’hamme Ali Oulhaci is a PhD student at LSIS laboratory Aix-Marseille University; his works include behaviors simulation, multi-agents systems, and learners’ assessment in serious games. He got his master at Paris Dauphine Unversity. Contact him at LSIS, Domaine universitaire de Saint Jérôme Avenue Escadrille Normandie Niemen 13397 MARSEILLE Cedex 20. Trygve Pløhn works as a lecturerer and a researcher at the Nord-Trøndelag University College. He obtained his MSc in Software Development, Information Technology from the IT University of Copenhagen in 2007. He is a PhD Candidate at the Norwegian University of Science and Technology. His main research interest is within Pervasive Games and Serious Games. Jakub Procházka, Ph.D. is an assistant professor at the Department of Corporate Economy and at the Department of Psychology, Masaryk University, Brno, Czech Republic. His current research focuses on psychology of leadership and leadership effectiveness. He teaches interactive courses in the field of organizational and work psychology. Dr Aristidis Protopsaltis is a Senior Researcher at the Institut für Lern-Innovation at Friedrich-Alexander-Universität ErlangenNürnberg. His background is in Cognitive Science, Serious Games, ICT and Education. He is involved in a number of European projects with focus on education, e-learning and Serious Games. He has published numerous peer-reviewed conference and journal papers. Wen Qi researcher at CELSTEC Open Universiteit. PhD in Men Machine Interaction. Research interests are in Virtual Environments, Serious Games (for learning) and Human Computer Interaction. He has worked in different research projects in both academia and industry. Those research projects were sponsored by US, European and national funding agencies. He is now active in gaming based learning. Rosa Reis teaches at IPP-ISEP, Computer Engineering Department. MSc on Information Systems in Education and PhD student on Informatics at University UTAD-Tras-os-Montes and Alto Douro, Vila Real. Researcher at GILT-Graphic Interaction & Learning Technologies. Researches application of techniques of software engineering in design of educational collaborative virtual environments. Involved in several National and European research projects, presently regular reviewer of several conferences and scientific journals. Christian Reuter studied Computer Science at TU Darmstadt and finished his Master Thesis about the “Development and Realization of Methods and Concepts for Multiplayer Adventures” in 2011 before he then joined the Multimedia Communication Lab. His research focus includes the Authoring-Platform “StoryTec”, especially its extension for multiplayer serious games. Tiago Ribeiro is an eclectic researcher seeking harmony between arts and technology. He has been collaborating internationally on research projects like LIREC and EMOTE, and also with CMU, focusing especially on non-verbal expression in robots both through animation and sound. He is currently in an early stage of obtaining his PhD, in which he pursues artist-oriented intelligent robot animation. Gabriela Rodríguez has a B.A. degree in Spanish Literature and a M.A. in Education. She is currently a PhD student at Turku University’s Faculty of Education, where she is part of a research group designing a mathematical Serious Game as a tool to develop flexible and adaptive use of arithmetic strategies amongst upper elementary children. Martin Ruskov is currently a PhD student at the UCL Deptartment of Computer Science. He has previously worked in the areas of interactive storytelling and authoring tools for multimedia publishing. For his PhD Martin explores how to develop effective serious games and measure the learning happening with their use.

xvii

Eric Sanchez is associate professor at the Ecole Normale Supérieure de Lyon, head of EducTice, a research team of the French Institute for Education and adjunct professor at the University of Sherbrooke, QC (Canada). His research work concerns the uses of digital technologies for educational purposes (e-learning, simulation, serious games). Luciano Santos has a graduate degree in informatics engineering from the Engineering School of the Porto Polytechnic (ISEP) in 2011, and is currently undertaking a Master’s degree in graphics and multimedia systems at ISEP. Ángel Serrano-Laguna, MSc, works for the Complutense University of Madrid as a researcher in the e-UCM e-learning group as well as being a PhD student. His current research interests are educational video games, learning analytics and the eAdventure project. He has published 8 academic papers related to these topics. Helga Sigurdardottir is a PhD candidate at the Nord-Trøndelag University College. She is attending the PhD program in Interdisciplinary Culture Studies at The Norwegian University of Science and Technology, Trondheim, Norway. Helga has a bachelor degree in Social Anthropology, a Master in Education (Program Evaluation) and a Teacher Certificate from The University of Iceland. Petr Smutný is assistant professor at the Department of Corporate Economy of Faculty of Economics and Administration, Masaryk University, Brno, Czech Rep. His current research focuses on managerial simulation games effectiveness and leadership effectiveness. He teaches several courses using game based approach. Currently he is vice-dean for external affairs of the faculty. Heinrich Söbke researcher in “Intelligent Learning” programme (www.intelligentes-lernen.de) at Bauhaus-Universität Weimar. Focuses on game based learning, where his background in computer science enables him to transfer software design principles into technical design of video games. Ws visiting scholar in Department of Curriculum & Instruction at University of Wisconsin, Madison, when worked on development of educational games at Morgridge Institute for Research. Mario Soflano is a researcher at University of the West of Scotland. His background education is in computer science. His main interests are computer games technology, games design, educational technology, web development, adaptive system and mobile games / software development. Pooya Soltani has a M.Sc. degree in Exercise Physiology from Shiraz University, Iran. He is now a PhD candidate of Sport Science at University of Porto. He’s interested in characterizing Exergames and their effects in three domains of physiology, biomechanics and psychology. He’s working at Porto Biomechanics Laboratory under supervision of Professor João Paulo Vilas-Boas. Narumon Sriratanaviriyakul (Cherry) is Senior Lecturer in Centre of Commerce and Management at RMIT University Vietnam and has 8 years of teaching experience in higher education in international universities. Her research interests include gamebased learning, online social network, case study methods, and technology in education. Martin Steinicke earned his BSc. and MSc. in Business Informatics at University of Applied Science HTW-Berlin. He works in the research project „Innovationsdramaturgie nach dem Heldenprinzip“ headed by Professor Carsten Busch and teaches DGBL. His work centers on game based learning in the business context and information & knowledge diffusion in social networks. Gunilla Svingby is a Professor of Education, at Malmö University, Sweden. I was professor at Lund University, Gothenburg, Oslo University, and Tromso University. Some research projects: A computer game on ethics as a learning environment, Continuous assessment and dynamic examination, Professional competence with simulations in teacher education, Learning by mobile games, Mathematics for the digital generation. Emilia Todorova has recently graduated in BSc Information Systems Development from Glasgow Caledonian University. She has worked on projects involving the Bologna Process, Quality Assurance and Education Policy. Her research interests are on improving learning and teaching, using technology within higher education and quality assurance in the EHEA. Ofut, Ogar Tumenayu: obtained his bachelor degree in Computer science from Cross River University of Technology, Calabar, Nigeria. He is currently studying for his master degree in Volgograd State Technical University Volgograd, Russian Federation. His scientific research is in field of Design and implementation of adaptive Educational Games. Herre van Oostendorp is Associate Professor Human-Media Interaction at the Institute of Information and Computing Sciences, Utrecht University. His research activities are on the domain of Human-Computer Interaction. He is a specialist on the areas of web navigation, hypertext comprehension, usability engineering and cognitive principles in serious game design xviii

Didin Wahyudin is a lecturer in Indonesia University of Education, Bandung. He received Master degree in Game Technology from Bandung Institute of Technology, Indonesia. He has experienced as a first responder in many disasters. Currently, He is a PhD student at School of Information Science JAIST Japan focusing on research of Mobile Game Based Learning. Ayelet Weizman Director of science education at Snunit center for the advancement of web-based learning, located at the Hebrew University of Jerusalem. Designing educational games and interactive learning and teaching materials in science on several websites. PhD in Planetary Sciences from Tel Aviv University and post doctorate studies in science education at Michigan State University. Viktor Wendel received his degree in Computer Science from the Julius-Maximilians-University of Würzburg in 2009. Since November 2009, he is working as a research assistant at the Multimedia-Communications-Lab at the Technical University of Darmstadt. Research topics are Game Mastering in Multiplayer Serious Games, and Collaborative Learning. Further, he is an editor for ACM SIGMM Records. Dr Nicola Whitton is a Research Fellow at Manchester Metropolitan University, specializing in the innovative use of learning technologies in Higher Education. Her particular interest is in the design and use of computer games with adult learners and she is the authors of Digital Games for Learning, a practical guide to educational game development. Amanda Wilson is a research student at the University of the West of Scotland. Her research focuses on how games based construction learning can be implemented into the curriculum within primary education in Scotland using Scratch. Andrew Sean Wilson worked in biomedical research for last twenty years. Interested in use of technology in medical research particularly in the management of musculoskeletal diseases. Designed and developed educational computer programs to help patients and practising doctors gain better understanding of how to manage these diseases. Sees game based learning as another way of assisting in this. Amel Yessad is PhD in computer science. Currently, she is an associate professor in the team MOCAH–LIP6 of the University Pierre et Marie Curie. Her research focuses on knowledge engineering, technology enhanced learning, and serious games. Dr Yessad is involved in several serious game projects. Ebru Yeniman Yildirim is a senior lecturer and Head of Computer Technology & Programming at Uludag University, Bursa, Turkey. She has written books on computer programming and managed many large-scale EU projects on Vocational Training. She is interested in e-learning, how new technologies impact on the teacher’s role in the classroom and game based learning. Matej Zapušek is employed as teaching assistant for computer science courses at University of Ljubljana, Faculty of Education. He is also a PhD student at University of Ljubljana, Faculty of computer and information science. His main field of interest is researching the possibilities for teaching introductory programming with intelligent tutoring systems. Symeon Zourelidis is a postgraduate student of the M.Sc. program "ICT in Education", in the National Kapodistrian University of Athens. He works as a director in primary school. He participates in teacher training course for the use and the application of ICT in the classroom. Research interest focuses on new technologies in education.

xix

Sit Down to Table and Confess who you are! Design of an Epistemic Game for Nutritional Education at Secondary School Réjane Monod‐Ansaldi1, Eric Sanchez1, Daniel Devallois1,2, Thomas Abad1,2, Pierre Bénech1, Anne Brondex1,2, Isabelle Mazzella1,2, Sandrine Miranda1,2, Claudie Richet1,2 and Céline Recurt1 1 French Institute of Education, Lyon, France 2 Lycée Madame de Staël, Saint Julien en Genevois, France rejane.monod‐ansaldi@ens‐lyon.fr eric.sanchez@ens‐lyon.fr Daniel.Devallois@ac‐grenoble.fr Abstract: Nutrition is a complex behaviour. The choices made depend on multiple dimensions as taste, physiological consequences, cultural aspects, or environmental and economic impacts. Rather than transmitting standards to follow, nutrition education might help people to take into account the complexity of the problem, to reflect on all the dimensions involved in nutrition and to make informed choices. The French‐Canadian research project Jouer pour apprendre en ligne aims at designing and studying digital epistemic games (Shaffer, 2007) that address such complex problems. Epistemic games are playful, complex, realistic, authentic and non‐deterministic learning situations that deal with ill‐structured problems (King & Kitchener, 1994). Rather than having a unique solution they have solutions that depend on the available information, knowledge and values. Within this context, we used the 3E model (Sanchez & al., 2012)‐ a tridimensional model for game design ‐ to design “Mets‐toi à table !” (“Sit down to table and confess who you are!”), a digital epistemic game about nutritional education. Our research methodology consists in a Design‐Based Research approach which allows identifying the key factors that have to be taken into account to enroll, entertain and educate the player/learner. Therefore, our work is based on a collaborative research which enables teachers and researchers to iteratively combine design and analysis within an ecological context. The results of this research are both pragmatic (the design of the game) and heuristic (understanding the impact of the game design on the learning process). The data collected encompasses the recording of the students during the game session and focus groups carried out with teachers and students. The analysis of the impact of the choices made for the game design permits to reconsider the design of the game. Indeed, the findings of the first iteration, which bring little play and little learning to the students, lead to modify the gameplay. In this paper we present the game, the methodology of the research and we discuss the key factors that should be taken into account to design a digital epistemic game. Keywords: digital epistemic game, ill‐structured problem, nutrition education, design‐based research

1. Introduction The nutritional choices made by each of us depend on multiple dimensions as taste, physiological consequences, cultural aspects, psychological dimensions or environmental and economic impacts (Vandenbroeck, Goossens, & Clemens, 2007). Rather than providing standards to follow, nutrition education might help people to take into account the complexity of the problem, to reflect on all the dimensions involved in nutrition and to make informed choices (Brixi, Gagnayre & Lamour, 2008). In France, the secondary school curriculum offers 15‐16‐year‐old students, as an optional course, the opportunity to carry out interdisciplinary approaches and to address complex scientific problems raised by our modern society (MPS, BO n°4, 29/04/2010). The course involves mathematic, physics and chemistry and biology and geology teachers. Science and food, is one of the themes of this course. We chose the context of this course to tackle the question of nutrition education, taking the opportunity to address the complexity of nutrition with an interdisciplinary approach. In this respect, complexity relates to what is woven together and there is complexity whenever the various elements that compose a whole are inseparable and interdependent (Morin, 2000). Regarding nutrition education, there is a stake to help students understand that their decisions are grounded on a complex set of values, cultural preferences and societal influences. They also need to understand that their choices concerning food can have a physiological impact (health) but also economic or environmental consequences. Therefore, our study aims at exploring the possibility to design a meaningful learning situation that offers students the opportunity to deal with complex ill‐structured problems (King & Kitchener, 1994). Our approach consists in designing a digital epistemic game to address such issue.

401

Réjane Monod‐Ansaldi et al. In this paper, we describe an empirical study which addresses the question of nutrition education though a Game‐Based Learning approach. This approach consists in designing a digital epistemic game. “Sit down to table and confess who you are!” offers the learner/player the opportunity to face the complexity of nutrition and to reflect about his/her choices and behavior. In the following we describe an empirical work dedicated to identifying the key factors that have to be taken into account to design such an epistemic game. We will present a three‐dimensional model as a background for the design of the game and the methodology of the study based on a collaborative work with teachers. Then, we will present the lessons learned after having tested two versions of the game with secondary students regarding our three‐dimensional model.

2. Digital epistemic games for learning complexity The French‐Canadian project « Jouer pour apprendre en ligne » (Gaming for online learning) (Sanchez et al., 2012) funded by the Canadian Social Sciences and Humanities Research Council aims at addressing two questions. (1) The relevance of digital epistemic games for designing playful learning situations adapted to the digital culture of a new generation of learners. (2) The impact of such learning situations on the personal epistemology of the learners regarding the nature of knowledge and the process of knowing. There is a growing body of research that focus on Digital Play‐Based Learning and there are arguments advocating the consideration of the contexts (Steinkuehler & Duncan, 2008) or the situations (Sanchez & Jouneau‐Sion, 2010) rather than the game as a digital artefact. From this stems the possibility of focusing on the interactions that emerge from the situation, rather than on the game itself (Mitgutsch, 2007). Therefore, following Shaffer (2006), we propose to avoid the widely used term serious game that focuses on the artefact used to play and we prefer the expression digital epistemic game to name playful situations designed with digital technologies that intend to foster epistemic interactions. Epistemic games are complex, realistic, authentic and non‐deterministic learning situations that address ill‐structured problems (King & Kitchener, 1994). Rather than having a unique solution they have solutions that depend on the available information, knowledge and values. Within this context, we designed “Mets‐toi à table !” (“Sit down to table and confess who you are!”), a digital epistemic game about nutritional education. The design of the game is based on a three‐dimensional model (3E) (Sanchez et al., 2012) which consists in a set of recommendations that permit to Enrol, Entertain and Educate the learner/player. The dimension Enrol encompasses the elements that are crucial to foster the students’ motivation to participate and to accept to take up the challenge. Enrolling the learner/player means fostering his/her motivation. Motivation is the point generally putted forward to underline the relevance of a Game‐Based learning approach. According to the self‐determination theory (Ryan & Deci, 2000), motivation results from different universal and innate needs, that include the need for competence, autonomy, and relatedness. The feeling of competence increases when students have to reach clear goals. Therefore, the devolution (Brousseau, 1986) of the problem becomes possible: the students accept to be responsible for solving the problem and take decisions that are motivated by their onsciousness of the situation rather than the teacher’s expectations (Ahuja, Mitra, Kumar, & Singh, 1995). The feeling of competence also depends on the difficulty that the learner player has to face. A too low or too high level of difficulty decreases the motivation. Enrolling the students also depends on the autonomy offered by the situation. Autonomy entails the freedom to make choices and to choose a strategy. Autonomy also depends on the resulting feedbacks of the decisions taken that enable the learner/player to assess the relevance of his/her strategy. Therefore feedbacks are crucial. At any time, they allow the learner/player to evaluate his/her decision and, therefore, to make changes or not. The need for relatedness entails competition as "game worlds are meritocracies" (Reeves, 2011) but also collaboration and the success may result from the ability to collaborate with other learners/players. The dimension Entertain relates the play factors of the game to make possible that the learner/gamer is involved in a playful experience. Entertaining the player is the first objective of a game and pleasure mainly results from the feeling of freedom as a core characteristic of a game (Brougère, 2005). Freedom means that the player/learner is allowed to take decisions and to shape his own strategy. This freedom should offer to the learner/player a high level of control and, despite chance is also a part of the game, what happens should be directly linked to his/her decisions. As a counterpart to freedom, there are arbitrary rules (ie. a set of norms and conventions shared by the players that limit freedom). Rules need to be acceptable, relevant (regarding

402

Réjane Monod‐Ansaldi et al. the model embedded in the game) and clear. Entertainment also results from earning points and getting rewards. They are important in order to maintain a player’s interest in a game’s challenges (Habgood & Overmars, 2006) by increasing the feeling of competence. Frivolity is another core dimension of games (Brougère, 2005). As actions or decisions do not impact the real world, play occurs in a safe space where the learner/player feels secure. The learner/player can make mistakes without real consequences and he is encouraged to continue trying (Gee, 2003). Games are also often based on the use of avatars that are projective identity. The avatar allows one to project one’s values and desires and endorses mistakes and failure. Entertainment also depends on humour and on the artistic value of the game. Educate, the third dimension of the model, encompasses more specifically the epistemic factors. As a meta‐ activity, a game is a model of a situation of reference and simulating a part of the world allows the player to explore a physical or human situation of this world (Egenfeldt‐Nielsen, 2006). Therefore, the relevance of game content is an important issue to deal with. Though that a game can be a metaphor, a "good” epistemic game is anchored in the real world. It should be authentic regarding the interactions enabled by the gameplay. The model of knowledge embedded into a game results from the transposition of a real situation (a situation of reference) into the context of the game. Therefore, this model ought to be relevant regarding the situation of reference (Kirriemuir & McFarlane, 2004). In addition, it is important that the game is adapted to the learning objectives and to the curriculum. An important issue that must be addressed is the link between the content and the gameplay (Habgood, 2007). Games are said to be intrinsic if core game mechanisms (gameplay) and learning content are integrated (i.e. the required knowledge to succeed in the game is the learning purpose). In an extrinsic game, content and gameplay are separated (i.e. the game entails alternate phases of play and school‐like exercises). Regarding learning, intrinsic games are known to be more efficient than extrinsic games (Ibid.). The role of the teacher during the game is recognized to be crucial as well. His/her role cannot be the role of a teacher anymore. Research in Digital‐Play‐Based Learning demonstrates that the knowledge developed within the game is mainly implicit and students do not develop declarative knowledge without reflection and debriefing (Garris, Ahlers, & Driskell, 2002; Habgood, 2007; Sanchez, 2011). Therefore the teacher is needed to help students become aware of the implicit knowledge that they gained. This step is called debriefing or "after action review" (Aldrich, 2009). It aims at bridging the game world and the real world in order to facilitate the transfer of the knowledge developed during the game experience. The issue of the assessment has been recognized to be paradoxical. On one hand, a game entails an embedded assessment used by the player/learner for recognizing achievements and failure. However, by assessing the learner/player with a typical educational point of view, there is a risk to kill the frivolity and the feeling of freedom and, therefore, to kill the game itself.

3. Methodology of the study: A design‐based and collaborative research Our research methodology consists in a Design‐Based Research approach (Design‐Based Research Collective, 2003), that aims to produce theories on learning and teaching, by interventionist iterative methods using design and experimentation in naturalistic contexts (Cobb et al, 2003). Design‐based research involves flexible design revision, multiple dependent variables, and capturing social interactions. Different participants take part in the design to bring their differing expertise into both producing and analyzing the design process and results. Our work is then based on a collaborative research which allows teachers and researchers to iteratively combine design and analysis within an ecological context. The research team includes six secondary teachers (2 mathematic teachers, 2 physics and chemistry teachers and 2 biology and geology teachers), two researchers and a pedagogical engineer. The collaborative work enabled to design of the game “Sit down to table and confess who you are!” and to discuss the key factors that have to be taken into account to enroll, entertain and educate the player/learner. The game design is based on the analysis of the knowledge and social practices in the field of nutrition and with respect to the gameplay from previous games. We drew a conceptual map of nutrition (Figure 1) to design the knowledge model of the game (ie. the concepts that will be included in the game).

403

Réjane Monod‐Ansaldi et al.

Figure 1: Simplified version of the conceptual map The skills related to this theme were also identified. Several focus groups were dedicated to discuss the knowledge model and the gameplay. This iterative and collaborative work enabled to explore different ideas while taking into account the numerous constraints of the challenge. A first version (first iteration design) appeared to be irrelevant regarding our objectives. The difficulties that we faced relate to the complexity of the theme that encompasses many different dimensions and the constraint to enable distant and asynchronous interactions between French and Canadian students on a digital platform (which is also used to record the digital traces of the learners’ activity). This failure led us to skip several constraints such as the use of the digital platform and the level of knowledge. Thus, the second version of the game being designed and tested is based on the lessons that we learnt from the first experimentation. Our methodology enabled to follow our pragmatic objectives (the game design) with more heuristic expectations (to understand the key factors involved that have to be taken into account to foster epistemic interactions) by organizing focus groups with teachers dedicated to discuss the design of the game or the observations that we made during the experimentations or information gained during focus groups with students. The two versions of the game are briefly described below. The two different versions of the game were experimented in three (2012) or four (2013) 75 minutes sessions, with four groups of students about age 15‐16. The data collected encompasses the recording and videotaping of the students during the game session, some students’ productions and focus groups carried out with teachers and students. Choices, actions and expressions of the learners/players were analysed referring to the three dimensions Enrol, Entertain and Educate of the game.

4. A first version of the game: A role playing game The first version of the game designed with the teachers was a role playing game. Each student (or pair of students) successively plays the role of the owner of a restaurant responsible for designing a menu, and the role of a customer choosing a restaurant. The owner of the restaurant whose menu is chosen by the largest number of clients wins the game. The game is driven by a teacher involved as the game master. The learners/players start by picking out "Restaurants cards" that indicate the type of restaurant and, therefore, specifies different constraints regarding the menus. Each team has 20 minutes to create the menu and to advertise it with a short text. Online resources on food and restaurants are available to help the owners of restaurants write their menus. Then, each player/learner shifts to the role of customer and votes individually for one menu except for her/his own. The master of the game scores and displays the results. He gives points to the three restaurants receiving the greatest number of votes. Then, the learner/player comes back to the role of the owners of a restaurant. A third step consists in a discussion: each team has to convince the customers regarding the quality of their menu, and, therefore, to get more votes. A new vote takes place and new scores are added to the previous ones. Then each team picks out a “Challenge card” among bonus, blank, or penalty and a new step, call challenge step, begins. Each team challenges another one by asking them to demonstrate that their menu matches the constraints. If the restaurant owner picked out a bonus card, he automatically wins the challenge. If he picked out a penalty card, he loses. A blank card leads to develop arguments in order to convince the other learners/players that the menu matches the constraints. The game master arbitrates the challenge and the new scores are added to the previous ones. The game ends with the announcement of the overall scores and the winner is declared to be the best restaurant.

404

Réjane Monod‐Ansaldi et al. We used the data collected during the game sessions, and some students and teachers focus groups conducted after the game sessions to analyse and assess this first version. Regarding the power of the game to enroll the students, the teachers noticed that the competition created a great motivation for students, but also generated problems. Some students are not comfortable in competing with their friends (probably due to the school context). Others developed avoidance strategies, completely unrelated to the learning objectives, as secret agreements to help each other or to make another team lose. These ways to cheat were also reported by students during the focus groups. For students, the game was somewhat funny, interesting and original, but they also identified different problems that occurred. They also deplored the academic dimension of the situation, considered by some of them to be more a school assignment than a game. Students asked for a game context different from the school context, for game material as game boards or hourglasses, for minimal intervention of teachers and for the possibility to choose their team. Types of restaurants and constraints were considered to be unbalanced: creating menus and finding arguments were more or less difficult depending on the restaurant. As a result, some students expressed a feeling of injustice. Student also pointed out the lack of action and the lack of diversity regarding the way of thinking during the game. In addition, the teachers also underlined the lack of feedback enabling the students to determine if the menu that they designed was relevant regarding the rules of the game and their autonomy was not totally insured. Therefore, the games failed to fulfil the requirements to foster entertainment. As a result, we all agreed to consider that this first version of the game had too little learning potential. The first result of our project was a game which was not adequate to enroll the learners/player, which failed to entertain them and with few educational potential. However, we learnt a lot and we gained new ideas for re‐ designing the game. Therefore, this first iteration was the source of many choices in the design of the second game version.

5. A second version of the game: Sit down to Table and Confess who you are! One of the main difficulties identified by the research team was to make the game intrinsic (Habgood, 2007) (ie: to foster interactions that mobilize all the targeted knowledge). Therefore, we decided to put our effort on the design of the gameplay. As a result, the gameplay was fully revised for the second version of the game. Each player is now represented by a character. The goal of the game consists in unmasking the other player and in finding his character. Each character is defined by his age, gender, weight, height, activity, personal choices and dietary restrictions. Using learning resources of the game, each player transcribed these features in puzzles, which must be unmasked by his partner. The last player unmasked win. The game material includes now "Characters cards", wildcards, and a game board with pawns, dice (Figure 2). Each "character card” specifies 13 features in four categories (physic, activity, choices and dietary restrictions), which are represented as boxes on the game board. Two other types of boxes are also present on the game board, leading to miss a turn or choose a category. Only a part of the game material is now available under a digital version and the computerization of the game is an ongoing process.

2a: Character card

2b: Game board

Figure 2: The game material

405

2c: Lexicon of pictograms

Réjane Monod‐Ansaldi et al. The game begins with the random drawing of a character card by each team. During the first thirty minutes, the teams mask their characters, by reformulating as far as possible its features in puzzles (Figure 3). Different resources are available for such a task (for example, the way to calculate the body mass index (BMI), information concerning proteins needs for different sports, or local fruits and vegetables production seasons…). Each team owns a wildcard that offers the possibility to ask once for teacher assistance. The two teams compete by throwing the dice, moving their pawns on the game board and asking the opponent to give a piece of the puzzle corresponding to a feature (according to the category referred to by the pawn on the game board). For example, if the pawn lands on a "physical box", a piece of the puzzle hiding weight, height, gender or age of the character must be provided. Information exchange is carried out through five game rounds, then a five minutes break allows each team to summarize and interpret the collected information using the game resources. If one team of students unmasks the opponent character, they win. If not, five new game rounds are played. A video summarizing the game steps is online (http://eductice.ens‐ lyon.fr/EducTice/recherche/jeux‐et‐apprentissage/mets‐toi‐a‐table/mets‐toi‐a‐table ).

Figure 3: Students masking their character’s features Taking into account the amount of resources (46 files of 1 to several pages), the use of computer is essential for simultaneous access to all resources for all teams and allows students to move easily and rapidly from one resource to another (hyperlinks). The results of this second experimentation permitted to state that this new version enables a better engagement of students. This engagement results from a greater level of autonomy offered to the students. They shaped their own strategies and they were able to describe these strategies after the game session. However, the problem which consists in providing the students with relevant feedbacks is not totally solved by this version of the game. We observed different feedbacks, provided by opponents, which were the opportunity for discussing concepts related to nutrition. Yet, the feedbacks are still, for a large part, under the responsibility of the teachers who decides what is relevant or not. The enrolment of students seems also to result from a good combination of collaboration (within pairs of players) and competition (with opponents). In addition, the level of difficulty can be adjusted depending on the resources that are used for the coding phase. This level of difficulty also depends on the system bend, namely the ability for a character to perform one or two deviations from his/her dietary habits. Overall, there was an atmosphere of play during the game sessions and this atmosphere results probably from the freedom offered to the students. However, the rules of the game need revisions. Some of them are simple and they have been respected but others are too complex and it is difficult to state if they have been upheld. Therefore, the autonomy of the students is not yet totally ensured.

406

Réjane Monod‐Ansaldi et al. The issue of the frivolity of the game appears now to have been adequately addressed. The character to be masked/unmasked is an avatar which does not reveal the dietary profile of the students and it is possible to play without consequences. Therefore the frivolity of the game is guaranteed. For this second version of the game, special attention has been paid to the graphical dimension of the game material and the students expressed their interest and also proposed new improvement. In a school context, this aspect has been recognized to be important as it enables to shape the game universe. According to the teachers, the educational properties of the game have improved with this second version. Indeed, during the game, some students made links between the different dimensions of nutrition. However, further investigations need to be carried out to appreciate to what extend the students learnt. Preliminary results show that the model of knowledge embedded into the game is still too simple. It also includes knowledge that is not directly linked to nutrition education. In addition, according to our observations, the students mainly considered independently the different dimensions of nutrition, which is totally opposite to our expectations. Furthermore, we observed that certain students managed to reveal the hidden characteristics of their opponents with a strategy based on a clever use of the dice and the game‐board but not on a relevant use of the knowledge related to nutrition. This aspect needs further improvements and the problem of educating the learner/player has not been totally solved yet. The role of the teacher varied during the game session. The beginning of the game was devoted to help the students use the resources. Later on, they have been more involved in the game itself by being responsible for providing scientific assistance according to the rule of the wildcard. The debriefing phase has not been analyzed yet. However, there is a common agreement for saying that this phase is crucial to help the students grasp the complexity of nutrition. Despite several weaknesses mentioned during the focus groups, this second version is now considered to be far more efficient to enrol, entertain and educate the students.

6. Conclusion The results of this empirical work confirm the complexity to design a digital epistemic game. There are many factors to take into account and, furthermore, they are not isolated elements but elements that are linked by complex relationships. Some elements are crucial. The autonomy of the learner/player is one of the most important issues to consider. Autonomy results from the possibility offered to the learner/player to assess his/her strategies. As a result, the design of a game should pay specific attention to the feedbacks provided during the play. Another difficult challenge that we faced is to combine a clear and easy to learn gameplay with a rich and authentic model of knowledge embedded into the game. A too complex gameplay leads to fail in enrolling the learner/player while a too simple model of knowledge leads to fails in educating him. Regarding this dimension, it is important to underline that designing interactions, both playful and epistemic, has been identified to be the most difficult challenge to tackle. As a result, making the game intrinsic should be one of the core challenges to tackle for the game designers. The visible improvement of the second version of the game that we experimented tends to demonstrate the relevance of a collaborative methodology. The involvement of teachers at the early stage of the design process offers some guarantees that the game will be adapted to the context. In addition, the capacity of researchers to record data offers the opportunity to get an objective idea of the impact of the game. Regarding this aspect, the ideas expressed by the students themselves appear to be particularly important to take into account and we now plan to involve students in the design process.

Acknowledgements References Ahuja, R, Mitra, S, Kumar, R, & Singh, M. (1995). Education Through Digital Entertainment ‐ A Structured Approach. Paper presented at the XXX Ann. Conv. Of CSI, New Delhi. Aldrich, C. (2009). Learning online with Games, simulations, and virtual worlds. Strategies for online learning. San Francisco, Ca: Jossey Bass. Brixi, O., Gagnayre, R. & Lamour, P (2008). Eduquer pour la santé autrement. Propositions en appui aux pratiques alternatives à l'œuvre. Editions Le Manuscrit, Paris. Brougère, G. (2005). Jouer/Apprendre. Paris: Economica.

407

Réjane Monod‐Ansaldi et al. Brousseau, G. (1986). Fondements et méthodes de la didactique des mathématiques. In J. Brun (Ed.), Recherches en didactique des mathématiques (Vol. 7). Grenoble: La Pensée Sauvage. Design‐Based Research Collective. (2003). Design‐based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5‐8. Garris, R, Ahlers, R, & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441‐467. Gee, JP. (2003). What Video Games Have to Teach us About Learning and Literacy? New York: Palgrave Macmillan. Habgood, J. (2007). The effective integration of digital games and learning content. PhD Thesis, University of Nottingham. Habgood, J, & Overmars, M. (2006). The Game Maker's Apprentice: Game Development for Beginners. Berkeley, Ca: APress. King, P M, & Kitchener, K S. (1994). Developping Reflective Judgement: understanding and Promoting Intellectual growth and Critical Thinking in Adolescents and Adults. San Francisco, CA: Jossey‐Bass Publishers. Kirriemuir, J, & McFarlane, C A. (2004). Literature Review in Games and Learning. FUTURELAB SERIES. Bristol. Mitgutsch, K. (2007). Digital play‐based learning; A philosophical‐pedagogical perspective on learning anew based on games. Paper presented at the Games in Action, Gothenburg, Sweden. Morin, E. (2000). Seven complex lessons in education for the future. Paris: Seuil. Reeves, T. (2011). Can Educational Research Be Both Rigorous and Relevant? Educational Designer(4). Ryan, R M, & Deci, E L. (2000). Self‐determination theory and the facilitation of intrinsic motivation, social development, and well‐being. American Psychologist 55, 68‐78. Sanchez, E. (2011). Usage d’un jeu sérieux dans l’enseignement secondaire : modélisation comportementale et épistémique de l'apprenant. Jeux sérieux, révolution pédagogique ou effet de mode ? Revue d’Intelligence Artificielle, 25(2), 203‐222. Sanchez, E, & Jouneau‐Sion, C. (2010). Les jeux, des espaces de réflexivité permettant la mise en œuvre de démarches d’investigation. Paper presented at the Ressources et travail collectif dans la mise en place des démarches d'investigation dans l'enseignement des sciences, Lyon, novembre. Sanchez, E, Jouneau‐Sion, C, L, Delorme, Young, S, Lison, C, & Kramar, N. (2012). Fostering Epistemic Interactions with a Digital Game. A Case Study about Sustainable Development for Secondary Education. Paper presented at the IOSTE XV International Symposium, Hammamet, Tunisia. Shaffer, D W. (2007). How computer games help children learn. NewYork: Palgrave. Steinkuehler, C, & Duncan, S. (2008). Scientific Habits of Mind in Virtual Worlds. Journal of Science Education and Technology, 17(6), 530‐543. Vandenbroeck, IP, Goossens, J, & Clemens, M. (2007). Foresight Tackling Obesities: Future Choices. Project Report. London: Government Office for Science.

408

Learning in Context Through Games: Towards a new Typology Alex Moseley University of Leicester, Leicester, UK am14@le.ac.uk Abstract: The use of real contexts in learning has been of central interest to educational developments such as experiential learning, case studies, work placements and simulations. They centre around the benefits of putting students in realistic situations, using real tools to solve real (or realistic) problems: immersing them in realistic activity rather than theoretical concepts. A growing number of learning games are making use of context in this way: initially focussed around serious games (many close in character to simulations) and most recently distilled into immersive and pervasive games, which mix real and imagined contexts for apparently deep learning experiences. This paper reviews the existing literature around the use of context in learning, considers the applicability to learning games, and proposes new theoretical developments through the consideration of four models for the integration of context into learning experiences (based on a typology of existing contextual learning experiences and games). To illustrate the models and begin to strengthen the typology, a number of existing case studies are referenced, and further research needs highlighted. Keywords: context, authentic, learning, games, models

1. Introduction The Latin origins of the word context give a strong clue to its strength within education: cotexere, to 'weave together'. As will be explored in this paper, context draws a number of facets together, and most importantly, gives meaning to the combined facets greater than the sum of their parts: weaving them together, adding colour and texture. This is not a new revelation, of course: in daily conversation one will hear a request to ‘give me some context’, or an example that begins ‘to put it in context...’. Context adds relevance; relates a concept to some familiar territory or setting. As such, it has enjoyed long and detailed study by psychologists, educators and social scientists across a number of domains and from a range of perspectives. By contrast, the role of context in learning games is a potential area of study that, to date, has not drawn focus, despite the fact that the game design process, and games themselves, use context to great effect (immersing players in other worlds, or in deep narratives, or realistic simulations). This paper aims to structure and catalyse research in this area through a study of the types of contextual learning possible within learning games and playful activities.

2. A study in context In their review of context across a number of domains, Edwards and Miller (2007, p266) identified five principle fields engaged in the debate and study of context: socio‐cultural psychology, applied linguistics, social anthropology, social studies of science, and organisational studies. Johnson (2002) adds the scientific sphere of neuroscience, drawing on principles from biology and physics. In addition to these core areas, study has been undertaken into context applied specifically to learners and study, known as learning context (and drawn from some of the principles from the core domains). Linguists study context from two perspectives: the linguistic context (what comes before and after in writing or speech) and the non‐linguistic (eg. social, temporal, locational aspects), and focus on how context is used (and generated) subconsciously during normal conversation: “whatever information listeners (or readers) use to make sense of what is said (or written)” (Mercer 2000, p20) – most importantly, anything not relevant to the conversation is left out: and this may include otherwise substantial elements such as location, time, physical appearance etc. These focussed, localised ‘contexts of use’ also vary continually, as each new piece of information within a conversation can add or alter the perceived context (Mercer 2000, p20‐21). Ethnographers have also considered context when studying cultural behaviour. Gilbert Ryle (1968) coined the terms thin and thick description to distinguish between an out‐of‐context and in‐context observation by ethnographers (he used the example of a twitch and a wink: whilst they may look the same, the cultural context will give them different meanings).

409

Alex Moseley In neuroscience, context is used to describe the relationship between different parts: the brain takes new content, “searches for meaning and when it finds it, it learns and remembers” (Johnson 2002, p23) – this searching for meaning is the process of applying a known context to new content (and making connections between them). This is an important principle for education: learning and teaching that helps the brain to make these connections will result in more effective learning. Biological connections of this nature are known as interdependence in science: there are strong parallels with the study of context in social anthropology, in particular in the way that groups in the same social or work environment or profession build meaning and development through their social connections (and interdependence on each other). This was observed by Wenger in his study of workplace practice (1998), and described in terms of a community of practice ‐ linked by tools, methods, place and other contextual aspects. Within this environment, new members of a group learn about the community by being immersed in the same context (using the same tools or methods in the same place as the existing members). Vygotsky’s earlier study of child development matches this model of learning in context, his zone of proximal development (Vygotsky 1978) describing this transfer region between beginners and experts. Sharing a strong context will see novice members form a zone of proximity with ‘elder’ members, and benefit from enhanced and more relevant learning as a result (Lave & Wenger 1991). The work of these social anthropologists in what came to be described as situated learning led to a branch of educational research focusing on learner contexts: looking in particular at things that might affect the learner around the learning activity itself; Hansman (2001, p44) neatly summarises: "learning in context is paying attention to the interaction and intersection among people, tools, and context within a learning situation". Edwards and Mercer (1987, pp68‐9) describe the process of education as the establishment of shared mental contexts: contexts that form during learning through communal experience of, and understanding of, an activity plus the context up to that point (previous experiences with the same people, or activities, or in the same space, etc.). Through these shared activities students might generate a common vocabulary and common methods; the context being the platform on which these developments can take place (p82). Johnson (2002) extends the context that needs to be considered when she notes that students "see meaning in schoolwork when they can connect new information with prior knowledge and their own experience" (p vii), and that such context "means much more... than events located in place and time. [It] also consists of unconscious assumptions we absorb growing up, of tenaciously held convictions we gain as if by osmosis, of a worldview than unobtrusively shapes our sense of reality" (p49). Various theoretical frameworks have been used to try to make sense of these wider learning contexts, the most notable of which are activity theory and actor‐network theory. Activity theory builds on Vygotsky's work around tool mediation ‐ how people are influenced not by an object, but by mediation through joint activity with shared cultural tools ‐ and considers tools, people and objects in an interconnected system. Russell (2002, p68) notes: "an activity system might be thought of as a context for behaviour and learning, but not in the sense of something that surrounds the individual's behaviour and learning... [it is] rather a weaving together of the learner with other people and tools into a web of sociocultural interactions and meanings that are integral to the learning". Actor network theory focuses closer on the actors, or linked individuals, with an emphasis on boundary objects that form the links between the actors (tools, activities, theories, etc.): the context is described by this overarching network of actors and the boundary objects that connect them. Whilst learner contexts focus on learners and their educational context, another branch of educational theory is concerned with contexts that are outside formal learning and academia, but that have an effect on that learning. As Lave (1996, p5) notes, we cannot isolate individual actions of learners from society and their relations with it. Situated learning, as Lave, Wenger and others have seen it, is learning situated in another ‐ possibly wider, possibly more directed ‐ context. Lave (1988) compared adult mathematical learning in a traditional school environment, with adults learning the same mathematical equations within a real grocery store setting. She found that learning in the grocery store context took place through handling items, special offers and sales whilst interacting with shop workers and customers; and that this method of learning was more effective than that in the traditional classroom setting. In her review of Lave's work, Hansman (2001, p45) concludes "real‐world contexts, where there are social relationships and tools, make the best learning environments".

410

Alex Moseley

3. Real‐world contexts Kolb (1984) looked at these real‐world contexts in a different way: seeing the same learning value in them as Lave, but using them as experiences within formal education; applying existing knowledge to a real‐life context. Drawing on earlier theorists, Kolb constructed an experiential learning cycle that describes a continual loop of testing out knowledge in a real context (applying it), then reflecting on the success or failure of this experience (which generates new knowledge), then re‐applying this new knowledge to a real context, and so on through the loop. It is easy to apply Kolb’s cycle to familiar experiences: playing computer games, for example, often involves trying to complete a level, or fight an opponent, or solve a puzzle. By trying one tack, failing, reflecting, then trying a new approach, and continuing to try new approaches each time (building on the previous attempts) until we succeed, we are engaging in experiential learning. Traditional apprenticeships, of course, are true experiential learning experiences that place the learner firmly within a real‐world community of practice; but such apprenticeships have now become rare except in certain areas (manual trades, mainly) due to the cost to employers, high numbers of students needing places, and incompatibility with modern curricula. Farmer et. al. (1992) describe a modern version: cognitive apprenticeships, through a five‐stage model to integrate real‐world experiences within education. Through modelling (observing experts in methods and tools), approximating (trying out the methods and tools themselves, then reflecting on their performance: both with support), fading (support for the doing‐reflecting is reduced), self‐directed learning (practicing doing‐reflecting alone) and generalising (to other related applications), students get to experience immersion in the real world context and its methods, whilst also benefitting from reflective, supported developmental activity within an educational setting. What if the whole learning process is embedded within a real‐world context? This is the aim of authentic education, which started to emerge in the early 1980s, but by the 1990s was gaining widespread interest. Shaffer and Resnick (1999) analysed this area to come up with four types of thickly authentic (from Ryle's earlier classifications) educational experience:

activities that are aligned with the outside world;

assessment that is aligned with instruction;

topics that are aligned with what learners want to know;

methods of enquiry that are aligned with the discipline.

(after Shaffer & Resnick, 1999:197‐199) For a learning experience to be thickly authentic, all of these conditions have to be met. Shaffer later combined this work with Collins and Ferguson’s (1993) ideas of epistemic forms and epistemic games. They distinguished these as “the difference between the squares that are filled out in tic‐tac‐toe and the game itself” (p25). The epistemic forms are the squares or the structure, the underlying context; whereas the epistemic games are played out upon it with particular “rules, strategies, and different moves that players master over a period of time” (p25). Shaffer took these ideas and, applying them to the theories of authenticity and communities of practice, suggested that an epistemic frame can be used to describe the “practice, identity, interest, understanding, and epistemology” (2005, p1) of particular communities. Taking one example, “Lawyers act like lawyers, identify themselves as lawyers, are interested in legal issues, and know about the law. These skills, affiliations, habits, and understandings are made possible by looking at the world in a particular way ‐ by thinking like a lawyer” (p1) – an epistemic frame for a lawyer would then feature all of these aspects. Epistemic frames for other professions, or interest groups, would be different – but all would have rich descriptions of the particular group. Epistemic frames therefore provide a very useful context in which to base educational activity: if the learning is designed within the epistemic frame, it will all be embedded within the context of the subject or profession the student is working in or aiming towards. Shaffer describes this type of learning by extending the definition of epistemic games – activities that use methods, tools and approaches from the profession or subject in order to solve problems that reflect those a professional would have to deal with: “they make it possible for students to learn through participation in authentic recreations of valued reflective practices” (2005, p4).

411

Alex Moseley Using the example of lawyers, students might be given a typical real world briefing and asked to run a moot court, being assessed on their professional performance – all activities and assessment fitting within the epistemic frame. It should be noted that Shaffer’s (and Collins and Ferguson before him) notion of an epistemic game is not intended entirely literally: the ‘game’ being merely a set of rules, methods, approaches and strategies. However, Shaffer in particular has applied his ideas to designing playful experiences, or games in the literal sense, to increase engagement and teach concepts within an authentic context (e.g. the game Madison 2200, described in Shaffer 2005, p4, although he admits this is strictly somewhat closer to a simulation than a game).

4. Context, learning games and learning activities The use of real‐world contexts to create authentic experiences has the potential to be used within learning environments in a variety of ways. Some methods exist already in tried‐and‐tested forms: traditional and cognitive apprenticeships and epistemic games as previously described, but also more commonly the use of case studies or fieldtrips ‐ core aspects of many subject domains. Much of this existing practice goes unnoticed or unrecognised, however, and as a result research into the effectiveness of context within an educational curriculum ‐ already limited ‐ is narrowed still further in looking at the newer forms described above. Is it more effective to immerse students in real‐world content at the start of a course, in the middle, or at the end? Should theoretical frameworks or factual information be provided before the immersion in context, after, or not at all? How should the curriculum integrate with the contextual elements: should assessment be within or outside the contextual later? Do game‐based approaches (a feature of many of the recent forms of contextual learning) provide particular affordances in this area? These and many other research questions are crucial if formal education is to equip students with the correct skills for modern professions; to ensure that graduate attributes are matched to industry requirements; and to help us understand how new forms of learning and teaching (including the continuing development of games for learning) might help. In order to bring structure to this area of study, to provide focus for research into particular approaches, and to help educators choose from a clearer range of options when developing new contextual learning programmes, a typology of approaches has been developed. Through an analysis of existing methods, four principle uses of context have been identified, and their key features described. In the case of each, current approaches that map to the model are identified (with selected case studies where available), and further development areas suggested.

5. Four contextual learning models Model A: Viewing Context The simplest method of introducing context, and one that is easily applied over existing fixed curricula, centres around the use of 'snapshots' of real context in the form of case studies or examples. Almost always brought in within a curriculum after theoretical or informational learning, to exemplify certain approaches (or provide a real example on which to analyse or reflect), the contextual element tends to be a discrete package: a different narrative or presentation style to the surrounding learning and teaching, and not connected directly to the assessment format. It is also usually brought into the classroom or learning environment, rather than taking the students to the real context.

Figure 1: Model A ‐ viewing context

412

Alex Moseley Existing Approaches The use of case studies from real (or realistic) contexts is the dominant form, used in a wide range of subject domains and in a variety of forms. They may be transcripts of conversation, reports of a particular event, audio‐visual material, company reports or case histories, etc. Usually the case studies are presented, and learners are then asked to reflect on and analyse the content, with teachers focusing or guiding discussion to match the intended learning outcomes. Further development work The effectiveness of case studies in "bridg[ing] the gap between theory and practice and between the academy and the workplace" (Barkley et al 2005, p182) has already been studied at length; however, the form of embedding and transfer activity are both in need of further research. Are videos more effective at generating the real context for the learner, than text transcripts or other forms? Should the case studies be presented in a different way, time or place to the rest of the curriculum? What are the most effective forms of transfer activity, to draw out lessons from the case studies into the wider curriculum? Model B: Using Contexts as Seeds This model describes a range of approaches that, for a certain portion of the curriculum, immerse learners within a real or realistic context for a period. Immersion goes beyond case studies to involve the students in real aspects of the context (locations, methods, tools, decisions, events etc.). In order to link the contextual element with the curriculum, theory or knowledge might be delivered before the contextual portion; or themes within the real(istic) context might be drawn out to link with the curriculum as 'lenses'. For example, a 'lens' of teamwork might be used to focus on that aspect of a group fieldwork project ‐ learners would be provided with the 'lens' details before entering the contextual portion, and would have it in mind as they perform in context, making it easier to reflect on teamwork aspects afterwards.

Figure 2: Model B ‐ using contexts as seeds Existing Approaches On a long timescale, work placements, fieldwork or years in industry/professions are an example of this model. Programmes in medicine (with periods in the real hospital context), teacher training (with a period of teaching practice in schools) and modern languages (with a year abroad immersed in the country of native language) commonly use this model, with training in theory and skills before the immersion, and reflection on practice and performance following it. Lenses are also commonly used (trainee teachers are asked to focus on particular planning and management skills, for example). Learning games that model particular contexts and simulations yet aren't developed for a specific curriculum conform to this model: they provide a realistic context at a certain point within the curriculum, provide

413

Alex Moseley immersion for the learner, yet weren't designed with specific learning outcomes or curriculum in mind. One of the first such games to be used within a training environment was the Monopologs inventory management simulation for the American Air Force (Renshaw and Heuston, 1957), but many off‐the‐shelf games and those involving detailed simulations of contexts are delivered in this way. The author ran a series of sandpits: rapid research project developments for creative professionals in the museum sector, in which a full research proposal was developed within two days. At the start of the sandpit, participants were given four visitor types to consider (their lenses) and immersed in a normal morning in a local museum, entering each gallery as 'visitors' and using the lenses to guide their movement and browsing. They then returned to the classroom to reflect on their experiences through the same lenses, before current theory and practice in visitor studies was introduced. Further development work Within this model are a range of possible orientations: the contextual element could take a number of forms in time, type and position within the curriculum: some may be better suited for particular domains or learning outcomes; some may provide more effective transfer of theory through applied practice to analysis and reflection. There is already some work in the area of embedding games within the curriculum (Whitton 2009, pp77‐88; Moseley and Jones 2012, p112) and growing evidence that the use of a reflective/analysis stage after playing a learning game or simulation provides more effective learning. Further experimentation of forms, and research into the many variables, is needed however. Model C: Applying Context In this model an activity or game is designed that adds an authentic layer to a significant portion of the curriculum: although principles or theories may be taught in the classroom as preparation for the contextual element, and assessment might take place outside of the authentic layer, back in the classroom. Within the authentic layer, the curriculum is delivered using contextual features: learning scenarios or activities are taken from those in the real context, matched to the learning outcomes; locations, objects, and real or simulated characters from the context might be used. Often, a detailed narrative or scenario will help to embed curriculum needs within a contextual frame. If assessment takes place in context, the assessment would be contextually‐relevant (for example, if the authentic layer is drawn from an engineering firm, the assessment might involve the creation of a design proposal and presentation to a commissioning panel). If out of context, it would reflect on the principles and frameworks defined earlier in the programme, as applied to the new and applied knowledge resulting from the contextual element.

Figure 3: Model C ‐ applying contexts Existing Approaches Learning games that have been designed specifically for a programme (around learning outcomes, or to cover specific aspects of the curriculum) fit into this model: contextual elements within the games can be used to generate authentic environments, yet the design parameters ensure a close fit with curriculum aims. This includes Schaffer's epistemic games (although some fuller implementations may fall into Model D). Where an

414

Alex Moseley existing, off‐the‐shelf, game matches the curriculum requirements and required context closely, that may also fit into this model: an example is the Civilisation commercial game that covers the development of societies over time, and has been used in a number of programmes covering history, sociology and politics (see eg. Squire and Barab, 2004). A growing number of board and card games that generate authentic contexts through simple design elements, and are designed and embedded into a programme, are another solid example of the model (Moseley and Whitton, 2013). Case studies that include an interactive component (where learners play a part in shaping the outcome of the case study, or create their own in‐context case study through study and modelling of others) would also fit into this model; as would roleplaying activities, where learners are given contextual roles and asked to play out events or activities from the real context. A range of other authentic‐learning approaches fit this model: in experiential learning, the use of metaphors or frame stories to structure contextual activities (and often to include reflection and assessment) are an effective example (see eg. Hildmann and Hildmann 2009). At the University of Leicester, final year Physics students take on the role of real‐world physicists and consider a research problem from literature or film (for example, how many seagulls would actually be needed to lift Roald Dahl's James's giant peach? ‐ Flood, 2013): their activity (working as research groups) and assessment (submit an academic paper to a journal) both occur within the authentic layer. Further development work Due to the potential range of approaches that might fit this model, further research and development would most usefully be focussed on the interfaces between the curriculum and the authentic layer: how are principles, theories or frameworks effectively set up before learners enter the authentic layer? Within the layer, what are the most effective methods for guiding or focussing learning (narratives, metaphors, theoretical lenses etc)? Is assessment more effective when embedded in the context, or as an out‐of‐context test of application/reflection/analysis? Model D: Designing In Context In this model the curriculum is developed with the real context firmly in mind: learning objectives are fully aligned to a real profession or role. Learners are therefore embedded in the context from the first day of the programme to the last, with theory, knowledge, practice, reflection, assessment and feedback all presented in an authentic form. The curriculum therefore tends to be delivered through a contextual narrative, or one or more scenarios ‐ designed to introduce the key theories, knowledge and skills through experiential learning methods. The real context will be evident in all aspects of the programme: learner roles, equipment, 'tools' (digital and non‐digital: eg. email, SMS text, letters, written reports, websites etc.: whatever is normal within the real context), events, etc. Assessment is likely to be designed into the real context, in line with the rest of the programme (and may be continuous or at fixed points); although learners might be brought out of context to apply theoretical or analysing lenses to their performance, and be assessed on these reflective elements.

Figure 4: Model D ‐ designing in context

415

Alex Moseley Existing Approaches The dominant traditional form of this model is apprenticeship: apprentices follow an (often unwritten) curriculum that will provide them with all learning opportunities and assessment within the authentic environment. This is where learning games have proved most interesting in the past decade, however. The development of immersive and pervasive gaming forms and their application to education has seen them used as authentic curriculum delivery forms in areas such as induction (eg. the ARGOSI project at Manchester Metropolitan University: Whitton 2009a), language learning (Connolly et. al. 2011), enterprise education (Brookes et. al. 2012) and transferable skills (Squire and Jan 2007). Based around alternate reality games (Whitton and Moseley 2012, p143) or emerging hybrid forms such as pervasive learning activities (Brookes and Moseley, 2012), such game forms embed learners in deep authentic contexts, mixing reality with fiction but using the same authentic tools, methods and activities throughout. Further development work There has been much consideration of the merits of apprenticeship, and this approach has its own strong area of interest and research. The area of immersive or pervasive games in education is, however, a relatively new one. Some research has already considered their effectiveness within education, particularly in the area of motivation (Moseley et. al. 2009), but this has highlighted a tendency for only a small proportion of learners to engage fully in the game and therefore learning context: and this is clearly an area for further study and development of the form. Aside from study of existing forms is the potential for the development of new forms, both games‐based and non‐games‐based, that provide effective ways to develop and deliver curricula entirely within a real or realistic context. For these reasons, this is certainly the most interesting of the four models in terms of further development work.

6. Conclusion The four models are presented here for two primary purposes:

to map existing contextual learning forms onto a framework, and encourage the study of elements within the model (in particular the methods of transfer into and out of contextual elements, the design frameworks that map curricula to real contexts, and the assessment forms) as well as overall effectiveness of particular approaches.

to provide a range of models for those teachers interested in adding contextual learning to their own programmes, ranging from simple non‐embedded approaches (model A) through to new or redesigned programmes that are developed in tandem with a real context (model D).

In particular, the role of learning games in models B through D is deserving of further study: why is it that games lend themselves well to the more embedded forms of contextual learning? Are they good starting points for teachers keen to embed context in their programmes, and if so, what are the key design steps for embedding such context? Finally, it is hoped that through application and analysis using these models, the models themselves will be tested, refined, and if necessary added to.

References Barkley, E. F, Cross, K. P. & Major, C. H. (2005). Collaborative Learning Techniques: A Handbook for College Faculty. San‐ Francisco: Jossey‐Bass. Brookes, S. & Moseley, A. (2012) Authentic Contextual Games for Learning. In N. Whitton & A. Moseley (eds) Using Games to Enhance Learning and Teaching: A Beginner’s Guide. New York: Routledge, 91‐107. Brookes, S., Moseley, A. & Underwood, S. (2012). Contextual Games: Pervasive Learning Activities in Enterprise Education (a case study). Presented at the ISBE 2012 Conference, Dublin, Ireland, 7‐8th November. [online] http://www.pla‐ academy.co.uk/home/wp‐content/uploads/2013/02/Universityof_957_Full_paper_final_2012.pdf [accessed 2/5/2013].

416

Alex Moseley Collins, A. & Ferguson, W. (1993). Epistemic forms and Epistemic Games: Structures and Strategies to Guide Inquiry. Educational Psychologist 28:1, 25‐42. Connolly, T.M., Stansfield, M.H. & Hainey, T. (2011). An Alternate Reality Game for Language Learning: ARGuing for Multilingual Motivation, Computers and Education, 57:1, 1389‐141. Edwards, D., & Mercer, N. (1987). Common Knowledge: The Development of Understanding in the Classroom. London: Methuen. Edwards, R. & Miller, K. (2007). Putting the Context Into Learning. Pedagogy, Culture & Society 15:3, 263–274. Farmer, J. A., Buckmaster, A., and LeGrand Brandt, B. (1992). Cognitive Apprenticeship: Implications for Continuing Professional Education. In H. K. Morris Baskett & V. Marsick (eds) Professionals’ Ways of Knowing: Findings on How to Improve Professional Education. New Directions for Adult and Continuing Education, 55. San Francisco: Jossey‐Bass. Flood, A. (2013). Think again, Roald Dahl: scientists hit on giant peach of a theory. The Guardian, 4 January 2013. [online] http://www.guardian.co.uk/books/2013/jan/04/roald‐dahl‐scientists‐giant‐peach [accessed: 1/5/2013]. Hansman, C.A. (2001). Context‐Based Adult Learning. New Directions for Adult and Continuing Education 89, 43–52. Hildmann, J. & Hildmann, H. (2009). Promoting Social Skills Through Initiative Games in the Classroom and Assessing Their Effects. In Proceedings of the 3rd European Conference on Games Based Learning: FH Joanneum University of Applied Sciences, Graz, Austria, 12 ‐ 13 October 2009. Reading: Academic Conferences, 180‐194. Johnson, E. B. (2002). Contextual Teaching and Learning. Thousand Oaks, CA: Corwin Press. Kolb, D.A. (1984). Experiential learning : experience as the source of learning and development. London: Prentice‐Hall. Lave, J. (1996). The Practice of Learning. In S. Chaiklin & J. Lave (eds) Understanding Practice : perspectives on activity and context. Cambridge: Cambridge University Press, 3‐34. Lave, J. (1998). Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life. Cambridge: Cambridge University Press. Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: University of Cambridge Press. Mercer, N. (2000). Words and Minds: How we use language to think together and get things done. New York: Routledge. Moseley, A. & Jones, R. (2012) Mapping Games to Curricula. In N. Whitton & A. Moseley (eds) Using Games to Enhance Learning and Teaching: A Beginner’s Guide. New York: Routledge, 108‐123. Moseley, A. & Whitton, N. (eds) (2013). New Traditional Games for Learning: A case book. New York: Routledge. Moseley, A., Whitton, N., Culver, J. & Piatt, K. (2009). Motivation in Alternate Reality Gaming Environments and Implications for Learning. In Proceedings of the 3rd European Conference on Games Based Learning. Graz: Academic Conferences. Renshaw, J. R. & Heuston, A. (1957). The Game Monopologs. Santa Monica, CA: RAND Corporation, 1957. [online] http://www.rand.org/pubs/research_memoranda/RM1917‐1 [accessed: 1/5/2013]. Russell, R. R. (2002) Looking beyond the interface: activity theory and distributed learning. In M. Lea & K. Nicholl (eds) Distributed Learning: social and cultural approaches to practice, London, Routledge with The Open University, 64–82. Ryle, G. (1968). The Thinking of Thoughts. In University Lectures, The University of Saskatchewan, Vol 18. Shaffer, D.W. (2005). Epistemic games. Innovate 1/6. [online] http://www.innovateonline.info/index.php?view=article&id=79 [accessed 20/4/2013]. Shaffer, D.W. & Resnick, M. (1999). ‘Thick’ Authenticity: New Media and Authentic Learning. Journal of Interactive Learning Research 10:2, 195‐215. Squire, K. & Barab, S. (2004). Replaying history: engaging urban underserved students in learning world history through computer simulation games. In Proceedings of the 6th international conference on Learning sciences (ICLS '04). International Society of the Learning Sciences 505‐512. Squire, K. & Jan, M. (2007). Mad City Mystery: Developing Scientific Argumentation Skills with a Place‐based Augmented Reality Game on Handheld Computers. Journal of Science Education and Technology, 16:1, 5‐29. Vygotsky, L.S. (1978). Mind and society: The development of higher mental processes. Cambridge, MA: Harvard University Press. Wenger, E. (1998). Communities of Practice: Learning, Meaning, and Identity. Cambridge, UK: Cambridge University Press. Whitton, N. (2009). Learning with digital games. A practical guide to engaging students in higher education. New York: Routledge. Whitton, N. (2009a). Alternate Reality Games for Orientation, Socialisation and Induction. [online] http://argosi.playthinklearn.net/ [accessed 28/4/2013]. Whitton, N. & Moseley, A. (eds) (2012). Using Games to Enhance Learning and Teaching: A Beginner’s Guide. New York: Routledge.

417

Let the Students Contruct Their own fun And Knowledge ‐ Learning to Program by Building Computer Games Peter Mozelius1, Olga Shabalina2, Christos Malliarakis3, Florica Tomos4, Chris Miller4 and David Turner4 1 Stockholm University, Sweden 2 Volgograd State Technical University, Russia 3 University of Macedonia, Greece 4 Glamorgan University, Wales mozelius@dsv.su.se Abstract: Computer programming is a core subject in most Computer science programmes at university level but many students have difficulties with both the understanding of theoretical concepts and with the learning of practical programming skills. Several studies have pointed out that there exist pedagogical and motivational problems resulting in high drop‐out rates and low learning outcomes. On the other hand today’s students have grown up in a digital and internet connected world where playing computer games is a common and appreciated spare time activity. It has been discussed during many years amongst teachers and researchers which important features a programming language should have to support learning and which the appropriate programming paradigms are for introductory programming courses. Less has been discussed and written about modifying the actual course content and use the recognised attraction and catalytic effect of computer games. The aim of this case study is to describe, analyze and discuss the concept of learning to program by game construction. In this case study two course analyses have been combined with a literature review on pedagogy for Game Based Learning (GBL Findings from the literature review show that the idea of students constructing knowledge in their interactions with their environment is not a new one and that games have been used in educational contexts long before the introduction of computers. The idea of Game based learning has support in the constructivist learning theory concept that was introduced theoretically at university level in the 1970s by Jean Piaget and Lev Vygotsky. Results from the study also indicate that the practical application of the game construction concept has been successful in the two investigated programming courses both when it comes to student motivation and learning outcomes. Furthermore, this game based pedagogy can motivate students not only to develop themselves as future programmers but also to become more innovative and entrepreneurial by improving the quality and performance of the games in order to deploy, promote and sell them. Keywords: game‐based learning, GBL, computer games, programming education, constructivism

1. Introduction In most Computer science programmes at university level programming is still a core subject but several studies on programming education show that many students are facing difficulties even in the understanding of basic programming techniques (Guzdial & Soloway 2002) (Järvinen, Ala‐Mutka and Lahtinen, 2005) (Eckerdal, 2009). This is not only related to theoretical programming concepts and research studies show that there are problems with the practical programming and code construction as well (Eckerdal, 2009). Some researchers see this mainly as a pedagogical problem but maybe as a psychological and motivational issue as well (Wiedenbeck, Labelle & Kain, 2004). Drop‐out rates from programming courses are high in general (Guzdial & Soloway 2002) and there is no consensus on how programming should be introduced. Some researchers have strongly argued for an early introduction of object‐oriented techniques and concepts (Kölling & Rosenberg, 1996) (Cooper, Dann & Pausch, 2003) meanwhile others completely reject the idea of object‐orientation for introductory courses (Decker & Hirshfield, 1994). It has been widely discussed amongst teachers and researchers which important features a programming language should have to support learning (Mannila & De Raad, 2006), and which programming paradigms that are appropriate for introductory programming courses (Vujusevic‐Janicic & Tosic, 2008). Less has been discussed and written about modifying the actual course content and use the recognised attraction of computer games. Many students find programming education difficult and at the same time boring. Programming courses often have a focus on syntax and algorithms without considering differences in students’ learning styles. (Jenkins, 2002) A solution recommended in studies is to redesign courses and give them a more flexible outline to allow different students to learn in different ways. According to (Fasli et al, 2009) a successful learning environment

418

Peter Mozelius et al. should provide a combination of challenge and entertainment for the students. What effect the playing of computer games could have on students’ ability to successfully execute given tasks has been studied studied by Pillay (2003). He claims that the completion of tasks through educational‐based computer games can increase the students’ cognitive abilities Playing games as a pedagogical approach was introduced at university level in the 1970s by Jean Piaget (1973) and Lev Vygotsky (1978) but a broader discussion on playing started earlier in 1938 when the Dutch historian and cultural theorist Johan Huizinga published his ideas on man as a Homo Ludens (Huizinga, 1938) where play is described as a basic cultural phenomenon. There is no model that successfully integrates educational theory and game design aspects but idea use of integrating flow theory with learning theory in computer games have been investigated by the Finnish researcher Kristian Kiili (2005). He has also studied how students can improve their learning process when they produce their own multimedia learning objects (Kiili, 2006). The use of Game Based Learning (GBL) in education has generally increased during the last decade but there is a risk for negative consequences with a stereotypical gamification (Kapp, 2012). There are two different approaches in GBL for programming education. In the first one the main idea is that students learn to program by constructing digital games. By using imagination and creativity in order to build their own executable computer games they will at the same time learn traditional programming techniques. In the second approach, they learn programming by playing educational or serious games where programming skills and knowledge can be learnt from the actual gaming (Ljungkvist, & Mozelius, 2012). This study will have a focus on the first approach: Learning to program by game construction.

1.1 Problem A dilemma in many Computer science programmes today is how to engage novice students in programming courses (Järvinen, Ala‐Mutka and Lahtinen, 2005) (Eckerdal, 2009) and that the courses at the same time must provide sound instruction in traditional programming techniques (Guzdial & Soloway 2002) (Haden, 2006). There is no clear answer to how programming courses should be designed to obtain a higher student engagement and at the same time keep the learning outcomes at a high level but research indicates that students find the programming courses boring (Jenkins, 2002) and that the generally low pass rates and learning outcomes might depend on students’ lack of motivation (Wiedenbeck, Labelle & Kain, 2004).

1.2 Aim The aim of this study is to describe and discuss the design and outcome of some programming courses based on the concept of learning to program by game construction.

2. A constructivist view of learning and teaching This study will consider the construction of computer games as an educational method to gain knowledge about basic programming by the novice students with different levels of basic knowledge and a manner to increase their motivation. For this reason, in 1970 Piaget emphasised the role of “knowing how” within the evolution of cognitive functions. Piaget (1970; 1980) argued that the intelligence and the cognitive development are in fact two different and concurrent processes, although related phases. Furthermore, Piaget (1970) argued that the constructivism of knowledge is an act of “creation of novelty”. Based on the above justification, this research suggests that the knowledge is determined by students’ creativity in learning programming by games construction. Piaget’s theory about the construction of knowledge supported also substantial criticism. Thus, referring to Piaget’s opinion on learning and child internal representation of things, Fox (n. d.) criticised the idea of general and synchronized development of all areas of the mind and indicated that recent studies demonstrated that knowledge develops independently in different areas of experience. However, Piaget (1970) argued that the pressure of the environment upon individuals requires the need of accommodation. Furthermore, an increase in knowledge happens when the individual ensures an equilibrium between accommodation and assimilation, in other words, adaptation (Piaget, 1970). Moreover, Piaget (1970) defined the intelligence as the assimilation, or the process of including his/her experience within a framework. Additionally, Fox (n. d.) recommended Vygotsky’s research on individual mind. Vygotsky (1978: 24) discovered that “...practical and abstract intelligence, occurs when speech and practical activity...converge”. Nevertheless, was Piaget (1970:

419

Peter Mozelius et al. 398) who defined intelligence as “...the development of an assimilatory activity... [whose] ...structures are elaborated by the interaction between itself and the external environment”. As a consequence, student’s mind develops independently according to their own experience. However, their intelligence progresses when social interaction by communication and practice takes place at once. Additionally, Piaget (1970) emphasised the role of active experimentation within the discovery process. This indicates that students’ active experimentation and communication emerge into knowledge by discovery and innovation. Piaget (1970) argued that within the sensori‐motor intelligence phase, a child explores the new situation. Similarly, a student explores a new task, in order to assimilate the knowledge. However, the mature student requires a period o concrete operation in order to be able to face “the formal operation period”. When the student deals with with a new situation, he/she has to find appropriate means to achieve his/her target. Thus, students will search for well‐known means as initial basis of knowledge, acquired previously and which they can use successfully to solve a new problem (Piaget, 1970). Vygotsky’s theory on games to support learning The use of games to support learning was introduced theoretically at university level in the 1970s by Jean Piaget (1973) and Lev Vygotsky (1978). Vygosky’s (1978) highlighted the existence of three main theoretical positions regarding the relations between child developments and learning process. The first theoretical viewpoint assumes that the learning process is an independent process. The second presumes that the learning and development are mixed, and the third opinion saw the process of learning as embedded within the development process. Further, Vygotsky (1978) recognised the relationship between learning and development and differentiated between the actual development and the possible development under guidance. This study will use construction of computer games in order to learn programming and will build on students’ basic information and knowledge, ensuring differentiation to reach the student potential development in the real world (Piaget, 1970,1980; Vygotsky, 1978). Vygotsky (1978) approached the importance of games and motivation these can give. Vygotsky also defined the meaning of action and play and explained the games as the prediction of future situations. Consequently, Vygotsky tried to present the role of imaginary situation for further learning and development. This, Vygotsky said, will assist student imagination, creativity and innovation. According to Kaffka (Vygotsky, 1978) the play is an imaginary world that can be transferred into life. It has practical consequences for student development and trust in approaching more confident situations within the real world. Methodology and Methods The overall approach for our research is the case study strategy where a case study is defined as an empirical inquiry investigating a real world contemporary phenomenon (Yin, 1989). Case studies are a strategy where the researchers explore and evaluates a programme, process or activity in depth using a combination of methods for the data collection (Creswell, 2009) in a setup where the different sources of evidence together should generate a deeper understanding of the investigated phenomenon (Remenyi, 2012). Case study research is more than any other kind of research relying on both primary and secondary data (Remenyi, 2012) and in this study the primary data from the two embedded case units are completed with secondary data from a literature study on constructivist pedagogy and game‐based learning. Case Study Unit 1 – Volgograd The experiment involved 40 students in total with an age of 18‐22, who studied software engineering during four years and got a bachelor degree after finishing their study. All the students studied development process using project‐oriented approach. 13 of them wished to choose educational games as their study projects. The results of students’ study were collected and analysed. All the courses that cover software development were included into the analysis (Programming Languages, Data Bases, Computer Graphics, Theory Translation and others, 25 in total). Students obtained scores in the range [61, 100] according to the Russian grading standard for every course they studied. Data has been collected from the teachers of corresponding subjects. Average improvement of scores during study period was measured. According to accomplished ANOVA analysis, the average improvement in a group of students involved into educational games development was higher (+3.81 versus +1.38) and this increase was statistically significant. The probability that all the students belong to the

420

Peter Mozelius et al. same group and this increase is accidental was computed with T‐test t=0.0483 is less than 0.05, which means that these two evaluated groups are significantly different with p=0.95. Case Study Unit 2 – Stockholm The focus in this unit has been on the course Multimedia programming in Python that is described in detail under 2.3. Data has been collected in a combination of semi‐structured interviews, document analyses and an analysis of student’s postings in on‐line forums. Two students that have taken the course at different time periods were selected for interviews. One of them had earlier knowledge of Python and the other had to learn Python from the beginning. The student that had to learn Python from scratch in now working in a company that develops and sells computer games, the other student is using Python to build an artefact for his Bachelor thesis. Both informants were committed when they took the course but they were selected because of their enthusiasm and not because they were top students in their course batches. The conducted interviews are semi‐structured and the informants will in this article be referred to as Student 1 and Student 2. Interviews were both recorded as mp3‐files with a successful result but as a backup notes were taken by pen and paper as well. Document analyses are based on the course syllabus and course statistics from 2009 ‐ 2012 retrieved from DSV’s internal information and course management system Daisy (Daisy, 2012). Student’s course evaluations in the Daisy system have been compared with student discussions in the Moodle virtual learning environment at: https://ilearn.dsv.su.se

3. Learning to program by game construction Case Study Unit 1 – Volgograd State Technical University Computer‐Aided Design department at Volgograd State Technical University graduates specialists in computer‐ aided systems design and development. The most part of special courses starting from the third year require a high level of programming skills. So thus students start learning programming languages from the very beginning: they study programming languages C, C++ during the spring semester of the first year and the autumn semester of the second one. After finishing the second year studying and passing the exams all the students are required to pass a month training course (normally in July). The main objective of this course is to gain practical skills in programming by doing some simple but real‐world tasks. Each student (or a small team of students) gets an individual task and works on it during one month under supervision of a teacher. During the third and the fourth year students use their skills while fulfilling different projects included in corresponding courses dealing with software development components like data bases, artificial intelligence and computer graphics. Over the last several years educational games have been used for teaching programming (Shabalina, Chickerur, 2010). A game‐based approach is integrated to the all stages of the learning process. The teaching methodology is based on a two‐fold use of the games. Games for teaching programing developed by students are used for teaching the next generation of students. After gaining basic skills in programming these students are again involved in the development of new educational games. Learning programming language by the use of games Existing games for training programing skills such as Colobot and its modification CeeBot (Epsitec Games, 2001), Robocode (Robocode, 2001), Robot Battle (Garage Games, 2002) and others are based on an idea of writing programming code and using it for controlling the behavior of some game objects. Thus the game process includes two types of action – training in programming and game actions that occur sequentially. It means that training programming in such games is almost the same as in traditional learning. In order to train skills in a game context an approach of fulfilling learning and gaming task in parallel (Shabalina et al., 2012) is applied in games for training programming skills The idea is based on writing programming code as step‐by‐step composition of a sequence of lexemes. To do this what a player must do at each step is to choose a lexeme from a set of choices which include at least on correct lexeme (from the perspective of fitting the possible task decision). Each lexeme is matched to a game object, and the rules of game are interpreted in such a way that a player could compose programming code from the lexemes by controlling the game objects. Step‐by‐step composition of programming code helps students to learn programming language syntax and understand the code structure and to keep their interest in playing a game.

421

Peter Mozelius et al. Examples of games for training basic programming skills The set of games for training basic programming skills was developed by second and third year students and present different realizations of the idea of composing programming code from separate lexemes in a game context. Popular mini‐games were used as game prototypes as these mini‐games are familiar to almost all players. Then the players do not have to learn the game rules because they already know them, they are ready to learn through that game immediately. In order for the player to compose the code for a task one step at a time a set of alternative lexemes is presented to the player at each step. The set includes some correct lexemes (one or more depending on the game options) and a set of lexemes that can be chosen randomly from a database of possibilities. The total number of lexemes depends on certain game rules. The set of alternative lexemes is dynamically generated as the set of correct lexemes on the next step depends on the lexeme chosen by a player on the previous step. After a player has chosen the next lexeme to add to the program code it is checked as a partial task solution. If the lexeme chosen by the player matches some correct solution it displays on the output window, if it doesn’t, the player is penalised, and gets a new set of lexemes to choose from. Screenshots presented in Figure 1 show some games for training programming skills developed by the students.

1a) The CSnake game

1b) The CRace game

Figure 1: Screenshots of games for training programming skills developed by students Case Study Unit 2 – Stockholm University At the Department of Computer and Systems Sciences at the Stockholm University there is a tradition of building computer games in programming courses, and after the establishment of a Computer Game Development Bachelor’s programme games are now seen as more constructive that distractive. But in a standard programming course on a Bachelor’s or Master’s programme there is not that much of games or game construction and programming techniques are taught and learned by more traditional exercises and assignments. University programmes in Sweden are a combination of spring and autumn semester, but in the last decades the number of summer courses given in a shorter and more informal summer trimester has increased. A summer course is seldom part of a curriculum or aligned to any programme and can for that reason be designed a bit different and more experimental. The course that will be described here is Multimedia programming in Python, a course where game construction with multimedia techniques is the way for students to learn important basic imperative programming techniques and how they should be implemented in the Python programming language. The is a distance course given in the Moodle environment where streaming lectures and tutorials combined with 2 course books provides the information for the students independent work with four assignments and a game project. Course Design Practically everything in the course is based on analysing and building games with two course books where all code examples are runnable computer games. The first book Python Programming for the Absolute Beginner (Dawson, 2010) introduces basic programming concepts like variables, selection, iteration, Boolean condition

422

Peter Mozelius et al. and data structures in commented code examples of classical computer games like Hangman and Tic Tac Toe. The involved programming techniques are also explained in streaming lectures and exemplified in recorded online lessons and tutorials. As in many other game‐themed introductory programming courses the first start‐ up assignment is relatively simple and on basic integer and floating‐point number arithmetic (Sung, 2008). Assignment 1 can later in the students’ project be extended to a mathematical educational game. In Assignment 2 the students should, with help from the course literature write code for the well‐known computer game Guess my Number (BFGL, 2010, a kid’s game where the player have to use the Computer science principle of ‘Divide and Conquer’ to get a good score. In Assignment 3 this should be enhanced with a graphical user interface built by the use of the Tkinter library (Lundh, 1999) that is an integrated part of the Python language. Assignment 4 introduces core Python multimedia features for the construction of a slideshow where images combined with sound stored in a data structure should be combined with a navigation system. For students that find the final project a difficult task Assignment 4 can later, for a grade of D or C, be extended to a multimedia quiz. For the grades of A and B the game idea must be more complex or involve techniques from outside the course syllabus.

Figure 2: Flag quiz (screen shot from a student project) To provide techniques and ideas for more complex game construction in the final project there is a second course book More Python Programming for the Absolute Beginner (Harbour, 2012) that introduces multimedia features in the add‐on library pygame. Like in the main book by Michael Dawson (2010) practically all chapters in the book are built around the Python code for complete and executable games. Most games in the book are classical computer games like Snake and Block Breaker. Compared to programming education in general the described course outline could be classified as more constructivist but not as pure constructivism. The frequent use of games and game construction is meant to support the Vygotskian ideas of stimulating learners’ imagination, creativity and innovation by games and play. Assignments are composed with a mix of closed and open exercises where the more closed assignments with a given game design should provide training of traditional programming techniques. The final open project assignment where students’ should design their own games is on the other hand given to stimulate students’ active experimentation and creation of knowledge by discovery and innovation in the spirit of Jean Piaget.

4. Findings and discussions The tradition of using games in educational contexts is much older than Computer science and several well described examples exist on how games and computer games can be used in teaching sessions and self

423

Peter Mozelius et al. learning. Theoretically, the concept of using games at university level was introduced in the 1970s by Jean Piaget and Lev Vygotsky. Game Based Learning for programming education can be divided into two main approaches. In the first one, students learn by constructing computer games when they use their creativity to design and implement. The second approach is based on students learning to program by playing games classified as educational games or as serious games. Students that participated in the experiment at Volgograd State Technical University and developed educational games expressed their opinion about the project. They consider game development as a very inspiring thing to do. The development of educational games helped them a lot in practicing skills they learned during studying. Also, most of the students noticed that they were motivated to learn many new things that are not included in the standard curriculum. Analysis of using game development for teaching programming shows that students gained knowledge and skills that to a considerable extent extend the educational program, and raised their qualifications. Students who participated in game development projects gained important professional skills such as dealing with real projects, software design, testing, debugging and development work with open libraries, Version Control Systems (VCS) and other modern tools, and also working with somebody else's code. Students who participated in game projects gained soft skills such as team working, project management, priority assignment and conflict resolution. Students and graduates have created two small enterprises and obtained financial support from the Russian government. The companies are actively developing and produce regular and educational games for different platforms – PCs, mobile devices and games for social nets. As a result, students have been able to gain work placements during their study. Compared to other summer courses given in distance mode at the Department of Computer and System Sciences Multimedia, programming in Python has the highest past rate ever. The course has been given four times with pass rates between 54 – 66 percent which is a good result even when the course is compared with other courses than just programming courses. Most students are males (around 90%) and between 20 to 35 years but in the last course batch the oldest participant was 69 years old. Most students classify the difficulty level as average, and a bit easier than programming courses in general. But with variations since many students have chosen to develop complex games in the final project. What the interviewed students liked most about the course was the freedom to design and develop their own game idea in the final project. Student 1 mentioned that some of his classmates chose the “easy way out” and completed the final project quite quickly by building some of the given alternatives for less complex games. On the other hand, he and other students spent several months on the design and implementation of games that had features that were not replicated by other games. Student 2 said that he appreciated the freedom to plan and implement his own unique game project in a way that has similarities with how he is now working for a software company where his main occupation is to construct computer games that sometimes are less complex than the game he built in the course project. Both students said that what might be missing in the course are assignments on the Python model for object‐orientation but, as an afterthought, they mentioned that this might be something for another separate course. On the final question regarding what parts of courses that teach Computer Science programs can be gamified, both students were a bit hesitant. However, after a while they both said that this would be an appreciated element in several courses, not necessarily in all.

5. Conclusions The use of games to support learning is an old concept, far older than the history of Computer science and programming education. As several pedagogues and researchers has found, games and gaming can be used in education to raise engagement. In the 21st century where students have grown up with computers, virtual environments and with digital game playing as part of their everyday life, the use of computer games in education seems to be this century’s learning through entertainment. Game Based Learning for programming education can be divided into two main categories. The first category is a pedagogical approach based on students learning by playing educational and serious games. In the other category discussed in this article students develop their programming skills by designing, implementing and testing their own computer games. Both described programming courses contain components that have roots in other pedagogical theories but the main common approach is constructivism. Courses on computer science and programming are like most courses at university level, built by a mix of pedagogical ideas, but to improve

424

Peter Mozelius et al. the students’ theoretical knowledge as well as their practical skills we believe that making is thinking and effective learning. The discussed concept of learning to program by game construction seems promising and our recommendation is that the described approach is worth to implement in more programming courses.

References Creswell, J. W. (2009) “Research Design, Qualitative, Quantitative and Mixed Methods Approaches”, Sage Publications Inc, ISBN: 978‐1‐4129‐6557‐6 Daisy (2012) ”DSV's internal information system” (retrieved 20/04/2013) http://dsv.su.se/en/education/comp/daisy/ Dawson, M. (2010) "Python Programming for the Absolute Beginner" Course Technology, CENGAGE Learning, ISBN10: 1435455002 Decker, R. & Hirshfield, S. (1994) "The top 10 reasons why object‐oriented programming can't be taught in CS 1" ACM SIGCSE Bulletin, 26 (1) Eckerdal A. (2009) “Novice Programming Students' Learning of Concepts and Practise”, PhD Thesis Uppsala University, available at http://www.avhandlingar.se/avhandling/6809751ebf/ Epsitec Games (2001) “Colobot” http://www.ceebot.com/colobot (retrieved 18/04/2013) Fasli, M. & Michalakopoulos, M. (2006) “Interactive Game based Learning”, Association for Learning Technology Newsletter, 1(6). Guzdial, M. & Soloway, E. (2002) “Log on education: teaching the Nintendo generation to program” Communications of the ACM, 45(4), pp. 17‐21. Haden, P. (2006) “The incredible rainbow spitting chicken: teaching traditional programming skills through games programming", ACE '06: Australian conference on Computing education Harbour, J. S. M. (2012) "More Python Programming for the Absolute Beginner" Course Technology, CENGAGE Learning, ISBN10: 1435459809 Huizinga, J. (1955, originally published in 1938) “Homo Ludens: A Study of the Play Element in Culture” Beacon Press, Boston Jenkins, T. (2002) “On the difficulty of learning to program”, In Proceedings of the 3rd Annual LTSN_ICS Conference, The Higher Education Academy, pp. 53‐58. Kapp, K. M. (2012) ”The Gamification of Learning and Instruction: Game‐based Methods and Strategies for Training and Education”, John Wiley & Sons, ISBN: 978‐1‐118‐09634‐5 Kiili, K. (2005). “Digital game‐based learning: Towards an experiential gaming model” The Internet and Higher Education, 8, 13–24. Kiili, K. (2006) ”Towards a participatory multimedia learning model”, Education and Information Technologies, 11(1), 21‐32. Kölling, M. & Rosenberg, J. (1996) "An object‐oriented program development environment for the first programming course", SIGCSE '96 Proceedings of the twenty‐seventh SIGCSE technical symposium on Computer science education, 83‐87 Lahtinen, E., Ala‐Mutka, K., Jarvinen, H. (2005) “A Study of Difficulties of Novice Programmers” In: Innovation and Technology in Computer Science Education 2005, pp. 14–18. Ljungkvist, P., & Mozelius, P. (2012) ”Educational Games for Self Learning in Introductory Programming Courses‐a Straightforward Design Approach with Progression Mechanisms” , Proceedings Of The 6th European Conference On Games Based Learning, ECGBL 2012 (pp. 285‐293) Lundh, F. (1999) “An Introduction to Tkinter”, (retrieved 15/03/2013) http://www.pythonware.com/library/tkinter/introduction/ Mannila, L. & De Raad, M. (2006) “An Objective Comparison of Languages for Teaching Introductory Programming” Baltic Sea '06 Proceedings of the 6th Baltic Sea conference on computing education research: Koli Calling 2006 Petley, R., Parker G. & Attewell, J. (2011) “The Mobile Learning Network: Getting Serious about Games Technologies for Learning", International Journal of Game‐Based Learning (IJGBL), Volume 1, Issue 4 Piaget, J. (1970) “The Principles of Genetic Epistemology”, Routledge & Kegan Paul Ltd., London Piaget, J. (1973) "To understand is to invent”, Grossman, New York (Original work published 1948) Piaget, J. (1980) “The origin of Intelligence in the Child”, Penguin Education, UK Pillay, H. (2003) “An investigation of cognitive processes engaged in by recreational computer game players: Implications for skills of the future.” Journal of Research on Computing in Education, 34(3) Remenyi, D. (2012) “Case Study Research”, Academic Publishing International Limited, Reading, United Kingdom ISBN: 978‐ 1‐908272‐40‐9 Robocode (2001) “Robocode” http://robocoderepository.com/ (retrieved 04/01/2011) Shabalina, O. & Chickerur S. (2012) "Competence‐based approach for effectively teaching software development curricula", Izvestiya VolgGTU, " Chapter Actual problems of control, cvomputing and informatics in technical systems", Volume 1, Issue 15 (102), p. 102‐111. Shabalina O., Vorobkalov P., Kataev A., Davtyan A. & Blanchfield P (2012) "Development of Computer games for Training Programming Skills", Proceedings of the 6th European Conference on Games Based Learning, Cork, Ireland , 4‐5 October 2012 / The University College Cork And Waterford Institute of technology, Ireland.‐ Cork, 2012.‐ P. 460‐471.

425

Peter Mozelius et al. Sung, K., Panitz, M., Wallace, S., Anderson, R. & Nordlinger, J. (2008) ”Game‐themed programming assignments: the faculty perspective”, 'SIGCSE '08: Proceedings of the 39th SIGCSE technical symposium on Computer science education' , ACM, New York, NY, USA Vujusevic‐Janicic, M. & Tosic, D. (2008) “The Role Of Programming Paradigms In The First Programming Courses” The Teaching Of Mathematics, XI (2) Vygotsky, L.S. (1978) "Mind in society: The development of higher psychological processes." Cambridge, Harvard University Press Wiedenbeck, S., Labelle, D. & Kain, V. N. R. (2004) "Factors affecting course outcomes in introductory programming", 16th Annual Workshop of the Psychology of Programming Interest Group Vygotsky, L.S. (1978) "Mind in society: The development of higher psychological processes”, Harvard University Press, Cambridge Yin, R. K. (1989 ‐ 2008) ”Case study research: Design and Methods”, Sage, Thousand Oaks

426

Towards Understanding the Instructional Value of Real‐Time Continuous Feedback From the use of Simulation Games Mathews Nkhoma, Jaime Calbeto, Narumon Sriratanaviriyakul, Thu Yein Win, Quyen Ha Tran and Thanh Kim Cao RMIT University Vietnam, Ho Chi Minh City, Vietnam mathews.nkhoma@rmit.edu.vn jaime.calbeto@rmit.edu.vn narumon@rmit.edu.vn thuyein.win@rmit.edu.vn quyen.tran@rmit.edu.vn thanh.cao@rmit.edu.vn Abstract: Simulation games have long been used as a teaching tool in the classroom environment mainly due to the high level of participation and engagement that students are able to generate from these, making the learning process more enjoyable and capable to replicate real‐life scenarios. When all is said and done, students are rewarded with a more authentic and complete learning experience. Feedback given during the simulation helps to motivate students to find better solutions to the problems being presented throughout the games and thus enhance their hands‐on knowledge on particular subjects. The purpose of this research is to provide empirical evidence of interrelations and impacts that exist between real‐time continuous feedback and simulation game performance as well as the interrelations and impacts that exist between real‐time continuous feedback and both students’ attitude and engagement towards learning. The research comprised 60 undergraduate students enrolled at the Centre of Commerce who had undergone at least three semesters of studying at various programmes at RMIT University Vietnam. For test purposes, the research employed a 3D IBM Business Process Management (BPM) simulation game, INNOV8 (more information is available at http://www‐ 01.ibm.com/software/solutions/soa/innov8/index.html) developed by IBM Academic Initiative. Students in the sample were asked to play this simulation game. A web‐based survey followed at the conclusion of the simulation game for the collection of data. The findings of the research concluded that students showed a favourable attitude towards learning through the simulation game. In addition, the real‐time continuous feedback given during the simulation game had a positive impact on the students’ cognitive learning outcomes. The originality of this research stems from the nature of the feedback being given to students in a real‐time continuous basis during the gameplay of a computer‐based simulation game to examine how this impacts students’ learning outcomes. Keywords: real‐time continuous feedback, simulation games, game‐based learning, serious games

1. Overview of simulation games Games are a form of entertainment and enjoyment, which have long had a negative connotation due to its richness in storyline, graphics, engagement, and interactivity, with a potential to cause serious addictions leading to health exhaustion in young adults, one case taking place in South Korea (Sutter, 2012). Nevertheless, a significant amount of research has also been carried out to examine the positive impacts in playing computer‐based games, one of which is the educational value (Connolly et al., 2012). A study on game‐ based learning by de Freitas (2006) shows that when there are clear educational objectives and goals, then games can be used effectively as a tool for teaching and learning, particularly Serious Games. This key point is supported by Zyda (2005), saying that Serious Games, with the implementation of theories and principles in the gameplay, break away from the conventional definition of games by adding to it an academic dimension. Michael and Chen (2006) as cited in Susi et al. (2007) reaffirm this pedagogic aspect of Serious Games emphasizing that the outcome of the game‐playing process is to have students actually learn from the game‐ play experience. Simulations games were known as the earliest application of educational games (de Freitas, 2006). With the inclusion of the entertainment aspect, simulation games offer users hands‐on experience from its open‐ended replication of real‐world scenarios and tasks which ask users to tackle the challenges given during the game play and find solutions that can improve the outcomes in the end (Susi et al., 2007; Gredler, 2004). Gredler (2004) raises three reasons why simulation games provide educational value and strengthen students’ learning experience, making it not only a popular teaching tool in academic but also in professional working environments. Firstly, simulation games require players to solve real‐life issues, thus helping to bridge theories with reality; secondly, simulation games help to spot out students’ misunderstanding and confusion about the

427

Mathews Nkhoma et al. abstract theories learnt in class leading to clarification or further explanation; and lastly, simulation games give students guidance to find better solutions when they face the problems in real life.

2. Positive impacts on learning outcomes A lot of researchers have examined the effectiveness of simulation games and how they impact on students’ learning outcomes. Interaction with simulation games provides a student‐centered learning environment because of its active and pragmatic approach to obtaining new knowledge and enhancing skills (Lainema, 2009, Mayer et al., 2010). An experiment carried out by Tan and Seng (2010) using computer games to teach difficult‐to‐understand IT concepts and algorithms was able to discover that students favored learning from interacting with simulation games and that they could comprehend and perceive the abstract concepts better. Ke (2009), on the contrary, failed to find a relationship between students’ engagement and attitudes towards simulation games with the learning outcomes when employing mathematical games to compare the effectiveness of computer games with the traditional teaching method when it came to boosting students’ math learning outcomes. Specifically, findings showed that although students had favorable attitudes towards learning through simulation games, there was no significant difference in the learning outcomes when comparing computer games and the traditional teaching method. The lack of objectives when evaluating the knowledge, skills that players obtain from simulation games and whether these meet the learning outcomes is pointed out by Anderson and Lawton (2007, 2009). Consequently, there is a need to examine and find out how and to what extent students’ attitudes towards simulation games could impact their learning outcomes. From what has been stated, the first hypothesis is proposed: H1: A positive correlation exists between the students’ performance on the simulation game and the students’ attitude and engagement toward learning through the simulation experience. Wouters et al. (2009) proposed a taxonomy of learning outcomes based on the authors’ literature review examining students’ interaction with Serious Games and its impact on learning experience with the following categories: cognitive skills, motor skills, affective skills, and communicative skills; with cognitive skills being studied the most by researchers (Connolly et al., 2012). The process of employing the knowledge and theories learnt in class to tackle problems given during the simulation makes the participants more likely to achieve desired learning outcomes. Anderson and Lawton (2007) conducted a research to examine the connection between students’ performance on the simulation games and their attitudes towards the use of simulation games as a learning tool along with their cognitive learning outcomes. Despite the success of the research, the sample size of only 25 students was too small to find correlations between students’ achievement in the games and their learning outcomes. Furthermore, the simulation game used in Anderson and Lawton’s research is designed in a way that students could easily achieve high results by making guesses without basing answers on knowledge. As a result, it is worth investigating whether a true level of knowledge learnt by the application of theories can be reflected through the students’ performance at the conclusion of the simulation games. In this research, a more feedback‐driven simulation called INNOV8, a 3D IBM Business Process Management (BPM) simulation game was incorporated. In this simulation, players are presented with business model problems and clues on how to solve them. When players make the wrong decisions, feedback and directions are provided to assist them in choosing the right ones in order to move forward with the game. The simulation is designed in a way that students will have to apply the theories they have learnt so as to achieve high scores rather than through guesswork. Furthermore, with 60 students participating in the research, the sample size was larger than the one from Anderson and Lawton’s (2010) making the quantitative findings more representative and accurate to the larger population (Vogt, 2007). From what has been stated, a second hypothesis is proposed:H2: A positive correlation exists between the students’ performance on the simulation game and the students’ perception of how much they learned from the simulation experience.

3. Feedback in simulation games It is emphasized by my researches that an effective framework and design for simulation games should incorporate a supportive feedback system. Garris et al. (2002) identifies the motivation for learning stems from

428

Mathews Nkhoma et al. the feedback provided while interacting with the simulation game, making the flow of the gameplay engaging and challenging and thus encouraging participants to perform better in order to achieve higher scores. Gobet et al. (2004), as cited in Dunwell et al. (2010), and Gredler (2004) also agree with this point saying that feedback motivates players to follow the flow of the game until the end and helps them achieve the desired learning outcomes. There are several justifications for feedback being crucial to the students’ achievement of the intended learning outcomes. Firstly, feedback helps students realize how far they are from their current performance to their desired learning outcomes and thus motivates them to learn harder in order to bridge that gap (Song & Keller, 2001). Secondly, students can be ambiguous when trying to comprehend new concepts, but with the timely guidance given by feedback, it makes students feel more confident and motivated in tackling problems given during the simulation (Paas et al., 2003; Moreno, 2004). Finally, feedback helps fine‐tuning students’ understanding of the new theories they learn in class to the right direction and thus, reducing students’ misconceptions while boosting their learning outcomes and performance (Mason & Bruning, 2001; Goodman et al., 2004; Mory, 2004; Narciss & Huth, 2004). One aspect of feedback that many researchers have vested time in examining is the timing feedback being given and its relationship to the learning experience and efficiency in learning (e.g., Gentry, 1990; Black and William, 1998; Clariana et al., 2000; Dihoff et al., 2004; Schooler et al., 2008; Epstein et al., 2010; Dihoff et al., 2012). Nonetheless, there is still an ongoing debate as to which type of feedback is more effective: immediate feedback which is given instantly after an action is made or delayed feedback which is given only after all the tasks have been completed for some time. Despite the inconsistent findings on feedback timing, it is argued that immediate feedback is more effective in the classroom context than delayed feedback (e.g., Dihoff et al., 2004; Jarvis and De Freitas, 2009; Brosvic et al., 2010). This is mainly because when struggling or when introduced to new theories, students are likely to have misconceptions about ideas introduced from the new lessons and thus less confidently examine problems with the the right understanding in how to solve them. Therefore, immediate feedback will adjust their understandings timely and improve the ability of students to give correct responses after making the initial incorrect ones (Mory, 2004). In this research, INNOV8 was used to examine students’ learning performance. INNOV8 is a simulation game that provides immediate feedback, or as referred in this research as ‘real‐time continuous feedback’. During the simulation, students will be presented with business problems and every time they submit their answers, they will be given feedback throughout the simulation until they reach the conclusion of the game. Consequently, it is essential to examine how using Serious Game in providing real‐time continuous feedback as a teaching method can affect students’ final game performance. From what has been stated, a third hypothesis is proposed: H3: A positive correlation exists between the students’ performance and the students’ perception of how much real‐time continuous feedback they receive from interacting with the simulation experience.

4. Research methodology Quantitative research method was employed in this research. The sample included 60 students who volunteered in teams of two to four players each to play INNOV8, a 3D IBM Business Process Management (BPM) simulation game, for a period of about 30 minutes. Participants were then requested to answer an online survey adapted from Rowe and Wood consisting of twelve closed‐ended questions as well as one optional open‐ended question (Rowe and Wood 2008). After the survey, data was collected and analyzed via SPSS Platform version 20. Linear regression is used to model the value of a dependent variable based on its linear relationship to one or more predictors in order to test the three hypotheses. This technique requires the input of the following quantitative data (as indicated in Table 1): One dependent variable, i.e. perceived performance from playing the game, represented as Measure 1, and various single independent variables (students’ attitude and engagement in learning from playing the simulation game represented as Measure 2; students’ perception of how well the simulation game reflected the disciplines of courses studied at RMIT Vietnam represented as Measure 3; students’ perception of how much real time continuous feedback they are receiving from the

429

Mathews Nkhoma et al. simulation game represented as Measure 4; and students’ preferred form of feedback when interacting with the game represented as Measure 5). The measures mentioned above are multi‐dimensional, thus they need to be grouped by factor analysis. To do so, Cronbach’s alpha coefficient was used to evaluate the scale reliability and also provide information about the relationships between individual items in the scale. Kaiser‐Meyer‐Olkin (KMO) and Bartlett test were used to measure sampling adequacy. Factor analysis was used in data reduction to identify a small number of factors explaining most of the variance been observed in a much larger number of observed variables. These factors were then used for hypothesis testing through regression analysis.

5. Data analysis Cronbach’s alpha coefficient was generated to evaluate the scale’s reliability and validate the consistency of items within the scale. According to Hair et al., the reliability coefficient is considered acceptable if it meets the value of 0.7. The Cronbach’s alpha coefficients shown in Table 1 reveal that the reliability of the scales to be acceptable since all of the coefficients are over 0.7. Table 1: Cronbach’s alpha coefficients Measure / Factor 1 2 3

Description Students’ perceived performance Students’ attitude and engagement toward learning based on 6‐scale Students’ perception of how well the simulation game reflected the discipline of courses studied at RMIT Vietnam

Number of Items 7 6 7

Cronbach’s Alpha .773 .775 .817

Students’ perception towards how much real‐time continuous feedback they received while playing the game

.730

Students’ preferred forms of feedbacks when interacting with the game

.798

Kaiser‐Meyer‐Olkin (KMO) and Bartlett test were used to measure sampling adequacy in order to determine whether the sample size is sufficient for factor analysis. Achieving a KMO value of 0.682 and a Bartlett’s Test Significance value below 0.001 (as seen in Table 2) consents for the dataset to be used for factor analysis. Table 2: KMO and Bartlett’s test Kaiser‐Meyer‐Olkin Measure of Sampling Adequacy.

.682

Bartlett's Test of Sphericity

Sig.

.000

The average number was then used to perform factor analysis. According to Hair et al., “recalling the concept of a summated scale, which is formed by combining several individual variables into a single composite measure” and “more commonly the average score of the variables is used as a replacement variable” because of two specific benefits that this approach provides: (1) "means for overcoming to some extent the measurement error", and (2) “ability to represent the multiple aspects of a concept in a single measure". Therefore, the factor analysis rendered 5 factors:

Factor 1: Students’ perceived performance (Feedback)

Factor 2: Students’ attitude and engagement toward learning based on 6‐scale (Anderson)

Factor 3: Students’ perception of how well the simulation game reflected the discipline of courses studied at RMIT Vietnam

Factor 4: Students’ perception towards how much real‐time continuous feedback they received while playing the game

Factor 5: Students’ preferred forms of feedbacks when interacting with the game

6. Hypothesis testing Five measures in terms of the five factors were used in this study. To drive toward the research outcomes, testing of the three hypotheses would follow the model below:

430

Mathews Nkhoma et al.

Figure 1: Research model Four regression analyses were employed in order to test the hypotheses proposed in this research, with results being summarized in Table 3. The first regression examined the relationship between students’ performance and their attitudes towards learning; the second between students’ performance and their perception of how much they learn; the third between students’ performance and their perception of real‐time continuous feedback received; and the last one between students’ performance and their preferred form of feedback when interacting with the game. Table 3: Regression analyses Hypothesis

Beta t‐value p‐value R ‐square

.327 .383

3.158

.003

.147

.454 .419

3.514

.001

.176

.541 .449

3.794

.000

.202

.317 .274

2.155

.035

.075

All of the four regressions are significant at the p<0.05 level (F‐value = 9.974; 12.350; 14.394; 4.642), confirming the integrity of the models. The coefficients of R square reveals that, ‘Attitude towards learning’ explains 14.7%, ‘Perception of how much they learn’ explains 17.6%, ‘Perception towards how much real‐time continuous feedback provided by interacting with the game’ explains 20.2%, and ‘Preferred forms of feedbacks when interacting with the game’ explains 7.5% in the variation for performance. All predictors are significant in explaining the positive relationships with the dependent variable which is the measure for performance, confirming H1, H2, and H3. As indicated in the regression analysis outcomes in Table 3, there is adequate statistical evidence to prove the fact that all the p‐values are under 0.05 and all the Beta coefficients are positive (0.383 for Measure 2; 0.419 for Measure 3; 0.449 for Measure 4; 0.274 for Measure 5) . Amongst these factors, it can be seen that ‘Perception towards real‐time continuous feedback provided by interacting with the game’ has the biggest positive effect on Performance. Of lesser but significant importance are the ‘Perception of how much they learn’ and the ‘Attitude towards learning’. Least significant is the ‘Preferred forms of feedbacks when interacting with the game’.

7. Discussion This research is carried out with the purpose to give the exploratory evidence of the correlations and the impacts of feedback, specifically real‐time continuous feedback, and simulation games along with the correlations of real‐time continuous feedback on students’ attitude and participation towards learning through the use of simulation game. As mentioned, Anderson and Lawton in 2007 also conducted a similar research but failed to find out the effectiveness of teaching and learning through the use of simulation games, chiefly because of the small sample size of only 25 students and the predictability of the questions employed in their survey, which allowed students to achieve high scores easily through guesswork. With a larger sample size of 60 students and a different simulation game that is more feedback‐driven in which the students are required to apply what they learn in class so as to obtain good performance, the results of this research statistically prove that there is a positive relationship between students’ game performance and their favorable attitudes

431

Mathews Nkhoma et al. towards the use of simulation games as a teaching method. Among 60 participants, 62% say that they had a good time playing the game and 55% state that the game was engaging and stimulating. The study’s results also confirm that students perceive stimulation games as a valuable method to help them improve their learning performance. Moreover, responses from the participants show that when students were presented with business problems during the simulation, they had to apply the theories they had learnt in order to move on to the next level and thus, the theories became clearer to them. With 93% of the respondents stating that they felt motivated to find better ways to tackle the business problems in order to achieve higher scores on their second play, it is a good sign that students’ cognitive learning outcomes can be positively driven by interaction with the learning medium of simulation games. As previously mentioned, the preference for using immediate feedback in the classroom context to encourage and positively influence students’ learning experience as well as findings from this research demonstrating the importance of feedback to students specifically via the incorporation of real‐time continuous feedback, not only confirms the hypotheses proposed in this research but also supports findings from previous studies about the preference of using immediate feedback in the classroom context to encourage students’ learning experience (e.g., Dihoff et al., 2004; Jarvis and De Freitas, 2009; Brosvic et al., 2010) as well as the reasons for the importance of feedback to students as discussed (Locke & Latham, 1990; Song & Keller, 2001; Mason & Bruning, 2001; Paas et al., 2003; Ashford et al., 2003; Mory, 2004; Narciss & Huth, 2004). The hypothesis testing has proven that students’ performance is most positively affected by ‘Students’ perception towards how much real‐time continuous feedback students are receiving from playing the game’. Statistics show that 65% of the respondents felt motivated to study more when they received feedback, and 85% reported that they could achieve better results thanks to the feedback provided. Undoubtedly, feedback was perceived as important by most of the participants and thus, they were obliged to take notice and study all the feedback provided during the gameplay. Furthermore, students’ ambiguity and anxiety during the simulation were lessened thanks to the feedback given which also motivated students to learn more. As a matter of fact, 87% of participants reported that feedback helped them to be aware of their current performance and therefore make better decisions in order to reach higher scores.

8. Conclusion The primary limitation consisted principally in the relatively small sample size. The research outcome is based on data collected from a total of 60 students participating in this study, and therefore the research outcomes might not be a true representative sample of the total student population studying at RMIT University Vietnam. A second limitation, also involving the demographics of the sample, consisted in permitting students from a variety of majors to take part in this research. Since this research did not use the game to test the knowledge learnt in the classroom but rather exposed them to a totally unfamiliar environment, perhaps, the results could be more significant if students who played the game had taken courses related to Business Process Management. Understanding with more precision the value that students can gain from the application of real‐time continuous feedback in the classroom, specifically in the form of computer‐based simulation games, requires more exploration. A follow‐up project to this research could incorporate a more defined sample of students enrolled in a Business Process Management course. From findings uncovered from this type of sample, we can then assess more clearly the value gained by students from real‐time continuous feedback toward better comprehension of the theory of the course content and hence the achievement of the learning outcomes in a course that has more to do with Business Process Management.

References Anderson, P. H. & Lawton, L. 2007. Simulation Performance and Its Effectiveness as A PBL Problem: A Follow‐up Study. Developments in Business Simulation and Experiential Learning, 34. Anderson, P. H. & Lawton, L. 2009. Business Simulations and Cognitive Learning : Developments, Desires and Future Directions. Simulation & Gaming, 40, 193‐216. Black, P. & Wiliam, D. 1998.Assessment and classroom learning.Assessment in education, 5, 7‐74. Brosvic, G. M., Epstein, M. L., Dihoff, R. E. & Cook, M. J. 2010. Acquisition and retention of Esperanto: The case for error correction and immediate feedback. The Psychological Record, 56, 4. Clariana, R. B. 1999. Differential Memory Effects for Immediate and Delayed Feedback: A Delta Rule Explanation of Feedback Timing Effects

432

Mathews Nkhoma et al. Connolly, T. M., Boyle, E. A., Macarthur, E., Hainey, T. & Boyle, J. M. 2012.A systematic literature review of empirical evidence on computer games and serious games.Computers & Education, 59, 661‐686. De Freitas, S. 2006. Learning in immersive worlds.London: Joint Information Systems Committee. Dihoff, R. E., Brosvic, G. M. & Epstein, M. L. 2012. The role of feedback during academic testing: The delay retention effect revisited. The Psychological Record, 53, 2. Dihoff, R. E., Brosvic, G. M., Epstein, M. L. & Cook, M. J. 2004. Provision of feedback during preparation for academic testing: Learning is enhanced by immediate but not delayed feedback. Psychological Record, 54, 207‐232. Epstein, B. B. & Brosvic, G. M. 2010. Immediate feedback assessment technique promotes learning and corrects inaccurate first responses. The Psychological Record, 52, 5. Garris, R., Ahlers, R. & Driskell, J. E. 2002. Games, Motivation, and Learning: A Research and Practice Model. Simulation & Gaming, 33, 441‐467 Gentry, J. W. 1990. What is Experiential Learning.Guide to business gaming and experiential learning, 9‐20. Goodman, J. S., Wood, R. E. & Hendrickx, M. 2004. Feedback specificity, exploration, and learning.Journal of Applied Psychology; Journal of Applied Psychology, 89, 248. Gredler, M. E. 2004. Games and Simulations and Their Relationships to Learning.Handbook of research on educational communications and technology, 2, 571‐581 Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E. 2010.Multivariate data analysis 7th edition. Upper Saddle River, NJ : Prentice Hall. Ke, F. 2008. Computer games application within alternative classroom goal structures: cognitive, metacognitive, and affective evaluation. Educational Technology Research and Development, 56, 539‐556. Lainema, T. 2009. Perspective Making Constructivism as a Maning‐Making Structure for Simulation Gaming. Simulation & Gaming, 40, 48‐67. Mason, B. J. & Bruning, R. 2001. Providing feedback in computer‐based instruction: What the research tells us. 15. Available: http://dwb.unl.edu/Edit/MB/MasonBruning.html [Accessed 15 November 2012]. Mayer, B. W., Dale, K. M., Fraccastoro, K. A. & Moss, G. 2010.Improving Transfer of Learning: Relationship to Methods of Using Business Simulation.Simulation & Gaming, 42, 64‐84. Moreno, R. 2004. Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery‐based multimedia.Instructional science, 32, 99‐113. Mory, E. H. 2004. Feedback research revisited. Handbook of research on educational communications and technology, 745‐ 783. Narciss, S. & Huth, K. 2004.How to design informative tutoring feedback for multimedia learning.Instructional design for multimedia learning, 181‐195. Paas, F., Renkl, A. & Sweller, J. 2003. Cognitive load theory and instructional design: Recent developments. Educational psychologist, 38, 1‐4. Rowe, A. D. & Wood, L. N. 2008.Student Perceptions and Preferences for Feedback.Asian Social Science, 4, 78‐88. Schooler, L. J. & Anderson, J. R. 2008.The disruptive potential of immediate feedback. Shute, V. J. 2008. Focus on formative feedback. Review of educational research, 78, 153‐189. Song, S. H. & Keller, J. M. 2001. Effectiveness of motivationally adaptive computer‐assisted instruction on the dynamic aspects of motivation.Educational technology research and development, 49, 5‐22. Susi, T., Johannesson, M. & Backlund, P. 2007. Serious Games ‐ An Overview. Sutter, J. D. 2012. Wired for success or destruction? [Online].CNN. Available: http://edition.cnn.com/interactive/2012/08/tech/gaming.series/korea.html [Accessed 15 November 2012]. Tan, B. & Seng, J. L. K. 2010. Game‐based Learning for Data Structures: A Case Study. International Conference on Computer Engineering and Technology.IEEE. Vogt, W. P. 2007. Quantitative research methods for professionals, Pearson/Allyn and Bacon. Wouters, P., Van Der Spek, E. & Van Oostendorp, H. 2009. Current practices in serious game research: A review from a learning outcomes perspective. Games‐based learning advancements for multi‐sensory human computer interfaces: techniques and effective practices, 232‐250. Zyda, M. 2005. From visual simulation to virtual reality to games.Computer, 38, 25‐32.

433

Learning Math as you Play: Comparing Arithmetic Performance Enhancement Induced by Game Play and Paper Exercises Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy Department of Communication Sciences, iMinds‐MICT‐Ghent University, Belgium ElenaPatricia.NunezCastellar@Ugent.be Anissa.All@Ugent.be J.Vanlooy@Ugent.be Abstract: One of the promises of video game training is that, compared to traditional training, it can be more engaging and entertaining (Boot et.al., 2008). However, besides entertainment, games have shown to have the potential to impact a larger variety of cognitive abilities. Previous research has consistently shown that several aspects in cognition such as visual short‐memory, multitasking and spatial cognition can be enhanced by game play. In a previous study, we found that playing Monkey Tales, a commercial game aimed at training arithmetic skills in children, helped second grade pupils to increase their accuracy in mental calculation as compared to paper exercises or no exercises. The present study aimed to explore how arithmetic performance enhancement induced by game play and paper exercises differs. In order to do this, we compared the performance gains that second graders achieved in a computer test made for assessing their math skills. We performed a combined analysis of the changes in two behavioral measurements: accuracy and reaction times. Children were tested at two points in time: before and after the three week period. We compared the reaction times and the accuracy improvements between these two moments and compared different items types (e.g. understanding tenths, understanding hundreds, even or odd up to 100 among other types). We found indirect evidence suggesting that arithmetic performance enhancement induced by game play and paper exercises might rely on slightly different cognitive mechanisms. Keywords: arithmetic training, mental calculation, educational game, traditional training, reaction times, accuracy

1. Introduction Video games are one of the more interesting and promising means to improve cognitive abilities, particularly with children. One of its promises is that, compared to traditional training, can be more engaging and entertaining (Boot et al., 2008), and effective (Wouters et al., 2013 ). Moreover, besides entertainment, games have the potential to impact a larger variety of cognitive abilities. Recently, research has consistently shown that several aspects in cognition such as visual short‐memory, multitasking and spatial cognition can be enhanced by game play (for a complete review, see Bavelier et al., 2012). In a previous study (Nuñez Castellar et al., submitted), we reported that playing Monkey Tales, a commercial game aimed at training arithmetic skills in children, helped second grade pupils to increase their accuracy in mental calculation as compared to paper exercises or no exercises.. However, the extent to which the positive changes induced by gaming or by paper exercises differ in its nature and characteristics is an issue that has not yet been explored. Specifically, based on previous research showing that video game playing can enhance working memory capacities and attention (Bavelier et al., 2012), in the present paper we explore whether by a detailed, combined analysis of the changes in accuracy and reaction times after game training and traditional training by means of math paper exercises, we can provide more informed description of how arithmetic performance enhancement induced by these two methods might differ. Specifically, there are reasons to believe that arithmetic performance enhancement induced by game play might be modulated by improvements in the domains of attention and working memory. Working memory is the ability to explicitly maintain a mental representation of some amount of information, while being engaged simultaneously in other mental processes (Baddeley, 2000). Research has demonstrated that working memory capacity increases from preschool through the elementary school years. Preschool children can hold three to four items of information, such as numbers, in working memory, whereas a typical fourth grader can hold five to six items (Kail, 1990). Although during the past decades it was traditionally assumed that working memory is highly heritable and unlikely to be influenced by environmental experience and opportunity (see Campbell et al., 1997), recent findings have provided evidence suggesting that children’s working memory can be enhanced by means of training (Klingberg et al., 2005; Holmes, Gathercole and Dunning, 2009; Turley‐Ames and Whitfield, 2004).

434

Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy Remarkably the study of Holmes, Gathercole and Dunning (2009) has demonstrated that attention training can lead to a significant boost of the academic mathematics performance of children. This study showed that IQ scores (both verbal IQ and performance IQ scores) did not show a comparable boost after working memory training, suggesting that, rather than leading to global performance enhancement, improvements in working memory seem to act locally, boosting arithmetical performance. Moreover, studies with clinical populations indicate the existence of a close relationship between working memory capacities and mathematical skills. For instance, studies investigating children with a mathematics learning disability(MD), have shown that they receive diminished scores on a variety of working memory tasks when compared with their same age pairs (Hitch and McAuley, 1991; McLean and Hitch, 1999; Siegel and Ryan, 1989; Swanson, 1993). Furthermore, recent studies have reported evidence suggesting that working memory and attention can be trained in normal adults by means of video gaming. For instance, it has been found that video game players are faster and more accurate in the monitoring and updating of working memory than non‐ video game players (Colzato et al., 2012). Green and Bavelier (2003) conducted a series of experiments on the effects of video game playing on visual attention comparing action video game players and non‐video game players, and found that video game playing experience enhances the capacity of the players’ visual attention system. Likewise several recent studies have demonstrated that action video game players have the ability to switch faster between tasks compared with non‐video game players (Karle, Water and Shedden, 2010, and Boot et al., 2008). Finally, a recent study has shown that performance gains are not restricted to the action game genre, but that playing Tetris, a casual puzzle game, can also improve attention, working memory and visuo‐spatial ability in young adults (Nuchi et al., 2013). Taken together, the results mentioned above suggest the existence of a close link between, working memory, attention and arithmetic skills, and that, remarkably, these cognitive abilities can be trained by means of game play. This creates important opportunities for using games for mathematics training but also questions as to how these different performance gains are related and how they compare with traditional methods for practicing mathematics. Hence, in the present study, we explore whether traditional methods and game training differ in terms of the cognitive processes that both are able to impact. In order to do this, we compared the results that second graders achieved in a test made for assessing their math skills. We conducted a combined analysis of the changes in accuracy and reaction times whilst considering different item types (e.g. understanding tenths, understanding hundreds, even or odd up to 100, etc.). Moreover, we explored whether the type of item that showed the largest improvements differed between game training and the traditional training group. Finally, results are compared with the ones of a group that did not receive any assignment (control group).

2. Methods 2.1 Participants Participants were drawn from a previous study. Overall arithmetic performance in a math test, as well as subjective measures like math anxiety, enjoyment and perceived competence from this sample have already been reported (Nuñez Castellar et al., submitted). However, that report did not examine individual reaction times and accuracy rates per type of arithmetical problem or item type. Children were recruited by sending letters to schools in the area of Ghent, Belgium. The parents interested in participating, registered via the Computer‐Aided Registration Tool for Experiments (CORTEX) (Elson and Bente, 2009). Parents gave written informed consent for their child’s participation. Children were tested at two points in time: before and after the three‐week period (Pre‐ and Post‐testing). During the first evaluation 88 second graders were tested. At the second moment of measurement, 84 were assessed (one child could not participate because of illness and three parents did not react to the repeated calls for post‐testing). From this sample some participants were excluded from the analyses because they were clinically diagnosed with disorders listed in the Diagnostic and Statistical manual of Mental disorders (DSM‐IV) (American Psychiatric Association, 2000), namely learning disability, ADHD, and dyslexia. Also participants who could not complete the task assignment and all participants who performed the computer math test at chance level or below, either in the pre‐ or the post‐test, were excluded from the analyses. In the present study data of 74 children are reported. As can be seen in Table 1, the groups did not differ significantly in terms of age, gender or game and study habits (see Table 1).

435

Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy Table 1: Socio‐demographic data and study and game habits by group

Educational Game (N=25)

Paper Exercises (N=23)

Control (N=26)

Male gender Age

n 18 Mean 7.52

n 15 Mean 7.26

n 18 Mean 7.35

Chi² 0.26 F 1.33

p .88 p .27

Median 4 4

Chi² 1.30 4.51

p** .52 .10

Mean 2h 02min 0h 53min 3h 41min

Mean 2h 10min 1h 08min 3h 42min

Mean 1h 30min 1h 05min 3h 11min

F 0.87 0.49 0.48

p .42 .61 .62

2h 48min

2h 23min

2h 21min

0.53

.59

Level education parents Education level father* Education level mother* Study and game habits Homework hours per week Math homework hours per week Gaming hours during the week Gaming hours during the weekend

*Four levels: Primary = 1, Junior High School/Middle School = 2, High School = 3, College/University = 4. ** Independent Sample Kruskall‐Wallis Test

2.2 Design Children were randomly assigned to three groups. One group was instructed to play through the entire educational game Monkey Tales in three weeks’ time (gaming group). A second group was instructed to complete a set of math drill exercises in the same period, equivalent in quantity and basic level of difficulty to the exercises in Monkey Tales (paper exercises group). Additionally we included a group that did not receive any assignment (control group).

2.3 Stimulus material 2.3.1 Educational game We used the 3D video game Monkey Tales (Larian studios, 2011), which exists in different versions for second to sixth grade and is used to support the learning of math. The main goal of this educational game is not to instruct but to improve mental arithmetic of children by motivating them to engage in drill exercises with increasing time pressure. Only by being faster than a monkey (artificial intelligence) they can go through all the game levels. Importantly, the game uses an algorithm that tries to establish where a child is on the learning curve, and then stimulates the child to make progress by progressively augmenting the difficulty of the exercises. For the present study we selected the Museum of Anything, which is meant for children in the 3rd grade (ages 8+) to repeat what they have learned in the 2nd. The educational game is divided into chapters and levels in which the player has to solve 3D puzzles (moving something that blocks the way or neutralize a laser for instance) and is challenged by a monkey to take part in a minigame (an educational math exercise in classic game format, e.g. 2D shoot ‘em up) which the player has to win to get to the next level (see figure 1). The game contains 42 basic and one final level. In order to complete all the levels of the game, children need to finish 322 math exercises whereby the exact number depends on how many times they need to replay a minigame because of mistakes.

436

Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy

Figure 1: Screenshot Monkey Tales 2.3.2 Paper exercises As one of the goals of the present study was to compare Monkey Tales with paper exercises, the latter needed to be as similar as possible to the former. Therefore, the educational publisher of the game, Die Keure, provided us with exercises based on their educational method that are equivalent in basic level of difficulty to the exercises included in Monkey Tales. Of around 1000 exercises we received, a sample of 340 exercises that were representative of the Belgium math curriculum for second graders were selected. The exercises were organized in ascendant order of difficulty (as it is done in the educational game), and were given to the parents of the children in a folder that they gave back to us at the post‐test.

2.4 Measurement of math performance: accuracy and speed Two equivalent versions of exams (test A and test B) for assessing the math skills of children of the second grade were provided by publisher Die Keure. These test were based on the academic curriculum for second grade in Belgium. We used the questions of these two tests to program a computerized version that allowed us to automatically measure not only the accuracy rates but also the reaction times of each item of the test in milliseconds. We programmed this computerized version using Tscope. Tscope is a C/C++ experiment programming library for cognitive scientists. It provides functions for graphics, sound, timing, randomization and response registration (Stevens et al., 2006). After a number of practice trials to become familiar with the multiple choice task, all children performed the computer math test in the pre‐ and post‐session. In each group, half of the children performed test A as pre‐test measurement and test B as post‐test measurement. The other half performed the tests in the opposite order.

2.5 Procedure The participants were tested at the beginning of May 2012 for the pre‐test session and at the end of May for the post‐test session. As previously described, children were randomly assigned to three groups. One group of children was instructed to finish the educational game Monkey Tales in three weeks’ time (Gaming group). Parents were instructed to help with the software installation and support the children while playing the game tutorial. However, they were explicitly asked not to help children with the math exercises. Moreover, the parents were asked to monitor on a weekly basis how far the children had progressed in the game, and to motivate them to play if needed. Importantly they were briefed about how to check the progress and detect when children had completed all the levels of the game. Finally, one week before the post‐test an e‐mail was sent as a reminder that, by the end of the week, the children should have completed the game. A second group of children was instructed to complete a set of math drill exercises on paper in three weeks’ time of equivalent quantity and basic level of difficulty as the exercises in Monkey Tales (Paper exercises group). Similarly to the parents of the group of children that was asked to play the educational game, the parents of this group were instructed to check and motivate the children to do the math drill exercises, but not to help them. One week before the post‐test, an e‐mail was sent as a reminder that by the end of the week children should have completed the math drill exercises. Additionally, we included a group that did not receive any assignment (Control group) but served as a comparison. The parents of this group were asked not to change anything from their normal routine and specifically they were asked not to let their children play any educational math games.

437

Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy All parents received the instruction to let the children continue to do their math homework as usual. The group that completed the paper exercises and the control group received the educational game at the end of the post‐test as a reward. All the parents were rewarded with 15 euros for their participation.

2.6 Data analysis 2.6.1 Accuracy improvements per item type In order to investigate whether the items in which the largest improvement was observed differed between groups (e.g. understanding tenths, understanding hundreds, even or odd up to 100, divide into equal parts, multiplication tables from 2 up to 10, we compared the accuracy in the pre‐test and the post‐test measurements for each item type. All the items of the Math test were included in the analyses except one for which in the pre‐test session 100% of the children gave a correct answer and therefore no improvement was possible. Afterwards we made a ranking based on the percentage of improvement including all the items to identify the ones in which the largest accuracy improvement was registered. 2.6.2 Correlation between accuracy and reaction time improvements Similarly to the accuracy improvements analyses, we compared the reaction times in the pre‐test and the post‐ test measurements for each item type. Only reaction times for correct responses were included in the analyses. Additionally, too fast reaction times ( < 300ms) and reaction times slower than 60 seconds were excluded. After calculating the reaction time improvements, we investigated the relationship between the accuracy improvements and the reaction time improvements. Therefore we performed a correlational analysis separate for each group: Monkey Tales, paper exercises and control. A significance level of 0.05 was used.

3. Results 3.1 Accuracy improvements Figure 2 shows the accuracy improvements per item type ordered by percentage of improvement for the three groups whereby the improvement of the control group can be considered as a baseline to interpret the improvement of the two experimental groups. As can be observed in the figure, when considering only the items ranked in the first positions, the graph shows that the largest accuracy improvement is observed for the group that made the paper exercises. For instance, the first ranked items on that group show an accuracy improvement above 30%. However, interestingly, when considering the pattern more globally, the results show that overall playing Monkey Tales lead to a larger sustained accuracy improvement in most of the items included in the math test. When we look at the number of items scoring accuracy improvements of 6% and above, we see that there are 18 for the Monkey Tales group as compared to only 13 in the math exercises group.

Figure 2: Accuracy improvements per item type

438

Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy

3.2 Correlation between accuracy and reaction time improvements Since not only accuracy but also reaction times can be analyzed to investigate performance enhancement we conducted correlation analysis between these two behavioral measures. The results revealed that for the group that played monkey tales, accuracy and reaction time improvements were strongly correlated r(37) = .46, p < .001. Likewise, for the group that made the paper exercises, reaction time improvements were correlated r(37) = .33, p < .05, but less strongly than for the game group. Additionally, when analyzing the relationship between accuracy and reaction time improvements for the control group, the association between these two measurements was found to be non‐significant r(37) = .08, p = 61. Based on the correlation analysis we selected for each group the 5 items in which the largest accuracy and reaction time improvement were observed. The results of this ranking can be found in table 1. As can be observed, there are important differences between the types of items that are ranked for each of the three groups. The results revealed that doing the paper exercises predominantly lead to improvements on items where second graders were required to solve addition problems. Playing Monkey Tales, on the other hand, lead to improvement in a variety of items including items that require second graders to make parity judgments and rehearse the multiplication tables of 7 and 8. The graph also shows the ranking for the group that got no assignment (control group). Table 2: Ranking items with the largest accuracy and reaction time improvements T = tens and U = units Ranking

MONKEY TALES Even or odd up to 100

PAPER EXERCISES CONTROL Divide into equal parts Multiplication table 7 (e.g. Find the correct number: (e.g. select the odd number) 12= . + . + . ) (e.g. 7 x 5) Tens and units Addition TU+TU up to 100 with Addition TU+TU up to 100 regrouping with regrouping (e.g. Which number has 7 (e.g. 55 + 29 = ) (e.g. 55 + 29 = ) units?) Multiplication table 8 Addition TU+TU up to 100 with Subtraction with 1 multiple of regrouping 10 up to 100 (e.g. 8 x 8) (e.g. 55 + 29 = ) (e.g. 86 – 10 = ) Multiplication table 7 Addition TU+TU up to 100 with Subtraction TU+TU = T regrouping (e.g. 7 x 5) (e.g. 55 + 29 = ) (e.g. 45 + 25 = ) Subtraction with up to 3 Subtraction TU‐U up to 100 Multiplication table 6 multiples of 10 up to 100 with regrouping (e.g. 75 – 30 = ) (e.g. 63 – 6 = ) (e.g. 6 x 5)

4. Discussion and conclusions In spite of the fact that the present study was explorative in nature, our results point to three interesting findings. First, our accuracy improvement analysis showed that, when considering only the items ranked in the top positions, the largest accuracy improvement can be observed in the group that made the paper exercises. Playing Monkey Tales, however, lead to a larger sustained accuracy improvement in terms of number of items included in the math test. In other words, rather than acting locally (leading to the improvement in few item types), playing the game had a global impact on the accuracy performance in a large variety of items. Although these results should be interpreted with caution given the exploratory nature of this study, an interesting venue for future research would be to investigate whether this global impact might be the result of secondary effects of the arithmetical training in other cognitive domains like working memory and attention. Since the content of the paper exercises and the game exercises were carefully matched ‐ not only regarding content but also in number ‐ it is unlikely that the differences we observed between groups relies on the type of arithmetical exercises. Rather, we suggest that there might be other mediating factors associated with game play that could better explain this pattern. For instance, an increased capacity to hold information items, such as numbers, in working memory could explain the broader positive impact for the game group in terms of variety of arithmetical problems for which improvement occurred.

439

Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy Secondly, the correlation analysis performed between accuracy and reaction time improvements, revealed that, although for both groups, Monkey Tales and paper exercise, the correlation between these two behavioral markers was significant, this association was stronger for the gaming group. Moreover, this relationship was absent for the children who only went to school and did not receive any kind of repetition exercises during the three‐week period. This is an important finding considering that, in order to be effective, a game aimed to train mathematical skills in children would be expected to have a positive impact for both behavioral measurements. The present study shows that, similar to the traditional method of paper math drills, Monkey Tales can lead to performance enhancement by making children perform faster and more accurate in mental calculations. Thirdly, a ranking of the five items in which the largest accuracy and reaction time improvement were observed, revealed that doing the paper exercises lead to improvements predominantly on items where children were required to solve addition problems while playing Monkey Tales lead to improvement in a broad range of item types. These findings provide indirect evidence for the idea that arithmetic performance enhancement induced by game play and paper exercises might rely on slightly different mechanisms. For instance, previous research in the field of cognitive psychology has shown that information about the parity of numbers is associated with their arabic representation and directly retrieved from long‐term memory when needed (Dehaene, Bossini, and Giraux, 1993). Thus, the parity judgment task involves memory retrieval as well as the selection of the response induced by the retrieved information. Interestingly, it was a parity judgment item that showed the largest accuracy improvement after playing through Monkey Tales. This suggests that videogame training boosted cognitive processes like memory retrieval and response selection ‐ according to Miyake (2000) one of the three major cognitive control functions ‐ being both crucial for parity judgment. Whilst listing the cognitive processes and the working memory load involved in the arithmetical problems reported in the present study is not at the core of this study, the previous example shows the utility of this kind of exhaustive analysis. Since previous research has shown that arithmetic, and more specifically numerical calculation, involves the use of working memory (i.e., keeping information available in the cognitive system) and attention (Rubinsten and Henik., 2009), we are convinced that detailed analyses of improvement considering different categories of arithmetical problems can be informative about the underlying mechanisms of performance enhancement. Finally, some limitations of the present study have to be acknowledged. Given its explorative nature, it provides a very first examination about how, in terms of cognitive processes, arithmetic performance enhancement induced by game play and paper exercises might be different, but also that there are several ways this study could be improved. First of all, our results strongly suggest that learning mathematics through game play might be mediated by improvements in the use of working memory resources and attention. This partly remains speculative, however, because we did not apply tests that measured working memory and attention directly and thus have only presented indirect evidence. Future research could incorporate such standardized measurements to further investigate this issue. Additionally, rather than aiming to be conclusive, the present study explored whether we could find indications that performance gains differ between traditional learning methods and game training. Consequently, several questions could be now experimentally investigated, for instance the extent to which are results are generalizable to other games and whether similar positive effects can be found with children of different ages. Finally, future research could investigate the impact of arithmetical training through games for children disadvantaged by learning difficulties or socioeconomic status.

Acknowledgements This work was performed in the context of the iMinds‐ICON Games at School project, which is cofunded by iMinds, a research institute founded by the Flemish Government. We thank Vicky Vermeulen and Nic Pappijn (Die Keure) for their support with the materials and information about the educational game.

References Association, A.P., 2000. Diagnostic and statistical manual of mental disorders: DSM‐IV‐TR: American Psychiatric Publishing, Inc. Baddeley, A., 2000. The episodic buffer: a new component of working memory? Trends in Cognitive Sciences, 4, 417‐423. Bavelier, D., Green, C.S., Pouget, A. & Schrater, P., 2012. Brain Plasticity Through the Life Span: Learning to Learn and Action Video Games. In S.E. Hyman (ed.) Annual Review of Neuroscience, Vol 35. 391‐416.

440

Elena Patricia Nuñez Castellar, Anissa All and Jan Van Looy Boot, W.R., Kramer, A.F., Simons, D.J., Fabiani, M. & Gratton, G., 2008. The effects of video game playing on attention, memory, and executive control. Acta Psychologica, 129, 387‐398. Campbell, T., Dollaghan, C., Needleman, H. & Janosky, J., 1997. Reducing bias in language assessment: Processing‐ dependent measures. Journal of Speech Language and Hearing Research, 40, 519‐525. Colzato, L.S., Van Den Wildenberg, W.P., Zmigrod, S. & Hommel, B., 2012. Action video gaming and cognitive control: playing first person shooter games is associated with improvement in working memory but not action inhibition. Psychological research, 1‐6. Dehaene, S., Bossini, S. & Giraux, P., 1993. The mental representation of parity and number magnitude. Journal of Experimental Psychology General, 122, 371‐371. Elson, M., & Bente, G, 2009. CORTEX ‐ Computer‐Aided Registration Tool for Experiments [online]. University of Cologne. Available from: http://cortex.uni‐koeln.de [Accessed Access Date Green, C.S. & Bavelier, D., 2003. Action video game modifies visual selective attention. Nature, 423, 534‐537. Holmes, J., Gathercole, S.E. & Dunning, D.L., 2009. Adaptive training leads to sustained enhancement of poor working memory in children. Developmental Science, 12, F9‐F15. Kail, R. & Park, Y.S., 1990. Impact of practice on speed of mental rotation. Journal of Experimental Child Psychology, 49, 227‐244. Klingberg, T., Fernell, E., Olesen, P.J., Johnson, M., Gustafsson, P., Dahlstrom, K., Gillberg, C.G., Forssberg, H. & Westerberg, H., 2005. Computerized training of working memory in children with ADHD ‐ A randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 44, 177‐186. Mclean, J.F. & Hitch, G.J., 1999. Working memory impairments in children with specific arithmetic learning difficulties. Journal of Experimental Child Psychology, 74, 240‐260. Miyake, A., Friedman, N.P., Emerson, M.J., Witzki, A.H., Howerter, A. & Wager, T.D., 2000. The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive psychology, 41, 49‐100. Siegel, L.S. & Ryan, E.B., 1989. The development of working memory in normally achieving and subtypes of learning disabled children. Child development, 973‐980. Stevens, M., Lammertyn, J., Verbruggen, F. & Vandierendonck, A., 2006. Tscope: AC library for programming cognitive experiments on the MS Windows platform. Behavior Research Methods, 38, 280‐286. Studios, L., 2011. Monkey Tales. Gent, Belgium: Die Keure Larian Studios. Swanson, H.L., 1993. Working memory in learning disability subgroups. Journal of experimental child psychology, 56, 87. Turley‐Ames, K.J. & Whitfield, M.M., 2003. Strategy training and working memory task performance. Journal of Memory and Language, 49, 446‐468. Wouters, P., Van Nimwegen, C., Van Oostendorp, H. & Van Der Spek, E.D., 2013. A meta‐analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105, 249‐265.

441

Serious Game Adaptive Learning Systems Chinedu Obikwelu and Janet Read University of Central Lancashire, UK coobikwelu@uclan.ac.uk JCRead@uclan.ac.uk Abstract: Serious games have evolved from the traditional one‐size‐fits‐all mode to a Dynamic Difficulty Adjustment (DDA) mode. DDA is an individualized approach that is based on the principle of adaptivity. For serious games, there is an emphasis on adapting hints and feedbacks according to the changing learner’s competence by using set rules. Adaptive Learning Systems (ALS) are sometimes referred to as Personalised Learning Systems (PLS).According to Karagiannidis and Sampson every PLS should answer the following questions ‐ What is the learning content being adapted? Which aspects of the learning experience ‘drive’ adaptations? What is the basis for adaptation? These questions make up the adaptation logic which differs from game to game and from model to model. This paper reviews the current literature by investigating different adaptation logics embodied in existing serious game Adaptive Learning Systems. These different Adaptive Learning Systems have been adopted by and proposed for serious games. The Adaptive Learning Systems investigated in this paper include the ‘NUCLEO’ framework which emphasizes collaboration in a Multi‐User Virtual Environment (MUVE) with role assignment and team formation adapted to learners; S.M.I.L.E which is more of an accessibility model accommodating player‐learner’s with handicaps by adapting quests with the uniqueness of allowing teachers to define educational games with stored educational materials; ‘Framework for Adaptive Game Presenters with Emotions and Social Comments’ which adapts emotions and social feedback; the Fine‐Tuning System (FTS) which is based on adaptive fading ‐ adapting scaffolds including feedbacks and hints based on the fading principle; and ALIGN which separates the game logic from the adaptation logic by creating reusable adaptation abstractions. The trend in assessment generation which is drifting from the traditional After Action Review (AAR) to assessment generation for adaptive interventions is also highlighted in this paper. Keywords: serious game; learning; adaptivity; personalised learning; feedback; adaptive learning system; adaptation logic

1. Introduction A possible means for increasing the effectiveness and enjoyment of an educational game is the introduction of adaptivity, allowing a game to be customized for a specific player based on an assessment of their state of learning or other characteristic(Mehm et al., 2012) these characteristics could include motivational states, gaming preferences and psycho‐pedagogical implications (M.D Kickmeier‐Rust & Albert, 2010). In current forms of technology‐enhanced learning, concepts of adaptivity, adaptability and personalization have increasingly become important (M.D Kickmeier‐Rust & Albert, 2010). User‐adaptation in e‐learning can be characterized as the ability of a system to personalize the learning experience to different individual conditions over time (Sancho, Moreno‐Ger, Ruben, & Baltasar, 2009). An interesting concept is Dynamic Difficulty Adjustment (DDA) which (Bailey & Katchabaw, 2005) described as the ability of a game to automatically adapt the difficulty level of gameplay to match the skills and tolerances of a player. In their work, they specified that gameplay including player character attributes; non‐player character (NPC) attributes; Game World and level attributes; puzzle and obstacle attributes could all be made adjustable. The success of adaptation is addressed at two distinct layers – the user modelling and adaptation decision‐ making layers (Brusilovsky & Sampson, 2004). The primary task for game‐based adaptive educational mechanisms is to guide and support the learner in acquiring knowledge by, for example, informing the learner, intervening when misconceptions occur or when the learning progress is unsatisfactory, and hinting or providing the learner with appropriate feedback (Kickmeier‐Rust & Albert, 2010). In addition tasks are required to be motivating, maintaining immersion and personalizing the game according to the preferences and needs of the learner (Kickmeier‐Rust & Albert, 2010). Accomplishing this goal requires a theoretical and technological approach that enables the game to assess cognitive states (e.g. competence states or motivational states), learning progress, possible misconceptions or undirected/ unsuccessful problem‐solving strategies (Kickmeier‐ Rust & Albert, 2010).

2. Stages in adaptation in serious games (Ismailovic, Haladjian, Kohler, Pagano, & Brugge, 2012) describe adaptivity in serious games as follows ‐ Adaptivity in serious games is an approach that enables a serious game to (A) learn from learner’s behaviour by (A1) intelligent monitoring and (A2) interpreting learner’s actions in the game’s world and (B) to intervene

442

Chinedu Obikwelu and Janet Read in the game’s world by (B1) automatically adjusting the learning content and (B2) adjusting the game elements according to (C) the students individual ZPD and using the principles of (D) the More Knowledgeable Other ‐ MKO, where ADAPTIVITY is a MKO for the learner according to the Social Development Theory. “According to this definition, the adaptivity process in a serious game consists of four stages:

Monitoring players (A1)

Learner characterization (A2)

Assessment generation (B1)

Adaptive intervention (B2)” (Ismailovic et al., 2012)

These adaptation stages are discussed in further detail in the following sections.

2.1 Monitoring players The proposed NUCLEO framework which is deeply rooted in socio‐constructive pedagogical theories (Sancho et al., 2009) emphasizes intelligent monitoring in a Multi‐User Virtual Environment. In this framework students are classified into four different types in relation to their attitude to learning Vermunt’s Meaning Directed (MD), Application Directed (AD), Reproduction Directed (RD), and Undirected (U) describe the types. Each player is given a role in relation to their attitude to learning ‐ based on the aforementioned student types (Vermunt, 1987). According to (Sancho et al., 2009) this classification of learners help to distinguish the students who need more intensive guidance through the learning process from those who are more capable of guiding their own learning experience.

2.2 Learner characterization This is often referred to as the user model. Game logs recording the players’ performance are used to create models of players’ actions, preferences or personality (Lopes & Bidarra, 2011). Given a game state, these models assess and predict the player’s desired experience for the next game state (Lopes & Bidarra, 2011). Models for the player experience and performance are then used to steer an adaptation and generation engine, which adjusts the appropriate game components to better fit both (Lopes & Bidarra, 2011).

2.3 Assessment generation The traditional approach to assessment generation is the After Action Review (AAR), which typically allows instructors and students to critically review the decisions made and actions taken during game‐play (Raybourn, 2007).This AAR has evolved to assessment generations in games required for adaptive interventions. A plausible example is that utilized in the S.M.I.L.E model (Divéky & Jurnecka, 2007). The model utilizes the Item Response Theory to predict how players react to tasks and questions they are given while solving quests, and thus is able to measure their level of forgetfulness by testing them on tasks and questions they have previously successfully completed and answered.

2.4 Adaptive intervention Pedagogical interventions and feedback are an integral element of the educational process (M. Kickmeier‐Rust & Steiner, 2010). Dynamically adjusting game elements according to individual player performance can contribute to make the game experience more unique and personal (Lopes & Bidarra, 2011). In serious games, adapting to specific skills is more important than to the global notion of difficulty or challenge (Lopes & Bidarra, 2011). Adaptive games have specialized (and usually ad‐hoc) approaches, where the game components are adjusted to encourage training a specific skill (Lopes & Bidarra, 2011). Individual learners/gamers are provided with appropriate personalized interventions, either in the form of personalized feedback or hinting or in the form of smooth alterations of the game (e.g. Its storyline, its pace, or its difficulty) (M. Kickmeier‐Rust & Steiner, 2010). The condition under which a certain adaptive intervention is given is developed on the basis of pedagogical and didactic rules while considering a strong integration in the game‐ play context (Michael D Kickmeier‐Rust, Hockemeyer, Albert, & Augustin, 2008). (M.D Kickmeier‐Rust & Albert, 2010) have classified intervention into the following

443

Chinedu Obikwelu and Janet Read

“Competence activation interventions: may be applied if a learner becomes stuck in some area of the problem space and some competencies are not used even though the system assumes that the learner possesses them.

Competence acquisition interventions: may be applied in situations when the system concludes that the learner lacks certain competencies

Motivational interventions: may be applied, for example, if the learner unexpectedly fails to act for a certain long period of time.

Feedback: may be utilized to provide the learner with information about the learning progress or the game.

Assessment clarification interventions: may be applied, for example in the form of a query, if the learner’s actions provide contradicting support for the assumption of a certain competence state.”

3. The adaptive learning systems The adaptation logic of any personalized learning system can be defined in the following terms (Karagiannidis & Sampson, 2004).

The constituents (what is being adapted?)

The determinants (what the adaptation logic is based on) This information is normally stored in the student/user model. Brusilovsky & Milan (2007) characterize the typical content of user models as “the user’s knowledge, interests, goals, background and individual traits”.

The rules (the logic that defines which constituents are affected by the determinants and how?).

In this section, some existing frameworks are discussed in relation to their adaptation logic in terms of constituents, determinants and rules

3.1 NUCLEO framework (Sancho et al., 2009) The learning strategy in NUCLEO is deeply grounded in a socio‐constructive pedagogical stream (Sancho et al., 2009). It combines Problem‐Based Learning (PBL) and Computer Supported Collaborative Learning (CSCL) in a framework that uses a multiplayer role‐playing video‐game as the delivery format (Sancho et al., 2009). 3.1.1 Constituents Team formation and individual role assignments linked to learning experience are being adapted 3.1.2 Determinant This information includes the following

The mark obtained by group in the mission

Individual mark in peer‐to‐peer evaluation

Frequency of use of specific role tools

3.1.3 Rul After every mission, the student model is updated collecting information from three different sources: the mark obtained by the group in the mission, the individual mark obtained by the student in the peer‐to‐peer evaluation (every student evaluates his/her teammates) and the frequency of use of the specific role tools. According to this information, students’ roles and teams can be reassigned, for instance, in order to reduce intra‐team conflicts or to give students the possibility of experiencing new social contexts.

3.2 Smart multipurpose interactive learning environment S.M.I.L.E (Divéky & Jurnecka, 2007)(Kajan, Bieliková, Divéky, Omelina, & Jurnecka, 2010) S.M.I.L.E combines the advantages from interactive educational materials and popular computer games by giving teachers (i.e. authors of educational materials) the ability to transform study materials into exciting educational games that can be played by users with disabilities including those with visual and hearing

444

Chinedu Obikwelu and Janet Read impairments(Divéky & Jurnecka, 2007). The system is based on an automatic adaptive game generation concept. For adaptivity, the system attempts to be different for different learners and groups of learners by taking into account information accumulated in the learner’s’ or group’s’ profiles. In order to deal with the challenge of uses with disabilities, for adaptability, the system can be customised by every user according to his needs and abilities. The discussion in this work is restricted to adaptivity. 3.2.1 The constituents Games and quests are adapted. The game is a persistent virtual world made up of quests based on all available educational materials, and therefore practically resembles one vast game that contains all possible educational quests. 3.2.2 The determinants This is the subject‐specific skill level of the student. 3.2.3 The rules Players are dynamically navigated into solving quests most suitable for their estimated knowledge level. Additionally, while solving quests, options of appropriate difficulty are chosen for players to solve (e.g., if a player has his estimated knowledge of physics at a low level, he will need to answer a simpler physics question in order to solve the particular quest – and vice versa). The framework uses Item Response Theory to predict how players will react to both tasks and questions they are given while solving quests, and thus to measure their level of forgetfulness by testing them on tasks and questions they have previously successfully completed and answered (Divéky & Jurnecka, 2007).

3.3 Framework for adaptive game presenters with emotions and social comments (Karouzaki & Savidis, 2012) This is a framework for building artificial game presenter characters with emotions, capable of delivering knowledgeable social comments, adapted to individual profiles and game progress for table‐top multiplayer computer games (Karouzaki & Savidis, 2012). The framework supports emotional facial expressions for the presenters, allowing them to convey their emotions and thus be more expressive than the majority of the commentary systems today. Presenters provoke social interaction in order to keep the players and the audience constantly motivated and alerted about the game progress (Karouzaki & Savidis, 2012). 3.3.1 3.3.1 The Constituents: Agents with adaptive social feedback including adaptive emotions. The social feedback includes humour, reward, sympathy, surprise, disappoint, enthusiasm, agony or anticipation. 3.3.2 The determinants Performance, progress and profile of each individual player 3.3.3 The rule The framework uses inputs such as game events and player profiles and keeps track of incoming information about game progress to decide and deliver player‐adapted and context‐sensitive social feedback. Dynamic Game Difficulty (DGD) balancing is employed where the player’s performance is monitored and used to adjust its difficulty accordingly (Karouzaki & Savidis, 2012).

3.4 Adaptive learning in games through non‐invasion ALIGN (Peirce, Conlan, & Wade, 2008) The idea behind the ALIGN system is the separation of the game logic and adaptation logic with flow experience overlapping. In this system, the adaptation is authored at a level abstracted from a particular game, thus could be reused within multiple games (Peirce et al., 2008). The separation of the adaptation and the game features is at the core of the ALIGN system (Peirce et al., 2008). This feature aims to reduce the cost of

445

Chinedu Obikwelu and Janet Read implementing adaptation through maximizing the reuse of domain expert authored adaptation (Peirce et al., 2008). 3.4.1 The constituents In this system, a diverse range of Adaptive Elements (AEs) can be adapted depending on the game. The Adaptive Elements (AEs) include Cognitive feedback; Instant meta‐cognitive feedback; Tendency meta‐ cognitive feedback; NPC Confidence/Prudence feedback; Affective/ motivational feedback; Knowledge based hinting; Progression hinting etc. 3.4.2 The determinants The information contained in the user/student model of the game for which an adaptation abstraction is used. 3.4.3 TherRules The adaptation logic for a particular game focuses solely on selecting desirable AEs as the AEs are already constrained by feasibility and appropriateness.

3.5 The fine‐tuning system (FTS) (Obikwelu, Read, & Sim, 2013) The FTS is based on faded scaffolding levels. According to (Merrill, 2011) faded scaffolding levels are of interest because they offer an interaction with the student in which the working memory load is not as heavy as for totally faded scaffolding levels – so there is a gradual transition to ‘full’ exercises “Fading can be used to anchor and stimulate Limited problem solving; Reflection; Self‐explanation” (Melis & Goguadze, 2004). The structure of the full scaffolding level, determines the possibilities of fading (Melis & Goguadze, 2004).Static fading and Adaptive fading are the two types of fading (Reisslein, Reisslein, & Seeling, 2006)

Static fading ‐ as the scaffolding levels are faded, the learners are required to attempt each scaffolding level at a predetermined rate. No attention is paid to whether or not the learner could keep up with the increasing problem‐solving demands (Reisslein et al., 2006).

Adaptive fading ‐ adapts the fading of scaffolding levels to the learner’s successes and failures in the problem‐solving process (Reisslein et al., 2006).Adaptive fading is the process of reducing (fading) the scaffolding level with every new problem continues until all the scaffolding levels are faded away and the learner has to independently solve the entire problem (Reisslein et al., 2006).

3.5.1 Constituent Learning support (scaffolds) including feedbacks and hints 3.5.2 Determinant Successful attempts, number of attempts and attempts per scaffolding level 3.5.3 Rule The scaffold is reduced whenever the player‐learner successfully solves a particular task at a particular scaffolding level – adaptive fading. The lesser the number of attempts at a particular task on a given scaffolding level, the faster the fading. An interesting measure elucidated from this framework is ‘attempts per scaffolding level’ being inversely proportional to the fading rate. The fading rate is expected to be a direct representation of the player‐learner’s learning rate.

4. Student/user modelling: Challenges and limitations Adaptive interventions in serious games are based on the determinants which are mainly the current state of the player‐learner’s knowledge – i.e. the Student Model. The representation of the player‐learner’s current state of knowledge is rather problematic as the determinants used in most ALS could expectedly be influenced

446

Chinedu Obikwelu and Janet Read by the player‐learner’s game preference, background and game‐play expertise. Questions that have arisen to this effect include

How do you show that the students are learning what you claim they are learning?

How do you know you are measuring what you think you are measuring? (Chen & Michael, 2005)

The solution to these problems could be in the initialization of the student model based on game‐related player‐learner’s characteristics – game preference, background and game‐play expertise. This initial classification of the player‐learners would help minimize the effect of the ephemeral data elucidated via game‐ play (showing the state of knowledge) upon which adaptive intervention is based. “The simplest kind of assessment classifies the student in terms of behaviour, preferences or background”. “Such a stereotype user‐ model (Rich 1979) is useful for ‘priming’ the modelling process when there is no prior experience on which to base expectations (Carr & Goldstein 1977, Clancey 1982a)”(Clancey, 1986). There is also the challenge of setting clear student modelling goals. According to (AILE, 2006) the goals of student modelling could include

Finding out what the student knows, believes, can do

Looking for evidence the user fails to exploit some knowledge

Looking for inconsistent beliefs, differences between student and domain model

It is important to identify a number of relevant criteria associated with the goal of student modelling. For example “Looking for inconsistent beliefs between student and domain model” has been described by (Brown, Burton, & Larkin, 1977) as the bug model. “In general a bug is some structural flaw (faulty part) manifested in faulty behaviour (a process). Thus the term bug is used to refer to the incorrect part of a constructed procedure” (Clancey, 1986).

4.1 Student modelling based on the Bug Model (Brown et al., 1977):

List possible inconsistent beliefs that can be made (code bugs) – bug library with ratings based on criticality of misconception

Describe the bug: The reason about what student would have to believe in order to exhibit behaviour indicating the bug.

Single out the bugs – This can be done through direct access to real‐time behaviours in unobtrusive ways within the context of the system (Knight, Buckingham Shum, & Littleton, 2013)

Measure by means of a formula: The list of captured inconsistent beliefs are combined with a formula for measurement purposes “The combining formula may be simple, and require only the addition of weighted or unweighted component scores or ratings. On the other hand, the formula may be complicated (taking, for example, conjunctive and disjunctive form)” (D.Royce, 1989)

4.2 Student modelling based on the Overlay Model A user’s knowledge of a subject is most often represented by an overlay model which is based on the structural model of the subject domain (Brusilovsky, 1996). The idea of the overlay model is to represent an individual user’s knowledge of the subject as an “overlay” (subset) of the domain model which reflects the expert level knowledge of the subject (Brusilovsky & Millan, 2007).

Break the knowledge for the given domain into elementary pieces (concepts) and show the relationship. These could include the learning activities.

Assign values to these elementary pieces for a representation of the user knowledge of the concept. “This can be just a binary value (known‐not known), a qualitative measure (good‐average‐poor), or a quantitative measure, such as a probability that the user knows the concept” (Brusilovsky, 1996). “An overlay model of user knowledge can be represented as a set of pairs “concept – value”, one pair for each domain concept”. (Brusilovsky, 1996)

The overlay model is the student model associated with most serious game ALS. It is envisaged that the student modelling for any serious game ALS would be more effective if the overlay model is rightly combined

447

Chinedu Obikwelu and Janet Read with the bug model. Further research into the appropriate combination of existing student models for effective student modelling is required.

5. Conclusion This work summarizes several Adaptive Learning Systems that are being used in serious games. Serious game Adaptive Learning Systems are expected to protect and enhance the immersion element of the game whilst directing the player‐learner to the game’s learning objective. Adaptivity whilst protecting and enhancing immersion is the prime challenge of adaptive scaffolding. Another challenge is deducing learning analytics (“learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.”(1st International Conference on Learning Analytics and Knowledge, 2011) (George & Long, 2011)) that could be associated with a particular ALS. At present, serious game ALS is focused on the adaptation of adaptive elements to the player‐learner’s competence models based on set rules. The extent to which they protect immersion and elucidate useful learning analytics from the student modelling is unclear at the moment.

References 1st International Conference on Learning Analytics and Knowledge. (2011). Learning Analytics & Knowledge. Retrieved June 2, 2013, from https://tekri.athabascau.ca/analytics/ AILE. (2006). Introduction to Student Modelling. Retrieved from http://www.inf.ed.ac.uk/teaching/courses/aile/lect6studmhoout.pdf Azevedo, R., Gromley, J. G., & Seibert, D. (2004). Does Adaptive Scaffolding Facilitate Students’ Ability to Regulate Their Learning with Hypermedia? Contemporary Educational Psychology, 29(3), 344–370. Bailey, C., & Katchabaw, M. (2005). An experimental testbed to enable auto‐dynamic difficulty in modern video games. Proceedings of the 2005 GameOn North …. Brown, J. S., Burton, R. R., & Larkin, K. M. (1977). Representing and using procedural bugs for educational purposes. ACM ’77 Proceedings of the 1977 annual conference (pp. 247–255). New York,NY, USA. Brusilovsky, P. (1996). Methods and techniques of adaptive hypermedia. User Modeling and User Adapted Interaction, 6(2‐ 3), 87–129. Brusilovsky, P., & Millan, E. (2007). User models for adaptive hypermedia and adaptive educational systems. The Adaptive Web (pp. 3–53). Berlin: Springer‐Verlag. Brusilovsky, P., & Sampson, D. (2004). Layered evaluation of adaptive learning systems Charalampos Karagiannidis. Engineering Education, 14, 402–421. Chen, S., & Michael, D. (2005). Proof of Learning: Assessment in Serious Games. Gamasutra. Clancey, W. J. (1986). Qualitative Student Models. Ann. Rev. Comput. Sci., 1, 381–450. D.Royce, S. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119–144. Divéky, M., & Jurnecka, P. (2007). Adaptive Educational Gameplay within Smart Multipurpose Interactive Learning Environment. Semantic Media …. George, S., & Long, P. (2011). Penetrating the Fog: Analytics in Learning and Education. Educause Review. Ismailovic, D., Haladjian, J., Kohler, B., Pagano, D., & Brugge, B. (2012). Adaptive Serious Game Development. Games and Software Engineering (GAS), 2nd International Workshop on. IEEE. Kajan, R., Bieliková, M., Divéky, M., Omelina, L., & Jurnecka, P. (2010). Learning with Smart Multipurpose Interactive Learning Environment. International Federation for …. Karagiannidis, C., & Sampson, D. (2004). Adaptation Rules Relating Learning Styles Research and Learning Objects Meta‐ data. Workshop on Individual Differences in Adaptive Hypermedia, 3rd International Conference on Adaptive Hypermedia and Adaptive Web‐based Systems (AH2004), Eidhoven, Netherlands. Karouzaki, E., & Savidis, A. (2012). A Framework for Adaptive Game Presenters with Emotions and Social Comments. International Journal of Computer Games Technology, 2012, 1–18. doi:10.1155/2012/929814 Kickmeier‐Rust, M., & Steiner, C. (2010). A glissade on the learning curve: multi‐adaptive immersive educational games. … for Education. Digital …, 361–366. Kickmeier‐Rust, M.D, & Albert, D. (2010). Micro‐adaptivity: protecting immersion in didactically adaptive digital educational games. Journal of Computer Assisted Learning, 26, 95–105. Kickmeier‐Rust, Michael D, Hockemeyer, C., Albert, D., & Augustin, T. (2008). Micro adaptive, non‐invasive assessment in educational games. In M. Eisenberg, Kinshuk, M. Chang, & R. McGreal (Eds.), Proceedings of the second IEEE International Conference on Digital Game and Intelligent Toy Enhanced Learning (pp. 135–137). Banff, Canada. Knight, S., Buckingham Shum, S., & Littleton, K. (2013). Epistemology, pedagogy, assessment and learning analytics. Proceedings of the Third International Conference on Learning Analytics and Knowledge ‐ LAK ’13, 75. doi:10.1145/2460296.2460312 Lopes, R., & Bidarra, R. (2011). Adaptivity Challenges in Games and Simulations: A Survey. IEEE Transactions on Computational Intelligence and AI in Games, 3(2), 85–99. doi:10.1109/TCIAIG.2011.2152841 Mehm, F., Konert, J., Göbel, S., Steinmetz, R., Authoring, A., Educational, D., In, G., et al. (2012). An Authoring Tool for Adaptive Digital Educational Games State of the Art, (September), 236–249.

448

Chinedu Obikwelu and Janet Read Melis, E., & Goguadze, G. (2004). Towards adaptive generation of faded examples. Intelligent Tutoring Systems, 1–18. Merrill, D. (2011). Systematic Development of an E‐learning Module for Teaching Writing. World Conference on Educational Multimedia, Hypermedia and Telecommunications (Vol. 2011, pp. 1132–1137). Obikwelu, C., Read, J., Sim, G.(2013). Children ’ s Problem‐Solving in Serious Games : The “ Fine‐ Tuning System ( FTS )” Elaborated, 11(1), 49–60. Peirce, N., Conlan, O., & Wade, V. (2008). Adaptive Educational Games: Providing Non‐invasive Personalised Learning Experiences. 2008 Second IEEE International Conference on Digital Game and Intelligent Toy Enhanced Learning, 28– 35. doi:10.1109/DIGITEL.2008.30 Raybourn, E. M. (2007). Applying simulation experience design methods to creating serious game‐based adaptive training systems. Interacting with Computers, 19(2), 206–214. doi:10.1016/j.intcom.2006.08.001 Reisslein, J., Reisslein, M., & Seeling, P. (2006). Comparing static fading with adaptive fading to independent problem solving: The impact on the achievement and attitudes of high school students learning. JOURNAL OF ENGINEERING …, (July). Sancho, P., Moreno‐Ger, P., Ruben, F.‐F., & Baltasar, F.‐M. (2009). Adaptive role playing games: an immersive approach for problem based learning. … Technology & Society, 12, 110–124. Vermunt, J. D. (1987). Learning styles and self‐regulation. Annual Meeting of the American Educational Research Association. doi:ERIC Document 285 900

449

Combatting Social Isolation and Cognitive Decline: Play a Physical or Digital Game? Daire Ó Broin and Ross Palmer Institute of Technology Carlow, Ireland daire.obroin@itcarlow.ie ross.palmer@itcarlow.ie Abstract: Two significant problems among the elderly community are those of social isolation and the cognitive decline that occurs as we age. These problems can be readily compounded (for example, reduced levels of attention are associated with an increased risk of falling). Keeping active is commonly advised to reduce these problems ‐ many senior groups exist that meet regularly and engage in activities that encourage social interaction. However, mobility and other problems mean that elderly people may not be able to attend such groups very often, if at all. To address this problem, we set out to create an experience that confers the same benefits (increasing social connectedness and inhibiting cognitive decline) without users having to leave their homes. Our user group selected one of the games they play together, Memory, a game that requires players to closely attend to their experience. We designed and developed a digital multiplayer version of Memory for browser and tablet targeted at elderly users. The game is integrated into a custom social network, and enables users to talk and play without having to be in the same physical space. It also incorporates user‐generated content from the social network. A pilot study has been carried out with two user groups across two European countries. It investigates whether the experience of playing a physical game between players in present the same physical space differs significantly from playing in different locations using the digital version of the game. In order to uncover differences, the study compared how intrinsically motivating each experience is. In the study, which uses within‐subjects design, the users played Memory with real cards together in the physical space and afterwards completed the Intrinsic Motivation Inventory (IMI). The users also played the digital version of the game via the social network, this time where users are in different locations using a PC, Smart TV, or Tablet, and communicate using VOIP, and afterwards completed the IMI. The data collected was analysed and used to inform semi‐structured interviews. This paper presents findings from the pilot study, and from these findings outlines the main study and future work in investigating and improving the game’s benefits. Keywords: games for seniors, social connectedness, cognitive decline, intrinsic motivation

1. Introduction Two of the most significant problems affecting elderly people are those of social isolation and the cognitive decline that occurs with age. Social isolation and loneliness in older people are problems that are becoming more and more widely recognised in international policy (Cattan et al., 2005). Numerous studies have shown the extremely negative effects social isolation can have on elderly people, such as cognitive decline (Zunzunegui et al. 2003, Bassuk, 1999), risk of dementia (Fratiglioni, 2000), and increased mortality (Steptoe et al., 2013). These problems can lead to further problems leading to further decline. For example, reduced levels of attention are associated with an increased risk of falling (O’Halloran et al., 2011). Keeping active is advised to alleviate these problems, and there are many senior groups that meet regularly and engage in activities aimed at encouraging social interaction. As a result of problems, such as that of mobility, elderly people may not be able to attend such groups frequently, or indeed at all. This paper describes work that forms part of a larger project, Join‐In (Join‐In, 2013), whose aim is to address the issues of social isolation, cognitive decline, and physical decline in elderly people using a social network portal customised for elderly people from which multiplayer games can be initiated. The use of games to promote well‐being in elderly people has been demonstrated, for example, by Goldstein et al, whose work showed that elderly people playing digital games for five hours a week for five weeks show improvements in reaction times, self‐esteem and sense of well‐being (Goldstein et al, 1997). Gaming also has the advantage of being multigenerational and facilitates the elder generation to socialise with the newer generations (e.g. their children and grandchildren). This paper describes the design of one of the multiplayer games developed, based on the card game Memory, which aims to address the issues of social isolation and cognitive decline. As this is a game selected by the user groups, it was necessary, before rolling out the main study, to investigate whether the experience of playing a physical game between players in the same physical space differs significantly from playing in different locations using the digital version of the game.

450

Daire Ó Broin and Ross Palmer A pilot study of this game has been carried out with two user groups in two European countries (Ireland and Germany). In order to uncover differences, the study compares how intrinsically motivating each experience is. In the study, participants played both the physical and digital versions of Memory and afterwards completed a subset of the Intrinsic Motivation Inventory (IMI) (Ryan et al., 1983). In the physical version, pairs of participants played together with real cards in the same physical space. In the digital version of the game, participants accessed the game via the social network portal using an iPad. This paper begins by describing the design of the Memory game; it then presents the design and findings from the pilot study. A discussion follows, and the main study is outlined, informed by the findings of the pilot study. The paper concludes by outlining future work.

2. Design of the game This section describes the requirements of the game gathered from the user groups and how some of the main requirements were met.

2.1 Requirement gathering and selection of a game The initial stage of the Join‐In project was to develop a social networking methodology for elderly persons who are little inclined to seek new contacts themselves. This target group were involved in activities to help them build and establish new social networks. User requirements were gathered from interviews, focus groups, questionnaires, and consultations with experts in elderly care. These requirements were ranked, and included:

Support for cooperative and competitive play

Possibility of communicating with other players

Simple to follow rules (that do not cause trouble for elderly users)

Beneficial for mental fitness

Means of tracking progress (for users and health professionals)

A simple and intuitive interface

Familiar elements (to ease access to the game)

Must be perceived as useful

Fun to play

Allow for multiple users at different locations to play simultaneously

Give positive feedback

Follow “design for all” principles, including

Adjustable speed/different levels of difficulty.

This can be used to accommodate users (including friends and family) with different ability levels and allow them to play together in a meaningful way.

Possible to play with limited fine motor skills

Both visual and aural feedback

Accessible from multiple platforms to enable wide target audience to play from their existing devices

In order to meet the needs and interests of the target group, the activities and games that are the favourites of the elderly user groups were elicited. Both board games and card games were frequently mentioned by the different user groups, and a multiplayer card game based on the game ‘Memory’ was finally settled on. The memory card game, also commonly known as the game of ‘concentration’, is a turn based card game. The game consists of ten pairs of cards arranged in four rows of five cards. Initially, the cards are all placed face down. Each player takes turns to try and match pairs of cards by selecting the two cards they wish to turn over. When two cards are matched, they remain drawn in a face up position and form no further part of the game. A player is rewarded for a successful match by getting another turn. The game ends when all ten pairs have been matched and the winner is the player with the most matches.

451

Daire Ó Broin and Ross Palmer

2.2 Meeting the requirements This section describes how some of the main requirements of the game were met, illustrating with some screenshots from the game. 2.2.1 Accessibility In order to maximise accessibility to the target audience, the game needed to be platform‐agnostic as far as possible. This was achieved by creating a browser‐based game using HTML5. This enables users to use most devices including tablets, Smart TVs, and PCs. The game is readily accessed from the customised social network portal, based on Elgg (http://elgg.org/), which has a simplified user interface suitable for elderly users. This is shown in Figure 1 – the coloured circle beside a friend’s name indicates if they are online and can be invited to play.

Figure 1: The social portal with simplified user interface suitable for elderly users; the coloured circle beside a friend’s name indicates whether they are online and can be invited to play 2.2.2 Adjustable speed/different levels of difficulty The game has adjustable levels of difficulty and allows players of different skills to play together by means of bonuses and handicaps. The simplest of these is beginning the game with some of the cards already upturned (Figure 2).

Figure 2: The main gameplay screen of the memory game; bonuses and handicaps enable players of different skills to play together; this example shows some cards already upturned for the player

452

Daire Ó Broin and Ross Palmer 2.2.3 Simple rules The memory game employs simple rules and already familiar game mechanics. It also opened the possibility of inter‐generational play between seniors and youngsters (such as grandparents and grandchildren) (Abeele and De Schutter, 2010).

Figure 3. Pop‐up notifications are used to give positive reinforcement messages 2.2.4 Positive feedback During play, a simple feedback system displays short‐lived notifications to the players. These notifications appear occasionally and praise successful card matches or offer encouragement when no matches are made (Figure 3). 2.2.5 Replayability The memory game has a number of selectable card ‘decks’ based on different themes (for example, nature, cartoon faces, national food dishes) as illustrated in Figure 4. To enhance replayability, users may create their own picture albums on the social portal and designate which albums they would like to use as card decks in the memory game. It is anticipated that this feature will offer opportunities for storytelling and offer a compelling reason for seniors to engage with the game on a recurring basis.

Figure 4. Players can choose which set of images to use as the in‐game card deck

453

Daire Ó Broin and Ross Palmer 2.2.6 Measure player performance A recent study (Grégoire et al. 2012) provides empirical support for the relationship between mindfulness and wellbeing where this relationship is partially mediated by personal goal setting. To enable players set personal goals, it is necessary to measure and record game performance. A set of metrics were devised to capture the player performance in terms of (i) skill level and (ii) engagement by the player. Skill level The computation of the skill level is derived from a player’s performance based upon a notional working model of how a player might perform. Each time a card is turned (by either player), it enters the players working memory, where the player remembers both the card and its position in the game. Cowan (Cowan, 2010) suggests that young adults have a working memory capacity of about 3 to 5 chunks of information (letters, digits or words) without rehersal. A baseline value of four cards (and their associated positions) is assumed as the maximum number of items a player can hold in their working memory. When a pair of cards is matched, we assume that both cards are removed from the player’s working memory. If the outcome of a turn is a matching pair of cards, a turn score is calculated using a devised algorithm. Player engagement A simple summation of the game score is used to set the experience level of a player. The more often players engage in games of memory, the more rapidly their experience level will grow (with an expectation that their skill level increases too). An approach similar to that used in commercial games is employed by not placing a cap on the experience level. The frequency of engagement is also important and this metric can be computed easily from the datetime stamp recorded after each game is completed. After each multiplayer game, the game score, the duration of play and a datetime stamp are recorded, and from this, the total duration for any given day and “streaks” (number of consecutive days) may be calculated.

2.3 Technical design The memory game server is written in Node.js (http://nodejs.org), a server‐side JavaScript framework for developing scalable network applications and uses socket.io to implement an asynchronous messaging system with the game clients, and provides an abstraction layer over Web Sockets and other communication schemes, depending on the browser capabilities.

3. The study This study is a pilot study of a larger study that is currently being rolled out to a much larger sample and rather than taking place in a lab environment will be used by the participants in their homes for about three months in four European countries (Ireland, Germany, Hungary, and Norway).

3.1 Objective The main objective of the study was to determine whether there is a significant difference in the participants’ experience between playing a physical game of Memory with co‐located players to playing the digital multiplayer version in different locations. A secondary purpose was to identify and address usability issues.

3.2 Study design The study employed within‐subjects design in which each subject is observed under each condition. In the study, participants played both the physical and digital versions of Memory and afterwards complete a subset of the Intrinsic Motivation Inventory (IMI) (Ryan et al., 1983). In the physical version, pairs of participants played together with real cards in the same physical space. In the digital version of the game, participants accessed the game via the social network portal using an iPad.

3.3 Measures The players’ experiences were assessed using the Intrinsic Motivation Inventory (IMI), a multidimensional instrument that measures the interest/enjoyment‚ perceived competence‚ effort‚ value/usefulness‚ felt pressure and tension‚ and perceived choice of participants when they were performing a particular activity

454

Daire Ó Broin and Ross Palmer (Ryan et al., 1983). Usability was also measured with a custom questionnaire and by observing the ease with which the participant completes the tasks she needed to do.

3.4 Setting The study took place in a number of laboratory rooms in the Institute of Technology Carlow, and also in a similar setting in Helmholtz in Germany. The participants were in different rooms and couldn’t see each other to simulate, as closely as possible, playing alone from home. Assistants were available to help them get setup.

3.5 Sample The participants, aged between 66 and 80, were recruited by contacting elderly groups, and had very little experience with technology. 18 people, roughly balanced by gender, participated in the study, although 2 of these made some omissions in the questionnaires and their data was not used as a result.

3.6 Procedure Participants are given a short introduction to the project. They are told that further refinement and input from the players is needed before the portal and games would be ready for playing at home. In pairs, they play both digital and physical versions of Memory; pairs alternated between playing the physical version or the digital version first. After each session (physical and digital), participants fill out the IMI, either with paper and pencil or digitally on a Google form. For the physical version, participants play a game of Memory using cards with the same pictures as the digital game and in the same format (a 4 x 5 grid). For the digital version, each participant is given an iPad (connected to Wifi), from which the social portal and subsequently the game can be readily accessed. One of the players invites the other to a game of Memory (players can see a list of their friends and a coloured circle indicating whether they are currently online as shown in Figure 1). Her partner gets a pop up notification (‘Daire invites you to a game of memory accept/reject’). She taps accept and the game begins. At this point the VOIP component starts up, and the participants can chat. The inviter is allowed to choose the deck of cards that they wish to play the game with. They then play for about 15 or 20 minutes (about two or three games) and are then given the IMI to fill in.

3.7 Results The results of the pilot study are depicted in Figure 5. The IMI scores are from 1 to 7, the higher the number, the higher the magnitude of the variable. In summary, the results indicate that the participants found the digital game to be more enjoyable than the physical one. They found the activity to be highly valuable and useful, and all participants noted they would be most willing to try it again.

Figure 5: IMI Scores are from 1 to 7; the results indicate that the participants found the digital game to be more enjoyable, and perceived competence was around the same in both physical and digital versions, suggesting no issues caused by the technology or the participants playing against each other from different locations

455

Daire Ó Broin and Ross Palmer

4. Discussion Prior to the study, there were two principal concerns. First, would the digital version of the game, where players were not in the same physical location as their opponent, pale in comparison with the experience of playing a physical version with the other player physically present? And second, would the lack of experience the participants had with technology be an issue (that is, negatively affect their experience?) In short, would there a significant difference between the physical and digital games (would the digital game simply not stack up against the physical game)? The results are encouraging as they suggest that participants enjoyed the digital version – indeed they enjoyed it more than the physical game. That is, though the participants were in separate locations, this did not have a negative effect on the participants’ enjoyment of the game. As this is the context for which the game was designed (users playing at home by themselves playing with other users also at home by themselves), this is an extremely positive result. Also, while there were some minor usability issues, such as the indicator for knowing whose turn it is not being clear to several participants (see Figure 2 – the player whose turn it is has a large dark blue rectangle around his name, and the other player has a small light blue rectangle around her name), the results suggest that usability or the technology were not a cause of major issues to the elderly users. This is suggested by the observation that the ratings of perceived competence, effort, and pressure/tension for the digital game closely approximated the corresponding ratings for the physical game. Moreover, all the usability issues identified in the study were minor and can be readily improved upon. Finally, the participants found the activity to be highly valuable and useful (in particular for memory and concentration) ‐ high ratings for value/usefulness were recorded for both versions of the game, and indeed somewhat higher for the digital version. Also the perceived choice (that is the degree to which the players felt they were choosing to do the activity) was high in both and again somewhat higher for the digital version. These two factors suggest a willingness of the participants to try the experience again.

5. Conclusion and future work This paper has described the design of a game for elderly users whose goal is to increase social connectedness and inhibit cognitive decline. A pilot study of the game with participants from the target group yielded positive results, and in particular that although the participants were in separate locations, this did not have had a negative effect on the participants’ enjoyment of the game compared to playing the physical version of the game in the same physical space as the other player. Moreover, usability and inexperience with technology did not negatively affect the experience the players had. The pilot study has alleviated the initial concerns, the questions now are: will this game and the other social portal games continue to be enjoyable over weeks and months, will people fit it into their lives, into their existing patterns, and will it lead to increasing social connectedness? These are the questions that the main study aims to answer. The main study is currently being rolled out in four European countries and aims to measure player engagement over a longer period (about three months) as well as assessing the players’ intrinsic motivation over time and the effect playing has on the participants, including well being measured pre and post game as in (Ryan et al., 2006), and whether over this time it will cause an increase in a player’s social connectedness, measured by an adaptation of Cohen’s Social Network Index (Cohen, 1997).

References AAL Joint Programme (2013), Join‐in. Online; http://www.aal‐europe. eu/projects/join‐in/. Abeele, V. V. and De Schutter, B. (2010) Designing intergenerational play via enactive interaction, competition and acceleration. Personal and Ubiquitous Computing, 14, 425–433. Bassuk, S. S., Glass, T. A., and Berkman, L. F. (1999) Social disengagement and incident cognitive decline in community‐ dwelling elderly persons. Annals of internal medicine, 131, 165–173. Cattan, M., White, M., Bond, J., and Learmouth, A. (2005) Preventing social isolation and loneliness among older people: a systematic review of health promotion interventions. Ageing & Society, 25, 41–67. Cohen, S., Doyle, W. J., Skoner, D. P., Rabin, B. S., and Gwaltney Jr, J. M. (1997) Social ties and susceptibility to the common cold. JAMA: the journal of the American Medical Association, 277, 1940–1944.

456

Daire Ó Broin and Ross Palmer Cowan, N. (2010) The magical mystery four how is working memory capacity limited, and why? Current Directions in Psychological Science, 19, 51–57. Fratiglioni, L., Wang, H.‐X., Ericsson, K., Maytan, M., Winblad, B., et al. (2000) Influence of social network on occurrence of dementia: a community‐ based longitudinal study. Lancet, 355, 1315. Goldstein, J., Cajko, L., Oosterbroek, M., Michielsen, M., Van Houten, O., and Salverda, F. (1997) Video games and the elderly. Social Behavior and Personality: an international journal, 25, 345–352. Gregoire, S., Bouffard, T., and Vezeau, C. (2012) Personal goal setting as a mediator of the relationship between mindfulness and wellbeing. International Journal of Wellbeing, 2. O’Halloran, A. M., P ́enard, N., Galli, A., Fan, C. W., Robertson, I. H., and Kenny, R. A. (2011) Falls and falls efficacy: the role of sustained attention in older adults. BMC geriatrics, 11, 85. Ryan, R. M., Mims, V., and Koestner, R. (1983) Relation of reward contingency and interpersonal context to intrinsic motivation: A review and test using cognitive evaluation theory. Journal of Personality and Social Psychology, 45, 736–750. Ryan, R. M., Rigby, C. S., and Przybylski, A. (2006) The motivational pull of video games: A self‐determination theory approach. Motivation and Emotion, 30, 344–360. Steptoe, A., Shankar, A., Demakakos, P., and Wardle, J. (2013) Social isolation, loneliness, and all‐cause mortality in older men and women. Proceedings of the National Academy of Sciences, 110, 5797–5801. Zunzunegui, M.‐V., Alvarado, B. E., Del Ser, T., and Otero, A. (2003) Social networks, social integration, and social engagement determine cognitive decline in community‐dwelling spanish older adults. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 58, S93–S100.

457

Sports Games’ Role for Learning Health Knowledge Kelly O’Hara1,2, Dulce Esteves1,2, Rui Brás1,2, Ricardo Rodrigues3, Paulo Pinheiro3 and Marco Rodrigues1 1 Sport Science Department, University of Beira Interior, Covilhã, Portugal 2 Research Centre in Sport Science, Health Science and Human Development, Vila Real, Portugal 3 Business and Economic Department, University of Beira Interior, Covilhã, Portugal NECE Research Centre ohara@ubi.pt desteves@ubi.pt rmmb@ubi.pt rgrodrigues@ubi.pt pgp@ubi.pt Abstract: Design learning environments in order to develop 21st century skills is crucial to create in learners “adaptive expertise” or “adaptive competence”. Educational/serious games, accomplished with appropriate learning environments, facilitate teaching through experience by offering immediate feedback and engaging the learner’s attention. The aim of this paper is to develop and applied serious games in health education, using physical exercise and sports as a learning environment. Methods: 201 high school students (15‐22 years, 17.56±1.37) were exposed, during eleven weeks, six hours/week to a learning sport‐game environment in other to promote physical activity and increase the awareness of its benefits. To evaluate the process effectiveness, two groups were formed, the control group (CG) and practical group (PG) that was exposed to serious games. To evaluate the learning process, a survey was developed and tested, resulting in 23 items distributed across the contents under investigation: Caloric Balance (CB); Posture (P), Heart Rate (HR). Student’s knowledge perception about each concept and their need for acquiring more knowledge related to health behaviors were also observed. A Pre and Post‐ intervention test was made. Results: Significant differences were observed between CG and PG in post‐intervention assessment (CB= p<0.000; HR p<0.000; P<0.000) where PG achieved better performance. Assessing the students’ knowledge perception about the contents (KP) and students searching for more knowledge information (SKI), for KP there is no significant difference between pre and post‐test (pre: p<.321 and post: p<.051). However for SKI after intervention results show that students of PG present significant difference (pre: p<0.172 and post: p<0.005) for searching knowledge information. Conclusions: Results show that a learning environment based on serious games can be very useful to motivate and promote students success, therefore should be integrated in school curriculum as a strategy of increasing school health and students’ wellbeing and quality of live. Keywords: learning environment; sport; serious games, health promotion, school‐based approach

1. Introduction Society has suffered a deep transformation from reliance on an industrial to a knowledge base. Design learning environments in other to develop 21st century skills is crucial to create in learners “adaptive expertise” or “adaptive competence”, i.e. the ability to apply meaningfully‐learned knowledge and skills flexibly and creatively in different situations (Dumont and Istance 2010, Spire 2008). To develop those competences, student’s need to learn how to generate, process and sort complex information; to think systematically and critically; to make decisions by weighing different forms of evidence; to ask meaningful questions about different subjects; to be adaptable and flexible to new information; to be creative; and to be able to identify and solve real‐world problems. The learning outcome can be implicit or explicit and is dependent upon the underlying game mechanics and how the content is integrated into the game so the learning is intrinsic to play. For Gee and Shaffer (2010) in formal education there are examples where games used with sufficient support are shown to be motivational and an aid to learning high level or complex skills. The authors argue that games are good for teaching and assessing because they provide appropriate challenges since game design is based on the deep human inclination to play games as a source for highly motivated learning (Gee, 2003). Other authors (Rodrigues et al 2012) consider that if built on previous information, serious games require problem solving and critical thinking. The use of serious games has not yet transferred to the classroom (Gee

458

Kelly O’Hara et al. and Shaffer 2010, Rodrigues et al 2012) because games teach and assess 21st century skills, such as problem solving, collaboration, negotiation etc., that are not the foundation of the current education system. Games with a direct link to the curriculum are more likely to be used in the classroom, especially if the game can provide appropriate assessment and fits into existing lesson structures (Rodrigues et al 2012, Popescu et al 2013). To Rodrigues et al (2012) the criterion for using a game is often whether it will make the teacher’s life easier and not developing learning skills. Serious games to health education are a growing domain, not only with simulators in medical schools, but also in games address to general population. For example the Wii Fit has been recognized as a way of training players in certain appropriate behaviors that will benefit their health (Ulicsak and Wright 2010). The use of serious games in high‐school context should comprehend not only the development of intellectual capacities (retained knowledge of concepts, memory, process and sort complex information; think systematically and critically, etc.) but also social skills (Wouters et al 2009) so, the learning environment should be based on complex problems with different information sources (virtual and real) and different tasks that require different abilities from the student. Those additional tasks must maintain the motivational aspect that serious games achieve. The use of physical exercise and sport is a possible strategy, since the majority of high school students willingly participate in physical activity, especially in the form of team games (Bak‐Sosnowska and Skrzypulec‐Plinta 2012). The creation of a learning environment that includes problem solving and team exercise tasks is particularly important for health education and exercise promotion. However it tends to decrease between ages 11 to 15 for most European countries (OECD 2012). If the learning outcome is understand the benefits of an active lifestyle for health and understand the caloric balance, in order to reversing the overweight and obesity prevalence in young’s and to promote an active lifestyle. Learning environment must comprehend movement and exercise. So, this investigation concerns the development and application of serious games for health education, using physical exercise and sports as a learning environment, concerning three major concepts: Caloric Balance (CB) – obesity is one of the biggest health problems and has substantially increased over the past 20 years (Howard‐Jones 2010). Between 1990 and 2009, the increase in adult population was 9 % to 17% in Organization for Economic Co‐operation and Development (OECD) members, 23% to 34% in US. Much adult health behavior, such as eating habits and physical activity patterns are established during childhood and adolescence (Magarey 2003, Singh 2008, Garber 2011). Understanding the caloric balance allows students to evaluate the amount of energy ingested and the consequent energy that need to be wasted (in physical exercise, for instance), in order to maintain the caloric intake equilibrated with energy spending. Heart Rate (HR) ‐ understanding cardiovascular system and it relationship with physical activity can fight sedentary behaviors. In 2001, only 32% of high school students participated in daily physical education classes (Centers for Disease Control and Prevention 2002). Understanding cardiovascular system and it relationship with physical activity can fight sedentary behavior’ and adequate the exercise intensity to individual capacities. Posture (P) ‐ in Europe 20 or 30% of adults are affected by muscle skeleton pains. In Portugal about 1.2 million of persons have back pain every week. Postural problems become evident in the course of growing. Therefore preventive measures, education habits and right life styles are needed (Jones 2003). The contents tasks considered meaningful real‐life problems since it has a key role to play in bolstering the relevance of the learning being undertaken, supporting both engagement and motivation. The learning environment comprehends the use of different tools, technological devices and information systems, in order to create different solicitations on students.

459

Kelly O’Hara et al. Since studies in high school (Yazzie‐Mintz 2007, National Research Council and the Institute of Medicine 2004) found that almost a quarter of students surveyed indicated more dissatisfaction than satisfaction with their school experience, our aim is to applied those games situations to young people who are unmotivated and disinterested with school activities.

2. Methods 2.1 Participants 201 high school students, aged between 15 and 22 years (17.56±1.37), both genders (Female=80, Male =121), were selected and randomly assigned to two groups, control group (CG) n= 99, and Practical Group (PG) n=102. The protocol was approved by the University of Beira Interior Research Ethics Boards, Portuguese Government Educational Ministry and School board. Parents were informed and gave their permission for the student’s participation.

2.2 Procedure All research interventions were conducted by the investigators. Before and after intervention both groups performed a test questionnaire to assess their basal knowledge contents (CB, HR and P). it was also assess students’ knowledge perception about the contents (KP) and students searching for more knowledge information(SKI). To ensure content and face validity, all questions were reviewed by an expert panel of professors and researchers not involved in the study. The board consisted of 3 sport scientists (with research experience), one expert researcher on market studies survey development and one expert researcher on knowledge management. To guarantee clarity, comprehension and time to complete all items, the survey was reviewed by 2 experts and 40 students of same age interval that did not participate in the investigation. The experimentation occurred during eleven weeks, for 6 hours. Game description The game develop by the research team uses physical activity to promote not only learning goals but also social, psychological, personal development and teamwork. The game conceptions began with concept selection that we intend to transmit to students (CB, HR and P). For each concept a game learning situations was created. The Game stated by introducing the student to the concept (CB or HR or P), the members of their team and the games rules. Students were divided in teams, since teamwork as the potentiality to promote cooperation, cohesiveness over competition and each team had to explore the problem– situations for solving the game. Through the course of the game they have information that they need to consult, reflect upon and debate to solve the mystery. The game‐situation environment are autonomous, players need team work, develop communication skills, perform a broad range of actions including performing self – experiments, interacting with other teams, reading and gathering background information on the concepts. In the current games, there are goals that players can achieve but the way they choose to play determines in which the actions can be performed. As a centered learning environment, criteria of challenge, curiosity, control; learning includes competence and direction in the face of novelty, complexity, and ambiguity. A Game example: Game 1 ‐ “Eating without being fat”. The aim was to show that calories are fundamental in supplying energy to the organism but if they are not completely used up they contribute to weight increase. Each pair of students received a card with a food type. First they had to identify the quantity of calories of this food type (using video information) and then define the effort that they have to make to burn the excess calories, in proportion to the amount of effort needed to complete the game. They were then asked to complete a circuit with more or less difficulties (different types of obstacles, travel distance, time do accomplishes the game), depending on the number of calories per effort.

460

Kelly O’Hara et al. Game 2 ‐ “Looking for the good fat cell”. The aim was to show that there are various types of fat cells (good and bad) and their consequent implications on the functioning of the organism. To transmit this knowledge, researchers created a circuit that combined speed, travel distance, balance and skill accuracy. To transmit the notion of bad fat cells, the student’s ability to complete the circuit with mobility was decreased by adding weight to their limbs and by using uncomfortable clothes. Motivation, social and collaborative learning, communication, knowledge about food, space / time and intensity of the situations were key variables to achieve the goals. The experimental design considerer the fundamental principles of learning environment [Dumont and Istance 2010, OECD 2007, Danish, 1996) trough:

encourage curiosity (exercises led to seeking and integrate information about body functioning);

perceived by students as relevant work to their own personal goals (exercises with important concepts for their own health, to improve performance, controlling anxiety a placement test);

challenging situations (exercises with progressive and competitive goals);

encourages teamwork (with scores exercises, need to communicate and exchange ideas);

the use of technological equipment for student motivation (exercises with Polar heart rate monitors);

demonstrate and explain how simple scientific concepts can improve everyday activities (exercises for postural awareness of the issues and implications on the day‐to‐day).

Considering the key Physical Activity and Health recommendations, games were performed with moderate‐to‐ vigorous intensity (O'Donovan et al 2010, Pangrazi 2003) controlled by heart rate monitoring (Polar Rs100 Basic Heart Rate and Timing).

2.3 Statistic analysis The data was analyzed using t‐Student test for independent variables. A p value of.05 or less was considered as significant. The data were analyzed using the software Statistical Package for the Social Sciences (SPSS 19 for Windows). A 5‐ point Likert‐type scale was used (1=Strongly disagree, 2=Disagree, 3=Neither agree nor disagree, 4=Agree and 5=Strongly agree).

3. Results At baseline, no significant differences were found between the CG and PG in terms of background concerning knowledge about the learning contents (CB= p<.22, HR=p<.525). After intervention, results show (Table 1) that for all contents there were significant difference (p<.000), between groups (CG and PG). Table 1: Comparisons between CG and PG for each knowledge content in post‐test PG.

Sig ta

CG. 56,7%

87,9%

‐8,235

.000

40,2%

53,9%

‐6,281

.000

57,7%

77,8%

‐11,188

.000

Content

Comparing pre‐ and post‐test (Table 2) it is clear that PG had significative better results for all Knowledge contents. Table 2: Comparisons between pre and post‐ test for each group (CG or PG)

HR pre_pos t CB pre_pos t P pre_pos t

CG.

PG.

Sig ta

‐1.9p.p.

‐27.9 p.p

‐6.864

.000

‐.6p.p.

8.4 p.p

‐3.694

.001

‐.6p.p

14.7p.p

‐7.963

.000

461

Kelly O’Hara et al. Assessing the students’ knowledge perception about the contents (KP) and students searching for more knowledge information(SKI), for students’ knowledge perception there is no significant difference between pre and post‐test (pre: p<.321 and post: p<.051). In respect to SKI after intervention results show that students of PG were searching for more knowledge information that students from CG with significant difference (pre: p<0.172 and post: p<0.005).

4. Discussion The main purpose of this research study was to examine high school unmotivated students’ achievement in health concepts learning through an educational games, and if game based learning can be effective in this kind of students population. This study showed that games can be utilized in formal learning environments to support students learning and simultaneously practice physical activity. Students achieved statistically significant learning gains when learning about health concepts. Considering the results about knowledge contents, our findings are in agreement with literatures (Jennifer et al. 2005, OECD 2007, Spires 2008, Dumont and Istance 2010, Howard‐Jones et al 2011) were learning can be optimized if students a) were involved in real life and meaningful problem‐based activities, b) if is based on situated practice, c) extended engagement of self as an extension of an identity to which the player is committee, d) participate in practical situations (games) that operates at a player’s “outer regime of competence” e) are challenging and f) participate in a cooperative team work. Our results suggest that the exposure to new content, new ways of presenting knowledge, context and real situations of interest (Dumont and Istance 2010), special is this kind of population. Create motivation and engagement in the learning process by exploring, interacting and collaborating are crucial for its exit (Yazzie‐ Mintz 2007). These results are in agreement with studies (Huebner, et al. 2000, Jennifer et al 2005) that suggest that students who feel connected to school are more likely to commit themselves to the learning process and to exhibit more positive behaviors and attitudes. Those, who perceive school as positive and relevant, are more likely to exhibit participatory classroom, such as seeking for new information about the learning contents. According to Zyda (2005), games based learning activities are designed to help achieving a balance between fun and educational value since it is one of active learning methodology. The use of games with learning purposes has also benefitted from the use of educational technologies (Padrós et al. 2011). Our experimental combine not only technology but also physical activity in other to develop problem solving competences, decision making. If school activity were treated in terms of learning, playing, then students could be more thoroughly engaged in the learning process (Popescu et al. 2013). Considering the obtain result for SKI, the proposed learning games had a positive effects in this problematic student’s (as already mentioned with poor motivation and performance, lack of participation in school activities (National Research Council and the Institute of Medicine 2004, Hafen et al. 2012). During and after intervention they were motivated and interested for acquire more information about the learned knowledge, as SKI results showed. This finding can be a contribution to the proposed holistic interventions (Larson 2000, McNeely 2002, Haraldssona 2008, Douglas et al. 2010) in other to promote well‐ being related to stress. For example, one of the HR games situations was to develop student’s ability to control the HR frequency in several contexts (stress, anxious, fear). This findings toward with research, that learning would be optimized if student’s were involved in real‐life and meaningful problem‐based activities. The proposed games situations contribute to the development of the new educational challenge on the relationship between physical activity and health promotion, showing how it can be possible to learn but also develop others skills in a motivating way.

462

Kelly O’Hara et al.

5. Conclusions The present study allowed us to help students to develop team work, seeking changes in lifestyle and health promotion behavior’s, learn new information, experimenting, discussing, taking decisions in different situations. This paper prepossesses a School‐Based Interventions that increases both physical activity level and knowledge about some fundamental health knowledge concepts in a motivated, playing and learning way, even for difficult student population. Considering the results, this kind of learning environment should be integrated in school curriculum as a strategy of increasing school health and students’ wellbeing and quality of live. Since learning content should be an ongoing part of playing the game, emerging from game play as a consequence of game play rather than an antecedent (Charsky and Ressler 2011), sport or physical activities games can be used as an effective tool because “game” is it essence. However more longitudinal studies are also recommended to consolidate results. New games situations should be develop and applied for a longer period of time in other to identified changes healthy behaviors.

Acknowledgements Thanks to teachers, students and school staff from Campos Melo High School, Portugal, who took part in this study.

References Bak‐Sosnowska, M., and Skrzypulec‐Plinta, V. (2012) ”Eating habits and physical activity of adolescents in Katowice ‐ the teenagers' declarations vs. their parents' beliefs”, Journal of Clinical Nursing, Vol. 21( 17‐18), pp. 2461‐2468, July. Centers for Disease Control and Prevention (2002) “Youth Risk Behavior Surveillance—United States, 2001”, MMWR Surveill Summ, Vol. 51, pp 1–64. Charsky, D. and Ressler, W. (2011). “Games are made for fun”: Lessons on the effects of concept maps in the classroom use of computer games. Computers & Education, Vol. 56, No 3, pp 604‐615. Danish, S.J., Nellen, V.C., and Owens, S.S., (1996) “Teaching life skills through sport: Community based programs for adolescents”. In J. L. Van Raalte, & B. Brewer (Eds.), Exploring Sport and Exercise Psychology. (American Psychological Association, Ed.) (pp. 205–225). Washington; DC. Douglas C. S., Ito, A., Gruenewald J. and YehH. (2010) “Promoting School Engagement: Attitudes Toward School Among American and Japanese Youth” Journal of School Violence Vol. 9, No 4, pp 392‐406. Dumont H. and Istance, D. (2010) “The Nature of Learning Using Research to Inspire Practice”, Edited by Hanna Dumont, David Istance and Francisco Benavides, OCDE. Garber, et al, (2011) "Quantity and Quality of Exercise for Developing and Maintaining Cardiorespiratory, Musculoskeletal, and Neuromotor Fitness in Apparently Healthy Adults: Guidance for Prescribing Exercise." Medicine & Science in Sports & Exercise, Vol. 43, No 7 pp 1334‐59. Gee, J. P. (2003). What video games have to teach us about learning and literacy. Palgrave MacMillan, New York. Gee, J. and Shaffer, D. (2010) “ Looking where the light is bad: video games and the future of assessment”, [online] Madison: university of Wisconsin‐Madison. URL: http://epistemicgames.org/eg/looking‐where‐the‐light‐is‐bad. Hafen, C., Mikami, J.P., Gregory, A., Hamre, B. and Pianta, R.C. (2012) “The pivotal role of adolescent autonomy in secondary school classrooms,” Journal of Youth and Adolescence, Vol. 41, No 3, pp 245‐255, December. Haraldssona, K., Lindgrena, E., Fridlundb, B., Baigia, A., Lydella, M. and Marklunda, B. (2008) “Evaluation of a school‐based health promotion programme for adolescents aged 12–15 years with focus on well‐being related to stress,” Journal of the Royal Institute of Public Health, Vol.122, No.1, pp 25–33, April. Howard‐Jones, P., Skevi D., Rafal B., Jee H.Y. and Ute L. (2011) "Toward a Science of Learning Games." Mind, Brain, and Education 5, no. 1, pp 33‐41. Chapter 8. Learning with technology— Richard E. Mayer The Nature of Learning Using Research to Inspire Practice, Edited by Hanna Dumont, David Istance and Francisco Benavides, OCDE. Huebner, E.S., Drane, W. and Valois, R.F. (2000) “ Levels and demographic correlates of adolescent life satisfaction reports”. School Psychology International, Vol. 21, No 3, pp 281–292. Jennifer,F., Blumenfeld, P., Friedel, J. and Paris, A. (2005) "School Engagement." Chap. 19 In What Do Children Need to Flourish?, edited by KristinAnderson Moore and LauraH Lippman. The Search Institute Series on Developmentally Attentive Community and Society, 305‐21: Springer US, Jones, G. T., Watson, K. D., Silman, A. J., Symmons, D. P., and Macfarlane, G. J. (2003) “Predictors of low back pain in British schoolchildren: a population‐based prospective cohort study”, Pediatrics, Vol. 111, No 4 , pp 822–828, April. Larson, R.(2000) “Toward a psychology of positive youth development,” American Psychologist, Vol.55, No.1, pp 170–183. McNeely, C., Nonnemaker, J. and Blum, R. W. (2002) “Promoting school connectedness: Evidence from the National Longitudinal Study of Adolescent Health,” Journal of School Health, Vol. 72, No.4 pp.138–146.

463

Kelly O’Hara et al. Magarey, A.M., Daniels, L.A., Boulton, T. and Cockington, R.A. (2003) “Predicting obesity in early adulthood from childhood and parental obesity”, Journal of the International Association for the Study of Obesity, Vol. 27, No.4, pp 505–513. National Research Council and the Institute of Medicine “Engaging schools,” The National Academies Press , Washington, DC. O'Donovan, G., Blazevich,A., Boreham, C., Cooper, A., Crank, H., Ekelund,U., Fox,K., et al. (2010)."The Abc of Physical Activity for Health: A Consensus Statement from the British Association of Sport and Exercise Sciences." Journal of Sports Sciences Vol. 28, No 6,pp. 573‐91. OECD (2007) “Understanding the Brain: The Birth of a Learning Science”. (O. For Economic Co‐operation & Development, Eds.) Learning pp. 13–18, OECD Publishing. OECD (2012) Health at a Glance: Europe 2012, OECD Publishing. Padrós, A. Romero, M. and Usart, M. (2011). “Developing serious Games: Form Face‐to‐Face to a Computer‐based Modality”, E‐learning papers, Vol 25. Pangrazi, R.P., Beighle, A., Vehige, T., and Vack, C. (2003) “Impact of Promoting Lifestyle Activity for Youth (PLAY) on children’s physical activity”. The Journal of school health, Vol.73, No 8, pp. 317–321. Popescu, M. M., Romero, M., & Usart, M. (2013). Serious Games for Serious Learning Using SG for Business, Management and Defence Education. International Journal of Computer Science Research and Application, Vol 3, No 1, pp 5‐15. Rodrigues, R.G., Pinheiro, P. and Barbosa,J. (2012)”Online Playability – the social dimension to the virtual world”. In M. M. Cruz‐Cunha (Ed.), Handbook of Research on Serious Games as Educational, Business and Research Tools, pp. 391– 421, Hershey, PA, USA: IGI Global. Singh, A.S., Mulder, C., Twisk, J.W., Mechelen, W. and Chinapaw M.J. (2008) "Tracking of Childhood Overweight into Adulthood: A Systematic Review of the Literature." Obesity Reviews, Vol. 9, No 5, pp. 474‐88. Spires, H.A., (2008) “Twenty‐first century skills and serious games: Preparing the N Generation” In Serious educational games: From theory to practice, ed. L.A. Annetta, 13–24., The Netherlands: Sense Publishers, Amsterdam. Ulicsak, M. and Wright, M. “Games in Education: Serious Games. A Futurelab literature review” in Serious games in education. Tech. rep. (2010). [online] URL http://media.futurelab.org.uk//resources/documents/lit reviews/Serious‐ Games Review.pdf Wouters, P., Van der Spek, E.D. and Van Oostendorp, H. (2009) “Current Practices in Serious Game Research: A Review from a Learning Outcomes Perspective”. In T. Connolly, M. Stansfield, & L. Boyle (Eds.), Games‐Based Learning Advancements for Multi‐Sensory Human Computer Interfaces: Techniques and Effective Practices (pp. 232‐250). Hershey, PA: Information Science Reference. Yazzie‐Mintz, E. (2007),” Voices of students on engagement: A report on the High School Survey of Student Engagement. Bloomington Center for Evaluation & Education Policy. Indiana University. Zyda, M. (2005) From visual simulation to virtual reality to games. IEEE computer, Vol. 38, No 9, pp 30–34.

464

A Multi‐Agent Architecture for Collaborative Serious Game Applied to Crisis Management Training: Improving Adaptability of non Played Characters M’hammed Ali Oulhaci1, 2, Erwan Tranvouez1, Sébastien Fournier1 and Bernard Espinasse1 1 Aix‐Marseille Université (AMU), LSIS UMR CNRS 7296, Marseille, France 2 Groupe SII, Société d’Ingénieur et de Conseil en Technologies, Aix‐en‐Provence, France ali.oulhaci@lsis.org erwan.tranvouez@lsis.org sebastien.fournier@lsis.org bernard.espinasse@lsis.org Abstract: Serious Games (SG) are more and more used for training in various domains, and notably in the crisis management domain. Crisis management can gather several hundred stakeholders, which can present various difficulties when organizing field exercises. Serious Game constitutes a more practical alternative with specific benefits concerning detailed players’ actions tracking during a simulated exercise. Moreover, Non Played Characters (NPC) can be used to adapt the crisis management exercise perimeter to the available stakeholders as well as specific training objectives. In this paper we present a general Multi‐Agent System (MAS) architecture providing support to the behavioral simulation as well as the monitoring and assessment of human players. To each NPC is associated a so called Game Agent designed to reproduce the behavior of the actor simulated. The Game Agents are based on a deliberative model (Belief Desire Intention) with added editing features to facilitate the scenario design phase. Thus an Agent editor allows a designer to configure agents’ behaviors illustrated in this paper with the case of crisis management scenario. The behavior simulation was implemented within the preexisting SIMFOR project, a serious game for training in crisis management. Keywords: serious game, multi‐agents system, multi‐agent simulation, crisis management

1. Introduction Serious Games (SG) are more and more used for training in various domains, and notably in the crisis management domain. Crisis management can gather several hundred stakeholders, which can present various difficulties when organizing field exercises. Serious Game constitutes a more practical alternative with specific benefits concerning detailed players’ actions tracking during a simulated exercise. Moreover, Non Played Characters (NPC) can be used to adapt the crisis management exercise perimeter to the available stakeholders as well as specific training objectives. The work presented in this paper focuses on adding NPC capabilities to the pre‐existing SIMFOR Serious Game (simulation and training, SIMulation & FORmation in French) dedicated to training actors (with various level of expertise) involved in a Crisis Management situation. NPC capabilities implies being able to simulate humans’ behaviors with whom human players interact. This paper addresses the modeling and software requirements needed to support these objectives. A general Multi‐Agent System (MAS) architecture has thus been proposed providing support to the behavioral simulation as well as the monitoring and assessment of human players. To each NPC is associated a so called Game Agent designed to reproduce the behavior of the actor simulated (role incarnated). The Game Agents are based on a deliberative model (Belief Desire Intention) which is quite usual in Multi‐Agent Systems for complex behavior modeling (and simulation). This feature can be summarized in describing an agent as pursuing multiple goals (possibly with different priorities), which can be attained by plans composed of a sequence of actions either applied in the virtual world (3D environment) or resulting in interactions between agents/human players (with a self‐evolutionary response). To facilitate the scenario design, we have implemented an agent editor which will allow a designer to configure agents’ behaviors (as well as dialogues) for a SG scenario, applied here to Crisis Management training. The next section presents the SIMFOR project, a serious game for training crisis management. Section (3) discuss the NPC issue and relates to different works in the field of multi‐agent systems and behaviors simulation. In section (4), we define our Game Agent model for the SIMFOR project and in section (5), we present a short game scenario in the SIMFOR project to illustrate how the implementation phase is supported. Finally we conclude and present future works for the SIMFOR project.

465

M’hammed Ali Oulhaci et al.

2. The SIMFOR project In this section, we briefly present SIMFOR project, a serious game for crisis management as well as the general architecture combining Intelligent Tutoring System (ITS) and Serious Game (SG) elements.

2.1 SIMFOR context SIMFOR (figure 1) is a serious game developed by SII 1 company in partnership with Pixxim 2 company, in response to serious gaming call for project launched by the French Secretary of State for Forward Planning and Development of the digital economy. SIMFOR provides a fun and original approach for learning crisis management as a serious game. SIMFOR is adapted to actors’ needs and enables learners to train for major crisis management by integrating multi‐stakeholder aspect (i.e. heterogeneous learning profiles). The project objective is to create a training environment that immerse users in a crisis management situation in real‐time context and realistic in terms of environment, self‐evolving scenarios and actors.

Figure 1: Screenshot from SIMFOR project SIMFOR is a multi‐player game and allows different people to learn skill (shared or specific) in the same game. This is possible because SIMFOR does not target only the specialists in the field of crisis management, but rather the non‐professional. Managing a major crisis can mobilize several hundred stakeholders, from the regional Prefect in his office to the firefighter in the field. These stakeholders are required to communicate and work together in order to restore a normal situation.

2.2 The general architecture of the system The SIMFOR architecture combines elements from the Intelligent Tutoring System and Serious Game domains (see (Oulhaci et al., 2013a) for a detailed presentation). Our goal is to associate the playful learning of SG and the different modules of an ITS (domain model, learner model, pedagogical model) to get the optimal learning environment. The SIMFOR architecture is composed of the following components: The SG module (SIMFOR): this module includes the 3D models, user interface (as a communication channel between the learner and the system), simulation module (for natural phenomena such as fire propagation), and data models. This module constitutes the former ”perimeter” of the Simfor SG.

http ://www.groupe‐sii.com http ://www.pixxim.fr

466

M’hammed Ali Oulhaci et al. The Behaviors Simulation module: which allows simulating humans behaviors to replace absent players with ”artificial” actors (Game Agent). The Evaluation module: the evaluation module aims to provide skills assessment of players in real time to the pedagogical module. The Pedagogical module: which plays the role of a virtual tutor accompanying the learners by providing support and help during (and after) their training. Knowledge representation module: All knowledge used or produced by the previous modules of our proposed architecture is stored in the following models:

The Domain model: the domain model represents the different concepts of crisis management and it’s segmented into parts representing a role or a skill to learn.

The learner model: for each learner or agent, a learner model is associated. This model represents the mental state of actors at a time t.

As this paper focuses on simulation of human actors in a SG, the following section exposes the scientific issue of NPCs in the SIMFOR project and some relevant work in this field.

3. Adaptive NPC for SIMFOR The SIMFOR project faces two issues:

The simulation of human behavior of NPC players.

The monitoring and evaluation of learners during their training.

The learner assessment was discussed in (Oulhaci et al., 2013b) and (Oulhaci et al., 2013a). To deal the heterogeneous aspect of the learner assessment (assess different skills and trades), we have proposed the concept of the ”Evaluation Space”. The guiding idea is to consider a SG scenario through different view, each corresponding to a particular evaluation objective. An ”Evaluation Space” thus gathers (homogeneous) information and primitives to manipulate these information in order to produce assessments, such as a Behavioral Space (for evaluating procedural knowledge) or Social Space (for evaluating actors interaction during a game scenario). Adding NPC capabilities to the SIMFOR SG implies being able to simulate actors’ behaviors with whom human players interact (in the best case scenario without knowing the virtual nature or not of other players). This step requires extracting from domain experts nominal behaviors which players are expected to follow, and express them in suitable format. The challenge of behaviors simulation is how to transform an expert domain nominal behavior to a SG NPC behavior? Given the number of stakeholders’ as well as their skill heterogeneity, designing a scenario to establish crisis management exercises is a complex task. We present a scenario example in section 3.1 to discuss this issue.

3.1 NPC and complex behavior simulation As a SG, SIMFOR aims at immersing players in a virtual world enabling them to pretend acting as they would (and should) do in a real emergency situation. Knowledge and skills involved in such situation are various in nature as well as in terms of evaluation means, but nonetheless must be all assessed in order to certify (or not) that players know their part of the job on which many lives may depend. To better understand the heterogeneous aspect of the behaviors simulation as well as the assessment needs, let’s consider a simplified example of emerging situation scenario. This scenario starts with a TDM (Transport of Dangerous Material) truck overturned after a traffic accident. The tank is damaged and hydrocarbon is spreading over the road. A witness to the accident gives the alarm by calling the CODIS (Departmental Center for Operational Fire and Rescue Services in French) which in turn must perform four missions consequently to the alert. First, CODIS has to send a Firefighter on the scene to retrieve information about the accident (”send firefighter”). Once information on the accident is received (transmitted by the Firefighter in the ground), confirming a TDM accident has occurred, the CODIS must secondly gives instructions to an Officer (firefighter) on the measures to be taken. In a TDM accident the Officer must give the intervention order (send another Firefighter with a

467

M’hammed Ali Oulhaci et al. fire truck). Then thirdly, the CODIS must complete an information sheet on the disaster that passes later through a fax to the Mayor, Prefect and the Sub‐Prefect (sending order is not important). Finally the last mission is to inform the OCP officer (Operational Command Post) once it is sent by the Prefect. This scenario excerpt illustrates the needs of the domain, for behaviors simulation (define nominal agent behavior), as well as for the assessment process (if the CODIS is played by a human). Actors can play the same role (Firefighter), but enact different behaviors (one collect disaster information, and the other should intervene on the disaster). Moreover, we have a trades which do not belong to a sole organization (given the large number of stakeholders in crisis management), like School Principal, Mayor, media, etc. In addition, the actors’ behaviors may differ from a scenario to another (depending of the disaster nature i.e. fire, earthquake...). Therefore, the scenario designer (domain expert) must specify the actors involved their associated behaviors, disasters consequences, etc, for each scenario exercise. In the next section we present some works in the field of behavior simulation and multi‐agents architecture and how our work relates to these works.

3.2 Related work In this section, we present some representative works related to behaviors simulation in Serious Games (SG). The Artificial Intelligence (AI) has always been present in video games, more or less elaborated (depending on the video game objective) (Bakkes et al., 2009). SGs borrow much from classic video games, but SGs are more than game, a SG is a game that is used to learn something. The tools and mechanisms borrowed to the video game must be more elaborated, like AI. The AI is used to simulate natural phenomena or human behaviors to get a realistic virtual word for training (Zyda, 2005). In (Buche et al., 2003), Buche propose MASCARET, a pedagogical multi‐agent system for virtual environment for training. The MASCARET model is proposed to organize the interactions between agents and to endow them with reactive, cognitive and social abilities to simulate the physical and social environment. The physical environment represents, in a realistic way, natural phenomena. The social environment is simulated by agents executing collaborative and adaptive tasks. The MASCARET model was applied to SECUREVI, an application for fire‐fighters training. The MASCARET model allows designing a complex organization (by defining role, organization, behaviors ...), but is difficult to transpose in situations where many and heterogeneous organizations interact. For each organization, we must define roles and procedures etc, which are not adapted for larger scale crisis management exercises. In addition, the SECUREVI project does not include any assessment solution. More specifically on multi‐agent organizational modeling, the MOISE+ model (Hübner et al., 2002) (Model of Organization for multi‐agent SystEms) considers organizational structure and dynamics of a Multi‐Agent System or MAS (for example for simulation purpose). This model adds an explicit deontic relation (to structure and dynamics) to make the (artificial) agents able to reason on the fulfillment of their obligations or not. In these models, obligations and permissions are entitled to roles (such as a firefighter has to extinguish a fire and may use a fire hose which a bystander may not). As training may require actors to detect wrong behaviors (during a collaborative task between a simulated actor and human player), we would also need to allow (voluntary) erroneous behaviors which is not covered by MOISE+.

To simulate NPCs in the SIMFOR project, we have opted for BDI architecture. The BDI (Beliefs, Desires, and Intentions) model is an agent modeling standard in the field of agent behavior modeling, inspired from the human reasoning process (Rao et al., 1995), and has been widely applied. A BDI model is based on the notion of capacity, skills, beliefs, purpose, desire, and intention‐plan. Agents aim at achieving their goals by executing plans depending of their current knowledge (beliefs). These concepts allow designing and programming agents with complex behaviors. Our goal in the SIMFOR project is to provide an agent architecture helping the main expert to design NPCs for a specific scenario. These NPCs can have a nominal behaviors (perform the expected behaviors) or intentionally erroneous behaviors. The NPCs must also adapt their behaviors related to other players (learners) actions and interaction (social abilities) as well as events from the environment (3D world) that can occur during the game.

4. A BDI architecture for SIMFOR The proposed model is composed of a set of agent, actions and facts. Each SIMFOR scenario is associated with several agents model that reflect the NPCs behaviors. An agent model is represented as follows:

Model(role) = {Goals; Plans; Facts; Dialogues}

(1)

A Game Agent will play a role in a scenario, and as such tries to achieve Goals (activated i.e. evaluated as reachable,

468

M’hammed Ali Oulhaci et al. by the context) by enacting its associated plans depending on its knowledge of the situation (defined as a list of declarative Facts). Each plan is composed of actions either directed toward the environment or other agents/actors (causing different type of ”effects” on the scenario). In the former case, interactions between Game Agent/human actors (or GA/GA to fully simulate a scenario) are codified by adaptive Dialogues as a set of Sentences (see section 4.2). Figure 2 synthesizes the general structure of the GA model with an UML metamodel.

Figure 2: The SIMFOR Game Agent metamodel On an architectural or software level, an agent Engine is defined in order enliven the agents (i.e. enact the agents life cycle). A Game Agent (GA) begins to update its Facts Base through its perception of the environment (events) and messages from other agents (or players), and will select the appropriate goal according to the situation (role, ...) then select the intention (plan) that will achieve this goal and finally execute the plan. For goal selection, the GA selects first realizable goals by studying the plans feasibility for each goal. Once the realizable goals known, the GA selects the highest priority goal. If there are several goals with the same priority, the GA selects a goal randomly between the highest priority goals. This process enables an agent to adapt its behavior to the state of the environment in a broad sense (i.e. information about the virtual world and the other actors whether incarnated by agents or not). The objective of this model is to provide adaptive NPC behaviors, but we also help the scenario de‐ signer(s) with a user‐friendly and efficient tool to configure these NPCs behaviors. Thus, a graphical editor tool has been developed as illustrated in section 5.1 applied to the risk management case study.

4.1 Action modeling Actions characterize what an actor can do during a crisis management situation. These actions thus serve as model to design agents’ behaviors as well as assessment data when comparing what is done to what should have been done by human actors. Regarding the Game Agent, an action has Preconditions (expressed by a set of facts

supposed to be present in the agent Facts Base) and Effects (see fig. 2). An action can be performed in several ways: one shot (action undertaken only once), cyclic (repetitive action such as ”check fire progression”) or performed at a time t. For each action the designer can specify the number of attempts (if the number is reached without success, the agent goes to the next action). Actions influence the environment through three kinds of Effects:

Physical effects (PE): such action influences/impacts the SIMFOR 3D environment (as would a human actor through its user interface).

Knowledge modification effect (KE): direct consequence of an action is facts modification (i.e. knowledge update) which in turn can abort a goal or validate other goals or actions preconditions.

469

M’hammed Ali Oulhaci et al.

Message Effect (ME): such effect reflects the social nature of agent and actors as in the SIMFOR SG, they communicate in order to carry out the collaborative task of managing a crisis situation.

Table 1: Example of crisis management actions and their effects Action

Effect type KE, ME

Phone Fax

KE, ME Radio

KE, ME Talk

Move

KE, ME PE

Daybook

Description This action allows joining a player by phone. If both interlocutors are human player, the communi cation will be oral via VOIP (Voice over IP). If one of the interlocutors is a GA, the communication will be done as a dialogue with textual phrases exchange (see section 5.2) This action allows sending a fax to one or more recipients. Faxes are represented by preformatted HTML documents related to crisis management (to be filled in with the right informations) This action allows joining a players by radio. The player must then select a channel and press the talk button to communicate with all actors listening to this channel. To add a radio action into a plan, the designer must specify the channel and the id of the dialogue that will be used (by GA). This action allows talking with the nearest players (with a defined perimeter) This action allows to move in the virtual environment. The players can move using the mouse or automatically by selecting a chosen address in address book (either on foot or using a vehicle). the daybook simulates a web portal that allows stakeholders to relay information on the disaster The player writes the information to be shared and will be available to all stakeholders..

Table 2 illustrates actions with various effects as defined in the SIMFOR SG. Some actions may only carried out by GA (NPCs). These actions help to enrich the simulation and make it more realistic to human actors. For example, a (virtual) mayor assistant may prepare a room for a press conference. This action will take some time during which the (human) mayor can not start the press conference. These kinds of actions can also trigger some physical change in the SIMFOR 3D environment, for example activate an emergency siren, and put in place a foam pad (firefighter action). These virtual actions are defined by the scenario designer detailing their execution time, preconditions and effects.

4.2 Dialogue modeling There are several ways to interact in SIMFOR (phone, fax, radio...) but if one of the interlocutors is a GA, the communication is done in the form of textual dialogue. Partial automated interaction may result in fixed interaction lacking flexibility to reflect the various situations actors and agent may face. In order to avoid that, we have defined a dialogue process as a set of possible Sentences (see fig. 2 ), each one characterized with a context pertaining to the agents’ perception of their environment. A dialogue is thus designed as sentence

470

M’hammed Ali Oulhaci et al. tree where each node is represented by a sentence. A sentence is characterized by a list of properties described in table 2. Table 2: Structure of a sentence Sentence Attribute Id parent Children Content Preconditions Transmitted information Display type

Roles list

Description identify the sentence in the Dialogue tree list of (possibly) preceding sentences (parent nodes in the Dialogue tree) list of (possibly) following sentences (children nodes) represent the displayed text in the dialogue box contains the preconditions (set of facts) required for the GA to answer this sentence. represents the fact transmitted when the phrase is answered (by GA or human player) defines if the phrase will be displayed for all roles, only roles in the role list or all roles except the roles in the role list a list of roles involved in the sentence, used for display

In section 5.2, we will see a dialogue example and how the dialogue can be adaptive and interactive.

5. Implementation In this section we present how we carried out the agent model and the corresponding editor tool, on a software level.

5.1 The agent editor The agent editor allows the designer to set up the NPCs behaviors. The SIMFOR scenario specifies the actors, means and disasters involved during the game as well as the associated events (fire accident for example). For each SIMFOR actors (i.e. roles), the designer must associate an NPC. If the actor is not played by a human participant, it will be simulated by a GA, and conversely the GA will be disabled. Moreover, if the game begins with a NPC and a learner want to join in an ongoing game and play the role of the NPC, the GA will be disabled. Also, if a learner plays a role and for some reason leaves the game, the NPC takes over and plays his role.

Figure 3: The Game Agent edition For each SIMFOR scenario, the designer (domain expert) uses the agent editor to specify facts, actions for each agent (GA) involved in the scenario (figure 3). Facts are used for actions’ preconditions and effects, for goal preconditions and for dialogues. The Facts base (Base de faits zone) represents the facts known by the GA at the beginning of the game. Action(s) represent the actions that can be realized by the GA (these actions will be used to define plans). The trigger events (Evénements déclencheurs) are facts which can make Goals realizable (cf. agent engine in section 4). A goal is defined by its name, a priority, preconditions, previous goal (goals that must be previously completed) and finally a set of plans (permitting to achieve this goal). Each plan is

471

M’hammed Ali Oulhaci et al. represented by a set of (ordered) actions. We will see in more details the dialogues between human players and NPCs in section 5.2.

5.2 The dialogue editor In crisis management scenario, the different stakeholders must collaborate to restore the normal situation. To do this, there are several interaction processes between actors during the game. In accordance with the Dialogue and Sentence concepts defined in section 4.2, the designer can make rich and interactive dialogue with adaptive response. Within the agent editor, a dialogue editor helps the dialogue and sentence design (figure 4) and saves it in a XML format. We can also import dialogues (of phrase) to create and reuse more complex dialogues. Figure 5 presents an example of dialogue. This dialogue represents an interaction between the actor Codis and the actor Fireman1. The Codis must inform the fireman1 of the accident, and the fireman1 can answer with two choices: Ask for the road closures or not. The response of Codis actor will depend on its fact base (precondition road blocked). In figure 4 we can see the tree structure of the dialogue as well as the different sentences properties (id, content, precondition etc as described in table 2).

5.3 The agent/environment interaction The Agent model designed for behaviors simulation is implemented as a library and is completely generic. As agent actions may influence/modify the 3D environment, a communication interface with the SIMFOR 3D environment is required. This communication is based on Commands, which drive the behaviors simulation model (GA), but are also exploited as learner traces for the learner support and skill assessment, For example, when the GA wants to perform a move action, the Agent engine sends a Move command with the necessary parameters (destination, means of transport used). SIMFOR processes the request: if the 3D avatar of the GA is near the vehicle and the vehicle is available, SIMFOR carries out the trip (as a 3D animation), otherwise, SIMFOR sends an error command. This process can relate to the MASQ (Multi‐Agent Systems based on Quadrants) model (Stratulat et al., 2009). The MASQ model separates the agent mind (decisions) and the agent body (actions). A mind corresponds to the internal structure of an agent or to the decision‐making component. The body, either physical or social is parts of the environment and is connected to minds. As with SIMFOR, the mind of the agent is represented by the GA, and the body of the agent is represented by the 3D avatar in the virtual environment. In the ext section, based on the example presented in section 3.1, we illustrate how GA behaviors as well as interactions between players and GA can be edited.

Figure 4: The dialogue and phrase edition

5.4 Case study To illustrate the behaviors simulation, we present a scenario example defined by a domain expert, which describes the interaction between GA and human player. For this, we resume the scenario presented in section 3.1. Considering the CODIS role, figure 6 shows an UML activity diagram of the GA. With the given scenario (defined in section 3.1), we have defined and attributed five goals to the GA that play the role of the CODIS.

472

M’hammed Ali Oulhaci et al.

Figure 5: A dialogue example from the TDM scenario

Figure 6: Codis activity diagram The first goal ”send firefighter” is triggered when the CODIS receives the alert (from a witness of the accident), who then sends a firefighter at the disaster scene. This goal is reached by calling the firefighter and ordering him to go to the disaster scene (dialogue process). The second goal ”give instruction to the officer” consists in informing the officer of the initial actions to undertake concerning the TDM. The CODIS calls the officer and with a dialogue process gives the procedures to follow. The third goal ”Follow up the fire‐fighter for disaster information” will be triggered if the firefighter is slow to transmit information. This goal is achieved by a cyclic action repeated until the disaster information is received. The fourth goal ”warn municipal officials” consists in warning municipal officials by filling in the information sheet of the disaster and faxing it to the Prefect, Mayor and Sub‐prefect. If the CODIS is played by a human player, the information sheet will be filled in manually by the player, if not, the GA will use a pre‐filled sheet based on its Facts base content (disaster information). Finally the last goal ”inform officer for PCO place”, aims at informing the officer of the PCO location, by selecting the right address (found in the Facts base) and calling the officer. This Behavior model can be used for behaviors simulation, but also for learner assessment, as it is part of the domain model (Anderson, 1988) and is used as a reference (overlay model) for the learner model (VanLehn, 1988) (learner’s actions and knowledge). During the game, we can follow each GA actions (figure 7), its current goal, current plan and the content of its facts base. We can also modify the GA behavior through the interface (figure 7), for example we can reset a goal, add new fact (to trigger some goal), etc. This flexibility in the control of the NPCs is very useful because crisis management is a collaborative process, and the GA behaviors can influence the learner performance. In (Oulhaci et al., 2013b), we have presented different kinds of learner assess‐ment, for example the collective assessment assesses the global performance of all the stakeholders. This performance takes into account the human players (learner) as well as the NPCs.

473

M’hammed Ali Oulhaci et al.

Figure 7: GA monitoring

6. Conclusion With the growing interest in SG for training purpose, the behaviors simulation of NPCs is increasingly relevant. In this paper, we have presented how behaviors simulation could challenge SG into better training simulation. Consequently, in order to address this challenge, we have proposed a BDI‐like Game Agent architecture to simulate the NPCs. The goal of this implementation is to cover all trades of crisis management stakeholders and facilitate agent programming for better design of crisis management scenario. This Game Agent model is integrated into SIMFOR project, a serious game for crisis management. The GAs interacts with the SIMFOR environment through command system as well as human players through a dialogues system. This integration is used for behaviors simulation but also plays a role in the learners’ assessment. The crisis management is collaborative process, and the learners’ player and GA must collaborate to restore the situation. The GAs behaviors can influence learners’ performance and the GAs behaviors can be intentionally erroneous to evaluate the learners’ behavior in reaction to these errors. Our immediate work in the SIMFOR project is to focus on the collaborative aspects in the field of cri‐sis management, based on an analysis of the interaction graph permitting real‐time interpretation for better pedagogical support. On a more medium‐term perspective, primary feedback on our conceptual and architectural proposition reveals sufficient genericity to consider applying our approach to other SG training situation.

References Anderson. J, (1988). ‘The expert module’. Foundations of intelligent tutoring systems pp. 21–53. Bakkes, S. et al. (2009). ‘Rapid and reliable adaptation of video game AI’. Computational Intelligence and AI in Games, IEEE Transactions on 1(2) :93–104.

Buche, C. et al. (2003). ‘MASCARET : pedagogical multi‐agents systems for virtual environment for training’. In Cyberworlds, 2003. Proceedings. 2003 International Conference on, pp. 423–430. IEEE. Hübner, J. F. et al. (2002). ‘MOISE+ : towards a structural, functional, and deontic model for MAS organization’. In Proceedings of the first international joint conference on Autonomous agents and multiagent systems : part 1, pp. 501–502. ACM. Oulhaci, et al. (2013a). ‘Intelligent Tutoring Systems and Serious Game for Crisis Management : a Multi‐Agents Integration Architecture’. In IEEE International conference on state‐of‐the‐art research in enabling technologies for collaboration in CT2CM Track. IEEE. Oulhaci, et al. (2013b). ‘A Multi‐Agent System for Learner Assessment in Serious Games : Appli‐cation to Learning processes in Crisis Management’. In Seventh IEEE International Conference on Research Challenges in Information Science. IEEE. Rao, et al. (1995). ‘BDI agents, From theory to practice’. In Proceedings of the first international conference on multi‐agent systems (ICMAS‐95), pp. 312–319. San Francisco. Stratulat, et al. (2009). ‘MASQ : towards an integral approach to interaction’. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems ‐ Volume 2, AAMAS ’09, pp. 813–820, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems. VanLehn K. (1988). ‘Student modeling’. Foundations of intelligent tutoring systems pp. 55–78. Zyda M. (2005). ‘From visual simulation to virtual reality to games’. Computer 38(9) :25–32.

474

Nuclear Mayhem – a Pervasive Game Designed to Support Learning Trygve Pløhn Nord‐Trøndelag University College, Steinkjer, Norway trygve.plohn@hint.no Abstract: Pervasive gaming is a new and emerging gaming genre where the games are not confined to the virtual domain of the computer, but integrates the physical and social aspects of the real world into the game and blends into the player’s everyday life. Games have in general proven to be useful in different types of learning situations. Given the nature of pervasive games, it may be possible to use that type of game as a tool to support learning in a university course by providing a gameplay where the students, by playing the game, expands the area of learning beyond the lecture hall and lectures and into the students everyday life. If this is possible, the area for learning will also become pervasive and be everywhere and anywhere at any time. To address this research area, a prototype of a playable pervasive game to support learning in university studies has been designed. The name of this game is Nuclear Mayhem and it is designed to support university studies in the development of Flash based Web‐games at the Nord‐Trøndelag University College, Norway. The game Nuclear Mayhem ran parallel with the course and started same day as the course began and ended when the students had completed the exam nine weeks later. The only mandatory activity the students had to do during the course was to participate in the game and to be allowed to attend the exam they had to complete the game within a given time limit. This paper presents the experimental pervasive game Nuclear Mayhem and how the game was designed to be pervasive and support the curriculum of the course. Analysis of log files showed that 87% of the logins in the game client was done outside of the time period that was allocated to lectures and lab exercises. Although most of the logins occurred during daytime, logins where registered in all the 24 hours of a day. These numbers indicate that the game became pervasive and a part of the students/players everyday life. The log files however also suggest that there is a need for a stronger link between the progress of the game and the course to get a better learning outcome. Interviews with the players indicate that they found the game exciting and fun to play, but that the academic tasks and riddles that they had to solve during the game was too easy to solve. The paper concludes that games such as Nuclear Mayhem are promising tools to support learning and transform the area for learning to become pervasive according to the players everyday life and suggest improvements in the game for the next versions. Keywords: pervasive games, education, serious gaming

1. Introduction The terms pervasive game and pervasive gaming is widely used on a lot of different types of games, toys and experiences (Magerkurth et al. 2005). Pervasive gaming is a gaming genre where the game is not confined to the virtual domain of the computer but extend the gaming experience out into the real world ‐ be it on city streets, in the remote wilderness, or a living room (Benford et al. 2005). The players must interact with the environment and with real objects to achieve certain goals (game objectives and missions). In contrast to traditional computer games, which take place in limited and well‐defined settings, pervasive games erase the boundaries between spatial, temporal, and social expansion (Lindt et al. 2007). Pervasive games are staged in reality and their main attractiveness is generated by using reality as a resource in the game (Waern et al. 2009). It can be difficult to motivate students to devote enough time working with the academic material in the curriculum throughout the courses. This leads to students not having the necessary academic maturity and understanding of the course material when the exam is approaching, and in spite of "pressure reading" the last week(s) before the exam it is ‐ for most of the students ‐ not possible to obtain a sufficient understanding of the subject to be able to get a good grade in the course. It is desirable that students work much more smoothly with the subject matter throughout the course instead of just "burst reading" when the exam is approaching. A solution to this problem might be to expand the area for learning outside of the lecture hall to become pervasive and enable learning to happen anytime and anywhere, for the duration of the course. Given the nature of pervasive games, being games where the players are in the game everywhere all the time for the whole duration of the game, such games may be useful as a tool and a platform to extend the area for learning beyond the lecture room and into the students everyday life in such a way that the area for learning becomes pervasive. The research question is therefore as follows: Can a pervasive game be used to expand the area for learning and awareness beyond the university classroom and into the students’ everyday life and enable learning to be anytime and anywhere?

475

Trygve Pløhn To address this research question a playable prototype of a pervasive game to support university studies was designed (Nuclear Mayhem) and the game went parallel with the course it was designed to support. The duration of the course, and thereby of Nuclear Mayhem, was nine weeks. The students had to complete the game within a given time limit to be allowed to attend the exam, and participation in the game was the only mandatory activity during the course. Apart from participation in the game, everything else was voluntary, including attending the lectures.

2. Related work There has been a lot of research on the use of games in education. Research has shown that games can be used to support teaching and learning (Jenkins et al. 2003) and that the use of games in education can improve skills in many different types of areas. Research done on how to use community‐building mechanics in games to achieve learning in education suggests that social gaming has the potential to revolutionize the way students learn (Hicks 2010). In higher education, games can be used in three areas (Wang and Wu 2009):

Games can be used instead of the mandatory/traditional assignments

Games can be used to increase participation and motivation of the students

Students may, by developing a game, learn about other topics such as for example game development, mathematics, physics, programming, game design and software development

Nuclear Mayhem tries to encompass all the three points above. There are no mandatory assignments or requirements in the course the game is designed to support except that the students must participate in the game and complete it within a specified deadline. Participation in the game is also intended to motivate students to spend more time on the subject, and as a part of the game, the player/student must develop a web‐based game and as a result of this learn programming and game development. Pervasive gaming is a research area that is becoming increasingly popular and more and more scientific articles are being published on this topic. Currently, pervasive gaming is mainly taking place within the research community and is not yet being widely used commercially. Research in pervasive gaming has so far been largely technology oriented where the motivation behind the development and design of the games has been to create games that are suitable as a platform for research on the technology one wants to explore. These include games like Mobio Threat (Segatto et al. 2008), The Drop (Smith et al. 2005), Epidemic Menace (Lindt et al. 2007) and Capture the Flag (Cheok et al. 2006) where the motivation mainly is to test and explore technologies and how technology can be applied to move games out in the real world. Research on the use of pervasive games to support learning or education (serious gaming) is a very interesting research area where some research has been done. The pervasive game The search for the professor (Spikol et al. 2009) was designed to introduce social web technologies and to support team building for a university course to beginning media technology students. The search for the professor shows promise to be a useful tool but needs a clearer integration into the course work. Some research has been done on how pervasive games can benefit from being game‐mastered rather than be fully automatic (Jonsson and Waern 2008). When a game is to be used in education it is critical that the game is aligned with the curriculum of the courses (Monroy et al. 2011). To ensure the best possible coordination with the topics of the course and the course progress Nuclear Mayhem had to be game‐mastered.

3. Nuclear Mayhem Nuclear Mayhem is a prototype of a pervasive game developed to support university studies in Multimedia and Web‐game technology at Nord‐Trøndelag University College, Norway. The pervasive game, Nuclear Mayhem, supports the course by providing a gameplay that is strongly related to the course syllabus. To be successful in the game, the players have to understand and master the topics in the syllabus. The game starts on the first day of the course and ends when the examination is conducted. During this period the players (students) are in the game 24/7 everywhere and anywhere, both when they are at the university taking classes, when they are at home in bed, out partying with friends or doing whatever students are doing in their leisure time.

476

Trygve Pløhn The game engages in the player's everyday life in many ways. Clues and tasks are found on Facebook, are sent by SMS to the player's phone in the middle of the night, is a part of the cityscape in Steinkjer (for example, tags that are placed in shop windows ‐ Figure 1), is a part of student life (some of the lecturers appear to have secret messages that players must obtain) and on the Internet (clues and tasks are spread across different websites that players need to find).

Figure 1: A paper note with a game clue is placed in the window at a hair salon in the main street of Steinkjer Several of the tasks that are carried out by the players during the game give points. For those players who are keen to win the game, it is an important strategy to be the first to perform the various tasks. The players can at any time see how many points they have in the game high score list. When the game is finished, the three players with the most points will be awarded.

4. Game story The game story was constructed from the ability to support the story by referring to real life events that had already occurred (reality hack), and the likelihood that something would happen related to the story that would be referred to by the news media (newspapers, television) and would be possible to implement as a part of the game story or game plot while the game was in progress. The strategy behind this design choice was that the use of real life events would help to create awareness about the game and make the game more pervasive and the game story more "real". At the time, Iran’s alleged nuclear weapons program was often mentioned in the news. There had already been a number of different events that could be used to substantiate the game story, and the issue seemed to be so relevant that it was highly probable that one or more events could happen during the game, that would be featured in the news media and thereafter could be implemented in the game. On the basis of these considerations the background theme chosen for the game story was Iran's nuclear program. Nuclear Mayhem and the academic connection The relationship between Nuclear Mayhem and the teaching of the curriculum of the course is illustrated in Figure 2. The top line represents the game Nuclear Mayhem and the bottom line represents the university course. The game and the course both start and end exactly at the same dates. The course starts with a presentation of the curriculum of the course and information on how to register as a player in the game Nuclear Mayhem. The students are told that participation in the game and completing the

477

Trygve Pløhn game within a given time limit is mandatory to be allowed to take the exam. After this information, the teaching of the course begins with the curriculum of week 1.

Figure 2: The alignment between the curriculum of the course and the pervasive game Nuclear Mayhem. To be able to complete level 1 in the game and register the secret code in the game client at the end of level, the student/player has to learn the topics that are taught in week 1 since the academic challenge in level 1 is directly related to this. Furthermore, the academic challenge at level 2 is directly related to what is discussed in lectures in week 2, the same for level 3/week 3, level 4/week 4 and so on. The last two weeks of the course, the students will conduct the exam project, which is to develop a functioning web game. At the same time period the players in Nuclear Mayhem have been assigned a mission where they shall create a web game that will be used to distract a guard, and that they – when the game is ready – will inform the saboteur group about this by registering the game URL in the Nuclear Mayhem game client. Registering the URL address in the game client completes the game. To succeed with Nuclear Mayhems main mission, the URL address must be registered within a given deadline, a deadline that coincides exactly with the deadline for submission of the exam project in the course.

Figure 3: The game client shows the countdown towards the deadline for both the game and the course This is the only time limit that exists in the game and the game client is reminding the players about this by displaying a timer that is counting down towards the deadline second by second.

478

Trygve Pløhn

5. Participants 17 students were attending the University Course that Nuclear Mayhem is designed to support, and all of the students were male. The age distribution of the students was from 20 years to 45 years. 17 students started Nuclear Mayhem and of those students, 15 live in the city where the game takes place and two students had to commute. A total of 16 students completed the game and of those, 14 students took the exam (included the two students that had to commute). None of the students had any previous experience with pervasive gaming.

6. Methods and procedure The nature of a pervasive game such as Nuclear Mayhem that is designed to be played anywhere at any time in the players everyday setting, is such that it is practically impossible to use an ethnographic approach where we study and observe the players while they are playing the game. An ethnographic approach to register and capture the players’ interactions with the game and all the potential situations of game play, would require that the players were observed 24 hours a day, both in their private and professional life, for the whole duration of the game. Furthermore it is not possible to study the players’ interactions with the environment and the environments ubiquitous artefacts since those are not directly accessible (Jegers and Wiberg 2006). Since an ethnographic method for evaluation is unsuitable, four other methods were used to overcome the methodological challenges:

a questionnaire

interview of selected individual players

system logs of user activities

observations made by game master during the game

The game was played during a 9 week period. In this period, all the activities that were done by the players via email, Web or SMS to interact with the game were logged. A questionnaire, with both open‐ended questions with free‐text answering and multiple‐choice questions, was used to capture the player’s subjective opinions of the game. The data collection was done via a Web‐based questionnaire that the players had to complete immediately after the game was completed. The questionnaire dealt with topics such as participation in the teaching program, previous programming experience, the use of mobile devices, previous gaming experience, the types of games you usually play, how you played Nuclear Mayhem, what you liked or disliked, what motivated or demotivated you to participate in the game and how the game managed to support the course. There are too few participants in the survey to have statistically significant results, but the answers still provide an indication of the players' attitudes and opinions. Based on the responses in the questionnaire five people were chosen for in‐depth interviews. The subjects were chosen based on their attitude towards the game. Of those who were selected for interview, there were two persons with a positive attitude, one person who was neutral and two persons who were negative to the game. The in depth interviews were conducted after the exam grade was set, and the interviewees were informed about this fact and that nothing they would say in the interview, whether it was positive or negative, would make any difference to their final grade. The interviews dealt with the respondents' general attitude towards games and the game Nuclear Mayhem in particular, how they felt about the game, the experience of the individual game plots, suggestions for improvements, the use of reality and the real world as part of the game and specific and detailed questions about what they perceived as good or bad during the game.

7. Results and discussion In the duration of Nuclear Mayhem the 16 players logged in at the game client a total of 610 times. Of those logins 80 of them where in the same time period as the lectures in the course (in the classroom or at the

479

Trygve Pløhn computer lab) and 530 logins were in the period where there were no lectures in the course (before or after lectures or at dates there were no lectures at all).

Figure 4: The percentage distribution of logins in the game client within and outside of the time period allocated to lectures in the course 87% of the logins that were made in the game client was outside the time period devoted to teaching the course. The fact that so much of the game client activity occurred outside the time allocated for lectures in the course, suggests that the game managed to expand the area for learning beyond the boundary of the lecture hall. Analysis of log files also show that although the majority of logins occurred in the period from 0800‐1600, players logged into the game client most of the 24 hours a day.

Figure 5: Number of logins in the game client during day and night time The fact that there was activity in the game client around the clock suggests that the game succeeded in becoming pervasive in relation to the player's everyday life in the period the game lasted. There was only one deadline in the game that players had to comply with and to get the best possible match between the teaching of the course and the game, the players had to play and complete the game as shown in Figure 3. Analysis of the log files indicate that not all players played the game as intended and this is illustrated in Figure 6. The figure above shows when the seven level‐codes were registered by each player in the game client. To achieve the best possible match between the game and the lectures in the course the line with the actual registrations and the desired registrations should match exactly. The reason that the lines do not match is that some of the players have completed the tasks in the game later than it was intended. The fact that some players did not play the game according to the planned schedule is unfortunate because they will then not be working with the academic challenges in the game at the same time that this is being taught in the course, and those students do not get the same learning outcome of playing the game as those who complete the game as planned. The use of a game story that is designed based on real events has certainly helped to reinforce the game story and make it more exciting and real. The use of real events, that was featured in newspapers and in news

480

Trygve Pløhn broadcast on TV while the game was in progress, as a part of the game, reinforced the game story, the game plot and created more awareness about the game.

Figure 6: Registration of the seven secret codes (level codes) in the game client Table 1: Quotes from the interviewed players/students on the topic of the use of the real world and real world events as part of the game and the game plot It makes it all the more realistic ... or more real ... For my part, I think it made it more real ... when you connect it to more realistic stuff so ... it increases the tension as well. ‐ Subject A It was a good plot, there was a lot of work put in it. It was good. It made it the more exciting ... ‐ Subject B The story itself was well supported, it was well made, and the details were good. It was exciting, it was. It increased the atmosphere of the game. ‐ Subject C I think that was good because it made the story more believable. Being able to read it in the newspapers made it a bit more credible. Real. ‐ Subject D ... story was perhaps a little too serious ... and then it might be a bit difficult to take it seriously … I think it might be good ... then it becomes a little more realistic ... ‐ Subject E

All the interviewed students were positive to the use of real events in the game, even those who did not like this type of game or were negative to the game. The only objection that was mentioned was a player who thought the game might be a bit too serious, but the same player also emphasized that this made the game more realistic. Whether the students/players experienced the game as fun to play or not, seems to depend on what kind of game they preferred initially. Those who are positive to this type of games experienced the game as very funny, while those who do not like RPGs were negative to the game even if they thought the game was well made. However, most of the players had some experiences in the game which they thought were interesting or fun regardless of whether they were positive or negative to the game. Table 2: Quotes from the interviewed players/students on the topic whether they experienced the game as fun game to play I think it was awesome ‐ Subject A ... I have never had a taste for RPG games, and maybe this game it's more like an RPG. ... It was very well made, but it was not for me. ‐ Subject B ... when you entered the code in the game client ...that was very exciting. It really was the highlight. That was fun. ‐ Subject C ... it was exciting to enter the code ... I knew it was correct but it was still ... hehe ‐ Subject D ... my biggest motivation was to win the game. ‐ Subject E

481

Trygve Pløhn Another factor that may have affect on the gaming experience is the fact that the prototype only had one way through the game that all the players had to follow, and that this could have lead to an experience of railroading (Jonsson and Waern, 2008), but none of those interviewed mentioned this as a problem. All interviewees mentioned that they got a learning benefit of participating in the game, but several of them pointed out that they felt they would have had a greater benefit with an ordinary arrangement of compulsory exercises. At the same time, several of the interviewees mentioned that their participation in the game resultet in them using more time trying to understand and review the programming code using the textbook than if they had not participated in the game. Table 3: Quotes from the interviewed players/students on the topic of the learning benefits of participating in the game ... the academic assignments were too easy ‐ Subject A ... to solve the problems in the game I used the textbook to compare the programming code and I learned something by doing this ‐ Subject B ... participation in the game was a motivating factor for me to sit down and study the programming code. ‐ Subject C ... I have learned about ActionScript 3.0 by participating in the game. I did not know how to program before. ‐ Subject D ... I would say that I have learned some ActionScript 3.0 and also something about pervasive gaming by participating in the game. ‐ Subject E

The learning benefit each player got from participating in the game also apperars to be dependent on their level of expertice. Hence it can be that each player would have a larger learning benefit with a more dynamic model that adapts the professional challenges to the player's skills. All players who finished the game managed to pass the exam.

8. Conclusion This paper has presented the concept and the prototype of the pervasive experimental game Nuclear Mayhem and showed how the game is designed to support learning in university studies. Participants in the game reported that they believe they gained a learning benefit from participating in the game but that there is a need for a stronger connection between the game and the course and a more dynamic academic challenge in the game that is better adapted to the players’ academic level. An instrument in ensuring a better match between the progress of the game and the lectures in the course would be to attach a deadline to each of the seven codes and a requirement for when each code has to be registered in the game client. A deadline on each of the seven secret codes will be easy to implement in the game. All of the players, including those who were negative to the game, mentioned that they experienced all or parts of the game as fun or motivational. Nuclear Mayhem has shown that this type of game has the potential to expand the learning space towards being pervasive and support and help learning, but to better evaluate the academic benefits of participation in the game there is a need to develop a model to measure the academic benefits of participation in this type of game in terms of the learning objectives in the course.

References Benford, S., Magerkurth, C. and Ljungstrand, P. (2005) 'Bridging the physical and digital in pervasive gaming', Commun. ACM, 48(3), 54‐57. Cheok, A. D., Anuroop, S., Lei, C. and Thang, L. N. (2006) 'Capture the flag: mixed‐reality social gaming with smart phones', Pervasive Computing, IEEE, 5(2), 62‐69.

482

Trygve Pløhn Hicks, A. (2010) 'Towards social gaming methods for improving game‐based computer science education', in Proceedings of the Fifth International Conference on the Foundations of Digital Games, Monterey, California, 1822386: ACM, 259‐ 261. Jegers, K. and Wiberg, M. (2006) 'Pervasive gaming in the everyday world', Pervasive Computing, IEEE, 5(1), 78‐85. Jenkins, H., Klopfer, E., Squire, K. and Tan, P. (2003) 'Entering the education arcade', Comput. Entertain., 1(1), 1‐11. Jonsson, S. and Waern, A. (2008) 'The art of game‐mastering pervasive games', in Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology, Yokohama, Japan, 1501803: ACM, 224‐231. Lindt, I., Ohlenburg, J., Pankoke‐Babatz, U. and Ghellal, S. (2007) 'A report on the crossmedia game epidemic menace', Comput. Entertain., 5(1), 8. Magerkurth, C., Cheok, A. D., Mandryk, R. L. and Nilsen, T. (2005) 'Pervasive games: bringing computer entertainment back to the real world', Comput. Entertain., 3(3), 4‐4. Monroy, C., Klisch, Y. and Miller, L. M. (2011) 'Emerging contexts for science education: embedding a forensic science game in a virtual world', in Proceedings of the 2011 iConference, Seattle, Washington, 1940845: ACM, 622‐629. Segatto, W., Herzer, E., Mazzotti, C. L., Bittencourt, J. R. and Barbosa, J. (2008) 'Mobio threat: A mobile game based on the integration of wireless technologies', Comput. Entertain., 6(3), 1‐14. Smith, I., Consolvo, S. and LaMarca, A. (2005) 'The Drop: pragmatic problems in the design of a compelling, pervasive game', Comput. Entertain., 3(3), 4‐4. Spikol, D., Pettersson, O. and Gerestrand, A. (2009) Designing Pervasive Games to Support University Studies in Media Technology, translated by 261‐263. Waern, A., Montola, M. and Stenros, J. (2009) 'The three‐sixty illusion: designing for immersion in pervasive games', in Proceedings of the 27th international conference on Human factors in computing systems, Boston, MA, USA, 1518939: ACM, 1549‐1558. Wang, A. I. and Wu, B. (2009) 'An application of a game development framework in higher education', Int. J. Comput. Games Technol., 2009, 1‐12.

483

StartUp_EU: Using Game‐Based Learning and Web 2.0 Technologies to Teach Entrepreneurship to Secondary Education Students Aristidis Protopsaltis1, Thomas Hainey2, Spiros Borosis3, Thomas Connolly2, Jesus Copado1 and Sonia Hezner1 1 Institut für Lern‐Innovation, Friedrich‐Alexander‐Universität, Erlangen‐Nürnberg, Germany 2 University of the West of Scotland, Paisley, Scotland, UK 3 Menon Network EEIG, Brussels, Belgium aristidis.protopsaltis@fim.uni‐erlangen.de thomas.hainey@uws.ac.uk thomas.connolly@uws.ac.uk spiros.borotis@menon.org.gr jesus.copado@fim.uni‐erlangen.de sonia.hetzner@fim.uni‐erlangen.de Abstract: The present paper reports on the pre‐piloting of the StartUp_EU Lifelong Learning European Project. StartUp_EU is a project for secondary schools that simulating the excitement and creative innovation of starting up a new company. The project developed a series of mini‐games, an encompassing Web2.0 platform and a collaborative competition for secondary school students, between 14 and 18 years of age, to inspire high tech entrepreneurship. The platform was based on the open source ILIAS Learning Content Management (LCM) platform and the mini‐games were developed in 3D Flash. Each mini‐game focused on a specific challenge/task associated with business start‐up. Students participated in a competition completing a series of 8 tasks simulating the process of arriving at an idea for a business to a real product and creating an elevator pitch. Students had to fill in a pre‐competition questionnaire at the beginning and a post‐competition questionnaire at the end of the competition. The questionnaires were completed online and the pre‐piloting lasted eight weeks, from the middle of January 2013 until the middle of March 2013. Forty seven (47) students took part in the pre‐ piloting with the vast majority of them never taken part in a competition about entrepreneurship and had limited or no experience with game‐based learning. This paper presents the results of the pre and post questionnaires. Keywords: game‐based learning, serious games, entrepreneurship, Web 2.0 technologies, education

1. Introduction Europe faces a number of challenges that can only be met if it has innovative, well‐educated, and entrepreneurial citizens who, whatever their walk of life, have the spirit and inquisitiveness to think in new ways, and the courage to meet and adapt to the challenges facing them (EACEA, 2012). It is critical for Europe to maintain a knowledge‐based economy and be at the forefront of technological, innovative entrepreneurship to maintain its competiveness. To meet these targets young people should study and seek careers in the scientific and technological fields and understand entrepreneurship. In a discussion entitled: ‘Educating the Next Wave of Entrepreneurs’ at the World Economic Forum 2011, it was concluded that the earlier an entrepreneurial spirit is encouraged the better the results for society. This project field was chosen to directly address the motivation of entrepreneurship and innovation within European secondary school students. StartUp_EU is designed to motivate secondary school students by simulating the excitement and creative innovation of a start‐up company. The project has created a series of educational games to foster the development of entrepreneurial skills on a Web2.0 technology platform where secondary school students will learn about entrepreneurship through inspiring and thought‐provoking videos, online workbooks covering business and marketing plans, and presentation skills. Students are then supported to develop their own business ideas collaboratively, and across Europe if possible. The process mirrors the idea creation, barriers and problems in developing new technology and building a company. The goal is to enable students to understand the problems and rewards of working in the exciting high tech area and inspire students to seek out careers in this vital European sector. Through reflection activities students will have the opportunity to understand what factors influenced their success or failure.

484

Aristidis Protopsaltis et al. According to the Key Competence Framework, the entrepreneurship key competence refers to an individual’s ability to turn ideas into action. It includes creativity, innovation and risk taking, as well as the ability to plan and manage projects to achieve objectives (Brag & Henry, 2011). The Web2.0 platform will provide a mechanism for students to turn their ideas into action and includes: communications, facilities for partner finding, inspirational videos, online materials, competition submission, and online presentation tools. The reality of creating a new idea, analysing its potentials and the real need, will be set against the problems and costs of developing the idea into a virtual product for a market. The students will have the option and ability to play the mini‐games autonomously. Digital computer games have now been around for over three decades and the term games‐based learning (GBL) has been attributed to the use of computer games that are thought to have educational value, however there has been much debate surrounding this theory (Pivec, 2009). Hainey, Connolly, Boyle and Stansfield (2011) show that GBL has been applied in a number of different fields such as medicine, languages and software engineering. Further research (de Freitas, 2006; de Freitas & Neumann, 2009; Egenfeldt‐Nielsen, 2005; Prensky, 2006; Squire, 2004; Squire & Jenkins, 2003) has shown that serious games can be a very effective as an instructional tool and can assist learning by providing an alternative way of presenting instructions and content on a supplementary level. GBL and Serious Games can promote student motivation and interest in subject matter resulting in enhanced learning effectiveness. Learning through games offers increased motivation and interest to learners through introducing fun into the learning process. Adding fun into the learning process makes learning not only more enjoyable and compelling, but more effective as well (Prensky, 2002, p. 4). One of the main characteristics of GBL is the fact that the instructional content is presented together with fun elements. A game that is motivating makes learners become personally involved with playing it in an emotional and cognitive way. By engaging in a dual level, learner attention and motivation is increased (Protopsaltis et al., 2011). Systematic literature reviews (Boyle et al., 2012; Connolly et al., 2012) have indicated that playing computer games confers a range of perceptual, cognitive, behavioural and affective, motivational impacts and outcomes where the most frequently occurring outcomes and impacts were affective and motivational followed by knowledge acquisition/content understanding. Computer games have also had an effect of students learning styles. Beck and Wade (2004) examined a large number of young professionals and found that their approach to learning was deliberately overlooking the structure and format of formal education. Young professionals extensively used trial and error, welcoming contribution and instruction from peers, and emphasising 'just in time' learning to fulfil their needs and complete their tasks. All of these skills are considered essential in the modern world and GBL can assist towards developing and practicing them. Furthermore, the next generation of jobs will be characterized by increased technology use, extensive problem solving, and complex communication (Levy & Murnane, 2004). These are skills that go beyond typical reading, writing, and arithmetic of years past. It is not only what students need to learn that is shifting, but also how and when they learn. Students of today are growing up with laptops, tablets, cell phones, and video calls, and they expect to use this technology in their daily interactions (NCREL & Metiri, 2003). Additionally, the skills required for success in games such as thinking, planning, learning, and technical skills are also sought by employers (Federation of American Scientists, 2006). Games are frequently cited as important mechanisms for teaching 21st century skills because they can accommodate a wide variety of learning styles within a complex decision‐making context (Squire, 2006), foster collaboration, problem‐solving, and procedural thinking (Johnson et al., 2011) which are important 21st century skills and important aspect of entrepreneurship.

2. Method 2.1 Participants Sixty three (63) participants, between 14 and 18 years of age, were registered to the StartUp_EU platform to participate in the initial piloting, forming 13 groups from 4 schools. The participants were recruited via their schools. The consortium partners contacted schools affiliated with their institutions to invite them to participate to the StartUp_EU pre‐piloting. 49 participants filled out the pre‐piloting questionnaire and after

485

Aristidis Protopsaltis et al. data refinement, 47 questionnaires were used for the analysis. 14 participants (29.8%) were female and 33 participants (70.2%) were male.

2.2 Material Different types of material were available to the students and teachers through the Web 2.0 platform. The materials were videos, mini‐games, how to guides and examples. Videos are videos of 2 to 3 minutes that set the task within the narrative framework and declare the task's goals and expected outcome. Mini‐games are self‐contained games to be played for stimulating certain skills to be applied in the challenge at hand and save the scores within the game. How to guides are short practical guides that assist in solving the tasks. There are three how to guides, one explaining the StartUp_EU competition, the second explaining the challenges and the third explaining how to use the platform. Examples are useful to enhance the quality of student work by modelling the expected output. The examples were taken from real IT companies (Apple, Dell, Microsoft, etc.). The game rules were presented in written documents describing the rules, roles, deadlines and assets. All the materials were made available through the StartUp_EU platform.

2.3 Apparatus Two types of apparatus were used in this study a Web 2.0 Learning Content Management System (LCMS) and five independent mini‐games. 2.3.1 Web 2.0 LCMS The StartUP_EU platform was based on the ILIAS 4.3.0 version. ILIAS is a SCORM compliant LCMS developed at the University of Cologne/Germany and is available as Open Source Software (OSS). The technical characteristics available to the users were the following:

Integration/upload of text, images, audio, video in various file formats including: (jpg, png, gif, mp3, wav, mp4, avi, pdf, doc).

File upload, material collection and creation of containers such as folders for topics. Uploads were restricted to a defined maximum size to prevent server overload.

Group communication enabled by the following facilities: messages that could be forwarded to a personal email account, forum, chat, communication with tutor/mentor/discussion board for individuals or teams to ask questions that could be answered by mentors or peers, a star system allowing students to rate each other’s responses and allowing mentors to provide qualitative feedback, group collaboration tools (wiki), links to Google Docs documents to allow collaboration, surveys to decide on questions within groups and a calendar to set deadlines for the submission of documents.

Group brokerage function ‐ groups can form online or offline, rights and roles are defined to support different users on the platform, groups are able to use a “notice board” to post “job offers” to recruit team members with particular skill sets and mentors/tutors can change the status of an individual’s group membership.

2.3.2 Mini‐games A set of educational mini‐games have been developed that allows students to practice and enhance their entrepreneurial skills as they develop their own business ideas collaboratively and autonomously across Europe. Mini‐games have been created to support a number of challenges including:

Sparking creativity

Building your company team

Understanding your clients

Marketing your product

How to develop your IT product.

Assessment in each mini‐game is through the use of quests and how well an individual or team performs determines their final score. Mini‐games can be played more than once to increase a player’s score (that is, increase their learning). For example, in the mini‐game ‘Understanding your clients’ the player plays the role of

486

Aristidis Protopsaltis et al. a salesman who sells a number of products. By driving a car through part of a city, the player has to find potential clients and match them to a suitable product. On finding a client, the player is allowed to ask a number of questions to help select the most appropriate product. Figure 1 provides an illustration of this mini‐ game.

Figure 1: StartUp mini‐game “Understanding your clients” As a second example, in the mini‐game ‘Marketing your product’, the player is given a marketing budget and has to identify characteristics of his clients that may indicate how the budget should be spent. For example, one client may travel by train and watch TV, while another might travel by train and use the Internet frequently. Given that two clients travel by train, this might suggest that some of the marketing budget should be allocated to train advertising with a lower spend on TV adverts and Internet adverts. Figure 2 provides an illustration of two screens for this game, one where the player is identifying characteristics of a client and the second screen showing an example of how the marketing budget is being allocated.

(a)

(b)

Figure 2: StartUp mini‐game ‘Marketing your product’: (a) finding out about the clients; (b) allocating the marketing budget

2.4 Procedure The students were set a series of 8 challenges, a preparation stage and a final stage which replicated the process of arriving at an idea for a business. The process involved building a business plan for taking an idea or pitch to the reality of a real product. Each challenge was related to a real problem or information needed to build the business plan and was introduced by a thought‐provoking video accompanied by a mini‐game and supportive material which explained in detailed how each challenge should be completed. All the material was uploaded onto the StartUp_EU platform and was accessible online. The pre‐piloting assessed three of the mini‐ games and the StartUp_EU platform.

487

Aristidis Protopsaltis et al. Students had to fill in a pre‐competition questionnaire at the beginning and a post‐competition questionnaire at the end of the competition. The pre‐competition questionnaire assessed their own experiences, attitudes, knowledge and skills, and their expectations of the StartUp_EU tools, while the post‐competition questionnaire focussed on assessing the usability of the tools, students’ motivation, pedagogy and the skills developed by the students. Additionally, both questionnaires assessed their perceptions towards the broad dimensions of entrepreneurship (EACEA, 2012). The questionnaires were completed online and the pre‐ piloting lasted eight weeks, from the middle of January 2013 until the middle of March 2013.

3. Results 3.1 Pre‐competition questionnaire 3.1.1 Experience with entrepreneurship education Students were asked to state to what degree they had had previous experience with entrepreneurship education. The metric used included the adjectives “never”, “1‐2 times per year”, “twice in a school term”, “once a month”, and “more than once a month”. There was also the option available not to declare anything as a separate option. The majority of participants (23, 48.94%) stated that they never had any experience of entrepreneurship education and 11 participants (23.40%) stated that they only had entrepreneurship education one or two times in a year. The results are as follows: Table 2: Experience with entrepreneurship education Past experience with entrepreneurship education More than once a month Once a month Twice in a school term 1‐2 times per year Never Not Applicable Total

Number 2 3 3 11 23 5 47

Percentage 4.26% 6.38% 6.38% 23.40% 48.94% 10.64% 100%

Students were also asked to describe briefly their previous experiences with entrepreneurship education programs. Some interesting comments presented by the Belgian students included the following: “For economy, we had to create a little company and create our own product.” “For an assignment for the economy class, we had to create a self‐made product. We also had to sum up all the costs and profits during the production.” “'Kinderen van dewindt' game, it was a fun game. You had to buy a building and start a business. Last year we played a game were we had to make our own company (buy a building hiring persons ...).” “We made a little virtual company and we made a business plan.” Students were also asked whether they have participated in a competition on entrepreneurship in the past, and if yes, whether they would do that again. Unfortunately only the 8,5% (4) of them had done that in the past, and these four people did not provide additional valuable information concerning their future intentions. 3.1.2 Experience with game‐based learning Students were asked to state to what extent they previous experience with GBL. The metric used included the adjectives “never”, “1‐2 times per year”, “twice in a school term”, “once a month”, and “more than once a month”. There was also the option available not to declare anything as a separate option. The results in Figure 3 indicate that the 40% of the students have no experience at all with GBL while another 31,91% only had experience once or twice per year.

488

Aristidis Protopsaltis et al.

Figure 3: Students past experience with game‐based learning 3.1.3 Self‐assessment of entrepreneurship own characteristics Based on research from the EACEA (2012) students were asked to self‐assess their own attitudes, knowledge and skills related to entrepreneurship. The skills included were: self‐awareness, self‐confidence, sense of initiative, risk‐taking, creativity, problem‐solving, knowledge on career opportunities and the world of work, knowledge of economic and financial literacy, knowledge on business organisation and processes, communication, presentation, planning, team work, exploring entrepreneurial opportunities, and design business projects. The cumulative results in percentages can be seen are shown in Table 3. Table 3: Students self‐assessment of their entrepreneurship related attitudes, knowledge, and skills. Self‐awareness Self‐confidence Sense of initiative Risk‐taking Creativity Problem‐solving Knowledge ‐ career opportunities and the world of work Knowledge of economic and financial literacy Knowledge on business organisation and processes Communication Presentation Planning Team work Exploring entrepreneurial opportunities Design business projects

Very low 0.00 0.00 0.00 0.00 0.00 4.26 0.00 2.13 4.26 0.00 0.00 0.00 0.00 6.38 2.13

Low Medium High 4.26 53.19 25.53 2.13 46.81 31.91 4.26 34.04 46.81 12.77 29.79 38.30 10.64 36.17 31.91 2.13 29.79 53.19 17.02 48.94 19.15 29.79 40.43 17.02 29.79 42.55 14.89 2.13 25.53 51.06 6.38 29.79 46.81 8.51 40.43 36.17 0.00 27.66 31.91 10.64 38.30 21.28 12.77 46.81 23.40

Very high 14.89 14.89 12.77 14.89 19.15 8.51 12.77 6.38 4.26 19.15 14.89 12.77 38.30 10.64 12.77

NA 1 2.13 4.26 2.13 4.26 2.13 2.13 2.13 4.26 4.26 2.13 2.13 2.13 2.13 12.77 2.13

The results revealed that students generally self‐assessed their entrepreneurship related attitudes, knowledge and skills from medium to high, with the only two exceptions being: knowledge of economic and financial literacy and knowledge on business organisation and processes which were rated medium to low. 3.1.4 Expectations from StartUp_EU Students were asked also to declare their expectations towards their participation in the StartUp_EU “exercise”. In particular, they were asked to declare which attitudes, knowledge and skills related to entrepreneurship they were expecting to develop through their participation in the StartUp_EU competition. The results are presented in Figure 3. Interestingly, the results showed that students expect – or would like – to develop mostly team working, creativity and problem solving skills through their participation in the StartUp_EU competition. Presentation skills, knowledge of business organization and processes, as well as their sense of initiative are close to the aforementioned first “group” of expectations with the rest of the skills, attitudes and knowledge. 1

NA: Not applicable OR I don’t answer OR I don’t want to answer

489

Aristidis Protopsaltis et al.

Figure 3: Students expectations in developing skills, knowledge and attitudes

3.2 Post‐test questionnaire Although a total of 63 students declared their participation initially to the StartUp_EU competition, only the 31 of them finally submitted the elevator pitch. Moreover, only the 27 of them completed the pre and post‐tests. This means that the analysis can be performed using the group who had filled in the pre and post‐test (27 participants) and the group with the additional 4 participants who had only completed the post‐test including that 27 giving 31 participants could be used for further exploration. Participants were requested to self‐ evaluate their participation in the StartUp_EU competition, filling in a questionnaire that included Likert‐type questions with the objectives of assessing the following:

Their intention to participate again in the StartUp_EU competition

Their usability assessment of the StartUp_EU platform

Their motivational usability to use StartUp_EU

The usefulness of StartUp_EU for learning entrepreneurship

The evaluation of the pedagogical model of StartUp_EU in its contribution of developing their entrepreneurial‐related knowledge, skills and competences

The separate evaluation of learning objects included into the StartUp_EU (videos, mini‐games)

Strongly, the assessment of the entrepreneurship‐related expectations was only a sub‐set of the complete study (analysis), which by nature was and should be more generic. The reason unfolds from the fact that, StartUp_EU employs a new approach to learning in the school‐level settings, which is currently applied to entrepreneurship learning. In line, and if effective, this approach could be used also to other key competences, like the social and civic competences. The group of 31 participants consisted of 19 males (61.29%) and 11 females (38.71%) and the group of 27 participants consisted of 11 females and 16 males (59.26%). 3.2.1 Intention to use StartUp_EU The students were asked whether they intended to use the StartUp_EU service to participate in the competition in the following year, given that they have access to the service for this period. The results were generally positive with the majority of the students from both groups specifying either strongly agree, agree or neutral. The results are shown in Table 4. 3.2.2 Usability assessment The usability assessment of the service was conducted based on the System Usability Scale (SUS) of Brooke (1996). SUS is a ten‐item attitudinal Likert scale that provides a global view of subjective assessments on usability. It is typically used after the respondent had had an opportunity to use the system being evaluated

490

Aristidis Protopsaltis et al. and yields to a single score representing a composite measure of the overall usability of the system being evaluated. SUS scores have a range of 0 to 100. In the case of StartUp_EU, the respective usability score was estimated to 63.87 for the 27 participant group and 63.15 for the 31 participant group out of 100 respectively, which is a generally positive result. Table 4: Intention to use StartUp_EU to participate in the competition in the next 12 months Answer NA or Empty Strongly Disagree Not Agree Neutral Agree Strongly Agree

27 participant group % 0.00% 0.00% 16.13% 48.39% 32.26% 3.23%

31 participant group % 0.00% 0.00% 14.81% 51.85% 29.63% 3.70%

The constituents of the usability score were also looked at to identify potential areas of improvement for the platform. Some of the key findings were the following:

6.45% in the 27 participant group and 7.41% in the 31 participant group found the service to be complex and that the functions were not very well integrated.

12.90% in the 27 participant group and 14.81% in the 31 participant group did not find the platform easy to use, stated that they did not feel confident using the service and thought that they had to learn a lot of things before they could proceed. Approximately 10% in each group stated that it was difficult to use and that they would like a technical support person to use StartUp_EU.

16.13% in the 27 participant group and 18.51% in the 31 participant group believed that most people would not learn to use StartUp_EU platform very quickly.

The results have shown that overall, minor changes are required to make the platform more appealing, less complex and to improve overall usability. 3.2.3 Motivational usability The participants were asked to assess the motivational usability of the StartUp_EU service. The respondents were asked to assess whether StartUp_EU service incorporates novel characteristics, whether it stimulated further inquiry and whether it is enjoyable and interesting. The results are generally positive showing that the StartUp_EU platform is generally enjoyable and interesting. The results are shown in Table 5. Table 5: Motivational usability of the StartUp_EU service

n=27

n=31

Question

Strongly Agree 7,41% 7,41% 3,23% 6,45% 6,45% 3,70%

Incorporates novel characteristics Stimulates further inquiry Is enjoyable and interesting Incorporates novel characteristics Stimulates further inquiry Is enjoyable and interesting

Agree

Neutral

18,52% 37,04% 58,06% 16,13% 38,71% 55,56%

59,26% 44,44% 29,03% 58,06% 45,16% 29,63%

Not Agree 11,11% 7,41% 9,68% 16,13% 6,45% 11,11%

Strongly Disagree 3,70% 3,70% 0,00% 3,23% 3,23% 0,00%

3.2.4 Usefulness of StartUp_EU The results indicate that approximately 77% of the students from both groups believed that the service is useful for learning entrepreneurship. Overall, the 51.62% from the 27 participant group and 48.15% from the 31 participant group believe that StartUp_EU is identical for entrepreneurship education. 3.2.5 Evaluation of entrepreneurship‐related characteristics The students were also asked to assess particular characteristics of the pedagogical model used that are related to entrepreneurship. The reader may remember that these characteristics were also assessed in the pre‐pilot phase (pre‐test questionnaire). The researchers decided to use the same measures so as to identify potential tensions in increase of these self‐assessment measures. To create more coherent results and identify the potential tensions, only the data from students having completed both the pre‐competition and the post‐ competition questionnaires are presented at this stage. The results are shown in Table 6.

491

Aristidis Protopsaltis et al. Table 6: Ratings of self‐assessed entrepreneurship characteristics in the pre and post‐competition Attribute Self‐awareness Self‐confidence Sense of initiative Risk‐taking Creativity Problem‐solving Knowledge of career opportunities/world of work Knowledge of economic and financial literacy Knowledge on business organisation and processes Communication Presentation Planning Team work Exploring entrepreneurial opportunities Design business projects

M before 3.59 3.81 3.70 3.63 3.37 3.81 3.30 3.11 2.89 3.78 3.74 3.70 4.19 3.19 3.30

M after 3.19 3.00 3.33 2.85 3.44 3.56 3.19 3.19 3.56 3.11 3.15 3.04 3.56 3.33 3.44

Difference ‐11.14% ‐21.26% ‐10.00% ‐21.40% +2.07% ‐6.56% ‐3.33% +2.57% +23.18% ‐17.72% ‐15.78% ‐ 17.83% ‐ 15.03% + 4.39% + 4.24%

The results suggest that participants became more hesitant towards entrepreneurship, however there is an increase in creativity, knowledgeable of economic and financial literacy, business organization and processes, inclination to explore entrepreneurial opportunities and design business projects. The decrease in scores of some of the characteristics was possibly due to lack of entrepreneurial experience of participants. 3.2.6 Evaluation of learning objects The participants were asked to assess the learning objects on the StartUp_EU platform, particularly, to assess whether the videos introducing each challenge were highly motivational, whether the mini‐games improved their learning of entrepreneurship, whether they were complicated, and whether they were useful in learning entrepreneurship. In terms of the videos, results indicate that participants did not agree whether the videos accompanying StartUp_EU were highly motivational or not, however slightly more of the participants indicated that they strongly disagreed or disagreed that the videos were motivational. The results are shown in Table 7. Table 7: Ratings of the introductory videos Rating Videos are highly motivational (n = 27) Videos are highly motivational (n = 31)

Strongly disagree ‐ disagree 45.16 % 44.44 %

Neutral 16.13 % 14.81 %

Agree ‐ strongly agree 38.71 % 40.74 %

Overall, there is a positive attitude towards the contribution and usefulness of mini‐games towards learning entrepreneurship and a quite clear opinion that the mini‐games are not complicated. The results are shown in Table 8. Table 8: Ratings of the attributes of the mini‐games Group Question The mini‐games improved my learning of entrepreneurship The mini‐games are useful for learning entrepreneurship The mini‐games are complicated

27 participant group SA A N 25.92 % 37.04 % 37.03 %

31 participant group SA A N 32.35 % 32.26 % 35.49 %

33.33 %

29.63 %

37.03 %

29.04 %

32.26 %

38.71 %

40.74 %

18.51 %

41.94 %

16.13 %

4. Discussion and conclusion This study focused on the assessment of the pre‐piloting use of the StartUp_EU service in school education. The objective was to acquire early input from the students so as to refine appropriately both the service and the evaluation tools. Interestingly, no major amendments to both questionnaires proved necessary. Though, interesting findings were surfaced for the StartUp_EU approach, despite the small sample (which hinders the generalization of the results).

492

Aristidis Protopsaltis et al.

4.1 Pre‐competition questionnaire The majority of the participants in the pre‐test (88.94%) stated that they had never had any experience of entrepreneurship education before or only experienced it once or twice a year. This education seemed to take place primarily in the context of an economy class, in a game to start a business or company and constructing a business plan. Only 8.5% of the participants had taken part in an entrepreneurship competition before suggesting that the StartUp_EU competition presents a novel learning opportunity for the participants. The majority of the participants (72.34%) also had never experienced or had only experienced GBL once or twice a year. This again indicates that the StartUp_EU platform presents a number of novel learning opportunities. Student self‐assessment of their entrepreneurship related attitudes indicated that they rated their skills from medium to high. Knowledge of economic and financial literacy and knowledge on business organisation and processes were the only exceptions to this and were rated medium to low. The results indicated that the students were relatively confident in terms of their knowledge and skills and had positive attitudes towards entrepreneurship. The participants expected to develop a number of skills from participating in the StartUp_EU competition including: team work, creativity, problem solving skills, presentation skills, knowledge of business organisation and processes and sense of initiative.

4.2 Post‐test questionnaire Students were generally positive about using the StartUp_EU service to participate in the competition in the following 12 months. The majority of the participants stated that they strongly agreed, agreed or were neutral indicating that the StartUp_EU service is appealing enough to get participants to use it in future competitions. In terms of usability of the service, the System Usability Scale score for both groups was approximately 64%, which is a generally positive result. Approximately 7% of the participants found the service to be complex with functions that were not well integrated. Approximately 14% did not find the platform easy to use and stated that they did not feel confident using the service and thought that they had to learn a lot of things before they could proceed. Approximately 10% in each group stated that it was difficult to use and that they would like a technical support person to assist to use StartUp_EU. 17% of participants believed that people would not learn to use the StartUp_EU platform very quickly. Overall, minor changes are required to make the platform more appealing, less complex and to improve overall usability. These results will feed into the next phase of platform development iteration. Participants generally found the platform to be interesting and enjoyable with regards to motivational usability. 77% of the participants believed that the service was useful for entrepreneurship education and approximately 50% of the participants believed that was identical for entrepreneurship education. With regards to participant self‐evaluation of entrepreneurship related characteristics, the results indicated that the participants became more hesitant however there was an increase in creativity, knowledge of economic and financial literacy, business organisation and processes, inclination to explore entrepreneurial opportunities and design business projects. The decrease in scores of some of the characteristics is somewhat alarming but that the consortium will look closely into it and will try to identify its causes. One possible explanation for this decrease might possible be the lack of previous entrepreneurial experience of participants which might led the participants to realize that entrepreneurial education and/or entrepreneurship was more complex than originally anticipated. However, the increase in self confidence regarding other entrepreneurship related aspects can offer a light of optimism since these characteristics are vital in today’s working environment and an integral part of entrepreneurship. In terms of the video learning objects, the results indicate that participants did not agree whether the videos accompanying StartUp_EU were highly motivational or not, however slightly more of the participants indicated that they strongly disagreed or disagreed that the videos were motivational. These results show us that additional or alternative videos are required. The Web 2.0 platform will be updated with more relevant videos in an effort to increase motivation and inspiration. These videos can be for example videos of well known successful entrepreneurs or motivational couches. Overall, there was a positive attitude towards the contribution and usefulness of mini‐games towards learning entrepreneurship and a quite clear opinion that the mini‐games are not complicated and the majority of participants believed that the mini‐games improved learning of entrepreneurship and that they were useful for learning about entrepreneurship.

493

Aristidis Protopsaltis et al. This paper has presented some of the results of a pilot study of a competition run on the StartUp_EU platform along with the learning objects presented on that platform in the form of accompanying material and mini‐ games. The results of the mini‐pilot have been analysed and used to improve the platform and learning objects for the main‐pilot which is currently in progress. Future research directions will involve completing the main pilot and analysing the results on a much larger scale to ascertain if the initial findings are consistent.

Acknowledgements This work has been co‐funded by the EU Lifelong Learning Programme under contract 518060‐LLP‐I‐2011‐1‐ UK‐COMENIUS‐CMP (StartUp_EU ‐ Be a High Tech Entrepreneur).

References Beck, J. C., and Wade, M. (2004). Got game: How the gamer generation is reshaping business forever: Harvard business school press. Boston, MA: Harvard Business School Press. Boyle, E., Connolly, T.M, Hainey, T., Hancock, F. and Boyle, J. (2012). "Engagement in digital entertainment games: a systematic review", Computers and Human Behaviour, Vol. 28, Issue 3, pp. 771‐780. Bragg, S., & Henry, N. (2011). Order 121 ‐ Study on Support to Indicators on Entrepreneurship Education: Final Report. Birmingham: GHK Brooke, J. (1996). "SUS: a "quick and dirty" usability scale". In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L. McClelland. Usability Evaluation in Industry. London: Taylor and Francis. Connolly, T.M., Boyle, E. A., MacArthur, E., Hainey, T. and Boyle, J.M. (2012). "A systematic literature review of the empirical evidence on computer games and serious games". Computers and Education, 59, 661 – 686. de Freitas, S. (2006). Using games and simulations for supporting learning. Learning, Media and Technology Special Issue on Gaming, 31(4), 343‐358. de Freitas, S. and Neumann, T. (2009). The use of ‘exploratory learning’ for supporting immersive learning in virtual environments. Computers and Education, 52(2), 343‐352. Education, Audiovisual and Culture Executive Agency. (2012). Entrepreneurship Education at School in Europe: National Strategies, Curricula and Learning Outcomes. Brussels: EACEA P9 Eurydice and Policy Support. Egenfeldt‐Nielsen, S. (2005). Beyond edutainment: Exploring the educational potential of computer games. University of Copenhagen, Copenhagen. Federation of American Scientists. (2006). Summit on educational games: Harnessing the power of video games for learning. Retrieved from http://www.fas.org/programs/ltp/policy_and_publications/summit/Summit%20on%20Educational%20Games.pdf Hainey, T., Connolly, T.M., Stansfield, M.H., and Boyle, E.A. (2011). The Differences in Motivations of Online Game Players and Offline Game Players: A Combined Analysis of Three Studies at Higher Education Level, Computers and Education, Vol. 57, Issue 4, pp. 2197‐2211. Levy, F. and Murnane, R. J. (2004). The new division of labor: How computers are creating the next job market. Princeton, NJ: Princeton University Press. North Central Regional Education Laboratory [NCREL], & Metiri Group. (2003). enGuage 21st century skills: Literacy in the digital age. Retrieved from http://pict.sdsu.edu/engauge21st.pdf Pivec, P. (2009). Game‐based Learning or Game‐based Teaching? Becta report. Prensky, M. (2002). The motivation of gameplay. On the Horizon, 10(1). Prensky, M. (2006). Don’t bother me mom, i’m learning. St. Paul, MN: Paragon House. Protopsaltis, A., Pannese, L., Pappa, D., and Hetzner, S. (2011). Serious Games and Formal and Informal Learning, eLearning Papers, elearningeuropa.info, n 25, (July 2011). Squire, K. (2004). Replaying history: Learning world history through playing civilization iii. Indiana University, Indiana, USA. Squire, K. and Jenkins, H. (2003). Harnessing the power of games in education. Insight, 3, 5‐33. Squire, K. (2006). From content to context: Video games as designed experiences. Educational Researcher, 35(8), 19‐29.

494

Measuring Effects of Reflection on Learning: A Physiological Study Wen Qi, Dominique Verpoorten and Wim Westera CELSTEC, Open University in the Netherlands, The Netherlands Wen.Qi@ou.nl Abstract: As an economical and feasible intervention, reflection demands learners using critical thinking to examine presented information, questioning its validity, and drawing conclusions based on the resulting ideas during a learning process. The aim of this study is to gain insight into the effects of practicing short, frequent and structured reflective breaks that are interspersed with the reading process of a learning material. It tries to reveal whether physiological signals can be used as appropriate indicators to reveal the actual changes of cognitive states while introducing different reflective breaks during learning. The recorded physiological signals include skin temperature, blood volume pulse, pulse volume amplitude, and pulse rate. The results show that while these embedded “reflection rituals” did not affect learners’ performance they had significantly impact on time on task, perceived learning and those learners' physiological (cognitive) states. Physiological data returned significant differences between the reading and reflection activity. Measurements of temperature and pulse rate are lower when covering the course equipped with additional reflection affordances while blood volume pulse and pulse volume amplitude are higher. In addition, applying statistics analysis to the physiological data exhumes significant differences between different types of reflection activities for those measurements including skin temperature, pulse volume amplitude and pulse. Keywords: reflective break, physiological signals, learning, reflection

1. Introduction Today’s teaching practice aims to educate the knowledge workers of the future how to master domain knowledge and develop transversal (domain‐independent) skills. The latter enables individuals to cope with requests for acquiring new knowledge and ongoing personal development. As an economical and feasible intervention, reflection is often embedded into a learning process and demands learners using critical thinking to examine presented information, questioning its validity, and drawing conclusions based on the resulting ideas. At the same time, learning through gaming format starts to gain popularity since it makes good uses of a competitive mechanism that pits the learners against each other or provides challenges for learners in order to motivate them to learn better. This paper probes two research topics in Gamed Based Learning: the effectiveness of physiological signals as indicators to monitor and correlate with the cognitive state during reflection and non‐reflection period and the potential benefits of tidy frequent and structured reflective breaks interspersed with the reading process of learning material to advance such a student/professional development.

1.1 Physiological signals and learning Scales or items referring to student workload or stress have commonly been recognized as an important variable in designing and proposing learning techniques. The major reason for measuring workload is to quantify the mental cost of performing learning tasks in order to predict learner performance. Mental workload can be affected by numerous factors that make it difficult to have definitive measurement. Physiological signals have been used by researchers as indicators of mental workload and stress for some time [Vicente, Thornton & Moray 1987; Wilson 2001]. Psychologists use physiological measurements as special identifiers of human emotions such as anger, nervousness, and sadness [Ekman, Levenson & Friesen 1983]. However, physiological data have not been employed widely to identify learners’ experience states, such as engagement and reflection.

1.2 Reflective breaks John Dewey [Dewey 1966] has stated” We do not learn from experience. We learn from reflecting on experience.” Reflective breaks have received attention from research when applied to face‐to‐face lectures [Di Vesta, Smith 1979; Ruhl, Hughes & Schloss 1987; Simpson 2004]. Despite the availability of theoretical models of reflection [Boud, Keogh & Walker 1985; Le Cornu 2009; Moon 1999] and a clear drift towards the promotion of thinking skills [Romainville 2007; Rychen & Salganik 2003], finding practical means to introduce the reflective habits to learners remains a challenge for researchers [Claxton 2006; Csap 1999] and

495

Wen Qi, Dominique Verpoorten and Wim Westera practitioners [Jaschnik 2011; Joseph 2003]. Researchers and practitioners have identified that there are several types of reflection breaks:

questioning: previous research highlights the importance of encouraging students to generate questions about the study material [Logtenberg, van Boxtel & van Hout‐Wolters 2011; Marbach‐Ad & Sokolove 2000; Pedrosa & Moreira 2009]. In this study, students deliberately and systematically exerted a questioning strategy called “student set the test”.

evoking: an evocation brings or recalls to the conscious mind what has been previously read. Conceptual works of the ”mind management” theory [Brown‐Frossard 2012; De la Garanderie 1989] suggest that this process of mental imaging allows readers to somehow transform what they have read into a mental object [Seel 2001; Vermersch 2009] and so doing to anchor it in their mind.

self‐assessing: research shows that self‐assessment can lead to significant enhancements in learning [Taras 2002] by developing students habit to evaluate the strengths and weaknesses in their own study.

2. Experiment design 2.1 Research questions In this comparative study, we have formulated two research questions that guided the experiment design.

We assume that the biofeedback measurement can bring extra information about possible contrasts between distinct activities performed within the learning process. Therefore, the first question was whether the absence/presence of reflection amplifiers impacts upon the physiological measurements of the control/treatment group. Reflection amplifiers refer to deliberate prompting approaches offering learners brief episodes of thinking while studying [Verpoorten, westera & Sprecht 2011].

In this study, we incorporate all three types of reflection breaks to establish learning as an object of attention and reflection and, so doing, to introduce students to essential components of academic literacy. For the questioning type of reflection in this study, students deliberately and systematically exerted a student set the test strategy. Therefore, the second research question was whether different types of reflective amplifiers do not have the same impact upon the physiological measurements.

Figure 1: The page design bundles content (purple frame) and affordances to develop thinking habits: reflective breaks (light green) and learning dashboard (dark green)

2.2 Method and materials The course designed for this study was a shortened version (1H) of the 4‐hour online course Seks en de evolutie (Sex and the evolution) created [Eshuis & Goltstein 2007] and offered in Dutch by the Open University the Netherlands. The course covered non trivial and interrelated notions and mechanisms as defined by Darwin and his followers: reproductive value, paternity uncertainty, mating strategies, differential investment in parenthood, etc. The course invited learners to use this theory as an interpretation grid of gender‐related behaviors observable in everyday life. The course consists of 5 chapters of 5 pages each, which contained about 200 words and one or two illustrations (Figure 1). In order not to bias the use of the different reflective breaks (see next section) by uneven levels of difficulty in the content, special attention was paid to ensure equivalence between all chapters. Each of them underwent the Flesch reading ease test [Flesch 1948] which returned an average comprehension difficulty level of 52 (SD = 4), which is comparable to the level of the Time news magazine. In addition, a systematic concept mapping procedure of each chapter ensured that they presented an even level of complexity regarding the number of new concepts introduced.

496

Wen Qi, Dominique Verpoorten and Wim Westera

Figure 2: The reflection portlets: question (2a), evocation (2b), self‐assessment (2c) The online course was delivered in the study at 2 different conditions: with and without reflective breaks (RBs). The study exposed participants to 3 types of RBs: questioning, evoking and self‐assessing. To support and condense the reflective processes of questioning, evocating and self‐assessing, 3 miniature Web applications (called portlets on the Liferay platform) were developed (Figure 2). They displayed, in a clear and graphical style.

The Question break portlet offered a note‐taking tool where the students wrote down their questions (Figure 2a).

The Evocation break portlet combined a “starts the evocation” button and a “stop the evocation” button (Figure 2b).

The Self‐assessment break portlet presented as a 5‐star visual scale (Figure 2c) that the students used to indicate their current level of mastery of a defined portion of content (for each level a standardized explanation was given).

Table 1: Compact view of the course chapters with offered reflective breaks Course chapter

Question breaks

Evocation breaks

Self‐assessment breaks

‐

Yes

‐

Yes

‐

Yes

During the study, the treatment group studied chapter 1 just like the control group: without any reflective break. This arrangement opened to participants a possibility of contrast within the learning experience and provided an internal yardstick to the chapters studied with support tools. In chapters 2‐3‐4, students got acquainted with one reflective technique (see Table 1). In chapter 5, all techniques were available. Based on their experience in the previous chapters, students could decide which one to use after each visited page. The students had to deliberately practice the offered RBs after each page visited or re‐visited. In order to consolidate this systematic reflective approach, a learning dashboard (Figure 3) was set up. It contained a built‐in reminder of the importance to practice the reflective breaks. A color scheme indicated whether or not the number of (re‐)visited page matched the number of use of the RBs. In case of match, the number appeared in green and in case of discrepancy in red.

Figure 3: The learning dashboard for chapter 4; In green, the number 4 mirrors that the student practiced self‐ assessment each time he/she visited a page of this chapter

497

Wen Qi, Dominique Verpoorten and Wim Westera

2.3 Physiological measures Physiological data was collected with the appliance Biofeedback 2000 x‐pert from SCHUHFRIED (Figure 4). This non‐invasive biofeedback system recorded the following physiological signals: a) skin temperature (TEMP), b) blood volume pulse, viz. the pulse component of the surface blood flow (BVP), c) pulse volume amplitude, viz. the amplitude of the blood volume pulse (PVA), and d) pulse rate (PR). The sampling pace was one measure every 25 milliseconds (Figure 5). The learning sessions of the students were also screen‐recorded with the software Camtasia in order to grab supplementary information about the sequencing of reading and reflecting periods.

Figure 4: During the experiment, the module is fastened with Velcro strap to the index finger of the non‐ dominant hand

Figure 5: Visualization of the 4 measured physiological signals for one subject (treatment condition)

2.4 Procedure Participants received a 15 euro iTunes voucher for their participation and were debriefed before leaving. Taken prior to the course, the background questionnaire evaluated the students’ pre‐knowledge of the course topic with 6 multiple‐choice questions. Meta‐cognitive ability was assessed for each student by their teacher on a 3‐item Likert scale. After a pre‐test, participants individually studied in one version of the course (with or without reflection breaks) according to a random distribution. Both groups were evenly invited to practice a thoughtful study freed from time pressure in order to gain as much mastery as possible of the learning material. The tracked data was the time in the course (total and per chapter), the number of pages visited (total and per chapter) and the number of time a reflective break was used. The logs also stored the choices made by learners in chapter 5 regarding the reflective breaks. After the course completion, students filled in a post‐test. This questionnaire gathered:

Evaluative feed‐back: open and closed questions collected students’ perceptions of overall satisfaction, sense of control, and feeling of learning. Questions relating to the instructional intervention were added for the participants to the treatment group.

Performance measures: a test assessed the knowledge and comprehension. Ten multiple choice questions were selected among a pool of questions tested by 137 students in a previous experiment based

498

Wen Qi, Dominique Verpoorten and Wim Westera on the same study material. The discrimination index was of .67 in average. For this index, values above 0.4 are desirable [McAlpine 2002], which located the test at a medium‐high level of difficulty. Three open questions asked students to comment on pictures with what they learned in the course. This was consistent with the design of the course that displayed carefully selected pictures on each page. A follow‐up questionnaire was administered one month after the experiment in an attempt to evaluate possible persistent effects. The follow‐up questionnaire asked students to give to an imaginary friend who ought to take the same course some advice regarding 8 study strategies, including the 3 reflective breaks. The perceived relevance of the strategies was rated with sliders on 100‐point scales, an asset available on the survey software Qualtrix.

3. Results The experiment convened 42 subjects at the same time. The sample population consisted of secondary‐school students physically present in computer rooms during the experiment. In that sense, the context of this study was close to regular schooling practice. It sought to provide more stable experimental conditions, more homogeneity in the sample and a contrast regarding the target audience of the reflective breaks. The data collected were the returns from the questionnaires (pre, post, follow‐up), the logging data and the physiological measures. Students who missed either the pre or the post questionnaire were removed from the analysis. It was the case for 2 participants in the treatment group. 40 test persons (mean age = 17 years old, 37% female, 63% male) composed the final sample: 21 participants in condition 1 (control) and 19 in condition 2 (reflective breaks). Table 2: Results for the physiological signals in with/without reflection amplifiers condition between the control and treatment groups) Results TEMP with RBs TEMP without RBs BVP with RBs BVP without RBs

Mean 33.65 30.98 49.37 49.52

Standard deviation (SD) 1.05 3.12 12.35 13.04

PVA with RBs PVA without RBs

31.18 34.53

19.57 24.8

PR with RBs

68.69

12.74

PR without RBs

60.11

12.85

p< 0.02 p<0.02 p<0.02 p<0.02

3.1 Physiological data Physiological data was collected from 4 students (2 at the control group without reflection breaks and 2 at the treatment group with reflection breaks) because limited amount of physiological sensors are available. T‐tests were conducted on 130773 paired sampled measures to compare TEMP, BVP, PVA and PR in the “with and without reflection breaks” conditions. This returned significant differences for the 4 physiological signals (Table 2). TEMP and PR are lower when covering the course equipped with additional reflection affordances while BVP and PVA are higher. The accuracy of the measure is acceptable (the observed difference in temperature is far above the variations that could be imputed to the recording system (0.01C) and the other measures embed compensations for interference and automatic averaging of data at baseline). Table 3: Mean and SD of the physiological signals with different reflection amplifiers TEMP (mean) (SD) BVP (mean) (SD) PVA (mean) (SD) PR (mean) (SD)

reading 33.350 1.193 49.334 11.184 28.415 17.928 66.893 11.553

questioning 34.055 0.590 49.441 17.524 45.817 25.509 68.721 17.096

499

evoking 33.722 0.717 49.389 11.276 26.190 15.747 70.912 12.383

self‐assessing 33.634 0.641 49.503 9.442 24.146 11.702 64.874 8.562

Wen Qi, Dominique Verpoorten and Wim Westera In order to refine the analyses, the screen recordings of the learning session from the subjects in the treatment group were analyzed to identify reading versus reflection periods. According to this timing information, the sampled physiological measures matching respectively each category were put together. Applying One‐Way ANOVA also exhumed significant differences (p<.0005) for the 4 physiological signals but with a slightly different pattern: in the periods of structured reflection (use of the reflection amplifiers), TEMP, PR and PVA are higher while BVP is lower (Table 2). Following the same process, the sampled measures corresponding to the periods of use of the different types of reflection amplifiers were contrasted against each other and with the reading activity (Table 3). Applying One‐Way ANOVA exhumed significant differences for three signals except BVP (p < .002).

3.2 Time on task Total time on task (Figure. 6) was descriptively higher in the group prompted to reflect (M = 52 min, SD = 9 min) than in the group without prompting (M = 26 min, SD = 12 min), and the difference was significant, t (38) = 7.46, p < .0001, d = 2.45.

Figure 6: Average time (in minutes) per chapter for the control and the treatment group

3.3 Performance Scores for the multiple‐choice questions revealed no significant differences between the control group (X = 4.5, SD = 2.24) and the treatment group (X = 4.7, SD = 1.59), t (38) = .41, p = .67, d = .08. A 3‐level scoring rubric was used to control the quality of the answers to the open questions:

trivial explanation of the picture;

explanation invocating the correct Darwinian concept;

explanation contextualizing the correct Darwinian concept in the overarching evolution theory.

The treatment group (X = 4.5, SD = 1.6) did not perform differently from the control group either (X = 3.7, SD = 1.7), t (38) = 1.54, p = .13, d = .05.

3.4 Feedback from learners 73% of subjects in the treatment group claimed that their learning experience in the course differed from their daily experience against 61% of subjects in the control group (relative percentages). Results showed that each of the 3 reflective breaks was foreign to about half of the sample population (Figure 7). 16% of the respondents answered almost never for all 3 reflective techniques.

500

Wen Qi, Dominique Verpoorten and Wim Westera

Figure 7: Familiarity level with the reflective breaks prior to the experiment

3.5 Perceived effect on time and mindfulness The reflective breaks were rated by the students on a 3‐point Likert‐type scale for their contribution to their study result and study time (1 = decreased the quality of my study/my study time, 2 = did not affect the quality of my study/my study time, 3 = increased the quality of my study time). Figure 8 illustrates the results.

Figure 8: Perceived contributions of the reflective breaks to study quality and time

4. Discussions In this study, we observed that physiological measurements differ between the conditions both at the global level of the course and when reading/reflection periods are contrasted. The observed variations indeed bring extra information to the study of reflection in formal learning. So far, reflective activity attached to this context has usually been inferred from performance changes or claims of students (scales, open questions) or think‐ aloud protocols [Veenman, Van Hout‐Wolters & Afflerbach 2006]. What we have learned here is that biofeedback measurements can be another dimension to the study of the phenomenon of reflection during learning. The findings of this study suggest that the cognitive states associated to different learning activities may be detected and recognized from physiological parameters. For instance, the externally‐imposed reflection remits seems to trigger internal responses traceable in physiological data. However, interpretation of these findings is not straightforward. It is not clear whether reflection is assimilated to some forms of

501

Wen Qi, Dominique Verpoorten and Wim Westera mediations, which can slow down certain body activities. Or on the contrary, reflection reverberates as a form of stress [McCraty, et al. 1999] because of its compulsory (this is an assignment) or/and perhaps unfamiliar character, which has effects on some physiological signals. Answering these questions goes beyond the scope of this study. It would require further interdisciplinary discussions combining both pedagogical and psycho‐physiological expertise. But, the findings related to the physiological measures in this paper should nevertheless be taken with prudence for the following reasons:

the results bear on a limited number of subjects with physiological measurements,

the huge amount of sampled observations can partly cause the significance.

It calls for further investigation into the relationships between reflection and learning performance through physiological measures that would be carried out with larger samples and with contrasted audiences of low and high performers in order to confront the way they study and practice reflection to their respective physiological coherence.

5. Conclusions Reflective break means to induce regular mental tingling for evaluating ones own learning, nurturing internal feedback [Butler & Winne 1995] and maintaining active commitment to the tasks at hand. This study has explored the potential benefits of reflective breaks for learning activities. The study also investigates whether physiological signals can be used as appropriate indicators to detect the actual changes of cognitive states while introducing reflection breaks during learning. The pattern of findings suggests that the benefit of a one‐ hour hand‐on session embedded with the reflective strategies is not significant in terms of enhancing cognitive performance. However, these reflective practices did bring an increased awareness and intensified their presence to the learning process. This study has demonstrated a physiological method that can be used to study the effects of reflection on learning. The study points out the future challenges faced by researchers while studying reflective break with physiological signals.

Reference Brown‐Frossard, R. (2012). Memorization Isnt A Lot Of Hocus Pocus. The International TEYL (Teaching English to Young Learners) Journal Retrieved June 1, 2012, from http://www.teyl.org/article14.html Boud, D., Keogh, R., & Walker, D. (1985). Reflection,Turning Experience into Learning. London: Kogan Page. pp. 100‐116. Butler, D. L., & Winne, P. H. (1995). Feedback and self‐regulated learning: A theoretical synthesis. Review of Educational Research Review, Vol. 65, No. 3, pp. 245‐281. Claxton, G. (2006). Expanding the capacity to learn: a new end for education? keynote speech, British Educational Research Association Annual Conference, University of Warwick, 6‐9 September 2005. Csap, B. (1999) Improving thinking through the content of teaching. In H. Hamers,J. van Luit & B. Csap (Eds.), Teaching and learning thinking skills (pp. 37‐62). Lisse: Swets and Zeitlinger. De la Garanderie, A. (1989). Defense et illustration de l’introspection [Defense and illustration of introspection]: Le Centurion. Dewey, J. (1966). Democracy and Education: an introduction to the philosophy of education. Free Press, New York. Di Vesta, F. J., & Smith, D. A. (1979). The pausing principle: Increasing the efficiency of memory for ongoing events. Contemporary Educational Psychology, 4(3), pp.288‐296. Ekman, P., Levenson, R.W. & Friesen, W.V. (1983). Autonomic Nervous System Activity Distinguishes among Emotions. Science, 221(4616): p. 1208‐1210. Eshuis, J. H., & Goltstein, G. P. H. (2007). Seks en de evolutie. Retrieved 18‐02, 2011, from http://www.ou.nl/eCache/DEF/2/22/850.html Flesch, R. (1948). A new readability yardstick. The Journal of applied psychology, 32(3), pp.221‐233. Jaschsik. (2011). Can students learn to learn? Inside Higher ED Retrieved June 1, 2012, from http://www.insidehighered.com/news/2011/01/31/ colleges try to use metacognition to improve student learning Joseph, N. (2003). Metacognitive awareness: investigating theory and practice, Academic Exchange Quarterly Retrieved June 1, 2012, from http://findarticles.com/p/articles/mi hb3325/is 4 7/ai n29059143/ Le Cornu, A. (2009). Meaning, Internalization, and Externalization. Adult Education Quarterly, 59(4), pp.279‐297. Logtenberg, A., van Boxtel, C.,& van Hout‐Wolters, B. (2011). Stimulating Situational Interest and Student Questioning through Three Types of Historical Introductory Texts. European Journal of Psychology of Education, 26(2), pp.179‐198. Marbach‐Ad, G., & Sokolove, P. G. (2000). Can undergraduate biology students learn to ask higher level questions? Journal of Research in Science Teaching, 37(8), pp.854‐870. Moon, J. (1999). Reflection in Learning and Professional Development. London: Kogan Page. McAlpine, M. (2002). A summary of methods of Item analysis . Bluepaper Number 2, University of Glasgow. ISBN 1‐904020‐ 02‐X.

502

Wen Qi, Dominique Verpoorten and Wim Westera McCraty, R., Atkinson, M., Tomasino, D., Goelitz, J., & Mayrovitz, H. (1999). The impact of an emotional self‐management skills course on psychosocial functioning and autonomic recovery to stress in middle school children. Integrative Physiological and Behavioral Science, 34(4), pp.246‐268. Pedrosa de Jesus, H.,& Moreira, A. C. (2009). The role of students questions in aligning teaching, learning and assessment: a case study from undergraduate sciences. Assessment & Evaluation in Higher Education, 34(2), pp.193‐208. Romainville, M. (2007). Conscience, metacognition, apprentissage : le cas des competences methodologiques [Consciousness, meta‐cognition, learning: the case of the methodological skills]. In F. Pons & P.‐A. Doudin (Eds.), La conscience chez l’enfant et chez l’lve [Consciousness in the child and in the student] (pp. 108‐130). Qubec: Presses de l’Universit du Qubec. Rychen, D. S., & Salganik, L. H. (2003). Key Competencies for a Successful Life and a Well‐Functioning Society: Hogrefe Publishing Ruhl, K. L., Hughes, C. A., & Schloss, P. J. (1987). Using the Pause Procedure to Enhance Lecture Recall. Teacher Education and Special Education: The Journal of the Teacher Education Division of the Council for Exceptional Children, 10(1), pp.14‐18. Simpson, W. (2004). The pause method in undergraduate auditing: An analysis of student assessments and relative effectiveness. In J. Ketz (Ed.), Advances in Accounting Education Teaching and Curriculum Innovations (Vol. 6, pp. 69‐ 85): Emerald Group Publishing Limited. Seel, N. M. (2001). Epistemology, situated cognition, and mental models: Like a bridge over troubled water. Instructional Science, 29(4), pp.403‐427. Taras, M. (2002). Using Assessment for Learning and Learning from Assessment. Assessment & Evaluation in Higher Education, 27(6), pp.501‐510. Vicente, K.J., Thornton, D.C. & Moray, N. (1987). Spectral Analysis of Sinus Arrhythmia: A Measure of Mental Effort. Human Factors, 29(2): pp.171‐182. Verpoorten, D., Westera, W., & Specht, M. (2011). Reflection Amplifiers in Online Courses: A Classification Framework. Journal of Interactive Learning Research, 22(2), pp.167‐190. Veenman, M., Van Hout‐Wolters, B., & Aﬄerbach, P. (2006). Metacognition and learning: conceptual and methodological considerations. Metacognition and Learning, 1(1), pp.3‐14. Vermersch, P. (2009). Describing the Practice of Introspection. Journal of Consciousness Studies, 16(10‐12), pp.20‐57. Wilson, G.M. (2001). Psychophysiological Indicators of the Impact of Media Quality on Users. In Proceedings of CHI 2001 Doctoral Consortium. Seattle, WA, USA. ACM Press. pp.95‐96.

503

Evaluation of Adaptive Serious Games using Playtraces and Aggregated Play Data Christian Reuter, Florian Mehm, Stefan Göbel and Ralf Steinmetz TU Darmstadt, Darmstadt, Germany christian.reuter@kom.tu‐darmstadt.de florian.mehm@kom.tu‐darmstadt.de stefan.goebel@kom.tu‐darmstadt.de ralf.steinmetz@kom.tu‐darmstadt.de Abstract: Adaptive Serious Games often feature complex algorithms and models, which influence the player’s progression through the game. These models include properties like pre‐existing knowledge or preferred playstyle and are matched with a pool of appropriately annotated parts of the game, such as assignments or scenes, during runtime. While being transparent for players, these models must be visualized for testing and evaluation purposes. In order to allow authors the retrospective interpretation of playtraces generated by a gaming session, we developed a replay component for adaptive serious games created with the authoring tool “StoryTec”. This method removes the need for continuous observation of individual players while retaining the same level of detail and being much more understandable compared to log files, especially for the non‐programming audience addressed by StoryTec. In addition to showing the player’s view, the state of the internal models and the progression through the story structure are also visualized. Sharing the same models and data structures as the authoring tool and making their runtime behaviour visible to the author, the replay component is therefore able to offer additional benefits compared to more generic methods like screen capturing or key recording tools. A complementary tool which is able to aggregate a large number of playtraces into one comprehensive spreadsheet for statistical analysis was also implemented. This allows authors to gain an overview over a great number of players in a shorter time compared to investigating them individually. In order to reduce the complexity of the result, the table contains aggregated information like the total time the players spent in each scene or the final value of variables at the end of their sessions. If authors detect an anomaly, they can then access more detailed information by loading the original traces into the replay component, which uses the same data format. Together these two components support the evaluation of adaptive serious games by means of user studies with the intended target audience, for example pupils. By combining them with our testbed for rapid prototyping named “StoryPlay”, we were able to provide a set of tools covering a broad range of evaluation tasks based on the same underlying models and data formats. Using these tools, it is possible to gain insights on how the adaption algorithms behave over a large number of players, e.g. which paths were taken by how many players or whether the time to solve a task as estimated by the author was matched. Keywords: Serious Games, evaluation, adaptation, playtrace, testbed

1. Introduction While the idea of Serious Games, i.e. games which serve an additional purpose besides entertainment like education or health, becomes more and more popular, there are still critical voices who question the effects of these kinds of games. Therefore it is necessary to provide hard evidence in this regard in order to fully establish this idea, especially when those games are meant to provide game‐based learning and training. This is usually done by means of evaluation studies, where the game is played by a large number of players and their results and / or gameplay experiences are recorded. Besides assessing the outcome of the games, user studies also allow the game’s creators to identify technical as well as design problems and measure the acceptance by the intended target audience. There are already many generic tools and methods available that support user studies of software in general as well as games and Serious Games in particular. Adaptive Serious Games are games that contain adaption mechanism that allow the game to alter itself depending on the player using it. This can happen by simply personalizing the presentation based on the user’s preferences or by taking his pre‐existing knowledge into account and restructuring the order and overall number of assignments / game scenes accordingly. When evaluating adaptive games, it is important to not only record the player’s view, but also the state of the underlying adaption models and algorithms, giving insight into how these behave over a wide range of users. An obvious benefit of this approach would be a case, where an adaptive game with multiple paths through its story is evaluated, showing that some paths are never chosen by the adaption engine. In order to assess whether this is an error and how to fix it, a close observation of the internal states is necessary.

504

Christian Reuter et al. The current state‐of‐the‐art regarding evaluations can be separated into two groups of methods. Generic methods and tools like observation, screen capturing or questionnaires only record the “outsiders view” and therefore work with arbitrary games. In contrast, specific approaches must be tailored towards a concrete game, but allow the recording of internal information. To fully evaluate adaptive games, it is therefore necessary to employ specific methods, which are closely linked to their internal models. A drawback of specific methods however is that they offer limited reusability and often must be adjusted for each game they are applied to, creating a trade‐off between the amounts of information an evaluation tool is able to provide and the costs associated with its use. In order to address this problem we therefore developed two tools based on the adaption models of our authoring tool “StoryTec” (Göbel et al. 2010). Since the games created with it are based on the same adaptation models, it is possible to evaluate all of them using the same set of tools. The tools also aim at reducing the effort for evaluating complex games with many players and are intended to be easily used by single non‐programming authors. In this paper we first discuss the current state‐of‐the‐art in regards to evaluation methods and tools for adaptive Serious Games. We then give a short overview over the authoring tool “StoryTec” and the associated “StoryPlay” Testbed, which build the foundation for this work. After that we describe our evaluation framework, consisting of a replay component and an aggregation tool for playtraces. We also show how our tool is used in the ongoing evaluation of a learning game for mathematics, focussing on explorative results like the overall playtime or anomalies in player behaviour.

2. Related work A good overview of software evaluation in general can be found in (Hilbert & Redmiles 2000). They argue that user interfaces automatically generate events that can be recorded and describe different types of evaluation goals and event data. Based on this the authors compared tools and techniques, which are able to analyse these recordings and can extract higher level information from them. Examples include the synchronization with other data sources such as video recordings, transformation (e.g. filtering), summarization (to decrease the amount of data) and the detection / comparison of subsequences of events. Another overview aimed specifically at games was done by (Nacke et al. 2009), differentiating between playability, which focusses on the game itself, and player experience, which concerns the player’s interaction with the game. They argue that good playability is necessary to conduct studies on player experience. The work then lists a number of methods for both types of evaluations. While playability is measured against heuristics, player experience can be assessed by a number of different objective (like biofeedback, gameplay data) and subjective methods (questionnaires). It is concluded that a combination of several methods yields more information and is therefore advisable. (Nacke et al. 2010) follows up on the topic of player experience in the regards to Serious Games. They also list several evaluation methods, grouped in individual (e.g. psychophysiological, gameplay data, player modelling and questionnaires) and context oriented (playability heuristics, also questionnaires) ones that take the environment in which the game is played into account. (Bruder et al. 2004) describes a certificate for computer‐based learning environments, which also takes the learning effects and outcomes into account – a criterion for Serious Games in particular. Individual approaches that work with games in general include work by (Ketkar & Youngblood 2010), where the movement of players in a 3D world was recorded. After that graph based algorithms were used in order to build player profiles. (Liu et al. 2011) used heatmaps to visualize players’ progress in games with discrete states, needing only the states and transitions between them as an input information. They also proposed the mapping of continuous spaces to state features, making their approach viable for games without discrete states while also defining higher level information. This mapping however must be done manually and the results of the visualization are highly dependent on the mapping. (Kim et al. 2008) developed a testing framework which can be used to interpret user triggered events, which are logged by a game. It is able to visualize anomalies on different levels of granularity. When linked to a video recording, they were able to find the reason behind sudden difficulty spikes and changed their game accordingly. They noted that it is important to decide which events to record, that the sequence of events is very important and that only the number of occurrences is often insufficient.

505

Christian Reuter et al. One example where data analysis was conducted for a commercial game can be found in (Medler et al. 2011). They used a graph based visualization of events to balance two heterogeneous teams in a multiplayer setting. Their server‐based system was directly integrated into the development process. Practical insights for the development of such tools were noted and clustered in the categories production (for example to build the tool parallel to the game), functionality (enabling faster analysis by aggregation) and game team integration (involving the game team in the tool development process). Other work by (Hullett et al. 2012) gathered game data from real players spanning 3 years after release of a game via the internet, requiring no additional collection setup. Their analysis produced insights for future games, such as popular or unused game modes and elements – to pick their focus for future releases to decrease development costs. Another visualization tool for game data is “StoryPlay”, formerly “Bat Cave” (Mehm et al. 2010). This tool is tailored for adaptive Games created with the authoring tool “StoryTec” and that visualizes the internal game state. Since the tool is closely linked to this work, it will be discussed in the next chapter.

3. Background The evaluation framework we describe in this paper is based on the internal models of the StoryTec‐authoring environment on the one hand and on the visualizations of the StoryPlay‐testbed on the other hand.

3.1 Adaptive games in StoryTec Games – or stories – build in StoryTec offer adaptation along three dimensions: the learner, player and story model (Göbel et al. 2010). The learner model takes the pre‐existing knowledge and dependencies between individual skills into account. Its main task is to make sure that the players do learn all skills taught by the game while guaranteeing that the learner has the right perquisites to understand this knowledge. To represent the dependencies between individual skills, the learning context is modelled as a graph based on the “Competence based Knowledge Space” (Korossy 1999). The player model is based on (Bartle 1996) and maps the players behaviour to four playstyles, “killer”, “achiever”, “socializer” and “explorer”. This allows different representations of same content, fitting the players preferred style of play. Each of these styles is modelled by a number in the interval [0, 1], based on how well the style fits the player. Lastly the story model uses a modified version of the Hero’s Journey (Göbel et al. 2009) to balance between adaptation and an interesting story as defined by the author. The fixed order of event types (like “call to adventure” or “return”) from the Hero’s Journey is preserved while allowing the events themselves to play out differently. During game creation, the author is able to annotate every scene along these three dimensions, for example describing which learning content they provide and thereby making them “Narrative Game‐based Learning Objects”. When the game is played, these models are constantly updated based on the players progress and behaviour by the Story Engine. Whenever a free transition to another scene is triggered, the engine calculates a score for every potential successor based on their annotation and the current state of the models as well as individual weights for each dimension. The scene which fits the player best is then selected.

3.2 Testbed StoryPlay When prototyping adaptive games created with StoryTec, it was necessary to visualize the state of these internal models. This made it possible to understand decisions of the Story Engine and, if the results seemed undesirable, change the game or the adaption algorithms accordingly. Since the models may seem very complex, especially for single untrained authors, we developed a rapid prototyping environment which is able to visualize the state of the models and the decisions of the Story Engine in an understandable way (Mehm et al. 2010). The testbed works directly on the story files created by StoryTec and is linked to the same models, reducing effort for rapid prototyping and eliminating the risk of mismatches or loss of information between the two tools.

506

Christian Reuter et al.

4. Evaluation framework Most evaluation methods described in the related work require internal play data from the game, which should include the state of the internal models in adaptive Serious Games. Since this data is already visualized by the testbed, our first approach used StoryPlay for small user studies. When working with practitioners it became clear however, that a rapid prototyping session by an author has different requirements than a general evaluation tool:

Authors usually work alone or in small groups, while evaluations ideally feature large groups of players.

Authors need lots of information regarding the internal models, while players participating in an evaluation get distracted by them.

Authors want to see the state of the models in real‐time, but during evaluations the models will be analysed retrospectively.

For rapid prototyping the amount of information is limited to that generated by one gameplay session at a time, while in evaluations an overview over a number of sessions is needed.

For rapid prototyping information has to be presented during play, while in an evaluation setting the information can be transformed before it is presented.

In order to address these requirements, we developed two additional tools that complement the authoring tool and testbed. The first one is a replay component, which was directly integrated into the testbed to give it the ability to review individual game sessions after they were played. This extension is accompanied by an aggregation tool, which combines the data from a large number of individual sessions into one spreadsheet for further analysis. Both new tools work with playtraces that are written to log files during a normal play session, where the internal models are hidden from the player. They use the same format / timestamp and are linked to the original story files as well as the models used by StoryTec, so there is no mismatch between these new components either. Therefore it is possible to analyse the aggregated view of the spreadsheet and then investigate anomalies in detail with the replay component without losing information. Based on the research questions asked in our previous evaluations of adaptive Serious Games, we decided that the playtraces had to contain at least the following information:

Change of the internal adaptation models along each dimension.

Time taken per scene (for example explanation, tasks, help‐screen).

If a scene was visited and how often.

User input (multiple choice, text, minigames and mouse movement).

We also decided to log every stimulus, which notes the execution of an event in the game. This will allow us to always reconstruct the whole playthrough in combination with the original story files. This way there will never be a loss of information, even if a feature was added without logging its information explicitly.

4.1 Replay component The replay component allows the testbed to replay game sessions recorded in playtraces retrospectively. It offers the same level of detail like a direct observation of the player, including mouse movements (Figure 1, left). In contrast to the original play session however, the state of the internal models is also visualized (Figure 1, right). The advantages compared to a direct observation are that the replays can be watched independently from the original game session and more than once. The internal game information and the evaluation setup is invisible for the player and does not require any additional effort besides the collection of the playtraces, which could even happen automatically while the game is played on a website. Compared to generic approaches like capturing a video feed, the playtraces can be much smaller by referencing simple events and having the replay component calculate the same result as the original game – which is only possible when tailoring the component to a specific game or game type.

507

Christian Reuter et al. Additional features of the tool include remote observation during live play and a check whether the story used during generation of a particular playtrace matches the version used during the original play, so no misleading results based on minor variations can be generated.

Figure 1: Screenshot of the replay component, showing the players view (including his mouse cursor symbolized by a white box) on the left and the internal state of some of the models (scene history, player model and parts of the knowledge space) on the right

4.2 Aggregation tool The aggregation tool in contrast is designed to provide an overview over a large group of players. Its task is to parse a number of playtraces created by a game and compile this data into one spreadsheet, which can be further analysed using common office tools. Calculating an average score for example does not require opening each playtrace separately, saving time and enabling large scale evaluations for single authors. By specialising the tool in reading our playtrace format, we were able to exploit the semantic of certain information, for example for aggregation. It is also able to distinguish between temporary and final values of the internal models and variables, although the event might technically be the same. The same differentiation can be done for answers a player has given. Important “higher level” questions that can be investigated using the tool include how the internal models evolve and if they converge over time. It can detect how often scenes were reached, giving the author feedback whether the adaptation produced different results for his players. It is also possible to detect unnecessary scenes, which are seldom reached, giving him a clue on where to focus when refining the game. Based on completion time, he can also estimate the difficulty of the game and gather information on the expected time for certain types of tasks or minigames, providing data for the creation of a game designed around a certain play time (e.g. during a school lesson). However certain information cannot be extracted automatically, because there are universal concepts which might be used differently in structurally similar games. One example showed us that an author used generic buttons to model multiple answers to task and to skip the task altogether. Since these buttons can be used in many different ways, we could not differentiate between solved and skipped tasks in a way that could be used over every game created with our authoring environment. When using a predefined template with a fixed semantic for such tasks however, the tool is even able to detect whether a task was solved correctly or not. In order to support very large games, we implemented a function which is able to filter certain events based on annotations. The aggregation tool also displays a warning, if multiple variations of the same story were used and splits the results accordingly to reflect the differences. We also added anonymous user IDs, which allowed the matching of play sessions to external data source like accompanying questionnaires. These IDs are

508

Christian Reuter et al. checked for uniqueness to prevent duplicate entries while complying with privacy regulations by being arbitrary identifiers.

5. Use Case: Evaluation of learning game for mathematics The first applications where our evaluation framework was used was a game called “Der Wechsel” (“The Switchover”), created for teaching about mathematic functions in school. The game combines the everyday use of math with elements of a crime story and optional minigames like puzzles or hidden object games. It was evaluated with multiple school classes of about 30 pupils each (age 14‐15, 55% male), where the pupils had roughly one hour to solve the game on their own. Afterwards the playtraces were collected and analysed using the aggregation tool. We will focus on explorative results in this paper, noting some interesting findings in regards to the overall gameplay, which we were able to quickly isolate by combining the aggregated play data with the replay tool. Since the game’s design did not use the adaptive features of the authoring framework, the adaptation models stayed constant over the course of the game. We could however verify that the evaluation workflow worked smoothly while including non‐expert authors and that even the analysis of non‐adaptive variables could produce helpful information, some of which will be discussed in the following paragraphs.

Figure 2: Graphs displaying the total playtime of each pupil (left) and the time it took them to solve a puzzle minigame (right) While the game was supposed to last one hour, analysis of the total playtime showed that most pupils needed about ten additional minutes to finish the game (Figure 2, left). However on outlier was obvious, who took less than 45 minutes. Further analysis with the replay tool then revealed that the pupil skipped all optional minigames and took lucky guesses at a lot of tasks instead of solving them, indicated by quick and often wrong answers when others paused to calculate their answers. Analysis of the time the players took to solve a puzzle minigame also yielded valuable feedback (Figure 2, right). Some players took up to twelve minutes to solve the minigame alone, although the game’s authors estimated that completing the puzzle would only take up to one minute (it only consisted of seven parts). Again the replay tool helped to answer this question by revealing that those players had supposedly solved the puzzle much earlier (all parts were in the right order), but were held back by a usability problem: The game would only acknowledge that the puzzle was solved when they aligned the combined parts with the upper left corner of the screen, but they centred them in the middle. In contrast, almost half of the players skipped the optional puzzle altogether in order to focus on the math questions, which is an interesting information when designing similar games in the future. Other important information is how well the pupils performed. The analysis tool allowed us to view the answers given for each task. We then grouped them into categories manually (Figure 3, left), for example revealing that the majority of players solved task 4.6 correctly. About a third made a mistake regarding the decimal digit (7200 or 7.2 instead of 720) and only one of ten pupils was completely wrong. This information then could be used by the teacher to discuss common mistakes in a subsequent lecture. In order to get a hint for the design of similar games the authors also check whether the pupils used a help function summarizing the learning content and giving the pupils hints. This was the case for almost every pupil; a majority even used the function more than once (Figure 3, right).

509

Christian Reuter et al.

Figure 3: How well a specific task was answered (left) and the number of times the help was used (right)

6. Conclusion The use case showed us that making internal game information visible to the authors in an easily understandable way is crucial for evaluating the intended effects (e.g. learning or training) and usability of serious games. If the data collection is tailored towards a specific game or technology, it is possible to automatically extract semantic information and reduce the overhead of the study at the same time. Offering aggregated data is great to get a quick overview, while providing a more detailed view for further analysis is crucial in the case of anomalies. When these different views are based on the same data, it is beneficial if they are provided by a suite of closely linked tools or even a single application. Using a model shared between the games created with a specific authoring tool, our evaluation environment is providing a middle ground between general purpose and highly specialized evaluation tools, balancing information detail and associated costs. While it is tailored to games created with the authoring tool, the general approach could be adapted for other models and environments as well. Future work will include the analysis of games that use the adaptive features more extensively, allowing us to evaluate the adaptation engine and algorithms themselves. We are also planning to offer the possibility to visualize aggregated data directly without using an external spreadsheet application. And since the evaluation framework is using the same models and data structures as our authoring tool, it would be interesting to couple them even further – for example by feeding the average completion time for each task back into the authoring tool. In a case where the players took more time than estimated, the author could then use this data to decide which tasks should be omitted in order to reach the expected completion time in a future version of the game.

Acknowledgements Parts of this work where funded and supported by the “Forum für interdisziplinäre Forschung” at Technische Universität Darmstadt in the project “Effekte mathematischer Lern‐ und Diagnoseumgebungen mit spielerischen Elementen” (May 2011 – September 2012).

References Bartle, R., 1996. Hearts, Clubs, Diamonds, Spades: Players Who Suit MUDS. Journal of Virtual Environments, 1(1), p.19. Available at: http://www.mud.co.uk/richard/hcds.htm. Bruder, R. et al., 2004. Third party certification of computer‐based learning environments. In EISTA: Int. Conf. on Education and Information Systems: Technologies and Applications. Orlando, Florida. Göbel, S. et al., 2009. 80Days: Adaptive Digital Storytelling for Digital Educational Games. In Y. Cao et al., eds. Proceedings of the 2nd International Workshop on Story‐Telling and Educational Games (STEG’09). CEUR Workshop Proceedings. Göbel, S. et al., 2010. Personalized, Adaptive Digital Educational Games using Narrative, Game‐based Learning Objects. In Proceedings of Edutainment 2010. pp. 438–445. Hilbert, D.M. & Redmiles, D.F., 2000. Extracting usability information from user interface events. ACM Comput. Surv., 32(4), pp.384–421. Available at: http://doi.acm.org/10.1145/371578.371593. Hullett, K. et al., 2012. Empirical analysis of user data in game software development. Proceedings of the ACM‐IEEE international symposium on Empirical software engineering and measurement ‐ ESEM ’12, p.89. Available at: http://dl.acm.org/citation.cfm?doid=2372251.2372265. Ketkar, N.S. & Youngblood, G.M., 2010. Graph‐Based Data Mining for Player Trace Analysis in MMORPGs. In Game Programming Gems 8 (AI Section). Charles River Media, pp. 335–352.

510

Christian Reuter et al. Kim, J.H. et al., 2008. Tracking real‐time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY, USA: ACM, pp. 443–452. Available at: http://doi.acm.org/10.1145/1357054.1357126. Korossy, K., 1999. Modeling Knowledge as Competence and Performance. In D. Albert & J. Lukas, eds. Knowledge Spaces: Theories, Empirical Research, and Applications. Psychology Press, pp. 103–123. Liu, Y.‐E. et al., 2011. Feature‐Based Projections for Effective Playtrace Analysis. In Proceedings of the 6th International Conference on Foundations of Digital Games. New York, NY, USA: ACM, pp. 69–76. Available at: http://doi.acm.org/10.1145/2159365.2159375. Medler, B., John, M. & Lane, J., 2011. Data Cracker : Developing a Visual Game Analytic Tool for Analyzing Online Gameplay. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY, USA: ACM, pp. 2365–2374. Available at: http://doi.acm.org/10.1145/1978942.1979288. Mehm, F. et al., 2010. Bat Cave: A Testing and Evaluation Platform for Digital Educational Games. In Proceedings of the 4th European Conference on Games Based Learning. pp. 251–260. Nacke, L.E. et al., 2009. Playability and Player Experience Research. In Proceedings of DiGRA 2009: Breaking New Ground: Innovation in Games, Play, Practice and Theory. DiGRA. Available at: http://www.digra.org/dl/display_html?chid=http://www.digra.org/dl/db/09287.44170.pdf. Nacke, L.E., Drachen, A. & Göbel, S., 2010. Methods for Evaluating Gameplay Experience in a Serious Gaming Context. International Journal of Computer Science in Sport, 9, p.2010.

511

Learning Effectiveness of Management Simulation Game Manahra Petr Smutný1, Jakub Procházka1, 2 and Martin Vaculík2 1 Faculty of Economics and Administration, Masaryk University, Brno, Czech Republic 2 Faculty of Social Studies, Masaryk University, Brno, Czech Republic psmutny@econ.muni.cz jak.prochazka@mail.muni.cz vaculik@fss.muni.cz Abstract: The subject of the study was to examine the learning effectiveness of a management simulation game called Manahra. The observed management game was a long‐term competition among several 20‐member teams. The team members acted in the roles of fictitious business managers and performed complex management tasks associated with the production and sales of passenger cars. The economic competition among the individual businesses took place in a simulated commercial market and the individual players were remunerated for their work with fictitious money. The amount of accumulated money was the criterion of their evaluation at the end of the game. The study consists of two relatively autonomous parts that worked together to answer the key question: does completion of management game lead to the development of managerial skills? The first part of the study presents the results of an original, yet unpublished research work. The research was conducted through a questionnaire survey. In the questionnaire, participants had to assess the level of their managerial skills. Individual managerial skills were included into the questionnaire based on an a priori created model of managerial competence. Self‐assessment was done using rating scales for each managerial competence separately. Players filled out questionnaires at the beginning and at the end of the game. The questionnaire distributed at the end of the game was extended by including a second scale in which players rated the magnitude of the change regarding their managerial competencies. The second part of the study presents the follow‐up research focused on obtaining qualitative responses from the players. They were again asked to fill out a questionnaire designed to obtain spontaneous reports about the development of their managerial competencies due to their participation in the game. This assessment was then compared with the initially established competency model and to the results of the original research. Keywords: gaming simulation, managerial skills, training, management, game, educational effectiveness

1. Introduction Numerous research studies (Bigelow 1991, Clark 2003) show that managerial skills can be acquired and developed, and that good results can be achieved by various training activities outside of the work environment (off‐site training). Large number of training activities is based on the theory of experiential learning (Paglis 2012). Management simulation games are primarily used to develop skills, therefore learning effectiveness is the main measure of their quality. Previous research, however, show rather ambiguous results. Some authors have documented the benefits of using this type of teaching method as being: practice in a quasi‐realistic environments with limited risks, increased creativity, faster decision making, better targeted analysis of competition (competitive analysis) and a better understanding of the relationships among business functions (cross‐functional understanding) (Chapman & Sorge 1999; Reibstein & Chussil 1997). A number of studies concluded that evidence is insufficient to show that the game experience can be potential learning (Faria & Wellington 2004, Wolfe & Jackson 1989). Factors that influence the effectiveness of simulation games can be divided into two main groups: a) the quality of the application (simulation design); and b) the quality of the game play (quality of implementation). The main feature of simulation games is the participation of real people in the role play. In this regard, Peters (1998) talks about the psychological validity as an important factor influencing effectiveness. The objective is to create a game environment which seems realistic to the participants. Furthermore, the additional goals are meant to affirm a high degree of similarity through objective parameters, and also to ensure a high degree of similarity of the subjective perception of reality among the players (Norris 1986). When creating a credible simulation game, it is therefore necessary to consider the experience and competencies of the participants.

512

Petr Smutný, Jakub Procházka and Martin Vaculík

2. Management simulation game Manahra A group of about 20 students make up the management of a fictitious car manufacturer, aiming to maximize the company’s accumulated profit during seven gaming rounds. Students learn the rules of the game in introductory seminars and, in a selection procedure, choose the managing director. The managing director appoints top management, divides the company into departments, and assigns management members to them. During the game they can release any member of their team or headhunt employees from another company and recruit them. For their work players receive remuneration in form of fictitious money and the sum they receive determines their final mark at the semester’s end. A successful company on the market can generate larger funds and its employees have a better chance of getting a better mark. The market computer simulation model takes the decisions of the carmaker’s management into account and then determines the overall demand for cars and the market share of individual car makers accordingly. At the beginning of the game the position of all car making companies is identical. During the game, students have a number of options for enhancing the performance of their company. They decide how many cars to produce in each round, set production costs, invest in research, add accessories to the standard equipment of a car to meet target customer group’s specifications, run advertising campaigns, and negotiate credit lines with banks. In every round, each car maker must complete financial statements, analyze the results of other companies, and pay out salaries. The entire workload is insurmountable for one person, or even for a small group. A successful company will involve all, or virtually all of the students in its tasks and functions. The managing director is the company’s main manager.

3. Study 1 3.1 Methodology and research sample The methodology of Study 1 is based on Kirkpatrick's model (Kirkpatrick, 1959), the presented results are consistent with findings on second level ‐ learning. The observed management simulation game allows the development of all components of managerial competencies ‐ knowledge (declarative, procedural or knowledge of type "know why") as well as skills. This has consequences also for the research methodology. Whereas knowledge can simply be measured using quizzes, such as tests, these are inappropriate for skills assessment (Boud 1992). Skills can be measured through case studies, evaluation of others (e.g., subordinates, colleagues etc.) or self‐assessment. Agut et al. (2003) reported the results that have confirmed that the method of self‐assessment of management competencies is, in terms of the quality of the data, equivalent to evaluation by subordinates. Also Hansson (2001), concludes that self‐assessment is sufficiently accurate and can be used to measure competencies. Therefore, we have also used the self‐assessment to evaluate skill levels. To collect data, a questionnaire was developed in which participants of the game evaluated their own level of managerial skills. Self‐assessments were carried out using rating scales individually for each managerial competence. The participants answered the questions concerning their own level of managerial skills listed. We used managerial skills that are part of a verified model of managerial competencies (Smutný 2007). Participants marked responses on rating scales, each varying from 0 to 100 points. Players completed the questionnaires before and after the game. The questionnaire distributed at the end of the game was extended by a second scale in which players assessed the magnitude of change in their managerial competencies. This enabled an extension of the classic two‐level "before and after" testing methodology to three‐level methodology. Sadri and Snyder (1995), referring to the series of experiments, state that such a procedure reliably leads to revealing progress due to changes in perception and evaluation of the factors. The participants were full‐time undergraduate students of Faculty of Economics and Administration at Masaryk University in Brno, Czech Republic. It was a relatively homogeneous group of students of the first three years of full‐time study. All participants completed the management simulation game as a compulsory part of their coursework. Only the data of those who filled out all questionnaires were considered for the data analysis. In total, data were obtained from the 195 participants, i.e., 60.19% of all participants. With the exception of bankers, all playing roles were represented, while owners were represented significantly less proportionately (in 41.67% of all owners in Manahra). Other roles otherwise were adequately represented.

513

Petr Smutný, Jakub Procházka and Martin Vaculík

3.2 Results Hypothesis 1: Completion of Manahra leads to positive changes in managerial skills evaluated through self‐reports. The data for verification of the hypothesis was obtained from a questionnaire filled out by players after completing the game. The questions concerned: What changes in managerial skills have you experienced as a result of completing Manahra? Table 1: Changes in managerial skills

Test Value = 0

t 18.49

Sig. (2‐ tailed) 0.00

Communication skills

N 195

Mean 8.73

Std. Dev. 6.59

Cooperativeness

195

11.73

8.50

19.27

Motivational skills

195

4.35

6.44

Cognitive skills

195

7.18

Organizational Skills

195

Flexibility / Adaptability

195

95% Confidence Interval

Lower 7.80

Upper 9.66

0.00

10.53

12.93

9.43

0.00

3.44

5.26

7.48

13.42

0.00

6.13

8.24

7.82

8.66

12.62

0.00

6.60

9.05

9.43

7.75

16.98

0.00

8.33

10.52

The results (see Table 1) show a slight improvement in all observed competencies, mainly between 5 and 10 points. The greatest improvement is reflected in cooperation, flexibility and communication skills. The fact that the measured progress is not zero is also confirmed by the results of the statistical analysis with the one‐ sample t‐test. The value of the test criterion for all competencies lies within critical values, always at a confidence interval of α <0.05. We therefore confirm the hypothesis that completion of Manahra leads to improvement in all monitored managerial skills. These results were validated through analysis, in which we observed differences in the evaluation of initial and final level of managerial skills of participants. Data for verification of the analysis were obtained from questionnaires filled out by the players before and after completing the game and answered the question: How do you rate your current level of managerial skills? Table 2: Changes in managerial skills – alternative analysis Paired Differences

Communication skills

initial 67.51 195

Std. Dev. Mean 12.71 6.51 16.41

Cooperativeness

final 78.79 195

12.06

initial 77.74 195

15.91

final 65.38 195

15.76

initial 58.80 195

18.87

final 71.21 195

14.48

initial 66.90 195

15.79

final 70.14 195

16.35

initial 64.69 195

19.74

final 76.16 195

13.52

initial 72.31 195

16.22

Motivational skills Cognitive skills Organizational Skills Flexibility / Adaptability

Mean N final 74.02 195

95% Confidence Interval

Std. Dev.

Lower

Upper

Sig. (2‐ tailed)

14.73

4.43

8.59

6.17

0.00

1.05

14.87

‐1.05

3.15

0.99

0.32

6.58

18.93

3.91

9.25

4.85

0.00

4.31

14.18

2.31

6.32

4.25

0.00

5.45

18.35

2.85

8.04

4.14

0.00

3.86

15.08

1.73

5.99

3.57

0.00

514

Petr Smutný, Jakub Procházka and Martin Vaculík The results (see Table 2) comparing the initial and the final evaluation of managerial skills show that the final level of all investigated managerial skills reported via self‐reports is higher than those at the initial stage. The statistical significance of differences was also assessed at significance levels for individual values of t‐test criterion. Differences can be declared significant for almost all studied competencies. Only in terms of cooperativeness, the significance level was α> 0.05 . According to the results, it can be concluded that completion of Manahra leads to positive changes in managerial skills. These changes are for the sake of convenience in the further text referred to as the effect of learning. Given the generally low level of practical experience in management of our participants, we examined whether, in addition to the effect of learning, completing the game has other effects as well. We tested the hypothesis that experience with managerial work gained in the game will lead to a more realistic assessment of participants' own managerial skills. Hypothesis 2: The reported final level of managerial skills does not exactly correspond to the initial level and magnitude of the change (i.e., there is an effect of self‐reflection). Data for the verification of this hypothesis was obtained from questionnaires filled out by the players before and after completion of Manahra. Of these, we obtained three kinds of reports: assessing the level of skills at the beginning, evaluating skills at the end of the game and reports concerning the perceived change in skills (assessed at the end of the game). With these values it was possible to calculate the quasi‐value of the initial skills and compare it with the value that players originally reported before the start of the game. Table 3: Correspondence of initial and ex‐post calculated quasi‐initial levels of managerial skills Paired Differences 95% Confidence Interval

Communication skills Cooperativeness

Std. Std. Mean N Dev. Mean Dev. initial 67.51 195 16.41 2.22 15.40 computed 65.29 195 13.56 initial

initial

Flexibility / Adaptability

Sig. (2‐ tailed)

0.04

4.39

2.01

0.05

10.68 16.05

8.42

12.95

9.30

0.00

‐2.23 19.00

‐4.91

0.46

‐1.64

0.10

2.87 14.62

0.81

4.94

2.74

0.01

2.38 19.12

‐0.32

5.08

1.74

0.08

5.57 14.41

3.54

7.61

5.40

0.00

66.90 195 15.79

computed 64.03 195 14.57

Organizational Skills

58.80 195 18.87

computed 61.03 195 15.70

Cognitive skills

Upper

77.74 195 15.91

computed 67.06 195 13.50

Motivational skills

Lower

initial

64.69 195 19.74

computed 62.32 195 15.68 initial

72.31 195 16.22

computed 66.74 195 13.87

Comparing the calculated values to the initial self‐reported level of managerial skills shows that there are differences in all observed managerial skills. These differences could be examined using the significance levels for each value of t‐test criterion. Communication and cognitive skills, cooperativeness and flexibility show the statistical significance of the differences. The remaining two show insignificant results. The results of the statistical analyses confirm the examined hypothesis. The final level of managerial skills does not correspond to the initial level and perceived magnitude of change. We call this difference the effect of self‐ reflection. Furthermore, it was demonstrated that the effect of self‐reflection is generally negative, thus there is a negatively directed re‐evaluation of the initial level of one’s own skills.

515

Petr Smutný, Jakub Procházka and Martin Vaculík The last step of the analysis was to examine the relationship of both effects. Examining the relationship of the two effects is logical, in particular with regard to the conclusions of Kolb‘s theory of learning. Hypothesis 3: The magnitude of the effect of self‐reflection varies differently among students who experience various degrees of the effect of learning. The participants were divided into two groups according to the magnitude of the effect of learning: group of students with above‐average value of the effect of learning (group 1) and a group of other students (group 0). The average value of the effect of self‐reflection in both groups of respondents was compared. Table 4: The relationship between differences and changes in managerial skills

Group Statistics

t‐test for Equality of Means 95% Confidence Interval

Communication skills

group N 0 75

Mean 2.57

Std. Dev. 16.05 15.97

120

‐2.34

124

‐15.50

18.62

‐18.83

20.47

6.47

20.72

2.45

18.64

105

‐2.05

16.47

‐7.53

18.17

Organizational Skills

105

2.93

18.28

‐4.53

18.74

Flexibility / Adaptability

‐67.01

17.53

119

‐61.39

16.76

Cooperativeness Motivational skills Cognitive skills

Sig. (2‐ tailed)

Lower

Upper

2.09

0.04

0.27

9.56

1.16

0.25

‐2.34

9.00

1.42

0.16

‐1.55

9.58

2.21

0.03

0.59

10.38

2.81

0.01

2.23

12.71

‐2.25

0.03

‐10.57

‐0.68

The results show that changes in self‐assessment (self‐reflection effect) due to the completion of Manahra are higher among respondents with higher levels of learning. By comparing the relevant values of the calculated significance level of p with the selected significance level α = 0.05, the null hypothesis can be rejected for all observed characters except for cooperativeness and motivational skills.

4. Study 2 4.1 Methodology and research sample In the first study, participants assessed their managerial skills and their development based on a checklist and description of individual skills. It is possible that while, under other circumstances, the participants did not think of their newly developed skills as being the result of Manahra; after seeing the checklist of skills, they might decide to indicate the development of some of them. Therefore, we conducted a second study, which identified whether players perceive the development of managerial skills even when asked an open‐ended and not close‐ended question about their skills development. New players were approached again through a questionnaire which had three parts. In the first part, participants answered questions concerning the degree of the development of soft and hard skills as a consequence of participating and completing Manahra. Participants answered on a five‐point scale ranging from "not at all“ to “very much." In the second part of the questionnaire, open‐ended questions were posed, the aim of which was to obtain spontaneous answers about the skills are being developed as a result of their participation in the game. The third part of the questionnaire included questions about the participants themselves ‐ we investigated their age, gender and degree and type of participation in the management game. The participants were full‐time undergraduate students at the University of Economics in Prague, Czech Republic. It was a relatively homogeneous group of students in their first three years of full‐time studies. The questionnaire was completed by a total of 126 players of Manahra, about 45% of all participants in the game.

516

Petr Smutný, Jakub Procházka and Martin Vaculík Slightly more men (53%) than women were represented in the sample. The average age of the participants was 21.44 years (SD = 2,08). Management positions (general manager, department managers, deputy manager) were held by 56 participants, 58 participants were regular employees of the company. The remaining participants had another position (banker, business owner) or non‐specific position. None of the participants in Study 2 was a participant in the first study.

4.2 Results According to the majority of the players, Manahra helps develop soft skills to an average degree. Only 5 (4%) of the players reported that Manahra did not help develop their skills at all. 28 players (22.2%) believe that Manahra developed their soft skills above average. At least one specific managerial skill was indicated by 86 players of Manahra as being developed (68.25%), while these players indicated an average of 1.47 (SD = 0.66) in terms of the development of different managerial skills. The most commonly reported skills as being developed were organizational skills, communication skills, cooperativeness and cognitive skills (Table 5). Table 5: Managerial skills and the number of players that reported them as developed

Sum

Rel.

Communication skills

126

22.22%

Cooperativeness

126

22.22%

Motivational skills

126

5.56%

Cognitive skills

126

18.25%

Organizational Skills

126

26.19%

Flexibility / Adaptability

126

1.59%

The players holding a managerial position indicated a higher level of development of soft skills than the players in non ‐ managerial positions (Table 6). We investigated whether this difference was demonstrated not only in terms of the perceived overall level of development of soft skills, but also in light of the idea that managers might be more likely to indicate particular skills as being developed. We focused on the four most frequently mentioned skills. The results of the analysis of variance shows that there is no difference between the position of manager and employee as to which skills were developed (Table 7). Therefore, we conclude that the player's role in the game affects the magnitude of the development of soft skills, but not other expected‐to‐ be‐developed skills. Table 6: The magnitude of development of soft skills for managers and non‐managers

Mean

Std. Deviation

Employee

2.64

Manager

Total

Confidence interval Lower

Upper

0.79

2.43

2.85

3.20

0.80

2.98

3.41

114 2.91

0.84

2.76

3.07

F (1, 112)

14.17

0.00

Table 7: Development of skills for managers and non‐managers Mean

Std. Deviation

Employee

0.17

Manager

0.30

Total

Communication skills

Cooperation Cognitive skills

Confidence interval LLCI

ULCI

0.38

0.07

0.27

0.46

0.18

0.43

0.24

0.43

0.16

0.32

Employee

0.28

0.45

0.16

0.39

Manager

0.14

0.35

0.05

0.24

Total

0.21

0.41

0.13

0.29

Employee

0.17

0.38

0.07

0.27

517

F (1, 112)

2.73

0.10

3.06

0.08

0.03

0.87

Petr Smutný, Jakub Procházka and Martin Vaculík Mean

Std. Deviation

Manager

0.16

Total Organizational skills

Confidence interval LLCI

ULCI

0.37

0.06

0.26

0.17

0.37

0.10

0.24

Employee

0.21

0.41

0.10

0.31

Manager

0.36

0.48

0.23

0.49

Total

0.28

0.45

0.20

0.36

F (1, 112)

3.22

0.08

Skill development was not affected by the department in which the employee in Manahra works. Players from different departments indicated a similar degree of development of soft skills and differences between various departments are not statistically significant (Table 8). Even if we focus on the four most frequently mentioned skills, one cannot say that some of them were developed to a greater extent in one department but not in another (Table 9) Table 8: The magnitude of development of soft skills based on the department in Manahra

Mean

Std. Deviation

Production

2.52

0.83

Confidence interval LLCI

ULCI

2.20

2.83

Marketing

3.04

0.90

2.68

3.39

Finances

3.03

0.80

2.75

3.31

Human Resources

2.85

0.67

2.54

3.16

110 2.86

0.83

2.71

3.02

Total

F (3, 106)

2.65

0.05

Table 9: The development of individual skills based on the department in Manahra N

Mean

Std. Deviation

Production

0.17

Marketing

Communication skills

Cooperation

Cognitive skills

Organizational skills

Confidence interval LLCI

ULCI

0.38

0.03

0.32

0.11

0.32

‐0.02

0.24

Finances

0.24

0.43

0.09

0.39

Human Resources

0.30

0.47

0.08

0.52

Total

110

0.20

0.40

0.12

0.28

Production

0.34

0.48

0.16

0.53

Marketing

0.22

0.42

0.05

0.39

Finances

0.26

0.45

0.11

0.42

Human Resources

0.10

0.31

‐0.04

0.24

Total

110

0.25

0.43

0.16

0.33

Production

0.10

0.31

‐0.01

0.22

Marketing

0.22

0.42

0.05

0.39

Finances

0.29

0.46

0.13

0.46

Human Resources

0.15

0.37

‐0.02

0.32

Total

110

0.20

0.40

0.12

0.28

Production

0.24

0.44

0.08

0.41

Marketing

0.19

0.40

0.03

0.34

Finances

0.32

0.47

0.16

0.49

Human Resources

0.25

0.44

0.04

0.46

Total

110

0.25

0.44

0.17

0.34

518

F (3, 106)

0.99

0.40

1.33

0.27

1.32

0.27

0.51

0.68

Petr Smutný, Jakub Procházka and Martin Vaculík

5. Discussion Methodology is important when reflecting the results. Various experiential learning courses and programs have been assessed in a variety of ways: objective learning tests, perceived learning measures, behavioral measures and others (Gosen and Washbush, 2004). The self‐assessment methodology used in our studies is compatible with measuring perceived learning. Our data represent respondents’ perceptions. The effectiveness of the game was not tested in any alternative way. Previous studies in the area of managerial skills (Agut, Grau and Peiro, 2001, 2003) have proven self‐assessment to be equivalent to evaluation by others in terms of reliability and validity of the data collected. The results of both studies show the development of the participants' managerial skills and they are complementary. In the case of organizational skills, communication skills, cooperativeness and cognitive skills, a fifth to a quarter of players spontaneously stated these skills as being developed. At the same time, by comparing the self‐assessment at the beginning and at the end of Manahra, the statistical significance of the development of these skills was confirmed. The results show an improvement of all the examined participants' skills. The results of the study support the conclusions of other authors on the applicability of simulation games to develop managerial skills. Those authors arrived to similar results while using self‐assessment (perceived learning measures) (Dedeke, 1999; James, 2000) and other research methods (Premi and Shannon, 2001). In particular, the results of Study 2 can be interpreted as a positive indicator of the structural quality of the analyzed management game. Although it is a complex management game (total‐enterprise game) in which players hold different positions, the same skills are being developed for the various positions. The players in leadership positions indicated a greater degree of development of their skills. This may not be the result of imbalances of Manahra, but rather that more proactive players, who are willing to invest more time and energy into Manahra, took those leadership positions. Manahra can therefore be described as a balanced simulation game. These results confirmed the findings of the previous studies of Smutný (2007). An important factor in the effective use of simulation games is the method of its implementation (Stainton, Johnson and Borodzicz, 2010). The simulation game used in our study is a part of mandatory class, as it is a part of seminars of ‘The Basics of Management‘ course. It is not linked to the lecture part of the course. Implementation of the game follows the standard division of the preparatory phase (lecturers present the basic aspects of the game, players are studying the rules), and the actual playing of the game (1 training round and 7 actual game rounds) and phase reflection (evaluation of play by the players and teachers). Faria (2001), based on extensive research, identified other important factors influencing the effectiveness of management games: students' exposure to management games (attitude of students), and the applicability of the players' game results to their own evaluation. The players participated in the management game as a part of university course, which was mandatory for most of them. This fact may have influenced their approach to the game and the motivation with which they come into the game. It can be assumed that some students would, if given a chance, have possibly preferred not to participate in the game. However, the players had strong external motives to participate in the game‐ credits gained for the course participation, the obligation to complete the game in order to get a higher education. One can also assume that some students would have, in the absence of external motivation, chose not to participate in the game. The effectiveness of simulation games could be even higher in individuals with prevailing internal motivation. The results that players achieve are dependent on their individual drive and on the success of the team as a whole. Players received a fictitious salary (which could have been increased through bonuses by the more successful teams), and the acquired money they could invest individually in a simulated stock market. The total value of the acquired assets is then the main criterion for evaluating players. The obtained result is then counted as the one third of their overall evaluation in ‘The Basics of Management‘ course. An important aspect of the beginning of the game is the fact that on the basis of information from previous studies and practice, the majority of our participants can be characterized as relatively inexperienced (Smutný, 2007). The game is, in terms of its content and difficulty of the tasks, adapted to the players. It is likely that the effect of learning would be lower if the players had more experience at the beginning and if they had had

519

Petr Smutný, Jakub Procházka and Martin Vaculík higher entry level managerial skills (see Washbush and Gosepund, 1994). The inexperience of the participants at the beginning of the game probably affected the existence and size of the effect of self‐reflection. The game creates an environment in which students recognize the complexity and nature of managerial work, and, after its completion, they are better able to evaluate the real level of their managerial skills. The reason may be that the completion of the game leads to increased cross‐functional knowledge (Scherpereel, 2013). This assumption was confirmed by the finding that players who achieved a higher level of learning, are more critical to the initial level of their skills after the game. We dubbed this phenomenon the effect of self‐reflection, and although it was observed statistically only in regards to some of the examined skills, it is important that it was demonstrated at those competencies that the participants spontaneously reported as being the most developed. This supports the hypothesis that magnitude of learning affects the degree of self‐reflection of the game participants.

Acknowledgements This article is part of a research „Effective leadership: Integrative approach“. The research has been funded by Czech Science Foundation (P403/12/0249).

References Agut, S., Grau, R., Peiro, J.M., 2001. Los sesgos de la autoevaluaci on de competencias. El caso de los gerentes de organizaciones turisticas (The biases of competences self‐assessment. The case of tourism industry managers). Revista de Psicologia Social Aplicada, Vol. 11, No. 1, pp 5–20. Agut, S. Grau, R., Peiró, J. M. (2003) ‘Competency needs among managers from Spanish hotels and restaurants and their training demands’. Hospitality Management, vol. 22, pp 281‐295. Bigelow, J. D., ed. (1991) Managerial Skills: Explorations in applied knowledge, Sage, Newbury park. Boud, D. (1992) The use of self‐assessment schedule in negotiated learning. Studies in Higher Education, Vol. 17, No. 2, pp 185 – 201. Chapman, K. J. and Sorge, C. L., (1999) ‘Can a simulation help achieve course objectives? An exploratory study investigating differences among instructional tools’, Journal of Education for Business, Vol. 74, No. 4, pp 225‐230. Clark, S. C., Callister, R. and Wallace, R. (2003) 'Undergraduate management skills courses and students' emotional intelligence', Journal of Management Education, Vol. 27, No. 1, pp 3‐23. Simulation & Experiential Exercises, Vol. 14, pp 43‐46. Dedeke, A. (1999) Design, integration and student evaluation of response papers in an introductory management course. Journal for Education of Business, Vol. 77, pp 211 – 214. Faria, A. J. (2001) ‘The Changing Nature of Business Simulation/Gaming Research: A Brief History.’ Simulation & Gaming. Vol. 32, No. 1, pp 97‐110. Faria, A. J. & Wellington, William (2004). ‘A Survey of Simulation Game Users, Former‐Users, and Never‐Users.’ Simulation & Gaming. Vol. 35, No. 2, pp 178‐207. Gossen, J.,Washbush, J. ‘A review of Scholarship on Assessing Experiential Learning Effectiveness. ’ Simulation & Gaming. Vol. 35, No. 2, pp 270‐293. Hansson, B. (2001) ‘Competency models: are self‐perceptions accurate enough?’ Journal of European Industrial Training, Vol. 25, No. 9, pp 428‐441. James, P. (2000) ‘The influence of a period of environment oriented work on students’ perception of their learning style. Environmental Education Research, Vol. 6, 157 – 165. Kirkpatrick, D. L. (1959). ‘Techniques for evaluating training programs.’ Journal of the American Society of Training Directors, 13, pp 3–9. Norris, D.R. (1986). ‘External validity of business games.’ Simulation & Gaming, 1Vol. 7, No. 4, pp 447‐459. Paglis, L. L. (2012) 'A review of managerial skills training in the classroom', Journal of Management Education. Vol. 36, No. Peters, V. (1998) ‘The Validity of Games.’ Simulation and Gaming, Vol. 29, No. 1, pp 20 – 30. Premi, J. and Shannon, S. I. (2001) Randomized controlled trial of an educational program for individualized learning. Journal of continuing Education in the Health Professions, Vol. 17, pp 245 – 249. Reibsein, D. J. and Chussil, M. J. (1997) Putting the Lessons before the Test: Using Simulation to Analyze and Develop Competitive Strategies, Wiley and Sons. Sadri, G. and Snyder, P. (1995) ‘Methodological issues in assessing training effectiveness.’ Journal of Managerial Psychology, Vol. 10, No. 4, pp 30 – 32. Smutný, P. (2007) Simulační hry jako nástroj zvyšování kvality lidského kapitálu podniku, unpublished thesis (doctoral dissertation), Masaryk University. Stainton, A. J., Johnson, J. E., Borodzicz, E. P. (2010) ‘Educational Validity of Business Gaming Simulation: A Research Methodology Framework‘ Simulation & Gaming, Vol. 41, No. 5, pp 705–723 Scherpereel, C. M. (2013) ‘Changing mental models: Business simulation exercises’, Simulation Gaming Vol. 36, No. 3, pp 388 – 403. Wolfe, J, and R. Jackson (1989). ‘An Investigation of the Need for Algorithmic Validity.’ Simulation & Games Vol. 20, No. 3 , pp 272‐291.

520

Using the Master Copy ‐ Adding Educational Content to Commercial Video Games Heinrich Söbke1, Thomas Bröker1 and Oliver Kornadt2 1 Bauhaus‐Universität Weimar, Faculty of Civil Engineering, InnoProfile: Intelligentes Lernen (Intelligent Learning), Weimar, Germany 2 Technische Universität Kaiserslautern, Faculty of Civil Engineering, Kaiserslautern, Ger‐ many heinrich.soebke@uni‐weimar.de thomas.broeker@uni‐weimar.de oliver.kornadt@bauing.uni‐kl.de Abstract: Successful development of educational video games has to overcome plenty of challenges. In addition to the requirements of a successful software development project, an attractive game experience has to be designed and imple‐ mented. However a failure in one of the most ambitious tasks of developing an educational game is known as "chocolate‐ dipped broccoli". This term was used by Bruckman (1999) to describe the unsuccessful integration of game mechanics and learning content. An alternative approach to avoid these problems is not new but still seems to be underrepresented in the field of Digital Game Based Learning: the use of existing, popular video games. Recent research has acknowledged several video games as educational media for the training of professional skills. Besides that video games can be used to spread domain knowledge. Such video games either already contain specific domain knowledge or they can be extended easily to embed domain knowledge. This paper presents case studies of four commercial video games, which are potentially eligible for educational extension. Each game represents a distinct category: Fliplife stands for simple, community enhancing Social Network Game (SNG) Triviador as a SNG is the synthesis of two well known board games (Risk and Trivial Pursuit). Ju‐ raShooter StGB ‐ already designed as an educational game ‐ exemplifies mobile device games. Lastly SimCity 5 is a repre‐ sentative of a classic strategy and simulation game with focus on systems. Each game is shortly introduced, our relevant gameplay experiences and the potential learning content is described. Finally we identify potential extensions regarding additional learning content for each game. Having these examples in mind we suggest a more systematic approach to use commercial video games as learning tools: if a game has been proven as a viable source for transfer of domain knowledge, we suggest categorizing it due to two characteristics: complexity of learning content and mainly attracted player types. Based on such a categorization a directory of games can be compiled. This directory then can be used in a concrete educa‐ tional scenario to identify appropriate games. As a categorization for learning content we suggest Bloom’s revised taxon‐ omy. A well known categorization for player types was proposed by Bartle. Although by far not every commercial video game can be enriched with educational content, the effort to customize existing games seems to be much smaller than the effort to create educational games from scratch. This paper argues for a systematic approach to facilitate commercial video games in educational settings as an additional option apart from individual implementations of educational games. Keywords: digital game based learning; COTS; commercial video games; SimCity, JuraShooter StGB, Fliplife, Triviador

1. Introduction Video games are a relatively new type of media. Originally just created for entertainment purposes they are now also used in educational settings. Gee (2005) refers to video games as learning machines. In the light of the huge impact on players learning sophisticated models to master a game, educators started to develop educational games. However this approach has seen a lot of failures. Egenfeldt‐Nielsen (2007) states that “edutainment started as a serious attempt to create computer games that taught children different subjects. Arguably, it ended up as a caricature of computer games and a reactionary use of learning theory.” Papert (1998) uses the picture of a Shavian reversal: “Shavian reversals — offspring that keep the bad features of each parent and lose the good ones — are visible in most software products that claim to come from a mating of education and entertainment”. Bruckman (1999) used the expression chocolate‐dipped broccoli for her observation that “fun is often treated like a sugar coating to be added to an educational core”. All these perceptions mirror at least partially the ultimate complexity of educational game development: Build‐ ing software is already a risky venture: Ambler (2010) found that only around half of IT projects are successful. The next hurdle to take is to design an attractive game: a commonly acknowledge method uses effort consum‐ ing cycles of development and play testing (Fullerton 2008). For educational games this process has to be ad‐ justed to certain learning targets and complemented with an appropriate context and content. Habgood & Ainsworth (2011) call a seamless interlacing of game and educational content “intrinsic integration”: learning

521

Heinrich Söbke, Thomas Bröker and Oliver Kornadt content should not appear as a foreign object in the game. Not being all enough, these complex tasks have to be covered with limited financial resources, because normally educational games do not reach those high sales revenues as their commercial counterparts do (Bröker et al. 2011). As one solution to this dilemma Young et al. (2012) “believe that commercial gaming companies and educa‐ tional researchers could mutually benefit by bringing academic content into the fictitious worlds originally created without educational content objectives in mind”. Commercial video games have been already used as educational tools: The Futurelab Project (Sandford et al. 2006) investigated the usage of “commercial off the shelf [(COTS)] computer games in formal education”. The Sims 2, Roller Coaster Tycoon and Knights of Honor have been those COTS games looked at in this study. The Sims 2 as a medium in formal educational settings have been studied by Peterson (2011) and Panoutsopoulos & Sampson (2012). A. N. Foster (2011) chose Roller Coaster Tycoon 3 as a tool for the transfer of disciplinary knowledge in economics and social studies. 1 Looking at all these examples at least three areas of game usage for educational purposes can be identified : First, games can be used as a medium to spur reflection. At Wabash College Portal was set on the booklist of an compulsory seminar, which “is devoted to engaging students with fundamental questions of humanity from multiple perspectives and fostering a sense of community” (Abbott 2010). Second games are used to teach professional skills. Poling (2010) facilitated StarCraft in a course about 21st Century Skills. Steinkuehler & Dun‐ can (2008) found that successful play of World of Warcraft requires “scientific habits of mind”. And third games are employed for construction of disciplinary knowledge. Squire (2003) taught history with a modified version of Civilization III. A case study of Moshirnia (2007) done with Civilization IV lead to similar results. So far we have argued that building successful educational video games is a highly challenging task. Then again there exist a lot of intriguing commercial video games. In this paper we want to propose the systematic inves‐ tigation of COTS games for either existing learning content or the possibility to add and embed appropriate content. We use a comprehensive definition of COTS video game: It is any digital game which could be used as 2 host for potential educational content but which currently is not holding that content . In addition to dedi‐ cated educational games this would be a further, effort saving approach to establish video games as educa‐ tional tools.

2. Case studies The proposed approach to use COTS games for educational purposes is based on a few assumptions: It may be possible that a game can embed educational content. For a systematic approach to classify a game its content should be graded according to a knowledge taxonomy – the knowledge dimension of the game. A well‐known taxonomy has been introduced by Bloom (1956) and modified by Anderson et al. (2000). This is not the only valid categorization – e.g. the already mentioned overview of Prensky (2007) classifies content specifically found in video games, but it is a taxonomy commonly utilized in educational contexts. Players are categorized by player type taxonomies, e.g. Bartle's player types (1996). In addition to these as‐ sumptions we hypothesize that the learning objectives are given as well as the structure of the player type distribution. Then it should be possible to choose an appropriate game, enrich it with content and use it in the considered educational setting. Such an approach would need a catalogue of COTS games capable of including educational content. Our case studies introduce four video games as potential elements for such a catalogue. We describe each game in four categories. Game description is a rough summary of the games main character‐ istics. It conveys a short impression of the game. In the section Gameplay experiences we summarize our per‐ sonal impressions of the gameplay as we think they are relevant to use that game in an educational context. Learning content is the caption for an analysis of the structure and type of potential educational content. The part Possible adoptions proposes appropriate changes of the game software to support the game’s usage as an educational tool.

Prensky (2007) gives a more comprehensive overview of different types of learning content including facts, skills, behaviour and creativ‐ ity. 2 This definition implies also educational games themselves ‐ as the example JuraShooter StGB demonstrates. The important characteristic we want to point out is the combination of an existing video game and additional educational content

522

Heinrich Söbke, Thomas Bröker and Oliver Kornadt

2.1 Fliplife Game description: Fliplife is a multiplayer, HTML5 based browser game (Fliplife 2012; Fliplife.com 2012). Unlike a typical social network game the relationship between two players is not fixed, but defined according to the amount of common actions: The more interactions two players have, the nearer becomes their relation‐ ship. Main objective of Fliplife is a simulation of the player’s life: professional career, education and free time. The player pursues a career by taking part in a well‐ defined set of projects. Spare time activities are sports and parties. These activities are rewarded with energy, which is needed for doing the projects. Education is pro‐ vided by multiple choice question based quizzes. Fliplife got special attention when it was said to be an as‐ sessment tool for a German trust (Söbke, Hadlich, et al. 2012). Gameplay experiences: The basic game mechanics of Fliplife seem to be very simple, although they can build the foundation for complex problems (Söbke, Bröker, et al. 2012). In general Fliplife shows the characteristics of an online third place: players meet online in their spare time, make acquaintances and friends (Soukup 2006). The game mechanics create a frame for communication with co‐players. Fliplife may be a low cost ver‐ sion of an online game which serves as a third place (Steinkuehler & Williams 2006).

Figure 1: Fliplife: project description Learning content: Fliplife contains domain knowledge: specialist’s expertise is represented by career paths and projects. Also the game objects are partly domain specific. However to play Fliplife successfully this knowledge is not mandatory. The so‐called university is another possibility to learn: The quizzes are quite demanding and require broad general knowledge. Possible adoptions: Müller (2012) has investigated possibilities to integrate domain knowledge in Fliplife using the domain of building physics, a discipline in civil engineering concerning the comfort in buildings. Her sugges‐ tions include a domain specific career path, a building physics institute as employer and projects from the field of building physics. Apart from descriptive texts, striking graphics could emphasize technical details. Beyond that the university can support learning by field specific quizzes and issuing field specific rewards and badges. Furthermore she suggests assignments which require the player to do a technical analysis of the projects (e.g. “Execute 3 heat protection projects.” where heat protection is not a shown attribute of a project, but derivable from the project description). For the integration of building physics learning objectives it seems appropriate to upgrade the player’s building with more technical details as it is done in the current version. An additional mini game could deal with wall structures: The player has to build wall structures from a given set of materials to meet certain requirements. Crowdsourcing is also an option mentioned by Müller: certain could cause play‐ ers to deal with technical details. Her main approach to complement Fliplife with domain specific know‐how is integration of specific information, graphics and items. As already mentioned succeeding in the game is not connected to any technical knowledge. Nevertheless it could bring domain affiliated persons together. If those persons are involved in the game, the game’s strength

523

Heinrich Söbke, Thomas Bröker and Oliver Kornadt as a virtual place could induce community building processes (“Community of Practice” (Lave & Wenger 1991)).

2.2 JuraShooter StGB ‐ Jagd nach dem Katzenkönig Game description: JuraShooter StGB ‐ Jagd nach dem Katzenkönig is an educational iOS game to memorize the crucial legal terms of criminal law (Lernfreak GbR 2012). This drill & practice game was produced 2011 by a group inspired and led by the german lawyer Raban von Buttlar. Its title refers to a well‐known and widely discussed criminal cause (Pötters 2009). Objective of the game is to unmask the alien cats of the Katzenkönig (“cat king”). This is done by instantly tapping the cats that carry parts of the correct answer to a law question. The player is rewarded for correct answer and gets extra points for faster answers and for answering in a cor‐ rect order. Questions itself are collected in packages. Each package focuses on a certain topic. Currently all content relates to the domain of law. Gameplay experiences: Our personal gameplay experiences have been dominated by audiovisual feedback caused by tapping the monsters. It seems to act both as a reward and a short cycled feedback. This feedback and the additional entry in the high score list are the main motivational elements of this game. Extrinsic moti‐ vation for learning the content vanishes during the play. Answering questions becomes a tool for reaching a high score – at least for a certain type of player. This conclusion is not yet backed up by scientific data. How‐ ever it is derived from the entries in high score lists: Top entries can only be reached when the player instantly taps the first appearance of each answer. That is when the correct answer is perfectly memorized and can be answered without delay (Buttlar et al. 2012).

Figure 2: JuraShooter StGB: Tapping definition elements Learning content: The learning content of this game is obvious. Additional packages of learning content can be added with an existing content editor. The editor does not simply support the text of the question and wrong and correct answers. It also allows determining structural attributes of a question, e.g. to take account of the order of the answer’s parts. Possible adoptions: Possible adoptions of the game include ‐ besides knowledge of other domains ‐ different types and structures of questions, e.g. matching or estimation questions. It could also be necessary to change the narrative (and accordingly graphics) because the main figure “Katzenkönig” is closely related to the law domain.

2.3 Triviador Game description: Triviador is a Social Network Game (SNG) available on the Social Network Service (SNS) Facebook (THX_Games_PLC 2011). It is an adoption of the well‐known board game Risk. While Risk uses dice for random decisions, Triviador replaces dice by questions in several forms. At the first stage a multiple choice

524

Heinrich Söbke, Thomas Bröker and Oliver Kornadt question is asked. If both players choose the right answer, an estimation question with a numeric answer is issued. The player with the nearest answer wins. In case of a tie the fastest answer wins. As a further game element players can use jokers which increase the probability of finding the correct answer in different ways. Gameplay experiences: Our gameplay experiences contained a similar trait as those already mentioned for JuraShooter StGB: Especially in the war mode the desire to win a duel outshines the awareness of probable learning. In the context of this game questions become a tool to win a match. They are not the primary pur‐ pose of playing the game, but an integrated game mechanic.

Figure 3: Triviador: Selecting bases Learning content: The learning content of the game is given by questions and their answers. Questions are taken randomly from a pool of more than 10.000 questions. So far there are no mechanics for enhancing learning: a player barely can choose the topic of questions. Also the game determines the rhythm of play – a player cannot reflect at her own pace about a question nor can she review a completed question. Possible adoptions: In general the displayed questions are chosen randomly from a large pool of questions. A so‐called Targeted Subject Booster allows the attacking player to choose the domain of the question. Further‐ more there are some country specific versions of Triviador that use a map of the country and country related questions. To enhance further Triviador’s characteristics of a learning tool the player should be able to choose preferred topics in his personal game options. During the matching phase of the game preferably two other players with same topic preferences are assigned. Furthermore the selection of questions could adhere to an algorithm of flashcard systems (e.g. Leitner (2011)). This approach could improve the educational benefit. Another feature of Triviador is the crowd sourcing of questions: When a player has reached a certain level she is allowed to enter own questions in several categories. These questions undergo a review by other players. After logging in they could opt for answering a certain amount of crowd sourced questions. They have to judge the question in several categories, e.g. if the spelling and grammar is correct and if they want to answer that question in the game. Both activities have educational value. The creation of questions means producing arte‐ facts and judging questions spurs reflection. We suggest tagging the questions as a further improvement: until now there is only a fixed set of domain cate‐ gories. Tags would introduce more and specialized categories. A question could bear more than one tag and belong to more than one (sub‐)category (Peters 2009).

525

Heinrich Söbke, Thomas Bröker and Oliver Kornadt

2.4 SimCity Game description: SimCity is a well‐known and popular simulation game. The first version has been published in 1989. Earlier this year (2013) SimCity 5 3 has been issued on a new technical foundation. SimCity is about building, developing and managing cities. Thus a player takes the role of an engineer when he interacts with the underlying systems. Will Wright, who originally designed SimCity, considers simulation games as tools for experimenting with their base systems (Wright 2007). SimCity has been used and investigated as educational tool (Adams 1998; Gaber 2007; Minnery & Searle 2012). Gameplay experiences: The attractiveness of SimCity has already been proven by millions of sold copies of every single version of the game. SimCity 5 now supports online multiplayer gameplay, e.g. it connects cities of neighboured players as sources and sinks of (material) streams. From the technical viewpoint of an engineer of 4 urban hydrology we examined the model elements of this discipline. We only found a rudimentary and sche‐ matic implementation of an urban hydrology system. D’Artista & Hellweger (2007) have examined SimCity 4 and marked its compliance with current urban hydrology models as insufficient. Compared to their findings the last version of SimCity contains more detailed models 5 . However, they still do not meet the requirements of realistic simulation models 6 . Learning content: An important learning goal connected to simulation games is the understanding of the un‐ derlying systems. Every simulation game is based on a model. Players reveal and learn the game’s model dur‐ ing gameplay. As our experiences suggest game models often do not meet requirements of reality‐like models, i.e. the structure of model does not reflect important real world elements. Devisch marks blurred, incorrect models as one argument against usage of simulation games for learning about real systems (2008). In general the genre Simulation Game has often been used as a pedagogical tool, which is demonstrated by different examples: Foster (2011) examined the knowledge construction using Roller Coaster Tycoon, Squire has used mods of Civilization III to teach social science. Mobility was built as an educational simulation game based on scientifically validated models for transport systems (Brannolte et al. 2000), but it has also reached attractive‐ 7 ness as a video game . Possible adoptions: The rule driven simulation engine GlassBox of Simcity 5 is highly configurable. It could be the base for other simulation games as well (Willmott 2012). Until now the API of GlassBox has not been dis‐ closed. If that would happen, adjusting model elements of SimCity could lead to realistic scenarios.

2.5 Questions: A valid game element for learning? Multiple choice questions (MCQ) are a main pedagogical element of three of the four above presented case studies. MCQs are often criticized. One argument is that they just contain knowledge at the recall level. And students learn to choose the right answer from a set of options instead of knowing all alternatives they have 8 . Scouller (1998) attributes surface learning to MCQ based examinations. Nicol (2007) presents a framework for MCQs as an educational tool. In general MCQs are considered as a valid assessment tool for the understanding of abstract concepts (Hopkins 1998). Iz & Fok (2007) gave an example for the domain of geomatics ‐ showing that it is possible to cover all levels of the cognitive domain of Bloom’s taxonomy with MCQs. Higher levels of complexity may require activities outside the test, e.g. doing calcula‐ tions to determine the correct answer. Furthermore they found a relation between the complexity of the question and the time students spent on answering that question. Also Simkin & Kuechler (2005) experienced difficulties in constructing MCQs of higher knowledge levels, but they confirmed that this is an practicable approach. 3

We refer to the current version of SimCity as SimCity 5, although it is officially named SimCity. Urban hydrology is – besides building physics and law ‐ one of the disciplines we are using for exemplifying usability of commercial video games as educational tools. 5 For example water pumps can be upgraded to filtration pumps. This allows the reuse of dirty water as drinking water. 6 As an example it is not possible to use surface water for drinking water catchment. Also storm water management is nonexistent in the game. 7 This was witnessed by a dialogue partner: when he heard about the authors’ affiliation to Bauhaus‐Universität, he mentioned the game Mobility which production has been scientifically accompanied by members of that University. He named this game a cause for many short nights of his former life. 8 As an example: the examination regulations of our faculty exclude MCQ based examinations explicitly as a valid test tool th (http://www.uni‐weimar.de/cms/fileadmin/uni/files/ka/mdu_akad/07/30_2007.pdf, last accessed April, 12 2013) 4

526

Heinrich Söbke, Thomas Bröker and Oliver Kornadt Palmer & Devitt (2007) stated, based on a study in the domain of medicine, that MCQ tests “are capable of withstanding the intellectual and statistical scrutiny imposed by a high stakes exit examination”. Mukhopadhyay et al. (2010) made a similar conclusion also for MCQs in medical education. MCQ may not be fun causing game elements. They are neither part of a problem solving, constructive game‐ play. But they are a valid educational instrument. And as far as they can be integrated in gameplay without affecting intrinsic motivation negatively, they can be used as an educational tool for the transfer of knowledge. Prensky (2011) defines an appropriate measurement for fun and learning in an educational game: “1. Is the game fun enough that someone who is not in its target audience would want to play it [...]? 2. Do people using it think of themselves as ‘players’ [...]? 3. Is the experience addictive? [...] 4. Are the players’ skills in the subject matter and learning content of the game [...] significantly improving at a rapid rate, and getting better the longer he or she plays? [...].” Due to our experiences these questions can be answered positively for all of our MCQ “enriched” examples – mostly without any restriction, but always to a certain grade.

2.6 Levels of knowledge complexity Game structure and game mechanics form the frame for knowledge that can be included in the game. So the complexity of the learning objectives depends on the game. The following discussion of an example uses Bloom’s revised taxonomy: JuraShooter StGB’s current learning content covers level 1 to 3 in the Knowledge Dimension: Factual Knowledge is included in the form of domain terminology. The game also contains Concep‐ tual Knowledge: legal structures and models are object of the questions, e.g. a player has to be familiar with the “legal model” of a second degree murder. An example for Procedural Knowledge is a list of necessary elements of an offense to constitute a punishable crime (“Knowledge of criteria for determining when to use appropriate procedures”). The Cognitive Process dimension is only supported in the first level: The player just needs to remember facts. The use of additional types of questions requires a change of the game software. However it would enable the game to include content of other Cognitive Dimension’s levels than level 1 “Re‐ member”. As an example ordering questions would require comparisons, which is located in level 2 “Under‐ stand”. As Triviador does support only a smaller set of question types (Selection question and Estimation ques‐ tion), the complexity of the knowledge is lower. Iz & Fok demonstrated that all levels of the Cognitive Dimension can be reached by answering questions. How‐ ever the attractivity of questions may be low when answering needs a lot of work outside of the game. Simula‐ tions games ‐ like SimCity ‐ allow to experiment with potential answers in the game itself. Because of versatile possibilities of interaction between player and software simulation games can include also higher levels of the taxonomy – at the cost of much more effort for development of the game.

3. Conclusions and discussion Educational games often do not meet the promise of a sensed effortless learning. Habgood (2007) observes "chocolate covered brocoli"‐like edutainment applications. Missing “intrinsic integration”, i.e. intrinsic motiva‐ tion cannot be consistently perpetuated during gameplay, is considered as an important reason (Habgood & Ainsworth 2011). These failures may result from the enormous complexity to create an “intrinsically inte‐ grated” educational video game. We propose an additional approach to employ video games as educational media for domain knowledge trans‐ fer: the systematic review of existing commercial off‐the‐shelf (COTS) video games. By using existing games the only challenge to overcome is to embed educational content while the game software has been produced and 9 the game itself has been proven as attractive . Furthermore successful games are considered to be learning machines teaching their content effectively (Gee 2005; Becker 2006). In a larger framework suitable video games could be classified according to two criteria: a first criterion is the potential learning content the game is capable to embed. We propose Bloom’s revised Taxonomy (Anderson et al. 2000) as an appropriate categorization. The second criterion is the predominant player type of the game. These classifications would allow selecting appropriate games for a defined learning content and a known set of learners. 9

For example often these games outperform their educational “counterparts” in terms of audiovisual presentation (Breuer 2012).

527

Heinrich Söbke, Thomas Bröker and Oliver Kornadt Of course this approach has to fight some imponderables. One of them is the question if the suggested tax‐ onomies are resilient in this context. Bartle’s taxonomy created for players of a Multi User Dungeon (MUD) may not be directly applicable to players of other game genres. Another item needing further research is the transfer to other context. Gee (2003) postulates that successful learning in games depends on context, i.e. the constructed knowledge is not transferred to other contexts. So players learn to master the game, but are not able to apply their knowledge in real world contexts (Rehm 2013; Niegemann 2009). This may become a prob‐ lem, as contexts given by commercial games cannot be adjusted easily. So the proposed classification may need to be extended by a context dimension. Also it may be questionable if necessary game related changes are feasible – the developer of a game may pursue other approaches. Concerning our sample there is a mixed situation: SimCity has been announced to be open for new content, but the GlassBox API has not yet been revealed. Triviador is open in parts: new questions can be added, but no new categories can be established and selected during gameplay. Fliplife is extensible theoretically, but de facto it is not yet clarified if the devel‐ oper company would support customized versions. This has already been done for JuraShooter StGB, which is the only ready for reuse game among our case studies. Another challenge of simulation games is the problem of simplified and blurred models, which do not suffi‐ ciently mirror domain knowledge. FarmVille (Zynga 2009) can be taken as an example: It is build for being extended with additional content (Mahajan 2010). But for crops and animals almost the only fact which is connected to reality is their name – unless they are not pure fantasy elements. Other important simulation model characteristics – as for example maturation periods – are only game specific. The proposed approach has still a lot of challenges to overcome. Games which can be enriched with educa‐ tional content are admittedly a rare species. On the other hand the possible gains of employing (modified) COTS games are compelling; even small achievements might be worthwhile. It is an additional option in the field of game based learning. A next step on our agenda is to extend JuraShooter StGB to the field of urban hydrology. While the extension of the other three examples is more hypothetical and still has to be substanti‐ ated, in the case of JuraShooter StGB the financial effort has been calculated: it is about one‐tenth of the origi‐ nal application.

References Abbott, M., 2010. Portal on the booklist. Available at: http://www.brainygamer.com/the_brainy_gamer/2010/08/portal‐ booklist.html [Accessed January 18, 2011]. Adams, P.C., 1998. Teaching and Learning with SimCity 2000. Journal of Geography, 97(2), pp.47–55. Ambler, S.W., 2010. 2010 IT Project Success Rates. Dr.Dobbs. Available at: http://www.drdobbs.com/architecture‐and‐ design/2010‐it‐project‐success‐rates/226500046# [Accessed May 10, 2012]. Anderson, L.W. et al., 2000. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educa‐ tional Objectives, Abridged Edition L. W. Anderson & D. R. Krathwohl, eds., Allyn & Bacon. Bartle, R.A., 1996. Hearts, Clubs, Diamonds, Spades: Players Who Suit MUDs. The Journal of Virtual Environments, 1(1). Available at: http://www.citeulike.org/user/drakkos69/article/3474752. Becker, K., 2006. Classifying Learning Objectives in Commercial Video Games. In Summer 2006 Institute ‐ Linking Research to Professional Practice. University of Calgary. Bloom, B.S., 1956. Taxonomy of Educational Objectives J. P. Keeves, ed., Longman London. Brannolte, U. et al., 2000. Aktualisierung und Erweiterung von Planspielansätzen im Verkehrswesen im Hinblick auf die Erstellung von Mobilitätsspielen, Weimar. Breuer, J., 2012. Broccoli‐coated chocolate? The educational potential of entertainment games. In W. Kaminski & M. Lor‐ ber, eds. Gamebased Learning: Clash of Realities 2012. München: kopaed, pp. 87–96. Bröker, T., Söbke, H. & Kornadt, O., 2011. Close the gap — Obstacles and solutions for the missing educational games in graduate education. In 5th European Conference on Games Based Learning. pp. 74–80. Bruckman, A., 1999. Can educational be fun? In Game Developer’s Conference, San Jose, California. pp. 75–79. Buttlar, R. von et al., 2012. Die Jagd nach dem Katzenkönig. In W. Kaminski & M. Lorber, eds. Gamebased Learning: Clash of Realities 2012. Kopäd, p. 384. D’Artista, B.R. & Hellweger, F.L., 2007. Urban hydrology in a computer game? Environmental Modelling & Software, 22(11), pp.1679–1684. Devisch, O., 2008. Should Planners Start Playing Computer Games? Arguments from SimCity and Second Life. Planning Theory & Practice, 9(2), pp.209–226. Egenfeldt‐Nielsen, S., 2007. Educational Potential of Computer Games (Continuum Studies in Education), Continuum. Fliplife, 2012. Fliplife. Available at: http://fliplife.com/ [Accessed April 26, 2012]. Fliplife.com, 2012. Was ist Fliplife? Fliplife for Business. Available at: http://business.fliplife.com/lerne‐fliplife‐kennen/was‐ ist‐fliplife/ [Accessed May 21, 2012].

528

Heinrich Söbke, Thomas Bröker and Oliver Kornadt Foster, A.N., 2011. The process of learning in a simulation strategy game: Disciplinary knowledge construction. Journal of Educational Computing Research, 45(1), pp.1–27. Fullerton, T., 2008. Game Design Workshop: A Playcentric Approach to Creating Innovative Games 2nd Revise., Morgan Kaufmann. Gaber, J., 2007. Simulating Planning: SimCity as a Pedagogical Tool. Journal of Planning Education and Research, 27(2), pp.113–121. Gee, J.P., 2005. Learning by design: Games as learning machines. E‐Learning and Digital Media, 2(1), pp.5–16. Gee, J.P., 2003. What Video Games Have to Teach Us About Learning and Literacy. Comput. Entertain., 1(1), pp.20–20. Habgood, M.P.J., 2007. THE EFFECTIVE INTEGRATION OF DIGITAL GAMES AND LEARNING CONTENT. University of Notting‐ ham. Habgood, M.P.J. & Ainsworth, S.E., 2011. Motivating Children to Learn Effectively: Exploring the Value of Intrinsic Integra‐ tion in Educational Games. Journal of the Learning Sciences, 20(2), pp.169–206. Hopkins, K.D., 1998. Educational and psychological measurement and evaluation 8. ed., Boston: Allyn and Bacon. Iz, H.B. & Fok, H.S., 2007. Use of Bloom’s taxonomic complexity in online multiple choice tests in Geomatics education. Survey Review, 39(305), pp.226–237. Lave, J. & Wenger, E., 1991. Situated learning: legitimate peripheral participation, Cambridge University Press. Leitner, S., 2011. So lernt man lernen. Der Weg zum Erfolg., Herder, Freiburg. Lernfreak GbR, 2012. Jura Shooter: Jagd nach dem Katzenkönig. Lernfreak. Available at: http://lernfreak.de/produkte [Ac‐ cessed July 16, 2012]. Mahajan, A., 2010. Rapidly Developing FarmVille. GDC 2010. Available at: http://de.slideshare.net/amittmahajan/rapidly‐ building‐farmville‐how‐we‐built‐and‐scaled‐a‐1‐facebook‐game‐in‐5‐weeks [Accessed September 10, 2011]. Minnery, J. & Searle, G., 2012. The effectiveness of computer games for planning education: A SimCity case study. In AN‐ ZAPS 2012. pp. 142–154. Moshirnia, A., 2007. The educational potential of modified video games. Issues in informing science and information tech‐ nology, 4, pp.511–521. Mukhopadhyay, M. et al., 2010. EVALUATION OF MCQs FOR JUDGMENT OF HIGHER LEVELS OF COGNITIVE LEARNING. Gomal Journal of Medical Sciences, 8(2), pp.112–116. Müller, N., 2012. Erweiterung von Fliplife mit bauphysikalischen Inhalten. Bauhaus‐Universität Weimar. Nicol, D., 2007. E‐assessment by design: using multiple‐choice tests to good effect. Journal of Further and Higher Education, 31(1), pp.53–64. Niegemann, H.M., 2009. Lernen durch Computerspiele : Das spielende Klassenzimmer? [Vortrag im Rahmen der Ringvorle‐ sung der Universität Erfurt “Spielemedien ‐ Medienspiele” am 08.12.2009]. , p.49. Available at: http://www.db‐ thueringen.de/servlets/DerivateServlet/Derivate‐19610/niegemann.pdf [Accessed October 9, 2012]. Palmer, E.J. & Devitt, P.G., 2007. Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? BMC medical education, 7, p.49. Panoutsopoulos, H. & Sampson, D.G., 2012. A Study on Exploiting Commercial Digital Games into School Context. Journal of Educational Technology and Society, 15(1), pp.15–27. Papert, S., 1998. Does Easy Do It? Children, Games, and Learning. Game Developer, 5(6). Peters, I., 2009. Folksonomies : Indexing and retrieval in Web 2.0, De Gruyter/Saur. Peterson, L., 2011. Values in Play–Interactional Life with the Sims, Poling, N., 2010. 21st Century Skills in Starcraft. Available at: http://www.honors.ufl.edu/courses/coursesfall10.html [Ac‐ cessed January 18, 2011]. Pötters, S., 2009. Strafrecht Classics – Der Katzenkönig ( BGHSt 35 , 347 ). JuraExamen, pp.1–7. Available at: http://www.juraexamen.info/strafrecht‐classics‐der‐katzenkonig‐bghst‐35‐347/ [Accessed April 10, 2013]. Prensky, M., 2011. Computer games and learning: Digital game based learning. In J. Raessens & J. Goldstein, eds. Handbook of Computer Game Studies. MIT Press, pp. 97–124. Prensky, M., 2007. Digital Game‐Based Learning, Paragon House, St. Paul, Minnesota, USA. Rehm, M., 2013. Do gamers change attitudes towards economics through playing manager games? Zeitschrift für Ökono‐ mische Bildung, (01), pp.162–176. Sandford, R. et al., 2006. Teaching with Games Using commercial off‐the‐shelf computer games in formal education. Fu‐ turelab. Scouller, K., 1998. The influence of assessment method on students’ learning approaches: Multiple choice question exami‐ nation versus assignment essay. Higher Education, 35, pp.453–472. Simkin, M.G. & Kuechler, W.L., 2005. Multiple‐Choice Tests and Student Understanding: What Is the Connection? Decision Sciences Journal of Innovative Education, 3(1), pp.73–98. Söbke, H., Hadlich, C., et al., 2012. Social Game Fliplife : Digging for talent – an analysis. In P. Felicia, ed. Proceedings of the 6th European Conference on Games Based Learning. Academic Publishing Limited, pp. 487–494. Söbke, H., Bröker, T. & Kornadt, O., 2012. Social Gaming – Just Click and Reward? In P. Felicia, ed. Proceedings of the 6th European Conference on Games Based Learning. Academic Publishing Limited, pp. 478–486. Soukup, C., 2006. Computer‐mediated communication as a virtual third place: building Oldenburg’s great good places on the world wide web. New Media & Society, 8(3), pp.421–440. Squire, K., 2003. Replaying History: Learning World History through playing Civilization III. Indiana University.

529

Heinrich Söbke, Thomas Bröker and Oliver Kornadt Steinkuehler, C.A. & Duncan, S., 2008. Scientific Habits of Mind in Virtual Worlds. Journal of Science Education and Tech‐ nology, 17(6), pp.530–543. Steinkuehler, C.A. & Williams, D., 2006. Where Everybody Knows Your (Screen) Name: Online Games as “Third Places”. Journal of Computer‐Mediated Communication, 11(4), pp.885–909. THX_Games_PLC, 2011. Triviador on Facebook. Facebook. Available at: http://apps.facebook.com/triviador [Accessed October 10, 2012]. Willmott, A., 2012. Inside GlassBox. In GDC Conference 2012. Wright, W., 2007. Will Wright makes toys that make worlds. Available at: http://www.ted.com/talks/will_wright_makes_toys_that_make_worlds.html [Accessed December 23, 2010]. Young, M.F. et al., 2012. Our Princess Is in Another Castle: A Review of Trends in Serious Gaming for Education. Review of Educational Research, 82(1), pp.61–89. Zynga, 2009. FarmVille. Available at: http://apps.facebook.com/onthefarm/.

530

An Application of Adaptive Games‐Based Learning Based on Learning Style to Teach SQL Mario Soflano, Thomas Connolly and Thomas Hainey University of the West of Scotland, UK mario.soflano@uws.ac.uk thomas.connolly@uws.ac.uk thomas.hainey@uws.ac.uk Abstract: The fact that each student has a different way of learning and processing information has long been recognised by educationalists. In the classroom, the benefits derived from delivering learning content in ways that match the student’s learning style have also been identified. As new modes of delivery of learning content such as computer‐assisted learning systems (e.g. eLearning) have become increasingly popular, research into these has also identified the benefits of tailoring learning content to learning styles. However, in games‐based learning (GBL), the adaptation based on learning style to enhance the educational experience has not been well researched. For the purpose of this research, a game with three game modes has been developed. 1) non‐adaptivity mode; 2) a mode that customises the game according to the student’s learning style identified by using a learning style questionnaire; and 3) a mode that has an in‐game adaptive system that dynamically and continuously adapts its content according to the student’s interactions in the game. This paper discusses the term adaptivity in a GBL context and presents the results of an empirical experimental study investigating the differences in learning effectiveness of different learning groups. The study was performed between three different game groups and a paper‐based learning group with 120 students in Higher Education learning SQL (Structured Query Language). The results show that the game developed, regardless of mode, produced better learning outcomes than those who learned from a textbook. Particularly for adaptive GBL, the learning effectiveness was identified to be higher with lower completion time compared to the other modes of the game. Keywords: adaptive GBL, adaptivity, learning style, SQL, NeverWinter Nights, RPG

1. Introduction The fact that each student has a different way of learning and processing information has long been recognised by educationalists (Kolb, 1984; Felder and Brent, 2005). In the classroom, the benefits derived from delivering learning content in ways that match the student’s learning style have also been identified (Smith and Renzulli, 1984; Price, 2004). As new modes of delivery of learning content such as computer‐assisted learning systems (e.g. eLearning) have become increasingly popular, research into these has also identified the benefits of tailoring learning content to learning styles (Miller, 2005). However, as Connolly and Stansfield (2006) have suggested, eLearning simply replicates the traditional education system (classroom style) and may be overly focussed on method of delivery, i.e. delivering materials over the web rather than on actual teaching and learning, and indeed motivating and engaging the students in the learning process. In contrast, games, particularly video games, appear to be able to engage people over extensive periods of time and also motivate them to re‐play the game repeatedly until they have mastered it (Kirriemuir and McFarlane, 2004). Therefore, some educationalists (for example, Prensky, 2006) have considered games to be a potential platform in supporting student learning and have turned their attention to what is now called games‐based learning (GBL). While many GBL applications have been developed in the last two decades, there remains a lack of empirical evidence to support the use of GBL for learning purposes (Connolly et al., 2012). Given that there appears to be genuine advantages for learning outcomes to be derived from the adaptation of teaching materials to learning styles in the classroom and remotely through eLearning, it may also be possible that GBL applications that are adapted to the individual’s learning style would improve learning outcomes. Kirriemuir and McFarlane (2004) have suggested that games, unlike classroom learning or eLearning, provide a different type of engagement as they demand constant interaction and generate a ‘flow’ that could assist in engaging students. It is therefore possible for students to adopt different leaning styles in GBL than they adopt in other learning settings.

531

Mario Soflano, Thomas Connolly and Thomas Hainey In the next section, adaptivity is presented followed by a review of previous empirical work in adaptive GBL. Section 3 discusses the evaluation of an adaptive GBL application on Section 3. The evaluation includes analysis in the difference in learning effectiveness and completion time between experimental groups. Section 4 provides conclusions from the study and discusses future research directions

2. Previous research 2.1 Definition of adaptivity In computing, there are two types of adaptation process: adaptability and adaptivity (Jameson, 2003). Adaptability refers to the ability of the student to ‘adapt’ to the system by explicitly customising the system according to their preferences (Bontcheva, 2002). On the other hand, adaptivity, which is usually used in the context of a user‐adaptive system, refers to the ability of the system to identify the student’s preferences or characteristics and customise the system accordingly; that is, the student implicitly influences the adaptation process (Mulwa et al., 2010). In modern computer systems, adaptability is usually implemented by providing customisation options that allow the student to customise the system according to their preferences. For example, in eLearning the student can choose a font size and font style associated with the learning materials. Conversely, adaptivity does not explicitly require input from the students and it is usually hidden from them. The students simply see the result of the customisation process provided by the system. Although both types can exist in a computer system, each type has differences in terms of its usage. Nowadays, adaptability through student customisation is widely used and exists in many computer systems. Whilst it is certainly useful in some circumstances, adaptability requires direct manipulation from the student that can result in an increase in the student’s cognitive load, especially if there are many options the student needs to choose (Oppermann, Rashev and Kinshuk, 1997). Conversely, adaptivity can capture the interactions between the student and the learning system and the adaptivity analyses the historical interactions before making an automatic adjustment. Adaptivity is considered to be less intrusive compared to adaptability as it does not require the student to make any changes and, as a result, the interaction between the student and the system can be maintained. This is useful especially in a system that has a considerable amount of elements that the student would need to manipulate. In addition, adaptivity can be used for a behavioural pattern analysis of the student’s interaction with the system. Such an analysis may be used to create different student experiences. The disadvantage of adaptivity is that the student does not have direct control in customising the system. For this particular research, adaptivity is used to refer to the system’s ability to automatically customise certain elements of the system based on a series of the student’s interactions with the system.

2.2 Adaptivity in GBL Reflecting back to the benefits of accommodating students’ learning styles on the learning outcomes in classroom‐style learning and in eLearning, the same benefits may also apply in GBL. However, not all elements of learning style theory can be adopted in GBL. According to Becker (2007) one of the main characteristic of games, regardless of the genre, is their interactivity, which requires players to actively interact with the game, and indicates that the ‘reflection’ element of the Felder‐Silverman learning style may not be relevant to GBL. This statement is also supported by Boyle, Connolly and Hainey (2011) who indicated that games provided an active, experiential, situated and problem‐based learning environment. According to Charles et al. (2005) and Melis and Monthienvichienchai (2004) adaptation can be incorporated into games through:

A player’s character: all actions undertaken by the character have implications; for example, if the character is wounded, the movement of the character is slower.

Non‐player character (NPC): the player can access this feedback by ‘talking’ to the NPC. Besides providing feedback, the conversation itself may be used to alter the story based on the selections the player has made in the conversation.

532

Mario Soflano, Thomas Connolly and Thomas Hainey

The game environment: adaptation through the game environment can be categorised into: customisation, contextualisation and personalisation (Melis and Monthienvichienchai, 2004). In the context of GBL, customisation is related to the functionalities of the learning system; for example, when the student reaches a particular level, a new control will appear. Contextualisation means that the content that is going to be delivered is adaptable according to the student’s performance, learning history or response to certain missions. Personalisation relates more to the student’s preference, for example, the font size.

Feedback/scaffolding: can be used to inform the student about their status in the game and to help the student to achieve the game and educational objectives (Jackson, Krajcik and Soloway, 1998).

2.3 Adaptive GBL applications ‐ empirical evidence A systematic literature search was conducted on electronic databases including: Association for Computing Machinery (ACM), Cambridge Journals Online, Institute of Electrical and Electronic Engineering (IEEE), Index to theses, IGI Global, Ingenta Connect, Science Direct, Springer Link, Wiley Online Library, Extended Academic asap, simulation and gaming and emerald. the following search terms were used: (adaptive or adaptivity or personalisation or personalization or “learning style”) and (“serious games” OR “games‐based learning”) The search returned 978 papers, however, after detailed analysis only 8 papers were identified as being relevant to adaptive GBL applications with empirical evidence and these are summarised in Table 1. Table 1: Existing empirical evidence in adaptive GBL Authors

Area

Felicia and Pitt (2007, 2008, 2009)

Maths

Conati and Zhou (2002), Conati and Zhao (2004)

Maths

Peirce, Conlan and Wade (2008)

Physics of optics

Lynch, Steen, Pritchard, Buzzell and Pintauro (2008) Hwang, Sung, Hung, Huang and Tsai (2012) Lee and Ko (2011) Yongyuth, Prada, Nakasone, Kawtrakul and Prendinger (2010) Demmel, Kohler, Krusche and Schubert (2011)

Food safety

Adaptivity Learning style and personality traits (The Big‐5 model). Cognitive theory of emotions (joy, distress, pride, shame, admiration and reproach). Number of interactions between the student and certain elements of the game. Learning style based on Dunn and Dunn’s theory. Felder‐Silverman (sequential ‐ global).

Plantation Logic programming Agriculture Languages

Successful rate of the code execution. Changes occurring in the environment caused by the student’s actions. Number of mistakes the student has made.

MathQuest was developed to teach maths (algebra) at secondary school level. Felicia and Pitt (2007, 2008, 2009) proposed a learning style and personality traits concept based on the MBTI and the Big‐5 model known as PLEASE (Personality, Learning styles, Emotions, Autonomy, Systematic approach and Evaluation). Using this model, Felicia and Pitt (2007) developed strategies that could be incorporated and implemented to the game design and game play. For example, for students with a high level of competitiveness, ranks and scores were displayed and for students with a high level of extraversion, frequent rewards were given. Felicia and Pitt (2009) conducted an experiment involving 80 secondary school students that investigated the effect of personality in learning through MathQuest. The experiment used a questionnaire on game preferences, an International Personality Item Pool (IPIP) personality questionnaire, pre‐test and post‐test of the subject matter and a questionnaire about the game. They found that adaptivity towards elements of the Big‐5 such as neuroticism and conscientiousness did not have a significant impact in learning outcomes, whilst adaptivity towards agreeableness, openness and extraversion seemed to benefit the students. They concluded that although “not all students will benefit from a video game but that when settings and options match their preferences and learning style, improvements can be obtained” (Felicia and Pitt, 2009, p.151). Prime Climb was developed to teach maths (factorisation) for grade 6 and 7 students (Conati and Zhao, 2004). To analyse the student’s performance and actions with the game, two layers of dynamic Bayesian networks were developed. The first layer, the short‐term student model, was used to capture changes in the student’s behaviour from one interface action to the next whilst climbing a specific mountain. Conversely, the second

533

Mario Soflano, Thomas Connolly and Thomas Hainey layer was used to record the student’s performance for each finished mountain. The game used cognitive theory of emotions (the OCC theory) as the basis for the student’s profile (Conati and Zhou, 2002). The OCC theory consists of 22 emotion types and in Prime Climb the emotional types that were used were joy, distress, pride, shame, admiration and reproach. To evaluate the game, Conati and Zhao (2004) used a control group (who played the original game) and an experimental group (who play the game with adaptivity) with ten participants for each group. The results showed that the experimental group had significantly higher gain in learning effectiveness compared to the control group and the control group hardly improved without any external guidance. However, in terms of the correlation between agent hints and learning effectiveness, there was no conclusive evidence. ELECTRA is an RPG game with micro‐adaptivity implemented through feedback and hints. The game uses adaptivity logs that record the interaction between the student and the game. Peirce, Conlan and Wade (2008) investigated the effectiveness of the ELECTRA adaptivity procedure and its impact on the student’s motivation. The experiment involved 49 participants split into four experimental groups (no hint, neutral hint, adaptive hint and counter‐adaptive hint). The study conducted pre‐tests and post‐tests of the physics of optics to measure the participant’s knowledge of the particular material. The post‐test also included game evaluation to measure the gaming experience. The results showed that the learning outcomes and game experience for the game with adaptive hints was better than the other groups although the difference was not statistically significant. Lynch et al. (2008) developed Ootle‐U to teach aspects of food safety. The game is adapted to the student’s learning style, based on Dunn and Dunn’s theory: motivation/learning enjoyment (M); persistence towards task completion (P); sense of responsibility (R); structure (S); alone versus peer (AP); auditory (A); visual (V); tactile (T); and kinesthetic (K). The authors suggested that M, P, R and S were related to emotional preferences while A, V, T and K could be categorised as perceptual preferences. Their experiment involved 217 participants to measure the learning effectiveness of the game. The effectiveness was measured by comparing the pre‐test and post‐test scores. The experiment also assessed the impact of learning style on learning effectiveness and the correlation between the learning achievement and the game. The results showed that the participants’ knowledge in food safety was improved although the results indicated that the improvement was not significant because the students had prior knowledge of the subject matter. With respect to learning style, the experiment indicated that a greater number of participants preferred to mix their learning methods rather than consistently use a single learning method. The authors suggested that perhaps the participants, who were considered as digital natives, interacted with various different media in their everyday life. Hwang et al. (2012) developed an adaptive GBL application based on learning style to teach about plantation. Their evaluation of the game involved 46 participants and they investigated whether learning in a game that could match the student’s learning style was better than a game without any consideration of the student’s learning style in terms of learning effectiveness and motivation. The experiment also evaluated the easiness and usefulness of the game. A pre‐test/post‐test experimental design was used with an equal number of participants in the control group (who played the game without any adaptivity) and the experimental group (who played the game with adaptivity based on the learning style identified before the game). The learning style adopted in this research was the ‘sequential/global’ element of the Felder‐Silverman learning style, identified using their learning style questionnaire, while a test sheet was used in the pre‐test and post‐test to measure the knowledge of the participant on the subject matter. The results showed that the adaptive version had a significantly higher learning effectiveness. Gidget is an autonomous agent in a GBL application developed by Lee and Ko (2011). The game itself teaches logic programming to novice programmers. Gidget has an adaptivity that is reflected in its expression based on the execution of the code (smile when the code is successful and look sad otherwise). The experiment involved 116 participants comprising 50 females and 66 males with an average age of 27.5 ranging from 18 to 59 years of age. The level of education of the participants were pre‐high school (<1%), high school (13%), college (23%), associate degree (3%), bachelor degree (38%), master degree (14%) and doctoral degrees (6%). The experiment used two groups: a control group that played the game with non‐adaptive feedback and an experimental group that played the game with adaptive feedback. The results indicated that Gidget with its expression had a positive effect on motivating the participants and the experimental group completed more levels compared to the control group in a similar completion time.

534

Mario Soflano, Thomas Connolly and Thomas Hainey Agrivillage is a GBL application based on Second Life and its objective is to teach agriculture. The game itself was developed by Yongyuth et al. (2010) and it has an adaptivity implemented through NPCs based on changes that occur in the environment caused by the student’s actions. The game was evaluated by 20 university students through the use of a questionnaire. While there was no pre‐test and post‐test of agriculture knowledge, the results suggested that the game had a positive impact on learning and raising awareness about the impact of agriculture on the environment. Demmel et al. (2011) developed a collaborative and adaptive game called weMakeWords that teaches alphabetical words and symbols related to Chinese, German or English. The adaptivity is implemented through scaffolding based on the number of mistakes the student makes. Their preliminary research, involving children aged between 4 and 8 years of age, investigated the effectiveness of the adaptive game. The data gathered in this research was analysed by looking at how many characters the students could draw after the game. The results showed that children remembered on average four Chinese symbols and their meanings and some could draw the symbols after playing the game for 15 to 30 minutes. From the empirical evidence generated from the adaptive GBL that has been discussed above, adaptivity has the potential for improving learning effectiveness in GBL. Moreover, it has also shown:

The current trend in adaptive GBL research is the investigation of the effect of theories used as the basis for the adaptivity. Such theories may be personality theories as researched by Felicia and Pitt (2009) or learning styles as used by Lynch et al. (2008) and Hwang et al. (2012).

The majority of research in adaptive GBL uses a pre‐test/post‐test experimental design. This is mostly used when measuring learning effectiveness as reflected in the differences between pre‐test and post‐test either within or between control and experimental groups. The analysis methods adopted in the studies are mainly quantitative.

The adaptive systems mostly use log files or a database to collect information about interactions between the games and the students.

3. Learning‐style‐based adaptive GBL For the purpose of this research, an adaptive GBL application based on learning style was developed. The game was intended to teach the basics of the database programming language SQL (Structured Query Language) while the learning style adopted in this game was the Felder‐Silverman learning style model, particularly the presentation elements (picture‐text). Felder‐Silverman learning style model has been widely used in eLearning and GBL and its reliability and validity have been tested. When compared to other learning style model, Felder‐Silverman model represents elements from most models which indicate the generalisability of the model. The selected genre of the game was role‐playing games and it was developed by using NeverWinter Nights 2 engine. In this study, there were three modes of the same game designed and developed:

a non‐adaptive mode of the game. This mode treats all students the same and takes no account of the student’s learning style.

an out‐of‐game adaptive mode. The characteristics of a student are identified by means of the Felder‐ Silverman learning style questionnaire completed in advance of playing the game and the game is then customised according to the student’s learning style.

an in‐game adaptive mode. In this mode, the student’s characteristics are identified during the gameplay. As it is possible for the student to change learning style in the course of the game, the game will have an adaptive system that can automatically customise the game in real‐time according to the student’s current learning style.

The difference between the modes concerns the nature of the adaptive approach adopted while the rest of the game elements such as storyline, game environment, controls and game interface, are identical. The adaptive approach itself was implemented through the presentation of the learning materials presented by the conversation system of the game.

535

Mario Soflano, Thomas Connolly and Thomas Hainey

3.1 Methodology To investigate the differences in the learning effectiveness, 120 university students with no knowledge of SQL voluntarily participated in the study. Only the participants who scored 0 in the pre‐test SQL test were selected in this study. The participants were 83 undergraduate students (69.2%) and 37 postgraduate students (30.8%) with 43 students from the University of the West of Scotland (35.8%), 46 students from Heriot‐Watt University (38.3%), 19 students from Napier University (15.8%), 9 students from University of Edinburgh (7.5%), 2 students from Glasgow University (1.7%) and 1 student from Strathclyde University (0.8%). The participants were also selected from various programmes. There were 78 participants (65%) from a computing programme, 1 students from accounting (0.8%), 3 students from business (2.5%), 10 students from engineering (8.3%), 1 student from finance (0.8%), 3 students from languages (2.5%), 1 student from law (0.8%), 7 students from life science (5.8%), 9 students from management (7.5%), 2 students from mathematics (1.8%), 1 student from medicine (0.8%), 1 student from nursing (0.8%), 1 student from psychology (0.8%) and 2 students from social science programmes (1.8%). For the purposes of analysis, the participants were categorised into two groups, computing and non‐computing students, because the number of participants in each of the non‐computing programmes were too low to be analysed individually. There were three experimental groups which represent each mode of the game and one control group which the students allocated in this group would learn SQL by using a textbook. Each participant was allocated randomly to one of the four groups. In this section, the groups will be referred to as: the paper‐based group, the non‐adaptive game group, the out‐of‐game LS group and the in‐game adaptive group. The participants were required to answer the SQL tests and those aspects of the game relating to the questionnaire. The SQL tests were conducted twice: in both the pre‐test and post‐test. In the pre‐test, the SQL test was used to identify the participants’ knowledge of SQL. Those who had no knowledge of SQL were selected for the experiment. The SQL post‐test was used to test the students’ knowledge of SQL after learning SQL through the game. The SQL pre‐test had similar questions to the post‐test. Both had 11 questions in total. The maximum points achievable for the SQL pre‐test and post‐test was 16. The analysis of the learning effectiveness to be conducted is a non‐parametric test according to the Levene statistic = 8.280 (p<0.001), while the analysis for the completion time is also a non‐parametric test according to the Levene statistic = 3.879 (p<0.024). A Levene statistic with p<0.05 means the homogeneity cannot be assumed. In this thesis, the confidence value used is 95% (p<0.05).

3.2 Results This section describes the results of the experiment starting from differences in learning effectiveness followed by the completion time analysis between the experimental groups. 3.2.1 Learning effectiveness Before learning effectiveness can be measured, the reliability of the data must be tested by an independent rater. To analyse the reliability of the data analysis the Cronbach Alpha test (Kinnear and Gray, 2008) was used. The test showed that the Cronbach Alpha value of the data is 0.990, which means the post‐test scores marked by the experimenter were consistent with the scores marked by the independent rater. The learning effectiveness of the paper‐based SQL learning for 30 university students has a mean of 2.6 (SD=3.8065). The mean score of the learning effectiveness from the non‐adaptivity game group, which consists of 30 university students, is 12.517 (SD = 4.690). The learning effectiveness of the out‐of‐game LS group, consisting of 30 university students, has a mean score of 12.35 (SD = 4.065). The learning effectiveness of the in‐game adaptive group, which also consists of 30 university students, has a mean score of 14.042 (SD = 2.354). When compared to any mode of the game, there is a significant difference in learning effectiveness between the groups. By using the Mann‐Whitney U tests, the learning effectiveness for the non‐adaptivity game group is significantly higher when compared to the paper‐based learning group (Z = ‐5.903, p<0.001). The same results also apply when comparing the paper‐based group and the out‐of‐game LS game group (Z = ‐5.995, p<0.001) and the in‐game adaptive group (Z = ‐6.495, p<0.001).

536

Mario Soflano, Thomas Connolly and Thomas Hainey However, the Mann‐Whitney U test shows there is no significant difference between the non‐adaptivity group and the out‐of‐game LS group (Z = ‐0.872, p<0.389). A similar result is found when comparing the learning effectiveness of the non‐adaptivity group to the game with the in‐game adaptivity (Z = ‐0.378, p<0.711) and between the out‐of game learning LS group and the in‐game adaptive group (Z = ‐1.546, p<0.124). So far, the tests have proven that the learning effectiveness for the modes of the game are significantly higher than those for paper‐based learning. Another statistical test is performed using the Kruskal‐Wallis test to compare all these groups (paper‐based, game without adaptivity, game with out‐of‐game LS and game with in‐ game adaptivity). The Kruskal‐Wallis test is significant beyond the 0.01 level (χ2 = 58.153, p<0.01). A summary of the comparison analysis is presented in Table 2. Table 2: Comparison of Learning effectiveness Groups

Learning Effectiveness between Groups

Paper‐based Group and Non‐adaptive Group

Non‐adaptive Group and Out‐of‐game LS Group

Z = ‐5.903, p<0.001 (Non‐adaptive Group > Paper‐ based Group) Z = ‐5.995, p<0.001 (Out‐of‐game LS Group >Paper‐ based Group) Z = ‐6.495, p<0.001 (In‐game Adaptive Group > Paper‐ based Group) Z = ‐0.872, p<0.389

Non‐adaptive Group and In‐game Adaptive Group

Z = ‐0.378, p<0.711

Out‐of‐game LS Group and In‐game Adaptive Group

Z = ‐1.546, p<0.124

All Groups

χ2 = 58.153, p<0.01

Paper‐based Group and Out‐of‐game LS Group Paper‐based Group and In‐game Adaptive Group

4. Conclusions and future directions Adapting the learning contents according to the student’s learning style has been identified to have a positive impact. An experiment was conducted to investigate the differences in learning effectiveness between different adaptive modes of a game and paper‐based learning and the difference in the completion time of the game groups. The results show that the game groups have significantly higher learning effectiveness compared to the paper‐based group. The in‐game adaptive game also produced higher learning effectiveness compared to other game groups. These findings are similar to the studies described in section 2.3. The results also indicate the potential of GBL as the supplement in teaching SQL. One of the considerations in designing an educational game lies in how to teach the learning materials without losing the ‘fun’ part of the game so the student can remain motivated to learn the materials. One way of maintaining motivation is to blend the materials and the game story in such a way that the learning materials are part of the challenge of the game. The findings here may be specific to the integration of these particular learning materials and the mechanics of the game therefore modifications in the game’s specification could give rise to different learning outcomes. However, the conversation system used to present the learning materials in the game is highly customisable and can provide a useful platform for exploring the game’s potential for teaching different disciplines or accommodating different learning styles. Further research based on this game and its variants may contribute empirical evidence of the beneficial effects of GBL and adaptive GBL in particular. The adaptivity may also be improved in future research particularly to address multiple learning styles in a game and to create more complex adaptivity in various elements of the game.

Acknowledgements This work has been co‐funded by the EU Lifelong Learning Programme under contract 519057‐LLP‐1‐2011‐1‐ UK‐KA3‐KA3NW (Ed2.0Work – European Network for the integration of Web2.0 in education and work) and as part of the Games and Learning Alliance (GaLA) Network of Excellence on ‘serious games’ funded by the Eurpean Union in FP7 – IST ICT, Technology Enhanced Learning (see http://www.galanoe.eu).

References Becker, K. (2007). Digital Game‐Based Learning Once Removed: Teaching Teachers. British Journal of Educational Technology, 38(3), 478‐488.

537

Mario Soflano, Thomas Connolly and Thomas Hainey Boyle, E.A., Connolly, T.M. Hainey, T. (2011). The Role of Psychology in Understanding the Impact of Computer Games, Entertainment Computing, 2(2), 69‐74. Bontcheva, K. (2002). Adaptivity, Adaptability, and Reading Behaviour: Some Results from the Evaluation of a Dynamic Hypertext System, AH’2002, 69‐78 Charles, D., McNeill, M., McAlister, M., Black, M., Moore, A., Stringer, K., Kerr, A., Kucklich, J. (2005). Player‐centred game design: Player modelling and adaptive digital games. Proceedings of DiGRA 2005 Conference: Changing Views – Worlds in Play, 285‐298. Conati, C., Zhou, X. (2002). Modelling Students’ Emotions from Cognitive Appraisal in Educational Games. Proceedings of th the 6 International Conference on Intelligent Tutoring Systems, London, UK, Springer‐Verlag, ACM Conati, C., Zhao, X. (2004). Building and Evaluating an Intelligent Pedagogical Agent to Improve the Effectivenesss of an Education Game. IUI'04, International Conference on Intelligent User Interfaces, Funchal, Madeira, Portugal, 6‐13. Connolly, T.M. Stansfield, M.H. (2006). Enhancing eLearning: Using Computer Games to Teach Requirements Collection and Analysis. Second Symposium of the WG HCI & UE of the Austrian Computer Society, 23 November 2006, Vienna, Austria. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59, 661–686. th Demmel, R. B., Kohler, B., Krusche, S., Schubert, L. (2011). The serious game: wemakewords. Proceedings of the 10 SIGPLAN symposium on New ideas, new paradigms, and reflections on programming and software, New York, USA, ACM, 109‐110 Felder, R. M., Brent, R. (2005). Understanding Student Differences. Journal of Engineering Education, 94(1), 57‐72. Felicia, P., Pitt, I.J. (2007). Evaluating the Effect of Personalities on the Design of Educational Games. Proceedings of the ECGBL Conference, Paisley, Scotland Felicia, P., Pitt, I.J. (2008). Personalising Educational Games to Students' Learning Styles. Proceedings of the International Technology in Education and Development Conference, Valencia, Spain Felicia, P. Pitt, I.J. (2009). Profiling Users in Educational Games. In Games‐based Learning Advancement for Multisensory Human Computer Interfaces: Techniques and Effective Practices (Eds: T.M. Connolly, M.H. Stansfield, E Boyle), Idea‐ Group, Hershey. Hwang, G. J., Sung, H. Y., Hung, C. M., Huang, I., Tsai, C. C. (2012). Development of a Personalized Educational Computer Game Based on Students’ Learning Styles. Special Issue on Personalized Learning, 60(4), Springer, 623‐638 Jackson, S. L., Krajcik, J. Soloway, E. (1998). The Design of Learner‐adaptable Scaffolding in Interactive Learning Environments. Proceedings of ACM CHI 1998 Conference on Human Factors in Computing Systems, 197‐194. Jameson, A. (2003). Adaptive Interfaces and Agents. In Human–Computer Interface Handbook. (Eds: Jacko, J. A., Sears, A.). Lawrence Erlbaum, Mahwah, 305–330 Lee, M. J., Ko, A. J. (2011). Personifying Programming Tool Feedback Improves Novice Programmers’ Learning. Conference on International Computing Education Research (ICER), Providence, Rhode Island, USA, ACM, 109‐116 Lynch, R. A., Steen, M. D., Pritchard, T. J., Buzzell, P. R., Pintauro, S. J. (2008). Delivering Food Safety Education to Middle School Students Using a Web‐Based, Interactive, Multimedia, Computer Program. Journal of Food Science Education, 7, Wiley, 35–42 Kirriemuir, J., McFarlane, A. (2004). Literature Review in Games and Learning. Report 8, NESTA, Futurelab, Bristol. Kolb, D. (1984). Experiential Learning. Englewood Cliffs, New Jersey. Prentice‐Hall Inc. Melis, E., Monthienvichienchai, R. (2004). They call it learning style but it's so much more. Proceedings of World Conference on e‐learning in Corporate, Government, Healthcare, and Higher Education 2004, Chesapeake, VA, 1383‐1390 Miller, L. M. (2005). Using learning styles to evaluate computer‐based instruction. Computers in Human Behavior, 21, 287‐ 306 Mulwa, C., Lawless, S., Sharp, M., Arnedillo‐Sanchez, I., Wade, V. (2010). Adaptive Educational Hypermedia Systems in Technology Enhanced Learning: A Literature Review. Proceedings of the 2010 ACM Conference on Information Technology Education, ACM Oppermann, R., Rashev, R., Kinshuk. (1997). Adaptability and Adaptivity in Learning Systems. in Knowledge Transfer, (Eds: nd Behrooz, A), pAce, London, 2 ed, 173‐179. Peirce, N., Conlan, O., Wade, V. (2008). Adaptive Educational Games: Providing Non‐invasive Personalised Learning Experiences. Second IEEE International Conference on Digital Games and Intelligent Toys Based Education; IEEE, 28‐ 35 Prensky, M. (2006). Don’t Bother Me, Mom, I'm Learning!: How Computer and Video Games Are Preparing Your Kids for 21st Century Success and How You Can Help. St Paul, Paragon House. Price, L. (2004). Individual differences in learning: Cognitive control, cognitive style, and learning style. Educational Psychology, 24, 681‐698. Smith, L. H., Renzulli, J. S. (1984). Learning Style Preferences: A Practical Approach for Classroom Teachers. Theory into Practice, 23, 44‐50. Yongyuth, P., Prada, R., Nakasone, A., Kawtrakul, A., Prendinger, H. (2010). AgriVillage: 3D Multi‐Language Internet Game for Fostering Agriculture Environmental Awareness. Proceeding of the International Conference on Management of Emergent Digital Ecosystems, NY, ACM, 145‐152

538

Can Moral Sensitivity be Enhanced by Game Play? Gunilla Svingby Malmö University, Malmö, Sweden Gunilla.svingby@mah.se Abstract: The question of the title may seem senseless. Computer games presumably do not foster moral sensitivity. On the contrary, the player often takes the role of soldier, thief, monster etc. with the aim of winning at any cost. And yet, digital games carry great potentials for learning. The possibility to act and engage in virtual situations may be used in many other contexts with the aim of handling difficult moral dilemmas, in which the “practicing of goodness” may lead to understanding and empathy. The study seeks an answer to the question of the title by means of the game, Men and animals. Moral development of the young generation is recognized as a growing problem by western countries. As the lives of young people are characterized by moral ambivalence, and competing moral discourses, values seem to be increasingly uncertain and fragmented. Researchers describe such morality as “situational” in contrast to rule based moral values. To develop a more coherent and reflected morality, people have to meet, discuss, and act in a variety of complex situation, leading to deepened “moral sensitivity”. The digital game Men and animals was developed to meet such demands. Embedded in a narrative, the game presents the students to authentic dilemmas that can be “solved” in more than one way. The solutions represent different moral values. The game is played in pairs allowing for discussion. To solve the dilemmas a range of opinions and information is presented. The game can be played several times allowing students to test the consequences of polar value positions. Playing takes 45 minutes. Effects and experiences of playing the game was studied involving 100 students aged 16‐19. By interviews and a questionnaire students’ moral reasoning was tested in various situations before and after playing the game. Observation of players’ discussions and arguments while playing was included. The choices made by each pair were registered and positioned on a scale from “Absolute animal rights” to “Animals have no rights at all”. At the end of the play students saw all their choices, and were invited to reflect on them in relation to the value scale. This gave room for reflection on the situational character of the choices, which for most pairs mirrored an ambivalent moral position. A majority of the students experienced the continuous demand for decision‐ making as highly engaging – and difficult. The game made the students question their initial general idea of the relations between men and animals. Before playing, a majority ascribed the same value to animals as to men. Having played, students’ decisions became more relational and situational, mirroring a more reflected position. Students themselves commented on the problem of understanding the moral components of a dilemma, as well as their own position. They articulated a feeling of not having thought of such things in depth, and interest in playing more games that challenged their presumptions. Keywords: game, learning, moral sensitivity

1. Background The question of the title may seem senseless. Computer games presumably do not foster moral sensitivity. On the contrary, the player often takes the role of soldier, thief, monster etc. with the aim of winning at any cost. And yet, digital games carry great potentials for learning. The possibility of engaging and acting in virtual situations may be used in other contexts, even with the aim of handling difficult moral dilemmas. Research shows that games offering simulations of real dilemmas have the power of supporting learning (Jenkins, 2005, Kirriemuir & Mc Farlane, 2003, Barab et al. 2007). This paper explores the potentials of a digital game Men and animals to enhance students’ moral sensitivity. Why is the enhancing of moral development by a game an interesting issue? As part of growing up, young people have to build their values in interaction with the world they live in. As life in the “post‐modern society” is characterized by moral ambivalence and competing moral discourses values seem to be increasingly uncertain and fragmented. Researchers describe the morality as “situational” in contrast to rule based moral values (Bagnall, 1998; Nussbaum, 1990). Accordingly, the moral development of the young generation is recognized as a growing problem in many western countries. Schools have been assigned an important role in helping students to develop in this respect. The Swedish curriculum, for example, states that it is essential for the school to make students embrace the values of the Declaration of Human Rights, to recognize the existence of conflicting values, and to stimulate the discussion of ethical dilemmas.

539

Gunilla Svingby Young people have to develop the ability to reflect on and articulate the morality of their own actions. They have to meet, discuss, and act in a variety of complex situations. By developing situational sensitivity education may contribute to enhancing a truly situational morality (Bagnall, 1998). What sort of educational experiences may contribute to moral sensitivity in a world of conflicting moralities and life styles? The traditional method of transmitting of rules from teacher to students is not effective enough, which is confirmed by teachers, who express difficulties in dealing with moral issues (Oscarsson & Svingby, 2005). The shortcomings of formal schooling in this area is, further, shown by the results of four successive Swedish national evaluation studies in which 2 ‐ 4000 students answered questionnaires that contained both open and closed questions, and simulated real‐life situations. On the whole, students expressed a positive interest in discussing complex moral dilemmas, and an overall positive attitude to the general curriculum values (Svingby et al, 1990; Svingby, 1993, 1998; Oscarsson & Svingby, 2005; Jönsson & Persson 2006). When students had to act in life‐like situations many, changed choice of action and the underlying value of their choice. Age and gender influenced the answers, and the tendency to change when confronted with a tangible situation. At the age of 12, both girls and boys answered in line with the established values, but neither group could explain their answers. At the age of 16, girls still – on a general level – agreed with the curriculum values, while about one third of the boys did not. When the values of the Declaration of Human Rights were tested in “real” situations, many boys shifted from the altruistic values to an egoistic position. The studies further, indicated that boys lacked the competence to explain the reasons of their choices, whereas girls to a higher degree were able to give explanations to their choice of action. It seems as if the boys, neither in the school nor in the world outside of the school, had met the challenges of exploring and discussing the moral basis of an action or the consequences of alternative actions.

2. Games and learning As computer games are part of the every‐day life of many European young persons, and they already use them to act out various types of morality, it can be argued that today the experience of digital gaming is part of the construction of a student’s identity and morality (Svingby, 2005). The importance of experience ‐ lived or simulated ‐ in forming values is established which means that gaming experiences might be used for formal learning. Researchers argue that games have a great potential for learning by linking virtual world problems to problems in the real world, and by allowing players to test different identities and roles (Jenkins, 2005; Gee, 2006). Being involved in simulated situations which are interpretations of the situations‐as‐lived, the player can test out consequences in a virtual world before s/he acts in the real world. Games may, thus, be a means to develop alternative moral actions (Nussbaum, 1990, Gee, 2006). The question is, however, what type of moral values are practiced in games. While commercial games often offer the involvement with characters like soldiers or thieves, there is no reason why a game could not let players act in the moral dilemmas that ordinary people encounter in their daily life, and thus experience a match between the game world and the way ordinary men sees and acts in the real‐life world. Such a game may offer the possibilities to confront their own values in the game world. Gee (2006) argues in line with this: Since video games are “action‐and goal‐directed preparations for, and simulations of, embodied experience” they allow language to be put into the context of dialogue, experience, images, and actions. Furthermore, good video games give verbal information “just in time” – near the time it can actually be used – or “on demand” – when the player feels a need for it and is ready for it (p. 17)

3. A prototype of a digital game on moral issues The digital game Men and animals was developed to meet such demands. The game is seen as a prototype of a type of digital games that may be developed and used for the enhancement of moral sensitivity in a row of life situations. The intent of the game is to let players aged 15‐18 explore the complex interrelationships between men and animals. The game is based on the assumption that to build a morality of his/her own a student has to be engaged in authentic/virtual problems that can be “solved” in more than one way. When playing the game, students will meet with and explore different value positions. In each situation the players take action according to one of five different moral values. The consequences of each choice will be mirrored on the

540

Gunilla Svingby health‐and happiness scale of the “game inhabitants”. In order to make the decisions more informed and reflected the game presents relevant factual information offers time for discussion and reflection. The game narrative portrays a small virtual city. The players act the character of Mayor of the city. A range of realistic situations are presented which forces the Mayor (the players) to take decisions on if and how to use animals for medical research, for tests of cosmetics, and as food. Before taking a decision, players can ask for advice from the five members of the ‘Citizenship committee’. These are virtual characters designed to represent the five positions on a scale from animal liberationist – free all animals – to the position that man can use all animals without any restrictions (Orlans, 1993). Additional facts are presented to players if they ask for it. Various long term health problems representing different degrees of self‐infliction are presented to the Mayor, who chooses among five different long‐term actions. The demand for ‘fast action’ is met, when unforeseen things occur, which demand immediate action. The game is constructed to give the players enough time to reflect on the situations before taking a decision with enduring effects. Quite substantial texts are presented, and players are given time to read, reflect, and discuss. At any time, the players can get an indication of how satisfied the villagers are and of their health status. To use animals for medical research will, as an example, give credit from those positive to man’s use of animals at large, but will create hostility from ‘animal liberationists’. Continuously, players can also observe their decisions in relation to the five value positions referred to above. The recommendation to play the game in pairs is in line with the request for discussion and reflection. Playing takes about an hour and the game can be played several times allowing students to test the consequences of polar value positions.

4. The issue of men and animals The specific context of the game is man’s use of animals. This issue has grown controversial during the last decades, which is illustrated by questions like ‘For what reasons and to what extent can humans use animals? How can you distinguish use from abuse?’ There are no simple answers to these and similar questions. Since Singer published his book Animal liberation (1975) the debate has been intense on man’s right to use animals for his own good. The debate has focused on a range of situations where animals are used (and abused) by man. The positions range from on one hand seeing man first of all as an animal amongst animals to on the other hand stressing man’s superiority to the rest of the creation, with an absolute right to use animals in his own interests. One background to the study was a representative Swedish study including about 2000 students aged 15‐16, who among other questions about morality and action, answered questions about men’s use of animals (Jönsson & Persson, 2006). Half of the students answered yes on the general question Is a human’s life worth more than the life of an animal? implying that the Human rights declaration should also apply to the animal world (Ibid, 2006). When confronted with specific, authentic situations the majority, however, gave preference to men. About sixty percent of the students, for example, accepted medical tests on animals to find cures to severe illnesses. But there were restrictions. All animals were not ascribed the same value, and human illnesses were graded. Banana‐flies and rats were accepted for research, whereas dogs and cats were not, and few accepted to use animals to find a cure to self‐inflicted illnesses. Boys and girls as groups answered differently. While a majority of the boys in the authentic situations said that a man’s life has a higher value than that of an animal, only a third of the girls supported the same opinion. The dominant argument forwarded by the girls is that animals should not suffer because of human’s disastrous ways of life. The answers, further, revealed that students’ decisions to a large extent were based on emotions and not on factual knowledge or on reflected values. For the majority of the students, the answers, thus, did not mirror a reflected moral attitude on the use of animals, and the majority of the students seemed unable to articulate the rationale behind their moral choices (Oscarsson & Svingby, 2005). Empirical studies The game was designed and researched according to an iterative model called Design Based Research (Barab & Squire, 2004). Four field studies were conducted with in all 151 students (Svingby, 2005; Bergman & Svingby,

541

Gunilla Svingby 2006; Svingby & Jönsson, 2007). The game was played in pairs for an hour as part of the ordinary teaching of Social Studies. Before playing, students answered a questionnaire about their attitudes to animals, and about their gaming experiences. Some of the questions were identical to the ones used by the National Evaluation Study of 2003 (ibid). The questions were a mix of open and multiple‐choice. Some questions portrayed authentic dilemmas. After having played students answered the questionnaire again. Students were also observed while playing, and in two of the empirical studies they were interviewed.

5. Research questions and data collection The overall aim of the studies was to investigate if the game, when used in a school setting, can help young people develop a more reflected, and informed moral attitude to the issue of man’s use of animals. As the game is seen as a prototype of a type of game designed to promote the experience of the reflection on moral issues of various kinds, the aim was secondly to get information about how students played the game, and how they reacted to this specific type of digital game. The following more specific questions were explored:

1. Do students take time to discuss before they take a decision? What type of discussions do students engage in?

2. Do students use the factual information offered by the game? Do they express a wish for more information?

3. Which moral positions do players choose? Is there a pattern?

4. Do students change their moral position towards man’s use of animals?

6. Design Research data was collected as follows: before playing the game, during play and after play from 148 students aged 15‐18, who played in pairs (78 women and 70 men) as is shown below: Questionnaire 1 Gameplay + Observation Questionnaire 2 (+Interview) Before playing: Students’ attitudes to man’s use of animals were collected by a questionnaire with multiple‐choice and open questions. Some of the questions portrayed authentic dilemmas. During playing: Students’ discussions when playing were gathered by observation/ field notes. In three of the studies, one researcher who was present in the classroom observed all discussions between the players while playing. In one study, each pair of players was individually observed. Students’ choices were registered. The choices made by each pair in the various situations were registered and positioned on a scale from Absolute animal rights to Animals have no rights at all. At the end of the game session, students were presented to all their choices, and were invited to reflect on them in relation to the value scale. After playing: Students’ experiences of playing the game were gathered by multiple‐choice and open questions and by interviews. Students’ attitudes to man’s use of animals were collected by a questionnaire with multiple‐choice and open questions. Some of the questions portrayed authentic dilemmas.

7. Results Playing the game

542

Gunilla Svingby The game was very well received by the students, who found it interesting and provoking. A majority of the students experienced the continuous demand for decision‐making as highly engaging – and difficult. In contrast, a group of experienced game‐players, mainly males, mentioned the striving for credits as the most engaging. The observations confirmed that this group played the game differently. They did not discuss the situations as moral dilemmas but rather how to get high scores on the health‐and‐happiness scale. Except for the male students mentioned above, most of the students were observed to be intensely engaged in discussions both when playing the ‘slow’ situations and the situations demanding immediate action. According to most students the embedded realistic situations where you have to make difficult decisions, and can see the effects of on people in the virtual community, was the most interesting aspect of the game. Comments like the following were frequent: ‘It makes you aware of how society has to deal with problems that its inhabitants evaluate differently’. ‘How difficult it must be to be a politician!’, and ‘It was an interesting game that made us think on ethics and animals right in a way we had never done before’. The fact that the game was played in the classroom added to its value. Use of the inbuilt and external information Students appreciated both that different opinions were present in each situation, and the factual information offered in the game. Many expressed an interest in acquiring more information on themes such as ‘how animals are treated in research laboratories’, ‘national and international laws on animal rights’, and ‘the causes of various types of human illnesses’. This information can be found via internet links connected to the game. Few students, however, used the links while playing, but they may be used by teachers in a follow up of the game. An example lesson is offered as a complement to the game.

8. Values On the general question of the worth of men and animals, the students answered in line with earlier studies (Oscarsson & Svingby, 2005). About half of the students, thus, agreed that ’A human’s life is worth more than the life of an animal.’ Playing the game seemed to have an effect of students’ reasoning about men and animals. Even if the initial attitude did not change much, answers to the open questions showed that playing had challenged students’ presumptions. The answers to the second questionnaire, thus, showed to be more situational, which was mirrored in the choices made in the game. Depending of both the problem and the specific animals and men involved, students’ choices varied on the scale, mirroring both situational morality and the fact that the situations and dilemmas were new to the students. Students, thus, to a greater extent differentiated between various types of animals ‐ and men. For example, more students accepted animal testing for medical research using rats and pigs, but not to cure ‘self‐inflicted’ illnesses: ‘Animals should not suffer because people got ill by drinking alcohol’. Further, after playing, the answers from female students, who initially were more restrictive to the use of animals differed less from those of the men. The majority of the students said that playing the game may have an impact on a player’s attitudes, by making you more observant, more conscious, and more sensible on the issue. That students changed quite little as measured by the questionnaire may be explained by the fact that the problems encountered by the game were new to them. A majority said that they had seldom thought of moral issues in depth, and that the issue of men and animals had not been discussed either at school or at home. The majority expressed an interest in playing more games that challenged their presumptions.

9. Conclusions The ambition with this project was to explore the potentials of a digital game to enhance moral sensitivity on the issue of men and animals. The results are limited to the specific content area, but the ambition is to see the game as a prototype of games for learning with the aim of offering virtual experiences of value laden situations to promote greater sensitivity, and competence in various moral dilemmas (Svingby 2009). In order to do so, the specific game was built on the assumption that when exposed to virtual but authentic situations which can be solved in several ways, young people are helped in the building of reflected values. Accordingly, the situations offered have to be difficult, mirroring the multitude of value positions possible, and the underlying moralities. The game, further, must be built so that the players will be informed as well as emotionally involved. The game design gave room for reflection on the situational character of the choices.

543

Gunilla Svingby The game content is very close to real life. The problems that the players are confronted with in the virtual world have their equivalents in the real world. The game brings questions of similarities and differences between men and animals, of human versus animal rights, of research ethics to the fore, questions that can be followed up in the classroom. As the players take the virtual identity of a decision maker, being constantly confronted with problematic situations that demand a choice between alternative solutions, most students are prompted into discussions of the ethical basis of a solution. The discussions were both general and specific: Shall animals have the same rights as men? In all situations? All animals‐ or only the ‘cute and curly’? Have some animals have more rights than others, and do some men have less rights than others? The impact of a game will always depend on to what extent the players engage in the moral problems of the game, or in the striving for the highest possible credits. As Gee (2003) argues, students’ projective identity, the identity they find most appropriate in the play situation, either reflects the players own values independent of the credit given, or it reflects the values that the player estimates will give the best credit. The studies show that most students did engage in the moral problems, but that a group of experienced males mainly looked for a way to gain the most credit. A revised game must adapt also to these players by making reflections on the dilemmas necessary to gain high credits. An hour is a short time to engage in difficult dilemmas. To really contribute to students’ moral sensitivity, the issues need to be discussed further in the classroom and outside of it, and maybe the game can be played more than once with the aim of exploring what will happen if you adapt all choices to a definite moral position. The study supports the assumption that a good simulation game on a central issue may be a valuable tool to stimulate students’ engagement in and learning of values, and indicates that the actual game can be a rewarding tool both for engaging students in moral problems, and for the deepening of the understanding of such problems. Further studies of actual learning and of the integration of games in the school learning system are needed to realize the full learning potential of games in education. The educational use of electronic simulation games works the same way‐‐not as a replacement for good teaching or tried‐and‐true methods, but as a tool that good teachers can use to spark learning and to provide a context for a range of other related experiences. More and more teachers are bringing games into their classrooms on these terms” (Jenkins, 2005).

References Bagnall, R. (1998) Moral education in a Postmodern World: Continuing Professional Education, in Journal of Moral Education, 27, pp 313‐331. Barab, S. A.,& Squire, K. (2004). Design‐Based Research. Putting a Stake in the Ground. Journal of the Learning Sciences. Volume 13, Issue 1, 2004. Barab, S. A., Sadler, T. D., Heiselt, C., Hickey, D. & Zuiker, S. (2007). Relating Narrative, Inquiry, and Inscriptions: Supporting Consequential Play. Journal of Science Education and Technology, 16 (1), 59‐82. Bergman, P. & Svingby, G. (2006). Utveckling och utprövning av ett elektroniskt spel om människors förhållningssätt till djur. Interimsrapport 2006‐02‐23. Malmö: Malmö University. Gee, J.P. (2003). What Video games Have to Teach Us About Learning and Literacy. USA: Palgrave Mac Millan. Gee, J.P. (2006). Why are Video Games Good for Learning? Madison, USA: University of Wisconsin‐Madison, Department of Curriculum and Instruction, Department of Educational Psychology. Jenkins, H. (2005). Getting Into the Game. Educational Leadership, Apr.2005, Vol. 62, Issue 7. Virginia, USA: Association for Supervision & Curriculum Development. Jönsson, R. & Liljefors Persson, B. (2006). Religionskunskap i årskurs 9: Rapport från den nationella utvärderingen av grundskolan 2003 (NU03). (Religious Studies in Year 9: The report of the national evaluation of the compulsory school 2003), Malmö: Malmö University: Educare. Kirriemuir, J. & McFarlane, A.E. (2003). Literature Review in Games and Learning, Report 7. Bristol: Nesta Futurelab. Nussbaum, M. (1990). Love´s Knowledge: essays on philosophy and literature. Oxford: Oxford University Press. Oscarsson, V. & Svingby, G. (2005). Sanhällsorienterande ämnen. Ämnesrapport (Citizenship education. Main report. The national evaluation, 2003) Stockholm: Skolverket. Orlans, F.B. (1993). In the Name of Science. Issues in responsible Animal Experimentation. New York: Oxford University Press. Singer, P. (1975). Animal Liberation: a new ethics for our treatment of animals. New York. Svingby, G., Lehndals, B.& Ekbom, D. (1990). Omvärldskunskap:SO. Bakgrund, Beskrivning av undervisningen. Fostran till demokrati. (Social Studies: Background, The teaching, Democratic values). Gothenburg: University of Gothenburg. Department of Education, 1990:02.

544

Gunilla Svingby Svingby. G. (1993). The result of the National evaluation of Social Studies, 1992. (Samhällsorienterande ämnen. Den nationella utvärderingen av grundskolan 1992). Stockholm: Skolverket, nr 17. 1993. Svingby, G. (1998). Vad är rätt och rättvist? Ungdomars val i etiska situationer som utryck för demokratisk kompetens. Samhällsorienterande ämnen. (What is right and what is just? The National Evaluation of Social Studies, 1995). Stockholm: Skolverket. Nr.125. Svingby, G. (2005). Teaching citizenship with a collaborative computer game. In Ross, A. (Ed.). Teaching citizenship, pp487 – 493. CiCe publication, London Svingby, G. & Jönsson, R. (2007). What is Use and What is Abuse of Animals? Malmö: Malmö University. Working paper. Svingby, G. (1997). Utvärdering av de samhällsorienterande ämnena. (The 1995 National Evaluation of the Compulsory School: Social Studies). Utbildning och Demokrati, tidskrift för didaktik och utbildningspolitik, 2/1997. Göteborg. Svingby, G. (2009). Practicing goodness by a computer game. A model for enhancing children’s pro‐social behavior. Scientific Project Plan. Malmö: Malmö University, Working paper.

545

Digital Educational Games: Adopting Pedagogical Agent to Infer Leaner‘s Motivation and Emotional State Ogar Ofut Tumenayu and Olga Shabalina Volgograd State Technical University, Volgograd, Russia ofuriti4u@yahoo.com O.A.Shabalina@gmail.com Abstract: Digital educational games (DEGs) possess the ability of providing an attractively and essentially motivating learning context. However, an adaptive learning game would increase the probability of a DEG being actually motivating and emotionally appealing. A pedagogical agent‐based environment suggests a new opportunity for computer mediated learning emphasizing virtual social relations between learners and computers. The overall goal of this work is to provide pedagogical agents with social intelligence, so that they can decide when is an appropriate time to interact with the learner, be sensitive to the motivational and emotional state of the learner, and try to develop a positive social relationship with the learner. The study examine the effects of the competency (low vs. high) and interaction type (proactive vs. responsive) of pedagogical agents as learning companions (PALs) on learning, self‐efficacy, and attitudes. Interactions are been analyzed as a series of interaction tactics, where the speaker seeks to address one or more informational, motivational, or social goals, and monitors the listener’s response to ensure that these goals are achieved. This is followed by brief look at Affective Human Agent Interaction to have Existing agent systems typically infer human affect by sensing and reasoning about the state of a game or an outcome related to an action taken by the user within the learning environment. Keywords: pedagogical agent, pedagogical agents as learning companions, interaction type, motivational and emotional state

1. Introduction Animated pedagogical agents can promote effective learning in computer‐based learning environments (Johnson, Rickel, and Lester 2000). Learning materials incorporating interactive agents engender a higher degree of interest than similar materials that lack animated agents (Moreno, Mayer, Spires, & Lester 2001). Animated agents can produce a positive affective response in the viewer, sometimes referred to as the persona effect (Lester, Converse, Kahler, Barlow, Stone, and Bhogal 1997). This effects would go long way to improve the motivation and emotional state of the learner. They exploit the computer human tendency to tread computer systems as if they are social actors (Reeves and Nass 1996), capable of expressing emotions and attitudes. It has also been acknowledge by researchers the important in tutoring of recognizing the learner’s affective and motivation state ( Lepper, Woolverton, Mumme, & Gurtner 1993). Inferring and tracking computation techniques are being developed for such state (De Vicente and Pain 2002). (Conati, C. & Zhou, X. 2003) and for influencing them positively e.g. to try promote a positive atti+tude about learning ( Del Soldato and du Boulay 1995). In a nutshell, this paper would report the idea of social interaction procedures between pedagogical agent and the learner in a DEG in order to improve motivations and emotions in game based learning. It main purpose is to discuss how pedagogical agents can help to make better learners’ levels of interaction with the course content.

2. How pedagogical agents (PAs) heighten effective motivations and emotions to foster learning 2.1 Pedagogical agents (PAs) in DEG Pedagogical agents are human‐like characters that are included in educational and instructional materials in order to somehow stimulate learning (Eliseo, Eunice, and Letícia 2007). Pedagogical agents can be defined as entities that can perform a task or a set of tasks in addition to performing pedagogical roles. Animated agents can mimic human gestures and emotion‐driven expressions, which can help engage students’ more than static avatars (Elizabeth & Debbie). When provided with shape, personality and abilities of human agents can play the role of human‐like characters.

546

Ogar Ofut Tumenayu and Olga Shabalina With the help of Artificial Intelligent Pedagogical agents can be autonomous with a way of thinking through knowledge representation and the use of rules. They can have abilities like moving, talking, feeling and reacting to user within their environment. “a learner can learn content through interacting with one or more pedagogical agents, who may provide information or encouragement, share menial tasks, or collaborate with the learner” (Kim and Baylor, 2006, pp. 224). Pedagogical agents allow communication and interaction in learning environments (Girara & Viccari 1998). Animated pedagogical agents, especially life like characters, have significant motivational benefits; play an important pedagogical role by acting as virtual learning companions (Maragos & Grigoriadou 2005). Agents can also have different roles to support education, for example observing the students' actions and assessing them, in addition to providing feedback, explanations and demonstrations to the learner (Hospers et al. 2003; Abbas & Sawamura 2009). In a study conducted by Kim and Baylor (Kim and Baylor, 2006, pp. 224) it was found that students who worked with high‐competency Animated Pedagogical Agents – in both proactive and responsive conditions – achieved higher scores in applying what they had learned and showed more positive attitudes toward the animated pedagogical agents. According to Atkinson (Atkinson 2002) voice animated pedagogical agents environments have better results in measures of learning than text‐only or audio‐only environments. Moreover, emotional agents can be used to support the student system interactions and provide human‐like tutoring (Nkambou 2006; Neji et al. 2008). In this way, the activities of pedagogical agents can create a situation where by learners’ engaged themselves in a more meaningful interaction in which can infer the learners’ motivation and emotional state (see Figure 1).

2.2 Pedagogical agents (PAs) can create emotion and motivation in learning process According to Scherer (Scherer 2000), emotion is the relatively brief episode on synchronized responses for most or all organic systems for the evaluation of an external or internal event as being of major significance. Some examples of emotions are anger, sadness, joy, fear, shame, pride, and desperation. According to the approach of Peter and Herbon (2006) emotions are represented by the circumflex model of emotion (e.g. Russell, 1980; Larsen and Diener, 1992). The circumflex model is divided into two continuous dimensions of pleasantness and activation. Pleasantness or valence is considered as a bipolar dimension with the two poles pleasantness and unpleasantness. Activation, also known as arousal, is considered as a unipolar dimension with the poles of low and high activation (e.g. Harcourt and Lang, 1995). Each emotional or affective state can be described in terms of these two independent dimensions. For example, the emotional state excitement could be characterized as highly activated and pleasant (Larsen and Diener, 1992). Studies on the effect of emotional states on learning outcomes and efficiency (in particular with respect to valence) have yielded ambiguous results (Bower, 1992). However, with respect to activation, research indicates unambiguously that a medium level of activation leads to a superior learning process in terms of efficiency and sustainability in comparison too high or too low activation levels (Revelle and Loftus, 1992). Frijda (1994)considers that the emotion is an intentional mental state, because it is “directed toward” an object, its intentional object. For instance: I am angry with John, but I admire Nicholas. Integrating pedagogical agents with emotional features into a learning environment, aim at capturing and managing the emotions expressed by the learner during a learning session. The inclusion of emotional features in pedagogical agents is with high importance for improving the level of interaction in man machine communication system. The works on affectivity and education have also considered the motivation of the student. Student motivation deals with the student’s desire to participate in the learning process (Ames, 1990).Formal studies show that motivation was seen as a personality trait, as something people had in higher or lower degrees, a depending part on their genetic nature and on their childhood experiences (Meece &, 2001). However, current researchers have established that motivation is sensitive to contexts and it can be fostered in digital educational game. This way, a lot of works have developed in the sense of fostering student’s motivation to learn in educational computing systems (Del Soldato & De Boulay,1995; Bercht & Viccari, 2000; De Vicent & Pain, 2002). Motivational aspects in Digital Educational Game can be addressed from two perspectives, the perspective of the learning game design (as a combination of typical game design with instructional design, which can be considered being more than the sum of its parts) and the perspective of the game’s dynamic and intelligent adaptation to the motivational state of the learner. The approach to motivational aspects in game‐ based learning can be utilized for both, design and adaptive features of the game (Elke Mattheiss 2009).

547

Ogar Ofut Tumenayu and Olga Shabalina

Figure1: Pedagogical agent techniques: DEG hosting PA Now pedagogical agents with a high sense of motivation and emotion state in digital educational games are likely to optimize interactivity and further enhance a better flow of the game, which can promote a focused attention of the player: Firstly, by making the aim to be achieved understandable at all point through direct interaction with the agents with a clear provided feedback without the player bordering to think on what to do. Secondly, the challenges would be address to suit the ability of the player in order to avoid negative experience like boredom or anxiety in order to encourage a better flow. And the player can now be in control of the situation. Thirdly, the game would be more instructional to play with defined procedure of actions, which would promote fluentness. A better usage would avoid the player engaging on inappropriate actions.

3. The competency of a pedagogical agent and how it interacts between peer 3.1 Impact of pedagogical agents interactivity By simulating human instructional roles, pedagogical agents may provide learners with similar social contexts. Given that human computer interaction is consistent with human‐to‐human interaction (Reeves & Nass, 1996), learners might become more engaged in learning tasks through social interaction with pedagogical agents. Taking into concentration the potential of pedagogical agents for learning, several studies have examined the instructional impact of pedagogical agent‐based learning environments. Learners exposed to an environment with a pedagogical agent demonstrated deeper learning and higher motivation than learners without an agent (Moreno et al., 2001). Students in a voice‐plus‐agent environment outperformed those in a text‐only environment and those in a voice‐only environment on both process and product measures of learning. Similarly, students in the voice‐plus‐agent environment perceived worked‐out examples as being less difficult than did their counterparts (Atkinson, 2002).

3.2 Pedagogical agents as learning companions (PALs) Baylor and Kim (2003c) effectively simulated agents as an expert, a motivator, and a mentor who served distinct instructional purposes. The expert agent provided learners with relevant information in the professional manner; the motivator agent provided verbal persuasion and encouragement, emphasizing affective affiliation with learners; and the mentor agent incorporated both qualities to simulate an ideal instructor. In contrast to the other agent roles, pedagogical agents as learning companions (PALs) adopt a peer

548

Ogar Ofut Tumenayu and Olga Shabalina metaphor, where the agent serves to learn “with” the learner, acting as a simulated peer. In this manner, PALs have the potential to strengthen social relations and to facilitate social interaction during the learning process. PALs are pedagogical is descript to be an agents with the specific role of peers. PALs are different from other learning companion systems. PALs with peer‐like appearances and behaviors interact with learners to simulate traditional peer mediated learning. PALs may be able to eliminate some limitations like pairing logistics and distractions. Bandura’s social cognitive theory (2001) supports the benefits of human peer partners in intellectual and social development. According to Bandura, a great deal of psychological modeling occurs when learners see their everyday associates as similar to themselves. Learners tend to enhance their self‐efficacy beliefs based on perceptions of peer models. Peer interaction can provide a free and open forum to facilitate a more active, and thus a more productive, exchange of ideas (Driscoll, 2000). The main function of PALs is to serve similar functions as human peers; and also, the functions of PALs can be designed to meet specific instructional goals. Here, the potential functions of PALs for learning and motivation are suggested in terms of social cognitive theories. According to distributed cognition, PALs can share learners’ cognition and flexibly function as artifacts (cognitive tools) or collaborating partners (social tools). PALs as cognitive tools can bring knowledge and skills that learners intend to learn and mechanical tasks to preserve the ample cognitive capabilities of the learners for higher mental activities (person‐plus cognition). PALs as social cognitive tools can provide learners with social contexts by discussing and sharing ideas and perspectives with the learners. In these ways, PALs could play significant roles as social intellectual partners for learning. Similarly, Salomon (1990) suggests two types of outcomes in technology‐mediated learning, effects of technology and effects with technology. Effects of technology include the acquisition of cognitive skills resulting from the use of technology. Effects with technology are cognitive changes occurring “while” the learner is working with computer software. PALs as both artifacts and collaborating partners support learners in attaining both outcomes.

3.3 PALs competency PALs can be designed to achieve different levels of competency depending on the theoretical perspective. A PAL could be designed to demonstrate knowledge and skills maximizing the effect of technology (Salomon, 1990) or to advance the learner’s knowledge in the zone of proximal development. Likewise, the concept of proxy agency (social modeling) supports PAL competency that is high, but not too high, to serve as a desirable social model for learning. As Bandura (2001) warned, a highly competent model could sometimes impede the cultivation of personal competencies because the learner might unduly rely on the model’s competency. In a review of literature on peer modeling, Schunk (1987) noted that similarity in competency between a learner and a model might serve as an important source of self‐efficacy information, especially in unfamiliar tasks, where the learner had little information on which to base self‐efficacy judgments. It would be apparently reasonable for learners, for especially probationer learners, to increase their self‐ efficacy in the task more by working with a low competency PAL than with a high Competency PAL. The educational advantage of this approach is worth using. There would be need for further research to confirm the findings of studies and to identify if the optimal level of PAL competency would interact with learners competency.

3.4 Interactive type of PAL The interaction that occurs among students is extremely dissimilar between a Digital Educational Game and the traditional classroom course. The agents’ format excludes physical interaction, which may contribute positively on learning. The interaction type suitable for PAL in these contexts is learner –learner Interaction. Learner‐learner interaction can be between one learner and a tutee or between several tutees. Goodman and his colleagues (1998) suggested a learning triangle that illustrates the types of interaction exchanges between a learner and a partner: clarify, critique, explain, question, evaluate, articulate, reinforce, and justify. Related to the content of interaction, Craig, Gholson, Ventura, Graesser, and Group (2000) examined the effect of learning companion dialogue versus monologue with 48 college students in a domain of computer literacy. Results showed that students who overheard the dialogue in which a virtual tutee asked a virtual tutor questions wrote significantly more propositions in free recall and asked significantly more questions in the transfer task than did students who overheard a monologue. Also, students who overheard the dialogue showed deeper‐level reasoning. In order to promote effective learning by using learner ‐learner, four types of peer behavior are necessary in a computer mediated environment: (a) participation, (b) response, (c) provision of affective feedback, and (d)

549

Ogar Ofut Tumenayu and Olga Shabalina short, focused messaging. Team work, or collaborative learning, involves students working together in groups to complete academic assignments (Alavi, 1994; Palloff & Pratt, 2001). Learner – learner interaction form would increase understanding of the course content and encourage critical thinking among learners. Also collaborative work may eliminate feeling of isolation and promote team work spirit within learners in digital educational games.

4. Improving PA interactivity: From PA techniques to tutor and tutee agents’ techniques This paper has argued strongly that pedagogical agents can provide or create motivations and emotions in DEG in order to infer learning. In order to achieve this we suggest an approach that considers Pedagogical agent techniques as Tutor and Tutee agent techniques to create a high social and collaborative DEG environment. The Tutor Agent (TrA) is engineered to interact directly with a learner and explicitly guide him/her through the domain. These pedagogical agents are applicable in teaching components and user interfaces. They encourage the learner by providing feedback within the learning environment. The Tutee Agent (TeA), similar to Pedagogical Agent as Learning Companions (PALs), serves as a Peer‐Mediated Learning Agent, this type of agents are integrated into the user interface to act as interactive partners in a learning his process. For a long time, educators have observed the benefits of peer interaction and implemented systematic peer‐mediated interventions to increase achievements and motivation. Peer‐ mediated learning (also called peer‐assisted learning) refers to learning through interactive help and support among peers (K. Topping & Ehly, 1998). The peer agent competency level is model in such a way that the peer agent intellectuality is a little bit higher than that of the learner, with these they can share information and collaborate with each other in a game based learning environment. With the combination of TrA and TeA in DEG would not only affected leaner‘s motivation and emotional state but would provide a fruitful and powerful way of enhancing learning practices by eliminating the problem of isolation and promoting team work spirit among learner’s in a digital learning environment (see Figure 2).

Figure 2: Improved pedagogical agent techniques: DEG hosting TrA and TeA

5. Discussion When we critical consider already existing DEG with Pedagogical agent functions we have every reason to believe that they are still some limitations. The most common among them is the absence of learner’s companion and lack of social interaction in the games. This limitation could promote isolation in game based

550

Ogar Ofut Tumenayu and Olga Shabalina learning. We have decided to diversify the functionality of a Pedagogical agent by introducing a Sub‐agent (TeA) as a learner’s companion. While the main agent (TrA) is engage in tracking the learner’s activities and inferring learner state, the TeA initiate interactions with the learner at the appropriate times and in appropriate manners. For instances, we want to enable an agent to intervene proactively when the learner appears to be confused or stuck. This kind of intervention would be better handle by TeA because of the type of interaction (Learner‐learner interaction) that exists between them. A proactive intervention is very important because sometimes the learner may feel reluctant to ask for help directly from the TrA. If the learner in any point in time have difficulty understanding the game, it may be appropriate for the TeA to interrupt and offer assistance to the learner. The competence of the TeA is in every level of the game higher than that of the learner, in order to continuously monitor the learner’s activities. If the TeA does not track the learner’s activities accurately it could interrupt the learner with assistance when it is not really needed, which could disrupt the flow of the game. The social interaction between TeA and the learner would be able to influence the learner motivation and emotional state.

6. Conclusion and future work In this paper, we highlighted the importance of boosting interactivity content in DEGs by adopting pedagogical agent to heighten motivation and emotional state for game‐based learning to become widely acceptable in classroom teaching and learning. DEGs have long proved to enhance learning in a digital environment. By creating a real‐classroom situation in a safe challenging environment, a DEG with learning companion agent would contribute immensely to knowledge, and skills acquisition through an attractive and motivational learning interface. The introduction of tutee agent when combined with the social interaction abilities of a pedagogical agent in DEG would promote more than just individual acquisition of knowledge: they would provide some learning affiliations and encouragement to the learner by sharing menial tasks, and also collaborate with the learner. Furthermore, we are currently developing a game prototype to demonstrate the effectiveness of our approach. There would be need for further research to confirm the findings of studies and to identify if the optimal level of tutee agent competency would interact with learners competency. If the two modules are integrated successfully, the games would be used for self‐learning.

References Atkinson, R.K.(2002) Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology, v. 94, n.2, pp 416‐427. Ames, C. (1990) Motivation: What Teachers Need to Know. Teachers College. 91, 3, pp 409‐21. Alavi, M. (1994).Computer‐mediated collaborative learning: An empirical evaluation.MIS Quarterly, 18, pp 159‐174. Baylor, A. L., & Ryu, J. (2003). The API (Agent Persona Instrument) for assessing pedagogical agent persona. Paper presented at the International Conference of Ed‐Media, Honolulu, Hawaii. Bower, G.H. (1992) ´How might emotions affect learning?´ in Christianson, S.A. (ed.) The handbook of emotion and memory. Hillsdale, NJ: Erlbaum, pp.3‐31. Bandura, A. (2001). Social cognitive theory: An agentic perspective. Annual Review of Psychology, pp 52, 1‐26. Craig, S. D., Gholson, B., Ventura, M., Graesser, A. C., & Group, T. R.(2000). Listening in on dialogues and monologues in a virtual tutoring session: Learning and questioning. International Journal of Artificial Intelligence in Education, 11, pp 242–253. Conati, C. & Zhou, X. (2003). A probabilistic model to assess user affect from goals, personality, and context. Proc. of the Intl. Conf. On Intelligent User Interfaces. New York: ACM Press. Driscoll, M. P. (2000). Psychology of learning for instruction. Boston, MA: Allyn & Bacon. De Vicente, A. and Pain, H. (2002). Informing the detection of the students’ motivational state: An empirical study. In S.A. Cerri, G. Gouardères, & F. Paraguaçu (Eds.), Intelligent Tutoring Systems: 6th International Conference, ITS 2002, pp 933‐943. Berlin: Springer‐Verlag. Del Soldato, T. and du Boulay, B. (1995). Implementation of motivational tactics in tutoring systems. J. of Artificial Intelligence in Education 6(4), pp 337‐378. Eliseo Reategui, Eunice Polonia, and Letícia Roland (2007). The role of animated pedagogical agents in scenario‐based language e‐learning: a case‐study, Author manuscript, published in "Conference ICL2007, September pp 26 ‐28, 2007, Villach : Austria (2007)" Frijda, N. (1994).Varieties of affect: emotions and episodes, moods, and sentiments. In Ekman, P. & Davidson, R. J. (Ed.). The Nature of Emotion. New York: Oxford Univ. Press, pp 59‐67.

551

Ogar Ofut Tumenayu and Olga Shabalina Gira, L. & Viccari, R. (1998). The use of agents techniques on intelligent tutoring systems. In XVIII International Conference of the Chilean Computer Science Society, pp.76. Goodman, B., Soller, A., Linton, F., & Gaimari, R. (1998). Encouraging student reflection and articulation using a learning companion. Paper presented at the 8th International Conference on Artificial Intelligence in Education, Kobe, Japan. 9 (3‐4), pp 237‐255. Hospers, M., Kroezen, E., Nijholt, A., op den Akker, R., & Heylen, D. (2003). Developing a generic agent‐based intelligent tutoring system. In Third IEEE International Conference on Advanced Learning Technologies (ICALT'03), pp 443. Harcourt, A.H and Lang, P.J. (1995) ´The emotion probe. Studies of motivation and attention´, American Psychologist, vol. 50, no. 5, pp 372‐385. Johnson, W.L., Rickel, J.W., and Lester, J.C. (2000). Animated Pedagogical Agents: Face‐to‐Face Interaction in Interactive Learning Environments. International Journal of Artificial Intelligence in Education 11, 47‐78. Kim, Y.; Baylor, A.L. Pedagogical Agents as Learning Companions: The role of agent competency and type of interaction. ETR&D, v. 54, n. 3, pp 223‐243, 2006. Larsen, R.J. and Diener, E. (1992) ´Promises and problems with the circumplex model of emotion´, in Clark, M.S. (ed.) Review of personality and social psychology: Emotion. Newbury Park: Sage, pp 25‐59. Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, S. T., Stone, B. A., and Bhogal, R. S. (1997). The persona effect: Affective impact of animated pedagogical agents. In Proceedings of CHI '97, 359‐366. Lepper, M.R., Woolverton, M., Mumme, D., & Gurtner, J. (1993). Motivational techniques of expert human tutors: Lessons for the design of computer‐based tutors. In S.P. Lajoie and S.J. Derry (Eds.), Hillsdale, NJ: Lawrence Erlbaum Associates. Computers as cognitive tools, pp 75‐105. Maragos, K. & Grigoriadou, M. (2005). Towards the design of intelligent educational gaming systems. In proceedings of AIED workshop5, at the 12th International Conference on Artificial Intelligence in education, Amsterdam, pp 35‐38. Meece, J. & McColskey, W. (2001) Improving Student Motivation. Serve. Available at: http://www.serve.org/Products/ProdPub.php. Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer‐based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), pp 177‐213. Nkambou, R. (2006). Towards affective intelligent tutoring system. In Workshop on Motivational and Affective Issues in ITS. 8th International Conference on ITS 2006, pp. 5‐12. Palloff, R. M., & Pratt, K. (2001). Lessons from the cyberspace classroom: The realities of online teaching. San Francisco: Jossey‐Bass. Peter, C. and Herbon, A. (2006) ´Emotion representation and physiology assignments in digital systems, Interacting with Computers, vol. 18, no. 2, pp 139‐170. Russell, J.A. (1980) A circumplex model of affect, Journal of Personality and Social Psychology, vol. 39, no. 6, pp 1161‐1179. Revelle, W. and Loftus, D. (1992) The implications of arousal effects for the study of affect and memory, in Christianson, S.A. (ed.) Handbook of emotion and memory. Hillsdale, NJ: Erlbaum, pp 113‐150. Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. Cambridge: Cambridge University Press. Scherer, K. (2000) Psychological models of emotion. In J. Borod (Ed.) The neuropsychology of emotion. Oxford/New York: Oxford University Press, pp 137‐162 Salomon, G. (1990). Cognitive effects with and of computer technology. Communication Research, 17(1), pp 26‐44. Schunk, D. H. (1987). Peer models and children's behavioral change. Review of Educational Research, 57(2), pp 149‐174. Topping, K., & Ehly, S. (Eds.). (1998). Peer‐assisted learning. Mahwah, NJ: Lawrence Erlbaum Associates.

552

Adapting the Complexity Level of a Serious Game to the Proficiency of Players Herre van Oostendorp1, Erik van der Spek2 and Jeroen Linssen3 1 Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands 2 Department of Industrial Design, Eindhoven University of Technology, Eindhoven, The Netherlands 3 Department of Human Media Interaction, University of Twente, Enschede, The Netherlands H.vanOostendorp@uu.nl e.d.v.d.spek@tue.nl j.m.linssen@utwente.nl Abstract: As games are continuously assessing the player, this assessment can be used to adapt the complexity of a game to the proficiency of the player in real time. We performed an experiment to examine the role of dynamic adaptation. In one condition, participants played a version of our serious game for triage training that automatically adapted the complexity level of the presented cases to how well the participant scored previously. Participants in the control condition played a version of the game with no adaptation. The adapted version was significantly more efficient and resulted in higher learning gains per instructional case, but did not lead to a difference in engagement. Adapting games to the proficiency of the player could make serious games more efficient learning tools. Keywords: serious game, proficiency, dynamic adaptation, learning efficiency, engagement

1. Introduction Serious games can be used to engender learning in a player, and two recent meta‐analyses have shown that the usage of serious games may even lead to superior learning compared with traditional (but passive) instructional methods (Sitzmann, 2011; Wouters et al, 2013). However a serious game is found to be primarily efficacious if a person is allowed to play the game multiple times (Sitzmann, 2011), a result that Wouters and Van Oostendorp (2013) argue underlines the notion that games are complex environments in which the player first has to learn how to control the game and the way in which it conveys the instructional material, before this material itself can be learned. Games in turn are products that have to be made beforehand and have a preset pace, and often do not take into account the individual learning rate. People learn at different speeds, which may lead to a number of problems. Firstly, the rich multimodal information of a game may overload the limited working memory capacity of a player, leading to incorrect learning (Moreno & Mayer, 2007), and some learners will therefore benefit from a slower pace in the presentation of instructional material in order to correctly organize all the new information that is coming in. Conversely, efficient learning may also be hindered by cognitive underload, where the learner is stimulated too little, for instance when a quick learner plays a game that has a slow pace in order to accommodate slow learners. Cognitive underload can lead to (passive) fatigue, which has been shown to result in disengagement from the task and higher distractibility and can subsequently degrade performance (Saxby et al, 2007; Paas, Renkl & Sweller, 2004). If a game were to actively prevent the player from becoming cognitively overloaded or underloaded, it could therefore both be more efficient. Secondly and closely related to this, Csikszentmihalyi (1988) posited that one can experience the feeling of flow, which is a feeling where someone is completely engaged in an activity to the point of losing self‐ consciousness and the activity becomes rewarding in its own right, and that this leads to the individual functioning at his or her fullest capacity (Shernoff et al, 2003). This is achieved when the provided challenge is optimally suited to the skills of the user; and as videogames are often stated to be engaging, with players reporting an experience of being completely absorbed in the game, they seem to be ideally suited to produce flow (Sweetser & Wyeth, 2005; Cowley et al, 2008). Flow has been shown to be positively correlated to learning (Webster, Trevino & Ryan, 1993); therefore, keeping players in a sense of flow by adjusting the challenge to their skills could improve learning (Liu et al, 2011).

553

Herre van Oostendorp, Erik van der Spek and Jeroen Linssen

Summarizing, if quick learners were able to progress in the game at a faster pace, for instance because the game recognizes their proficiency and adapts the game accordingly, engagement in performing the task could be enhanced which in turn results in a higher efficiency of the game. Similarly, a slower pace for slower learners would also improve engagement and efficiency for them. In this paper we will examine in an experimental study whether adapting a serious game to the proficiency of players improves learning and engagement. But first we will discuss in next section different aspects of adaptivity in general, how we monitored or assessed proficiency of players and how we implemented adaptivity in a dynamic way in the serious game Code Red Triage.

2. Adaptivity 2.1 Aspects of adaptivity In line with Lopes and Bidarra (2011), we can distinguish several components of adaptation: 1) the game world and its objects can be varied; e.g. the layout of the game world can be made simpler for underachieving players (Walker, 2009); 2) the game play mechanics; e.g. adjusting shooting difficulty by providing player aim assistance, according to individual skills (Brathwaite & Schreiber, 2009); 3) adapting the attributes of the non‐ player characters in the game; e.g. when domain knowledge is automatically gathered by the game, based on Artificial Intelligence‐techniques, in order to offer more challenging behavior of the non‐player characters (Bakkes, Spronck & van den Herik, 2009); 4) game narratives; e.g. adapting the sequence of events to the pace or behavior of the player (Roberts & Isbell, 2008); and 5) game scenarios ‐ more or less similar to the previous one; adapting the flow of events and actions within a game. For instance, monitoring the players actions and based on that certain points in the plot are included in the game (or not) (Niehaus & Riedl, 2009). A next issue in creating adaptive games is to decide on the method of generating the content. Lopes and Bidarra distinguish two general methods. First, offline adaptivity (or customized content generation); adjustments are made considering player‐dependent data, but prior to initiating the gameplay. Secondly, online adaptivity, i.e. adjusting the game to its players, in real time, as they play. A further discussion on the way adaptation can be implemented in games and the associated challenges can, for instance, be found in Lopes and Bidarra (2011). Though in the (game) industry and academia now many different adaptive (serious) games are developed, and progress has been made, empirical research to effects of adaptivity in terms of learning and engagement are still scarce. In this paper we will remedy this and present results of an empirical study on the learning and affective effects of a game with dynamic adaptivity. That is, a game where the challenges of, or difficulties caused by, the game are increasing, and at a rate dependent on the proficiency of the player (online adaptation). We will mainly be concerned with varying the attributes of the non‐player characters.

2.2 Assessing the proficiency of players For the principle of fitting the instruction to the learner's proficiency level to be implemented in serious games effectively, it is important first that the proficiency should be assessed and secondly that the challenge should be adapted to the player automatically in a non‐obtrusive way. Automatically assessing and adapting the challenge or difficulty of a game to the proficiency of a player is slowly becoming commonplace in entertainment games. For instance in Guitar Hero 5, a musical instrument simulation game, the player needs to hit the correct notes of a song with good timing. The game adds more notes and places a greater emphasis on timing when the player performs well, or vice versa when the player performs badly. Racing games like Mario Kart and Need for Speed, implement a simple adaptation known as ‘rubber banding’: when the player lags behind the other racing contestants, they will slow down in order to let the player catch up with them – when the player is up front, his opponents will become faster and try to keep up with him. Here, we will elaborate on two modes of assessing that are most relevant to our research. Firstly, one interesting avenue in which a game can be adapted to the player was undertaken by Yun et al (2009), who used an infrared camera that was mounted on a TV displaying the game. This camera (overtly) recorded the faces of the participants while they were playing a game that revolved around shooting enemy robots. Looking at the heat signatures from the supra‐orbital region of the face, they were able to derive how much apparent stress the game exerted on the player during game play. At the same time, the player reported at set intervals whether they found the game too easy, just right or too difficult, and whether they were enjoying the game or

554

Herre van Oostendorp, Erik van der Spek and Jeroen Linssen

would like to quit. This research is relevant to our own for two reasons. One, they discovered that people who found the game too difficult and wanted to quit actually had lower stress levels than when the game was moderately difficult. They argued that this is due to the player becoming disengaged with the game, thereby corroborating the previously made assertion that too high a challenge leads to cognitive overload and is detrimental to the engagement or flow experience. Two, a version of the game where the game automatically assesses and adapts to the stress level of the player was shown to lead to higher engagement and better in‐ game performance (in terms of how many robots were defeated) than in conditions with preset difficulty levels, even for the easy difficulty level. Another interesting example of how to adapt the game to the player is the entertainment game The Elder Scrolls 4: Oblivion. Here, the player roleplays a character in a large and open medieval fantasy world. As the player encounters new locales, performs quests and defeats monsters, his or her character will gradually become stronger and gain better weapons and items (see further Shute et al, 2009). Because the game features an open world for the player to explore freely, this traditionally leads to problems where the player may encounter monsters that are far too strong for his or her avatar to defeat at that point in time. To counter this and provide the optimal experience for everyone, the player’s adversaries in the game also progress in power at the same rate as the skill level of the player. Contrarily to what would be expected, many gamers criticized this feature, as it made them feel that their actions were largely inconsequential (Bostan & Ogut, 2009); they were not getting stronger than their enemies and therefore they didn’t feel like they were mastering the game. Above we mentioned two different techniques of assessing the player proficiency within the game. The first was a more overt technique, where in real life settings the player would have to install an infrared camera for it to work; the second example featured so‐called ‘stealth’ assessment (Shute et al, 2009), that is, a more covert assessment that is coupled to the naturally occurring moves of the player in the game. In essence, all games are an assessment device, in that progressing past an obstacle is contingent on acquiring the needed knowledge of how to do so. As digital games are played on computers, which require that every game rule and in‐game problem encountered is computable, determining whether the player succeeded is often easily quantifiable.

2.3 Dynamic adaptivity in the serious game Code Red Triage As indicated we want to study whether the online adaptation of the challenge or difficulty of a learning experience to the proficiency of players, improves learning and enhances engagement. Following Bailey and Katchabaw (2005) we use the term dynamic adaptivity to designate online adaptation of game experiences in terms of complexity and matching that to the proficiency of players. In order to test this hypothesis we used the serious game Code Red Triage, a total conversion mod of Half‐Life 2 (Van der Spek, Wouters & Van Oostendorp, 2011). The game is designed to teach the triage procedure, a procedure for medical first responders to prioritize the victims of a mass casualty event according to how urgently the victim needs medical attention. The mobility (sieve) triage taught here is a relatively simple procedure, where it takes the first responder between one and five steps to determine the severity of the victim’s injuries. When the game starts, the player finds himself in an empty train station with signs of recent panic. Here, he learns that he is a medical first responder who has received a call that a bomb has gone off on a subway platform. The player is then told to find the subway platform and perform the triage procedure on the victims. Upon reaching the subway platform, a visible timer starts counting down from seventeen minutes. When the timer reaches zero, the game ends. This timer was added to instill a sense of immediacy and stress; in practice almost every participant is able to triage all victims comfortably within this time. At the subway platform, the player can then walk up to a victim and press a button to enter the triage menu, which consists of eight buttons for triage actions, and four buttons for the four different triage categories (see Figure 1). Pressing a triage button will give a few lines of general information on what the action entails and approximately at what stage in the procedure it should be used, and a line with specific information on how the action affected the victim the player’s looking at. After choosing a few triage actions the player should be able to have an idea how heavily injured the victim is and assign a triage category. Once this is done, the victim changes color to depict the chosen category and the player receives a score showing how well he did, as well as a few lines telling him whether or not he forgot to take procedure steps, took steps in the wrong order, took unnecessary steps and whether it was done within the allotted time (between 10 and 55 seconds), see Figure 2 for a screenshot. The in‐game score that can be obtained per victim ranges from 0 to 100 and is based on the previous four criteria.

555

Herre van Oostendorp, Erik van der Spek and Jeroen Linssen

Figure 1: Triage menu in the game Figure 2: Feedback after categorizing a victim in a triage category In the case of Code Red Triage, we already have a measure to assess how well the player is performing in the game, namely the in‐game score, which provides us with an objective measure of whether the player is able to correctly apply the procedure to a given victim case. The player’s performance can therefore be seen as an indication of their proficiency level. We can thus use the above mentioned covert method to assess the proficiency of players here. We used this in‐game score to adapt the difficulty of the game to the proficiency of the player. In Code Red Triage, there are a total of six paths with an increasing number of steps in the triage procedure that are taught with the game, but there are multiple victims for any given path. As the victims are encountered in increasing order of complexity (i.e. the number of steps needed to come to a correct categorization), these groups of victims are called ‘victim tiers’. In the set of victims 6 tiers or levels of complexity were distinguished. In other words, the attributes of the non‐player characters were varied in complexity. If a player scores above a preset threshold, he or she has proven to have a certain level of proficiency and can move on to a more complex victim tier. In the adaptive condition of Code Red Triage this was operationalized as the game deleting all remaining victim cases within the same tier, if the player scored higher than a threshold value for that victim. The threshold was determined with the data from a pilot experiment, by rounding up the average score per victim tier. A player who was unable to triage a victim case and scored below the threshold, received one or more of the remaining cases of that tier before going to the next level of complexity. In other words, more successful players could attain the most complex case in less cases, and consequently learn to perform the triage more efficiently. In the control version of the game all (19) cases were presented in a gradually increasing complexity. We hypothesize that players feel more engaged by the dynamic adaptive version, because the game always remains challenging (compared to a control version), and secondly we expect in the dynamic adaptive version of the game that players are able to learn more efficiently, because redundant learning experiences (triage cases) can be skipped.

3. Method 3.1 Participants In total 28 individuals of university‐level education, 19 male and 9 female, participated in the experiment, and were randomly assigned to the adaptive game condition (n=14), and the control condition (n=14). Average age was 22.86 with a standard deviation of 5.68.

556

Herre van Oostendorp, Erik van der Spek and Jeroen Linssen

3.2 Materials To measure the learning of players, three types of instruments were used. The in‐game score formed the first measure: an indication of the progression of the player in the game. Statistics from the game that were logged furthermore included triaged victims, number of triaged victims, tier of victim, time per victim, total time, score per victim and total score. Second and third, we measured how much a participant learned in the game with two measures: a pen‐and‐paper knowledge test and a structural knowledge assessment. The knowledge test was in the form of eight verbal and eight pictorial multiple choice questions where the player had to answer questions related to the triage procedure by choosing one of four alternatives (total score range 0‐16). Whereas the knowledge test measured how well the participant could reproduce declarative knowledge, the structural knowledge assessment determined how the information was organized on a deeper, more structural level. Here, a computer program called PCKNOT was used, that let participants rate the degree of relatedness of pairs of concepts from the triage procedure. These ratings could subsequently be used to elicit a participant’s knowledge structure with the Pathfinder metric (Schvaneveldt, Durso & Dearholt, 1985) and compared to the knowledge structure of experts; resulting in a similarity measure that indicated how well the participant had organized the information of the triage procedure structurally (Gomez, Hadfield & Housner, 1996). The score range varies from ‐1 through 0 to +1. Pathfinder has been successfully by Day, Arthur Jr. and Gettman (1994) to measure learning from a complex videogame, and found that it was also predictive of skill retention and skill transfer. For further information see Wouters, Van der Spek and Van Oostendorp (2011). In our case we focused on 8 important concepts from the triage procedure and consequently 28 pairs were presented for the related judgments. The created networks were compared with the referent structure that was derived by averaging the elicited knowledge structures of the current researchers. The engagement of players was measured by using the subscale of the ITC Sense of Presence Inventory (ITC‐ SOPI), which indicates the participant’s feelings of engagement with a twelve item five‐point Likert scale (Lessiter et al, 2001). If the challenge of the game is better adjusted to the abilities of the player, one would expect the player to be drawn into the game more, which we hoped to see expressed in the scores on this subscale. The reliability of the ITC‐SOPI Engagement questionnaire appeared to be relatively low, Cronbach's coefficient α = 0.59.

3.3 Apparatus and procedure The game was played on a 17” laptop at a resolution of 1920 x 1200 with circum‐aural headphones in a room with the lights turned off. The graphics settings were set at their maximum and the game ran at a constant 60 frames per second. The participants were asked to perform the structural knowledge assessment with the PCKNOT software. Then, the knowledge test was administered. Before playing the game, the participants were given instructions about Code Red Triage and were informed about its goal. Nothing was revealed to them about the condition they took part in. Playing the game from start to finish took each participant at most 25 minutes: a few minutes for the entry level, a few more for the hallway part and a maximum of 17 minutes was allowed for the metro platform part, in which the triages took place. The scores participants reached in the game gave information about their performance (see also section 2.3). Directly after the participants finished playing the game, they were asked to fill out the engagement questionnaire. They were then asked to do the structural knowledge assessment and knowledge test as before, but with the questions in a different order. Finally, the participants were thanked for their cooperation and they received a coupon for their work. An overview of the procedure can be seen in Figure 3.

Figure 3: Procedure of the experiment

4. Results Engagement The mean scores and standard deviations of the engagement questionnaire are mentioned in Table 1. An ANOVA showed no significant effect of condition on the ITC‐SOPI engagement questionnaire, F(1,26) < 1.

557

Herre van Oostendorp, Erik van der Spek and Jeroen Linssen

Learning Efficiency There are several ways to determine whether learning was more efficient in the adaptive condition. A reliable measure for efficiency is to divide the posttest scores of the participants by the number of victim cases triaged, giving us an indication of how much the participant has learned per unit of instruction, and whether this would be higher in a game that adapts the information presentation to the player’s proficiency. Another way would be to divide learning performance by total time spent playing the game. However some players navigate more efficiently than others towards the platforms etc, which blurs what we want to measure. We therefore decided to use learning performance divided by the number of cases triaged, as a purer measure of learning efficiency. An ANCOVA with the pretest as covariate, condition as fixed factor and posttest score divided by the total number of victims triaged as dependent variable showed that condition had a significant effect on both the knowledge test (F(1,25) = 21.98, p < .001, d = 1.81) and the structural knowledge assessment (F(1,25) = 5.05, p < .05, d = .89). The means on these relative measures and standard deviations of these tests are listed in Table 1. Table 1: Mean engagement and efficiency scores on knowledge test and structural knowledge assessment (sd).

Control Condition

Adaptive Condition

Engagement (1‐5)

3.63 (.33)

3.66 (.45)

Knowledge test

.57 (.19)

1.02 (.30)

Structural knowledge assessment

.015 (.004)

.028 (.019)

In‐game score The total in‐game score was significantly higher for the control condition (M = 777.7, SD = 321.2) than for the adaptive condition (M = 316.4, SD = 107.8), F(1,26) = 25.95, p < .001, however this more or less follows from the result that participants triaged significantly less victims in the adaptive condition.

5. Conclusion and discussion We hypothesized that a serious game that dynamically adapts its challenge, or complexity presentation, to quick learners could make a serious game more engaging and more efficient. The first part of the hypothesis was not confirmed, while the second part was confirmed; participants in the adaptive game version learned significantly more per victim case than in the control condition, and were therefore more efficient. We found no difference in the engagement ratings. If the improved learning per unit of instruction was due to less disengagement from the task, one would expect this to appear from the results of the engagement questionnaire. We propose four explanations why we did not find a difference in engagement. Firstly, when participants had to appraise their engagement just after playing the game, they lacked knowledge of the other condition and thereby a reference point. The intervention itself may be too small next to all the other determinants of engagement, such as the game’s setting, world, expectations, control interface, et cetera, to show up as a difference on the rating scale, but the adaptive version may still be preferred when the conditions were placed side by side. A second explanation could be related to the fact that we only asked participants to appraise their engagement after the game. It is unclear whether a continuous measurement of a participant’s engagement, for instance with an infrared camera as in the research by Yun et al (2009), as we mentioned in the introduction, would have resulted in higher ratings throughout the game in the adaptive version. Thirdly, people may play games for different reasons; a higher challenge could lead to higher engagement in some players, whereas it has the opposite effect on others. Lastly, and perhaps as a result of the previous explanation, we found that the homogeneity of the engagement questionnaire (Cronbach’s alpha) was low. Perhaps this measurement problem contributed to the fact that we did not find an effect of engagement. We saw that participants learned more per victim case in the adaptive condition compared to the control condition. It could be that the moment a participant grasps the procedure to resolve a victim case pertaining to a certain tier, the information presented in the following victims in that tier is redundant, at least to a point that it does not improve learning of the procedure anymore, making the adaptive version more efficient.

558

Herre van Oostendorp, Erik van der Spek and Jeroen Linssen

In order to determine whether the adaptive condition not only made learning the instructional material more efficient, but also leads to deeper learning (Graesser et al, 2009), other experiments should be set up such as e.g. a study where learning is also measured after a longer delay or with transfer tasks. One last observation concerns the relation between engagement and learning; the results found indicate that an increase in engagement does not seem necessary to enhance learning efficiency. Also the correlation between engagement and learning efficiency appeared to be low and not significant (p > .05) for both groups of participants. However for this finding too, the same remarks as before should be made concerning the measured engagement of players. All in all, a rather simple alteration of a serious game where it dynamically adapts the presentation of complexity to the player’s performance and thereby its challenge has been shown to markedly improve the efficiency thereof. This is a promising result for serious games developers that worry about the comparative efficiency of their game, as well as for researchers interested in improving games with the aid of more sophisticated adaptation engines. It can also be a useful result for entertainment game developers, as many games need to incorporate tutorial levels that are necessary for players to understand the game, but are not a lot of fun to play, especially upon repeated playthroughs. A dynamic adaptive version that adapts to the player's proficiency could greatly speed up these mandatory instructional sequences and (possibly) make them more challenging.

5.1 Future research Above we already mentioned two limitations to our study, viz. that it is impossible to conclusively state whether dynamically adapting to the player’s performance only resulted in more efficient instruction, or also in deeper learning, and that it is unclear whether participants differed in engagement during gameplay. In addition, another limitation of our experimental setup that warrants future research is that we did not measure retention over longer time periods. Participants in the adaptive game version received less practice and consequently less opportunity to internalize the information. Therefore there is a real possibility ‐ or even danger ‐ that the participants in the dynamic adaptive condition remember less of the instruction after several weeks. Regarding dynamic adaptation itself, in this study we did make some specific choices during the design and implementation process. We focused on the nature of the non‐player characters and let them vary in number of steps needed to perform a correct triage. Several alternatives are open for continued research to the role of dynamic adaptation. For instance, the set of buttons for executing the triage actions could be adapted, that is, starting simple and increasing over time, depending on performance. Or the feedback given to players could be adapted, e.g. stating more or less explicitly what went right and what went wrong during performing the triage. Finally, another option for making training procedures adapt themselves to participants is the notion of adaptability. In this form of offline adaptation participants indicate themselves what direction they want to practice and what part of the procedure they want to repeat. These are questions that still need to be examined in the future.

Acknowledgements This research has been supported by the GATE project (http://gate.gameresearch.nl/), funded by the Netherlands Organization for Scientific Research (NWO).

References Bailey, C. and Katchabaw, M. (2005) An experimental testbed to enable auto‐dynamical difficulty in modern video games. In Proceedings of the 2005 GameOn North America Conference (pp.18‐22). Bakkes, S. Spronck, P. and van den Herik, J. (2009) Rapid and reliable adaptation of videogame AI. IEEE Transactions Computational Intelligence and AI in Games, 1(2), 93‐104. Bostan, B. and Ogut, S. (2009) Game challenges and difficulty levels: lessons learned From RPGs. In International Simulation and Gaming Association Conference. Retrieved from http://www.silentblade.com/presentations/Bostan_Ogut_Full_Paper.pdf Brathwaite, B. and Schreiber, I. (2009) Challenges for game designers. Boston, MA: Charles River media. Cowley, B., Charles, D., Black, M. and Hickey, R. (2008) Toward an understanding of flow in video games. Computers in Entertainment 6(2), article 20, 1‐27

559

Herre van Oostendorp, Erik van der Spek and Jeroen Linssen

Csikszentmihalyi, M. (1988) The flow experience and its significance for human psychology. In M. Csikszentmihalyi & I.S. Csikszentmihalyi (Eds.), Optimal experience: Psychological studies of flow in consciousness (pp. 15‐35). Cambridge, MA: Cambridge University Press. Day, E. A., Arthur Jr., W. and Gettman, D. (2001) Structural knowledge assessment: Comparison of referent structures. Journal of Educational Psychology, 86(5), 1022‐1033. Gomez, R. L., Hadfield, O. D. and Housner, L. D. (1996) Conceptual maps and simulated teaching episodes as indicators of competence in teaching elementary mathematics. Journal of Educational Psychology, 88(3), 572‐585. Graesser, A.C., Chipman, P., Leeming, F. and Biedenbach, S. (2009) Deep learning and emotion in serious games. In U. Ritterfeld, M. Cody & P. Vorderer (Eds.), Serious Games: Mechanisms and Effects (pp. 81‐100). New York: Routledge. Guitar Hero 5 [Computer software]. Santa Monica, CA: Activision Blizzard. Half‐Life 2 [Computer software]. Bellevue, WA: Valve Corporation. Lessiter, J., Freeman, J., Keogh, E. and Davidoff, J. (2001) A cross‐media presence questionnaire: The ITC‐sense of presence inventory. Presence: Teleoperators and Virtual Environments, 10(3), 282‐297. Liu, M., Horton, L., Olmanson, J. and Toprac, P. (2011) A study of learning and motivation in a new media enriched environment for middle school science. Education Technology Research Development, 59, 249‐265. Lopes, R. and Bidarra, R. (2011) Adaptivity challenges in games and simulations: a survey. IEEE Transactions on Computational Intelligence and AI in Games 3(2), 85‐99. Mario Kart [Computer software]. Kyoto, Japan: Nintendo. Moreno, R. and Mayer, R. (2007) Interactive multimodal learning environments. Educational Psychology Review, 19(3), 309‐326. Need for Speed [Computer software]. Redwood City, CA: Electronic Arts. Niehaus, J. and Riedl, M.O. (2009) Scenario adaptation: an approach to customizing computer‐based training games and simulation. In Proceedings AIED Workshop Intell Edu. Game. Brighton, UK (pp. 89‐98). Paas, F., Renkl, A. and Sweller, J. (2004) Cognitive load theory: Instructional implications of the interaction between information structures and cognitive architecture. Instructional Science, 32(1), 1‐8. PCKNOT [Computer Software]. Las Cruces, NM: Interlink Inc. Roberts, D.L. and Isbell, C.L. (2008) A survey and qualitative analysis of recent advances in drama management. Int. Trans. Sci. Appli., 3(2), 61‐75. Saxby, D. J., Matthews, G., Hitchcock, E. and Warm, J. S. (2007) Development of active and passive fatigue manipulations using a driving simulator. In Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting (pp. 1237‐1241). Santa Monica, CA: Human Factors and Ergonomics Society. Schvaneveldt, R. W., Durso, F. T. and Dearholt, D. W. (1985) Pathfinder: Scaling with network structures (Memorandum in Computer and Cognitive Science, MCCS‐85‐9, Computing Research Laboratory). Las Cruses: New Mexico State University. Shernoff, D. J., Csikszentmihalyi, M., Shneider, B. and Shernoff, E. S. (2003) Student engagement in high school classrooms from the perspective of flow theory. School Psychology Quarterly, 18(2), 158‐176. Shute, V. J., Ventura, M., Bauer, M. I. and Zapata‐Rivera, D. (2009) Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow. In U. Ritterfeld, M. Cody, & P. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295‐321). New York, NY: Routledge. Sitzmann, T. (2011) A meta‐analytic examination of the instructional effectiveness of computer‐based simulation games. Personnel Psychology, 489‐528. Sweetser P. and Wyeth, P. (2005) GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment, 3(3), 1‐24. The Elder Scrolls 4: Oblivion [Computer software]. Rockville, MD: Bethesda. Van der Spek, E. D., Wouters, P. and Van Oostendorp, H. (2011) Code Red: Triage Or COgnition‐based DEsign Rules Enhancing Decisionmaking TRaining In A Game Environment. British Journal of Educational Technology, 42(3), 441‐ 455. Walker, J. (2009) Left 4 Dead 2: Exclusive RPS Hand on Preview. Rock Paper Shotgun, PCGaming since 1873. Online available: http://www.rockpapershotgun.com/2009/06/01left‐4‐dead‐2‐exclusive‐rps‐preview Webster, J., Trevino, L. K. and Ryan, L. (1993) The dimensionality and correlates of flow in human‐computer interactions. Computers in Human Behavior, 9(4), 411‐426. Wouters, P.J.M., Van der Spek, E.D. and Van Oostendorp, H. (2011) Measuring learning in serious games: a case study with structural assessment. Educational technology research and development, 59(6), 741‐763. Wouters, P.J.M., & Van Oostendorp, H. (2013) A meta‐analytic review of the role of instructional support in game‐based learning. Computers & Education, 60, 412‐425 Wouters, P.J.M., Van Nimwegen, C., Van Oostendorp, H. and Van der Spek, E.D. (2013) A Meta‐analysis of the Cognitive and Motivational Effects of Serious Games. Journal of Educational Psychology, 105, 249‐265. Yun, C., Shastri, D., Pavlidis, I. and Deng, Z. (2009) O' game, can you feel my frustration?: Improving user's gaming experience via stresscam. In Proceedings of the 27th international conference on Human factors in computing systems (CHI '09) (pp. 2195‐2204). New York: ACM.

560

Designing Casual Serious Games in Science Ayelet Weizman Snunit center for the advancement of web‐based learning, The Hebrew University, Jerusalem, Israel ayeletw@snunit.org.il Abstract: One of the problems in science education is students’ low motivation and difficulty to see the connection between science learning and their daily life. In addition, many science concepts are not intuitive, and involve complicated relations between parameters. We previously presented (Weizman & Broza, 2012) evaluation of a model integrating digital educational games with interactive learning units, where we found that students reported experiencing both learning and enjoyment. The current study focused on games in science, and examined the relation between the game design parameters and the learning experience based on the Flow model. We define these games as casual‐serious, since they require short playing time and relatively low budget to develop, while based on educational purposes. The goal of educational games is to give players the right balance between skill and challenge that will not be too difficult or too easy, enabling a flow experience. The research goal was to investigate the relation between the game design parameters and the learning experience. We assumed that by changing the game parameters the relation between challenge and skill will be influenced, as expressed by the flow experience, which is related to the learning experience. We studied the design process of two games in science, aiming to teach complicated concepts, like the idea that good health is based on balanced nutrition combined with physical activity. The influence of game parameters on students’ learning experience was investigated through a pilot process in which each game was piloted in a primary school class, calibrated and piloted again. Flow was evaluated using online questionnaires. We found a significant change in the challenge to boredom rate between the three phases of the pilot. Students’ responds indicate assimilation of social practices and understanding of the relation between parameters. Findings show that careful design of game parameters influences the flow experience as well as the learning experience and understanding of content goals and social values. We conclude that casual‐serious games designed according to the described design principles may be useful in science teaching, since they provide an enjoyable way to deal with complicated concepts involving relations between parameters. By balancing game parameters it is possible to create a flow experience and lead to meaningful learning of science content and skills. Keywords: serious‐games, educational‐games, flow, science‐learning, game‐based learning, casual‐games

1. Introduction 1.1 Games and motivation According to OECD data, about 60% of the students in the USA, Canada and many countries in Europe think that school is boring. Moreover, studies show that one of the reasons for low achievements in science is negative attitudes towards science in school (Osborne, Simon, & Collins, 2003). One of the reasons for low motivation in school is that students seldom see the relation of school science to everyday life, especially when the topics are abstract or complicated (Foster, 2008). Computer games, on the other hand, are one of the favorite activities that teenagers choose to spend their time on. Moreover, while playing digital games youths often seem to be fully engaged in the experience and unaware of their environment as in the flow theory (Csikszentmihalys, 1975).

1.2 Flow and game‐based learning in science The state of flow was defined by Csikszentmihalys (1975) as a balance between challenge and skills, or anxiety and boredom. Originally it referred to activities like rock climbing, music composing or dancing, but later the model was extended to other contexts including digital environments. One of the implications of flow experience in digital environments is increased learning (Webster, Trevino, & Ryan 1993). A recent report (Clark et al., 2013) evaluated three aspects of learning: cognitive, intra‐personal and inter‐personal in previous studies of game‐based learning, and found that using games in science learning may improve cognitive competencies compared to learning in traditional methods. Another finding shows that the effectiveness of game influence on learning depends on game design. The challenge for game developers is to design the game parameters in a way that will enhance flow, while increasing the level of challenge according to the increase in skills, in a way that will keep the player in the flow zone all along the playing time.

561

Ayelet Weizman Kiili (2005) suggested a framework for flow in computer‐mediated environments referring to three stages of flow: Flow antecedents, Flow state and Flow consequences (Table 1). Educational‐game developers should take into account all the components comprising the flow framework. Table 1: Kiili’s (2005) three stage game flow model Game flow antecedents Game flow state Flow consequences Challenge to Skill balance Concentration Content learning gains Playability Sense of control Exploratory behaviors Gamefulness Time distortion Clear Goals Loss of self‐consciousness Immediate Feedback Autotelic experience Frame story

1.3 Design principles for educational games The main design principle for educational games was formulated by Gee (2003, 2005): in order to achieve meaningful learning game goals must be aligned with learning goals. This principle is the key for designing a game that is educational and enjoying at the same time. When goals are truly aligned the motivation to learn will be internal and unite with the motivation to play. Another principle for educational games is integrating the learning content in the game (and not stopping the game flow or taking a break in order to teach something). In a simulation‐game, which is a common genre in science games, the factors in the game must correlate with a scientific model, and represent the relations between them. In addition, success in the game cannot be coincidental, but should depend on knowledge of content or skills that are relevant to the learning goals. The potential in the influence of digital educational games on motivation, engagement and learning of science content and skills was investigated in several studies (e.g Kiili, 2005, Zheng, 2012, Pavlas, 2010), that found a th positive relation in specific contexts. Zheng (2012) found that 5 class students who played a digital game in science reported on flow experience and showed significant achievements in content learning. Pavlas (2010) found significant relation between flow experience and declarative knowledge in a science game.

1.4 Casual‐serious games Serious games are usually defined as games with a goal besides entertainment (for example: PeaceMaker), and recently many serious games have educational goals. Usually these games are complicated, expensive, and require extended playing time. On the other hand, casual games are usually not identified with educational games. They are characterized as simple, easy to learn, and easy to play, and attract a different audience than serious games (Orji, Vassileva & Mandryk, 2012). But actually, if we aim to integrate digital games in school learning, isn’t it more realistic to develop casual‐serious games that will be easy for teachers to adopt, will be possible to play within a lesson time‐frame, but at the same time will be based on educational learning goals?

2. Methodology 2.1 Game design In the following sections we describe the design process of two casual‐serious games in science, dealing with the subject of “healthy and active lifestyle”, which is included in the science curriculum. The games were developed by Snunit center for the advancement of web‐based learning, in cooperation with the ministry of health, the ministry of education and the ministry of sport and culture in Israel. The games are part of a website dealing with health in the kids‐governmental portal. The games in this portal are digital educational games that can be defined as casual‐serious games since they are low‐budget and require a relatively short playing time. In previous studies (Weizman & Broza 2012) we described our model for integrating learning with enjoyment in digital games in “Al‐Hagova” website. The games were shown to create positive learning experience for children playing after school hours. The model incorporates the games with separate learning units in the

562

Ayelet Weizman website, in a way that the free‐choice learning from the game is supported by explicit learning in the interactive units. In formal contexts the mediation may be done by the teacher using the learning units or any other teaching strategies. In the current study we explore the connection between game parameters and the learning experience in two casual‐serious games in science. The research question is: How can we influence flow experience and learning experience through the design of game parameters? 2.1.1 The game “Healthy and Happy” The game deals with the balance between nutrition and sportive activities in everyday life. The learning goals include:

Developing awareness to the significance of healthy lifestyle

Developing awareness to the connection between nutrition and daily activities

Enabling decision making and problem solving in this context.

These learning goals require high order thinking skills in addition to understanding complicated concepts like the food pyramid. The game is based on the genre of caring games. The player chooses a character (boy or girl with several choices of appearance characteristics), and has to care for its nutrition according to the food pyramid, to give it enough food and drink and at least 60 minutes a day of physical activities.

Figure 1: First screen of the game “Healthy and Happy” Figure 1 shows the game’s first screen including the game main parameters: the food pyramid (according to the Israeli ministry of health), food, drink and activity parameters. The game’s timeline is one day from morning to evening, divided to five sections representing five meals. Food items can be chosen from four categories – as shown in Figure 2. Each item has a nutritious combination determined by true values of the food groups it contains. The levels of the food pyramid parameter are filled according to the player’s choices of food items. The design principle of aligning game goals with learning goals is applied here in the choice of parameters, the relations between them, and the conditions for success and failure in the game. The feedback (Figure 3) refers to specific choices of the player in the game, and describes the achieved level of each parameter in the game, so that the player can improve her gameplay next time.

563

Ayelet Weizman

Figure 2: The main screen of the game “Healthy and Happy”

Figure 3: Feedback for the first level in the game 2.1.2 The game “Couch Potatoes Defense (60 minutes)” This game deals with the significance of physical activity during daily life, as part of a healthy lifestyle. The message of the game is that you have to be active at least 60 minutes a day in order to be healthy and happy. The game aims to clearly convey that these 60 minutes can be easy and fun, and there is a large variety of activities you can do without changing your everyday routine. The game is based on the genre of defense games, and specifically inspired by the long established Tower Defense game. The "enemies" are lazy characters advancing on a conveyor belt, whose aim is to go to sleep as soon as possible, while avoiding physical activities. The player's goal is preventing them from being lazy by placing various activities in specific positions along the conveyor belt.

564

Ayelet Weizman

Figure 4: The main screen of the game “60 minutes” The principle of aligning game’s goals with learning goals is applied in this game as well, and expressed in the choice of parameters (e.g. accumulation of minutes to reach 60 minutes a day for each character, and setting defense categories that represent physical activity categories), in the relations between parameters and in the conditions of success and failure in the game (e.g. in order to win you have to combine various activity categories).

2.2 Research framework Each game was piloted and revised in a process that lasted about a month: The game was piloted in a primary school class (4th to 6th grade, about 20 students in each class), game parameters were calibrated according to the feedback, and then it was piloted again in another class. The pilots were conducted in a computer room, with children siting in couples or individuals near computers. After a short explanation they were asked to play the game about 20 minutes, and then fill an online questionnaire. Finally there was a general discussion that we recorded. Game parameters that we could calibrate include the conditions for failure in each level, the number of challenges appearing simultaneously in the game and the time intervals between them. For example, in the game “Healthy and Happy” we could change the decrease pace of the drinking parameter, thus increasing or decreasing the time pressure in the game. In the second game parameters that were varied include time spans between the appearance of characters, their speed and number. The first game “Healthy and Happy” was published (free) online on June 2012, and we continued to collect data through the online questionnaire for several months. 60 kids aged 8‐14 filled the health game questionnaires, while data from Google Analytics for the same time period shows 12,646 unique visitors who played the game 6.5 minutes on average. Therefore, for this game there are three phases: A‐ first pilot, B‐ second pilot, C‐ after publication. We used Zheng (2012) game flow survey to evaluate the flow experience. The learning experience was evaluated using the Weizman & Broza (2012) questionnaire, combined with students’ responses in the oral discussion regarding the level of learning and description of the game’s goal and what they understood from playing it.

565

Ayelet Weizman

3. Findings Figure 5 presents players’ learning experience in the first game in each of the three phases (A, B, C). The learning experience was estimated through a measure of attitudes towards four categories: Level of understanding, Interest in the game, Enjoyment and Challenge. In the challenge category there is a significant change between the three phases‐ in each phase the game is appreciated as more challenging.

Figure 5: Players’ attitudes towards the game “Healthy and Happy” Presenting the data in a “flow tunnel” graph (Figure 6) emphasizes the development in the challenge to skill rate. In phases A and B the level of challenge is similar, but the level of interest is different. It seems that the required skill level in phase B was too low in relation to the students’ skills. In phase C the challenge level increased, but not to a level of anxiety. The level of enjoyment was high in all three phases. For the second game, “60 minutes” we had only two phases of pilot so far, but there is already evidence for the influence of game parameters and their calibration on students’ flow and learning experience: In figure 7 we compare the number of students who reached each of the five levels of the game. In an ideal game we would expect a normal distribution to account for high level of Gamefulness. The results show an improvement in this aspect in the second phase. Other components of flow are compared in Table 2. All of them show improvement from phase A to phase B, yet, these are still preliminary findings and we intend to continue the analyses in the following months. Further evidence for learning can be found in students’ answers to open questions and during the open discussion. Already in the first pilots it was clear that the main message of the game was understood. In the game “Healthy and Happy” students connected between varied nutrition and physical activity to happy and healthy lifestyle. For example: “I learned in the game about nutrition, that it is important to eat varied food from all the levels of the food pyramid. I also learned it is important to be active in order to feel better.” Examples for learning from the second game: “After playing this game I know more possibilities of physical activity to do during the day”, “I learned about the importance of doing at least one hour of physical activities every day”.

4. Summary and conclusions Previous studies found a positive influence of digital games on students’ learning of science concepts in certain conditions related to game design and flow experience. We assumed that by careful design of the game parameters it will be possible to use the game environment for significance learning combined with a positive experience.

566

Ayelet Weizman

Figure 6: Challenge/skill rate in the game “Happy and Healthy” for each of the three phases, compared to the “flow tunnel” diagram (Csikszentmihalys, 1975) Table 2: Comparison of flow components in 2 phases of the pilot for the game “60 minutes” Flow component Challenge to skill balance Clear goals playability Attention focus Sense of control Loss of time and self‐ consciousness Learning new content Deepening knowledge Change of attitudes

Phase A 68% 91% 66% 77% 69% 73%

Phase B 94% 100% 88% 94% 87% 81%

29% 61% 43%

44% 69% 57%

Figure 7: The number of students who reached each level in the game “60 minutes”

567

Ayelet Weizman In this study we examined the relation between game parameters to the learning experience from the game in two games in science dealing with health that can be defined as casual serious games. We followed the design principles of aligning game goals with learning goals, and calibrated the game parameters to achieve a flow experience. In a series of pilots for the two games we found an improvement in the flow experience of primary school students that was associated with evidence for content learning and change of attitudes. This is a preliminary study that requires further research mainly in clarification of learning expectations and outcomes. These findings have implications for integrating digital games in science teaching, which we believe have a great potential for improving science learning experience in schools. Educational games enable positive experience of dealing with complicated concepts, understanding relations between parameters, training problem solving in relevant contexts, and practicing scientific skills and high order thinking skills. Using effective digital games in science teaching may develop students’ internal motivation for learning, and help in clarifying the relevance of science learning to their daily life.

References Clark, D., Tanner‐Smith, E., Killingsworth, S & Bellamy, S. (2013). Digital Games for Learning: A Systematic Review and Meta‐Analysis (Executive Summary). Menlo Park, CA: SRI International. Csikszentmihaly, M. (1975). Beyond boredom and anxiety. San Francisco: Jossey‐Bass. Csikszentmihaly, M. (1991). Flow: The psychology of optimal experience. New York: Harper Perennial. Foster A. (2008). Games and motivation to learn science: Personal identity, applicability, relevance and meaningfulness. Journal of Interactive Learning Research. 19(4): 597‐614. Gee, J. P. (2003). What video games have to teach us about learning and literacy? Palgrave Macmillian Publishing, New York. Gee, J. P. (2005). Learning by Design: good video games as learning machines. E‐Learning, Vol 2 (Number 1), p. 5‐16. Gee, J. P. (2007). Good video games + good learning: Collected essays on video games learning and literacy. PETERLANG Publishing, New York. Kiili, K. (2005a). On Educational Game Design: building blocks of flow experience. PhD thesis, Tampere University of technology, Pori, Finland. Kiili, K. (2005b). Digital game‐based learning: Towards an experiential gaming model. The Internet and Higher Education, 8 (1), 13‐24. Orji, R., Vassileva, J. & Mandryk, R.L. (2012). LunchTime: a slow‐casual game for long‐term dietary behavior change. Personal and Ubiquitous Computing DOI 10.1‐11. Osborne, J, Simon, S. & Collins, S. (2003). Attitudes towards science: a review of the literature and its implications, International Journal of Science Education, 25:9, 1049 – 1079. Pavlas, D. (2010). A Model of Flow and Play in Game‐based Learning: The Impact of Game Characteristics, Player Traits, and Player States. PhD thesis, University of Central Florida, Orlando, Florida. Prensky (2001). Digital game‐based learning. USA7 McGraw‐Hill. Shelton, B. & Scoresby, J. (2011). Aligning Game Activity with Educational Goals: Following a Constrained Design Approach to Instructional Computer Games. Educational Technology Research and Development. Vilas (Ed.), Education in a Technological World: Communicating Current and Emerging Research and Technological Efforts, Formatex Research Center. Webster, J., Trevino, L. K., & Ryan, L. (1993). The dimensionality and correlates of flow in human‐computer interaction. Computers in Human Behavior, 9, 411–426. Weizman, A. & Broza, O. (2012) Learning and enjoyment during after‐school hours; Evaluation of an integrative model. ECGBL 2012, Cork, Ireland. Zheng, M. (2012) Fifth Graders’ Flow Experience in a Digital Game‐Based Science Learning Environment, PhD thesis, North Carolina State University. Zheng, M. (2011), Examining upper elementary students’ gameplay experience: A flow theory perspective, in A. Mendez‐

568

Designing a Collaborative Serious Game for Team Building Using Minecraft Viktor Wendel, Michael Gutjahr, Philipp Battenberg, Roman Ness, Sebastian Fahnenschreiber, Stefan Göbel and Ralf Steinmetz Technische Universität Darmstadt, Darmstadt, Germany viktor.wendel@kom.tu‐darmstadt.de gutjahr@psychologie.tu‐darmstadt.de p.battenberg@googlemail.com us93buza@rbg.informatik.tu‐darmstadt.de uh93ozoc@rbg.informatik.tu‐darmstadt.de stefan.goebel@kom.tu‐darmstadt.de ralf.steinmetz@kom.tu‐darmstadt.de Abstract: For collaborative learning scenarios to be successful many factors are necessary, like group formation, the setup of the group task, and the team members' ability and willingness to work or collaborate in a team. With easily moddable popular sandbox games like Minecraft being available today, new opportunities for Serious Games arise, especially in the field of multiplayer games. In this paper, we propose an approach for a game‐based solution of collaborative learning. This approach focuses on soft skills, especially communication as well as on improving both the motivation and the ability to collaborate and work in a team. We created a Minecraft mod for a collaborative gaming experience focusing on solidarity and teamwork. We designed a special obstacle course for a set of four players using especially designed game mechanics to improve collaboration skills. Communication skills are required as well as the ability to work in a team in order to win the game. Our hypothesis is that the mod can be used as a team forming and motivational tool in the context of collaborative learning by increasing the willingness to collaborate with other people by playing the game. We performed an evaluation in which seven groups of four random players (total 28 participants) played our game. Before and after the gaming session, the players played a version of the prisoner’s dilemma game in order to test their willingness to cooperate before and after playing the game. In a control group, participants worked together at solving a jigsaw puzzle instead. Results show that our Minecraft mod provides a better game experience and group experience than a non‐digital game with cooperative aspects. It also shows the impact of the participants’ working and social background in terms of an initial willingness to cooperate. Keywords: Serious Games, collaborative learning, Minecraft

1. Motivation In recent years a new form of creating Serious Games has emerged. Modding APIs (Minecraft) or game/level editors of successful games (e.g. Neverwinter Nights, Starcraft 2) enable easy and rapid development of Serious Games. One prominent example is MinecraftEdu 1 , which was created as a modification to Minecraft for use in the classroom. The concept of collaborative learning is being used in schools and institutes of higher learning as well as in various training scenarios today, ranging from mere group works over concepts like mutual teaching (learning by teaching) to collaborative working on complex projects. Whereas soft skills like the ability to work in teams and to communicate with group members are vital, they can be trained specifically by using collaborative learning principles. So, it seems promising to combine the concept of Serious Games with the concept of collaborative learning. Games, especially multiplayer games, inherently offer many of the features which are necessary for collaborative learning to be successful, like common goals, or the necessity to communicate with fellow players. In previous work, we described first approaches for collaborative multiplayer Serious Games focusing on either collaboration itself (Wendel et al. 2013) or for using such a game for learning specific contents (Wendel et al. 2010). In this paper, we propose a new approach for designing collaborative multiplayer Serious Games with the purpose of training soft skills, focusing on the ability and the willingness to collaborate and to work in a team. 1

minecraftedu.com/

569

Viktor Wendel et al. This concept is based both on previous work, and on concepts for collaborative learning (Johnson and Johnson 1994) and for collaborative gaming (Zagal et al. 2006). Our hypothesis is that our game design can be used to create games which encourage collaborative and team focused behavior and subsequently for a training of those soft skills. We decided to implement these concepts by creating a Minecraft mod. The basic functionality is identical with Minecraft, thus enabling an easy access. Our hypotheses are as follows: A digital serious multiplayer game for cooperation...

provides better user experience for the player ...

leads to more trust in the other group members ...

leads to more cooperative behavior ...

than a common game that includes aspects of multiplayer cooperation (e.g. a puzzle). We performed an evaluation with 28 participants. One group played our game for 25 minutes. A second group of participants worked together at solving a puzzle game for 25 minutes instead of playing our game. Results showed, that the participants were much more willing to collaborate using our game.

2. Related work 2.1 Digital educational games In the field of Digital Educational Games, or game‐based learning, the research is mainly focusing two aspects: Motivation for the use of Serious Games for learning, and Serious Game design. The use and potential of Serious Games has been argued by (Gee 2003), (Prensky 2003), (Squire 2003) and (Van Eck 2006) in terms of learning, or (Delwiche 2006), (Steinkühler 2004), or (Mansour and El‐Said 2008) in terms of Serious Games design. (Harteveld 2011) provides useful guidelines looking at the design process from the three dimensions reality, meaning, and play. A Serious Game design document is provided in (Bergeron 2006).

2.2 Collaborative learning This work is mainly based on concepts derived from (Johnson and Johnson 1994) which identified five essential elements which foster cooperative work in face‐to‐face groups. (Zea et al. 2009) provide first concepts for introducing collaborative learning techniques into educational video games. Apart from that approach, Computer‐supported Collaborative Learning (CSCL) research mainly focuses on e‐learning tools, like shown in (Onrubia and Engel 2009), or (Larusson and Altermann 2009). Concepts from those traditional CSCL fields of application might be transferable to Serious Games, too.

2.3 Motivation in games (Prensky 2002) argues that games can provide the necessary motivation for people who otherwise are not properly motivated to learn. However, in order for (Serious) games to be motivational, they need to fulfill several requirements. The game needs to provide the right amount of challenge (Gee 2003), (Lepper and Henderlong 2000). The concept of 'Flow' as first proposed by (Csikszentmihalyi 2000) and later refined especially for games by (Sweetser and Wyeth 2005) becomes important in this context.

2.4 User experience and collaboration User experience is a versatile construct that describes the whole experience a user has by playing a game (Nacke 2009). This includes aspects of cognition, emotion, physiology, etc. This is to say many aspects like positive and negative emotion, cognitive load and arousal, usability, immersion and flow (and many more) are part of this experience. To measure trust and cooperative behavior we used the prisoner's dilemma game approach of (Sheese and Graziano 2005). This approach allows differing between trust and cooperative behavior.

570

Viktor Wendel et al.

2.5 Creating games with Minecraft Today, Minecraft is already being used in various projects in schools (Duncan 2011), (Bayliss 2012), (Levin 2011), (Shifter 2012). The most prominent example is MinecraftEdu, a modification of Minecraft especially for a use in the classroom. As an official modding API is not yet released, the most promising alternative is the use frameworks Bukkit 2 and Spout 3 . As tutorials and documentation are very rare, the most important resources are the respective APIs. Using the Spoutcraft client instead of the original Minecraft client, client‐sided modifications are also possible without touching the client itself. The complete modification is done inside a server‐sided plugin similar to Bukkit.

3. Approach 3.1 Game design foundations Our game design is based on the five essential elements which according to (Johnson and Johnson 1994) foster cooperative work in face‐to‐face groups:

Positive Interdependence: knowing to be linked with other players in a way so that one cannot succeed alone

Individual Accountability: individual assessment of each student's performance and giving back the results to both the group and the individual

Face‐to‐Face Promotive Interaction: Promoting each other's success by e.g. helping, encouraging and praising

Social Skills: Interpersonal and small group skills are vital for the success of a cooperative effort

Group Processing: Group members discussing their progress and working relationships together

We also take into account the rules and pitfalls as to regard when designing collaborative games as stated by (Zagal et al 2006):

Lesson 1: To highlight problems of competitiveness, a collaborative game should introduce tension between perceived individual utility and team utility.

Lesson 2: To further highlight problems of competitiveness, individual players should be allowed to make decisions and take actions without the consent of the team.

Lesson 3: Players must be able to trace payoffs back to their decisions.

Lesson 4: To encourage team members to make selfless decisions, a collaborative game should bestow different abilities or responsibilities upon the players.

Pitfall 1: To avoid the game degenerating into one player making the decisions for the team, collaborative games have to provide a sufficient rationale for collaboration.

Pitfall 2: For a game to be engaging, players need to care about the outcome and that outcome should have a satisfying result.

Pitfall 3: For a collaborative game to be enjoyable multiple times, the experience needs to be different each time and the presented challenge needs to evolve.

The use of these fundamental elements for a game design for collaborative multiplayer Serious Games is described in more detail in (Wendel et al. 2013). In this paper, we address how a game can be designed based on these rules and guidelines in order to create a collaborative multiplayer game specifically for enhancing teamwork and collaboration.

3.2 Game design for team building Based on the foundations stated in the previous section, we derived the following game elements for creating a necessity for collaboration inside a team‐based multiplayer game. Our design is for a team of four players.

http://bukkit.org/ http://get.spout.org/

571

Viktor Wendel et al. 3.2.1 Minecraft (sandbox) mechanic By the nature of the game, players are able to control their character in the way they want. They cannot be forced to certain decisions. At several points throughout the game, players will have to choose between selflessly helping the team (by getting themselves into danger) or staying safe. However, the team can never force one player to do something. This implements Lesson 2. 3.2.2 Creation of a common goal The players are provided with a background story telling them what they need to do in order to win the game and what failure means. The players' task is to save the 'last gnome on earth' (idea from the Left For Dead 2 gameplay mutation). If they are not able to save that gnome in time, the game will be lost. There are only two possible outcomes: victory and failure. By providing the players with such a narrative background, a foundation for the fact that players need to care about the outcome is created. This way we address Pitfall 2. Moreover, it becomes clear that the game cannot be won alone. If the players achieve their goal, they all won, if someone fails, the whole team fails. This refers to Positive Interdependence. 3.2.3 Player separation According to (Reuter et al. 2012), ‘Player Separation’ is a concept for multiplayer puzzles. In our concept, we use player separation to prevent one player from being able to solve all tasks (Pitfall 1). When players are physically divided only those players being present at position x can solve the respective puzzle there. Thus, we also address Lesson 4; we bestow players with different abilities, whereas the abilities are based on their location. 3.2.4 Gnome handling Only one player at a time can carry the gnome. This player will continuously slow down until he/she cannot move at all. Furthermore, this player cannot jump. This mechanic forces players to hand over the gnome between each other in order to be able to move the gnome forward and to overcome certain obstacles. This mechanic implements Positive Interdependence as well as Lesson 1. Players might always decide to take over the gnome (team utility), getting into danger of being caught by enemies following them or just to try to stay safe by not carrying the gnome. Here, it is important for players to learn about the consequences of their choices (Lesson 3). Staying safe while the fellow player, which is carrying the gnome, becomes too slow, thus getting caught by enemies might lose the game. 3.2.5 Team highscore At the beginning and at the end of a game session, the players can see the highscore including their own score provided they won the game. This rather simple method prevents Pitfall 3, as it creates a motivation to beat the highscore, thus to play the game again. As the collaborative puzzles are changing from game session to game session, the challenge is renewed each game, making another game session interesting, again. 3.2.6 Need for communication Resulting from the player separation, players often possess information which the other part of the team needs (see Section 3.3). This makes communication inevitable. Also, telling another player that he/she needs to take over the gnome is necessary. Several puzzles require the players to coordinate their actions (see Section 3.3), thus making communication essential. This requires Social Skills and Group Processing. During this procedure, also Face‐to‐Face Promotive Interaction takes place. We did, however, not directly incorporate Individual accountability into our design, as is seems rather counter‐ intuitive to reward selfish actions. Furthermore, it should not be important which player pressed a button but rather that the team figured out which button to press together.

572

Viktor Wendel et al.

Figure 1: Colored lever puzzle

3.3 Collaborative puzzles The collaborative puzzles we designed are based on the concepts for multiplayer puzzles as described in (Reuter et al. 2012). We make use of the concepts of 'Player Separation' and 'Heterogeneous Resources'. Using these, we designed the following puzzle types:

Lever Puzzle: This puzzle is based on player separation. Players of both parts need to pull levers in order to open doors, etc. This simple puzzle prevents single players from being able to advance alone.

Lever Color Puzzle: An extension to the previous puzzles is coloring levers. Players need to know which levers the have to pull. The information is given to the other part of the team such that they need to find that information and provide their teammates with it. The information is a resource which at first is only available to one part of the team (see Figure 1).

Gnome Button: We introduce a special button which can only be pressed by the player possessing the gnome. This supports the 'Heterogeneous Resources' feature.

Simple Math Puzzle: Again, using division of information, we designed math puzzles in the form of: 'Find d'. Hints are given in the form of 'd=c+5', ' c=b‐3', 'b=a+1', and 'a=8' with one part of the team having the first and the third hint and the other part having the other hints. This puzzle can only be solved if the team can give information back and forth.

Heavy Block: At some points, the players need to place a block in order to create a step to climb. Therefore, a heavy block needs to be moved from a starting point to its destination. The player carrying the block cannot move so that the players need to form a human chain to pass the block on. This requires all four players to move and work together.

Difficult Terrain: This puzzles type requires the gnome to be carried to several locations to press the gnome button there. The terrain is designed in a way such that one has to move a long way alone. However, handing over the gnome at special locations shortens the distance drastically. This requires all four players to move and work together.

4. Implementation 4.1 Spout and Bukkit As it seemed reasonable to be able to use our Mod with future Minecraft versions, we decided not to mod the Minecraft client itself, but instead to use the Bukkit framework. This way, it is possible to create a Mod which can be used together with other available mods due to the modularized use of mods in the Bukkit framework. The access to the client is limited without modding the client itself. Therefore, we decided to use spout, which is a framework enabling multiplayer Bukkit plugins with an access to the client when using the SpoutCraft client. The SpoutCraft client is a modified version of the Minecraft client. Thus, using the SpoutCraft client and SpoutPlugin as a plugin for Bukkit, a server‐side API for client side changes is available.

4.2 Level design All players start the game together in one room. From there they can decide which two players will walk through the hallway carrying the gnome. The remaining two players will help them from upstairs. Once two players used the one‐way teleporters towards the gnome hall, and they pick up the gnome, the game actually

573

Viktor Wendel et al. starts. The players then move along the hallway as shown in Figure 2. Throughout the game, the players will solve colored‐levers‐puzzles (colored circles), or math puzzles (numbers). At the points with the gnome symbol, the gnome is handed over to the other two players, effectively switching roles. The icon with the four players means that the ‘Player Separation’ is revoked and all four players meet to solve a task. The pyramid is a puzzle where all players need to build stairs with heavy steps by forming a human chain. During the red arrow parts, the players are under time pressure, as they are chased by zombies. This is in the beginning during the colored‐lever‐puzzles and the math puzzles and in the last part as to create a thrilling finale. Players need to run while being chased by zombies. They need to hand over the gnome frequently while pulling a lot of levers in order to open doors. Moreover, the terrain is difficult in this part so that handing over the gnome becomes more vital.

Figure 2: Schematic level overview

4.3 Modded features Following, we describe in detail the main features which were created for this mod in order to implement our game design. 4.3.1 Building restrictions Free placement of blocks as well as destroying blocks was forbidden. This was necessary in order to prevent players from creating shortcuts.

574

Viktor Wendel et al.

Figure 3: Gnome Sockets 4.3.2 Gnome handling To prevent deadlocks due to Minecraft game mechanics (i.e. items despawn (disappear) after a few minutes when dropped), it was forbidden to throw or drop items. The gnome and the heavy blocks are passed from one player to another by clicking on the receiving player if he/she is close enough. In addition to that, special gnome related blocks (sockets) have been created. In Figure 3, those blocks are shown: from left to right:

StartSocket: Players receive the gnome from there

SleepSocket: The gnome can be placed there (this is used for puzzles where all four players need to be able to move)

SleepSocket with gnome: The gnome can be taken back from there

EndSocket: The players need to bring the gnome there to win the game

4.3.3 Logic elements The most important part of our mod is the logic‐packet. It enables the freely definable combination of levers and buttons with doors and triggering of events without the 'redstone'‐related delay 4 . Our logic system is based on two elements: triggers and responses. They are connected via a context. A context can have more than one trigger or response. They can be connected via logical ANDs and ORs. Whenever a trigger is fired, it sends an event to its context which checks the logical condition and sends a signal to the responses if the condition becomes true (see Figure 4). 4.3.4 Zombie control Our game design requires some sort of pressure at some points in the game. We create that pressure in form of time pressure. The players need to solve puzzles while being chased by zombies. Zombies are entities which are originally available in Minecraft. Therefore, a ZombieManager package was created providing functionalities for spawning zombies at desired locations, despawning zombies, setting targets and movement speed.

Figure 4: Logic control 'Redstone' is part of the vanilla Minecraft used to create logical circuits

575

Viktor Wendel et al.

5. Evaluation 5.1 Participants 28 participants attended the study. Their age ranged from 21 to 45 years (m=25.81; sd=5.16). Two were female, 24 were male, two more did not name their age and gender. The mean time of playing video‐games a week was 9.04 hours (sd=10.64). One group was excluded due to one member of the group having major problems using the Minecraft game controls, which made meaningful play impossible for the group. So 12 participants joined the experimental and 12 participants joined the control group.

5.2 Design The study has a 2‐factorial design between subjects. The independent variable is the kind of cooperative game (puzzle/ Minecraft). As dependent measurement a user experience questionnaire, a questionnaire asking for the quality of group‐cooperation and a version of the prisoner's dilemma game was used (see Table 1). Table 1: Prisoner’s dilemma game Decision of other player Share Do not share

Own decision Share Small candy nothing

Do not share Choice of big candy Small random candy

5.3 Stimuli The two games used were our Minecraft mod and a jigsaw puzzle consisting of 2000 parts. Each participant got about a quarter of the parts, including some curbs. A jigsaw puzzle was selected, because it is a common, well known game that includes aspects of cooperation (like communication, division of cognizance and work), can be done for an adjustable time and presents the success of the cooperation to the player all the time. The user experience questionnaire includes 7 sub‐scales of user experience (negative emotion, positive emotion, cognitive load, motivation, immersion, flow and arousal). Each sub‐scale includes three items (e.g. frustration, anger, boredom for negative emotion). Each of these items has to be answered on a 10 point scale. An evaluation of 145 of these user experience questionnaires using different games and settings showed a Cronbach's Alpha of .93, this is to say the questionnaire is measuring one homogeneous construct. The questionnaires used to calculate reliability came from different studies conducted or overseen by the second author.The group‐cooperation questionnaire is based on the design elements for collaborative multiplayer Serious Games presented in (Wendel et al. 2012). It includes 15 items like 'the communication in the group was good'. The prisoner's dilemma game has been conducted with the reward system shown in Table 1. The game includes two questions: At first, in respect to (Sheese and Graziano 2005): 'please mark with a cross the answer you think your opponent will choose'. This enables to differ between uncooperative behavior that is caused by fear (of the opponent’s choice) and hostility. The second question was 'please mark with a cross the answer you like to choose'. In contrast to common versions of the prisoner's dilemma game the opponent was unknown (chosen randomly from the group members) because it is aimed to measure trust in the group and not in a specific person.

5.4 Aggregation For the evaluation the data has been aggregated as followed. For each participant the mean of the items of the UX and the group cooperation questionnaire was built. For the prisoner's dilemma game the deviation between the two rounds was build. Therefore cooperate was rated with a value of '+1' and not cooperate was rated with a value of '0'. The value of the second round was subtracted from the value of the first round. So, participants who did not change their behavior had a value of '0', those how chanced from cooperate to not cooperate have a value of '‐1' and those changing from not cooperate to cooperate have a value of '+1'.

576

Viktor Wendel et al.

5.5 Analysis To analyze the data a two tailed independent samples t‐test with the kind of game as independent measurement has been conducted for the UX questionnaire and the group‐cooperation questionnaire. The prisoner’s dilemma game was tested by a two tailed ANOVA (between subjects).

5.6 A Priori It was more fun (t(22)= ‐1.93; p=.066) to play the Minecraft Mod (m=6.70; sd=0.90) than to play the puzzle (m=5.85; sd=1.23). The group‐interaction has been rated more positive (t(22)=1.99; p=.060) by playing the Minecraft Mod (m=8.15; sd=0.70) than by playing the puzzle (m=7.42; sd=1.05). Neither the expectation of the behavior of the other nor the own chooses of the prisoner's dilemma game are different in dependency of the kind of game (p>.20).

5.7 Discussion 5.7.1 Hypotheses We hypothesized that our digital multiplayer game will lead to a better user experience than a common non‐ digital game with cooperative aspects. In fact, the user experience questionnaire as well as the group‐ cooperation questionnaire showed that the experience was better while playing the digital multiplayer game. So this hypothesis seems to be true. We also hypothesized that the digital multiplayer game will lead to more trust in the group members as well as to more cooperative behavior. But trust and behavior did not differ between the two games ‐ so this hypothesis has to be rejected. 5.7.2 Shortcoming One reason for the not significant differences in the prisoner's dilemma game may be the sample of the participants. Students and lecturers were accustomed to cooperative work. In this case they were even accustomed to each other. So they mostly started the prisoner's dilemma game with ‘cooperate’ and ended with ‘cooperate’ as well. In fact, 20 of the 24 participants did so. This assumption is also based on the cooperation values showing a mean value of 7.43 for the puzzle and a mean value of 8.15 for the game (on a 10 point scale). One could conclude that initial group‐cooperation and trust was very good throughout all groups ‐ perhaps too good to be improved by a 25 minute lasting game.

6. Conclusions In this paper, we presented an approach for a multiplayer Serious Game for enhancing teamwork abilities. Our game design is based on design guidelines found in literature. It is designed in a way to support collaborative behavior in a game through the overall setup, special collaborative puzzles and the use of mechanics like player separation and heterogeneous resources. We implemented our design as a Minecraft Mod. We decided to create a plugin for Bukkit using Spout‐ Plugin. Thus, we created a Minecraft level based to our game design. We also created an editor in order to enable further level creation for non‐programmers. We evaluated our concept using our Minecraft Mod prototype. Results showed that our Minecraft Mod provides a better game experience and group experience as a non‐ digital game with cooperative aspects. But to show a significant increase of trust and cooperative behavior a sample of participants might be required, that is not familiar to each other and therefore will not be so trustful to each other right from the start.

References J. Bayliss, J. (2012) Teaching game AI through Minecraft mods. In Games Innovation Conference (IGIC), 2012 IEEE International, pages 1‐4, Rochester, NY, USA. Bergeron, B. (2006) Developing Serious Games (Game Development Series). Charles River Media. Csikszentmihalyi, M. (2000) Beyond Boredom and Anxiety. Jossey‐Bass. Delwiche, A. (2006) Massively Multiplayer Online Games (MMOs) in the New Media Classroom. Educational Technology & Society, 9(3):160‐172. Duncan, S.C. (2011) Minecraft, Beyond Construction and Survival. Well Played, 1(1):1‐22. Gee, J.P. (2003) What Video Games Have to Teach Us About Learning and Literacy. Computers in Entertainment (CIE), 1:20. Harteveld, C. (2011) Triadic Game Design. Springer‐Verlag New York Inc.

577

Viktor Wendel et al. Johnson, D. and Johnson, R. (1994) Learning Together and Alone, Cooperative, Competitive, and Individualistic Learning. Needham Heights, MA: Prentice‐Hall. Larusson, J. and Alterman, R. (2009) Wikis to Support the Collaborative Part of Collaborative Learning. International Journal of Computer‐Supported Collaborative Learning, 4(4):371‐402. Lepper, M.R. and Henderlong, J. (2000) Turning" Play" into" Work" and "Work" into "Play": 25 Years of Research on Intrinsic Versus Extrinsic Motivation., chapter Intrinsic and extrinsic motivation: The search for optimal motivation and performance, pages 257‐307. Academic Press, San Diego. Levin, J. (2011) The Minecraft Teacher. online via http://minecraftteacher.net/. Mansour, S. and El‐Said, d.M. (2008) Multi‐Players Role‐ Playing Educational Serious Games: A Link between Fun and Learning. The International Journal of Learning, 15(11):229‐240. Nacke, L. (2009) Affective Ludology: Scientific Measurement of User Experience in Interactive Entertainment. PhD thesis, Blekinge Institute of Technology, Karlskrona, Sweden. Onrubia, J. and Engel, A. (2009) Strategies for Collaborative Writing and Phases of Knowledge Construction in CSCL Environments. Computers & Education, 53(4):1256‐1265. Prensky, M. (2002) The Motivation of Gameplay: The Real Twenty‐first Century Learning Revolution. On the horizon, 10(1):5‐11. Prensky, M. (2003) Digital Game‐based Learning. Computational Entertainment, 1(1):21. Reuter, C. and Wendel, V. and Göbel, S. and Steinmetz, R. (2012) Multiplayer Adventures for Collaborative Learning With Serious Games. In P. Felicia, editor, Proceedings of the 6th European Conference on Games Based Learning, pages 416‐423. Acad. Conf. Ltd. Schifter, C. (2012) Minecraft in an english class. Poster at 6th European Conference on Games Based Learning (ECGBL). Sheese, B. and Graziano, W. (2005) Deciding to Defect. The Effects of Video‐game Violence on Cooperative Behavior. Psychological Science, 15(5):354‐357. Squire, K. (2003) Video Games in Education. International journal of intelligent simulations and gaming, 2(1):49‐62. th Steinkuehler, C.A. (2004). Learning in Massively Multiplayer Online Games. In ICLS '04: Proceedings of the 6 international conference on Learning sciences, pages 521‐528. International Society of the Learning Sciences. Sweetser, P and Wyeth, P. (2005) GameFlow : A Model for Evaluating Player Enjoyment in Games. Computers in Entertainment (CIE), 3(3):1‐24. Van Eck, R. (2006) Digital game‐based learning: It's not just the digital natives who are restless. Educause Review, 41(2):16. Wendel, V. and Babarinow, M and Hörl, T and Kolmogorov, S. and Göbel, S. and Steinmetz, R. (2010) Transactions on Edutainment IV, volume 6250 of LNCS, chapter Woodment: Web‐Based Collaborative Multiplayer Serious Game, pages 68‐78. Springer. Wendel, V. and Gutjahr, M. and Göbel, S. and Steinmetz, R. (2013) Designing Collaborative Multiplayer Serious Games: Escape From Wilson Island ‐ A multiplayer 3D Serious Game for collaborative learning in teams. In Education and Information Technologies, 18(2): 287‐308. Zagal, J.P. and Rick, J. and His, I. (2006) Collaborative Games: Lessons Learned From Board Games. Simulation and Gaming, 37(1):24‐40. Zea, N.P. and Sánchez, J.L.G. and Gutiérrez, F.L. and Cabrera, M.J. and Paderewski, P. (2009) Design of Educational Multiplayer Videogames: A Vision From Collaborative Learning. Advances in Engineering Software, 40(12):1251‐1260.

578

Application of the Principles of Gamification to Facilitate Acquisition of Self‐Management Skills in Young People With Long‐ Term Medical Conditions Andrew Wilson 1 and Janet McDonagh 2 1 School of Computing, Telecommunications and Networks, Faculty of Technology, Engineering and the Environment, Birmingham City University, Millennium Point, Birmingham, UK 2 University of Birmingham & Birmingham Children's Hospital NHS Foundation Trust, Birmingham, UK andrew.wilson@bcu.ac.uk j.e.mcdonagh@bham.ac.uk Abstract: With improvements in health care children diagnosed with long term medical conditions are now more likely to live longer with the possibility that their condition will follow them into adulthood. As the young person grows older they will eventually have to transfer their care to the adult setting. Failure to plan and coordinate this has been associated with poorer health outcomes and disruption to their care. Transition planning encourages both health literacy and health promoting behaviours in an age and developmentally appropriate way. In order to gauge the attainment of these skills assessment tools have been developed. They all have items of commonality (understanding the condition, self‐ management, adherence to treatment and communication) but many are disease specific and address items that are specific to their country of origin (i.e. medical insurance). A series of transitional readiness checklists have been developed at Birmingham Children’s Hospital which are more focussed to young people in the UK. The domains they measure include knowledge, self‐advocacy, transferring to adult care, health and lifestyle, activities of daily living, school and vocation, leisure and managing emotions. Engaging and encouraging young people to work towards attaining the required skills and the assessment of their competency in them beyond the health care setting remains a challenge in every day clinical practice. The processes would therefore benefit from a more objective assessment for the doctor and structured in way to be more fun for the young person. Gamification is the term used to describe the use of game mechanics in non‐game contexts with the aim of trying to improving engagement and motivation. In this paper we will discuss how the dynamics of gamification (progress, feedback and behaviour) and game mechanics can be mapped to the self‐care and self‐ management skills associated with Birmingham Children’s Hospital transitional care checklists. This will include how badges can be used to indicate a young person’s progression towards attaining increasing levels of knowledge for example about their condition, the effects of their condition on their body as well as improving understanding of their treatments and any side effects associated with it. Other examples will include how trophies can be used to indicate successful understanding of a series of concepts at a particular developmental stage in a young person. This paper focuses on discussing the application of gamification to the Birmingham Children’s Hospital transitional care checklists. However this initial framework will give both clinicians and health care workers an insight into the use of gamification in a health care setting and provide a basis for application to other areas of knowledge and skills acquisition where engagement in the processes can be challenging. Keywords: gamification, adolescent, chronic conditions, transition, self‐care

1. Introduction It is estimated that between 10‐12% of young people in the UK may be affected by a long term medical condition (Coleman et al., 2011) which can persist with them into their adult life. Care initially starts in family centred clinics but as the young person grows older they have to transfer to the adult setting. This normally occurs between 16 and 18 years of age in the UK but depends more upon local policy rather than the young person’s confidence in being able to cope in the adult setting. Studies investigating the pre‐transfer period identified an increased risk for negative health outcomes. This included both increased incidence of graft loss (Harden et al., 2012) and chance of mortality in transplant patients (Annunziato et al., 2007), worsening disease activity in young people with long term rheumatic conditions (Hersh et al., 2009), or they are just lost in the health care system and consequently fail to attend clinics (Wacker, 2005; Yeung, 2008). Those that experience such lapses in care have also been reported to have an increased risk of subsequent readmission to hospital (Yeung, 2008; Nakhala, 2009). Even in those with stable disease, poorer vocational outcomes have been reported post transfer (Duguépéroux, 2008).

579

Andrew Wilson and Janet McDonagh The disruption to care is complicated by the fact that young people are undergoing those developmental changes that are normally associated with adolescence. It is also a time when patterns of health promoting as well as health risk behaviours are established which can be maintained throughout life (Sawyer at al., 2007). Health risk behaviours are as common if not more common in those individuals with long term conditions (Suris et al 2008) and there is the potential for greater adverse health outcomes from them (Sawyer at al., 2007; Philpott, 2011). Non adherence to treatment is common among young people with long term conditions (Hanna, 2012) and can be related to their engagement in health risk behaviours. Transition addresses the medical, psychosocial as well as educational and vocational needs of the young person. During transition the responsibilities for care moves from the parent to the young person requiring them to acquire a series of knowledge and skills that are important for them to function independently in adult care. These include understanding of health and condition specific issues, self‐management skills, effective information‐seeking skills, managing psychological and general health, effectively utilising the health care available to them, coping with social issues, maintaining education and vocation aspirations as well as living independently. These skills need to be acquired in an age and developmentally appropriate manner. They also depend on young person’s cognitive skills for example La Rosa and colleagues (2010) suggest that young children requiring organ transplants should be able to understand their condition and need for treatment, during early adolescence they should understand side effects associated with their treatment and the need for urgent care, whereas young adults should be able to schedule their own appointments, communicate with their care team and understand the need to move their care to adult centres (Table 1). Table 1: A time line of knowledge acquisition for young people with solid‐organ transplants Time Period Childhood, before transplant, early post‐transplant period. Pre‐teen period / early adolescence. Adolescence / young adulthood.

Behaviour Learning.

Activities Understanding medical condition, need for treatment, medication and frequent follow‐up.

Increasing knowledge and responsibility. Providing self‐care.

Understand indications for medications, reasons for urgent care, awareness of medical consequences of treatment. Recite medications and indications, call for appointments and refills, identify and call for urgent concerns, effectively communicate health concerns, take medications on own, anticipate and accept eventual need for transfer of care.

Adapted from La Rosa and colleagues (2010). Transitional readiness assessment tools have been developed to assist clinical care teams in evaluation a young person’s progress towards attaining the necessary skills associated with successful transfer to the adult care system. This paper will give an overview of some of the assessment tools currently available and will discuss how the application of gamification to one specific tool developed by Birmingham Children’s Hospital (UK) would benefit the process by creating a more objective framework for the clinical team as well as encapsulating the processes in a more fun way in order to incentives the young person engagement.

2. Measurements of readiness to transition Gauging a young person’s readiness to transfer their care depends upon their self‐belief, parental involvement as well as the assessment of competency by the health care team. Several assessment tools including the Paediatric Transition Readiness Score (PTRS) and Pediatric Acceptance Readiness Score (PARS) (La Rosa et al., 2010), Transition Readiness Assessment Questionnaire (TRAQ) (Sawicki et al., 2011) and The University of North Carolina (UNC) Traxnsition Scale (Ferris et al., 2012) have been developed to assist in ascertaining transitional readiness. Many of the items in these tools are common to one another for example knowledge and understanding of the condition and its treatment, self‐care and self‐management, communications with the care team and understanding appropriate support services. However they may also be disease specific and have aspects that are more relevant to their country of origin for example medical insurance and financial aspects of care. Figure 1 illustrate the “Getting Ready!” early transitional readiness checklist that has been developed for young people with rheumatic disease at Birmingham Children’s Hospital (UK).This is one of three plans which also includes “Moving along!” and “Moving up!” for mid and late adolescence. The domains assessed include

580

Andrew Wilson and Janet McDonagh knowledge, self‐advocacy, transferring to adult care as well as health and lifestyle, activities of daily living, school and vocation, leisure and managing emotions. Although not formally validated they continue to be used with all young people attending the adolescent rheumatology service from age 11 years irrespective of condition. These assessment tools are based on self‐reporting making them subjective and rely on the young person motivating themselves to maintain positive health behaviours. By implementing mechanics associated with gamification the process could be made more fun for the young person and objective for the clinician.

Figure 1. Examples of some of the questions associated with the checklists (“Getting Ready!” early adolescence) for assessing transitional readiness from the young person’s [A] and the health professionals [B] perspective. E, M and L denote early, mid and late adolescent development.

3. Gamification Gamification is a term that has been used to describe the use of game mechanics and game design techniques in non‐game contexts. The three main game dynamics associated with gamification progression, feedback and behaviour represent processes that can promote the knowledge, learning and behaviours that are important in acquisition of self‐management skills. Table 2 (a and b) shows the game dynamic and mechanics associated with gamification and how these can be related to the positive self‐care and self‐management behaviours which are required in young people for successful transfer of care. Table 2a: Mapping of gamification mechanics to self‐care behaviours Game Dynamic Progression

Game Mechanic

Mechanic definition

Achievements

Levels

Accomplishments which are unlocked after succeeding in a given task. A reward for amassing points.

Points

Numerical values given to

581

Examples relating to self‐care / management behaviours Successfully attaining self‐care and self‐ management skills. The attainment of more advanced knowledge and skills as the young person develops. Incentives to achieve skills and a measure of

Andrew Wilson and Janet McDonagh Game Dynamic

Feedback

Game Mechanic

Mechanic definition

Progression

actions. Success as measured through the completion of tasks.

Appointments Bonuses

Cascading Information Theory Combos

A predetermined time and place to participate in a game. Rewards for completing a series of challenges.

Release of information in small parts to ensure appropriate levels of understanding. Undertaking another action after successfully completing a previous one.

Examples relating to self‐care / management behaviours success. Demonstration that the skills are being attained and that satisfactory progress is being made toward transfer of care to the adult system. Encouraging the maintenance of regular contact with the clinical care team. Encouraging positive behaviour for example adherence to treatment regimes, appointments or demonstrating acquisition of skills in a timely manner. Providing information in an age and developmentally appropriate manner. Demonstrating an understanding of a series of skills or skills that are associated with one another for example knowing the types of medicine they need to take as well as their side effects.

Table 2b: Mapping of gamification mechanics to self‐care behaviours Game Dynamic Feedback

Game Mechanic Countdown

Quests

Reward schedules

Behavioural

Behavioural momentum Blissful productivity

Community collaboration Discovery

Epic meaning

Mechanic definition A time limit to accomplish a task. A series of challenges or obstacles a player must overcome. The timeframe and mechanisms through which rewards (points, prizes, level ups) are given. The tendency to keep doing activities. Being happy as a result of working hard and undertaking rewarding activities. The game community works together to solve a riddle, a problem or a challenge. The love of exploring and discovering something new and being surprised by it. The motivation to achieve something great and awe‐ inspiring.

Infinite gameplay

No explicit end.

Ownership

Used to create loyalty.

Status

The rank or level of a player.

Urgent optimism

Extreme self motivation to tackle an obstacle with the

582

Examples relating to self‐care / management behaviours A way of encouraging behaviours which are potentially lagging behind what would be expected / anticipated for that young person. A way of reducing anxiety when visiting new clinical care settings and care teams. Motivation, incentives and recognition of accomplished tasks and acquired skills.

Encouraging a state where positive health care behaviours become the norm for example adhering to treatment, maintaining a good diet and taking regular exercise.

Providing peer support, sharing information and experiences to reduce the feeling of isolation and being alone. Learning about their condition, treatment and how they can manage their care independently. A feeling of achievement that the young person has taken an active role in their own care and that they have successfully prepared themselves for adult life and the adult care system. Care for long term conditions is ongoing and will change as the person grows older. Maintaining positive health behaviours improves the chances of a better quality of life. Taking responsibility for self‐care as the emphasis shifts to the young person away from the parents. Demonstrating that the person has achieved the required skills and knowledge in relation to their peers and that it is achievable. The impetus to acquire important skills as quickly as possible even though they may be challenging

Andrew Wilson and Janet McDonagh Game Dynamic

Game Mechanic

Mechanic definition expectation of success.

Virality

“Spreading the word” in order to engage a wider audience.

Examples relating to self‐care / management behaviours for example having joint injections without general anaesthetic. The success of the system encourages more young people to participate in it and take an active role in its development and evolution.

Adapted from Gamification Wiki (source: http://gamification.org/wiki/Game_Mechanics)

3.1 Progression The Birmingham Children’s Hospital adolescence plans monitor progress and achievements from both the young person’s and doctor's perspectives. The young person rates themselves as to whether they feel they can manage the task on their own or if they need more advice and support with the doctor recording that the young person feels that they have attained the skills. By applying a point based system to each item in the domain this would give a measure of achievement and progress. For example if a young person demonstrates their confidence in being able to describe their condition at a level that the doctor would associate with early adolescence they would be rewarded with points and a badge indicating this achievement (Figure 2). Subsequently if they demonstrate their confidence in being able to describe their condition at a level that the doctor would associate with mid adolescence they would receive further points and a higher level badge. When all items, for example demonstrating the ability to successfully describe their condition, understanding the effects on their body and understands their medicines / regimes, within in a domain (knowledge) at a particular stage of adolescence have been completed this would unlock an achievement in the form of a trophy (Figure 2). Higher value trophies (for example bronze, silver and gold) are acquired as a result of demonstrating higher levels of knowledge associated with older adolescence. When all items within a domain are completed a further reward (a shield) can be earned signifying mastery of that domain at all levels.

Figure 2: Badges and trophies Badges indicate attainment of increasing levels of knowledge associated with a particular item of understanding. The trophies indicate attainment of all the required individual items of skill at a particular level (early, mid or late adolescence). There is also an overall reward (shield) for completing all items within a domain.

3.2 Feedback Clinical appointments are in important point of contact between doctor and patient in order to monitor progress and provide feedback. However failure to keep appointments is a common occurrence which results in disruption of care but also causes additional burden on clinical services. This could be enhanced further by applying gamification techniques by awarding points for attending clinical appointments which would reinforce their importance. Some of the items in the checklists (Figure 1) lend themselves to aggregation. For example the act of 1. Being able to describe both the condition and effects on the body, 2. Knows medicines / regimes and side effects of medicine and 3. Preparation for joint injections without general anaesthetic and has joint injections without

583

Andrew Wilson and Janet McDonagh general anaesthetic. If these combinations or (combos) have been completed or completed in shorter than expected time frame bonuses would be rewarded. Where there is a delay in the time that certain skills would be expected to be acquired the countdown mechanic could be used to add an incentive for the young person to engage and acquire those skills. Consideration should still be given to the fact that the young person should acquire skills in an age and developmentally appropriate manner, which reflects the cascading information theory where information is released in small parts ensuring that appropriate levels of understanding are met. Orientation to the adult system where the young person and parents visit the new environment and meet the new care team can be a cause for concern and anxiety. However by encapsulating this into a quest the uncertainty and the unknown is given an element of fun and sense of achievement by accomplishing it. Gamification uses rewards as a way of recognising achievements. There is a danger that rewards can be over used devaluing the achievements but if insufficient are used the incentive to engage with the process is lost. Reward schedule and ratios are used to manage this with randomness and surprises potentially being the most effective. Badges and trophies are the most common examples of reward however digital prizes, virtual goods and objects found or taken within the course of a game have been used and can be traded or given away giving the game an extra dimension. In terms of gamification engagement is considered the prime metric. As many are web based this is often measured by unique visitors, page views per visitor, time spent on site, total time spent per user, frequency and depth of visits, participation and conversions. Although these can be used to demonstrate engagement with the process in terms of knowledge and skill acquisition they are not as useful. Therefore other activities such as keeping blogs, diaries, and uploading videos demonstrating activities as well as participating in answering questions in on line forums can be more indicative. This user generated content (UGC) is an important part of many websites. Prompt and real‐time feedback is an important to the young person. When quizzes were incorporated into an interactive multimedia computer program that was designed to teach young people about juvenile idiopathic arthritis these were praised for helping to support consolidation of knowledge about the condition (Wilson & Young, 2009). Leader boards are a tool used to give a comparison of progress and feedback in relation to others. This can have a positive effect by motivating people to improve their performance. However it can also have negative effects where it is felt that the targets are unachievable. Marczewski (2013) suggests the use of both absolute and relative leader boards where the former would indicate progress against all others where the latter would indicate performance within a domain and / or within a developmental stage (i.e. early, mid or late adolescence).

3.3 Behaviour Education is an integral part of care for young people with long term medical conditions and forms the basis for positive health behaviours. However many young people and their family find it difficult to get the information they need and it is not always in appropriate style for them and therefore multiple formats have been requested (Shaw et al., 2004). Status is an important mark of achievement in games and provides an opportunity to identify where help and assistance can be found for new players. In terms of gamification for health care this provides and opportunity to identify mentors who can provide advice and support. Despite the importance of collaboration and social interaction for the young people anonymity is also very important. Feelings of being different and not "fitting in" can lead to social isolation and stigma. There is evidence that people with disabilities use virtual worlds and play online games as a way of combating social isolation and loneliness (Kamel Boulos et al., 2007). Anonymity in these worlds can be facilitated by the use of avatars, online characters that reflect the player’s persona whilst maintaining their privacy and hiding identity. The premise of gamification is to make activities more fun and as a result improve the chances of engagement. By using this approach in young people with long term health conditions the aim would be to foster feelings of epic meaning (the motivation to achieve something great and awe‐inspiring) and ownership of their own care (self‐care). The clinical care team would be able to influence the young people’s actions with the anticipation

584

Andrew Wilson and Janet McDonagh that the individual increasingly takes on responsibility for their care as it shifts from dependence upon the parent to themselves in states that reflect the mechanics of behavioural momentum and blissful productivity.

4. Technology The Internet has established itself as a major source for young people for discovering information about their health. This has evolved into dedicated youth‐centred online programmes to facilitate self‐management. These websites include interactive activities, self‐monitoring and reflection, youth‐to‐youth information sharing and social support, and accurate, accessible and developmentally targeted health specific information (Scal et al., 2010). Features such as discussion boards, stories of hope, and video clips help to provide support and reduce the feelings of isolation in their illness (Stinson at al., 2010). Computer games have also been used successfully to promote health education and improve both self‐management skills and self‐efficacy in young people as reviewed by Wilson & McDonagh (2012). Their ability to foster social interactions allows young people to learn about their illness with their peers and can stimulate discussions about health care with friends, family and the clinical team (Lieberman, 2001). Technology is therefore an important medium for the delivery of care to young people. The combination of online delivery and game mechanics is an important part of gamification. Services such as Badgeville (http://badgeville.com/) and gamificationU (http://gamificationu.com/) already provide resources to start creating game based systems.

5. Discussion and conclusion Maintaining the engagement of young people in their self‐care as they grow older is challenging as other factors increasingly become prominent in their lives. However the implications of not taking care of themselves are a poorer quality of life and delaying the inevitable transfer of care to the adult system. Creating a process that allows both the young person and the doctor to monitor progress and create incentives for maintenance of care in a fun way and which the young person can relate to would be advantageous. By implementing the principles of gamification this might help address these issues. However there are some factors that need to be taken into consideration. Engagement itself may be dependent upon age and stage of development that is younger people may be more likely to enjoy and participate in the challenges whereas interest may wane as the person matures. There is also the element of novelty, the challenges faced in order to maintain that game’s uniqueness and how different genders perceive and react to the game based system. Therefore there is the possibility it may only be effective within a certain time frame. As the system is a game and based upon achievements there is always the chance that people will try to cheat in order to gain the rewards which would be counter productive to the requirement that the young people are learning and achieving a set of skills and behaviour which are important to their care. As with any game development, implementation and refinement of a range of complex mechanics and balancing them to make it fair, fun and in this case affecting behavioural change is a time consuming process. Despite these reservations even if there are limitations to the effectiveness of the system as long as there are improvements in the overall acquisition of skills and in the confidence of the young person to care for themselves it would be successful and as a result has the potential to engage a wider audience. This paper has discussed the application of gamification to the Birmingham Children’s Hospital transitional care checklists. This initial framework will provide clinicians and health care workers with an insight into the use of gamification in a health care setting and provide a basis for application to other areas where engagement with knowledge and skills acquisition can be challenging.

References Annunziato, R.A., Emre, S., Shneider, B., Barton, C., Dugan, C.A. and Shemesh, E. (2007) “Adherence and medical outcomes in pediatric liver transplant recipients who transition to adult services”. Pediatric transplantation, Vol, 11 No. 6, pp 608‐614. Coleman, J., Brooks, F. and Treadgold, P. (2011) Key data on Adolescence 2011: The Latest Information and Statistics About Young People Today, Association for Young People's Health, 8th edition. Duguépéroux, I., Tamalet A., Sermet‐Gaudelus, I., Le Bourgeois, M., Gérardin, M., Desmazes‐Dufeu, N. and Hubert, D. (2008) “Clinical changes of patients with cystic fibrosis during transition from pediatric to adult care”, Journal of Adolescent Health, Vol. 43, pp 459‐465.

585

Andrew Wilson and Janet McDonagh Ferris, M.E., Harward, D.H., Bickford, K., Layton, J.B., Ferris, M.T., Hogan, S.L., Gipson, D.S. McCoy, L.P. and Hooper, S.R. (2012) “A Clinical Tool to Measure the Components of Health‐Care Transition from Pediatric Care to Adult Care: The UNC TR(x)ANSITION Scale”, Renal Failure Vol. 34, No. 6 pp 744‐753 DOI: 10.3109/0886022X.2012.678171. Hanna, K.M. (2012) “A framework for the youth with type 1 diabetes during the emerging adulthood transition”, Nursing Outlook, Vol. 60, No. 6, pp 401‐410. Harden, P.N., Walsh, G., Bandler, N., Bradley, S., Lonsdale, D., Taylor, J. and Marks, S.D. (2012) “Bridging the gap: an integrated paediatric to adult clinical service for young adults with kidney failure”, British Medical Journal, Vol. 344, e3718. Hersh, A.O., Pang, S. Curran, M.L., Milojevic, D.S., and von Scheven, E. (2009) “The challenges of transferring chronic illness patients to adult care: reflections from pediatric and adult rheumatology at a US academic center”, Pediatric Rheumatology Online Journal. Vol. 7, No. 13. doi: 10.1186/1546‐0096‐7‐13. Kamel Boulos, M.N., Hetherington, L. and Wheeler, S. (2007) “Second Life: an overview of the potential of 3‐D virtual worlds in medical and health education”. Health Information and Libraries Journal, Vol. 24, pp 233‐245. LaRosa, C., Glah, C., Baluarte, H.J. and Meyers, K.E.C. (2011) “Solid‐Organ Transplantation in Childhood: Transitioning to Adult Health Care”. Pediatrics, Vol. 127, No. 4 pp 742 ‐753, doi: 10.1542/peds.2010‐1232. Lieberman, D.A. (2001) “Management of chronic pediatric diseases with interactive health games: theory and research findings”, The Journal of Ambulatory Care Management, Vol. 24, No. 1, pp 26‐38. Marczewski A. (2013) “Gamification a Little on Leaderboards”, [online] Available at http://marczewski.me.uk/2013/01/21/gamification‐a‐little‐on‐leaderboards/ (accessed 18th March 2013). Nakhla, M., Daneman, D., To, T., Paradis, G., and Guttmann, A. (2009) “Transition to adult care for youths with diabetes mellitus: findings from a Universal Health Care System”, Pediatrics, Vol. 124, No. 6, pp 1134‐41. Philpott, J.R. (2011) “Transitional Care in Inflammatory Bowel Disease”, Gastroenterology and Hepatology, Vol. 7, No. 1, pp 26‐32. Scal, P., Garwick, A. W., & Horvath, K. J. (2010) “Making Rheumtogrow: The rationale and framework for an Internet‐based health care transition intervention”, International Journal of Child and Adolescent Health, Vol. 3, No. 4, pp 451‐461. Sawicki, G.S., Lukens‐Bull, K., Yin, X., Demars, N., Huang, I.C., Livingood, W., Reiss, J., Woodm D. (2011) “Measuring the Transition Readiness of Youth with Special Healthcare Needs: Validation of the TRAQ—Transition Readiness Assessment Questionnaire”, Journal of Pediatric Psychology, Vol. 36, No. 2, pp 160‐171. Sawyer, S.M., Drew, S., Yeo, M.S. and Britto, M.T. (2007) “Adolescents with a chronic condition: challenges living, challenges treating”, Lancet, Vol. 369, No. 9571, pp 1481‐1489. Shaw, K.L., Southwood, T.R., McDonagh, J.E. on behalf of the British Paediatric Rheumatology Group. (2004) “User perspectives of transitional care for adolescents with juvenile idiopathic arthritis”, Rheumatology (Oxford), Vol. 43, No. 6, pp 770‐778. Stinson, J., McGrath, P., Hodnett, E., Feldman, B., Ciaran, D., Tucker, L., Hetherington, R., Tse S., Spiegel, L., Campillo, S., Gill, N. and White, M. (2010) “Usability Testing of an Online Self‐management Program for Adolescents With Juvenile Idiopathic Arthritis”, Journal Of Medical Internet Research, Vol. 12, No. 3, pp e30, doi: 10.2196/jmir.1349. Suris,J.C., Michaud, P.A., Akre, C. and Sawyer, S.M. (2008). “Health risk behaviors in adolescents with chronic conditions”, Paediatrics, Vol. 122, pp e1113‐8. Wacker, A., Kaemmerer, H., Hollweck, R., Hauser, M., Deutsch, M.A., Brodherr‐Heberlein, S., Eicken A, and Hess, J. (2005) “Outcome of operated and unoperated adults with congenital cardiac disease lost to follow‐up for more than five years”, American Journal of Cardiology, Vol. 95, No. 6, pp 776‐9. Wilson, A.S. and McDonagh, J.E. (2012). “Moving on: Use of Computer Games During Transitional Care for Young People with Long Term Medical Conditions”. Proceedings of the 6th European Conference on Games Based Learning. Cork, Ireland. October 2012. ISBN 978‐1‐908272‐69‐0 / ISSN 2049‐0992 Wilson, A.S. and Young, S.P. (2009) The rise of the computer as an assistive technology for education in chronic musculoskeletal disease. (eds Demir O and Celik C). IN Multimedia in Education and Special Education. Nova Science Publishers Inc. Yeung, E., Kay, J., Roosevelt, G.E, Brandon, M. and Yetman, A.T. (2008) “Lapse of care as a predictor for morbidity in adults with congenital heart disease”. International Journal of Cardiology, Vol. 125, pp 62‐5.

586

Development of an Implementation Framework for Games‐Based Construction Learning Using Scratch in Primary Education Amanda Wilson, Thomas Hainey and Thomas Connolly University of the West of Scotland, UK amanda.wilson@uws.ac.uk thomas.hainey@uws.ac.uk thomas.connolly@uws.ac.uk Abstract: Games‐based Construction Learning (GBCL) and games‐based learning (GBL) are being promoted within the curriculum for excellence (CfE). Children in the Upper Primary (Primary 4 onwards) are now being encouraged to construct their own games. These changes have been made from the previous curriculum and are causing frustration with some teachers who believe that they have been left without adequate resources and professional development to introduce such approaches into their lessons. Surveys have shown that while teachers are making some use of GBL in their class very little is being done with GBCL. With environments such as Scratch, Kodu and Gamemaker, game construction is becoming more accessible to children at a younger age. This paper will present an implementation framework for the introduction of GBCL into PE by utilising Scratch. It will discuss the literature surrounding GBCL in PE before presenting a generalised framework for the introduction of GBCL into PE. The framework is based on empirical work carried out in 3 primary schools within Glasgow in 2011/2012 and aims to give teachers a starting point for using GBCL in the classroom. Keywords: primary education, curriculum for excellence, games‐based construction, scratch, review, pedagogy

1. Introduction Children in the Upper Primary in Scotland (Primary 4 onwards) are now being encouraged to construct their own games within the CfE. These changes have been made from the previous curriculum and are causing frustration with some teachers who believe that they have been left without adequate resources and professional development to introduce such approaches into their lessons. Barriers that have been identified when trying to implement GBL within the curriculum are a lack of knowledge in using and implementing the technology and games (Pivec and Pivec, 2008; Groff, Howells and Cranmer, 2010; Robertson, 2012). Van Eck (2006) suggests from a review of the literature three ways of introducing GBL into educational establishments either through the students creating their own games, through the students playing serious games or throughout the use of Commercial off the Shelf (COTS) games. While studies have been undertaken within primary, secondary and tertiary education, there is still insufficient empirical evidence to properly substantiate the use of computer games technology as a recognised educational approach (Meluso, Zheng, Spires and Lester, 2012; McClarty et al., 2012). With applications like Scratch (Resnick, Kafai and Maeda, 2003), game construction is becoming more accessible to children and although there are studies such as Adventure Author (Robertson and Good, 2005) and Storytelling Alice (Kelleher, Pausch and Kiesler, 2007) that enable children to create their own interactive stories that other children are then able to play, game construction has been relatively unexplored within the classroom (Baytak and Land, 2011). This paper will discuss the literature surrounding GBCL in PE in the next section before presenting a generalised framework for the introduction of GBCL into PE.

2. Related work 2.1 Games‐based construction learning Papert‘s theory of constructionism regards learning as a process in which the learner actively constructs his or her knowledge by interacting with the subject matter. The constructionist perspective puts game construction in the hands of children to encourage their knowledge through developing objects (Papert, 1991). In the game construction approach, the aim of the game construction tool is to support such an activity by providing an appropriate environment. Kafai (2006) suggested that constructionists have focused their efforts on providing

587

Amanda Wilson, Thomas Hainey and Thomas Connolly students with greater opportunities to construct their own games and to construct new relationships with knowledge in the process, rather than embedding lessons directly in games. By adapting the definition of GBL by Tang, Hannegan and Rhalibi (2009) who view GBL as “the innovative learning approach derived from the use of computer games that possess educational value or different kinds of software applications that use games for learning and education purposes such as learning support, teaching enhancement, assessment and evaluation of learners”, GBCL can be defined as “an innovative learning approach that uses appropriate tools in order to allow games to be constructed to support learning and teaching”.

2.2 Literature review A literature review was carried out to identify previous game making educational research using a number of electronic databases: ACM, Science Direct, IEEE (Institute of Electrical and Electronics Engineers) Computer Society Digital Library (CSDL), Taylor and Francis, Proquest (including Applied Social Sciences Index and Abstracts, Proquest Computing and ProQuest Dissertations and Theses A and I), EBSCO (including CINAHL PsycINFO, SocINDEX, Library, Information Science and Technology Abstracts), Cambridge Journals, ERIC, Ingenta, Infotrac and Web of Knowledge. The following search terms were used: (making OR construction OR building OR design OR creation) AND (game AND education) Kafai’s Minds in Play (1995) is viewed as a notable piece of work within games construction. The study looked at children as game designers over a six‐month period and investigated how they developed as learners during this time. Over this period for one hour a day, 16 9‐10 year old children worked together with the researcher and their class teacher to create fraction games in Logo to teach younger children. Kafai argued that by using the games design they can start to construct their own knowledge and their relationship to it. By taking Kafai’s research as a starting point the literature search started from 1995 onwards. The initial search collated 1,848 entries in total. Focusing on empirical work undertaken with games construction within PE 10 papers were returned (see Table 1). By reviewing the studies undertaken in GBCL it can be seen that using GBCL in the class is still an emerging field and is starting to grow with the advent of more child‐centred games construction tools such as Scratch and Kodu. The ten papers found cover seven studies and the games constructed within these papers fall into two categories either the children making the games are creating educational games for other children to use (Baytak, 2011; Kafai, 1995; Mendes and Romao, 2011; Vos, van der Meijden and Denessen, 2011; Koutsikos et al, 2012) or they are creating original games (Robertson, 2012; Wilson, Hainey and Connolly, 2012) with the focus being on what programming concepts they have used to create their games. Table 1: GBCL empirical literature Authors

Software Used

Study

Kafai, Franke, Ching and Shih (1998) Kafai and Ching (1996)

Logo

Robertson and Howells (2008)

Adventure Author

Baytak and Land (2011) Baytak (2009)

Scratch

Mendes and Romao (2011)

Software specifically created for the study

This study investigated game design as a learning environment with a 5th grade class creating fraction games for younger children. It showed that they were able to undertake the task with more sophisticated games being developed over time. An exploratory study into the use of game making within a class of 30 P6 (9‐10 year old) children, to develop aspects of the successful learner strand of the CfE. The study showed that game making does provide the opportunity for children to achieve successful learning, however to fully explore this lessons need to become teacher led. This study looked at the use of Scratch with a class of 5th grade children to create science games for younger pupils. It showed that children were able to create games within a short period of time and were able to make use of programming concepts. This paper discussed an authoring tool created by the research team called t‐games. It allows children to author their own games and uses a learning by teaching approach.

588

Amanda Wilson, Thomas Hainey and Thomas Connolly Authors

Software Used

Study

Vos, van der Meijden and Denessen (2011)

Software specifically created for the study

Robertson (2012)

Adventure Author

Wilson, Hainey and Connolly (2012)

Scratch

Koutsikos et al (2012)

Kodu

This study compared the difference in learning between children playing games and children making games. The researcher took both lessons within four elementary schools comprised of 235 students from 9 classes split over 5th/6th grade age 10‐12 years. 5 classes constructed the games and 4 classes played the games. It looked into the motivation of students who play vs. students who create games. Findings showed that the children who created games were more motivated in their learning. Using adventure author over a 6 week period with a class of 11‐12 year olds this study looked at the analysis of games created by children during the project and also focuses n gender issues within the class showing that girls scored higher than boys of the storytelling aspects of their games. The study looked at the use of Scratch in PE or games construction and the programming concepts that children used. It evaluated games created during an 8 week study undertaken in a school with 60 children participating. The results showed that children were constructing functional games with a variety of programming concepts being demonstrated. This pilot study was conducted in a class with 25 children using Kodu. The children were asked to design games based on an environmental theme with the objective being to introduce the principles of object‐oriented programming, and to assess the satisfaction and interest generated through this.

3. Methods 3.1 Literature search This framework has been developed to help address the research question “What framework would be suitable to implement Scratch within the Curriculum at upper school PE?” The framework presented in this chapter will act as a starting point for Head teachers in PE who may wish to introduce GBCL into the curriculum but are unsure of where to start. The research aims to bring about change within ICT lessons in upper Primary Education. This will be achieved by undertaking the research in stages with different classes in the upper primary school (classes between Primary 4 and Primary 7). The objective is to work in collaboration with the class teacher to investigate how their class works with Scratch and then develop a framework that teachers will then be able to use for themselves to introduce game making into their class without the researcher being directly involved in the class lessons in the final stage. After this study has been undertaken the results from that and the literature review will provide an implementation framework. While an initial literature search performed did not provide any implementation frameworks for GBCL by extending it to include ICT it then highlighted other frameworks and models within ICT in education that may be a suitable starting point. Table 2 provides an overview of implementation frameworks/models for ICT in education. Table 2: Implementation frameworks/models for ICT in education Author Kahn (2001)

Framework/Model A framework for web‐based training

De Freitas and Oliver (2006)

Four dimensional Framework

Tearle (2004)

Theoretical framework for identifying factors important for ICT implementation E‐capacity model

Vanderlinde and Braak (2011)

589

Purpose To help plan, design, develop and implement a learning environment Help practitioners evaluate the use of GBL within their particular learning context To identify important factors in relation to ICT implementation Designed to help with the integration of ICT process in schools

Amanda Wilson, Thomas Hainey and Thomas Connolly Kahn’s (2001) framework is designed to be used within education and training it has 8 dimensions pedagogical, technological, interface design, evaluation, management, resource support and ethical, and each dimension will vary in use according to the scope of learning taking place. The De Freitas and Oliver (2006) four dimensional framework looks at how educators can evaluate games in order to incorporate them into their own learning contexts. This framework is designed for educators to use in advance of their lesson planning. By looking at the particular context in which the learning is happening, the educator can take into account factors that may affect the lessons such as their own knowledge, technical support and access to tools. The second dimension takes the learner into account and focuses on attributes such as age and what stage of education they are at. Dimension three focuses on the game itself and the interactivity it has with the learner. Lastly the fourth dimension is pedagogical and this concerns how the game is embedded and supports the curriculum. Tearle (2004) proposes a theoretical framework that covers the influences on implementation of ICT in a school, by examining the school as a whole taking into account leadership, staff and internal processes. The implementation process then is considered with factors such as timing of the implementation, resources available, knowledge of ICT and support and training. Before looking at the individuals involved within the implementation which takes into account attitudes, belief and skills. Vanderlinde and Braak (2011) propose a conceptual model of e‐capacity that they see will help schools foster change through the optimisation of sustainable ICT. The model encompasses four layers that look at Leadership; this is seen to be an important factor in the success of any ICT implementation. Following on ICT related school conditions are considered this takes into consideration school policies, schools vision for ICT, support for ICT (including technical support) and having coordination of ICT. The next layer incorporates ICT related teacher conditions which encompasses professional development and teachers’ competence in ICT. Lastly the teachers’ actual use of ICT is seen as an extra layer and considers how teachers are making use of ICT in their classrooms. By reviewing these frameworks, key areas that should be included in an implementation framework are:

Leadership

Infrastructure

Professional development for teachers

Pedagogy.

3.2 Teacher survey A survey of primary teachers undertaken within 2 local council areas in Scotland (Razak et al, 2012) shows that while teachers are making use of GBL they are not making as much use of GBCL with only 9% of the 104 teachers stating they used both GBL and GBCL and 3% stating they used GBCL only. The survey also highlighted obstacles that teachers in Scotland face when trying to implement GBL in their class, which were lack of skills and training, lack of PCs and technology, difficulty identifying a suitable game and time and curriculum constraints. Other surveys on teachers show similar obstacles that are faced when trying to implement GBL/GBCL (see Table 3). Given the lack of literature surrounding GBCL in the classroom environment and implementation of GBCL in the classroom an initial study was undertaken to better inform a framework of implementation. Table 3: Obstacles faced by teachers Obstacles Lack of Technology Teacher Skills Suitability and Time constraints

Authors Williamson (2009) Stuart (2005), Sandford et al (2006) Wastiau, Kearney and Van den Berghe (2009)

3.3 Findings from study 1 The first study was undertaken in three different schools that all had different set ups with regards to their ICT suites and the ways in which lessons were delivered. Table 4 provides an overview of each school and their

590

Amanda Wilson, Thomas Hainey and Thomas Connolly differing ICT situations. Initial meetings with schools were set up with the schools head teacher after interest shown through the questionnaire sent to teachers. The meetings were used as a way for the researcher to inform the head teacher about the research and to evaluate what the schools were doing in terms of ICT work at present. The framework looks to address the barriers to implementation and give clear guidance on implementation lessons with Scratch in PE. From Study 1 it can be seen how these barriers can start to be addressed (see Table 5). Table 4: Comparison of schools Does the school have and ICT suite? How many PCs within the suite? Do the classrooms have PC’s? How many PCs in each class? Do teachers have access to laptops? How many laptops does the school have? Class sizes Does the school have dedicated ICT teacher Who is responsible for ICT lessons? Does the school have timetabled ICT lessons Does the school have ICT lesson plans? Are the teachers aware of CfE in relation to GBCL? Have teachers had any experience of GBCL? How does the head teacher feel about implementing GBCL? How do teachers feel about implementing GBCL?

School A Yes 20 Yes 1 Yes 14 – all teachers

School C No n/a Yes 2 Yes 7 teachers

25 Yes

School B Yes 9 Yes 2 Yes 9 – 2 in ICT room and 7 teachers 21 No

ICT teacher Yes

Class teacher No

Yes No

In progress No

Keen

Keen – some reservations in teachers confidence levels

Reservations due to lack of technology

27 No

Table 5: Addressing barriers to implementation Barrier Lack of technology

Issues noted during Study 1 3 different settings, gives good range to identify minimum requirements.

Teacher skills

Confidence in abilities Lack of knowledge of CfE with GBCL Lack of knowledge of GBCL tools Curriculum requirements – planning with staff/management Schools with no timetable were flexible, school that was timetabled changed timetable to suit.

Suitability Time Constraints

Overcoming the barrier Audit of equipment available in school Advanced planning CPD sessions Guidelines for implementing Scratch School ICT plans School ICT plans Advanced Planning

4. GBCL implementation framework Based on the teacher questionnaire, observational findings from Study 1 and the literature review, an initial model and framework for implementation of GBCL within the ICT curriculum is presented in Figure 1 and table 6. This model shows the various stages for best practise when schools would like to introduce GBCL into the curriculum and represents a high‐level summary of the framework presented in table 6, which provides a more detailed view of implementation of GBCL within the ICT curriculum. The model has seven stages school audit, GBCL tools, pedagogical develop, implement, evaluate and review. Overall, the implementation is led by head teachers and at each stage they should be showing strong leadership in order to encourage staff and progress with the implementation. The school audit phase addresses an individual school’s needs. It allows head teachers to assess what technology they have available to them and how best in can be utilised. This is extremely important – all schools have different setups when it comes to ICT suites as seen from the three schools within the initial study. School C is a school that, like a lot of

591

Amanda Wilson, Thomas Hainey and Thomas Connolly other schools, has the minimum that education services provided them with (2 PCs per class and a laptop for each staff member), however, with sufficient advance planning the school was able to ensure the class had the equipment for use on the day. Current ICT curriculum also needs to be taken account of, what are teachers doing currently in school with classes and how does this fit in with CfE and GBCL. Teachers’ skills need to be examined – do they have any experience of using GBCL tools and are they keen to use GBCL tools. On occasion external factors may also influence what is happening in school.

Figure 1: GBCL implementation model Once there is a clear picture of what is available in terms of skills and technology the next step is to find a suitable games construction tool to use within the class. This will be determined by the freedom schools have within their school network. Schools generally are limited by local authority policies and this has an impact on whether they are able to install their own software or have to go through a process within a managed network of getting software approved and installed. However, the latter option can be time consuming and costly for schools. Web‐based tools may be an option although again this relies on the infrastructure being able to support it. During Study 1, Scratch was available on one of the school’s networks and it was at that time possible to install it on the other two schools’ networks. Given the age of the classes that were undertaking the study (P4‐P7, age 8‐12 years) it was felt that Scratch was the most suitable tool to use. Step 3 considers the pedagogical aspects of implementation. Goals and objectives have to be established from the start, ideally within the school’s ICT plans with all staff having an input. Detailed lesson plans would take into account the number of available PCs/laptops the number of children in the class and what goals the teacher has for the lessons as well as curricular outcomes; e.g. are they going to make games with a particular theme (a topic for instance they are working on), will it be for another class in the school or simply to develop skills (collaborative working in primary is greatly encouraged. When a suitable tool has been selected, the next step is continuous professional development (CPD). If teachers’ skills need to be developed then CPD sessions may be required as well as detailed guidelines for use

592

Amanda Wilson, Thomas Hainey and Thomas Connolly of the tool in the class. This requires support and encouragement from head teachers in order to allow their staff the necessary time to become acquainted with the chosen game construction tool. Once teachers have addressed this, the lessons can then be implemented. Initial observations showed the confidence of teachers in their own abilities at the start of the researcher‐led lessons was low. However, after a couple of lessons and observing the confidence the children were showing through their work and collaborative working skills this encouraged the teachers and they were more participative within the lessons from then on. Initially the researcher led lessons comprised of a 5 minute talk from the researcher that then allowed the children around 40 minutes on the PC working on their game. From working alongside teachers of classes from Primary 4 through to Primary 7, teachers felt that this may not suit all classes particularly with the younger children and that possibly a more rigid approach was required. Aspects of the lessons at all ages required modifying and it was important to ensure progress was being made and that children were not spending too much time focussing on one particular aspect of the game they were creating. Observations by the researcher noted that sometimes a lot of time could be spent on the design stage of the game and while a lot of effort was going into the design no progress was being made with the actual functionality of the game. It is suggested that initially children should spend some time working on the design but they should have functionality and then when they have a functional game they could go back and modify the design of the game. Support from head teachers during the implementation is required in order for lessons to run smoothly particularly when situations arise, such as in school C when teacher laptops have to be utilised clear guidelines must be given and fellow teachers should be supportive of the teacher undertaking the lessons. It was noted in observations by the researcher that while teachers in school C sent their laptops along to the classes for use they had not been following the guidelines given to them by the head teacher – as such some laptops were inaccessible due to being locked by a teacher or they weren’t charged enough to last for the one hour lesson. After the lessons have been implemented an evaluation of the work has to be undertaken. This is important to let teachers see how the children are progressing and if they are meeting their learning intentions. Assessment can be undertaken by the children themselves either by peer assessment of the games or by self assessment. The assessments would be developed before any lessons take place and incorporate the learning intentions of the class teacher. The final stage of the model after the lessons have taken place is to review the implementation. This examines how teachers feel about the lessons and how the children did in the lessons. After Study 1 had been undertaken the teachers who had participated felt much more confident about their abilities. They felt that the children did get a lot out of the lessons though felt that lessons could be modified to suit the class that they were working with. Children also enjoyed the lessons and felt more confident in their abilities with game construction. Table 6: GBCL implementation framework 1) School audit Current ICT curriculum in school ICT infrastructure Teachers ICT skills

External influences 2) GBCL tools Tools available Staff learning 3) Pedagogy Context Detailed lesson plans

Points to consider It is important for all teachers to be aware of the CfE Experiences and Outcomes relating to ICT in particular game construction. Schools should be aware of what they have available to them and when there is It is important to discuss with teachers what skills they have and what skills they would like to develop. It is also important teachers are aware of game construction within the curriculum, This would include visits from Inspectors which can have an impact on school policy If the schools are on a managed network are they able to access any of these tools and at what cost to the school? Are they web based or do they need to be installed? Is there going to be a big learning curve for staff when introducing a new tool What tool is going to be used and how is it going to be embedded within the curriculum? Guides on using GBCL will help teachers become more comfortable

593

Amanda Wilson, Thomas Hainey and Thomas Connolly 1) School audit

4) Professional development

Points to consider when teaching they will give them a detailed introduction and show the main points often raised within lessons.

Teacher skills

Teachers should be comfortable using tools in lessons to construct games if they are not the head teacher has to put CPD in place for teachers to enhance their skills.

5) Implementation

Look to engage teachers

While only one or two teachers from the school may be participating it is useful for other teachers to support their colleagues.

Lessons to suit age and stage

Teachers will have to plan lessons according to the age and stage they are teaching.

Support of management

Head teacher should be encouraging teachers to ensure support for any teachers leading lessons this includes infrastructure problems and ensuring any problems are dealt with accordingly.

6) Evaluation

Game evaluation

Clear learning intentions – are they being met when the final game has been constructed.

Peer assessment

Peer review of games – do they hold interest of the person playing

Self assessment

Children evaluating their own games by way of statements and providing their assessment of how they did

7) Review

Teacher feedback

Reviewing lessons and making changes as necessary for teaching again. Teachers’ confidence in delivering lessons should be growing and skills continually improving.

Children’s feedback

Children reviewing lessons to help teachers make changes for future lessons.

5. 5 Discussion and future work By examining the literature it is clear that GBCL is only starting to become more widely used within PE. However, teachers feel there are many obstacles they face when trying to implement GBCL in school. The study undertaken within the three schools looked at areas such as the schools infrastructure and what software could run on the schools PCs. The lessons were then incorporated into the curriculum to help overcome what teachers felt were the barriers stopping the implementation of GBCL in school. The implementation framework presented has been developed to give PE establishments’ guidance on the implementation of GBCL within their school. It has shown the barriers that can be overcome to implement GBCL namely lack of technology, suitability and the teachers’ own skills. Further studies will involve PE establishments implementing the framework within their school.

Acknowledgements We would like to thank the staff and children of Royston Primary, Carntyne Primary and Alexandra Parade Primary in Glasgow for their participation in the study. This paper is a scientific publication as part of the Games and Learning Alliance (GaLA). GaLA is a Network of Excellence on ‗serious games‘ funded by the European Union in FP7 – IST ICT, Technology Enhanced Learning (see http://www.galanoe.eu). GaLA gathers the cutting‐edge European research and development organizations on ‘serious games‘, involving 31 partners from 14 countries. The University of the West of Scotland is one of the partners.

References Baytak, A. (2009). An investigation of the artifacts, outcomes, and processes of constructing computer games about environmental science in a fifth grade science classroom (Doctoral dissertation, The Pennsylvania State University). Baytak, A. and Land, S. (2011). An investigation of the artifacts and process of constructing computers games about environmental science in a fifth grade classroom Educational Technology Research and Development Springer Boston Vol 59 issue 7 pp 765‐782

594

Amanda Wilson, Thomas Hainey and Thomas Connolly de Freitas, S. and Oliver, M. (2006). How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Computers and Education 46 (3), pp. 249‐264. ISSN 0360‐1315. Groff, J., Howells, C. and Cranmer, S. (2010). The impact of console games in the classroom: Evidence from schools in Scotland. Futurelab, UK. Kafai, Y. B. (1995). Minds in play: Computer game design as a context for children's learning. Routledge. Kafai, Y. B., Franke, M. L., Ching, C. C. and Shih, J. C. (1998). Game design as an interactive learning environment for fostering students' and teachers' mathematical inquiry. International Journal of Computers for Mathematical Learning, 3(2), 149‐184. Kafai, Y. B., and Ching, C. C. (1996). Meaningful contexts for mathematical learning: The potential of game making activities. In Proceedings of the 1996 international conference on Learning sciences (pp. 164‐171). International Society of the Learning Sciences. Kafai, Y. B. (2006). Playing and Making Games for Learning: Instructionist and Constructionist Perspectives for Game Studies. Games and Culture 1: 36‐40. Kelleher, C., Pausch, R. and Kiesler, S. (2007). Storytelling alice motivates middle school girls to learn computer programming, ACM, San Jose, California, USA. Khan, B. H. (2001). Web‐based training. Englewood Cliffs, NJ: Educational Technology Publications. Kirriemuir, J. and McFarlane, A. (2004), 'Literature Review in Games and Learning', Nesta Futurelab series, Bristol. Koutsikos, L., Holeva, V., Zourelidis, S., Dova, M. and Patrikakis, C. (2012). Information and Communication Technology in th Greek Primary Schools: A Pilot Application In Proceedings of the 6 European Conference on Games Based Learning. Cork, Ireland, 4‐5 October 2012 McClarty, K. L., Orr, A., Frey, P.M., Dolan, R.P., Vassileva, V. and McVay, A. (2012). A Literature Review of Gaming in Education, Pearson Meluso, A., Zheng, M., Spires, H. A. and Lester, J. (2012). Enhancing 5th graders’ science content knowledge and self‐ efficacy through game‐based learning. Computers and Education, 59(2), 497. Mendes, L. and Romão, T. (2011). Children as teachers: a tangible approach. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology (p. 12). ACM. Papert, S. (1980). Mindstorms: children, computers, and powerful ideas, Basic Books, Inc. Papert, S. (1991). Situating Constructionism. In I. Harel & S. Papert (Eds.) Constructionism Norwood, NJ, Ablex Publishing. Pivec, P. and Pivec, M. (2008). Games in Schools, report on final literature review ISFE‐EUN Partnership. Razak, A. A., Connolly, T. M., Baxter, G. J., Hainey, T. and Wilson, A. (2012). The Use of Games‐based Learning at Primary th Education Level within the Curriculum for Excellence: A Combined Result of Two Regional Teacher Surveys, 6 European Conference on Games‐based Learning (ECGBL), 4‐5 October 2012, Cork, Ireland. Resnick, M., Kafai, Y. and Maeda, J. (2003). A Networked, Media‐Rich Programming Environment to Enhance Technological Fluency at After‐School Centers in Economically‐Disadvantage Communities., United States. Robertson, J. and Good, J. (2005). Adventure Author: An Authoring Tool for 3D Virtual Reality Story Construction. In Proceedings of the AIED‐05 Workshop on Narrative Learning Environments, 63‐69. Robertson, J. and Howells, C. (2008). Computer game design: Opportunities for successful learning, Comput.Educ., Vol. 50, No. 2, pp. 559‐578. Robertson, J. (2012). Making games in the classroom: Benefits and gender concerns. Computers & Education, 59(2), 385‐ 398. Sandford, R., Ulicsak, M., Facer, K. and Rudd, T. (2006). Teaching with Games: Using commercial off‐the‐shelf computer games in formal education, (Bristol: Futurelab). Stuart, R. (2005). Learning Through ICT, Glasgow City Council, Glasgow. Tang, S., Hanneghan, M. and El Rhalibi, A. (2009). 'Introduction to Games‐Based Learning', in Liz Boyle Mark Stansfield, Thomas Connolly (ed.), Games‐Based Learning Advancements for Multi‐Sensory Human Computer Interfaces: Techniques and Effective Practices (IGI Global), 1‐77. Tearle, P. (2004). A theoretical and instrumental framework for implementing change in ICT in education. Cambridge Journal of Education, 34(3), 331‐351. Vanderlinde, R., and van Braak, J. (2010). The e‐capacity of primary schools: Development of a conceptual model and scale construction from a school improvement perspective. Computers & Education, 55(2), 541‐553. Van Eck, R. (2006). Where do we go from here? Ten critical areas to guide future research in DGBL. Workshop for the annual meeting of the Games, Learning, and Society Conference, June 15–16, Madison, WI. Vos, N., van der Meijden, H. and Denessen, E. (2011). Effects of constructing versus playing an educational game on student motivation and deep learning strategy use. Computers & Education, 56(1), 127‐137. Wastiau, P., Kearney, C. and Van den Berghe, W. (2009). How are digital games used in Schools? Main results of the study, Synthesis Report European Schoolnet, EUN Partnership AISBL. Williamson, B. (2009). Computer games, schools, and young people: A report for educators on using games for learning. Futurelab, UK. Wilson, A. Hainey, T. and Connolly, T. (2012). Evaluation of Computer Games Developed by Primary School Children to th Gauge Understanding of Programming Concepts. In Proceedings of the 6 European Conference on Games Based Learning. Cork, Ireland, 4‐5 October 2012

595

596

PhD Research Papers

597

598

Game Literacy Revisited: Developing Critical Play in Schools Rafael Marques de Albuquerque1 and Shaaron Ainsworth2 1 PhD student funded by CNPq – Brazil 1, 2 Learning Sciences Research Institute, School of Education, University of Nottingham, Nottingham, UK Lpxrmd@nottingham.ac.uk Shaaron.Ainsworth@nottingham.ac.uk Abstract: Digital games, as other technologies, are evaluated as monsters or heroes from different researchers. Such effects are not caused by digital games per se, but by the interaction between player and digital game. The objective of media education is to improve this interaction, what is also described as developing media literacy, and in the specific case of digital games, game literacy. This paper proposes a new concept of game literacy in order to inspire an alternative game education practice within media education programmes. Using the media education and game literacy literature as a basis, we integrated the work of the critical pedagogue Paulo Freire (1970/2012) and game studies that claim both positive (e.g. system thinking, problem solving) and negative (e.g. addictive behaviour, violent affection) effects of gaming. Seven characteristics shape the proposal: (i) The work of Paulo Freire emphasises criticality, rather than previous studies that proposed an emphasis on creative, cultural or protective approaches to media. (ii) The inclusion of the potential negative effects of digital games that are pointed by researchers in classroom discussions, considering that criticality towards the negative effects may minimize them. (iii) The inclusion of positive usage of digital games claimed by researchers, considering that developing agency and criticality towards the positive potential may support learners to reach them. (iv) Regarding positive and negative effects, the proposal considers the agency of learners to understand, judge and act on their own game habits, rather than imposing gaming rules or values. (v) The curriculum is created in a dialogic process; meaning that both educationists and students collaborate on defining the relevant themes. (vi) The educational practice involves learners’ experiences with digital games, in order to enable them to transfer their criticality to their actual practice. (vii) Methodologically, the teaching method is based on coded learning objects that are decoded by learners in a process of problematization, that leads to experimentation in gaming habits and, possibly, change. Keywords: game literacy, game education, media education, digital games, critical play

1. Introduction Over the last decades digital games and education have seemed to be at war. On one side of the war, game designers want to develop incredibly engaging and profitable games. On the other side, media scholars, educationist and parents try to protect children against the potential harm of playing games such as violence, addiction, stereotypes, isolation and obesity. This war has also manifested in academia, where many publications present evidence of the harms of games, whereas others claim benefits of games such as complex problem solving, contact with highly contextualised rich simulated environments, learning through mistakes, development of design skills, and wider “life lessons” such as optimism, self‐esteem, tolerance etc. This paper discusses the potential role of schools in this war. Media education argues that schools should prepare students to develop media literacy and digital games are included in its programme. This paper proposes a concept of game literacy that aims to inspire classroom practice of game literacy teaching in schools, and that merges three perspectives: (i) some of the existing literature of game literacy, (ii) the work of the critical pedagogue Paulo Freire, and (iii) the research claims regarding the effects of gaming – including the positive and the negative. This proposal is both theoretical – by developing an understanding of what could be considered a game literate individual – and pragmatic, because the concept is methodologically oriented towards practice.

2. Literature review Four sections are presented in this review. The first and the second present, respectively, the claims regarding negative and positive effects of gaming. These ideas are presented because our concept of game literacy includes awareness of both its potential for eliciting positive and negative effects, based on the assumption that digital games are not good or bad per se, but depending of the nature of their interaction with players. Rather than reaching final conclusions based on this review, our intent is to clarify some of the topics that could be discussed in classrooms in media education practice, and also to make evident the relevance of such

599

Rafael Marques de Albuquerque and Shaaron Ainsworth intervention. The third and fourth sections present aspects of media education and game literacy literatures in order to clarify the background of our proposal.

2.1 Dangerous gaming? According to the survey in US with 1,102 teenagers carried out by Lenhart et al. (2008), 97% of teenagers (12‐ 17 years old) play computer, web, portable, or console games. The Kaiser Family Foundation study (2010) with more than 2,000 teenagers in the US found that, on average, each one spend 90 minutes playing digital games in a typical day (73 minutes on consoles and 17 minutes in computer games). As games and digital technologies are becoming more ubiquitous, these numbers have been increasing in the recent years. Consequently, it is only natural that scholars (as well as parents and teachers) are worried about the possible harm that such frequent engagement with digital games could cause. Violence within digital games is probably one of the biggest of these concerns. Researchers have reviewed a number of studies – both correlational and experimental – and concluded that the exposure to violent games is causally related to heightened levels of aggression, including aggressive behaviour, cognition and affection, as well as decreased empathy and prosocial behaviour (Anderson and Bushman, 2001; Anderson et al., 2010). However, this claimed link is questioned by others. The studies that associate games and violence, however, were also severely criticised. Based on the work of Ritter and Eslea (2005), Jones (2005) and Ferguson (2010a), some of the main critiques are: (i) the tests used in violence research usually present a situation that is suggestive for aggression for the participant. Violence studies usually (ii) ignore the perceptions and motivations of participants and (iii) consider artificial situations that are not real violence – such as rough‐and‐tumble play – as evidence of actual violence. Furthermore, some studies (iv) ignore the possible benefits of fictional violence in digital games such as psychological development, self‐esteem and wellbeing. Jones and Ferguson also discuss the exaggerated panic of the violence in games in terms of society, and not as an isolated phenomenon within individuals. Moreover, studies that tried to conduct more realistic research found no evidence of relationship between violent game exposure and aggressive affect (Valadez and Ferguson, 2012). Ferguson (2010b) also found that other important variables – for example, depressive symptoms – were not considered in previous studies and may be better predictors of violence than violent games usage. There are other concerns about the potential harm of playing digital games. These include stereotyped representation or underrepresentation regarding gender (Brenick and Henning, 2007) and ethnicity (Burgess et al. 2011) which may strengthen discriminatory conceptions of players (Kirsh, 2010). Another concern is addictive behaviour towards gaming (Grusser et al. 2007). Although Skoric et al. (2009) found no relationship between game addiction and lower academic performance of students, it stills require further discussion, mainly in an individual level. Furthermore, media usage and obesity are also associated (Strasburger, 2010), although television probably has a more significant impact than gaming on this issue. In summary, it seems that the simple and straightforward fears of early studies concerning and automatic causal link with violence may be overstated. However, we should not dismiss the potential that playing digital games can have harmful effects.

2.2 Glorious gaming? In his book, Johnson (2005) describes a shift in the paradigm about the effects of video games, from something bad towards something good. It is far from a homogeneous position amongst scholars, but several studies suggest that digital games have the potential to be beneficial and, in some cases, there is empirical evidence of these claimed benefits. According to Gee (2007), good games are intellectually challenging, they stimulate players to think about interactions between variables in complex systems replete of contextualized information to be articulated. Based on the uncertainty of the dynamics of the challenge, it was claimed that some games offer the player the chance to develop a behaviour of enlightened risk taking and entrepreneurship (Shaffer et al., 2005; Gee, 2007), that includes generation of innovative solutions (Shaffer, 2006). This is not only because such games reward these kind of behaviour, but also because players have the chance to learn how to learn from their mistakes (Gee, 2007). Players also have to learn how to manage different levels of goals, hypothetically

600

Rafael Marques de Albuquerque and Shaaron Ainsworth creating a sense of priority and management (Johnson, 2005; Gee, 2007). Gee (2007) and Shaffer (2006) claimed that some game experiences have a similar reasoning to scientific enquiry, and the different ways to manage knowledge and perform different identities, e.g. roleplaying a city manager or an airplane pilot, enable diverse learning experiences. Digital games can be an ethical experience as well. Simkins and Steinkuehler (2008) found that roleplaying games have the potential to be used as spaces of practice of ethical decisions, in which critical decision making might flourish. According to the reviews of Greitemeyer (2011) and Gentile et al. (2009), several studies – including correlational, longitudinal and experimental studies – presented evidence that games with prosocial content may affect both cognitive and affective variables, increasing prosocial outcomes and decreasing antisocial outcomes. Lenhart et al. (2008) found that the kind of games that teenagers play is strongly related to teens’ interest and engagement in civic and political activities. The context surrounding digital games may also be beneficial. Digital games offer an environment to develop effective social practices (Shaffer et al. 2005). Based on an ethnographic study, surveys and experiments, Steinkuehler and Williams (2006) suggested that online games may replace other socialization places, offering the opportunity to develop bridging social capital, and more rarely, even bonding social capital. Moreover, players have the chance to engage in literacy practice in online forums and websites related to gaming (Steinkuehler, 2007; 2010; Thorne et al., 2009), and in countries where English is not the first language, there is evidence and self‐reports of English learning by gaming (Skoric et al. 2009; Cruz et al. 2012; Vintetjärn, 2008). Players that choose to get involved in map creation, modding games, game creation, fan art or fan fiction, also develop several skills in order to produce it, such as writing, design, art creation and programming (Squire, 2011). Evidence of cognitive improvement from digital gaming is also available. Spence and Feng (2010) published a review in which empirical studies suggest that gaming may improve: (i) spatial cognition, (ii) size of the attentional visual field, (iii) visual spatial resolution, (iv) contrast sensitivity, (v) visuomotor coordination/speed, and (vi) visual memory recall. Moreover, Dye et al. (2009) stated that evidence showed that action game practice may reduce reaction times without sacrificing accuracy. Although it may sound very optimistic and the authors did comment that research points to several cognitive benefits of gaming, they also remind us that it varies immensely to each game and game genre, and that more research is needed. All these benefits are more likely to be achieved in some contexts, with some games and with some players than others. Consequently, we argue there is a role that school can play to support students to have a more positive and less harmful use of digital games. The context for improvement of the interaction between student and media is media education.

2.3 Media education The appeal of media education is based on the importance that media has on young people’s lives, and it can make schooling more relevant to pupils’ lives. Media does not only influence a citizen’s relationship with information and news, but also tastes and judgments, pleasure, and expressions of identity (Burn and Durran, 2007). The main focus of media education is to change the interaction between learners and media, and not to use media to teach, which is another interesting initiative, but not media education (Buckingham, 2003). Consequently, the use of media for teaching – as educational/serious games, for example – is not discussed in this paper. The qualities of the interaction with media that are usually desired in media education are creative, critical and cultural. The creative aspect comes from the original meaning of literacy – that is considered the ability to read and write, therefore, to produce. Buckingham (2003) emphasizes that creative engagement with media is related to an active citizenship; because such approach towards media gives learners the possibility to share ideas, and not only consume them. The critical approach to media suggests that learners should develop mindfulness towards media, rather than automaticity (Potter, 2004). It also implies that the consumer should be capable of reflecting about what is behind media – such as political and commercial interests. The cultural approach suggests that learners can understand games as cultural artefacts that are embedded in other aspects of culture (Burn and Durran, 2007),

601

Rafael Marques de Albuquerque and Shaaron Ainsworth and that popular culture such as digital games is also culturally valuable (Buckingham, 2003). The inclusion of games within media education programmes, however, requires specific discussion.

2.4 Game literacy There are a few published empirical studies exploring game literacy. For example, Squire (2008) presented ‘good’ games to students, asked them to play and debriefed the experience, discussing how games are made, how they represent real world contexts, and how they involve a mix of expertise and art expressions such as art, music and programming. The game, in this context, was carefully selected and would be an example with the function to illustrate and inspire further discussion. Following the logic that game literacy includes being creative in regard to games some perspectives proposed that game creation activities in schools could be the basis of game literacy development. The experience described in Burn and Durran (2007) and Buckingham and Burn (2007) enabled students to create a game. The assumption is that by carrying out the role of designers and developers of games in an environment supported by teachers, students would develop a critical and cultural understanding of games. Zimmerman (2007) and Owston et al (2009) also related game literacy development through game creation, but they proposed that game creation is beneficial because it develops other skills. The former claims that it may develop a sense of system, play and design, and the latter, that game creation develops the skills of writing, searching for relevant information, questioning and editing. However, we query the reliance on game creation. Game literacy is sometimes defined by what it should be – based on previous concepts of literacy, digital literacy (Zwieten, 2012) or media literacy (Burn and Durran, 2007; Buckingham and Burn, 2007), and it implies that learners should be able to create digital games in order to be game literate. We question this assumption, asking if game creation is a desired outcome to the same degree that the creation of other kinds of media such as video and websites. Consequently, we focus on developing practices that can be detached from or sequenced with game creation. We also suggest that game education should avoid relying on games that are unknown to learners, and prefer to work with the games that students play at home. This is consistent with the aim to transform schooling into something more relevant for students. Accordingly, it could be expected that the inclusion of these games in classroom would improve the chances that students transfer their new understandings to actual gaming habits in their homes. The emphasis on the actual game practice of students also inspired the perspective of Klimmt (2010), however in different ways. He had suggested that game literacy should look forward the formation of normative players by teaching strategies that would allow them to avoid the potential harm of games – such as aggressive and discriminatory behaviour and game addiction. Protective approaches to media literacy are common in media education history and are nonetheless severely criticised by Buckingham (2003), Potter (2004) and Gutiérrez and Tyner (2012) mainly for (i) victimising students, treating them as vulnerable, defenceless and uncritical individuals, for (ii) ignoring all the potential benefits of media usage, and (iii) for trying to impose external values and procedures. Accordingly, we agree with his emphasis on everyday games whilst disagreeing with the desire to inform normative play. Approaches aiming to develop a more positive usage of existing games have also been proposed. Partington (2010) experimented the creation of what he called Games and Me posters in an English class. In the posters students made explicit their game knowledge and the relevance of games in their lives. The intent was to avoid working with some kind of canon of games, focusing in the cultural background of the students and their experiences, in order to “extend students’ beyond their experiences, so that they are able to apply them independently in other contexts” (p. 85). In the context of high education Zagal (2010) developed two techniques in order to improve the understanding that students have of games. One of these approaches was the GameLog website, which mixed the ideas of electronic portfolio and blogging in order to support students in reflecting about their gaming experiences. The game literacy proposals of Partington (2010) and Zagal (2010) are the closest to my proposal, because they work with the games that learners spontaneously play in their leisure time. However, the former proposed isolated activities that are not included in a wider plan or game literacy and media education, and

602

Rafael Marques de Albuquerque and Shaaron Ainsworth the latter aimed to prepare game scholars and professionals of the game industry. Our proposal is a more extensive plan of activities, and is designed for schools. Moreover, the theoretical emphasis is different, as we aim to merge critical pedagogy and media education into a more complete proposal of game literate citizen.

3. Game literacy revisited This paper proposes a concept of game literacy that can be used to design school practice, probably within media education programmes. We build on previous concepts of game literacy and media literacy, and add two main theoretical contributions: the work of the critical pedagogue Paulo Freire and the game literature that has researched the effects and potentials of commercial games on players. Paulo Freire discussed media very briefly, and when he did so, his emphasis was on television and radio (Freire and Guimarães, 2011), not digital games. The context in which he developed his work was by teaching literacy for adults in poor regions in Brazil, and it is very different from the context we are proposing in this paper; therefore, we will draw on his work, but are aware of the need for adaptation both in his theoretical and practical contributions.

3.1 Game literate individuals To be able to develop a pragmatic concept of game literacy applicable in school contexts, we need to understand what is desired from the student, as a citizen, regarding his/her relationship with digital games. This decision is a moral one; and we want to draw on the idea of Potter (2004) when he suggests that the only value of media education is criticality. It means that rather than teaching what learners should do with their games, they shall receive opportunities to develop their criticality towards it, thus, being able to actively decide how they use media. To Freire (1970/2012) it means to respect of the potential of human beings of being better, of using their own will and intelligence to act in their life and change it accordingly. The potential harm of digital games, in this context, is prevented by criticality, not submission. It means that all the potential harm of digital games should not be “solved” by repression or by obedience to rules, and not ignored either, but problematized and contextualised into the students’ lives, empowering them with the agency to deal with potential risks. The potential benefits of digital games are also presented and experienced in classrooms as possibilities, not guidelines. As Gee (2007) claims, an active and critical approach to games may enable players to develop a very beneficial gaming habit, and although family, school, friends or media may stimulate a more critical play, it is a decision of the learner. The literate player, thus, doesn’t necessarily develop games, write fan fiction or create deep social relationships in online games, for example. Neither s/he necessarily chooses the most complex games, nor must have the most restricted gaming hours. However, the literate player is aware of the possibilities, is able to critically analyse games and the roles that games may play in one’s life, and has agency to change his/her reality according to his/her will and actual possibilities, understanding the consequences of such choices. Game literacy is one specific aspect of a critical subject in general. It is unlikely that game education alone will develop complete critical subjects. However, the formation of critical subjects is a goal that may be embedded in diverse aspects of life, including other school spaces, family, politics and possibly also game design and development (Frasca, 2001). This paper only proposes what media education could do in order to bring about greater criticality towards game practice. Hopefully this work shall contribute with other initiatives towards the formation of critical subjects.

3.2 Game literacy methodology This proposal is oriented toward practice and action in schools. To Freire (1970/2012), the dialogic nature of education means an exchangeable relationship between theory and practice and between schooling and daily life of learners. It means that the boundaries between everyday experience of learners and school practice shall be blurred (Lewin, 2004), and that formal and informal learning become related, not dichotomous (Kent and Facer, 2004). In the experiential learning model of Kolb (1984), school moment would be a moment of abstract conceptualization that stimulates the active experimentation, concrete experience and reflective observation at home. Kiili (2005) also proposed that experiential learning could be used to conceptualise game

603

Rafael Marques de Albuquerque and Shaaron Ainsworth based learning, but suggested that reflective observation and abstract conceptualization should be supported by computer based tutors in educational games. We propose that this role could be played by teachers to change the relationship that students have with commercial games. Ideally, the dynamic of experiential learning would be incorporated by students and they would be able to engage in the whole process by their selves, at home, reflecting and learning critically through gaming. The curriculum, according to Freire (1970/2012), should be a dialogic process that empowers learners to contribute with relevant aspects of their lives (Pelletier, 2009), as well as the aspects that educationists believe that are important. The dialogic definition of the curriculum aims to design educational processes that are oriented towards action and change, in which the school experience is transferred to daily life. To educate is to create consciousness of the learners’ relationship with the world and not the world alone (Freire, 1970/2012), and in this case, it means the relationship with digital games. It is the rationale for use the games that learners play, rather than emphasise the presentation of alien games or software that will probably be accessed only in the classroom environment, during the course duration. The teaching method, based on Freire (Ibid.), includes the production of coded learning objects that represents the relevant topics, called generative themes. Some examples of themes are violence in games, or games as accurate simulations of reality. The generative themes are, however, coded, meaning that they are presented as a situation‐problem, as a challenge to be understood, and the learner perceives it as an external situation. Through the discussion about the topic, the learning object is decoded and the theme is problematized. It enables the learners to see themselves as included in the theme that seemed to be external and different from their reality. It offers the chance to perceive their realities from a distance, therefore, to critically analyse it, and to relate it to their own experiences. Afterwards, the discussion leads into action and practice towards change, and in this case, towards active experimentation, concrete experience and reflective observation (Kolb, 1984), that is, basically, critical play. Critical play can allow learners to understand that their relationship with digital games is a relevant aspect of their lives that sets up positive and negative possibilities, which depend on the learners themselves. By stimulating learners to engage critically with digital games, possibly more learners will benefit from gaming, and less will find the habit to be harmful.

4. Final considerations In this paper we presented the first model of a new proposal of game literacy that is oriented towards action within media education. It approximates media education, media effects studies and the work of the critical pedagogue Paulo Freire. This proposal comes to develop – or at least to present an alternative to – the role of digital games within media education, and will be carried out in the next years and enriched with empirical studies that will collaborate towards a more mature theoretical basis and work proposal.

Acknowledgements This project is financially supported by CNPq, National Council for Scientific and Technological Development ‐ Brazil

References Anderson, C. A., Bushman, B. J. (2001). "Effects of Violent Video Games on Aggressive Behavior, Aggressive Cognition, Aggressive Affect, Physiological Arousal, and Prosocial Behavior: A Meta‐Analytic Review of the Scientific Literature." Psychological Science 12(5): 353‐359. Anderson, C. A., A. Shibuya, et al. (2010). "Violent Video Game Effects on Aggression, Empathy, and Prosocial Behavior in Eastern and Western Countries: A Meta‐Analytic Review." Psychological Bulletin 136(2): 151‐173. Brenick, A., A. Henning, et al. (2007). "Social Evaluations of Stereotypic Images in Video Games: Unfair, Legitimate, or ''Just Entertainment''?" Youth & Society 38(4): 395‐419. Buckingham, D. (1996). "Critical pedagogy and media education: a theory in search of a practice." Journal of Curriculum Studies 28(6): 627‐650. Buckingham, D. (2003). Media Education: literacy, learning and contemporary culture. Cambridge, Polity Press. Buckingham, D. and A. Burn (2007). "Game Literacy in Theory and Practice." Journal of Educational Multimedia and Hypermedia 16(3): 323‐349. Burgess, M. C. R., K. E. Dill, et al. (2011). "Playing With Prejudice: The Prevalence and Consequences of Racial Stereotypes in Video Games." Media Psychology 14(1): 289‐311.

604

Rafael Marques de Albuquerque and Shaaron Ainsworth Burn, A. and J. Durran (2007). Media Literacy in Schools: Practice, Production and Progression. London, Paul Chapman Publishing. Cruz, D. M., D. K. Ramos, et al. (2012). "Jogos eletrônicos e aprendizagem: o que as crianças e os jovens têm a dizer? Contrapontos 12(1): 87‐96. Ferguson, C. J. (2010A). "Blazing Angels or Resident Evil? Can Violent Video Games Be a Force for Good?" Review of General Psychology 14(2): 68‐81. Ferguson, C. J. (2010B). "Video games and youth violence: a prospective analysis in adolescents." Journal of youth and adolescence 40(4): 377‐391. Frasca, G. (2001). Videogames of the Opressed: Videogames as a means for critical thinking and debate. Master`, Georgia Institute of Technology. Freire, P. (1970/2012). Pedagogia do Oprimido. Rio de Janeiro, Nova Fronteira. Freire, P. and S. Guimarães (2011). Educar com a mídia: Novos diálogos sobre educação. São Paulo, Paz e Terra. Gee, J. P. (2007). What video games have to teach us about learning and literacy. New York, Palgrave MacMillan. Gentile, D. A., C. A. Anderson, et al. (2009). "The effects of prosocial video games on prosocial behaviors: international evidence from correlational, longitudinal, and experimental studies." Personality & social psychology bulletin 35(6): 752‐763. Greitemeyer, T. (2011). "Effects of Prosocial Media on Social Behavior: When and Why Does Media Exposure Affect Helping and Aggression?" Current Directions in Psychological Science 20(4): 251‐255. Gutiérrez, A. and K. Tyner (2012). "Educación para los medios, alfabetización mediática y competencia digital." Comunicar: Revista Científica de Educomunicación 38(19): 31‐39. Johnson, S. (2005). Everything Bad is Good for You: How Today's Popular Culture Is Actually Making Us Smarter. New York, Riverhead Books. Jones, G. (2003). Killing Monsters: Our Children's Need for Fantasy, Heroism and Make‐believe Violence. New York, Basic Books. Kaiser Family Foundation (2010). Generation M2: Media in the Lives of 8‐ to 18‐Years Olds, The Henry J. Kaiser Family Foundation: 85. Kent, N. and K. Facer (2004). "Different worlds? A comparison of young people's home and school ICT use." Journal of Computer Assisted Learning 20(1): 440‐455. Kiili, K. (2005). "Digital game‐based learning: Towards an experiential gaming model." Internet and Higher Education 8(1): 13‐24. Kirsh, S. J. (2010). Media and Youth: A developmental perspective. Chichester, Wiley‐Blackwell. Klimmt, C. (2009). "Key Dimensions of Contemporary Video Game Literacy: Towards A Normative Model of the Competent Digital Gamer." Eludamos. Journal for Computer Game Culture 3(1): 23‐31. Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, Prentice Hall. Lenhart, A., J. Kahne, et al. (2008). Teens, Video Games, and Civics. Washington, PEW INTERNET & AMERICAN LIFE PROJECT: 76. Lewin, C. (2004). "Access and use of technologies in the home in the UK: implications for the curriculum." Curriculum Journal 15(2): 139‐154. Owston, R., H. Wideman, et al. (2009). "Computer game development as a literacy activity." Computers & Education 53: 977‐989. Partington, A. (2010). "Game literacy, gaming cultures and media education." English Teaching: Practice and Critique 9(1): 73‐86. Pelletier, C. (2009). "Games and Learning: What's the Connection?" International Journal of Learning and Media 1(1): 83‐ 101. Potter, W. J. (2004). Theory of Media Literacy: A Cognitive Approach. Thousands Oaks (US), Sage Publications. Rittern, D. and M. Eslea (2005). "Hot Sauce, Toy Guns, and Graffiti: A Critical Account of Current Laboratory Aggression Paradigms." Aggressive Behavior 31(1): 407‐419. Shaffer, D. W. (2006). "Epistemic frames for epistemic games." Computers & Education 46(1): 223‐234. Shaffer, D. W., K. R. Squire, et al. (2005). "Video games and the Future of Learning." PHI DELTA KAPPAN 87(2): 105‐111. Simkins, D. W. and C. Steinkuehler (2008). "Critical Ethical Reasoning and Role‐Play." Games and Culture 3(3‐4): 333‐355. Spence, I. and J. Feng (2010). "Video games and spatial cognition." Review of General Psychology 14(2): 92‐104. Squire, K. (2008). Video‐game Literacy: A Literacy of Expertise. Handbook of Research on New Literacies. J. Coiro, M. Knobel, C. Lankshear and D. J. Leu. New York, Lawrence Erlbaum Associates Taylor & Francis Group: 635‐669. Squire, K. (2011). Video Games and Learning: Teaching and Participatory Culture in the Digital Age. New York, Teachers College Press. Steinkuehler, C. (2007). Massively Multiplayer Online Gaming as a Constellation of Literacy Practices. The Design and Use of Simulation Computer Games in Education. B. E. Shelton and D. A. Wiley. Rotterdam, Sense Publishers: 181‐208. Steinkuehler, C. (2010). "Video Game and Digital Literacies." Journal of Adolescent & Adult Literacy 54(1): 61‐63. Strasburger, V. C., A. B. Jordan, et al. (2010). "Health effects of media on children and adolescents." Pediatrics 125(4): 756‐ 767. Thorne, S. L., R. W. Black, et al. (2009). "Second language use, socialization, and learning in Internet interest communities and online gaming." The Modern Language Journal 93(Focus): 802‐820.

605

Rafael Marques de Albuquerque and Shaaron Ainsworth Valadez, J. J. and C. J. Ferguson (2012). "Just a game after all: Violent video game exposure and time spent playing effects on hostile feelings, depression, and visuospatial cognition." Computers in Human Behavior 28(1): 608‐616. Vintetjärn, B. (2008). Video Games – a Source of Knowledge? An investigation amongst youths about their thoughts on their video gaming habits. Master, Malmö University. Zagal, J. P. (2010). Ludoliteracy: Defining, Understanding and Supporting Games Education. ETC Press. Zimmerman, E. (2007). "Gaming literacy Game Design as a Model for Literacy in the 21st century." Harvard Interactive Media Review 1(1): 30‐35. Zwieten, M. v. (2012). Video Games Literacy: a Theoretical Framework. M. A. in New Media and Digital Culture, Utrecht University.

606

A Systematic Literature Review of Methodology Used to Measure Effectiveness in Digital Game‐Based Learning Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy iMinds, MICT, Ghent University, Ghent, Belgium Anissa.All@Ugent.be ElenaPatricia.NunezCastellar@Ugent.be J.Vanlooy@Ugent.be Abstract: In recent years, a growing number of studies is being conducted into the effectiveness of digital game‐based learning (DGBL). Despite this growing interest, however, it remains difficult to draw general conclusions due to the disparities in methods and reporting. Guidelines or a standardized procedure for conducting DGBL effectiveness research would allow to compare results across studies and provide well‐founded and more generalizable evidence for the impact of DGBL. This study presents a first step in this process by mapping current practices through a systematic literature review. The review included peer‐reviewed journal and conference publications between 2000 and 2012. Other inclusion criteria were that (1) the study’s primary aim was effectiveness measurement of cognitive learning outcomes, (2) the focus was on digital games and (3) a pre‐post design with a control group was used. Twenty‐five publications were found eligible for this study. Important differences were found in the number of control groups used and the type of intervention implemented in the control group (e.g. traditional classroom teaching, use of multimedia, computer‐based learning, paper exercises, other games, or no intervention). Regarding the implementation method of the DGBL intervention in the experimental group, two approaches can be distinguished: stand‐alone intervention or as part of a larger program. Moreover, a wide variety of effectiveness measures was used: measures for learning outcomes were complemented with time measurements and/or with self‐reported measurements for self‐efficacy and motivation. Learning effect calculation also varied, introducing pre‐test scores in the analysis, conducting a separate analysis on pre‐ and post‐test scores or conducting an analysis on difference scores. Our study thus indicates that a variety of methods is being used in DGBL effectiveness research opening a discussion regarding the potential and requirements for future procedural guidelines. Keywords: effectiveness, digital game‐based learning, cognitive learning outcomes

1. Introduction In recent years, attention for the use of digital games has grown in a wide range of sectors. Digital games that do not primarily aim at entertainment have been deployed in the field of education, health and wellbeing, government, NGOs, corporate, defence, marketing and communication (Sawyer and Smith 2008). This growing interest in digital game‐based learning (DGBL) has resulted in an increasing amount of publications on the topic (Michael and Chen 2005). One important aspect in this field of research is effectiveness measurement (Connolly et al. 2012) whereby effects of DGBL and contributing elements on learning outcomes are assessed. An important limitation in this field is the incongruity of study designs (Kharrazi et al. 2012), which makes comparison across studies problematic. A consistent approach in effectiveness measurement would create the possibility to map important aspects of effectiveness on a more general level. Furthermore, uniformity in effectiveness studies on DGBL would help us gain better insight in validity and reliability of single studies. The present study takes a first step in the development of standardized guidelines by mapping the methods currently being used in effectiveness research on DGBL.

1.1 Defining effectiveness In the literature, learning is often clarified on the basis of the generated outcomes (Gagne 1984). An effective instructional method can thus be described as a method which has a positive impact on learning achievement and therefore learning outcomes (Joy and Garcia 2000). An instructional method has been defined by Salomon (Cited by (Clark 1994) p. 23) as “Any way to shape information that activates, supplants or compensates for the cognitive processes necessary for achievement or motivation.” Effectiveness of an instructional method can thus refer to either learning outcomes and/or motivation. According to Salomon (1993) the relationship between a medium used to teach and learning is an interaction between cognitive processes and characteristics of the mediatized environment. Medium and learning content are therefore inherently connected, implying that characteristics of the medium can influence the learning outcome (Kozma 1994). A characteristic that has been detected as an important aspect in the learning potential of digital games is their intrinsically motivating character (Garris et al. 2002), meaning that the activity in itself is engaging and no

607

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy external reward for performing the activity is expected (Jenkins 2009). Intrinsically motivating activities create an enjoyable and fun experience, increasing the likelihood of repetitive usage (Ritterfeld et al. 2009). Another aspect of effectiveness is transferability, which refers to the transfer of knowledge in a formal context to situations in real life (Kozma 1994). When the transfer between a formal context to real life situations is low, this is defined as inert knowledge (Whitehead 1959). According to several authors inert knowledge is often due to usage of traditional teaching methods, which are outdated in that respect (Renkl et al. 1996). Garris et al. (2002) state that, in the context of DGBL, this transfer can be stimulated by organizing a debriefing session after gameplay.

1.2 Effectiveness studies in DGBL Typically, an experimental design is implemented to assess learning outcomes in a DGBL context by comparing a game‐based approach with another type of instruction and/or no intervention. The types of interventions to which the game‐based approach is compared can vary, which implies that results will ultimately depend on the particular comparison that is made (Bleumer et al. 2012). According to Campbell et al. (1963) the best experimental methodology for establishing whether learning has taken place is a pre‐test post‐test approach, including both an experimental and a control group. Questionnaires are typically used to assess the motivational aspects of DGBL, gauging the motivations of participants for learning via the intervention received and their interest in participation (Hainey 2010). Questionnaires are also implemented to assess other affective outcomes, such as attitudes. Moreover, some studies use in‐game assessment – referred to as stealth assessment – which is a technique that aims at accurately and dynamically measuring the player’s progress (Shute et al. 2011). Finally, qualitative methods such as interviews and observation have also been used in the context of effectiveness studies of DGBL. Three types of effectiveness studies in DGBL can be distinguished based on learning goals embedded in digital games (Bleumer et al. 2012). Specifically, digital games can aim at either knowledge transfer (cognitive learning outcomes), skill acquisition (skill‐based learning outcomes) or attitudinal and behavioural change (affective learning outcomes). Games aimed at knowledge transfer are typically implemented in education. For example, some studies have found a positive impact of the use of digital games to teach math (Bai et al. 2012) and language (Yip and Kwan 2006). Digital games aimed at skill acquisition are typically implemented in a training and corporate context. Several studies have observed an impact of playing games to practice managerial skills (Corsi et al. 2006). Games aimed at behavioural change are typically implemented in the health sector. An example of this are the healthy eating games influencing the diet and physical activity of children (Baranowski et al. 2008). Games aimed at attitudinal or behavioural change are implemented to raise awareness on a certain topic, such as poverty (Neys et al. 2012). According to Kraiger et al. (1993) these different types of learning outcomes require different types of assessment. Including studies aimed at the three learning outcomes would result in an extra level of heterogeneity, depending on the type of outcome that is assessed. Therefore, we will focus on one type of learning outcome in this study, that is cognitive learning outcomes.

2. Method In the present study the Cochrane method was used to carry out our systematic literature review (Higgins et al. 2008). This review method has its origins in health research and aims to study the effectiveness of interventions for prevention, treatment and rehabilitation. According to Cochrane, four dimensions of study characteristics can be distinguished: 1) participants (e.g. characteristics of the sample involved), 2) intervention (contents, format, timings and treatment lengths, intervention(s) in control group(s)), 3) methods (e.g. applied research methods) and 4) outcome measures (e.g. instruments used to measure a certain outcome) and results (Higgins et al. 2008). This distinction was also made in the present study. Search engines used for our review were Web of Knowledge, EBSCO Host and the International Bibliography of the Social Sciences. The following search string was used: ((Edu* OR serious OR learn* OR digital game based learning) AND ((dig* OR video OR computer) AND game) AND (assess* OR effect* OR measur*)). This search identified 54 publications dealing with effectiveness of DGBL aimed at cognitive learning outcomes. The review included peer reviewed journal and conference publications between 2000 and 2012. Other criteria for

608

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy inclusion were that (1) the study’s primary aim was effectiveness measurement of cognitive learning outcomes, (2) the focus was on digital games and (3) a pre‐post design with a control group was used. Eight studies had a post‐only design with a control group and 21 studies had a pre‐post design without a control group which were all excluded. Eventually, 25 studies with a pre‐post design and control group were considered eligible for analysis. A quantitative content analysis was conducted using SPSS. The codebook for this analysis was created inductively, using qualitative coding in nVivo. For this, open and axial coding (Glaser and Strauss 2009) were used for analysing procedure and methods sections of the studies based on Cochrane guidelines.

3. Results 3.1 Participants 16

14 12 10 8

6 4

2 0 not specified Children

teenagers young adults

Frequency

Figure 1: Subjects included in study (n = 25) 16 14 12 10 8 6 4 2 0

11 5

5 2

Figure 2: Inclusion criteria (n = 25 The average sample size of participants in studies reviewed was 220 (SD = 284). Although not all the studies reported the number of participants included by group (8% did not), our results showed that when reported the average number of participants was 105 (SD = 163) in the experimental and 84 (SD = 92) in the control group. Although four studies reported participants’ mean age, most studies defined subjects based on types of people, such as ‘university students’. Sixty‐five per cent of the studies included children, 24% teenagers and 12% young adults (Figure 1). Inclusion criteria for participation were thus mostly school‐related (e.g., ‘majoring in math and science). Several studies only included a certain subgroup, including participants based on ability (e.g., low achievers), socioeconomic status or a certain health condition (Figure 2).

3.2 Intervention Experimental groups (EG) were compared to a control group (CG) that either included participants that did not get an intervention (24%), got an intervention using another instructional approach (56%), or were compared

609

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy to several control groups, combining both (16%). One study did not provide any information on interventions implemented in the CG (Table 1). Table 1: Interventions in control group (n = 25) Intervention in control group(s) Traditional classroom teaching Traditional classroom teaching, with the use of multimedia Computer‐based application, such as an educational website Other game not related to the subject of the game implemented in the EG Paper and pencil exercises No intervention Not specified

N 12 1 4 2 3 10 1

% 48 4 16 8 12 40 4

Frequency

In the larger part of the studies (64%) DGBL was implemented in a formal context (e.g., in school during school hours), 8% in an informal context (e.g., home setting) and 12% in a semi‐formal context (Figure 3) referring to an implementation in a formal institution, such as a school, but where gameplay occurred outside of school hours. Sixteen per cent did not specify the context of play and 56% did not specify the gameplay composition (Figure 4). Twenty‐four per cent let participants play individually, 4% individually in competition, 24% cooperatively and 4% in a cooperative competition, meaning groups of participants played together against other groups of participants. One study implemented all four gameplay conditions. 18 16 14 12 10 8 6 4 2 0

2 not specified

formal context

informal semi‐formal context context

Frequency

Figure 3: Context of play (n = 25) 18 16 14 12 10 8 6 4 2 0

6 1

Figure 4: Gameplay composition (n = 25) Forty per cent of the studies did not report on the presence of an intermediary, referring to a teacher or researcher present during gameplay. In 56% of the studies an intermediary was present. One study did not include an intermediary. The average implementation period was 9 days (SD = 6), with a minimum of 1 day and a maximum of 23 days. Average total interaction time with the game is 12.4 hours (SD = 14.8), with a minimum of 30 minutes and a maximum of 64 hours.

610

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy Games were either implemented as a stand‐alone intervention (28%) or were embedded in a larger program (48%). Twenty‐four per cent did not specify implementation. Table 2 gives an overview of program specifications. Table 2: Specifications about games embedded in a larger program (n = 12) Program specifications Introduction

N % 5 20

Description An introduction concerning game content and gameplay was provided by an intermediary. This does not refer to an in‐game introduction A training session before the intervention was provided

Training of participants before intervention Extra material

Online platform

Game task formulation

Required reading

Procedural help by intermediary

The participants received help concerning the actual gameplay. This does not relate to content

Guidance by intermediary

The participants received guidance during gameplay in order to contextualize the game in the broader learning context

Supplement of course Debriefing

6 3

24 12

Gameplay occurred next to the classes A debriefing session was provided

Extra material such as articles, extra exercises, extra reading material, etc. were freely available The game was part of a larger educational online platform Certain tasks were formulated during gameplay The participants were expected to read next to gameplay

Several studies implemented the game as a supplement of a course. However, half of these provided extra time for the experimental group to interact with the game in addition to the courses, therefore spending additional time with the learning content.

3.3 Method All studies reviewed implemented an experimental design. Forty‐for per cent used a randomized controlled trial; 24% randomly assigned subjects while 20% randomly assigned classrooms to one of the conditions. Twelve per cent did not randomly assign participants to experimental and control group(s), but ‘matched’ participants in groups based on certain characteristics such as previous test scores, and 44% did not specify on group assignment of participants.

3.4 Measures Less than half (44%) implemented standardized tests, six of these only used standardized tests while 5 studies combined standardized tests with tests developed by the researchers. Twenty per cent of the studies reviewed only implemented tests developed by the researchers and 24% used school tests or exams (‘student achievement’) as an accuracy measure. Two studies used both test scores and student achievement as an accuracy measure. Twenty‐eight per cent did not report on the similarity between the pre‐ and post‐test measurements. Forty per cent employed the same test before and after the intervention, 8% changed the sequence of the questions and 8% used a similar test (e.g., other questions with the same type and difficulty levels). The latter did not report on how similarity of parallel tests were assessed. Sixteen per cent used a dissimilar pre‐ and post‐test, such as midterm exam scores and final exam scores. Two studies also implemented a mid‐test and four studies a follow‐up test. Table 3 gives an overview of measures used in the studies. Thirty‐six per cent of the studies reported on how scoring on tests occurred. Three studies (12%) included an independent coder, of which two controlled for inter‐rater reliability. One study used several, non‐ independent coders to control for inter‐rater reliability.

611

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy Table 3: Measures used for determining effectiveness (n = 25) Objective measurements Accuracy Test scores Student achievement Time measurements Time on task Subjective measurements

N 19 16 5 2 2 N

% 76 64 24 8 8 %

Self‐measurements Self‐efficacy topic Self‐efficacy general

8 4 2

32 16 8

Perceived educational value Affective Measurements Motivation

2 N 10

8 % 40

Motivation towards educational intervention Post‐only, EG Post‐only, EG and CG Pre‐ and post, EG and CG Motivation towards learning/educational content Post‐only, EG and CG Pre‐post, EG and CG Other

7 3 2 2 3 2 1 2

28 12 8 8 12 8 4 8

Attitudes towards school Teacher expectations

1 1

4 4

The larger part of the studies (76%) did a check on pre‐existing differences between experimental and control group(s) and 36% of the studies included in this review reported on effect size. Twenty‐four per cent did not report on statistical analysis. Table 4 shows how analysis of tests occurred. Table 4: Data‐analysis (n = 18) Data analysis Absolute test scores comparison

N 7

% 28

Absolute test scores, adding pre‐ test scores to the analysis

13 52

Difference scores Item accuracy

10 40 1 4

Gain/loss scores

Percentage of improvement

Error rates

Description Absolute pre‐test and post‐test scores of EG and CG are compared separately Absolute test scores of EG and CG are compared, taking the pre‐test scores into account Use of a specific scoring system

The number of points the gained/lost between the pre‐and post‐test Percentage of improvement between pre‐and post‐test

612

Example from studies reviewed …the independent samples t‐test was applied to examine whether the differences between the mean scores of the control and experimental groups in the pre‐test and post‐test were statistically significant (Yip and Kwan 2006) …pre‐test scores on the specific subject tested were introduced as covariates in order to control for initial levels of the ability’ (Rosas et al. 2003)

Each factor was rated ‐1 if performance changed from correct to incorrect, 0 if there was no change and +1 if it changed from incorrect to correct. Then, each subject’s scores were summed to create a summary difference score…(Coles et al. 2007) …paired‐samples t tests were conducted to compare the treatment and control gain scores from pre‐test to post‐test…(Kebritchi et al. 2010) …the percentage of improvement was calculated from the primary scores by subtracting the pre‐test result from the post‐test result and then dividing the difference by the maximum result of the test (Ketamo 2003)

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy Data analysis Out‐of‐game error rates

N 1

% 4

Description Number of mistakes made in the pre‐ and post‐test are compared

In‐game error rates

In game measurement of number of errors during gameplay are compared

Example from studies reviewed the educational effect…by comparing the number of mistakes of the students of the VR‐ENGAGE sub‐ groups with the number of mistakes of the students of the respective sub‐groups that had used the simple ITS (Virvou et al. 2005) …the position and location of the mouse onscreen were recorded every 10th second. How successful children were at solving the computer assignments immediately or after one or more repetitions can be derived from these registrations (Van Der Kooy‐ Hofland et al. 2012)

4. Discussion The results of the present study show that studies vary on different dimensions of the study design, presenting a heterogeneity of methodologies. Homogeneity is, however, an important prerequisite when conducting meta‐analyses, impeding generalizing conclusions on the effectiveness of DGBL (Higgins et al. 2008). Differences were not only found between study designs, but also in reporting. Regarding the participants dimension, several studies only used certain subgroups (e.g. certain ability, certain socioeconomic status). This does logically narrow results to this specific subgroup (Campbell et al. 1963), which is problematic, however, when generalizing claims on DGBL effectiveness are made. Therefore, reporting on inclusion criteria for recruitment and how sampling occurred, is essential. When considering the intervention dimension, studies firstly differed on the type of intervention implemented in the CG. The interpretation of the contribution of the intervention to the EG does, however, depend on the activities performed in the CG (Campbell et al. 1963). Considering that intervention in the CG can influence results and interventions implemented in CG differed across studies, comparison between results becomes problematic. Secondly, implementation of DGBL in the EG differed between studies, either implementing them as a stand‐alone intervention or in a larger program. When embedded in a program, elements of the program differed across studies as well (e.g. introduction, debriefing, extra material, required reading, etc.). The addition of other elements to the intervention can result in multiple treatment interference (Campbell et al. 1963), however, meaning the achievement gain might not be solely attributable to DGBL, but could be influenced by other activities that are part of the intervention. Results of studies implementing only DGBL and studies implementing DGBL in a larger program are thus not comparable. Thirdly, implementation of DGBL differed by the presence of an intermediary and the role of this intermediary. Most studies did not report on whether or not an intermediary was present. When an intermediary was present, they were either present to solely supervise or were present with the purpose of providing procedural help and/or guidance during gameplay. The role of the intermediary was either filled by the classes’ teachers or a researcher. Who the intermediary is (e.g. someone more familiar such as a teacher or a total stranger) and how he or she interacts with the participants is a potential confound when assessing the effect of the DGBL intervention (Leary 1995). While all studies implemented an experimental design, differences were found in the participants’ assignment to the EG and CG which was done with or without randomization or by ‘matching’ in order to attain similarity of both groups. It was, however, not clear whether this matching occurred randomly. When using matched random assignment, the participants’ scores on a measure of relevance (for example: pre‐test) are obtained in order to randomly assign participants belonging to a certain level to the conditions (Leary 1995). According to Campbell & Stanley (1963), matching is not a preferable method. In the context of educational research, randomization of the classroom as a unit is more preferable, because classrooms can then be classified for analysis on the basis of factors such as schools, teacher, subject, time of day, mean intelligence level, etc. (Campbell et al. 1963). According to Leary (1995), however, randomization of schools will jeopardize internal validity, as groups will likely differ on multiple dimensions. This is an issue that merits further discussion, considering randomized controlled trials are difficult to implement with small sample sizes, which are often a reality in DGBL effectiveness studies. How effectiveness was measured, also differed between studies, complementing learning outcome measures with affective measures and time measures. Some studies reported very specifically on how scoring on tests occurred in. According to Campell & Stanley (1963) different instruments and different scorers, can yield other

613

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy results. Every measuring tool also entails a certain measurement error, fuelled by transient states (e.g. participants’ mood, level of fatigue), stable attributes (e.g. misunderstanding questions, individual differences in motivation), situational factors and characteristics of the measurement itself (e.g. ambiguous questions, test that induce fatigue). Incomplete information on and differences between the length of certain tests, formulating of questions or the time of measurement account for an incomparability of results across studies as well. Use of independently developed standardized tests could provide more ‘stable’ measurements and create comparability across studies. This is, however, a difficult exercise, considering the wide range of topics covered by DGBL. Although, standardized tests could be implemented to assess affective learning outcomes, such as motivation. While the larger part of the studies implemented the exact same test pre‐ and post‐intervention, others changed the sequence of the questions. Implementing the same questions in the pre‐and post‐test can however lead to a test‐retest practice effect (Campbell et al. 1963). According to Crawford et al. (1989) this is due to retention of specific test material by the participants. Other studies used similar tests, meaning these consisted of questions of the same type and difficulty level. While practice effects can still occur using a parallel version of a test on different points in time (e.g. pre‐ and post‐test), these generally tend to be smaller (Anastasi 1961). Certain studies also used dissimilar tests, when for example student achievement in school (e.g. exam scores) was used as a measure. This seems problematic, considering assumptions on the comparability of both tests cannot be made, making any significant achievement gains possibly invalid. Differences in similarity of pre‐and post‐test across studies, is another reason why it is difficult to compare results across studies. Important differences were also found when considering data analysis techniques. While indeed several analyses can be used to measure the effect of the intervention, an analysis of covariance (ANCOVA) with pre‐ test scores as a covariate, is a more preferable method (Campbell et al. 1963). In the context of randomized controlled trials, ANCOVA reduces error variance and in the context of nonrandomized designs, it adjusts mean scores of the post‐test to differences between groups on pre‐test scores (Dimitrov and Rumrill 2003). Furthermore, missing information on implementation of the intervention(s) impedes replication of certain studies, which is a basic principle of empirical research in order create the opportunity to falsify obtained results (Popper 2000). Finally, incomplete information on sampling, similarity between interventions of the EG and CG and similarity between pre‐and post‐tests, puts validity of certain results in doubt. In general, we can conclude that comparisons between studies are problematic as a result of heterogeneity in study designs, heterogeneity in reporting, incomplete information and biased results, impeding generalization. Standardized guidelines on sampling, activities in control groups, implementation of DGBL in the experimental group, measures, scoring, analysis and reporting on these elements could contribute to homogeneity in the research field and create insight in the validity of studies.

5. Limitations and further research The selection and coding of publications was conducted by one researcher, which can be considered a limitation of this study. This study also is limited to digital games aimed at cognitive learning outcomes. Further research should thus be conducted on methodologies used in digital games aimed at skill acquisition and behavioural or attitudinal change. An interesting venue for future research is exploring the possibilities for the development of a standardized procedure to measure effectiveness of DGBL. Relevant issues to investigate in this context are gathering input from experts in the methodology field in order to detect preferable methods for measuring learning effectiveness (e.g. number of control groups, activity in control group, implementation of DGBL, implementation period, etc.). Further, such a procedure should be adjusted to the requirements of the people who would benefit from this procedure and actually use the procedure. Therefore, involvement of relevant stakeholders in the process of developing the procedure is desirable.

Acknowledgements This PhD project is funded by IWT, the Flemish government agency for Innovation by Science and Technology (IWT).

614

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy

Appendix 1: Studies included in review Anderson, J. and Barnett, M. 2010. Using Video Games to Support Pre‐Service Elementary Teachers Learning of Basic Physics Principles. Journal of Science Education and Technology, 20(4), 347‐362. Bai, H., et al. 2012. Assessing the effectiveness of a 3‐D instructional game on improving mathematics achievement and motivation of middle school students. British Journal of Educational Technology, 43(6), 993‐1003. Coles, C. D., et al. 2007. Games that "work": using computer games to teach alcohol‐affected children about fire and street safety. Res Dev Disabil, 28(5), 518‐530. Din, F. S. and calao, J. 2001. The effects of playing educational video games in kindergarten achievement. . Child Study Journal, 31(2), 95‐102. Kajamies, A., Vauras, M. and Kinnunen, R. 2010. Instructing Low‐Achievers in Mathematical Word Problem Solving. Scandinavian Journal of Educational Research, 54(4), 335‐355. Kanthan, R. and Senger, J.‐L. 2011. The Impact of Specially Designed Digital Games‐Based Learning in Undergraduate Pathology and Medical Education. The Impact of Specially Designed Digital Games‐Based Learning in Undergraduate Pathology and Medical Education, 135, 135‐142. Ke, F. 2008. Computer games application within alternative classroom goal structures: cognitive, metacognitive, and affective evaluation. Educational Technology Research and Development, 56(5‐6), 539‐556. Kebritchi, M., Hirumi, A. and Bai, H. 2010. The effects of modern mathematics computer games on mathematics achievement and class motivation. Computers & Education, 55(2), 427‐443. Ketamo, H. 2003. An Adaptive Geometry Game for Handheld Devices. Educational Technology & Society, 6(1), 83‐94. Lorant‐Royer, S., et al. 2010. Kawashima vs “Super Mario”! Should a game be serious in order to stimulate cognitive aptitudes? Revue Européenne de Psychologie Appliquée/European Review of Applied Psychology, 60(4), 221‐232. Miller, D. J. and Robertson, D. P. 2010. Using a games console in the primary classroom: Effects of ‘Brain Training’ programme on computation and self‐esteem. British Journal of Educational Technology, 41(2), 242‐255. Miller, D. J. and Robertson, D. P. 2011. Educational benefits of using game consoles in a primary classroom: A randomised controlled trial. British Journal of Educational Technology, 42(5), 850‐864. Moreno, J. 2012. Digital Competition Game to Improve Programming Skills. Educational Technology & Society, 15(3), 288‐ 297. Moshirnia, A. 2007. The Educational Potential of Modified Video Games. Issues in Informing Science and Information Technology, 4, 511‐521. Papastergiou, M. 2009. Digital Game‐Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation. Computers & Education, 52(1), 1‐12. Parchman, S. W., et al. 2000. An Evaluation of Three Computer‐Based Instructional Strategies in Basic Electricity and Electronics Training. . Military Psychology, 12(1), 73‐87. Poli, D., et al. 2012. Bringing Evolution to a Technological Generation: A Case Study with the Video Game SPORE. The American Biology Teacher, 74(2), 100‐103. Rastegarpour, H. and Marashi, P. 2012. The effect of card games and computer games on learning of chemistry concepts. Procedia ‐ Social and Behavioral Sciences, 31, 597‐601. Rosas, R., et al. 2003. Beyond Nintendo: design and assessment of educational video games for first and second grade students. Computers & Education, 40, 71‐94. St Clair‐Thompson, H., et al. 2010. Improving children's working memory and classroom performance. Educational Psychology, 30(2), 203‐219. Suh, S., Kim, S. W. and Kim, N. J. 2010. Effectiveness of MMORPG‐based instruction in elementary English education in Korea. Journal of Computer Assisted Learning, 26(5), 370‐378. Van der Kooy‐Hofland, V. A., Bus, A. G. and Roskos, K. 2012. Effects of a brief but intensive remedial computer intervention in a sub‐sample of kindergartners with early literacy delays. Read Writ, 25(7), 1479‐1497. Virvou, M., Katsionis, G. and Manos, K. 2005. Combining Software Games with Education: Evaluation of its Educational Effectiveness. Educational Technology & Society, 8(2), 54‐65. Yang, Y.‐T. C. 2012. Building virtual cities, inspiring intelligent citizens: Digital games for developing students’ problem solving and learning motivation. Computers & Education, 59(2), 365‐377. Yip, F. W. M. and Kwan, A. C. M. 2006. Online vocabulary games as a tool for teaching and learning English vocabulary. Educational Media International, 43(3), 233‐249.

References Anastasi, A., 1961. Differential psychology: Individual and group differences in behavior. Macmillan. Bai, H., et al. 2012. Assessing the effectiveness of a 3‐D instructional game on improving mathematics achievement and motivation of middle school students. British Journal of Educational Technology, 43(6), 993‐1003. Baranowski, T., et al. 2008. Playing for real: video games and stories for health‐related behavior change. American journal of preventive medicine, 34(1), 74. Bleumer, L., et al., 2012. State of play of digital games for empowerment and inclusion: a review of the literature and empirical cases. Spain. Campbell, D. T., Stanley, J. C. and Gage, N. L., 1963. Experimental and quasi‐experimental designs for research. Houghton Mifflin Boston.

615

Anissa All, Elena Patricia Nuñez Castellar and Jan Van Looy Clark, R. E. 1994. Media will never influence learning. Educational Technology Research and Development, 42(2), 21‐29. Connolly, T. M., et al. 2012. A systematic literature review of empirical evidence on computer games and serious games. Computers & Education. Corsi, T. M., et al. 2006. The real‐time global supply chain game: New educational tool for developing supply chain management professionals. Transportation Journal, 61‐73. Crawford, J., Stewart, L. and Moore, J. 1989. Demonstration of savings on the AVLT and development of a parallel form. Journal of Clinical and Experimental Neuropsychology, 11(6), 975‐981. Dimitrov, D. M. and Rumrill, J., Phillip D 2003. Pretest‐posttest designs and measurement of change. Work: A Journal of Prevention, Assessment and Rehabilitation, 20(2), 159‐165. Gagne, R. M. 1984. Learning outcomes and their effects: Useful categories of human performance. American Psychologist, 39(4), 377. Garris, R., Ahlers, R. and Driskell, J. E. 2002. Games, Motivation, and Learning: A Research and Practice Model. Simulation & Gaming, 33(4), 441‐467. Glaser, B. G. and Strauss, A. L., 2009. The discovery of grounded theory: Strategies for qualitative research. Transaction Books. Hainey, T., 2010. Using Games‐Based Learning to Teach Requirements Collection and Analysis at Tertiary Education Level. Higgins, J. P., Green, S. and Collaboration, C., 2008. Cochrane handbook for systematic reviews of interventions. Wiley Online Library. Jenkins, H., 2009. Confronting the challenges of participatory culture: Media education for the 21st century. The MIT Press. Joy, E. H. and Garcia, F. E. 2000. Measuring Learning Effectiveness: A New Look at No‐Significant‐Difference Findings. JALN, 4(1), 33‐39. Kharrazi, H., et al. 2012. A Scoping Review of Health Game Research: Past, Present, and Future. Games for Health Journal, 1(2), 153‐164. Kozma, R. B. 1994. Will Media Influence Learning? Reframing the Debate. . Educational Technology Research and Development, 42(2), 7‐19. Leary, M. R., 1995. Introduction to behavioral research methods. Brooks/Cole Pacific Grove, CA. Michael, D. R. and Chen, S. L., 2005. Serious games: Games that educate, train, and inform. Muska & Lipman/Premier‐ Trade. Neys, J., et al., Poverty is not a game: behavioral changes and long term effects after playing PING. ed. 13th annual conference on the International Speech Communication Association, 2012 Portland. Popper, K. 2000. Science: conjectures and refutations. Readings in the Philosophy of Science: From Positivism to Postmodernism, 9‐13. Renkl, A., Mandl, H. and Gruber, H. 1996. Inert knowledge: Analyses and remedies. Educational Psychologist, 31(2), 115‐ 121. Ritterfeld, U., Cody, M. and Vorderer, P., 2009. Serious games: mechanisms and effects. New York: Routledge. Salomon, G. 1979. No distribution without individuals' cognition: a dynamic interactional view. Sawyer, B. and Smith, P. 2008. Taxonomy for Serious Games. Digitalmil, Inc& Serious Games Initiative/Univ. of Central Florida, RETRO Lab. Shute, V. J., Rieber, L. and Van Eck, R. 2011. Games... and... learning. Trends and issues in instructional design and technology, 3. Whitehead, A. N. 1959. The aims of education. Daedalus, 88(1), 192‐205. Yip, F. W. M. and Kwan, A. C. M. 2006. Online vocabulary games as a tool for teaching and learning English vocabulary. Educational Media International, 43(3), 233‐249.

616

Investigating Collaborative Games to Teach Mathematics‐Based Problem Solving in the Classroom Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield School of Computer Science, University of Nottingham, Nottingham, UK psxra6@nottingham.ac.uk gail.hopkins@nottingham.ac.uk peter.blanchfield@nottingham.ac.uk Abstract: Learning mathematics, and mathematical problem solving in particular, is considered a challenge for some pupils at primary schools. When solving mathematical problems several skills are needed and commonly some children face difficulties in areas like memory, language and spatial aspects. This PhD research aims to investigate whether collaborative computer games can support and motivate UK Key Stage 2 pupils in fostering better understanding of mathematical problem solving. As part of achieving this overall aim, two initial studies are reported in this paper. The research described here has taken a user‐centred design approach and has been two‐fold. Firstly, an initial study was conducted to gain an understanding of the existing teaching environment, methods used and problem solving experiences of both children and teachers in a Key Stage 2 teaching environment. The investigation involved observations of problem solving classes and questionnaires which sought to explore mathematics problem solving difficulties and the role of collaborative games in the actual learning environment. The findings from this initial study indicate that games and collaborative working in mathematics classes could be significant in supporting and enhancing Key Stage 2 pupils in their learning of mathematical problem solving. Therefore it is suggested that collaborative mathematical games could be effective instructional tools for enhancing learning and understanding of mathematics concepts, arithmetic and problem solving. The use of collaboration within such a tool may promote the sharing and discussion of problems and so may be helpful for students who have difficulty in mathematics. The second study aimed to investigate whether and how Key Stage 2 children engaged in a TM commercially available collaborative problem solving computer game, Lego Harry Potter and to examine their levels of collaboration and motivation within the game. The purpose of this study was to gather requirements for a future collaborative computer game designed to teach mathematics problem solving. A mixed‐methods approach of observation and questionnaires has been used. Key Stage 2 pupils were observed during collaborative computer game play and an analysis of their behaviour and interaction was conducted in order to identify the ways in which they collaborated and the elements of the game which promoted motivation and collaboration. Questionnaires were used to investigate elements of the game play which encouraged engagement and collaboration. The findings from the first study show that some, but not all, children found mathematical problem solving challenging but that collaboration and game playing is used and enjoyed during their classroom‐based learning of this. The second study showed that the collaborative computer game did engage the players. Elements of role play, suspense and mystery encouraged motivation and collaboration but there were issues at times with some players not knowing what to do in the game and at other times with providing enough challenge for two collaborating players. Resulting requirements gathered from this include the necessity for clear rules and goals, a playful environment with challenge, mystery and fantasy and a story line that flows through the game. Keywords: games‐based learning, collaborative learning, mathematics, problem solving

1. Introduction Mathematics in primary schools and problem solving in particular are considered challenging for some pupils (Tay Lay Heong 2005). Wu et al. (2009) confirmed that the most common difficulties that students encountered in mathematics problem solving are a lack of confidence, weakness in understanding the terms and mathematical language, confusion in determining mathematical operations and lack of conceptual and procedural skills in problem solving. There is significant research evidence that collaboration between peers during mathematics lessons can facilitate active learning among children and can be effective in improving children’s mathematical abilities (e.g, Artut and Tarim, 2007; Tarim and Akdeniz, 2008). Furthermore, many researchers have indicated that computer games can motivate children to learn mathematics and can deliver good support for learning (Quinn, 1994; Betz, 1995; Moreno, 2002; Liu and Chu, 2010; Charsky and Ressler, 2011). The overall research aim of this PhD is to investigate whether collaboration in game play can support and motivate UK Key Stage 2 (7‐11 years) pupils in fostering better understanding of mathematical problem solving. Initial research towards this aim is reported here which is two‐fold: firstly a classroom‐based study is reported which examines activities and problems that occur in mathematics classrooms and to study how

617

Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield games and collaboration are used in the classroom. Secondly, observations of pupils using a collaborative computer game were conducted in order to investigate the extent to which collaboration and motivation to play were present. Ultimately, the aim of this PhD is to build and test a collaborative game which is designed to teach mathematical problem solving to UK Key Stage 2 children, in part based on the findings of the studies reported in this paper.

2. Literature review In schools, learning mathematics can be a major challenge for students because it depends on several skills such as the ability to connect information to a concept and to transform problems into mathematical sentences (Tambychik et al. 2010). Problem solving is considered as being at the heart of mathematics (Cockroft 1982). It is a complex process which can involve many sub‐problems (NCTM 2000) and where the target is obvious but not easy to reach Altun (2005). Pupils’ failures at solving problems are not often the result of a lack of mathematical knowledge but an indication of the ineffective use of their knowledge (Harskamp and Suhre 2007). Cooperative learning techniques in the classroom have been shown to encourage interaction between children (Gillies, 2006) and research also suggests that collaboration can benefit children in terms of achievement (Webb et al., 2009). In the past, teaching mathematics was often based on traditional teacher‐centred methods. However, more recently the importance of collaboration in contributing to the development of pupils’ mathematical thinking and problem solving abilities has been stressed (Johnson and Johnson 1989; Lohman and Finkelstein 2000). Serious games in the field of education are widespread and offer an inspiring area of research into instructional tools which can be used to enhance learning (Egenfeldt‐Nielsen 2005; Ke and Grabowski 2007; Coller and Scott 2009) and problem solving (Pivec and Pivec 2008). Educators believe that children learn best during play and that games give them that opportunity (Gee 2003; Morgan and Kennewell 2005). Using technological devices such as computers in the classroom has also proven to be effective in achieving motivation (Ke 2008), collaboration (Cortez et al. 2009) and learning results (Carbonaro et al. 2008; Plass et al. 2010a). Collaborative game play activities which have mathematical principles and problems embedded in them can offer special opportunities for mathematics investigations (Gee 2004; Childress and Braswell 2006; Mikropoulos 2006). Harnessing educational technologies like games in mathematics learning can make the learning process more effective (Mor et al. 2004; Simpson et al. 2006) and can present academic subjects in a more learner‐centred, accessible, enjoyable and thus, more effective manner (Malone 1980; Kafai 2001; Prensky 2003).

3. Experimental design This research was conducted over a period of two years and took place initially in the classroom where observations and questionnaires (of both teacher and pupils) were used to determine pupils’ attitudes to mathematics problem solving, collaboration and games within the classroom. Subsequently a games‐based study was conducted whereby pupils were observed playing a commercial computer game, LegoTM Harry Potter, in order to study pupils’ behaviours in terms of motivation to play the game and collaboration within the game. Again, questionnaires were used to support these observations.

3.1 The classroom study 3.1.1 Methods Observations were conducted on 31 Year 5 pupils (7‐10 years old) at a local primary school. Observations took place twice a week for a period of 4 months. Observations focussed on the types of skills used and activities performed in the mathematics classes and, in particular, looked at the difficulties experienced with mathematics and especially problem solving topics, and the use of collaboration and games to overcome these.

618

Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield At the end of the observation two questionnaires were distributed; one for pupils and one for teachers. The pupil questionnaire was completed by 23 respondents. This aimed to identify students’ attitudes towards mathematics learning and aspects of collaboration. The student’s questionnaire had fifteen closed questions, each with answers represented as a "smiley face", "sad face" and "neutral face". Pupils were given help with reading and understanding the questions. Four teachers completed a questionnaire which aimed to investigate their views of the mathematics learning environment with regard to pupils’ skills, activities and difficulties during classes, especially with problem solving topics. The teacher’s questionnaire included twelve open ended questions which asked specifically about problems in mathematics classes, problem solving topics and use of mathematical software, games and collaboration. 3.1.2 Results Based on the observations in the classroom it was evident that pupils varied in their approaches to learning mathematics. Some engaged much more readily than others who were observed to become bored and to work slowly. Some children clearly lacked confidence as was evidenced by their unwillingness to answer questions in class even when they knew the right answer. However, encouragement and rewards such as stickers helped somewhat to overcome this. Some children were able to give correct answers but could not explain their reasoning. Collaboration between children varied; whilst children worked more slowly and were less motivated working alone, some were reticent in discussing answers and appeared more competitive. It was observed that different kinds of number games were used in the classroom in a collaborative manner. For example, one game involved the children standing in a circle taking turns to count forwards or backwards by 10 and they appeared to respond well to this. Other solutions included the use of pictograms with related questions which the children found enjoyable. The pupils’ questionnaire results are given in Table 1 and the qualitative data from the teachers’ questionnaires are presented in the text below. It can be seen from the results that out of 23 respondents more than half the children thought they were good at maths and more than half enjoy maths, although five children acknowledged difficulty in problem solving. The majority of pupils stated that they found mathematics exercises interesting and fun and results also indicated that they were strongly in favour of game playing in the classroom. Collaboration was viewed favourably (n=19) but less than half like to share their answers with friends. Table 1: The classroom study ‐ pupils’ questionnaire results Questions

Yes

Don’t know

Do you enjoy mathematics classes?

Do you find mathematics exercises interesting and fun? In your opinion are you good at maths? Do you find problem solving sessions difficult? Do you play any games in the classroom that involve maths? Does playing the games help you understand maths better? Do you use any computer software in maths classes? If you had this software at home would you use it? Do you like asking your teacher when you are not sure about your solution? Do you look at corrections in your maths work and try them again? Would you like a computer game that helps you to understand maths better? Do you like to discuss your answers with your friends? Do you enjoy to work together with your friends to solve maths exercises? Do you like to share your answers with your friends? Do you think working with your friends will help you to get better at problem solving?

18 12 5 15 13 12 20

0 2 6 3 2 4 2

5 9 12 5 0 7 1

619

Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield Teachers' answers regarding problem solving difficulties confirmed that there are several issues which contribute to this. They reported that problem solving is a difficult subject in Key Stage 2 as there is often no set way of solving a problem. Moreover, different skills must be taught to children and they must choose the most appropriate way to solve the problem by themselves. In addition, language can also be an issue with problem solving. For example, children can find the language of problem solving exercises difficult to understand. Furthermore, children can also struggle to explain what they have done and how they found a solution. Teachers indicated that involving different kinds of games in the classroom engaged pupils in problem solving. They also mentioned that some students need physical interaction for their learning, so games would help them. 3.1.3 Discussion The observation of pupils in the classroom and the questionnaire results indicated that mathematics is a difficult subject for some pupils and this was especially so with problem solving. However, it should be noted that not all children found mathematics challenging. This leaves the teacher with the task of teaching children at different ends of a spectrum of ability, and we propose that this provides a strong argument for the role of collaboration between pupils. Even when able to give the correct answer to a problem, often children could not explain in words how they achieved that answer, but during collaborative sessions children were given the opportunity to work through this with each other and to learn to express how they achieved their answers, and many children were observed to respond well to these sessions. According to Flick and Bell (2000), science involves a lot of difficult concepts which can be hard for primary school pupils to understand. These difficulties can be related to differences in the degree of idea complexity, abstract topics and the lack of pupils’ experiences. Pupils often face difficulties in mathematics especially in mathematics problem solving (Tay Lay Heong 2005). Underwood et al. (2000) showed that the performance of children who work collaboratively will be higher than children working individually. Additionally collaboration can benefit children of different ages (Hogan and Tudge 1999), mixed ability groups (Garton and Pratt 2001) and groups with children who have different levels of confidence (Tudge et al. 1996), motivation (Gabriele and Montecinos 2001) and are of different gender (Strough et al. 2001). The observation results gave a clear view that using different kinds of games can create an environment for learning where challenge and motivation of pupils are increased, especially for those who tend to get bored with traditional methods of instruction. This is confirmed by research suggesting games offer engagement for long periods of time (Lenhart and Kahne 2008). The teachers’ questionnaire results showed that physical interaction is important for some children and games provide this physical involvement. Further to this, the results showed that there is no set method for solving a problem and that different skills are needed including the ability to understand the language of problem solving. A games‐based environment could allow pupils to try out solutions more readily and more imaginatively and to engage with the problem rather than engaging with the language. In light of this study, collaboration is considered as key for achievement in problem solving tasks due to the activities that can be involved within the collaboration. Children like to interact and communicate with each other in order to share information and ask for advice. Moreover, collaboration among children will allow them to share ideas and learn from each other. Also, we propose that combining collaboration with computer game play can provide an effective environment within which children can physically interact to collaborate, share ideas and help each other to solve problems. As such, the second study reported in the next section aimed to investigate the extent to which such a computer game could encourage collaboration and motivation.

3.2 The game‐based study 3.2.1 Methods A mixed‐methods study was conducted of children playing LegoTM Harry Potter at two different primary schools. The purpose of this was to examine collaboration during game‐play and more specifically, communication and interaction and how these contributed to motivate pupils to complete tasks and reach desired goals within the game.

620

Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield Collaboration within games is considered a key component as it can boost motivation for learning while enhancing game performance (Wood et al., 1976; Walqui, 2006; Luckin, 2008). As such, observations focussed on collaboration and motivation during game play. Twenty seven groups, each consisting of two children from Key Stage 2, were observed playing the game for 25 minutes. The children’s ages ranged from 7 to 10 years old and groups consisted of male, female and mixed gender. Observations were coded according to collaboration and motivation and the codes and their frequencies are shown in Table 2. A further 30 children, aged between 8 and 10 years old, played the game in pairs for 25 minutes before completing a questionnaire. Again, groups consisted of male, female and mixed gender. The aim of the questionnaire was to investigate elements of the game play which encouraged motivation and collaboration. It also asked children about their fun whilst playing the game, whether they found it challenging or complex and whether they enjoyed collaborating during game play. Open ended questions asked them about why they gave a particular answer.

3.3 Results The results from the observations can be seen in Table 2 and Figure 1. Codes have been divided into those related to collaboration and motivation. Collaboration has been coded in terms of physical and verbal interaction. The results from the questionnaire study are shown in Table 3. Table 2 The game‐based study ‐ coded observations of game play Code

Meaning (Collaboration)

Collecting objects

Manipulating objects

Gesture

Joint manipulation

Conversation

Instruction

Sharing knowledge

Questioning

Disagreement

arg

Arguing

Sub‐category

Frequency

Code

Meaning (Motivation)

Sub‐ category

Frequency

Enthusiasm

_______

Physical movement

_______

Goal seeking

_______

Discovery

_______

Immersion

_______

Withholding information

_______

Competition

_______

Concentration

_______

Fun

_______

Aggression

_______

Challenge

Copying

Physical interaction Physical interaction Physical interaction Physical interaction Verbal interaction Verbal interaction Verbal interaction Verbal interaction Verbal interaction Verbal interaction

_______ _______

3 2

The results from the observations demonstrate that pupils definitely engaged in collaborative activity; in particular sharing knowledge, giving instructions and questioning each other, with a frequency of 32, 25 and 13 respectively. Pupils also spent a lot of time manipulating (19) and collecting (10) objects. Verbal interaction was high and there was clear evidence of collaboration in the dialogue that was recorded: Player 1: “I can’t do that” Player 2: “Because you need to listen to me” Player 1: “I did, but I still can’t do anything” Player 2: “No you did not because this is supposed to be done by helping”

621

Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield

Figure 1: The game‐based study ‐ coded observations of game play In terms of their motivation to play the game, evidence of enthusiasm was high (22) and a fair amount of discovery was evident (12). This was also reflected in some of the dialogue: Player 3: “Is this Lego Harry Potter?” Researcher: “Yes” Player 3: “Ohh, I have never played it” Researcher: “Do you like to play?” Player 3: “Yes, of course” Researcher: “Why do you like Harry Potter Lego Game?” Player 3: “I like it because it’s funny” Researcher: “Why you think it’s funny?” Player 3: “Because you can smash things around you in the game and you can fly” It was noted that those children who were familiar with the Harry Potter character tended to have a better idea of how to play the game than those who weren’t and they were observed to be helping those less familiar with the game characters. It was also observed that the act of one child explaining a solution to another enabled them to vocalise their reasoning and helped their problem solving. Table 3: The game‐based study ‐ game play questionnaire results Questions

Answers & Gender Yes Don’t know No B G B G B G

Did you have fun playing the game?

Was the game complex?

Was the game challenging?

Did you feel that time passed quickly? Would you like to have more time playing the game? Did you find the game exciting? Do you prefer to play with your friends? Do you prefer to play with children of the same gender?

622

Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield The questionnaire results showed clear evidence that children enjoyed playing LegoTM Harry Potter (n=29) and that they found the time passed quickly (n=26). All 30 children stated that they wanted to have more time playing the game. Twenty seven of the 30 children agreed that they preferred to play with friends. With respect to complexity and challenge, children were less certain, with only 5 agreeing that the game was challenging and only 2 finding it complex. Some did, however, state that they felt the hardest part was knowing what to do and that due to the complexity they needed instructions on how to play. Others stated that there was not enough challenge, that working together meant there was less challenge and that the game was too easy. Another problem was that children did not like the split screen, which appeared when the two players moved away from each other in the game. They stated that they preferred to have their own view of the game world and not to see both players’ views simultaneously. In terms of genre, pupils found the elements of adventure and fantasy very engaging and different characters within the game through which they could role play were important. 3.3.1 Discussion The children collaborated well with a high level of verbal interaction and were observed to be motivated to play the game. This level of engagement is something which is important to harness in an educational game that aims to teach problem solving. In terms of complexity and challenge, there was evidence that this was lacking for some children but that others just didn’t have enough instruction in how to play the game and this is something which needs careful consideration in the design of a future maths‐based educational game. Challenge and complexity need to be incorporated in such a game to engage children, but children also need to understand each stage of the game and its goals and requirements and be able to progress through the game. Therefore, the presence of clear goals and well‐structured rules for playing the game are considered as important requirements to allow progression through the game and the level of challenge needs to be such that children do not find the game too easy when collaborating with others. Also, we propose a fantasy genre for our future maths‐based educational game, which incorporates mystery and suspense and allows the players to take on the role of different characters to achieve their goals. The proposed collaborative game will be networked and provide players with their own separate views of the game world.

4. Conclusion and future work The research reported in this paper demonstrates that using collaborative games could play a significant part in teaching mathematics‐based problem solving in the classroom. Computer games can offer an environment which is motivating and allows collaborative learning. However, evidence shows that children prefer to choose computer games which are engaging more than games which are specifically educational (Facer 2003; Kerawalla and Crook 2002; Kinzie and Joseph 2008) and this engagement is key to a game’s success. The design and development of the teaching game has begun. The research findings reported in this paper form the basis for the design of the game which will be a fantasy, role‐playing game which includes the collaborative problem solving approach. This has been shown to provide a good basis for getting students to explore how to solve problems and will increase the motivation of those less willing to engage with this process. It will also allow those who understand how to solve problems to be able to express this process more clearly to others.

Acknowledgements The researchers would like to thank the staff and children of Southwold Primary School, Nottingham and The Kabin After School Club, All Hallows Primary School, Nottingham for taking part in this study. This research is supported by the Ministry of Education in Saudi Arabia.

References Altun, M. (2005) Teaching Mathematics, Erkam Press, Bursa. Artut, P. D., and Tarim, K. (2007) Effectiveness of jigsaw II on prospective elementary school teachers, Asia‐Pacific Journal of Teacher Education, Vol 35, No.2, pp 129–141. Betz, J. A. (1995) Computer games: increases learning in an interactive multidisciplinary environment, Journal of Educational Technology Systems, Vol 24, No.2, pp 195–205. Carbonaro, M., Cutumisu, M., Duff, H., Gillis, S., Onuczko, C., and Siegel, J. (2008) Interactive story authoring: a viable form of creative expression for the classroom, Computers & Education, Vol 51, No.2, pp 687–707.

623

Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield Charsky, D., and Ressler,W. (2011) Games are made for fun: lessons on the effects of concept maps in the classroom of computer games, Computers & Education, Vol 56, No.3, pp 604–615. Childress, M. D. and Braswell, R. (2006) Using Massively Multiplayer Online Role‐Playing Games for Online Learning, Distance Education, Vol 27, No.2, pp 187‐196. Cockroft, W. H. (1982) Mathematics Counts: Report of the Committee of Enquiry into the Teaching of Mathematics in Schools, London: HMSO. Coller, B. D., and Scott, M. J. (2009) Effectiveness of using a video game to teach a course in mechanical engineering, Computers & Education, Vol 53, No.3, pp 900‐912. Cortez, C., Nussbaum, M., Woywood, G., and Aravena, R. (2009) learning to collaborate by collaborating: a face‐to‐face collaborative activity for measuring and learning basics about teamwork, Journal of Computer Assisted Learning, Vol 25, No. 2, pp 126–142. Egenfeldt ‐ Nielsen, S. (2005) Beyond Edutainment: Exploring the Educational Potential of Computer Games, PhD thesis, IT University of Copenhagen. Facer, K. (2003) Screenplay: Children and computing in the home. London: RoutledgeFalmer. Gabriele, A. J. and Montecinos, C. (2001) Collaborating With a Skilled Peer: The Influence of Achievement Goals and Perceptions of Partners' Competence on the Participation and Learning of Low‐Achieving Students, The Journal of Experimental Education, Vol 69, pp 152‐178. Garton, A. F. and Pratt, C. (2001) Peer assistance in children's problem solving, British Journal of Developmental Psychology, Vol 19, pp 307‐318. Gee, J. (2003) What Video Games Have to Teach Us About Learning and Literacy, New York: Longman. Gee, J. (2004) Situated Language and Learning: a critique of traditional schooling, London: Routledge. Gillies, R.M., and Boyle, M. (2006) Ten Australian elementary teachers’ discourse and reported pedagogical practices during cooperative learning, The Elementary School Journal, Vol106,No.5 , pp 429–451. Harskamp, E., and Suhre, C. (2007) Schoenfeld’s problem solving theory in a student controlled learning environment, Computers and Education,Vol 49, No. 3, pp 822–839. Johnson, D. W., and Johnson, R. (1989) Cooperation and competition: Theory and research. Edina, MN: Interaction Book Company. Kafai, Y. B. (2001) “The educational potential of electronic games: from games‐to‐teach to games‐to‐learn”, [online], Cultural Policy Center, University of Chicago, http://culturalpolicy.uchicago.edu/conf2001/papers/kafai.html. Ke, F. and Grabowski, B. (2007) Game playing for maths learning: cooperative or not?, British Journal of Educational Technology, Vol 38, No.2 , pp 249‐259. Ke, F., and Hoadley, C. (2008) Computer games application within alternative classroom goal structures: cognitive, metacognitive and affective evaluation, Educational Technology Research and Development, Vol 56, No. 5, pp 539‐ 556. Kerawalla, L., and Crook, C. (2002) Children’s computer use at home and at school: Context and continuity, British Educational Research Journal, Vol 28, No6, pp 751–771. Kinzie, M. B., and Joseph, D. R. D. (2008) Gender differences in game activity preferences of middle school children: Implications for educational game design, Educational Technology Research & Development, Vol 56, No. 5/6, pp 643– 663. Lenhart, A., Kahne, J., Middaugh, E., Alexandra, M., Evans, C. and Vitak, J. (2008) Teens Video Games and Civics, Research study conducted for the PEW Internet and American Life Project . Liu, T. Y., and Chu, Y. L. (2010) Using ubiquitous games in an English listening and speaking course: impact on learning outcomes and motivation, Computers & Education, Vol 55, No. 2, pp 630–643. Lohman, M., and Finkelstein, M. (2000) Designing groups in problem‐based learning to promote problem solving skill and self‐directedness, Instructional Science, Vol 28, No. 4, pp 291‐307. Luckin, R. (2008) The learner centric ecology of resources: a framework for using technology to scaffold learning, Computers and Education, Vol 50, No. 2, pp 449–462. Malone, T. (1980) What makes things fun to learn?, Heuristics for designing instructional computer games, Palo Alto, USA. Mikropoulos, T. (2006) Presence: a unique characteristic in educational virtual environments, Virtual Reality, Vol 10, No.3, pp 197‐206. Mor, Y., Hoyles, C., Kahn, K., Noss, R. and Simpson, G. (2004) Thinking in Progress, Micromath, Vol 20, No. 2, pp 17‐23. Moreno, R. (2002) Who learns best with multiple representations? cognitive theory implications for individual differences in multimedia learning. Paper presented at World Conference on Educational Multimedia, Hypermedia, & Telecommunications. Denver, CO. Morgan, A. and Kennewell, S. (2005) The Impact of Prior Technological Experiences on Children’s Ability to Use Play as a Medium for Developing Capability with New ICT Tools. Becta: ICT Research Bursaries. NCTM (2000) Principles and standards for school mathematics. Reston, VA: NCTM Publications. Pivec, M. , and Pivec, P.(2008) Games in Schools. Learning. ISFE‐EUN Partnership. Retrieved from http://games.eun.org. Plass, J.L., Homer, B.D., Milne, C., Jordan, T., Ng, F., and Schwartz, R. (2010a) An Efficacy study on the integration of Optimized simulations into the high school Chemistry curriculum. Paper presented at the 2010 Annual Meeting for the American educational research Association (AERA), April 2010 in Denver, CO. Prensky, M. (2003) Digital game‐based learning. ACM Computers in Entertainment, Vol1, No.1,pp 1–4.

624

Reem Al‐Washmi, Gail Hopkins and Peter Blanchfield Quinn, C.N. (1994). Designing educational computer games. In K. Beattie, C. McNaught, and S. Wills (Eds.), Interactive multimedia in university education: Designing for change, Ambserdam: Elsevier, pp 45‐57. Simpson, G. Hoyles, C., and Noss, R. (2006) Exploring the mathematics of motion through construction and collaboration, Journal of Computer Assisted Learning, Vol 22, pp 1‐23. Strough, J., Berg, C. A. and Meegan, S. P. (2001) Friendship and Gender Differences in Task and Social Interpretations of Peer Collaborative Problem Solving, Social Development, Vol 10, pp 1‐22. Tambychik, T., Meerah, T. S. M., and Aziz, Z. (2010) Mathematics Skills Difficulties: A Mixture of Intricacies, Procedia ‐ Social and Behavioral Sciences: Elsevier, Vol 7, No.1,pp 171‐180. Tarim, K., and Akdeniz, F. (2008) The effects of cooperative learning on Turkish elementary students’ mathematics achievement and attitude towardsmathematics using TAI and STAD methods. Educational Studies in Mathematics, Vol67, No.1, pp 77–91. Tay Lay Heong (2005) Problem Solving Abilities and Strategies in Solving Multistep Mathematical Problems among Form 2 Students, Kertas Projek Sarjana, Universiti Malaya. Tudge, J. R. H., Winterhoff, P. A. and Hogan, D. M. (1996) The Cognitive Consequences of Collaborative Problem Solving With and Without Feedback, Child Development, Vol 67,pp 2892‐2909. Underwood, J., Underwood, G. and Wood, D. (2000) When does gender matter?: Interactions during computer‐based problem solving, Learning and Instruction, Vol 10, pp 447‐462. Walqui, A. (2006) Scaffolding instruction for English learners: a conceptual framework, International Journal of Bilingual Education and Bilingualism, Vol 9, No. 2, pp 159–180. Webb, N., Franke, M., De, T., Chan, A., Freund, D., and Shein, P., (2009) Explain to your partner: Teachers’ instructional practices and students’ dialogue in small groups. Cambridge Journal of Education, Vol 39, pp 49–70. Wood, D., Bruner, J., and Ross, G. (1976) The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, Vol 17, No.2, pp 89–100. Wu, S. S., Chang, T. T., Majid, A., Caspers, S., Eickhoff, S. B., and Menon, V. (2009) Functional heterogeneity of inferior parietal cortex during mathematical cognition assessed with cytoarchitectonic probability maps. Cerebral Cortex, Vol19,pp 2930–2945.

625

Training Flexible and Adaptive Arithmetic Problem Solving Skills Through Exploration With Numbers: The Development of NumberNavigation Game Boglárka Brezovszky, Erno Lehtinen, Jake McMullen, Gabriela Rodriguez and Koen Veermans Centre for Learning Research, University of Turku, Finland bogbre@utu.fi ernoleh@utu.fi jamcmu@utu.fi gabrod@utu.fi koevee@utu.fi Abstract: Traditional teaching methods often fail to develop the desired type of arithmetic problem solving expertise that is, flexibility and adaptivity in problem solving. Working with numbers, and exploring different number patterns and solution methods could represent a good basis for developing a more flexible and complex understanding of numbers and operations, which could result in the development of flexible and adaptive problem solving skills. Our aim when developing NumberNavigation (NNG) educational digital game was to provide an engaging and adaptive context for exploration with numbers. In the game, players progress by strategically selecting different sequences of number‐operation combinations, using the four basic arithmetic operations within the domain of natural numbers (1‐100). Players need to collect raw materials for building settlements, by navigating their ship through different maps placed over the 100 square. The game challenges players to strategically select the most optimal path and continuously adapt their choices. In this explorative case study, the goal was to test core game features and their relationship with in‐game arithmetic flexibility. A trial game version was tested in two consecutive sessions with three elementary school children. Methods of data collection included video recorded observation, open‐ended interview questions, and the screen capture of gameplay. Aiming to trigger more variation in players’ use of number‐operation combinations, we have tested three different game modes. In exploration mode players had to choose their moves so they bypass islands placed over the maps as obstacles. In the minimum moves mode, players had to make their journey using the least amount of moves, and in the minimum energy mode players had to reduce the magnitude of numbers used for navigating their ship. Results show that players used various number‐ operation combinations and they were able to adapt their in‐game problem solving strategies according to the changing rules of the game. The position of harbours‐islands‐targets proved to be a strong predictor of the amount of variation in number‐operation combinations and of players’ explorative tendencies. Overall, results suggest that basic game features are promising in triggering exploration with numbers and engaged practice over an extended period. Results of the study will be used to inform the future development of NNG. Keywords: adaptive expertise, arithmetic problem solving flexibility, game design, strategy

1. Background Developing adaptive expertise in arithmetic problem solving is one of the main objectives of elementary school mathematics education. Due to the complexity of this skill, traditional instructional methods might not be sufficient in this process. Although, there are different views regarding the meaning of adaptive expertise, it is usually agreed that the development of fluent and flexible procedural skills together with a deep conceptual understanding of numbers and operations are both necessary requirements (Baroody 2003; Schneider, Rittle‐ Johnson & Star 2011; Verschaffel et al. 2009). Flexibility and adaptivity in arithmetic problem solving represent strong markers of arithmetic expertise. Flexibility refers to the ability to apply various strategies during problem solving while adaptivity describes the strategic choice of the most optimal type of solution for a given problem which may depend on various factors such as problem characteristic, individual differences in problem solving skills, or the norms and rules of the context where the problem is solved (Verschaffel et al. 2009). Research shows that while placing emphasis on the development and practice of problem solving strategies can enhance flexibility, and that students of various skill levels can acquire and use different problem solving strategies, in practice these skills are rarely transferred to new problems. Despite students’ familiarity with various methods they often disregard both the characteristics of the task and their individual strategy performance when selecting problem solving strategies (Torbeyns et al. 2009). Similarly, despite

626

Boglárka Brezovszky et al. understanding when and how different problem solving strategies are more adaptive than others, students often fail to apply this knowledge in practice (Blöte, Klein & Beishuizen 2000). The discrepancy between the ability to understand different problem solving strategies and the ability to apply them adaptively in new problem situations is often explained as a shortcoming of instructional methods that place too much emphasis on teaching how to solve problems, concentrating more on the surface features of a problem and less on the underlining principles. As the development of flexible and adaptive skills in arithmetic problem solving largely depends on the ability to understand numbers and operations, instead of learning to choose from a set of different methods, the focus should rather be placed on developing well‐connected knowledge of numbers and operations (Baroody 2003; Threlfall 2009). By exploring and experimenting with different number combinations, children might develop a more richly connected conceptual understanding of numbers and the relations between different numbers and operations. Thus, important features and relationships between numbers in a problem might become more visible, and so more optimal solutions could be more readily recognized (Schneider, et al. 2011). Providing children with learning environments which trigger active exploration with numbers and reflection on problem solving processes, should enhance the development of stronger connections and the understanding of underlining relationships between number and operations and as a result trigger the development of flexible and adaptive problem solving skills (Baroody 2003; Verschaffel et al. 2009). For example, in their study Canobi, Reeve & Pattison (2003) examined 5‐8 year old children’s understanding of commutativity and associativity principles in addition problems and found that irrespective of age, those children who understood that numbers can be decomposed and recombined had a higher accuracy and more flexibility in their problem solving procedures. In equation solving, Rittle‐Johnson & Star (2007) found that reflection in terms of comparing and contrasting different solution methods proves to be an efficient method for noticing important features of a problem leading to the development of more flexible problem solutions and exploration of different alternatives. Additionally, there is evidence that using different mental models such as the hundred square or the mental number line might be beneficial for triggering the exploration of alternative solutions and the understanding of underlining principles between numbers and operations (Beishuizen 1993; Blöte, Klein & Beishuizen 2000). Games by their nature are ideal for triggering exploration and discovery and can provide a motivating context to experiment with numbers (Devlin 2011). Additionally, educational computer games can provide novel pedagogical approaches to deal with educational content that has been challenging for traditional teaching methods (Siewiorek et al. 2012). However, up to date there is no educational digital game with the explicit aim to trigger flexibility and adaptivity in arithmetic problem solving. Examples of games which aim to develop mathematical skills, and which build on a strong theoretical background in their game design providing empirical evidence of their efficiency are rare (Young et al. 2012). One relatively well‐researched domain in the field of games and mathematics is represented by games that aim to develop early numerical understanding by strengthening the mental representations of numbers. The mental number line has been successfully used in developing both the spatial representation of numbers and basic arithmetic problem solving skills of young children (Kucian et al. 2011). Other game designs with similar aims focused on strengthening the connection between different representations (e.g. symbolic and non‐ symbolic) of numbers (Wilson et al. 2006; Räsänen et al. 2009). Although these games are exemplary for building upon strong behavioural and neuro‐cognitive evidence in their game design, and provide adequate empirical evidence regarding the learning gains associated with gameplay, they are also highly specialized environments aiming to help young children with more severe mathematical difficulties (developmental dyscalculia). Arithmetics constitutes only a limited part of these games, and includes only basic addition and subtraction within a limited number domain. Furthermore, what might be a highly efficient game design for treating developmental dyscalculia in young children might not work for more various samples and mathematical skills. Unfortunately, in the domain of more advanced arithmetic skills, the amount of empirically tested games is even smaller. What is more, even existing games are often criticized for their inability to integrate their game design features with their mathematical learning aims (Young et al. 2012). A meaningful educational game

627

Boglárka Brezovszky et al. design requires embedding the educational content within the core game features (Devlin 2011) in a way that it becomes an integral part of the gameplay instead of a necessary extra placed on top of a game (Habgood 2007). Until now, there has been only one example of an educational digital game with published empirical results that manages to create a fusion between educational content and game design, as well as target the development of complex arithmetic skills such as the discovery and understanding of number patterns (Habgood & Ainsworth 2011).

2. Aims Our aim when developing NumberNavigation Game was to create a motivating game environment that would provide children a context to experiment with numbers and arithmetic operations. It was our goal to design a game where strategic choice in selecting the most optimal sequence of number‐operation combinations would represent those core game features by which players make meaningful progress. In this study, we have tested a prototype of NNG in order to gather rich qualitative data to be used in our future game design processes. Our aim was to explore the type of emerging game patterns that could suggest the presence of in‐game arithmetic problem solving flexibility and to explore those game features or combinations of game features where these patterns are more prominent.

3. Design principles of NumberNavigation game In NNG the player has to navigate a ship through different maps and collect raw materials for building settlements. Navigation takes place with the help of arithmetic operations that bring the ship to different locations in a map, which is based upon the hundred square. Within each map, players have to collect four types of building materials (wood, brick, stone and iron) placed in different locations and return them to the harbour. Harbours are located on a fixed number location within a map, which changes across different maps. Figure 1 illustrates a map from NNG, here the harbour is located at number 4 and the player has to pick up wood from number 94. In this example, a player might instinctively try to navigate on a straight line, adding 90 to reach the target, but this move is blocked by an island. A blocked move is signalled by a red path and a cross (see Figure 1). Islands always represent challenges the player has to consider when selecting a move. Alternatively, the player might consider 4*3 as the first correct move, which is signalled by a green path along which the ship moves to the next location, number 12. In NNG, apart from the islands, the efficiency of the sequence of moves a player uses towards the target depends on additional challenges. The game has three modes: exploration, minimum moves and minimum energy. In the exploration mode, the only challenge is to select moves so that the ship bypasses the islands. In the minimum moves mode the goal is to reach and return the target using the least amount of moves (operations). In the minimum energy mode the aim is to reduce the magnitude of numbers the player types into the command box to navigate the ship from one number to the next. For example, in case of the map presented in Figure 1, if the map is in moves mode, then continuing the journey from 12 to 48 would be a good option, reaching the target in three moves: 4*3=12; 12*4=48; 48+46=94. However, if the map is in energy mode, then this option would cost the player 53 energy units (the total value of numbers used to navigate the ship: 3, 4 and 46). If needed, this energy could be reduced by considering alternative options and using for example the following path which would only take 37 energy units: 4*11=44; 44‐6=38; 38*2=76; 76+18=94. Naturally, many alternatives are available at once, and it is the task of the player to decide which option would be more efficient for reaching the target, considering the position of harbours, materials, and islands, as well as individual maps’ game modes. As the task is to reach then return each material, the player can always use the same numbers and inverse operations on the return, but also has the opportunity to re‐evaluate and adapt the original strategy and chose a better path on return.

4. Methods 4.1 Participants Game testing took place in two consecutive sessions. During the first session J. (age 11) played the game individually while during the second session T. (age 9) and E. (age 11) played the game collaboratively. All three participants were male elementary school students confident in using digital games (both educational and recreational) and had a self‐reported positive attitude towards mathematics.

628

Boglárka Brezovszky et al.

Figure 1: Feedback in case of right move and blocked move in NNG

4.2 Instruments and procedure Data was collected using the methods of video‐recorded observation, interviews and recording of the gameplay. Players were recorded using a digital video camera. On‐screen activity was recorded using Camtasia Studio 5.1.0 screen capture software. Video data from the two sources were synchronized and analysed simultaneously using Transana 2.42 software. Interview questions consisted of open‐ended questions regarding players’ game experience. Game sessions lasted around one and a half hours, including instructions and interview questions. The game was played in three different modes introduced in the same order. First the exploration mode, followed by playing the game according to the rules of the minimum moves mode, and lastly playing the game according to the rules of the minimum energy mode. During collaborative play, participants had an extra option to decide which game mode they would like to apply for the last map (choice mode). The tested game version included three maps with 4 different target materials in each map. The tested maps and the location of harbours and target materials within each map is presented in Figure 2. Through the article, maps will be referred to according to their names used in Figure 2 (Map A, B & C).

Figure 2: Position of harbours and target materials within the three tested maps of NNG

5. Results Table 1 presents the sequence and number of maps that players completed during the three different game modes as well as the amount of moves (arithmetic operations) made. Aside from successful moves, blocked moves were also logged and differentiated. Blocked moves include both operations that were impossible to

629

Boglárka Brezovszky et al. execute due to islands blocking the path as well as operations which did not fit within the frame of natural numbers 1‐100. During collaborative play, players chose to complete the last map under the minimum moves mode. Table 1: Sequence and number of maps in different game modes; total number of clear and blocked moves Sequence & Number of Maps Moves Blocked moves Ind. Coll. Ind. Coll. Ind. Coll. Exploration A,B,C A, B 92 57 18 9 Minimum moves A,B C, A, B 52 68 9 4 Minimum energy C C, A, B 19 67 1 8 Choice ‐ C ‐ 16 ‐ 4 Total 6 9 163 208 28 25 Game Mode

Note: Ind. = Individual gameplay; Coll. = Collaborative gameplay As Table 1 shows, compared to the total number of moves the incidence of blocked moves was relatively low. Reasons for blocked moves were more often technical than mathematical, as from certain positions paths looked possible but were blocked by land. Total time on task (playing the game) was 1 hour for the individual game, and 1 hour 20 minutes for the collaborative game. Completing all three maps in one game mode took around 25‐30 minutes irrespective of the mode. Completing one map could take 5 to 10 minutes depending on the difficulty. The time needed for retrieving a material varied from 50 seconds to 5‐8 minutes, indicating that regardless of the game mode, certain quests proved to be more challenging than others. In order to identify signs of in‐game arithmetic flexibility and possible effects of the different game modes on players’ strategies, the types of number‐operation combinations used were explored across different game modes. Figure 3 and Figure 4 show the use of numbers and number‐operation combinations in each of the three game modes for individual and collaborative gameplay. There was a clear trend in using 10 and multiples of 10 during both game sessions. This trend becomes even stronger in case of larger numbers >40 and >50 where the players almost exclusively used round numbers (numbers ending in 0). Looking at the use of numbers over the two game sessions, five numbers seem to emerge as frequently used operands: 1, 5, 9, 10, 11, and 40. From these frequently used numbers, the use of 1 and 5 can be explained by the fact that they represent easy moves from an arithmetic perspective. The high incidence of 40 is an artefact of Map B where +40 and ‐40 was very frequently used for all four target materials and during both gam