Kahoot!ing in the Office - Process Book

&

Learning from Play at Work: The Art of Kahoot!ing in the Office

Zimu Guo Veronika Hanson Corynne Umeda Emily Zeng

Table of Contents Overview 03 Executive Summary 04 Team Bios 05 Process Overview

Design Process 06 11 12 14 16 17 18

Research Design Prototype Test Iteration User Journey Map Final Solutions

Reflection 22 23 28 30

Future Directions Final Reflections Sources Cited Appendix

2

Executive Summary Corporate trainers use Kahoot!, a gamified learning platform to promote learning retention among their trainees. Trainers often use their PowerPoint presentation materials to come up with Kahoot! quizzes and then launch these quizzes for training assessment during PowerPoint presentations. This creates a disjointed experience for the users, as they have to switch back and forth between the two platforms. So we tasked ourselves with the question:

How might we help corporate trainers prepare and hold training presentations in a more efficient and enjoyable way?

Through our extensive research consisting of literature review, competitive analysis, heuristic evaluation, forum analysis, and interviews with corporate trainers, we have been able to create a detailed user journey map that addresses trainers’ specific pain points in their current workflow. Besides, we learned that trainers also desire more flexibility in identifying participants.

Thus, our goal was to provide a seamless user experience for corporate trainers when using Kahoot! and Microsoft Office 365 products in their training preparation and execution. To do so, we created an interactive prototype that:

1.

Integrates Kahoot! as an add-in in PowerPoint to improve trainers’ workflow

2.

Uses Kahoot! meeting invites in Outlook Calendar to allow for tracking of trainee performance

3

Team Diversify & Conquer

Zimu Guo | UX Designer

Veronika Hanson | UX Researcher

Corynne Umeda | UX Engineer

Emily Zeng | UX Researcher

Zimu is a UX designer who loves solving complex problems to provide the best solutions for people. Zimu’s background in business, psychology, web design, and UX helped her develop a multidisciplinary skill set. Zimu specializes in interaction design, product strategy, problem solving to create intuitive user experience.

Veronika’s research experiences range from initial identification of problem spaces through background research, to persona development and user testing with a large toolkit of qualitative and quantitative methods. Her advertising experience has helped her become an active listener and analytic, advocating for consumer needs and improved usability. Her favorite part is connecting the dots in data and identifying key insights that will guide a project.

With a background in computer engineering, Corynne has experience with various coding languages and concepts, physical computing devices like the Arduino and Raspberry Pi, and SQL relational databases. She has dabbled with the Adobe Creative Suite and data visualization design.

Additionally, Corynne has experience in leadership and management through various officer roles in college clubs. She is very excited for the opportunity to collaborate and pursue her passion of creating the best user experiences with technology.

In her six years of research experience, Emily has gained skills in survey design, conducting interviews, data analysis and visualization, and presenting findings. In addition, she has published two first-authored research articles on mobile app usability. Emily believes great design starts with putting the user first and utilising research to unmask their goals and behavioral patterns.

4

Process Overview

RESEARCH

DESIGN

TEST

ITERATE

5

Research In-depth interviews (n=6) and past Kahoot! research revealed that trainers regard Kahoot! as a fun and easy-to-use tool. Most trainers use PowerPoint for training presentations and would find its integration with Kahoot! helpful. Trainers also desire more flexibility in identifying participants.

6

Research

Literature Review

Competitive Analysis

In our literature review, we gathered information from academic publications on learning retention, educational games, and Kahoot!. We also looked at case studies of Kahoot! for business to guide our initial research objectives.

We learned that learning retention is a significant problem space in both academic and corporate environments (Kohn, 2016).

We conducted a competitive analysis to find out what similar products were offering. Though there are many similar learning tools (e.g., Quizlet, Socrative, Quizizz, Poll Everywhere, Mentimeter), Kahoot! stands out.

Gamified learning can improve engagement and motivation (Perrotta, Featherstone, Aston, and Houghton, 2013). This finding is validated in Kahoot! case studies; corporate trainers notice that Kahoot! keeps trainees motivated and helps reinforce learning materials (Cabral, 2018; RunstrÓ§m, 2018; Shelly, 2018; Bodo, 2018; Bansal, 2018; Silva, 2018; Golubeva, 2017).

However, these case studies are limited in that they do not explore problem areas of Kahoot! and in-depth ways to improve the platform.

Kahoot! provides the fun and competitiveness of gameplay. In addition, Kahoot!’s unique interface allows players to submit answers on their personal device after viewing questions on the presenter screen. Since users no longer concentrate on their device for reading questions, this provides users with the opportunity to collaborate and participate more in group activities (Bawa, 2018). Another advantage is that Kahoot! Pro and Plus have features that are tailored to businesses.

We also learned about another competitor called Send Steps, which is a PowerPoint add-in, through talking to a trainer. However, the process for creating a question was drawn out.

7

Research

Forum Analysis

Subject Matter Expert (SME) Interview

We analyzed popular posts on Kahoot! forums to determine user needs and pain points. Although these posts were mainly about Kahoot! in the education space, this information is still valuable for elucidating usability issues that corporate trainers might also experience.

Kahoot!ers wanted:

We conducted a 30-minute SME interview with a Microsoft sales leader who plans and carries out global training sessions with large audiences. Overall, the trainer felt that Kahoot! helps trainees take away key points from dry, long, and boring training sessions in a way that “drives engagement and smiles�.

1, Customizability of questions and answers Users wanted more types of questions (e.g., matching, freeform answers) and freedom in changing the number of possible answers.

Some challenges of using gamification in the workplace that the trainer mentioned is the perception of winners and losers in games, which might not be the best fit in environments where competition is not integral.

2. Extension of Kahoot! outside of the classroom Kahoot! for business addresses user requests to enable students to play Kahoot! on their own time.

The trainer desires cross-comparisons of data from each training session done at a high-level because they did not need to track individual performance. They also wanted more reliable game-joining because the system sometimes kicks out audience members, and they cannot rejoin the game.

3. Identification of Participants Sometimes nicknames hinder teachers’ ability to identify students and easily track their progress.

8

Research

Initial User Interviews

Heuristic Evaluation

We conducted two 30-45 min online user interviews with three corporate trainers who have used Kahoot! in their trainings. Our main takeaways are that:

We focused our evaluation of the Kahoot! platform from the trainer’s perspective by each going through the quiz creation and presentation process and comparing it to 10 usability heuristics (Nielsen, 1995). Heuristics violated were ranked from minor to critical (Sauro, 2013). Major problems areas include:

Pros ✓ ✓ ✓

Cons —

— —

Trainers find Kahoot! easy to use Kahoot! keeps the audience engaged Kahoot! brought healthy competition to the office

Trainers thought that there was a lack of flexibility with questions and answers (e.g., character and format limits) Technical glitches prevented people from joining Usability issues and limitations with data reporting analytics

Key Suggestion ★ A trainer wanted a Kahoot! integration inside PowerPoint, which was their primary presentation software

1.

Correct Answer Selection - Moderate No clear and direct instructions telling users to select the correct answers before proceeding.

2.

Saving Quiz - Critical Kahoot! doesn’t allow users to save quiz until at least one question is created, which might cause users to lose content after filling out initial quiz information (e.g., description, tags, etc.).

3.

Game Options - Critical Heavy text buried under game options with no visual hierarchy. Some were uncomprehended elements.

9

Research

In-Depth User Interviews

Other Research

We conducted a total of 6 in-depth phone and online interviews with trainers who have used Kahoot!. Overall, Kahoot! helped trainers accomplish their goals, promoting team bonding and improving learning outcomes.

Our research informed our user journey map that is centered around a trainer’s persona. We also conducted usability testing of our first prototype with five users.

These will be discussed in the following sections.

On the other hand, trainers felt limited by question word limits, question types, and lack of music variety.

Trainers suggested being able to create a question without creating an entire quiz, increasing the question time frame to more than two minutes, and consolidating the reports into one.

Most trainers have not engaged with Kahoot! in the context of Office 365 products, but they would find the integration of the two to be useful.

“It would be nice to be running your PowerPoint presentation that could link up to a Kahoot! quiz” P5

10

Design Client feedback on our concept sketches helped us consolidate our integration ideas for PowerPoint and discover an integration opportunity for Outlook as a response to users’ needs for flexibility in identifying trainees to track learning outcomes. Sketches were developed into an interactive prototype.

Design

Prototype v.1

1

2

Creating Kahoot! in Powerpoint

1.

Kahoot as a Powerpoint add-in feature with customized ribbon

2.

Incorporating Kahoot!’s current design for quiz creation on the current slide

3.

3

Choose a quiz type on the task pane

12

Design

2 1

Prototype v.1 New functions in Powerpoint’s top ribbon and in presenter view

1.

Choose quiz type from ribbon as a drop-down menu

2.

Add anonymize participants option

3.

“Go to: Slide 2” will take trainers to the linked slide for this Kahoot! question. Add a “Pause Kahoot!” option in trainer’s presenter view

3

3

13

Test In the usability test (n=5) of our prototype, most tasks were successfully completed and received a low difficulty rating. Most usability problems related to the lack of microinteractions, design inconsistencies between the add-in and PowerPoint, and confusion around the Kahoot! workflow.

14

Test

Usability Testing Successes: ● All 5 participants successfully completed 6 out of 9 tasks ● The majority of the participants successfully completed the other 3 tasks ● All tasks received a median difficulty rating no lower than 5 on a scale of 1 (very easy) to 7 (very difficult)

Issues: ● Lack of microinteractions ● Design inconsistencies between the add-in and PowerPoint ● Confusion around the Kahoot! workflow

15

Iteration The final iterated design addressed usability issues by making the Kahoot! add-in adhere to PowerPoint design guidelines, simplifying the quiz creation process, adding the ability to link quizzes to Kahoot! events from Outlook, and improving the overall workflow.

User Journey Map

Before

Background Kevin is a trainer at a large technology company. He has a training event with 100 participants coming up in a week.

He plans on using Kahoot! to better engage his audience and reinforce the content from his Powerpoint.

Afterwards, he also wants to gather feedback and measure learning outcomes.

Goals and Expectations 01 Motivate trainees and increase the energy levels during training sessions 02 Increase knowledge retention and have fun at the same time

17

Iteration

Final Solutions

Create Kahoot! Event in Outlook As a solution, we added a Kahoot! event option on Outlook’s ribbon. This helps trainers create a Kahoot! event invite in Outlook, so that when the attendees join the game by the link provided in the email invite, the participants’ linked corporate IDs are revealed to the trainer during gameplay and reporting. This is important for mandatory training, when tracking individual performance is desired.

18

Iteration

Final Solutions

Link Kahoot! Event in Powerpoint

Trainers can select “Import Kahoot!� on the top ribbon to link to or search for existing Kahoot! events in Powerpoint.

Trainers can also create new events.

19

Iteration

Before

Final Solutions

Create Content and Kahoot!

Questions inside PowerPoint

After

We included four options for the types of questions available to trainers: multiple choice, drag & drop, poll, and open-ended.

Trainers can select each type through the ribbon to create Kahoot! questions. For example, after “Drag & Drop” is selected, the question creation page pops up.

We also improved the question creation page from Kahoot!’s existing design: we reconstructed the sequence of creating questions, labeled the necessary steps for completing the task, added icons to the number of potential answers and the “link slide” function to increase usability and discoverability, to make it more intuitive for trainers to use Kahoot! inside Powerpoint.

20

Iteration

Final Solutions Present and Play Kahoot!

Trainers can run the Kahoot! quiz in PowerPoint presentation mode. More options, such as pausing a Kahoot! for participants to discuss the questions and linking Kahoot! questions to specific slides in the Powerpoint were added into the Powerpoint presenter view.

21

Future Directions The scope of this project was limited due to the time constraint of Capstone (two university quarters). If we had more time, we would extend our project to cover the following: 1.

Get Kahoot!’s feedback to evaluate solution’s fit with the business strategy.

2.

Evaluate the iterated prototype with trainers who have used Kahoot!.

3.

Explore other Office 365 integrations with Kahoot! (e.g, Excel) to gather performance and feedback data for analysis and add-ins’ improvement.

4.

Attend conferences for corporate trainers that Kahoot! participates in to recruit a focus group of trainers to gain further research insights.

5.

Focus on accessibility by making sure the prototype follows the best practices of inclusive design and evaluating the prototype against Microsoft’s list of accessibility traps.

6.

Build a functional Kahoot! add-in in PowerPoint and Outlook.

7.

Conduct additional research to fill in the gap of remote training use cases (evaluate the need for Skype and Microsoft Teams integrations).

22

Reflection

23

Reflection

Contribution to the HCD Field

Implications

Research in gamified learning is more extensively covered in education, but there is little information on the application of gamified learning in the workplace. Our research provides rich contextual data into how trainers use gamification in the workplace and the pain points they face.

Our end product address the needs of trainers by elevating both the productivity and joy of corporate training. Kahoot! as an add-in brings fun and interactive gameplay into Microsoft Office productivity tools, which satisfies trainers’ and trainees’ needs and goals.

Our research and design insights into how Kahoot! can be integrated with Office 365 products for the corporate space can be extended into the educational setting as well. For example, teachers commonly use PowerPoint presentations to present their material and could benefit from a well-designed Kahoot! add-in as well. From our forum analysis, we found that teachers also need to track individual performance sometimes and the nicknames are a barrier to doing so. Teachers also need to explain answers to students during Kahoot! reviews, so they could also benefit from the pause and link slides functionality.

The project’s process also brings value to the HCD community by demonstrating an innovative approach to collaboration. We worked with two clients at the same time and balanced their business goals and design requirements with user needs. Starting the project as an independent team gave us the advantage of being an unbiased intermediary, which allowed us to find novel and meaningful integration solutions where both of our clients’ needs intersected. Also, since our collaboration with Kahoot! was entirely remote, our project could make for an interesting case study in the field of Computer Supported Cooperative Work.

Our solutions can be applied with some minor adaptations as needed to fit organizations’ requirements. For instance, the Outlook integration of Kahoot! may not be applicable in all educational settings, but colleges may find Kahoot! integration with their own email services helpful.

Some ethical implications of our design regard being able to identify individuals when trainers choose to “de-anonymize” participants. We discuss the potential dangers in the next section.

24

Reflection

Limitations, Dangers, & Opportunities Although we strived to recruit a large and diverse pool of participants, we were limited to several phone interviews with a handful of participants due to the time constraint of our research phase. Even though the corporate trainers we interviewed came from various backgrounds (medical, technology, sales), we realize that the research results may not be representative of the entire trainer population in all fields and do not account for all racial, socio-economic, or gender differences. More larger-scale studies (e.g., surveys) are needed to capture such variations and find out if they are meaningful for the context of this product.

Besides, we haven’t tested how our proposed design will affect trainees’ user experience or performance outcomes, so there is an opportunity to do additional research with this group of users to reach a more holistic understanding of the area. In addition, we did not explore design considerations to address pain paints of trainers engaged in remote training, which is a big problem space for companies with multiple locations, presenting another opportunity for more research.

We focused on designing for accessibility and inclusiveness when building out our final prototype, including using specific colors and fonts that follow Microsoft's design guidelines (Microsoft, 2018). However, more post-design research is needed to check against all existing accessibility guidelines, as the scope of our work was limited to design concepts.

Since Kahoot! involves data collection (e.g., user identities) and data analysis (e.g., Kahoot! report), there is a risk of personally identifiable information (PII) breach. We were concerned about data protection and were unsure how Kahoot! and Microsoft collect and store users’ data in their systems, so there is a need for more clarity on what PIIs are being collected. why they are being collected, and security measures taken to protect them. The key to eliminating danger also lies in how overall data within organizations is protected in the system. Since data governance policies in the US are state and sector specific, and Kahoot! is based in Norway, integrations should aim to comply with the GDPR (General Data Privacy Regulation), the EU’s data governance policy (Voigt & Von Dem Bussche, 2017).

25

Reflection

Personal Growth - Group Besides all the professional opportunities to do research and design for the domain that has not been studied much yet, we got plenty of opportunities to learn and grow as researchers and designers during our Capstone process.

As a team, we learned to collaborate more effectively by supporting each other and providing honest and straightforward critique on the work for mutual benefit.

We learned to scope down as we go for better result and to be nimble with our methods. When we weren’t able to do contextual inquiry as one of our research methods as we couldn’t get access to local participants, we quickly re-calibrated to find alternative methods to address our research question. When we had difficulties when connecting with one of our sponsors, we leaned towards the support we got from our Capstone lectures. We took the initiative to connect with potential participants via email and social media.

We also learned to take every opportunity to get feedback from and collaborate with our sponsors to make sure we are going in the right direction and ensure our goals are aligned. Besides that, our sponsor's design guidelines have greatly helped us create a more inclusive final design.

26

Reflection

Personal Growth - Individual

“I learned a lot about how to work with multiple different clients and stakeholders during this project. I also learned a lot about teamwork, compromise and being more assertive with my ideas.” - Corynne

“The capstone allowed me to conduct applied research in a massive problem space and work with professional stakeholders. Talking to users gave me many insights on potential solutions, but narrowing them down and ensuring they were research-driven was challenging!”. Emily

“The deadlines were tight, but they gave me an opportunity to learn to work more efficiently with the team and stakeholders, to be clear expressing my perspective and asking for feedback, to compromise and stay patient with temporary ambiguity.”- Veronika

“Capstone gave me a unique opportunity to explore a vast problem space and solve complex problems collectively. Having a workshop with our stakeholders on brainstorming and ideation provided valuable feedbacks that pivoted our project to the right direction.” - Zimu

27

Sources Cited Bansal, N. (2018). Liftoff: Break barriers and build relationships at work with Kahoot!. Retrieved from https://kahoot.com/blog/2018/02/19/liftoff-case-study-build-relationships-work-kahoot/ Bawa, P. (2018). Using Kahoot to Inspire. Journal of Educational Technology Systems, 0047239518804173.

Bodo, R. (2018). Master the art of virtual training with Kärcher Kohn, A. (2016). Brain science: The forgetting curve–the dirty secret of corporate training.

Cabral, A. (2018, March 26). Double knowledge retention in sales training? Easy with games!. Retrieved from https://kahoot.com/blog/2018/03/26/gamify-sales-training-double-knowledge-retention/ Golubeva, D. (2017). Getty Images: “Kahoot! Is built into our organizational culture”. Retrieved from https://kahoot.com/blog/2017/12/05/getty-images-case-study-kahoot/

Guidelines - Windows applications. (2018). Retrieved from https://docs.microsoft.com/en-us/windows/desktop/uxguide/guidelines Nielsen, J. (1995). 10 usability heuristics for user interface design. Nielsen Norman Group, 1(1).

28

Sources Cited Office Add-ins platform overview - Office Add-ins. (n.d.). Retrieved from https://docs.microsoft.com/en-us/office/dev/add-ins/overview/office-add-ins Perrotta, C., Featherstone, G., Aston, H., & Houghton, E. (2013). Game-based learning: Latest evidence and future directions.

Slough: NFER. RunstrÓ§m, L. (2018). End all your training sessions on a high note with Kahoot!. Retrieved from https://kahoot.com/blog/2018/10/25/training-tips-tech-data-sweden/ Sauro, J. (2013, July 30). Rating the severity of usability problems. Retrieved from https://measuringu.com/rating-severity Shelly, C. (2018). Fairtrade: mastering policies through play and teamwork. Retrieved from https://kahoot.com/blog/2018/08/22/fairtrade-case-study-policy-training/ Silva, L. (2018). 7 secrets of engaging sales training from Facebook LATAM. Retrieved from https://kahoot.com/blog/2018/01/08/facebook-latam-case-study-sales-training/ Voigt, P., & Von dem Bussche, A. (2017). The EU General Data Protection Regulation (GDPR). A Practical Guide, 1st Ed., Cham: Springer International Publishing.

29

Appendix Figma prototype: https://www.figma.com/file/G4pXpkxZdMFppMHhxlJdFW/Kahoot-and-Microsoft-Integration-Process-Book Final design walkthrough: https://drive.google.com/open?id=1LC_3xmgbd-t1dAAeYxcqq3Gkar0S_JoD Promotional video: https://drive.google.com/open?id=1r22fDUCwqE8PmfG3akiG-Ma3RmaniIJu

30

Thank You!