Executive Summary
The BrightVision Consultants agreed to evaluate the current lessons of the “Electronic Problem-Solving Training” (ePST) developed at the University of Georgia. This online intervention aims to support survivors of traumatic brain injury (TBI) by enhancing their problem-solving skills. The project goal is to provide insight into the lessons' strengths and weaknesses and offer constructive suggestions for improvements. The evaluation focused on factors such as language quality, accessibility, user engagement, effectiveness of interactive elements, and the microlearning approach.
BrightVision Consultants were granted operational freedom regarding the deliverable containing evaluation feedback. The long-term goal is to utilize the feedback and evaluation tools provided by BrightVision Consultants to enhance the quality of both future microlearning modules and those currently in development for TBI survivors.
To meet the project goal, we developed an evaluation form with a mixedmethods data collection approach, presenting both quantitative and qualitative data. Quantitative data was collected using survey questions with a Likert scale ranging from one (strongly disagree) to five (strongly agree). Qualitative data came from open-ended survey questions. These questions helped review the content and design for quality, relevance, and clarity, ease of navigation, visual appeal, microlearning approach, and effectiveness of the interactions. We captured experiential data from both mobile devices and desktop computers.
Executive Summary
Key Findings and Recommendations
• Strengths
High user engagement and effective interactive elements.
• Weaknesses
Identified areas for improvement in language clarity and visual appeal.
• Recommendations
Specific suggestions for enhancing language quality, accessibility, and interactive components.
Impact and Future Steps
The feedback provided will be instrumental in refining the current ePST modules and guiding the development of future microlearning modules for TBI survivors. The continuous improvement process aims to ensure that the ePST remains an effective and engaging tool for its users.
Content Evaluation
The reviewed lessons generally present high-quality and relevant content that aligns with the learning objectives and activities.
Some examples include:
1. In all of the lessons, the segmented structure is marked by visual graphics such as check marks, which help users track their progress effectively.
2. In all of the lessons, the use of a persona adds a personal touch and makes the content more relatable.
3. Several lessons feature introductory testimonials from individuals affected by Traumatic Brain Injury (TBI), which helps engage learners from the onset and makes the content more relatable.
4. Several lessons included music with logo intros and multimedia themes which enhanced the overall learning experience due to the engaging and dynamic presentation.
Content Evaluation
Design Evaluation
The evaluated lessons exhibited aspects of both effective and ineffective learning design.
Positive aspects included effectively using multimedia elements such as images, audio, and video to enhance learning. The structure of many lessons was generally intuitive with ample opportunities for interactive engagement, which will be well-received by learners. The visual design of the lessons featured appealing layouts that aid in focusing on content. Elements like badges will increase learner engagement and provide a motivational impact.
In addition, the microlearning format, with its bite-sized chunks of information and interactive elements, is effective and will be appreciated by learners. This approach makes learning more manageable and allows learners to consume content at their own pace.
Design Evaluation
However, we highlighted several common design issues across multiple lessons:
1. Navigation problems were a recurring concern, especially on mobile devices.
• We frequently encountered difficulties locating and interacting with elements referenced in the content, such as arrows and tabs.
• Instructions were frequently unclear with inconsistent navigational clues, which may lead to confusion among learners.
For example, in Lesson 2.1, the instructions indicate to select each tab for more information, however there is not a tab interaction present.
Design Evaluation
1. Inconsistencies in design elements will potentially cause frustration among learners and fall short of compliance with 508 accessibility standards.
• These inconsistencies included varying font sizes, alignment, and color schemes which impacted the readability and aesthetic coherence of the learning materials, specifically for those with low vision or color blindness.
• Improving accessibility features for learners with disabilities, including those with fine motor skill challenges or traumatic brain injury (TBI), is crucial. Ensuring navigable elements across various devices will create a more inclusive learning environment.
For example, the font sizes in the screenshot below would be considered too small for a learner with low vision.
3. Functionality issues also detracted from the overall user experience.
• These included malfunctioning interactive elements, problems with scrolling functions, and issues with multimedia interactions on mobile devices.
4. In some cases, AI voices and facial expressions did not align with the tone of the content, which may be distracting for learners.
Design Evaluation
Despite these challenges, the structured approach of the lessons and the effectiveness of interactive elements when functioning correctly contribute significantly to the overall effectiveness and engagement of the learning experience. While the microlearning approach shows promise in engaging learners and delivering content effectively, the persistent usability and technical issues undermine its full potential.
Addressing these challenges—such as improving navigation consistency, optimizing design for mobile usability, and ensuring technical functionalities work seamlessly across devices—is crucial for enhancing user experience, maximizing the effectiveness of the microlearning format, and increasing learner satisfaction. Additionally, maintaining accessibility standards across all aspects of the lessons will improve inclusivity and accommodate diverse learner needs.
Additional Areas for Evaluation
Additional Areas for Evaluation
Appendix A. Original client request:
Client Matt Schmidt Professor matthew.schmidt@uga.edu
Background
I'm in the UGA CoE in Workforce Ed and Instructional Tech, specifically in Learning, Design, and Technology: https://coe.uga.edu/academics/concentrations/learning-design-and-technology/ Slack page for ePST UGA project: https://app.slack.com/client/T01KPK0RZPV/C078QSAD4BE
The primary objective for your team is to evaluate the existing ePST modules. Because the microlearning modules are in active development, you will need to request the latest versions from Yueqi Weng, the PhD student who is designing them. I have informed her about your work and copied her here.
As to the scope of the evaluation, specifically, I would like you to:
Review the Content: Assess the quality, relevance, and clarity of the information presented in the modules to the best of your abilities. Provide feedback on any areas that might require revision or enhancement, particularly focusing on language quality, accessibility, and user engagement.
Evaluate the Design: Analyze the learning design and user interface of the modules. Consider aspects such as ease of navigation, visual appeal, microlearning approach, and the effectiveness of the interactive elements. Provide Recommendations: Based on your evaluations, offer constructive suggestions for improvements. This could include recommendations for content adjustments, design changes, or the addition of new features to enhance the learning experience.
Identify Additional Areas for Evaluation: Be proactive and use your knowledge and skills to identify other potential areas where evaluation would be useful. Your fresh perspective is invaluable in uncovering aspects we might have overlooked. Your team will not be responsible for creating new modules. However, your insights and feedback will be invaluable for refining and improving the existing content.
I am available via email or via Slack (I added all of your team to the "epstevaluation" channel) to discuss this further and answer any questions you might have. Unfortunately, I am traveling extensively this month, so setting up a Zoom meeting is not going to work for me in the short term. Longer term, I am available after Jul. 8. I am happy to provide detailed responses to any questions you have. Looking forward to working with you and your team!
Dr. S.
Appendix B. The Core Team:
Julie
Wyatt, Project Manager
Julie holds a M.P.A from the University of North Georgia and is now pursuing her M. Ed. for Instructional Design and Development at the University of Georgia. Julie is currently an academic advisor at Kennesaw State University and plans on developing e-learning modules to support higher education departmental trainings. She enjoys breaking down complex ideas into easily understandable material for adult learners and takes pride in her organizational skills.
Brooke Oliver, Instructional Designer
Brooke holds a Bachelor of Science in Interaction Design and a Minor in Technical Communication from Kennesaw State University. She is now pursuing her M.Ed. In Instructional Design and Development at the University of Georgia. Brooke is a full-time Instructional Designer and Developer in the Medical Device Education industry. Brooke’s area of expertise is designing and developing interactive eLearning and continuing education modules for healthcare professionals.
Christelle de Beer, Instructional Designer
Christelle holds a Bachelor of Commerce in Industrial and Organizational Psychology, a Bachelor of Arts in Psychology, a Postgraduate Diploma in Education Technology, and is currently pursuing an M. Ed. In Learning, Design, and Technology with an emphasis on Instructional Design and Development from the University of Georgia. Christelle is currently a Learning Experience Designer at the North-West University in South Africa. She specializes in online training and development and content creation.
Appendix B. The Core Team:
Jennifer Hoosier, Instructional Designer
Jennifer holds an MA in TESOL from Azusa Pacific University and is currently pursuing an M.Ed. in Learning, Design, and Technology with an emphasis on Instructional Design and Development from the University of Georgia. Jennifer has extensive experience with international students and second language learners and has been in higher education administration since 2017. Her professional work focuses on creating training that is both highinterest and pedagogically sound.
Maria Paula Borras-Patino, Instructional Designer
Paula received her BA in Financial Engineering from UNAB in Colombia in 1999. After working in the banking industry, she transitioned to HR and Safety, which served as a bridge into the training world. She is currently pursuing her M.Ed. in Learning, Design, and Technology with an emphasis in Instructional Design and Development from the University of Georgia, with an expected graduation in December 2024. Paula now works for Coca-Cola as a full-time Training Consultant. She enjoys working with SMEs and excels in active listening, conflict resolution, and fostering a positive and engaging work environment.
Appendix C. Quantitative Data:
Appendix C. Evaluation:
Lesson 2.1 Assess The Problem
• Conflictive numerations. It says "Welcome to Lesson 3.1" but it should be 2.1
• The "Select each tab below for more information" is a non-existing feature.
2.1 Slide 3
Lesson 2.1 Assess The Problem
• The signal principle is used effectively.
• Recommendation: The color contrast could be enhanced for the signal words.
2.1 Slide 6
• This video explains how to go to the next slide. Recommendation: If the down arrow has more contrast there would not be the need to show this video.
• The screencast is somewhat busy/distracting.
2.1 Slide 6, at the end of the tutorial.
Lesson 2.1 Assess The Problem
• Aesthetics: Alignment and font size.
• Instruction on "click continue…" could be closer to Ruth to avoid clicking the arrow and miss the assess the problem information.
2.1 Slide 7
• Spatial contiguity could be enhanced by moving the options close to the question.
• Aesthetics Recommendation: Make answer boxes the same size.
2.1 Slide 7
Lesson 2.1 Assess The Problem
• Recommendation: Remove the continue button and add the next step as part of the conversation, for example Ruth may say "ok, here is a summary of all steps" and show the list. Having continue next to the down arrow could send the user to the wrong slide.
2.1 Slide 7
Recommendation: Remove "at once" from title.
2.1 Slide 7
Lesson 2.1 Assess The Problem
• The back arrow takes you out of the conversation in slide 7 into slide 6
• Recommendation: There is an opportunity to include a practice here.
• Recommendation: Make fonts size more homogeneous.
• Design feels inconsistent (including fonts size).
• Punctuation could be modified to make the computer voice sound better.
2.1 Slide 7
Lesson 2.2 Brainstorm Solutions
• This is a better way to give the instruction of selecting the bottom right arrow to continue, compare to the one used in 2.1
• Design Feels inconsistent.
• Font size is better in 2.2 2.2
• The A-B-C-D-E-F concept was never described before.
Lesson 2.2 Brainstorm Solutions
• Recommendation: Rephrase- “They might help you overcome those barriers.”
2.2
Recommendation: Consider adjusting ratio of fonts in box 1 and 2- The font is too small. 2.2
Lesson 2.2 Brainstorm Solutions
• Recommendation: Change punctuation to make it sound better.
• Recommendation: Consider narrowing the space between the person and the text.
Lesson 2.2 Brainstorm Solutions
• Make complete sentences.
• Recommendation: Include the word ideas in 1. List as many "IDEAS" as you can.
2.2
Lesson 2.3 Consider and Chose
• The font is bigger on 3rd option.
2.3
• Recommendation: Consider rephrasing the text. “Jane is an individual with TBI. She feels frustrated when things are out of her control or schedule. Let's hear her story."
10.
2.3 Video in slide
Lesson 2.3 Consider and Chose
• Recommendation: Consider rephrasing the text for the computer voice to sound more natural. “Yesterday, I felt accomplished because I completed many tasks from my checklist. I enjoy having this feeling throughout my day."
Video in Lesson 2.3 slide 10.
• Pop up window feels inconsistent with the previous visual design.
• Recommendation: Consider adding a transparency banner similar to the one use in the previous slide to maintain consistency.
Video in Lesson 2.3 slide 10.
Lesson 2.3 Consider and Chose
• Recommendation: Consider rephrasing
• Make complete sentence by adding a period at the end of each sentence.
2.3
• Typo: sped to spend.
• Add periods at the end of each sentence.
2.3
Lesson 2.3 Consider and Chose
• Recommendation: Consider rephrasing “ I will have less stress and improve my well being.” and “I will have to accepts the fact that it might not happen every day.”
• Recommendation: Rephrase #5 to: "Is it difficult to do
Lesson 2.3 Consider and Chose
• Recommendation: Congratulation! You earn a badge for lesson 2.3!
• Consider rephrasing: "you are able" to "In this lesson, you learned to:" 2.3
Lesson 2.4:
Evaluate the outcome or Develop and do?
• Recommendation: Use same size as in the previous introductory slides. 2.4
• First screen shot is the end of lesson 2.3 Where is stated that lesson 2.4 name is "Evaluate the outcome." but the name of lesson 2.4 is "Develop and do."
• Recommendation: Title Case for titles. 2.4
Lesson 2.4: Develop and do
• Recommendation: Rephrase to “Think about what the outcomes will be if you achieve your goal fully, partially, or not at all."
• Incomplete sentences.
2.4
• Step 2- wrong title
• Recommendation: Rephrase step to: How to develop and Execute.
2.4
Lesson 2.4: Develop and do
• Select (the) Continue button below… or Select “continue” to talk with Ruth… 2.4
• Incomplete sentences. Recommendation: Rephrase them. 2.4
Lesson 2.4: Evaluate the outcome or Develop and do?
• Recommendation: Provide the option to go back.
• Recommendation: Rephrase “Does it difficult to do it? “To Does it seem difficult to do? Or Is it difficult to do? 2.4
• Recommendation: Move the note pad and person to the left to unclutter the right side. 2.4
Lesson 2.5 Evaluate the Outcome
Lesson 2.5 Evaluate the Outcome
Recommendation: Here is an opportunity for recognition not recall. Perhaps showing her quote or just removing the sentence that mentions the quote. Based on Nielsen' 10 usability heuristics- Recognition rather than recall.
2.5
• The steps seem to be in incorrect order. Step one should be step 3.
• Recommendation: Title case.
2.5 Slide 6.
Lesson 2.5 Evaluate the Outcome
• Recommendation: Change font size to a bigger one.
2.5 Slide 7
Recommendation: Rephase sentence.
Recommendation: remove” that” from: So, I told my colleagues “that”
2.5
Lesson 2.5 Evaluate the Outcome
• Recommendation: Keep consistent the deign of the pop-up boxes.
2.5 Story in slide 7
Lesson 2.5 Evaluate the Outcome
• Recommendation: Change the size of the of font. In the current example it is too small
2.5 Story in slide 7
Lesson 2.6 Flexibility in Problem-Solving
• Recommendation: Change flex for flexible. 2.6
• Recommendation: add “ the” to all mid-banners in slide 6 or remove “button”. 2.6
Lesson 2.6 Flexibility in Problem-Solving
• First screenshot is from lesson 2.6 slide 6 Recommendation: Face bold and Center “ Let’s Learn about”
Lesson 2.6 Flexibility in Problem-Solving
• Recommendation: Change font size to a bigger one.
2.6 Slide 8
• Recommendation: Rephrase to "It's really good to let Jeremy keep trying to figure out how to handle his emotions on his own. But what if he's still having a tough time with it?"
2.6 Slide 8
Lesson 2.6 Flexibility in Problem-Solving
• Recommendation: Aesthetically, there is a lot of white space underneath the character. Recommendation: Scroll the character down so that her head is not cut off, creating a better sense of space.
Lesson 3.1 Part 1: Assess the Problem in Depth Using the "5 W’s"
• The first screenshot is from lesson 3.1 slide 7 and the second screenshoot is from lesson 2.5 slide 7. Design consistency is affected in these slides. The Player and back and next buttons are different.
• Recommendation: Maintain player design consistency.
3.1 slide 7
Lesson 3.1 Part 1: Assess the Problem in Depth Using
the "5 W’s"
• Recommendation: Add word “a” to “I'm going to introduce to you the importance of a detailed problem statement.”
3.1.1
• Player design is not consistent.
• Player is missing the back and forth arrows.
3.1.1 Slide 7
Lesson
3.1 Part 1: Assess the Problem in Depth Using the "5 W’s"
• Next button does not work.
3.1.1 Slide 12
• Recommendation: Align text inside brown boxes.
3.1.1 Slide 13
Lesson 3.1 Part 2: Creating a Problem Statement
First screenshot above is from lesson 3.1 part 2, compared to bottom screen shot from 3.1 part 1. The slide side bar is no longer part of the design. This bar is a great addition, and we recommend keeping it. It allows the user to navigate freely throughout the module and informs the user as to how much has it been completed/or is left to complete. Also the up and down arrows were removed. All subsequent modules and slides from here on have omitted the bar and up and down arrow 3.1, Part 2
Lesson 3.1 Part 2: Creating a Problem Statement
• This should read: Welcome to Lesson 3.1 Part 2, not Part 1
3.1, Part 2
• Recommendation: remove the 'Select Next' tab as it creates a cluttered design, especially since the 'NEXT>' option is already available just below.
2.1 Slide 7
Lesson 3.1 Part 2: Creating a Problem Statement
• Arrows are no longer there, therefore video showing how to advance with the arrow is irrelevant. Recommendation to add the arrow as sometimes scrolling down does not always work.
3.1 Part 2
• Font is too small and difficult to read. Recommendation: Remove/vanish Ruth to make room for the activity information.
3.1 Part 2
Lesson 3.1 Part 2: Creating a Problem Statement
• The initial design of this screen is not consistent with previous ones. Second screenshot.
• Recommendation: Title case for this title. First screenshot
3.1 Part 2
Lesson 3.1 Part 2: Creating a Problem Statement
Recommendation to rephrase:
Great job so far; you have thoroughly assessed the problem. Now, let's help others assess their problems.
3.1 Part 2
Lesson 3.1 Part 2: Creating a Problem Statement
• Aesthetics: Alignment and font size.
• Instruction on "click continue…" could be closer to Ruth to avoid clicking the arrow and miss the assess the problem information.
3.1 Part 2
Lesson 3.1 Part 2: Creating a Problem Statement
• 3.1 Part 2 the font size is not consistent. Compare 2.1 to font in 3.1 part 2.
3.1 Part 2
Lesson 3.2 Part 1: Effective Brainstoming Techniques
• Add "s" to Technique_.
3.2. Part 1
• Recommendation: There is a community always ready to support you/ or There is a community that is always ready to support you.
3.2. Part 1
Lesson 3.2 Part 1: Effective Brainstoming Techniques
• Add "s" to Technique_
3.2. Part 1
Recommendation: Slow down pace of video
• Recommendation: Slow down pace of video.
3.2. Part 1
Lesson 3.2 Part 2: Effective Brainstoming Techniques
• Recommendation: Add "s" to ones: in Result 1: "Carefully consider the pros and cons of each solution you come up with so that you can save time by not including unhelpful ones later.
3.2. Part 1
• Recommendation add: "s" to Technique_
3.2. Part 1
Lesson 3.2 Part 1:Considering Inaction as a Viable Option
• Recommendation: Should the title be "Doing Nothing Strategy"? See Red box.
3.2 Part 2
Recommendation: Center top text box above small boxes Match font size of small text boxes.
3.2 Part 2
Lesson 3.2 Part 1:Considering inaction as a viable option
• Recommendation: Enable previous button
3.2 Part 2
Lesson 3.3: Make Informed Choices
• Recommendation: Add: "an" In this lesson, you will learn how to make an informed choice by:
Lesson 3.3: Make Informed Choices
• Recommendation: Rephrase to: You have completed this section. 3.3
• Recommendation: Enlarge font to information shown in the red box.
Lesson 3.3: Make Informed Choices
• Recommendation: Rephase to “Making an Informed Choice “or Making Informed Choices.
Lesson
• Recommendation: Verify if the intent is having that questions (in the red box) come from the user and not from Taylor.
Lesson 3.4: Detailed Planning and Execution
• Recommendation:… will you provide to him?
• Recommendation: Take it easy, you may try more than once to solve the problem.
3.4
Recommended rephrasing: #1 to: "Get details and organize your thoughts as much as possible.“ #2 to: "Be concrete with the plan for you to follow through."
#3 to: "Decide what could happen based on the achievement of your goal." or" Decide what will happen based on the achievement of your goal."
Rephrase #4 to : "Try it once you have finished the plan of action."
3.4
• Recommendation: 3.4
• Recommendation: 3.3