HEURISTICS EVALUATION REPORT

Yahoo Newsroom

Report #4: Heuristic Evaluation Report March 26, 2018

Team Jabberwocky Wyatt Clement, Saebom April Kwon, Jessica McCauley, Blake Wagner, Chuyao Wang

Link to Google Doc Link

Word count 2,473

Table of Contents

Executive Summary

3

Introduction

4

Methods

4

Findings and Recommendations

5

Key Finding 1: The app’s main functionalities should be ​better explained on first use

Recommendation

Key Finding 2: There are no Undos or Confirmations for user actions

Recommendation

Key Finding 3: Flexibility for Vibes should be improved

Recommendation

Key Finding 4: In their Newsroom feeds, users are not able to filter source, keywords or topic Recommendation

Key Finding 5: ​News reading experience could be improved

Recommendation

Discussion

10

Conclusion

10

Appendix A: Heuristic Evaluation Template

13

Appendix B: Group Heuristic Evaluation Results

16

Appendix C: Individual Heuristic Evaluations

21

Table A: Wyatt Clement Heuristic Evaluation

Table B: Saebom April Kwon Heuristic Evaluation

Table C: Jessica McCauley Heuristic Evaluation

Table D: Chuyao Wang Heuristic Evaluation

Table E: Blake Wagner Heuristic Evaluation

Executive Summary Reimagined in 2016, Yahoo Newsroom (YN) was designed to offer a customizable and socially-involved news reading experience. A UX research team at Oath Inc. (a subsidiary of Verizon Communications that houses Yahoo!) has partnered with our team at the University of Michigan to conduct a multi-phase user study to obtain an increased understanding of user behavior, experiences, and preferences in relation to existing mobile news applications, broadly; as well as with YN, specifically. In this phase of the study, each of our team members conducted a heuristic evaluation of YN. Jakob Nielsen’s usability heuristic considerations served as a framework for our evaluations. About three dozen unique heuristic issues were identified across 5 individual analyses. Each issue was assigned a severity rating score based on a 5-point Likert scale: “0-1” no/minor usability problem to “4” major usability problem that requires immediate attention. Seventeen heuristic issues were assigned severity scores that suggested major usability concerns, four of which deserved prompt attention. In the report, the issues were synthesized into ​5 key findings, and corresponding recommendations were advanced for consideration in addressing these findings. The key findings and related recommendations are as follows: 1) there was uncertainty centered on the functionality of primary features -- this could be resolved by providing an explanation of these features in onboarding; 2) undoing actions and confirmation of completed actions is not possible/nonexistent -- including this functionality/feedback should be considered; 3) search is limited to “Vibes” or communities -- providing more search filtering options (e.g., by topic, article, source, etc.) should be considered; 4) user Newsroom feed cannot be filtered or sorted -- including this functionality promotes user autonomy and should be considered; and finally, 5) the reading experience lacks user control and aesthetic cohesion across article types -- strategies and techniques used by competitors should be considered.

Introduction Yahoo Newsroom (YN) is a mobile news app with a focus on video content, personalization, and social features. Its news feed offers a mix of original content and stories aggregated from sources around the web. The interface is a traditional news feed design, reminiscent of comparable mobile news apps, as well as social networking sites such as Facebook. In the current phase of the study, our UX research team at the University of Michigan conducted a heuristic evaluation of YN to identify problems/shortcomings/areas of improvement using Jakob Nielsen’s 10 usability heuristic considerations as a framework. Our research team conducted five individual heuristic evaluations of YN. The following topic areas and key questions were of particular interest in our analyses: 1) Onboarding/First-time user experiences ● How easy is it for someone to get starting using the app? ● What aspects of YN are easy for users to quickly understand? ● What aspects of YN are difficult for users to quickly understand? ● Where can we improve the first-time user experience? ● How easy is it to find and follow things they care about within the app? ● What do they expect from a news app and how does YN compare to other news products they are using?

2) The wicked problem of media distrust ● Does YN convey a sense of trust and credibility? Why or why not and how can we continue to improve this? ● How do the community aspects of topics influence trust, credibility, and validity of news content within the app?

The results from the evaluations were synthesized and s​everity rating scores were specified for each of the heuristic issues identified. Severity ratings were based on a 5-point Likert scale, from no/minor usability problem to major usability problem that requires immediate attention. The heuristic issues were then organized in a table based on level of severity (​Appendix B​).

Recommendations are provided in regards to how the main heuristic issues our team identified can be addressed.

Methods Heuristic evaluation is a method of evaluating a system or interface that relies on knowledgeable evaluators, as opposed to existing or target end-users. These evaluators go through the system and note usability issues, using heuristics, or rules based on established good practices, as a point of reference. We opted to use Jakob Nielsen’s well-established set of 10 usability heuristics for user interface design (1994) to guide our evaluation of the Newsroom app (​Table 1​).

#

Heuristic

1

Visibility of system status

2

Match between system and the real world

3

User control and freedom

4

Consistency and standards

5

Error prevention

6

Recognition rather than recall

7

Flexibility and efficiency of use

8

Aesthetic and minimalist design

9

Help users recognize, diagnose, and recover from errors

10

Help and documentation

Table 1. 10 Heuristics for User Interface Design (Nielsen, 1994)

First, we met as a group to discuss how to interpret these heuristics in relation to this product and to decide what research questions and user tasks to focus on. Given that Newsroom is relatively narrow in its scope and functionality, we decided not to limit our individual assessments to only those areas relevant to onboarding and trust in the media. Instead, we noted all issues that we encountered, then later prioritized those issues that were of particular relevance to these key areas of focus and interpreted our findings through the lens of our research questions.

1. Following a Vibe(s) a. Follow a Vibe(s) in “Explore” b. Check the vibes that I am following in my profile c. Share the vibe link

2. Browsing a Newsroom Timeline a. Browsing the Newsroom timeline b. Picking an article with personal preference c. “Like” the article i.

Check the liked article in the profile

3. Reading a Story a. Reading the chosen article to the end b. Add a reaction to the article c. Repost the article in the vibe

Based on this shared understanding, each group member conducted a separate heuristic evaluation, assessing the system autonomously and providing written comments and severity ratings for all issues encountered. We used Nielsen’s severity scale, which ranges from 0 - ‘I don't

agree that this is a usability problem at all’ to 4 - ‘Usability catastrophe: imperative to fix this before product can be released’ (​Table 3​).

Rating

Definition

0

I don't agree that this is a usability problem at all

1

Cosmetic problem only: need not be fixed unless extra time is available on project

2

Minor usability problem: fixing this should be given low priority

3

Major usability problem: important to fix, so should be given high priority

4

Usability catastrophe: imperative to fix this before product can be released

Table 3. Severity Rating Scale (Nielsen, 1994)

Once we had each completed our individual assessment of the system, we reconvened as a group and discussed the most severe (severity rating of 3 or 4) and most frequent (identified by more than one evaluator) issues that had been identified across our five heuristic evaluations.

We also discussed issues with a severity rating of 1 or 2 that were only identified by a single evaluator. This enabled us to assess severity more reliably as a group, ensuring that we did not overlook any potentially significant usability issues. It also provided us with a better collective understanding of the system, which is important as we begin to make decisions about tasks for usability testing.

Next we calculated a group severity rating (mean of the ratings of each evaluators who identified the issue) for each issue and ranked them based on severity. A summary description of each significant usability issue was synthesized from individual evaluator comments. Using this final ranked list, we were able to prioritize issues and generate key findings and recommendations based on our research questions.

Findings and Recommendations Based on our research questions and the tasks, our heuristic evaluation revealed a number of key findings and their respective explanations, along with corresponding recommendations, are specified below:

1. Familiarization of key functionalities 2. Enabling undoing actions 3. More flexible user controls for Vibes 4. More personalizing options for user Newsroom feed 5. Enhancing basic reading experience

Key Finding 1: The app’s main functionalities should be better explained on first use Heuristic #2 Match between System and the Real World

Severity Score: 4

During the heuristic evaluation, we identified issues with understanding the app-specific terms such as “Vibes”, “Reactions”, and “Repost”. Even with our previous experience using the app, we encountered conflicts between our own understandings of these concepts and the way the app utilizes them. This is due to a similarity between icons used by Newsroom and those used by other mobile news and social media apps. Confusions included “Reaction” with “Likes” because of the heart icon (“❤”), “Repost” with “Share/Comments” because of the two arrows pointing each other (“ˬ”). It is possible that these issues would be amplified for first-time users.

Additionally, the location of “(Community writing) Guidelines” in the “Repost” page does not follow established conventions for most users. Most social media platforms embed their guidelines in a side of the community page and they only highlight several restricted points that users should be aware of (​Figure 2​).

Figure 1. Yahoo Newsroom App Guidelines

Heuristic #10 Help and Documentation

Figure 2. Reddit’s Subreddit Guidelines

Severity Score: 3

Yahoo Newsroom provides a separate Help page that is connected to an external Yahoo webpage. Nevertheless, the contents are limited to specific circumstances (mostly on getting to know the general application functionality) and there is no availability for on-demand assistance or direct contact to the staff.

Recommendation These findings suggest several points to consider while developing the app’s onboarding process. To get users more accustomed to these app-specific terms, we recommend the Yahoo

Newsroom team develop more interactive, on-demand tutorials made visible to users upon first use, with better visual aids for longer-term memory. Moreover, the application development team might consider changing the terms, or enhancing the visibility of the terms and their respective icons for the users.

Key Finding 2: There are no Undos or Confirmations for user actions Heuristic #3 User Control and Freedom

Severity Score: 4

We found that, in general, undo options for social interaction functions are not provided. While users can get instant feedback when they repost a story or mute a user as a small bar in either the bottom or the top of the application, the feedback does not include the option to "Undo".

Heuristic #5 Error Prevention

Severity Score: 3

Furthermore, the system does not appear to give many warnings in the first place. No confirmation is asked for when you report abuse, which could make mistaken reports more difficult to prevent. Users may prefer the system to treat these interactions with a bit more caution.

Recommendation The community nature of Newsroom may require a more attentive interface. Especially for social interaction features, more considerate design should be applied for user actions in the Vibes and articles, even small ones like muting users. The visibility of feedback for these actions should be improved and the feedback should provide an undo function.

Key Finding 3: Flexibility for Vibes should be improved Heuristic #6 Recognition rather than Recall

Severity Score: 2

During search, users are only allowed to search by Vibe, not by article. Moreover, the feature does not provide autofill suggestions even though there are recommended cards based on what the user typed in, which means the users need to remember their previous search keywords.

Heuristic #7 Flexibility and Efficiency of Use

Severity Score: 2

When reposting an article, users are provided with a list of Vibes to which users can repost the article. However, are also shown Vibes that they are unable to repost to (because posting is only available to admins), and those they do not follow. Separately, we had conflicting views regarding why the application does not allow users to repost an article without following the Vibe.

Heuristic #7 Flexibility and Efficiency of Use

Severity Score: 4

We also discovered that there are few features that allow users to engage with the Vibes they follow. For instance, the app does not have a notification adjustment feature. However, as we found during our comparative analysis, most of the app’s competitors enable their users to connect to the built-in OS settings so they can adjust the notification settings.

Recommendation Even though Vibes are an important element that should be a focus of the app, users might want to browse articles rather than communities. This function can also lead to opportunities to connect users to the recommended Vibes from the article search. Moreover, there should be more investigation into what kind of articles users want to repost in what Vibes: do they only want to share within Vibes they follow, or outside of their Vibes as well? This question could be examined through usability testing.

Key Finding 4: In their Newsroom feeds, users are not able to filter source, keywords or topic Heuristic #3 User Control and Freedom

Severity Score: 3

Even though users are able to follow certain Vibes, it occurred to us during the evaluation that the Newsroom timeline is apparently not organized based on these Vibes, but mostly by seemingly general breaking/trending news articles. Because there is no feature to sort or filter the user’s timeline by source, topic, or keywords, it may be more difficult for users to identify articles of interest.

Heuristic #4 Consistency and Standards

Severity Score: 3

Currently, there are a number of sponsored articles or advertisements in the user Newsroom feed. In terms of layout, these promoted articles have very similar card design to normal news articles (with the only difference being the color of the “Promoted” text), which places a burden on users to distinguish ads from articles. (​Figure 3, 4​)

Figure 3. Promoted Article in Newsroom Feed

Figure 4. News Article in Newsroom Feed

Recommendation As the newsfeed is in many ways the face of the app, the user experience could be enhanced to better fit user needs. To make news content more user-oriented, the application should have more filtering and muting options based on particular sources, topics, or user-specified keywords.

Additionally, the Newsroom team may want to reconsider the way in which ads are displayed in the feed.

Key Finding 5: The app should pay attention to news reading experience Heuristic #3 User Control and Freedom

Severity Score: 2

We found that the experience of reading articles on Newsroom does not stand out from other apps. In fact, it lacks some important functionality. For example, there was no navigation to go to a certain section or to the top, which is available in other document reading apps like Dropbox Paper or Google Drive.

Heuristic #1 Visibility of System Status

Severity Score: 2

Similarly, the application does not provide any supporting visual cues that users might need while reading. When an article is connected to an external webpage, users are not warned that a redirect is about to occur. Additionally, the ability to swipe while reading to view related articles is not made visible, and an accidental, unintentional swipe may lead to some frustration for users.

Heuristic #4 Consistency and Standards

Severity Score: 2

While the app retrieves articles from different web sources, the app does not fully adjust the layout of the content in a consistent manner. Sometimes, we faced the situation where the title styles are applied to the body of the article probably due to algorithmic bugs. (​Figure 5​) This issue could result in users developing a lower opinion of the app.

Figure 5. When a parse the external webpage, sometimes the layout is wrongly applied

Recommendation Newsroom’s reading experience should be further refined. Swiping gestures should either be made more apparent to users, or eliminated, and there should be more informative indicators when an article links to an external site. Further, the styling algorithm applied when content is pulled from the web may need to be revisited to ensure more consistency across the app.

Beyond this, additional visual aids may help to improve the reading experience. For this particular functionality, the platform can refer to other document sharing apps to see their navigation design elements within the articles. For example, Google Drive supports user in-document navigation with touch-and-drag interaction so they can see the headings in the article.

Discussion Though each of our team members performed the evaluation based on the same framework (i.e., Nielsen’s 10 usability heuristic considerations), there was no pre-defined method or approach to how the individual analyses were to be conducted (e.g., what features to focus on, in what order, etc). This is reflected in our individual findings, as there was notable variation in identified issues.

Had there been a more standardized approach to our analyses, perhaps our findings would have been more robust. In an ideal setting, a comprehensive interaction map would be used, and individual interaction pathways would be identified and labeled, and each team member would comb through each pathway in a methodical, sequential manner.

Moreover, there was a layer of subjectivity in the severity scores given to each of the issues identified. Severity scores were prioritized based on what our team considered significant/non-significant and may not reflect or align with the opinions/interests of YN developers.

Nielsen’s heuristic considerations are widely used and respected among industry leaders and academics, alike. However, with any framework there exists limitations. Because our analyses were based solely on this framework, our mental models may have been rigid/limited. For example, there may have been significant issues that fell outside of Nielsen’s considerations that may not have been identified/addressed. The next phase of our study is usability testing, which will provide us with the un-adulterated/influenced perspectives on the usability of YN. This will play a vital role in our final determinations of the most significant findings and recommendations in which we will present to our client.

Conclusion Our UX team conducted a total of five individual heuristic evaluations on YN. Seventeen usability issues were identified across our analyses, 4 of which demand prompt attention. Five key findings resulted from a collective analysis, and recommendations were offered for consideration as possible next steps to address each finding. ​The key findings and related recommendations are as follows: 1) there was uncertainty centered on the functionality of primary features -- this

could be resolved by providing an explanation of these features in onboarding; 2) undoing actions and confirmation of completed actions is not possible/nonexistent -- including this functionality/feedback should be considered; 3) search is limited to “Vibes” or communities -- providing more search filtering options (e.g., by topic, article, source, etc.) should be considered; 4) user Newsroom feed cannot be filtered or sorted -- including this functionality promotes user autonomy and should be considered; and finally, 5) the reading experience lacks user control and aesthetic cohesion across article types -- strategies and techniques used by competitors should be considered. In the final phase of our study, we will be conducting a series of in-person consumer usability testings which could substantiate our findings and/or uncover further heuristic issues -- ultimately leading to the refinement of our recommendations.

References Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), ​Usability Inspection Methods​. John Wiley & Sons, New York, NY.

Appendix A: Heuristic Evaluation Template

Review Checklist

Rating Comments

1. Visibility of system status Can the user tell where they are in the system?

Does every page have a title explaining the contents of the screen?

Does the system give appropriate feedback for user actions?

Does the user know how to get back in the system?

Can the user tell what options are available for what to do next?

Does the system use language that is likely to be familiar to the user?

Does the system follow real-world conventions?

Is the information displayed in a natural and logical order?

Is the interaction of curating intuitively understandable for users?

Is the interaction of reading an article intuitively understandable for users?

Can the user undo or revert mistaken input or paths?

Can the user safely abandon tasks that go away?

Can the user do things in the order that they want to do them?

Does the user know where to find the articles/topics they want?

Does the system comply with relevant standards (universal, platform, app category)?

Can the user do perform similar tasks by follow the same steps?

2. Match between system and the real world

3. User control and freedom

4. Consistency and standards

Is the system consistent internally (ex. Color, typography, words, icons, information display)

Do forms with similar components have similar layouts?

Does the system warn users when they are about to take a “dangerous (i.e. Delete, remove,)” action?

Does the system expect users to remember earlier decisions/actions when completing later steps?

Are error messages displayed in a proactive fashion?

Does the system force users to remember obscure commands or sequences?

Does the system expect users to remember earlier decisions/actions when completing later steps?

Is autofill utilized?

Are drop down menus utilized?

Is an internal search functionality present?

Is bookmarking for an article enabled?

Does the system force expert user to do things in slow, inefficient ways?

Does the system allow people with different roles/styles/preferences to get things done?

Does the system allow users to tailor frequent actions?

Is autocomplete utilized in forms/search on the site?

Are there ways to input of different styles/types?

Is there excess visual clutter and/or text that is irrelevant to users’ main tasks?

Is the visual design distracting or hand to pars, resulting in extra time to process information? (ex.

Color, typography, information display)

5. Error prevention

6. Recognition rather than recall

7. Flexibility and efficiency of use

8. Aesthetic and minimalist design

Does the visual design of the app pleasurable?

9. Help users recognize, diagnose, and recover from errors

Are the error messages expressed in a way that users can understand?

Does the error message provide them with a clear course of action?

Does the platform provide the functionality ‘going back’ when any error is happened?

Does Auto-corrected result for misspelled search?

Is there a help document?

Can users find the help document?

Is it helpful?

Is it well written, well organized, searchable?

Does the platform have a repository for frequent Q&A?

Does the platform provide a tutorial in the beginning of use?

When errors do occur, are users made aware of them?

10. Help and documentation

Appendix B: Group Heuristic Evaluation Results

H# Question

Rating (0-4)

# of Report

Summary of issue

10 Does the platform provide a tutorial in the beginning of use?

4

3 No tutorial is provided for getting started with the app, only provide a minimal description of suggested functions in the beginning

7 Does the system allow users to tailor frequent actions?

4

2 Very little personalization outside of Vibes. There is no setting feature for notification.

7 Does the system allow people with different roles/styles/preferenc es to get things done?

4

3 Can the user undo or revert mistaken input or paths?

4

4 Generally, undo function is not applied. No way to undo unfollowing vibes and users are not provided "Undo" right after they repost a story or mute a user. It's not too difficult to navigate backwards through the app. There are some nuisances about opening articles in succession mentioned above. Biggest issues are inability to delete posts/reactions, and the weird thing with posting mentioned earlier. Unfollowing and unliking are easy.

3 Can the user do things in the order that they want to do them?

3

3 Users cannot customize the contents they want to see in the timeline (in source- and topic- wise)

4 Is the system consistent internally (ex. Color, typography, words, icons, information display)

3

2 It is hard to distinguish the promoted articles with others, similarly, reposted articles and original articles look very same.

2 Does the system follow real-world conventions?

3

2 Video stories use “orange” progress bar; real-world: white; -Video stories “hold-tap” advances to next segment; real-world: pauses segment; Video stories zoom-in style not suggestive of video; Video thumbnail opens up into text story (with video)

5 Does the system warn users when they are about to take a “dangerous (i.e.

Delete, remove,)” action?

3

2 The system does not appear to give many warnings at all. No confirmation is asked for when you unfollow a Vibe, and you are forced to then find this Vibe again manually if you made a mistake. Users cannot confirm the actions for reposting in vibes, as well as reporting an article.

6 Does the system force users to remember obscure commands or sequences?

3

2 There is no summary or action history search for user actions in profile, so a user should browse the timeline for post-action (removal, edit).

For "Search" in the user profile, the users might expect the search based on their history but the function is the same with general search

8 Is there excess visual clutter and/or text that is irrelevant to users’ main tasks?

3

2 Lots of randomly larger modules, moving images, slideshow sections. These can get in the way of scrolling the feed with often duplicated articles and seemingly unrelated to my interests. Overall, there are too many ads and promoted articles in the timeline.

8 Is the visual design distracting or hand to pars, resulting in extra time to process information? (ex. Color, typography, information display)

3

2 The app is very busy visually, with large modules, some of which feature gifs or autoplaying videos.

9 Are the error messages expressed in a way that users can understand?

3

2 For video malfunction, error messages have been pretty technical, and not very user friendly: Lightray Settings? Feedback? Why Copy to Clipboard?

10 Is it helpful?

3

2 The web help documentation is hard to grasp and not on-demand, and some interactive tutorials (account security tutorial is not working)

2 Is the interaction of reading an article intuitively understandable for users?

3

The biggest issue is switching between articles and then trying to go back to the landing page. You're essentially forced to keep pressing the close button until all the articles you've opened are gone.

2 Does the system follow real-world conventions?

3

Location of community guidelines does not follow established conventions.

4 Can the user do perform similar tasks by follow the same steps?

3

The way of showing an article is not consistent; usually it shows a block with an article, but sometimes a list item connected to link box

7 Does the system allow people with different roles/styles/preferenc es to get things done?

3

Not many customization options available beyond Vibes. Users who like to read based on the source of an article are mostly forced to scroll around until they find the sources they like.

1 Can the user tell where they are in the system?

2

3 Not always. Links out of app to articles; especially confusing with Yahoo! News links. It is not clearly stated that "Read Full Story" in an article means you going into other web application to read the article

1 Does the system give appropriate feedback for user actions?

2

3 When not following any vibes, the "my vibes" button in the hamburger menu doesn't give any respond when selected, neither a notification to follow vibes.

2 Does the system use language that is likely

2

3 Vibes – real world: “energies/feelings”; YN: “topic areas”; Reactions – real world: “emoji visuals”; YN: “comments”

to be familiar to the user?

6 Is autofill utilized?

2

3 User only can search by vibes, not by an article. Also, the search feature does provide suggestions based on what you typed in, but it's not exactly autofill, and it doesn't seem to remember what you've searched for previously

10 Does the platform have a repository for frequent Q&A?

2

3 Users can only give feedback through the help document but the contents is a bit limited.

3 Can the user do things in the order that they want to do them?

2

2 Users cannot go to top inside the articles. Similarly on 'Search' page, if you scroll down then you have to scroll all the way back up.

4 Do forms with similar components have similar layouts?

2

2 Thumbnails of some vibes have animations, while others are still, without no clear why they do; Same for videos, some are playing in the feed while others have to be clicked on.

5 Does the system expect users to remember earlier decisions/actions when completing later steps?

2

2 The system doesn't seem to track what stories you read that well, so the suggestions for posting are not based on your recent reading behavior. Unfollowing a vibe forces you to find that vibe again manually if you made a mistake (no recently unfollowed category)

1 Can the user tell what options are available for what to do next?

2

2 The options for navigating between stories are not particularly clear. In the article, a user cannot know that swiping interaction is available.

7 Is autocomplete utilized in forms/search on the site?

2

2 In "Select Vibes", users should see the vibes that are unavailable to post; Cannot post to vibe without following vibe

3 Can the user do things in the order that they want to do them?

2

There are no social media connection on "Share", which seems to make it useless.

4 Is the system consistent internally (ex. Color, typography, words, icons, information display)

2

While "..." icon is the same for others' articles and my reposted article, their functionalities are different; When going into user's profile picture, the title is "Message History" (should be "Comment History"?)

3 Can the user undo or revert mistaken input or paths?

2

3 Can the user safely abandon tasks that go away?

2

Could not change gender or the date of birth.

Unfollowing vibes does not go into effect until app is restarted.

6 Is bookmarking for an article enabled?

2

4 Is the system consistent internally (ex. Color, typography, words, icons, information display)

1

2 Is the interaction of curating intuitively understandable for users?

1

The auto-slides are quite intuitive to use, the process bar is too narrow to notice, and I don't know how I should operate the slides at the first use

3 Does the user know where to find the articles/topics they want?

1

Newsroom feed appears to be populated at random, but icons for followed vibes are available from the landing page. You can also navigate directly to the Vibes page from almost anywhere on the app.

4 Can the user do perform similar tasks by follow the same steps?

1

Processes appear consistent. Navigating to the profile page using the bottom navigation bar versus the hamburger menu changes the way in which the user can navigate away from the profile page.

4 Is the system consistent internally (ex. Color, typography, words, icons, information display)

1

The font type of links are different than other contents.

Sometimes headlines and body texts are not in vertical order. Sometimes longer source names, titles or links (Google+) are not properly aligned or limited

4 Do forms with similar components have similar layouts?

1

The only two forms that are really included are the reposting and posting functions, which appear nearly identical

8

1

The icon to toggle to the keyboard on the 'Post story' page [A] - is this recognizable? Some other icons ("gif", "report") are not visually attractive and so small to touch.

9 When errors do occur, are users made aware of them?

1

We've encountered very few errors, but when we have, we've been told about them

9 Does Auto-corrected result for misspelled search?

1

There does not appear to be any autocorrect, but that might be intentional in some cases

Does the visual design of the app pleasurable?

1 Does every page have a title explaining the contents of the screen?

0

If you like or post a story, it will appear on your profile for later viewing. Not sure how long this will stay there, though, and isn't clearly indicated as a bookmarking feature.

2 Use of color and typeface is consistent throughout.

Layout changes slightly depending on the screen, but retains the modular feel. Occasionally the close button switches from the right to the left corner depending on the screen. However, we could also find inconsistent use of blue/purple and purple colored to distinguish users from news sources.

10 Is there a help document?

0

Appendix C: Individual Heuristic Evaluations Table A: Wyatt Clement Heuristic Evaluation

Review Checklist

Rating Comments

1. Visibility of system status Can the user tell where they are in the system?

0

Does every page have a title explaining the contents of the screen?

1

Overall, appears to provide good feedback, although some instances may be a bit too subtle

Does the system give appropriate feedback for user actions?

0

Does the user know how to get back in the system?

0

Can the user tell what options are available for what to do next?

2. Match between system and the real world Does the system use language that is likely to be familiar to the user?

1

Terminology seems standard. Vibes would be the only confusing thing I can think of, but that's going away

Does the system follow real-world conventions?

Is the information displayed in a natural and logical order?

2

Deeper in the app things seem pretty organized. The landing page is somewhat messy with no real organization to the stories presented

Is the interaction of curating intuitively understandable for users?

2

If this is about curation by the users, it's a bit confusing. Unless you already have knowledge about how the app works, the two-step process for posting/re-posting obscures what exactly you're doing. Plus, when you go to post and start typing in the upper field before selecting a story, there doesn't seem to be an easy way to select a story without closing the post screen and starting over.

Is the interaction of reading an article intuitively understandable for users?

3

Reading articles seems pretty straightforward. The biggest issue is switching between articles and then trying to go back to the landing page.

You're essentially forced to keep

pressing the close button until all the articles you've opened are gone.

3. User control and freedom Can the user undo or revert mistaken input or paths?

2

It's not too difficult to navigate backwards through the app. There are some nuisances about opening articles in succession mentioned above.

Biggest issues are inability to delete posts/reactions, and the weird thing with posting mentioned earlier.

Unfollowing and unliking are easy.

Can the user safely abandon tasks that go away?

Can the user do things in the order that they want to do them?

Does the user know where to find the articles/topics they want?

1

Newsfeed appears to be populated at random, but icons for followed vibes are available from the landing page.

You can also navigate directly to the Vibes page from almost anywhere on the app.

4. Consistency and standards Does the system comply with relevant standards (universal, platform, app category)?

0

Can the user do perform similar tasks by follow the same steps?

1

Processes appear consistent.

Navigating to the profile page using the bottom navigation bar versus the hamburger menu changes the way in which the user can navigate away from the profile page.

Is the system consistent internally (ex. Color, typography, words, icons, information display)

2

Use of color and typeface is consistent throughout. Layout changes slightly depending on the screen, but retains the modular feel. Occasionally the close button switches from the right to the left corner depending on the screen. Some videos play in the feed while others have to be clicked on.

Do forms with similar components have similar layouts?

5. Error prevention Does the system warn users when they are about to

3

The system does not appear to give

many warnings at all. No confirmation is asked for when you unfollow a Vibe, and you are forced to then find this Vibe again manually if you made a mistake.

take a “dangerous (i.e. Delete, remove,)” action?

Does the system expect users to remember earlier decisions/actions when completing later steps?

2

The system doesn't seem to track what stories you read that well, so the suggestions for posting are not based on your recent reading behavior.

Unfollowing a vibe forces you to find that vibe again manually if you made a mistake (no recently unfollowed category)

Are error messages displayed in a proactive fashion?

2

The error messages that I've encountered have been few and far between, but when they appear, they are generally pretty unhelpful

6. Recognition rather than recall Does the system force users to remember obscure commands or sequences?

2

Not a lot of sequences that need to be remembered frankly.

Does the system expect users to remember earlier decisions/actions when completing later steps?

Is autofill utilized?

1

The search feature does provide suggestions based on what you typed in, but it's not exactly autofill, and it doesn't seem to remember what you've searched for previously

Are drop down menus utilized?

2

All search functionality is free text.

However, the system provides suggested stories for posting

Is an internal search functionality present?

0

A search function exists, yes

Is bookmarking for an article enabled?

2

If you like or post a story, it will appear on your profile for later viewing. Not sure how long this will stay there, though, and isn't clearly indicated as a bookmarking feature.

7. Flexibility and efficiency of use Does the system force expert user to do things in slow, inefficient ways?

2

Not sure what would qualify as an "expert user" in this case. The system isn't all that complex with regard to what you can do with it, so I wouldn't say it's particularly slow for anyone

Does the system allow people with different roles/styles/preferences to get things done?

3

Not many customization options available beyond Vibes. Users who like to read based on the source of an article are mostly forced to scroll around until they find the sources they like.

Does the system allow users to tailor frequent actions?

Is autocomplete utilized in forms/search on the site?

Are there ways to input of different styles/types?

8. Aesthetic and minimalist design Is there excess visual clutter and/or text that is irrelevant to users’ main tasks?

2

Lots of randomly larger modules, moving images, slideshow sections.

These can get in the way of scrolling the feed

Is the visual design distracting or hand to pars, resulting in extra time to process information? (ex.

Color, typography, information display)

2

The app is very busy visually, with large modules, some of which feature gifs or autoplaying videos.

Does the visual design of the app pleasurable?

2

The modular design is fine but the app is so busy that it can get a bit annoying

9. Help users recognize, diagnose, and recover from errors When errors do occur, are users made aware of them?

Are the error messages expressed in a way that users can understand?

1

Error messages have been pretty technical, and not very user friendly

Does the error message provide them with a clear course of action?

Does the platform provide the functionality ‘going back’ when any error is happened?

0

Errors can be navigated away from pretty easily. That goes for most of the navigation in the app in general

Does Auto-corrected result for misspelled search?

2

There does not appear to be any autocorrect, but that might be intentional in some cases

10. Help and documentation Is there a help document?

Can users find the help document?

Is it helpful?

0

2

There is a help document that can be accessed through the hamburger menu

It appears to provide decent answers to several frequent issues

Is it well written, well organized, searchable?

0

Well organized by categories. Search function is available

Does the platform have a repository for frequent Q&A?

1

Yes, this is provided, though it is a bit limited

Does the platform provide a tutorial in the beginning of use?

4

No tutorial is provided for getting started with the app

Table B: Saebom April Kwon Heuristic Evaluation

Review Checklist

Rating Comments

1. Visibility of system status Can the user tell where they are in the system?

1

Does every page have a title explaining the contents of the screen?

Does the system give appropriate feedback for user actions?

Does the user know how to get back in the system?

Can the user tell what options are available for what to do next?

There is no indication of how long the article is

2

It is not clearly stated that "Read Full Story" in an article means you going into other web application to read the article

2

In the article, a user cannot know that swiping interaction is available

3

It is hard to connect "Reaction" to heart icon

2

Users cannot check who are the members of a vibe

2. Match between system and the real world Does the system use language that is likely to be familiar to the user?

Does the system follow real-world conventions?

Is the information displayed in a natural and logical order?

Is the interaction of curating intuitively understandable for users?

2

There is no clear connection in video stream between videos to other videos or to news article

Is the interaction of reading an article intuitively

2

Some articles have "Also Read"

understandable for users?

paragraphs and links in an article, which annoys users during their reading

3. User control and freedom Can the user undo or revert mistaken input or paths?

Can the user safely abandon tasks that go away?

Can the user do things in the order that they want to do them?

Does the user know where to find the articles/topics they want?

3

Users are not provided "Undo" right after they repost a story

3

Users cannot customize the contents they want to see in the timeline (in source- and topic- wise)

3

There are no social media connection on "Share", which seems to make it useless

4. Consistency and standards Does the system comply with relevant standards (universal, platform, app category)?

Can the user do perform similar tasks by follow the same steps?

Is the system consistent internally (ex. Color, typography, words, icons, information display)

3

The app sometimes link the user to Yahoo webpage without notice and the different layout distract users

3

The way of showing an article is not consistent; usually it shows a block with an article, but sometimes a list item connected to link box

2

Sometimes longer source names, titles or links (Google+) are not properly aligned or limited

1

The font type of links are different than other contents

2

While "..." icon is the same for others' articles and my reposted article, their functionalities are different

2

When going into user's profile picture, the title is "Message History" (should be "Comment History"?)

3

It is hard to distinguish the promoted articles with others, similarly, reposted articles and original articles look very same

Do forms with similar components have similar layouts?

2

Icons for "slides" are not clear

2

Thumbnails of some vibes have animations, while others are still, without no clear why they do

3

Users cannot confirm the actions for reporting an article (the action is directly initiated)

3

There is no confirmation for repost in vibes

5. Error prevention Does the system warn users when they are about to take a “dangerous (i.e. Delete, remove,)” action?

Does the system expect users to remember earlier decisions/actions when completing later steps?

Are error messages displayed in a proactive fashion?

Does the system force users to remember obscure commands or sequences?

Does the system expect users to remember earlier decisions/actions when completing later steps?

2

6. Recognition rather than recall

Is autofill utilized?

Are drop down menus utilized?

2

There is no summary for user actions

User only can search by vibes, not by an article

Is an internal search functionality present?

Is bookmarking for an article enabled?

7. Flexibility and efficiency of use Does the system force expert user to do things in slow, inefficient ways?

Does the system allow people with different roles/styles/preferences to get things done?

Does the system allow users to tailor frequent actions?

Is autocomplete utilized in forms/search on the site?

Are there ways to input of different styles/types?

4

Users sometimes should go to other web applications to read the full article

4

For "Search" in the user profile, the users might expect the search based on their history but the function is the same with general search

4

There is no setting feature for notification

2

In "Select Vibes", users should see the vibes that are unavailable to post

8. Aesthetic and minimalist design Is there excess visual clutter and/or text that is irrelevant to users’ main tasks?

Is the visual design distracting or hand to pars, resulting in extra time to process information? (ex.

Color, typography, information display)

Does the visual design of the app pleasurable?

3

There are often duplicated articles and seemingly unrelated to my interests

2

The gradient bar for trending articles are not intuitively understood, users might expect another information from the visualization

3

Sometimes headlines and body texts are not in vertical order

1

Some icons ("gif", "report") are not visually attractive and so small to touch

3

Overall, there are too many ads and promoted articles in the timeline

9. Help users recognize, diagnose, and recover from errors When errors do occur, are users made aware of them?

Are the error messages expressed in a way that users can understand?

3

For video malfunction: Lightray Settings? Feedback? Why Copy to Clipboard?

Does the error message provide them with a clear course of action?

Does the platform provide the functionality ‘going back’ when any error is happened?

Does Auto-corrected result for misspelled search?

10. Help and documentation Is there a help document?

2 Can users find the help document?

Is it helpful?

Does the platform provide a tutorial in the beginning of use?

3

The web help documentation is hard to grasp and not on-demand

3

When a user searches a keyword in help, it links to login page (even though the user is logged in the app)

Is it well written, well organized, searchable?

Does the platform have a repository for frequent Q&A?

Even though there is basic help statement within the app, the contents are very limited

3

The application only provide a short description of suggested functions in

the beginning

Table C: Jessica McCauley Heuristic Evaluation

Review Checklist

Rating Comments

1. Visibility of system status Can the user tell where they are in the system?

Does every page have a title explaining the contents of the screen?

Does the system give appropriate feedback for user actions?

2

Does the user know how to get back in the system?

Can the user tell what options are available for what to do next?

2

The options for navigating between stories are not particularly clear

Does the system use language that is likely to be familiar to the user?

2

The icon to toggle to the keyboard on the 'Post story' page [A] - is this recognizable?

Does the system follow real-world conventions?

Is the information displayed in a natural and logical order?

'My Vibes' in side menu doesn't work and provides no feedback

2. Match between system and the real world

Is the interaction of curating intuitively understandable for users?

Is the interaction of reading an article intuitively understandable for users?

3

How to curate is intuitive, but it is not clear if/how curating affects personalization

3. User control and freedom Can the user undo or revert mistaken input or paths?

3

No way to undo unfollowing Vibe

Can the user safely abandon tasks that go away?

2

On 'Search' page, if you scroll down then you have to scroll all the way back up

Can the user do things in the order that they want to do them?

Does the user know where to find the articles/topics they want?

4. Consistency and standards Does the system comply with relevant standards (universal, platform, app category)?

3

Location of community guidelines does not follow established conventions

Can the user do perform similar tasks by follow the same steps?

Is the system consistent internally (ex. Color, typography, words, icons, information display)

1

Newsroom screen has different style top bar than other screens

Do forms with similar components have similar layouts?

5. Error prevention Does the system warn users when they are about to take a “dangerous (i.e. Delete, remove,)” action?

Does the system expect users to remember earlier decisions/actions when completing later steps?

Are error messages displayed in a proactive fashion?

3

If a user unfollows a Vibe, the only way they are able to follow it again is remembering which Vibe it was and finding it again

Does the system force users to remember obscure commands or sequences?

Does the system expect users to remember earlier decisions/actions when completing later steps?

6. Recognition rather than recall

Is autofill utilized?

3

Search function on 'Post story' page is a broad internet search that does not really support users in finding a story they are looking for

Are drop down menus utilized?

Is an internal search functionality present?

Is bookmarking for an article enabled?

7. Flexibility and efficiency of use Does the system force expert user to do things in slow, inefficient ways?

Does the system allow people with different roles/styles/preferences to get things done?

Does the system allow users to tailor frequent actions?

2

Very little personalization outside of Vibes

Is autocomplete utilized in forms/search on the site?

Are there ways to input of different styles/types?

8. Aesthetic and minimalist design Is there excess visual clutter and/or text that is irrelevant to users’ main tasks?

3

Yes, ads and promoted articles

Is the visual design distracting or hand to pars, resulting in extra time to process information? (ex.

Color, typography, information display)

Does the visual design of the app pleasurable?

9. Help users recognize, diagnose, and recover from errors When errors do occur, are users made aware of them?

Are the error messages expressed in a way that users can understand?

3

Not always

Does the error message provide them with a clear course of action?

3

Not always

Does the platform provide the functionality ‘going back’ when any error is happened?

Does Auto-corrected result for misspelled search?

Is there a help document?

Can users find the help document?

10. Help and documentation

Is it helpful?

3

Only for specific situations

Is it well written, well organized, searchable?

Does the platform have a repository for frequent Q&A?

Does the platform provide a tutorial in the beginning of use?

3

Very minimal

Table D: Chuyao Wang Heuristic Evaluation

Review Checklist 1. Visibility of system status

Rating Comments

Can the user tell where they are in the system?

0

The tab bar shows clearly where the users are

Does every page have a title explaining the contents of the screen?

Does the system give appropriate feedback for user actions?

2

When not following any vibes, the "my vibes" button in the hamburger menu doesn't give any respond when selected, neither a notification to follow vibes.

Does the user know how to get back in the system?

0

Every detail page has a cancel button to get back

Can the user tell what options are available for what to do next?

3

I can't tell where to follow a news resource, and the "contributor" in the vibe is a little bit confusing

2. Match between system and the real world Does the system use language that is likely to be familiar to the user?

Does the system follow real-world conventions?

2

The name "vibe" is confusing at the first use. And "contributor", are they the editors of the news or the news resource?

Is the information displayed in a natural and logical order?

0

The organization of the information is pretty clear

Is the interaction of curating intuitively understandable for users?

1

The auto-slides are quite intuitive to use, the process bar is too narrow to notice, and I don't know how I should operate the slides at the first use

Is the interaction of reading an article intuitively understandable for users?

1

Hard without a tutorial

Can the user undo or revert mistaken input or paths?

2

If I post some texts mistakenly, there is no option to delete it immediately, I can only navigate to my profile to do it.

Can the user safely abandon tasks that go away?

2

There is no draft box for posts

Can the user do things in the order that they want to do them?

2

Users cannot go to top inside the articles

Does the user know where to find the articles/topics they want?

3

Users can only search by vibes, they cannot search article titles or news resources

3. User control and freedom

4. Consistency and standards Does the system comply with relevant standards (universal, platform, app category)?

Can the user do perform similar tasks by follow the same steps?

3

The font and other visual design elements are pretty consistent, but there is no "vibe", which is a key feature of the mobile app, in the webpage version.

Is the system consistent internally (ex. Color, typography, words, icons, information display)

0

Yes, the visual design elements are consistent

Do forms with similar components have similar layouts?

3

The reply to the comments have different design, the mobile version is "10 replies" and the website version is "show replies(10)"

Does the system warn users when they are about to take a “dangerous (i.e. Delete, remove,)” action?

2

When I'm about to delete my post, the system shows a warning of confirmation

Does the system expect users to remember earlier decisions/actions when completing later steps?

2

It doesn't help you remember where you had reached in the article.

5. Error prevention

Are error messages displayed in a proactive fashion?

Does the system force users to remember obscure commands or sequences?

Does the system expect users to remember earlier decisions/actions when completing later steps?

1

The system does pretty well when users posting a news-related post. It shows the news title and picture after the users select which news to post with

Is autofill utilized?

2

No but the content below the search bar would update according to the search content

6. Recognition rather than recall

Are drop down menus utilized?

Is an internal search functionality present?

Is bookmarking for an article enabled?

1

Every article is not very long, so maybe the designer assumed the users would finish reading in a short period without going back to the article

7. Flexibility and efficiency of use Does the system force expert user to do things in slow, inefficient ways?

Does the system allow people with different roles/styles/preferences to get things done?

Does the system allow users to tailor frequent actions?

Is autocomplete utilized in forms/search on the site?

Are there ways to input of different styles/types?

4

Users can only post texts, no memes, no picture, no emoji

Is there excess visual clutter and/or text that is irrelevant to users’ main tasks?

3

The way of ads inserting is kind of annoying and distractive

Is the visual design distracting or hand to pars, resulting in extra time to process information? (ex.

Color, typography, information display)

0

No, the visual design is pretty concise

8. Aesthetic and minimalist design

Does the visual design of the app pleasurable?

9. Help users recognize, diagnose, and recover from errors When errors do occur, are users made aware of them?

4

No, the system doesn't give any instructions

Are the error messages expressed in a way that users can understand?

Does the error message provide them with a clear course of action?

Does the platform provide the functionality ‘going back’ when any error is happened?

Does Auto-corrected result for misspelled search?

10. Help and documentation Is there a help document?

0

Yes, there is a document

Can users find the help document?

0

Yes

Is it helpful?

0

Maybe

Is it well written, well organized, searchable?

4

It's full of texts and I don't think any users would spend time reading it.

Maybe it should insert several infographics into the help document

Does the platform have a repository for frequent

Q&A? Does the platform provide a tutorial in the beginning of use?

4

No, only very limited instruction. It may take time for first-time users to figure out what is a "vibe"

Table E: Blake Wagner Heuristic Evaluation

Review Checklist

Rating Comments

1. Visibility of system status Can the user tell where they are in the system?

2

Not always. Links out of app to articles; especially confusing with Yahoo! News links

Does every page have a title explaining the contents of the screen?

Does the system give appropriate feedback for user actions?

Does the user know how to get back in the system?

2

When navigating out of app to article there are two options to return: 1) back button in top-left; and 2) "X" in top-right corner

Can the user tell what options are available for what to do next?

2

If not following vibes, "my vibes" in hamburger menu is dead (no feedback)

Does the system use language that is likely to be familiar to the user?

2

-Vibes – real world: “energies/feelings”; YN: “topic areas”; -Reactions real world: “emoji visuals”; YN: “comments”

Does the system follow real-world conventions?

3

-Video stories use “orange” progress bar; real-world: white; -Video stories “hold-tap” advances to next segment; real-world: pauses segment; -Video stories zoom-in style not suggestive of video; -Video thumbnail opens up into text story (with video)

Is the information displayed in a natural and logical order?

3

-Post story default screen is text; must toggle to tiny news tab to find article to post

2. Match between system and the real world

Is the interaction of curating intuitively understandable for users?

Is the interaction of reading an article intuitively

understandable for users?

3. User control and freedom Can the user undo or revert mistaken input or paths?

4

-Could not change gender or DOB

Can the user safely abandon tasks that go away?

4

Unfollowing vibes does not go into effect until app is restarted

Can the user do things in the order that they want to do them?

3

-Post story default screen is text; must toggle to tiny news tab to find article to post

Does the user know where to find the articles/topics they want?

3

Vibe search: does not populate similar topics/keywords

4. Consistency and standards Does the system comply with relevant standards (universal, platform, app category)?

Can the user perform similar tasks by following the same steps?

Is the system consistent internally (ex. Color, typography, words, icons, information display)

2

Inconsistent use of blue/purple and purple colored to distinguish users from news sources

Do forms with similar components have similar layouts?

3

Vibe search: does not populate similar topics/keywords

5. Error prevention Does the system warn users when they are about to take a “dangerous (i.e. Delete, remove,)” action?

Does the system expect users to remember earlier decisions/actions when completing later steps?

Are error messages displayed in a proactive fashion?

3

If a user unfollows a Vibe, the only way they are able to follow it again is remembering which Vibe it was and finding it again

Does the system force users to remember obscure commands or sequences?

Does the system expect users to remember earlier decisions/actions when completing later steps?

6. Recognition rather than recall

Is autofill utilized?

3

Search function on 'Post story' page is a broad internet search that does not really support users in finding a story they are looking for

Are drop down menus utilized?

Is an internal search functionality present?

3

Is bookmarking for an article enabled?

Recent activity/history does not work

7. Flexibility and efficiency of use Does the system force expert user to do things in slow, inefficient ways?

Does the system allow people with different roles/styles/preferences to get things done?

3

2/3

Does the system allow users to tailor frequent actions?

Is autocomplete utilized in forms/search on the site?

2/3

Are there ways to input of different styles/types?

-Lag when toggling to "Post" navigation tab; -Post story default screen is text; must toggle to tiny news tab to find article to post Cannot post to vibe without following vibe

Vibe search: does not populate similar topics/keywords

8. Aesthetic and minimalist design Is there excess visual clutter and/or text that is irrelevant to users’ main tasks?

Is the visual design distracting or hand to parse, resulting in extra time to process information? (ex.

Color, typography, information display)

2

Does the visual design of the app pleasurable?

Video articles provide unnecessary panning/movement that can deter from the text

When errors do occur, are users made aware of them?

Are the error messages expressed in a way that users can understand?

9. Help users recognize, diagnose, and recover from errors

Does the error message provide them with a clear course of action?

4

“Oh no…” error message in account settings – “start over” tab button is dead

Does the platform provide the functionality ‘going back’ when any error is happened?

Does Auto-corrected result for misspelled search?

Is there a help document?

Can users find the help document?

10. Help and documentation

Is it helpful?

3/4

Account security tutorial is not working

Is it well written, well organized, searchable?

Does the platform have a repository for frequent Q&A?

Does the platform provide a tutorial in the beginning of use?