A news credibility evaluation application for Facebook
Context:
Academic project for my Human-AI Interaction class.
Led the research and documentation for this project.
The team consisted of 3 members and the project spanned 12 weeks.
What were we trying to solve?
Social media is a particularly effective platform for the spread of mis/disinformation. Several tools exist to check the credibility of news articles on these platforms. However, their adoption and usage are still not widespread.
What challenges prevent their adoption? How can we design these tools to help users become more critical evaluators and consumers of news on social media?
Our solution
NewsGuide provides a compact credibility assessment survey in the user's news consumption environment on Facebook. The detailed summary also provided quick access to related news and videos to encourage users to be more critical consumers of news online.
Research
Exploratory Research - Phase 1
To understand the context of the problem clearly, we started by exploring two main areas. For each area, we used exploratory research methods as listed below:
a. What were our user's news consumption and credibility evaluation habits?
b. What are the existing tools in market for news credibility evaluation?
Insights:
Our literature review and screener yielded the following statistics:
90%
Get news on digital devices
68%
Experience news fatigue
86%
Do not fact-check often
31%
Get news on Facebook
From our competitive analysis, we learnt of the two broad categories of news evaluation tools online : Fact-checking tools and Credibility evaluation tools. We decided to focus on those typically used as both.
Next, we conducted interviews to hear directly from users. Our interviews centered around three themes. The findings from our interviews are listed below:
"Too much information online"
"Hard to distinguish the truth"
"Will only evaluate news if interested or invested"
"No time to evaluate"
Google Search
Discussions with people
Trusted/official sources
Cross-check with multiple sources
Wikipedia, YouTube and tools like Snopes
These insights from our exploratory research helped us to redefine our goals for the next phase of our research:
Exploratory Research - Phase II
a. Observation Study
With a good understanding of the problem space, we could proceed to the next phase of our research. Now that we knew of people's news consumption habits, we wanted to observe them evaluate news articles. We asked them to perform the following two tasks:
The observational study helped us narrow down the key factors that are most essential to users in evaluating the credibility of a news article. This was also consistent with our findings from the exploratory research as well. However, we still needed insight into the relative importance of these factors to different people so as to generalize our findings.
b. News Factor Assessment Survey
To rank the importance of the factors, we conducted a survey to generalize our findings better.
Design
Phase 1:
With the research phase completed, we had an outline of our extension in mind, along with a clear idea on how to present the information in order of its importance. We jumped into building a mid-fidelity design roughly based on Logically's design along with inputs from Newstrition.
User flow
The primary user flow for using this application is as follows:
The user encounters an indication on the top-right corner of a news article in Facebook. The extension computes a credibility score and displays it.
On hovering on the credibility score, a small window options up with a quick credibility summary.
On clicking on the "More" button, a detailed description of the credibility assessment summary opens up.
Mapping evaluation factors to the UI
We ensured that each of the factors which we discovered to be important for credibility evaluation was included in the result detailed summary. Here is how we mapped it
Heuristic Evaluation
Lastly, we conducted a quick Heuristic Evaluation and decided on a color scheme to come up with the first iteration of our high-fidelity prototype.
Testing
To gather feedback on our high-fidelity design, we conducted two tests as follows:
Usability Testing
We gathered 10 participants to perform 5 tasks. Each task evaluated the ease-of-use and effectiveness the following five screens:
A/B testing
We also conducted A/B testing to experiment with the way information was presented. We wanted to understand what kind of information made most sense to people to evaluate credibility effectively - would a label such as "Trustworthy" help or would a numeric score be better?
Reflection
This project helped me see the necessity of being flexible with research frameworks. We had to go back to our research phase several times to ensure that we had clear ideas to shape our design.
Innovation vs re-inventing the wheel - our competitive analysis showed us what already works in existing tools which helped us make quicker and more effective design decisions.
The several methods and research phases involved made this project a very valuable lesson in time management as well.