Rudiments of digital research

From Why Google Can’t Save Us by Sam Wineburg, Joel Breakstone, Sarah McGrew, and Teresa Ortega. Abstract.

The Stanford History Education Group has prototyped, field tested, and validated a bank of assessments that tap civic online reasoning—the ability to judge the credibility of the information that floods young people’s smartphones, tablets, and computers. We developed 56 tasks and administered them to students across 12 states. In total, we collected and analyzed 7,804 student responses. From pre-teens to seniors in college, students struggled mightily to evaluate online information. To investigate how people determine the credibility of digital information, we sampled 45 individuals: 10 PhD historians, 10 professional fact checkers, and 25 Stanford University undergraduates. We observed them as they evaluated websites and engaged in open web searches on social and political issues. Historians and students often fell victim to easily manipulated features of websites, such as official-looking logos and domain names.

Elaborating in the report:

The Stanford History Education Group has prototyped, field tested, and validated a bank of assessments that tap civic online reasoning—the ability to judge the credibility of the information that floods young people's smartphones, tablets, and computers. We developed 56 tasks and administered them to students across 12 states. In total, we collected and analyzed 7,804 student responses. From pre-teens to seniors in college, students struggled mightily to evaluate online information... The results of our study, combined with findings from other researchers provide evidence that the myth of the digital native is precisely that—a myth... Overall, young people's ability to reason about information on the Internet can be summed up in one word: Bleak. These "digital natives" may be able to flit between Facebook and Twitter while simultaneously uploading a selfie to Instagram and texting a friend. But when it comes to evaluating information that flows through social media channels, they are easily duped...

To investigate how people determine the credibility of digital information, we sampled 45 individuals: 10 PhD historians, 10 professional fact checkers, and 25 Stanford University undergraduates. We observed the groups as they evaluated live websites and searched for information on social and political issues. Viihat we found was not what we expected. Both historians and students often fell victim to easily manipulated features of websites, such as official-looking logos and domain names, and took at face value claims that were anchored to dubious scholarly references.... Paradoxically, a key feature of lateral reading is not reading. Fact checkers intelligently ignored massive amounts of irrelevant (or less crucial) text when making an informed judgment. In their evaluations, historians and students relied on common but deeply flawed weak heuristics, like viewing a domain designation as a proxy for trustworthiness.

From an earlier report.

In a study conducted by Eszter Hargittai and her colleagues at Northwestern University, 102 college students went online to answer questions about things that matter to them—like how to advise a female friend who's desperate to prevent pregnancy after her boyfriend's condom broke. How did students decide what to believe? One factor loomed largest: a site's placement in the search results. Students ignored the sponsoring organization and the article's author, blindly trusting the search engine to put the most reliable results first.

Students Can't Google Their Way to the Truth, Students would be wise to learn the strategies fact-checkers use to evaluate online information, write Sam Wineburg and Sarah McGrew.
—Getty
Research we've conducted at Stanford University supports these findings. Over the past 18 months, we administered assessments that tap young people's ability to judge online information. We analyzed over 7,804 responses from students in middle school through college. At every level, we were taken aback by students' lack of preparation: middle school students unable to tell the difference between an advertisement and a news story; high school students taking at face value a cooked-up chart from the Minnesota Gun Owners Political Action Committee; college students credulously accepting a .org top-level domain name as if it were a Good Housekeeping seal.

One task asked students to determine the trustworthiness of material on the websites of two organizations: the 66,000 member American Academy of Pediatrics, established in 1930 and publisher of the journal Pediatrics, vs. the American College of Pediatricians, a fringe group that broke with the main organization in 2002 over its stance on adoption by same-sex couples. We asked 25 undergraduates at Stanford—the most selective college in the country, which rejected 95 percent of its applicants last year—to spend up to 10 minutes examining content on both sites. Students could stay on the initial web page, click on links, Google something else—anything they would normally do to reach a judgment.

"For every political question swirling in this election, there are countless websites vying for our attention."
More than half concluded that the article from the American College of Pediatricians, an organization that ties homosexuality to pedophilia and which the Southern Poverty Law Center labeled a hate group, was "more reliable." Even students who preferred the entry from the American Academy of Pediatrics never uncovered the differences between the two groups. Instead, they saw the two organizations as equivalent and focused their evaluations on surface features of the websites. As one student put it: "They seemed equally reliable to me. ... They are both from academies or institutions that deal with this stuff every day."

Ironically, many students learned so little because they spent most of their time reading the articles on each organization's site. But masking true intentions and ownership on the web has grown so sophisticated that to rely on the same set of skills one uses for print reading is naive. Parsing digital information before one knows if a site can be trusted is a colossal waste of time and energy.

When they asked fact-checkers to perform the same function, they identified three major differences between digital native's naive searchers and the more sophisticated searching of fact-checkers.

• Landing on an unfamiliar site, the first thing checkers did was to leave it. If undergraduates read vertically, evaluating online articles as if they were printed news stories, fact-checkers read laterally, jumping off the original page, opening up a new tab, Googling the name of the organization or its president. Dropped in the middle of a forest, hikers know they can't divine their way out by looking at the ground. They use a compass. Similarly, fact-checkers use the vast resources of the Internet to determine where information is coming from before they read it.

• Second, fact-checkers know it's not about "About." They don't evaluate a site based solely on the description it provides about itself. If a site can masquerade as a nonpartisan think tank when funded by corporate interests and created by a Washington public relations firm, it can surely pull the wool over our eyes with a concocted "About" page.

• Third, fact-checkers look past the order of search results. Instead of trusting Google to sort pages by reliability (which reveals a fundamental misunderstanding of how Google works), the checkers mined URLs and abstracts for clues. They regularly scrolled down to the bottom of the search results page in their quest to make an informed decision about where to click first.

All useful guidance though I would note that the authors are themselves naive in their appeal to authority. Established organizations with broad membership are usually reasonably centered in their knowledge and opinions and that is useful to know. But new insights and different ways of looking at things often emerge at the edges and the borderlands.

If you want to know the received wisdom, go to the established sites. If you want innovation and insight, go to the borderlands. It might be repugnant there, and you can certainly waste a lot of time with dross, but they sometimes are bell weathers of emerging knowledge or insight.