Read a Statistical Claim

"Study finds coffee drinkers live longer." A headline like that compresses months or years of work into one sentence. It names a result. It rarely says who was studied, how the study was run, what was counted, or how sure anyone can be.

Four gaps matter most when you read a reported finding:

• Who was studied. How many people, and how they were chosen. A few hundred volunteers can tell a different story from a random sample of thousands.
• How it was studied. Whether researchers assigned people to groups at random or watched what people did on their own. The first design, done well, gives the most direct evidence about cause.
• What was measured. "Health" in a headline might mean a blood test, a questionnaire answer, or a death certificate.
• How uncertain the result is. Every estimate drawn from a sample carries chance error, and studies usually report it as a margin of error, a confidence interval, or a p-value.

You might expect a finding published in a scientific journal to have settled these questions already. Replication research suggests caution. In one large project, teams repeated 100 studies published in 2008 in three psychology journals. Ninety-seven percent of the original studies had reported statistically significant results, meaning results unlikely under the assumption of no effect. Only 36 percent of the repeats did. On average, the repeated effects were about half the size of the originals (Open Science Collaboration 2015).

That result doesn't show that the original studies were wrong. The authors warn that a failed repeat doesn't by itself prove the first result was a false positive, and a successful repeat doesn't prove the explanation behind it (Open Science Collaboration 2015). It shows that one published study is less certain than it looks. The figures come from one field and one set of journals, and how often findings hold up in other fields is still being studied.

Part of the trouble lies in how results are reported. In 2016 the American Statistical Association issued a statement on p-values, the most quoted measure of uncertainty in research. It stressed that a p-value doesn't measure the size of an effect or its importance, and that conclusions shouldn't rest only on whether a p-value passes a fixed threshold (Wasserstein and Lazar 2016). A headline that says "significant" often means only that the threshold was passed.

In this project you'll answer the question people ask when they forward a headline: is this real? You'll pick one news report of a study or poll, trace it to the original, and write a 600 to 900 word assessment of its sample, design, measurement, and uncertainty. You'll end with one careful sentence on what the evidence supports. An answer that accepts every headline leads people to act on weak findings. One that dismisses every study teaches them to ignore good evidence too.

This content reflects the field as of October 2026.

References

• Open Science Collaboration. 2015. "Estimating the Reproducibility of Psychological Science." Science 349 (6251): aac4716.
  • Free: Utrecht University repository (PDF)
• Wasserstein, Ronald L., and Nicole A. Lazar. 2016. "The ASA's Statement on p-Values: Context, Process, and Purpose." The American Statistician 70 (2): 129–133. doi:10.1080/00031305.2016.1154108.
  • Free: UC Berkeley copy (PDF)

Deliverable Overview

You'll write one assessment, 600 to 900 words long, of a single news report about a study or a poll. The assessment traces the reported claim back to its source and judges the evidence behind it. It has seven required components:

1. The claim as reported, with the news source
2. The original study or poll, with a link
3. The sample: its size, how it was chosen, and its likely biases
4. The design: randomized, observational, natural experiment, or survey
5. The measurement: what was measured and how
6. The uncertainty: the margin of error, confidence interval, or p-value as reported, read correctly
7. A verdict: what the evidence supports, in one careful sentence

A short heading for each component makes the assessment easy to follow. Citations are welcome. A simple format works: author or organization, year, title, and where to find it, such as a link. A reference list doesn't count toward the word limit.

Detailed Requirements

The Claim as Reported

Quote or closely paraphrase the claim the news report makes, and name the outlet and the date. Record the claim as the report states it, including any causal wording such as "cuts," "causes," or "linked to." You'll compare that wording against the study at the end.

The Original Source

Give the title of the study or poll report, who produced it, when, and a link. A journal article, a preprint, a government report, or a pollster's release all count. A preprint is a paper posted before peer review, so say so if yours is one. If the news report links only to another news story, keep following the trail until you reach the source.

The Sample

State how many people, places, or records the study used, and how they were chosen. A random sample gives every member of the population a known chance of selection. A convenience sample uses whoever is easy to reach. Samples made up of people who choose to respond, as in call-in surveys, are often unreliable (Illowsky and Dean 2023, sec. 1.2). Then name the likely biases. Sampling bias arises when some members of the population are less likely to be chosen than others. Refusal to take part can also leave a sample that no longer represents the population (Illowsky and Dean 2023, sec. 1.2).

The Design

Name the design and say why it fits. In a randomized experiment, researchers assign participants to groups by chance, which spreads other influences evenly across the groups, so a difference in outcomes can be credited to the treatment (Illowsky and Dean 2023, sec. 1.4). In an observational study, researchers record what people already do. A lurking variable, a factor outside the study's focus that affects the outcome, can then explain an association. In a natural experiment, chance events or policy changes treat groups of people differently, and researchers use that difference to estimate cause and effect (Nobel Prize Outreach 2021). A survey asks a sample questions in order to describe a larger population.

The Measurement

State what was measured and how. A headline's "anxiety" might be a diagnosis, a score on a questionnaire, or a single yes-or-no question. Note whether the outcome was measured directly or reported by participants, and whether it matches what the headline names.

The Uncertainty

Report the uncertainty the source gives, and restate it in plain words. A margin of error is the amount added to and subtracted from a poll's estimate to give a range. A poll reporting 52 percent support, plus or minus 3 points, gives a range of 49 to 55 percent. A confidence interval is a range built by a method that captures the true value for a set share of well-chosen samples, usually 95 percent (Illowsky and Dean 2023, chap. 8). Any one interval might miss. And a margin of error covers only sampling error, the chance variation from drawing a sample. Nonsampling errors, such as slanted question wording or a faulty count, aren't in it (Illowsky and Dean 2023, sec. 1.2).

A p-value is the probability, under a specified model, usually one with no real effect, of a result at least as extreme as the one observed. It isn't the probability that the hypothesis is true, and it doesn't measure the size or importance of an effect (Wasserstein and Lazar 2016). A result called statistically significant has a p-value below a chosen threshold, often 0.05 (Illowsky and Dean 2023, chap. 9).

The Verdict

Write one sentence that says what the evidence supports, at the strength the evidence supports it. Name the population, the design's reach (association or cause), and the uncertainty. A careful verdict might read: "In a randomized trial of 400 adults over 60, the program lowered falls over one year, though the interval runs from a small effect to a moderate one." Then make sure nothing elsewhere in the assessment claims more than that sentence.

Success Criteria

Your assessment meets the requirements when:

• The reported claim and the original source are both given
• The sample's size, selection, and likely biases are described
• The design is named correctly
• What was measured, and how, is stated
• The uncertainty is reported and read correctly
• The verdict is one careful sentence that doesn't go beyond the evidence

Step-by-Step Guidance

Step 1: Choose a recent news report of a study or poll with a traceable source. Look for a report that names the researchers, the journal, or the polling organization. A report that says only "a new study" may be hard to trace, so set it aside unless you can find the source.

Step 2: Find the original study or poll report. Follow the link in the report, or search for the title, the lead author, or the pollster's name. Many journals show at least the abstract for free, and pollsters often post their full results and method.

Step 3: Record the sample, design, and measurement from the source. Take these from the methods section of a paper, or from the methodology note of a poll, not from the news story. Note the sample size, how participants were chosen, whether anyone was assigned to groups, and how the outcome was measured.

Step 4: Find the reported uncertainty and restate it in plain words. Look for the margin of error, a confidence interval, or a p-value. Write one sentence that says what it means and what it doesn't.

Step 5: Write your verdict in one careful sentence, then the full assessment. Writing the verdict first keeps the rest of the assessment aimed at it. Then write the seven components, with a short heading for each.

Step 6: Check every claim in your assessment against the source. Reread the source beside your draft. Confirm each number, each description of the design, and each word in the verdict, and cut anything the source doesn't support.

Resources

• OpenStax: Introductory Statistics 2e: a free textbook. Its sections on sampling (sec. 1.2) and experimental design (sec. 1.4), and its chapters on confidence intervals (chap. 8) and hypothesis testing (chap. 9), cover the terms you'll use (Illowsky and Dean 2023).
• The ASA's statement on p-values (PDF): a short statement of six principles for reading p-values (Wasserstein and Lazar 2016).
• Notes or practice work of your own on samples, study designs, and margins of error.

Common Pitfalls

• Assessing the headline instead of the study. The headline is a summary written for clicks and speed. Your assessment is of the source, so every judgment should rest on what the study or poll report says.
• Treating "significant" as "large." Statistically significant means the result passed a threshold, not that the effect is big or matters. With a large sample, a tiny effect can be significant (Wasserstein and Lazar 2016). Report the size of the effect alongside its significance.
• Ignoring how the sample was chosen. A large sample doesn't fix a biased one. Ten thousand self-selected volunteers can still differ from the population in ways that matter.
• Reading a confidence interval as a range of certainty. A 95 percent confidence interval comes from a method that works for about 95 percent of samples, and this one might be among the misses. Values near either end of the range are also compatible with the data.
• Dismissing a study only because it's observational. Some questions can't be studied by random assignment; no one can be assigned a birth order (Illowsky and Dean 2023, sec. 1.4). Observational evidence can be strong. Judge how well the study accounts for other explanations, and say what it can and can't show about cause.

References

• Illowsky, Barbara, and Susan Dean. 2023. Introductory Statistics 2e. Houston, TX: OpenStax. CC BY-NC-SA.
  • Free: OpenStax: Introductory Statistics 2e
• Nobel Prize Outreach. 2021. "The Prize in Economic Sciences 2021." Press release, October 11, 2021.
  • Free: NobelPrize.org
• Wasserstein, Ronald L., and Nicole A. Lazar. 2016. "The ASA's Statement on p-Values: Context, Process, and Purpose." The American Statistician 70 (2): 129–133. doi:10.1080/00031305.2016.1154108.
  • Free: UC Berkeley copy (PDF)