Can you spot the problem with these headlines? (Level 1) - Jeff Leek and Lucy McGowan
- 431,410 Views
- 17,162 Questions Answered
- TEDEd Animation
In our videos, we briefly discuss the design of a study, for example paying attention to whether there is a control group. There are many types of studies but two big types are randomized, controlled trials (RCTs) and observational studies. RCTs are the gold standard, because they let us prove that an exposure, like taking a drug or changing a diet, causes a change in your health. Observational studies can support the hypothesis that the drug or diet causes the change, but you usually need a lot more information. Hill’s criteria are a set of extra bits of information that might lend more credibility to an observational study (see Lucy’s fun blog post on this for more).
In addition to the design of the study, it is important to notice the sample size.Bigger studies usually let us measure results more accurately, especially when the result is likely to be noisy or hard to get a good handle on. To demonstrate this, a science journalist, John Bohannon, conducted a study that found that eating chocolate lead to weight loss. The problem? This study was only conducted on only 16 participants and John purposefully looked at many different outcomes before he found a significant one. (Check out John’s write up of this study).
If the impact of the exposure, like a drug or diet, is pretty small we call that a small effect size. The smaller the impact, the harder it is to reliably find it with a study. When small effect sizes are combined with small sample sizes it makes it really, really hard to figure out if we found something real or not in the data. Similarly, in a very large study, we may be able to detect a very small effect that is statistically significant, but not actually scientifically meaningful. For example, remember the study we discussed that examined whether a new drug prolonged the life of patients with pancreatic cancer - while the result was statistically significant,
the actual effect was only a difference of 10 days.
Finally, it is important to pay attention to the outcome being analyzed. Many times, surrogate outcomes are used. Our example is based on a real case from the medical literature. In the mid-twentieth century, researchers started publishing studies suggesting that a B-vitamin called niacin might help prevent heart disease by raising levels of beneficial cholesterol. By 2009, more than 700,000 prescriptions for niacin were being written in the U.S. every month. There was just one problem. It turns out that niacin doesn’t help prevent heart disease — but it does increase the risk of serious side effects. The original niacin studies concluded that the vitamin would reduce the risk of heart disease because it raised the levels of the healthy form of cholesterol, and people with higher levels tend to have fewer heart attacks. But they didn’t measure heart attack risk directly. When other scientists eventually did, it turned out that niacin’s effect on cholesterol didn’t actually lead to fewer heart attacks.
A great way to dig deeper is to read the research article behind the headlines. Read Jeff’s blog post for tips on tracking down the original research.
We have only scratched the surface of things to think about when evaluating scientific studies. If you are interested in pursuing this area further, we have a short massive online open course (MOOC) that dives a bit deeper into these issues. The course is designed to be completed by busy students and professionals, with short lessons and quizzes that can be completed on the go. We take real examples from the medical literature and dive into the text to understand the study design, statistical methods, and results. The whole course can be completed in about four hours. We leave out the mathematical detail and focus on the conceptual ideas. Anyone can pick this course up and gain a better understanding of how data and statistics are used in the medical literature. You can find the course on Leanpub: Understanding Data and Statistics in Medical Literature.
Lucy D’Agostino McGowan and Jeff Leek
Create and share a new lesson based on this one.
More from Media Literacy
Can you outsmart a troll (by thinking like one)?
lesson duration 05:01
Why people fall for misinformation
lesson duration 05:16
This one weird trick will help you spot clickbait
lesson duration 05:38
How to spot a misleading graph
lesson duration 04:10