Skip to main content

The method that can "prove" almost anything - James A. Smith


3,088 Questions Answered

TEDEd Animation

Let’s Begin…

In 2011, a group of researchers conducted a study designed to find an impossible result. Their study involved real people, truthfully reported data, and commonplace statistical analyses. So how did they do it? The answer lies in a statistical method scientists often use to try to figure out whether their results mean something, or if they’re random noise. James A. Smith explores p-hacking.

Additional Resources for you to Explore

P-values are part of a method in science called ‘hypothesis-testing’. Hypothesis testing involves defining a null-hypothesis: a statement that would be true if nothing happened in your experiment. For example, imagine you want to test whether people who ate chocolate were heavier than those who didn’t: the null hypothesis would be that there is no difference in weight between people eating and not eating chocolate.

We’d also state an alternative hypothesis. This reflects what we think might be going on: for example, the alternative hypothesis would be that there is a difference in weight between people eating and not eating chocolate.

The p-value provides evidence against the null hypothesis. If it’s low, such as below 0.05, then it’s quite unlikely that the null hypothesis is correct, so we reject it. 

But P-values are really hard to understand, and even leading scientists struggle to explain what they mean. They tell you the probability of observing data at least as extreme as the data you observed in an experiment, assuming that the null hypothesis is true. It’s easy to think that they also tell you the probability that the null hypothesis is true, but that’s not quite right, because the probability we calculate assumes that the null hypothesis is true.

Some people think that the widespread use of and misunderstanding about p-values leads to problems with the reproducibility of scientific findings. You can learn more about problems with reproducibility in science in this video.

Don’t lose faith in science though – it’s one of the best ways that we have to learn about and improve the world. And partly because it’s so important, there is a new field called ‘metascience’, that aims to better understand and improve the way that science is done.

Next Section »

About TED-Ed Animations

TED-Ed Animations feature the words and ideas of educators brought to life by professional animators. Are you an educator or animator interested in creating a TED-Ed Animation? Nominate yourself here »

Meet The Creators

  • Educator James A. Smith
  • Director Anton Bogaty
  • Narrator Addison Anderson
  • Music Salil Bhayani, cAMP Studio
  • Sound Designer Amanda P.H. Bennett
  • Director of Production Gerta Xhelo
  • Editorial Director Alex Rosenthal
  • Producer Bethany Cutmore-Scott
  • Editorial Producer Elizabeth Cox
  • Fact-Checker Jennifer Nam

More from Math In Real Life