On a new episode of ID the Future, guest host Eric Anderson welcomes medical engineer and scientist Rob Stadler to begin a two-part discussion about the critical need for a new approach to evaluating the strength of evidence in science. Drawing from 30 years of experience in a field where lives depend on rigorous regulatory standards, Stadler explains how he developed six criteria to distinguish between high-confidence and low-confidence scientific claims. These criteria evaluate both the quality of the experiment and the quality of the scientist.
To illustrate the difference between high and low confidence science, Stadler shares a humbling story from his career in medical devices. When evaluating a heart device feature intended to reduce atrial fibrillation, his team conducted a retrospective study by looking back at a massive database of 37,000 patients to see who had the feature turned on, finding an impressive 50 percent reduction in the heart problem. But because they were simply looking at old data, they had no control over confounding factors, like why a doctor chose to turn the feature on for one patient but not another. Four years later, a prospective study proved that the setting actually had no impact at all. So the smaller, more controlled study provided a high-confidence result that completely overturned the misleading findings of the much larger, but less rigorous, retrospective study.
Stadler also contrasts the cautious, legally-mandated language of medical claims with the often absolute, overstated confidence found in evolutionary biology textbooks. He argues that by applying these practical filters to scientific claims, we can learn to critically assess science news headlines and understand why certain fields lack the evidentiary rigor demanded by the medical world.
Download the podcast or listen to it here. This is Part 1 of a two-part conversation. Look for Part 2 next!
Dig Deeper
- Get a copy of Stadler’s book The Scientific Approach to Evolution!
- Stadler is a script writer on our Long Story Short animated video series. Watch them all here.