<p><span id="docs-internal-guid-bb3b2ca0-baa5-9f8a-3944-25d195d2a871"> </span></p> <p dir="ltr"><span>1. Science needs some tough love (fields vary, but some enable and encourage unhealthy habits). And “good cop” approaches aren't fixing “phantom patterns” and “noise mining” (explained below).</span></p> <p dir="ltr"><span>2. Although everyone’s doing what seems “scientifically </span><a href="http://pllqt.it/PCy5my" target="_blank"><span>reasonable</span></a><span>” the result is a “machine for producing and publicizing random </span><a href="http://pllqt.it/PCy5my" target="_blank"><span>patterns</span></a><span>,” statistician Andrew </span><a href="http://www.slate.com/articles/health_and_science/science/2013/07/statistics_and_psychology_multiple_comparisons_give_spurious_results.html" target="_blank"><span>Gelman</span></a><span> says.</span></p> <p dir="ltr"><span>3. Gelman is too kind; the “reproducibility </span><a href="http://www.bbc.com/news/science-environment-39054778"><span>crisis</span></a><span>” is really a producibility problem—professional practices reward production and publication of unsound studies. </span></p> <p dir="ltr"><span>4. Gelman calls such studies “dead on </span><a href="http://andrewgelman.com/2016/06/26/29449/" target="_blank"><span>arrival</span></a><span>,” but they’re actually dead on departure, doomed at conception by “flaws inherent in [their] original </span><a href="http://pllqt.it/jzjgFR" target="_blank"><span>design</span></a><span>” (+much that’s “poorly </span><a href="http://pllqt.it/JwmA5m" target="_blank"><span>designed</span></a><span>” gets published).</span></p> <p dir="ltr"><span>5. Optimists say relax, “science is </span><a href="https://fivethirtyeight.com/features/science-isnt-broken/" target="_blank"><span>self-correcting</span></a><span>.” For instance, Christie Aschwanden says the “replication crisis is a sign that science is </span><a href="https://fivethirtyeight.com/features/failure-is-moving-science-forward/" target="_blank"><span>working</span></a><span>,” it’s not “</span><a href="https://fivethirtyeight.com/features/science-isnt-broken/" target="_blank"><span>untrustworthy</span></a><span>,” it’s just messy and hard (it’s “in the long run… </span><a href="https://www.sciencenews.org/blog/context/debates-whether-science-broken-dont-fit-tweets" target="_blank"><span>dependable</span></a><span>,” says Tom Siegfried).</span></p> <p dir="ltr"><span>6. “Science Is </span><span><a href="http://www.slate.com/articles/health_and_science/science/2017/08/science_is_not_self_correcting_science_is_broken.html" target="_blank">Broken</a></span><span>” folks like Dan Engber ask, “how quickly does science self-correct? Are bad ideas and wrong results stamped out [quickly]... or do they last for generations?” And at what (avoidable) cost?</span></p> <p dir="ltr"><span>7. We mustn’t overgeneralize—physics isn’t implicated, instructively it’s intrinsically less variable, (all electrons behave consistently). Biology and social science aren’t so lucky: People ≠ biological billiard </span><a href="http://bit.ly/1zmwOY4" target="_blank"><span>balls</span></a><span>. </span></p> <p dir="ltr"><span>8. Richard Harris’s </span><a href="http://www.pbs.org/newshour/updates/why-bad-science-is-plaguing-health-research-rigor-mortis-richard-harris/"><span>book</span></a><span> </span><em>Rigor Mortis </em><span>argues “sloppy science… wastes billions” (~50% of US </span><a href="https://www.ft.com/content/cc8b60ce-4225-11e7-82b6-896b95f30f58?mhq5j=e2" target="_blank"><span>taxpayer-funded</span></a><span> biomedical research budget, ~$15 billion squandered).</span></p> <p dir="ltr"><span>9. Harris blames </span><span>ultra-competitive “publish first, correct </span><a href="https://www.ft.com/content/cc8b60ce-4225-11e7-82b6-896b95f30f58" target="_blank"><span>later</span></a><span>”</span><span> games, and heartbreakingly abysmal experimental </span><a href="https://books.google.com/books?id=iRk9DQAAQBAJ&printsec=frontcover&dq=rigor+mortis&hl=en&sa=X&ved=0ahUKEwjw96zhr-bUAhUE8j4KHfDPDNkQ6wEIKTAA#v=onepage&q=abysmal%20experimental%20design&f=false" target="_blank"><span>design</span></a><span>, that can threaten </span><a href="http://pllqt.it/Q83Lbd" target="_blank"><span>lives</span></a><span> </span><span>(Gelman concurs, “Clinical trials are </span><span><a href="http://andrewgelman.com/2017/07/11/clinical-trials-broken-heres/" target="_blank">broken</a>.</span><span>”).</span></p> <p dir="ltr"><span>10. Harris sees </span><span>“no </span><a href="http://www.npr.org/books/titles/522223284/rigor-mortis-how-sloppy-science-creates-worthless-cures-crushes-hope-and-wastes#excerpt" target="_blank"><span>easy</span></a><span>” fix. But a science-is-hard defense doesn’t excuse known-to-be-bad practices.</span></p> <p dir="ltr"><span>11. Engber’s “bad ideas and wrong results” are dwarfed by systemic generation-spanning method-level ills. For instance, Gelman calls traditional statistics “</span><a href="http://retractionwatch.com/2017/02/09/traditional-statistics-often-counterproductive-research-human-sciences/" target="_blank"><span>counterproductive</span></a><span>”—badly misnamed “statistical significance” tests aren’t </span><span>arbiters “of </span><a href="http://www.chronicle.com/article/A-New-Theory-on-How/240470" target="_blank"><span>scientific truth</span></a><span>," though they’re widely used that way. </span></p> <p dir="ltr"><span>12. Psychology brought “statistical significance” misuse to light recently (e.g.,the TED chart-topping “</span><span>power </span><a href="http://www.slate.com/articles/health_and_science/science/2016/01/amy_cuddy_s_power_pose_research_is_the_latest_example_of_scientific_overreach.html" target="_blank"><span>pose</span></a><span>”)</span><span>, but Deirdre McCloskey declared "statistical significance has ruined empirical… </span><a href="http://bit.ly/2rMHJNe" target="_blank"><span>economics</span></a><span>" in 1998, and traced concerns to 1920s. Gelman wants us to “abandon statistical </span><a href="http://www.stat.columbia.edu/~gelman/research/unpublished/abandon.pdf" target="_blank"><span>significance</span></a><span>.” </span></p> <p dir="ltr"><span>13. Yet “</span><a href="http://andrewgelman.com/2015/09/03/name-this-blog-post-a-contest/" target="_blank"><span>noise mining</span></a><span>” abounds</span><span>. Fields with inherent variability, small effects, and noisy measurements drown in datasets with phantom patterns, unrelated to stable causes </span><span>(see Cornell’s </span><span>“world-renowned eating... </span><a href="http://andrewgelman.com/2017/06/15/pizzagate-gets-even-ridiculous-either-not-read-previous-pizza-buffet-study-not-consider-part-literature-later-study-found/" target="_blank"><span>expert</span></a><span>”)</span></p> <p dir="ltr"><span>14. No “statistical </span><a href="https://en.wikipedia.org/wiki/Statistical_alchemy" target="_blank"><span>alchemy</span></a><span>” (Keynes, 1939) can diagnose phantom patterns. Only further reality-checking can. </span><span>“Correlation doesn’t even imply </span><a href="http://andrewgelman.com/2014/08/04/correlation-even-imply-correlation/" target="_blank"><span>correlation</span></a><span>” beyond your data. Always ask: Why would this pattern generalize? By what causal process(es)?</span></p> <p dir="ltr"><span>15. Basic retraining must emphasize representativeness and causal stability. Neither bigger </span><a href="http://bigthink.com/errors-we-live-by/is-all-the-truth-we-need-in-the-data" target="_blank"><span>samples</span></a><span>, nor randomization necessarily ensure representativeness (see, mixed-type stats </span><a href="http://bigthink.com/errors-we-live-by/what-is-the-cause-of-soup-stats-blender-errors" target="_blank"><span>woes</span></a><span>, pattern </span><a href="http://bit.ly/24OvTgk" target="_blank"><span>types</span></a><span>).</span></p> <p dir="ltr"><span>16. Journalism that showcases every sensational-seeming study ill-serves us. Most unconfirmed science should go unreported—media exaggerations damage public </span><a href="https://amp.theguardian.com/science/2017/jun/28/exaggerations-threaten-public-trust-in-science-leading-statistician-david-spiegelhalter" target="_blank"><span>trust</span></a><span>. </span></p> <p dir="ltr"><span>17. Beyond avoidable deaths, and burned dollars, there’s a substantial “social cost of junk </span><a href="http://andrewgelman.com/2017/05/28/things-dont-really-need-social-cost-junk-science/" target="_blank"><span>science</span></a><span>” (e.g., enabling the science is </span><span>“</span><a href="http://pllqt.it/PVStZw" target="_blank"><span>bogus</span></a><span>” deniers).</span></p> <p dir="ltr"><span>18. Great science is occurring, but the “free play of free </span><a href="http://www.thenewatlantis.com/publications/saving-science" target="_blank"><span>intellects</span></a><span>” game, fun though it is, is far from free of unforced errors. </span></p> <p dir="ltr"><span>19. “Saving </span><a href="http://www.thenewatlantis.com/publications/saving-science" target="_blank"><span>science</span></a><span>” (Daniel Sarewitz) means fixing the game—not scoring points within it.</span></p> <p><span> </span></p> <p><em>Illustration by <a href="http://juliasuits.net/" target="_blank">Julia Suits</a>, </em>The New Yorker<em> cartoonist & author of </em>The Extraordinary Catalog of Peculiar Inventions</p>
Keep reading
Show less