Friday, April 20, 2018

Vox Popoli: The 11 percent metric (Modern science is actually less reliable than flipping a coin.)

Modern science is actually less reliable than flipping a coin. The Wall Street Journal reports on scientific efforts to address the reproducibility crisis:
Half the results published in peer-reviewed scientific journals are probably wrong. John Ioannidis, now a professor of medicine at Stanford, made headlines with that claim in 2005. Since then, researchers have confirmed his skepticism by trying—and often failing—to reproduce many influential journal articles. Slowly, scientists are internalizing the lessons of this irreproducibility crisis. But what about government, which has been making policy for generations without confirming that the science behind it is valid?

The biggest newsmakers in the crisis have involved psychology. Consider three findings: Striking a “power pose” can improve a person’s hormone balance and increase tolerance for risk. Invoking a negative stereotype, such as by telling black test-takers that an exam measures intelligence, can measurably degrade performance. Playing a sorting game that involves quickly pairing faces (black or white) with bad and good words (“happy” or “death”) can reveal “implicit bias” and predict discrimination.

All three of these results received massive media attention, but independent researchers haven’t been able to reproduce any of them properly. It seems as if there’s no end of “scientific truths” that just aren’t so. For a 2015 article in Science, independent researchers tried to replicate 100 prominent psychology studies and succeeded with only 39% of them.

Further from the spotlight is a lot of equally flawed research that is often more consequential. In 2012 the biotechnology firm Amgen tried to reproduce 53 “landmark” studies in hematology and oncology. The company could only replicate six. Are doctors basing serious decisions about medical treatment on the rest? Consider the financial costs, too. A 2015 study estimated that American researchers spend $28 billion a year on irreproducible preclinical research.

The chief cause of irreproducibility may be that scientists, whether wittingly or not, are fishing fake statistical significance out of noisy data. If a researcher looks long enough, he can turn any fluke correlation into a seemingly positive result. But other factors compound the problem: Scientists can make arbitrary decisions about research techniques, even changing procedures partway through an experiment. They are susceptible to groupthink and aren’t as skeptical of results that fit their biases. Negative results typically go into the file drawer. Exciting new findings are a route to tenure and fame, and there’s little reward for replication studies.

It's always ironic how the IFLS crowd isn't even remotely up to speed on current science while simultaneously pointing and shrieking about how everyone with substantive and valid criticism of scientistry simply "doesn't understand science". You can see this in the comments of the most recent Voxiversity on Christianity and Western Civilization. Richard Dawkins has repeatedly argued that eyewitness testimony should not be used in the courtroom because it is insufficiently reliable, but by his own metric, the expert testimony of a scientist should barred from the courtroom as well because science is considerably less statistically reliable.

As for the idea that science can even theoretically serve as a basis for moral guidance, the grand windmill at which Sam Harris has been jousting in futility for the last 10 years, that has become even more obviously ridiculous than even his most brutal critics believed at the start. One would do nearly four times better to simply flip a coin; indeed, statistically speaking, one's optimal strategy is to listen to what scientists advise, then do precisely the opposite.

Of course, in retrospect, this should have always been obvious. Look at the average scientist. Do you think following his advice on women or doing the precise opposite is more likely to lead to a desirable outcome? Do you trust his philosophy on fitness, or on any other aspect of life? These are individuals whose entire perspectives on life, the universe, and everything are constructed on an illusion of a nonexistent solidity.

And the great irony is that scientistry now stands condemned by its beloved scientodific metric. The New Atheists reasoned that religious faith must be false on the basis of presuming the eyewitness testimony and documentary evidence to the contrary being false, but now we actually know, we do not merely reason, that it is faith in science that is false due to irreproducibility.