Most scientific papers are probably wrong

[Ace Pace]

Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.

John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.

"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.

In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.
Massaged conclusions

Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.

Odds get even worse for studies that are too small, studies that find small effects (for example, a drug that works for only 10% of patients), or studies where the protocol and endpoints are poorly defined, allowing researchers to massage their conclusions after the fact.

Surprisingly, Ioannidis says another predictor of false findings is if a field is "hot", with many teams feeling pressure to beat the others to statistically significant findings.

But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research.

"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.

Journal reference: Public Library of Science Medicine (DOI: 10.1371/journal.pmed.0020124)

[DPDarkPrimus]

I'm just going out on a limb and say he's including "scientific" studies done for commercial gain... and not simply actual for-science science?

[kheegster]

It's fantastic how they generalise to '50% of all scientific papers are wrong' from what seems to be a study of medical/life science papers.

News flash: some sciences actually have (gasp!) analytical or experimental methods to verify their validity, and do not actually have to rely on human subjectivity for their results. If 50% of semiconductor research is wrong, he probably wouldn't have a computer to tar things with his brush.

[AdmiralKanos]

It's cited from PubMed. It's obviously based on medical studies, and we've all known for a long time that medical studies are heavily influenced by nonscientific factors and poor controls. Hell, the fact that they're generally paid for by the company which stands to lose millions of dollars if they don't produce the desired results is rather damning evidence against their integrity already.

[Son of the Suns]

When the ability to make publishable results will little money and/or time is a factor in most research, the results is exagerated or completely false claims.

[kheegster]

I'm just extremely pissed off that they weren't more specific about the nature of the research they were referring to. Medical research is a soft science that is heavily subject to non-scientific factors, as Wong had already pointed out.

Hard science is considerably more rigorous, yet by talking about 'all scientific papers' in the arficle, I can just see fundies pouncing it on it as 'evidence' that science is bunkum.

[FireNexus]

So, wait... Does this count as a scientific study that's wrong?

[Crown]

It's cited from PubMed. It's obviously based on medical studies, and we've all known for a long time that medical studies are heavily influenced by nonscientific factors and poor controls. Hell, the fact that they're generally paid for by the company which stands to lose millions of dollars if they don't produce the desired results is rather damning evidence against their integrity already.

Add to the fact that he is specifically talking about press releases before validation (replication) I think it's pretty clear; don't believe all the immediate 'first scoops' presented in the media. Wait for validation.

At least, that's the message I got.

I'm just extremely pissed off that they weren't more specific about the nature of the research they were referring to. Medical research is a soft science that is heavily subject to non-scientific factors, as Wong had already pointed out.

Hard science is considerably more rigorous, yet by talking about 'all scientific papers' in the arficle, I can just see fundies pouncing it on it as 'evidence' that science is bunkum.

Dude, relax. The guy making the claim is a medical practitionor, as is the other guy quoted, what other branch do you think they're talking about?

So, wait... Does this count as a scientific study that's wrong?

[Admiral Valdemar]

PubMed has quite a lot (a shitload) of scientific papers that most certainly do require rigorous scientific analysis. You don't test the effects of biofilms on hip joint implants based on wishy-washy reasoning, you have to actually think about it, same with drugs and other conditions. That doesn't mean there aren't a lot of papers that are meant to stir up the water a bit and get people thinking and which don't have any real concrete experimentation in them either due to the nature of the problem or other constraints.

You'll find most journals will publish papers that are soon refuted as being totally wrong. Guess what? That's the scientific process. The whole idea is that journals are a forum for specialists to overlook these proposals and see whether they will revolutionise the subject or should be binned. Of course there will be wrong proposals, as if scientists, especially medical technicians, are exempt from being fallible. I fail to see what this guy is going on about other than painting a somewhat false image of what is really going on. I can see people jumping on this and saying "Well, looky here, you trust science folks when they get things wrong half the time?". The advances in science and engineering over the last two centuries easily show something is being done right.

[Son of the Suns]

So, wait... Does this count as a scientific study that's wrong?

Zing* a cookie for the funny man.

[tharkûn]

Hell, the fact that they're generally paid for by the company which stands to lose millions of dollars if they don't produce the desired results is rather damning evidence against their integrity already.

The company stands to lose billions of dollars if they fake the data and the inevitable class action lawsuit comes along. Amoral, greedy corporate suits want good science, backing something unsafe will eventually burn you far worse than dumping the product now.

I'm just extremely pissed off that they weren't more specific about the nature of the research they were referring to. Medical research is a soft science that is heavily subject to non-scientific factors, as Wong had already pointed out.

You can do hard medicine. If I publish the tertiary structure of a protein (useful for all kinds of medicine) I've simply done a hard science problem on a soft target. I use the same machines (NMR or X-Ray) that a non-biological scientists would use, get the same type of readings, and run the same mathmatical processing. The problem comes in what you are working with statistical methods and the scientists don't have good setups/techniques.

Hard science is considerably more rigorous, yet by talking about 'all scientific papers' in the arficle, I can just see fundies pouncing it on it as 'evidence' that science is bunkum.

No it isn't. Hard science deals with problems easier to quantify and test. When you publish bad medicine it gets bitchslapped - either by another researcher or when it gets applied to strict double-blinds. The vetting process in medical science continues long after hard science calls it a day, as it should be. The problem comes from studying problems that you can't reduce to simplicity like in hard science. It is much easier to tell if I got an energy spike at this many eV than if 10% of my patient population responded better with the drug than placebo.

I fail to see what this guy is going on about other than painting a somewhat false image of what is really going on. I can see people jumping on this and saying "Well, looky here, you trust science folks when they get things wrong half the time?". The advances in science and engineering over the last two centuries easily show something is being done right.

Bad stastistics. Far, far too many medical researchers have phenomenal training in medicine, but piss all for understanding good stastistics. I can't begin to count the number of times I've seen scattershot correlation searches published when a cursory examination of statistics lays odds on the fact that their "discovery" is nothing more than looking for too many things in too small a population.

[drachefly]

The company stands to lose billions of dollars if they fake the data and the inevitable class action lawsuit comes along. Amoral, greedy corporate suits want good science, backing something unsafe will eventually burn you far worse than dumping the product now.

Unless they can be shielded from lawsuits by corrupt (or confused) laws passed by legislators in the pockets of the drug companies. This possibility is tantalizing, and some drug companies rely on it when their wish-based results are dashed.

Bad stastistics. Far, far too many medical researchers have phenomenal training in medicine, but piss all for understanding good stastistics. I can't begin to count the number of times I've seen scattershot correlation searches published when a cursory examination of statistics lays odds on the fact that their "discovery" is nothing more than looking for too many things in too small a population.

Exactly! Before going to graduate school, I worked at a biological research center. Those biologists were mostly horrible at statistics, with very few exceptions. They would use a P-value very naively, they didn't compound probabilities correctly (needed to review 7th grade math)... I could go on for a while. I'm sure that this is the source of most of the trouble here -- it's very hard to get a truly statistically significant sample when you're working with living things, which are very noisy.

[Wicked Pilot]

Well this is why we have peer reviewed scientific journals in the first place.

[drachefly]

When an entire field of science is ignorant of statistics, and the older generation is the more ignorant, it takes a while for the requirement for good statistics to seep in.

[tharkûn]

Unless they can be shielded from lawsuits by corrupt (or confused) laws passed by legislators in the pockets of the drug companies. This possibility is tantalizing, and some drug companies rely on it when their wish-based results are dashed.

That only covers your ass in the US, and even then is extremely expensive. If you seriously fake your data these days it will be found out and your European subsidaries will be hammered if nothing else.

The thing is drugs are continiously monitored. After the pills hit the market statistics continue to be tabulated, side effects recorded, etc. Within a short period of time you will have a dataset including millions of patients (for the really big money makers). At that point any handwavium employed is going to be shown to be fradulent because the data is three orders of magnitude better.

[Admiral Valdemar]

Bad stastistics. Far, far too many medical researchers have phenomenal training in medicine, but piss all for understanding good stastistics. I can't begin to count the number of times I've seen scattershot correlation searches published when a cursory examination of statistics lays odds on the fact that their "discovery" is nothing more than looking for too many things in too small a population.

I can vouch for that. While medicine is not a "soft science" (who the hell conjured up that bullshit view?), it does have a lot more people working in the field that won't be using standard techniques normal lab techies will use. I knew a fair few medical students at my uni and they were taught the same statistical protocols as me, but they then went and focused on the more usual medical aspects of the course such as diagnosis of conditions, drug categories and effects, physiological classifications etc. Sure, it was all science, but not something that you really need to publish a paper for peer-reviewed journals. It'll make you a great medic, it just won't create the R&D type scientists you come to think of when reading work in Nature or Science.

What I'd like more than a better focus on this stuff for pure med students is for every Layman out there to learn the scientific method over until they never forget it. That way you'd see a lot less pseudo-scientific bullshit out there and far more rational minds.

Exactly! Before going to graduate school, I worked at a biological research center. Those biologists were mostly horrible at statistics, with very few exceptions. They would use a P-value very naively, they didn't compound probabilities correctly (needed to review 7th grade math)... I could go on for a while. I'm sure that this is the source of most of the trouble here -- it's very hard to get a truly statistically significant sample when you're working with living things, which are very noisy.

While it is true biologists typically went into biology to avoid the maths of physics and the formulae of chemistry, many are very good at statistics. Modern statistics was created by biologists because the classical system was inadequate, as you say, to use with what biology deals with. It's funny that many are hounding the physics community to accept the biological form of statistics to solve their current problems because their system is simply flawed and too "old skool". I was never any good at maths, I did, however, make it a mission of mine to do well at the important number crunching section in my course since it's a nice big chunk of anything you do in that science.

[wolveraptor]

He means before peer-review, right? That seems obvious. The scientist who wrote the theory rarely finds out what's wrong with it him/herself. Otherwise, they never would've written it. Hence, peer-review.

[Admiral Valdemar]

Even then, some papers are published and are not going to get universal acceptance. Publishers will try and get as many top scientists as they can to peer-review papers in their free time, but they won't match the whole science community, so it isn't uncommon to get pieces published that are later proven false either immediately or because a new new theory or data set somes in. The last thing you want to do is become Pons & Fleischmann and forget to look over your supposed amazing results.

[wolveraptor]

So the best way to ascertain the veracity of a theory is to see how long it lasts? Then this study is deliberately misleading, as it doesn't tell you that most of the false theories are identified and trashed immediately.

[Darth Wong]

Hell, the fact that they're generally paid for by the company which stands to lose millions of dollars if they don't produce the desired results is rather damning evidence against their integrity already.

The company stands to lose billions of dollars if they fake the data and the inevitable class action lawsuit comes along. Amoral, greedy corporate suits want good science, backing something unsafe will eventually burn you far worse than dumping the product now.

The company does not have to fake data; they only have to use insufficiently rigorous test methods. And this happens all the time, whether you admit it or not.

[ClaysGhost]

It's funny that many are hounding the physics community to accept the biological form of statistics to solve their current problems because their system is simply flawed and too "old skool".

Have you been reading New Scientist again? You might want to look to that.

[Darth Wong]

No one should take New Scientist seriously; they have shown that they have no editorial standards.

[ClaysGhost]

Definitely. The article I remember made a claim that Bayesian statistics were unknown in astronomy; but in the further reading section, they linked to a famous astronomy paper that is packed with Bayesian stats (Spergel et al., the WMAP results). It seems they will let any fool write an article beyond their competence, these days.

[Darth Wong]

By the way, to address tharkun's argument that no pharmaceutical corporation would be stupid enough to risk major lawsuits for negligence, I would like to point out that prior to Enron, he would have undoubtedly argued that no corporation would be stupid enough to risk prosecution for incredibly blatant fraud either. Corporate suits have a history of short-term thinking and arrogant, foolish decisions, and then trying to use their influence to cover up for those decisions. Why do you think there's such a powerful political panic to shut down liability lawsuits even though the health care industry's liability payouts are not rising?

The idea that Vioxx is some kind of isolated incident is sheer bullshit. The list of drugs "tested" and marketed by their manufacturers and then later discovered to have serious life-threatening side effects is long and distinguished: Vioxx, Zyprexa, Crestor, Prempro, Baycol, Accutane, and Rezulin are all fairly recent examples. And Oxycontin is still being gleefully marketed even though it is basically a form of heroin. In the case of Rezulin, it was being marketed in the US even though it was already banned in Britain for safety reasons. So don't tell me that drug companies would never risk public harm for short-sighted short-term profiteering; tharkun's brazen drug industry apologism is wearing thin.

[tharkûn]

The company does not have to fake data; they only have to use insufficiently rigorous test methods. And this happens all the time, whether you admit it or not.

Right a jury is never going to hammer your ass for gross negligence and depraved indifference, both of which are viable charges. Every drug company we've worked for has been quite adamant that their overriding concern is not to be sued; several went so far as to inform us that if they were sued due to deficienies on our part they'd be sueing us.

If it happens "all the time", would you care to provide say three examples in the last five years where companies used tests below the level of rigor required by the FDA (or other applicable regulatory body)?