Episode Transcript
Interviewer: We'll talk about irreproducibility in science, up next on The Scope.
Announcer: Examining the latest research and telling you about the latest breakthroughs, the Science and Research Show is on The Scope.
Interviewer: I am talking with Dr. Tom Parks, Vice President for Research at the University of Utah.
Irreproducibility in science, these are scientific discoveries made by one group that can't be replicated by another. There has been a lot of discussion around this lately. Is this as big of a problem as people are making it out to be?
Dr. Parks: I do think it's a serious problem. There has been developing literature over the past ten years that I've been aware of, and I assume some of it is probably known earlier than that, that there are major flaws in the way research in certain fields, particularly biomedical science and social behavioral science, that cause a lot of published results to not be replicable or sufficiently reliable.
Interviewer: And what's the scale of the problem, do you think? Is it widespread?
Dr. Parks: Estimates are that as much as half or more of published research is not replicable, and a recent study estimated that $28 billion a year of federally-funded research is, you could say, wasted on research that cannot be replicated.
Interviewer: Those are big numbers.
Dr. Parks: Those are big numbers
Interviewer: And it could be undermining a trust in science by the general public, do you think?
Dr. Parks: I know it is. I mean, one of the things that has surprised me is that I've been following this for ten years and yet I find that many scientists are not aware of this issue, yet there have been pretty prominent articles in general interest magazines like The Economist, The New Yorker, The Atlantic, The Wall Street Journal and so forth that have laid out these issues quite clearly. So a lot of educated general public is aware of this, legislators are aware of it, funding agencies are aware of it, journals are aware of it, but many practicing scientists are not aware of this problem.
Interviewer: And that could be a contributing factor, I guess.
Dr. Parks: I think so, but I think part of the reason, either people are not aware or they're in denial about it because the consequences for scientists of changing, of making research more replicable will be significant.
Interviewer: And what has brought us to this point? What are some of the causes, do you think?
Dr. Parks: There are a number of them. In method sections we don't provide enough information to make a study easily replicated. We don't make available all of the data from results. We don't restrict ourselves to testing our priori hypotheses on an experiment where people often go in and look post hoc after the experiment is done and find significant effects, and the statistical power that is the sample size of many studies, a shockingly large number of studies have statistical power that is too low to reliably generate reliable results.
Interviewer: So a lot of problems, a lot of causes.
Dr. Parks: Yes, and I should also mention that we don't publish negative results. We don't register every study. I mean, we don't know how many studies have been done in a particular field because only the positive ones are ever published.
Interviewer: No, you used to run a lab. Were you ever in a position where you couldn't replicate someone else's results? Dr. Parks: I can't think of an example where we did that, but I think that is part of the problem. Individual investigators are rewarded for publishing new and original innovative findings and ideas, and there's really no reward for replicating previous studies. That is a big part of the problem with replicability.
Interviewer: So how do you think we could even overcome this problem?
Dr. Parks: In clinical trials fields or the consequences of poor replicability or reliability have consequences for human patients. They've sort of moved ahead with this. Trials have to be registered so that people know when a trial is underway, and then federally-funded trials have to report all of their results, whether positive or negative. I think journals are coming to expect more detailed methodology and eventually, I believe, federally-funded research will be required to have all of the data posted online and made available to anyone who wants to reanalyze it or replicate it.
I think journals will begin requiring larger sample sizes, more statistical power in studies. There are other more subtle issues with the specifications of reagents and the consistency of reagents and analytical methods and so forth. There's a lot of contributors to poor replicability but I've mentioned the ones that I think are the most significant.
Interviewer: So those are kind of sweeping changes. I mean, how likely do you think it is that those will happen?
Dr. Parks: We already see politicians on both left and right arguing with scientific evidence when it doesn't suit their ideological stance, and this will give them another way to argue against evidence that they don't like. I think the funding agencies, to maintain the funding, will have to insist on some of the changes that I just mentioned, and I think the journals, they're the ones who are publishing research that can't be replicated and I think they will eventually put in higher standards for getting papers published.
Interviewer: Does knowing about this issue taint how you look at new research that's coming out? If there's some new exciting discovery on the cover of Nature, do you sort of take it with a grain of salt?
Dr. Parks: Many years ago a grizzled old senior scientist said in a talk I went to, "If it's true it isn't new, and if it's new it isn't true," and at the time I thought that was a cynical old person's thing to say. But that's more or less my position now. As lots of people remark, cancer has been cured in mice thousands of times for every new therapy that gets to people. I think people should be very skeptical
Scientists by nature are very skeptical. We haven't been sufficiently skeptical about our own work until now, but I think it's coming.
Announcer: Interesting, informative, and all in the name of better health, this is the Scope Health Sciences Radio.