Bias, by the numbers
Debates on policy, whether they relate to the safety and effectiveness of medicines or class size in schools, often rely on these reviews. And they can be invaluable.
The Cochrane Collaboration is among the best known international organisations providing these reviews in healthcare. In his popular book Bad Science, Dr Ben Goldacre claims that meta-analyses like these have "saved the lives of more people than you will ever meet" (p. 54). A thorough literature review has the potential to make revelations that years of research has overlooked.
But these reviews are prone to bias, according to University of Leicester statistician Alex Sutton. That’s because research with positive findings is more likely to be published than work that is inconclusive or disproves a hypothesis. Which is unfortunately unlikely to change.
But Professor Sutton and his colleagues in our Department of Health Sciences are working on a different solution: developing statistical tools for identifying and quantifying bias, which he demonstrated in his recent Professorial Inaugural Lecture entitled Analysing the data you haven’t got.
So it’s all in the numbers; using these tools (if you must know, Professor Sutton uses novel contour enhanced funnel plots and a regression based adjustment method) literature reviews can compensate for missing results and can paint a more accurate picture of the research out there. Which can only be a good thing, because evidence-based policymaking is becoming more common in all walks of life.