The Problems With Meta-Analysis

I had written a more mathematical blog in May 2009, denoting the logical and mathematical/statistical problems with meta-analyses, but since that time many more meta-analyses have been published, and the public has discussed these results as if they were clinical fact. It is important to understand that the results of a meta-analysis should be presented only as a hypothetical clinical result, to be tested forwards in a properly designed clinical format, and not accepted as proven fact (such as the recent suggestion that women who ingest calcium supplements increase their risk of heart disease). In brief, a meta-analysis collects several studies of the same problem, none of which reaches clinical or statistical significance, in the hopes that the sum can be greater than its parts, and that combining non-significant studies can reach a significant result!

Some readily understandable problems with meta-analyses:

1) You are never told which studies the author rejects as not being acceptable for his/her meta-analysis, so you cannot form your own opinion as to the validity of rejecting those particular studies.

2) The problem of the Simpson Paradox, or the Yule-Simpson Effect: sometimes all the included studies point in one direction as being clinically significant, but the meta-analysis points in exactly the opposite direction. This has been discussed, for instance, in "Chance" magazine, published by Dartmouth University, where they showed different ways of calculating Derek Jeter's batting average, with differing results, using the same data in each case.

3) There are two different statistical models or assumptions by which the analyzer combines the effects of the individual studies: the fixed effects model and the random effects model. Each model makes different assumptions about the underlying statistical distribution of observed data, so each calculation produces different results.

4) There are two different methods for measuring the effect of the clinical intervention: standardized mean difference or correlation. Each method produces a different end result.

5) If we look at #3 and #4, we see immediately that there are four possible combinations of analyses, leading to four different conclusions for the same set of studies. No one paper shows all four combinations and all four possible results.

6) Finally, the choice of what constitutes a "significant' effect in any of the included studies is purely arbitrary. When this question was studied by clinical psychologists, no two analytical scientists reached the same conclusions of what was significant in all the included studies.

We therefore see that the result of any meta-analysis is largely dependent on the analyzer, and the reader never has enough data to redo the analysis, so the results have to be taken on faith, which is hardly a scientific result.

"There are three kinds of lies: Lies, Damn Lies, and Statistics" - Mark Twain

George Thomas, M.D., Ph.D.  has a Ph.D. in physics as well as M.D. Dr. Thomas has written publications in both physics and medical journals, is a reviewer for both physics and medical journals, a member of science and medical honor societies, a former physics professor and then medical professor at a medical school. He has been on the editorial board for both physics and medical journals, been an encyclopedia author, worked on government-sponsored research and has acted as a contract reviewer for a number of years, as well as has performed volunteer work with a chronic disease group. Dr. Thomas has been in private practice of family medicine for over 25 years. His practice is located in the New York City region. Dr. George Thomas can be reached at ghthomas3@aol.com.

This blog is also published by George Thomas, M.D., Ph.D. (Physics) at http://www.ghthomas.blogspot.com/.

Dr. Thomas can be reached by e-mail at ghthomas3@aol.com, or by snail mail at P.O. Box 247, Hillsdale, N.Y., 12529

The concepts discussed here are based upon the author's personal professional experiences with patients, or upon his review of the pertinent medical and/or physics literature. Before acting on anything written here, you should discuss it with your personal physician as well as your personal physicist.