Microarray-based clinical research has generated both unrealistic hype and excessive skepticism. We reviewed 90 microarray studies in which gene expression data are analyzed for relationships with cancer outcomes published through the end of 2004 (A Dupuy, RM Simon. J Natl Cancer Inst 2007;99:147-57). The statistical analysis was examined in detail for the 42 articles that were published in 2004. Statistical analyses fell into three distinct types: outcome-related gene finding, class discovery, and supervised prediction. We focused on serious problems, ones that can result in erroneous or misleading conclusions.
For outcome-related gene finding, the most common and serious flaw was an inadequate, unclear, or unstated method for controlling the number of false-positive claims of differentially expressed genes. For class discovery, the most common and serious flaw was a spurious claim that the expression clusters were meaningful for distinguishing different outcomes, when the clustering itself was based on genes selected for their correlation with outcomes. For supervised prediction, the most common serious flaw was invalid use of partial cross-validation, using the full-dataset to select the informative genes rather than performing gene selection from scratch for each loop of the cross-validation. At least one of these three major flaws was present in 50% of the 2004 publications and these flaws were present frequently in papers published in the highest profile journals.
We proposed guidelines for statistical analysis and reporting, based on the most common and serious problems identified. Such guidelines are critically needed as microarray studies are a fast-growing area for both basic and clinical research with an exponentially growing number of publications. As demonstrated by our results, common mistakes are pervasive in studies published in good-quality, peer-reviewed journals. To make the guidelines as practical as possible, they are presented as a checklist of “Do’s and Don’ts.”