Biases, biases everywhere

When I skim through list of cognitive biases , I wonder why papers like Over-optimism in bioinformatics: an illustration (hat tip: Neil) do not appear more often. Here’s the abstract:

MOTIVATION: In statistical bioinformatics research, different optimization mechanisms potentially lead to “over-optimism” in published papers. So far, however, a systematic critical study concerning the various sources underlying this over-optimism is lacking. RESULTS: We present an empirical study on over-optimism using highdimensional classification as example. Specifically, we consider a “promising” new classification algorithm, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. While this approach yields poor results in terms of error rate, we quantitatively demonstrate that it can artificially seem superior to existing approaches if we “fish for significance”. The investigated sources of over-optimism include the optimization of data sets, of settings, of competing methods and, most importantly, of the method’s characteristics. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should always be demonstrated on independent validation data. AVAILABILITY: The R codes and relevant data can be downloaded from http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/overoptimism/, such that the study is completely reproducible.

Emphasis in the last sentence is mine. It’s the first time I saw a statement that research is reproducible, even though publishing in peer-reviewed journal means reviewers made sure it is.

Anyway, I’m looking for more examples of biases in research process. If you have some, please let me know.