Forget pretty much everything you have heard in statistics classes about using statistical inference to interpret whether an experimental effect is ‘real’ or not. Want to know if an effect is real? Replicate, replicate, replicate.
Only if an effect replicates—preferably in a different lab, by different researchers—can the scientific community start to believe that the effect is real.
Replication is thus the most important statistic. Period.
But, psychology as a discipline has notoriously eschewed publishing failed replication attempts. So it’s great to see that the topic of replicability in psychological science has been given center-stage in a special issue of the high-impact journal “Perspectives on Psychological Science”: http://pps.sagepub.com/content/7/6.toc?etoc
There are many exciting articles in this special issue—not least an article that I contributed to with the Open Science Collaboration (http://pps.sagepub.com/content/7/6/657.full.pdf+html), which outlines the OSC’s project aiming to assess the proportion of studies in psychology that are replicable.
Given the importance of this issue to all psychologists (especially given the ‘year of horrors’ that faced psychology as a science last year – http://www.ejwagenmakers.com/2012/Wagenmakers2012Horrors.pdf) I hope all readers find this of interest.