The hard but necessary task of gathering order-one effect size indices in meta-analysis

Abstract

Meta-analysis of studies with two groups and two measurement occasions must employ order-one effect size indices to represent study outcomes. Especially with non-random assignment, non-equivalent control group designs, a statistical analysis restricted to post-treatment scores can lead to severely biased conclusions. The 109 primary studies included in 4 metaanalyses were recovered, and their authors were contacted to request the raw data to calculate the order-one effect size indices. From this total we only got 13 primary studies. The results with the raw data analysis were compared with those performed with the order-zero and order-one indices. Despite the difficulties for gathering the data, the few data sets analyzed show that if the meta-analysis is performed with order-zero indices, the results can be severely misleading