New IES report: Guidelines for educational research

Here's the abstract from a new set of guidelines (Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations) on dealing with educational research that attempts to address multiple hypotheses ( by Peter Z. Schochet).

This report presents guidelines for addressing the multiple comparisons problem in impact evaluations in the education area. The problem occurs due to the large number of hypothesis tests that are typically conducted across outcomes and subgroups in these studies, which can lead to spurious statistically significant impact findings. The guidelines, which balance type I and type II errors, involve specifying confirmatory and exploratory analyses in the study protocols, structuring the data by delineating outcome domains, conducting t-tests on composite domain
outcomes, and applying multiplicity correction procedures to composites across domains. Guidelines are discussed for subgroup analyses, designs with multiple treatment groups, power analyses, and reporting impact findings. The report also provides background for applying the guidelines, including a detailed discussion of the multiplicity problem, statistical solutions that are found in the literature, and weighting options for forming composite domain outcomes.

You can find the full report here: http://ies.ed.gov/ncee/pubs/20084018.asp