Comparing Propensity Score/Flexible Modeling Approaches for Hierarchical Data with Complex Sampling

Propensity score analysis, a technique used to control bias in the estimated treatment effect of observational studies, is popular in the biological sciences, economics, and the social sciences. In a review of the literature, however, many Programme for International Student Assessment (PISA) studies using this technique suffered from moderate to severe methodological shortcomings, including misspecification of the covariates to include in the propensity score model (e.g., interaction terms or polynomials), an inadequate handling of plausible values and missing data, and a failure to address the complexities associated with data originating from a nested, multistage sampling design. In sum, the studies failed to adequately account for hidden bias in observational studies (Rosenbaum, 2010). The purpose of this study is three-fold. First, this study proposes a "best practices" for analyzing PISA data where the goal is to remove treatment effect bias caused by observed and hidden bias. To this end, several propensity score techniques will be compared, including flexible non-parametric modeling approaches. A simulation study incorporating the final student weights specified in the PISA manuals and technical reports (OECD, 2009; OECD, 2012) will be used as a basis to compare bias across the different methods. Secondly, this study will investigate what is the best method for handling missing data and what is the preferred technique for estimating the standard errors of the average treatment effect (ATE). Thirdly, this study will employ propensity score techniques to compare mathematics performance among public, private, and religiously-affiliated private schools in the U.S. and Canada to study the cause effect of type of schooling on academic achievement. Findings from this study will contribute to the literature on propensity score analysis in the context of large-scale assessment with complex sampling.