Statistical Approaches to Missing Data in High-Stakes Evaluations

A test score that was never recorded is just an empty cell in a spreadsheet. But in a licensing exam, a clinical trial, or a national school assessment, empty cells can change who passes, which treatment gets approved, and how a country judges its schools.

Sterne and colleagues (2009) call missing data a persistent problem in medical research. This is actually a problem in all fields, but its effect on findings has often been underestimated. The basic problem is this: when some values are absent, the remaining observations may not represent the full population. If dropouts differ from stayers, analyses based only on available cases can mislead.

This article describes how statisticians handle missing data, why some traditional shortcuts fail, and which modern methods help researchers defend their decisions. Information here is drawn from peer-reviewed statistical literature and official assessment documentation.

Donald Rubin (1976) gave the field a useful framework. Three categories matter: missing completely at random, missing at random, and missing not at random.

The method choice partly depends on which mechanism is plausible.

Rubin (1976) showed that under certain conditions the missingness mechanism can be ignored for inference about parameters of interest. Those conditions require a defensible MAR assumption, an ignorable mechanism, and distinct parameters for the data model and the missingness process.

Schafer and Graham (2002) note that MAR is often misunderstood. Observed data alone usually cannot tell us whether missingness is MAR or MNAR. Researchers have to justify the assumption using subject-matter knowledge, study design, auxiliary information, and sensitivity analysis.

A hypothetical licensing body tests 500 candidates on a practical exam scored out of 100. Four hundred sit the exam; 100 do not.

Suppose the observed candidates average 74. For illustration, assume the 100 absent candidates would have averaged 55.

Group Candidates Average practical score

Sat the practical 400 74.0

Did not sit 100 55.0 (assumed)

Everyone 500 70.2

The overall mean would be:

xˉ=400(74)+100(55)500=70.2xˉ=500400(74)+100(55)=70.2

If the licensing body uses only the 400 observed scores, it reports 74. That is 3.8 points above the illustrative overall mean. The arithmetic is fine, but the overall mean rests on an assumption about scores nobody observed. In a real exam, we would not know the absent candidates’ average without more information or a defensible model.

The point is straightforward: if missing cases differ systematically from observed cases, analysing only observed cases can give a false picture of performance.

Schafer and Graham (2002) reviewed several traditional approaches. Two deserve attention.

Listwise deletion, or complete-case analysis, drops every observation with missing data on any variable used in the analysis.

It is easy to run. It can also throw away useful information, reduce power, and introduce bias.

Complete-case analysis can be unbiased when data are MCAR. But MCAR is not necessary for every complete-case analysis to be valid. Depending on the model, the estimand, and the missingness mechanism, some regression estimates can remain valid under weaker conditions.

So the question is not whether complete-case analysis is always bad. It is whether its assumptions hold for the research question at hand.

Mean substitution replaces each missing value with the variable’s observed mean. The usual version keeps the variable mean unchanged. It also shrinks variability and ignores uncertainty about the missing observations. Standard errors, confidence intervals, and tests can then mislead.

Sterne et al. (2009) warn against treating complete-case analysis as automatically adequate and against failing to report how much data are missing and how they were handled.

Neither shortcut should be chosen just because it is convenient. The method has to fit the question, the available information, and the assumptions the researcher can defend.

Maximum likelihood estimates the parameter values that make the observed data most likely under a specified probability model.

With incomplete data, researchers can often estimate parameters from the observed-data likelihood without filling in every missing value. Under a suitable model and missingness assumptions, this can use the available information better than deletion.

The expectation-maximization (EM) algorithm (Dempster, Laird, and Rubin, 1977) is one way to do maximum-likelihood estimation with missing data. It alternates two steps:

The algorithm repeats until convergence. Under standard conditions, the observed-data likelihood does not decrease from one iteration to the next. That does not guarantee a global maximum, so convergence needs proper checking.

EM usually does not plug in one guessed value for each missing observation and then treat it as real. The E-step carries uncertainty about the missing information.

Maximum likelihood and EM are related but not the same. Maximum likelihood is an estimation principle; EM is an algorithm for some estimation problems.

Schafer and Graham (2002) treat maximum likelihood and Bayesian multiple imputation as important methods for incomplete data. Graham (2009) gives practical advice, including repeated measures, grouped data, and many-variable settings.

Multiple imputation takes a different route. It does not fill each gap once and pretend the value is known. It builds several plausible completed datasets.

Each imputed value comes from a predictive distribution based on observed information and the imputation model. This carries uncertainty about the missing values and, when appropriate, uncertainty about model parameters.

The usual process has three stages:

The last stage matters. Differences across imputed datasets show how missing-data uncertainty affects conclusions.

mice is a widely used R package for multiple imputation by chained equations. Van Buuren and Groothuis-Oudshoorn (2011) describe the package and its statistical basis.

Each variable with missing data is imputed using a model suited to that variable and its available predictors. A continuous outcome might use linear regression; a binary variable might use logistic regression. The chained-equations procedure cycles through the variables, updates their missing values, and produces completed datasets for analysis and pooling.

Multiple imputation is not automatically reliable because it produces several datasets.

Sterne et al. (2009) stress correct imputation models. If a model misses important relationships or data features, estimates can be biased.

Allison (2000) shows that imputation algorithms can behave differently under different data-generating processes. That is a warning against assuming every procedure works equally well everywhere. It is not evidence that multiple imputation is generally ineffective.

Graham (2009) emphasizes auxiliary variables: observed variables that help predict missingness or the missing values themselves, even if they are not central to the study. Good auxiliary variables can improve power and reduce bias, but they should be chosen from study design, substantive knowledge, and observed relationships.

The imputation model should also preserve important data features: interactions, nonlinear relationships, clustering, and the structure of the final analysis model.

Some educational assessments face a different problem. Individual answers may not be “missing” in the usual sense; instead, each student answers only a subset of items, and researchers want to estimate underlying proficiency.

The U.S. National Assessment of Educational Progress (NAEP) uses plausible values for population and subgroup estimates.

Plausible values come from a model that combines item responses, background information, and the assessment’s population and sampling design. They are draws from a model-based distribution of plausible proficiency values, not ordinary replacements for unanswered questions.

Mislevy (1991) developed a randomization-based framework for inference about latent variables from complex samples, which provides relevant statistical foundations.

Plausible values are not precise individual test scores. They support group-level estimates and proficiency-related relationships. When analysts use multiple plausible values, they compute the statistic for each plausible-value dataset and combine results using the appropriate rules. 

The number of plausible values and the analysis rules can vary by assessment programme and cycle. Analysts should follow the technical documentation for the specific NAEP assessment rather than assuming one historical implementation applies to all.

The lesson for high-stakes evaluation: when the quantity of interest is latent or not precisely measurable for each person, the analysis must reflect that uncertainty.

Maximum-likelihood methods and standard multiple imputation often rest on a defensible MAR assumption when the missingness mechanism is ignored.

The trouble is that observed data alone usually cannot establish MAR. In high-stakes settings, candidates who skip an assessment, patients who leave a trial, or households that do not return a survey may differ from those who stay.

Schafer and Graham (2002) review methods for nonignorable missingness. Graham (2009) gives practical guidance on attrition and missing-data analysis.

Also run sensitivity analyses under plausible MNAR scenarios. For example, ask how conclusions change if missing exam scores are systematically lower than MAR would predict, or if dropouts have worse outcomes than similar patients who stay.

The goal is not to prove one missing-data assumption correct. It is to see whether key findings and decisions hold up under alternative, scientifically credible assumptions. If conclusions change substantially, report that uncertainty and consider it before making a high-stakes decision.

Researchers and statisticians can use this checklist when designing or analysing a study.

Missing data are not just empty cells. They can affect estimates, confidence intervals, policy evaluations, clinical decisions, and fairness in high-stakes assessments.

The right method depends on the research question, data structure, missingness mechanism, and assumptions that can be defended. Maximum likelihood and multiple imputation are powerful, but they still require careful modelling and sensitivity analysis.

The goal is not to fill every gap. It is to use what is available well and to be honest about what remains unknown. In high-stakes evaluation, transparent reporting and defensible assumptions are essential if conclusions are to be reliable, uncertainty understood, and decisions based on the best evidence the data can provide.