Estimating and evaluating
diagnostic models

With the R package measr

W. Jake Thompson, Ph.D.

What are diagnositc models

• DCMs are psychometric models for the purpose of classification

• Fine-grained, multidimensional reporting

• Results that are instructionally useful for teachers, parents, and students (Thompson & Clark, 2024)

DCMs with Stan

• Stan is free and open-source
• Existing documentation for implementing DCMs
  • Case study using the DINA model
  • Paper describing LCDM implmentation
• Access to other packages in the Stan ecosystem
  • loo, tidybayes, posterior
• Need an interface with R
  • Automate script generation
  • Facilitate model evaluations

Data: Pathways for Instructionally Embedded Assessment (PIE)

• 15 items, measuring 3 attributes
  • Level 1, 2, and 3 skills in a grade 5 learning pathway
  • Skills follow a linear hierarchy: Level 1 -> Level 2 -> Level 3
  • Described by ATLAS (2025)
• Available in the dcmdata package

# remotes::install_github("r-dcm/dcmdata")
library(dcmdata)

?pie

Specify a DCM

• Choose from a wide variety of measurement models (e.g., LCDM, DINA, C-RUM)

• Define attribute relationships through the structural model (e.g. HDCM, Bayesian Network)

• Set prior distributions

pie_hdcm_spec <- dcm_specify(
  qmatrix = pie_ft_qmatrix,
  identifier = "task",
  measurement_model = lcdm(),
  structural_model = hdcm("L1 -> L2 -> L3")
)

Estimate a DCM

• Wraps Stan via rstan or cmdstanr packages

• Multiple estimatiom methods supported: MCMC, variational inference, or optimization

pie_hdcm <- dcm_estimate(
  dcm_spec = pie_hdcm_spec,
  data = pie_ft_data,
  identifier = "student",
  method = "mcmc",
  backend = "cmdstanr",
  chains = 4,
  iter_warmup = 5000,
  iter_sampling = 500,
  parallel_chains = 4,
  adapt_delta = .99,
  file = "fits/pie-model"
)

Check model fit

• Posterior predictive model checks (PPMCs) to evaluate the fit of model to data

• PPMCs available at the model and item level

Evaluate reliability

• Multiple types of reliability indices
  • Profile-level classification
  • Attribute-level classification
  • Probabilities

measr_extract(pie_hdcm, "classification_reliability")
#> # A tibble: 3 × 3
#>   attribute accuracy consistency
#>   <chr>        <dbl>       <dbl>
#> 1 L1           0.964       0.994
#> 2 L2           0.919       0.977
#> 3 L3           0.930       0.877

Compare models

• Model comparisons with leave-one-out cross validation (LOO) with the loo package

• Equal fit between our two models indicates that our HDCM with the enforced hierarchy among the learning pathway levels is supported

loo_compare(
  pie_lcdm,
  pie_hdcm,
  model_names = c("LCDM", "HDCM")
)
#>      elpd_diff se_diff
#> HDCM 0.0       0.0    
#> LCDM 0.0       2.8

Learn more: r-dcm.org

Slides

 wjakethompson.com
 wjakethompson@ku.edu
 0000-0001-7339-0300
 in/wjakethompson
 @wjakethompson
 @wjakethompson.com
 @wjakethompson@fosstodon.org
 @wjakethompson
 @wjakethompson

Acknowledgements

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305D210045 and R305D240032 to the University of Kansas Center for Research, Inc., ATLAS. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.