Out of sample log-likelihood and accuracy for mirt item response theory models

26 Views Asked by At

I'm trying to compute out of sample accuracy and log-likelihood of fitted IRT models (that is, predictive accuracy and log-likelihood for data not used to fit the model). I'm using the fscores() function to get score estimates, and then probtrace() with the estimated factor scores to get probabilities for accuracy and log-likelihood computation.

I am having some issues of 2-factor multidimensional models performing much worse than expected. I am not sure if there is something wrong with my thinking, if I'm using the package in the wrong way or if there is an issue with the package?

Below is a working example with the LSAT7 data for which I'm using the training data to test my own log-likelihood computation. Why is my log-likelihood for the 2 factor model so much larger than the one retrieved from mirt using logLik()?

library(mirt)

data <- expand.table(LSAT7)
itemstats(data)

mod1 <- mirt(data, 1)
mod2 <- mirt(data, 2)

mirt_measures <- function(dat, mod, start0 = FALSE, inverse_coded = FALSE) {
  thetas <- fscores(mod, response.pattern = dat)
  probs <- get_mirt_probs(thetas[, grepl("^F", colnames(thetas))], mod)
  log_lik <- acc <- 0
  for (item in seq_along(probs)) {
    if (inverse_coded) { # multidimensional model has revers relationship between prob and score
      dat[, item] <- max(dat[, item]) - dat[, item]
    }
    for (person_id in seq_len(nrow(dat))) {
      if (!is.na(dat[person_id, item])) {
        score_ind <- dat[person_id, item] + 1
        log_lik <- log_lik + log(probs[[item]][person_id, score_ind])
        acc <- acc + (which.max(probs[[item]][person_id, ]) == score_ind)
      }
    }
  }
  acc = acc / (nrow(dat) * length(probs))
  return(c("ll" = log_lik, "acc" = acc))
}

get_mirt_probs <- function(theta, mod) {
  probs <- list()
  for (item in 1:extract.mirt(mod, "nitems")) {
    extr <- extract.item(mod, item)
    probs[[item]] <- probtrace(extr, theta)
  }
  probs
}

logLik(mod1)
logLik(mod2)
mirt_measures(data, mod1)
mirt_measures(data, mod2)
0

There are 0 best solutions below