Compute the PSIS LOO CV - a measure of model fit - of a rater fit object.

Source: R/model_comparison.R

Compute the PSIS LOO CV - a measure of model fit - of a rater fit object.

# S3 method for rater_fit
loo(x, ..., cores = getOption("mc.cores", 1))

Arguments

x

A rater_fit object. All model types are currently supported except the basic Dawid-Skene model fit with grouped data.

...

Other arguments passed.

cores

The number of cores to use when calling the underlying functions. By default the value of the mc.cores option.

Value

A loo object.

Details

This function is somewhat experimental; model comparison is always difficult and choosing between variants of the Dawid-Skene model should be largely guided by considerations of data size and what is known about the characteristics of the raters. loo is, however, one of the leading methods for Bayesian model comparison and should provide a helpful guide in many situations.

When calculating loo we always use the relative effective sample size, calculated using loo::relaive_eff to improve the estimates of the PSIS effective sample sizes and Monte Carlo error.

For further information about the details of loo and PSIS please consult the provided references.

References

Vehtari, A., Gelman, A., and Gabry, J. (2017a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413--1432. doi:10.1007/s11222-016-9696-4 (journal version, preprint arXiv:1507.04544).

Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2019). Pareto smoothed importance sampling. preprint arXiv:1507.02646

Examples


# \donttest{
fit_ds <- rater(anesthesia, "dawid_skene", verbose = FALSE, chains = 1)
fit_ccds <- rater(anesthesia, "class_conditional_dawid_skene",
                  verbose = FALSE, chains = 1)

loo_ds <- loo(fit_ds)
#> Warning: Some Pareto k diagnostic values are slightly high. See help('pareto-k-diagnostic') for details.
loo_ccds <- loo(fit_ccds)
#> Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.

# To compare the loos easily we can use the loo_compare function from the
# loo package:
library(loo)
#> This is loo version 2.6.0
#> - Online documentation and vignettes at mc-stan.org/loo
#> - As of v2.0.0 loo defaults to 1 core but we recommend using as many as possible. Use the 'cores' argument or set options(mc.cores = NUM_CORES) for an entire session. 

loo_compare(loo_ds, loo_ccds)
#>        elpd_diff se_diff
#> model1   0.0       0.0  
#> model2 -11.7       3.2  

# The documentation of the loo package contains more information about how
# the output should be interpreted.
# }