kfold, reloo, and non-gaussian group-level effects

[avehtari]

This worked in January 2024

fit_hiert <- brm(y | trials(n) ~ 1 + (1 | gr(id, dist='student')), family = binomial(),
                 data = dat, refresh=0)

kcv_hiert <- kfold(fit_hiert, K=48, save_fits=TRUE)

but now it gives an error

Error: Option sample_new_levels = 'gaussian' is not available for non-gaussian group-level effects.

Preferably, kfold would be fixed to work non-Gaussian group-level effects.

If non-Gaussian group-level effects are not supported, then it would be nice to clarify that kfold() does not support non-Gaussian group-level effects. Now the error message refers to argument sample_new_levels which is not mentioned in the kfold() documentation and it's not clear where to find possible instructions to go forward.

The same for reloo

fit_hiert <- add_criterion(fit_hiert, criterion='loo', reloo=TRUE, overwrite=TRUE)

[paul-buerkner]

What happens if you pass sample_new_levels = "uncertainty" to kfold? Does that work?

[avehtari]

What happens if you pass sample_new_levels = "uncertainty" to kfold? Does that work?

Sorry for not mentioning that it seems to work. My issue was that it is not mentioned in kfold() and add_criterion() documentation, and naturally even better would be to switch to sample_new_levels = "uncertainty" automatically if the group-level effects are non-Gaussian.

[paul-buerkner]

Yeah, especially the documentation would make sense to add. Will do it soon.