Crossvalidation and bootstrap prediction error for linear and generalized linear models

Install required packages

boot

\(k\)-fold crossvalidation

Simulate data

Crossvalidation

[1] 10.44508 10.26059

Leave-one-out crossvalidation

CVE = mean(PRESS)

[1] 10.3501 10.3460

\(k\)-fold crossvalidation of logistic regression using Brier score

Generate data and fit logistic regression.

\(k\)-fold crossvalidation using Brier score \(B\) as a strictly proper score.

[1] 0.5549060 0.5533345
[1] 0.1680421 0.1670967

Bootstrap prediction error with optimism correction

Function to fit logistic regression and calculate Brier score. Take into account that logistic regression might not converge for resample due to perfect separation: Turn warnings into errors and use try() to handle errors.

Bootstrap logistic regression, calculate optimim in prediction error.

Error : (converted from warning) glm.fit: fitted probabilities numerically 0 or 1 occurred
Error : (converted from warning) glm.fit: algorithm did not converge
Error : (converted from warning) glm.fit: algorithm did not converge
Error : (converted from warning) glm.fit: fitted probabilities numerically 0 or 1 occurred
Error : (converted from warning) glm.fit: fitted probabilities numerically 0 or 1 occurred
Error : (converted from warning) glm.fit: fitted probabilities numerically 0 or 1 occurred
Error : (converted from warning) glm.fit: algorithm did not converge
Error : (converted from warning) glm.fit: fitted probabilities numerically 0 or 1 occurred
Error : (converted from warning) glm.fit: algorithm did not converge
Error : (converted from warning) glm.fit: algorithm did not converge
Error : (converted from warning) glm.fit: fitted probabilities numerically 0 or 1 occurred
[1] 0.1494605
[1] -0.02089767

Optimism-corrected bootstrap estimate of prediction error.

[1] 0.1703581

Useful packages

Packages caret and ipred provide more crossvalidation and bootstrapping options like the +.632 bootstrap.

Detach (automatically) loaded packages (if possible)

Get the article source from GitHub

R markdown - markdown - R code - all posts