我在caret中定制了summary函数来计算Brier分数。计算工作正常,但我没有选择最好的模型作为Brier分数最低的模型。
library(data.table)
N <- 1000
X1 <- rnorm(N, 175, 7)
X2 <- rnorm(N, 30, 8)
X3 <- rnorm(N,0,1)
X4 <- rnorm(N,50,3)
X5 <- rnorm(N,2,1)
X6 <- rnorm(N,10,2)
X7 <- runif(N,0,1)
length <- sample(1:5,N,T)
Ycont <- 0.5*X1 - 0.3*X2 +0.01*X3 + 0.2*X4+0.24*X5+X6+X7*0.002 + 10 + rnorm(N, 0, 6)
Ycateg <- ntile(Ycont,3)
df <- data.frame(id=1:N,length,X1, X2,X3,X4,X5,X6,X7, Ycateg)
df$Ycateg=ifelse(df$Ycateg==1,"current",ifelse(df$Ycateg==2,"default","prepaid"))
df=setDT(df)[,.SD[rep(1L,length)],by = id]
df=df[ , time := 1:.N , by=id]
df=df[,-c("length")]
head(df)
customSummary <- function (data, lev = NULL, model = NULL) { # for training on a next-period return
Y_dummy = model.matrix( ~ data[, "obs"] - 1) # create dummy - for each level of the outcome
Y_pre=as.data.frame(data[ , c("current","default","prepaid")])
Brier=(as.numeric(Y_dummy) - Y_pre)^2
Brier_all=sum(Brier)
names(Brier_all)="Brier Score"
return(Brier_all)
}
# which type of cross validation to do
fitControl <- trainControl(method = 'cv',number=5,classProbs=TRUE,summaryFunction=customSummary, selectionFunction = "best" )
# tuning parameters
grid <- expand.grid(mtry = 1:5 )
cv=train(as.factor(Ycateg)~.,
data = df,
method = "ranger",
trControl = fitControl,
tuneGrid = grid
)
cv
产生什么:
......
mtry Brier Score
1 181.02207
2 92.22158
3 85.66351
4 81.85301
5 79.73677
Brier Score was used to select the optimal model using the largest value.
The final value used for the model was mtry = 1.
到目前为止,我使用
trainControl
和
selectionFunction = "best"
所以我的主要方程是,如何选择具有最低Brier分数的模型?
