代码之家 › 专栏 › 技术社区 › mlevy

插入符号中的折叠与训练错误

r-caret r

mlevy · 技术社区 · 8 年前

在模型调整中使用交叉验证,我从 caret::train 的 results 对象并自行计算其上的错误 pred 对象我想了解它们的不同之处,以及理想情况下如何在模型选择、绘制模型性能等方面使用折叠错误率。

这个 pred公司 对象包含未折叠的预测。文件很清楚 trainControl(..., savePredictions = "final") 保存最佳超参数值的折叠预测:“一个指示器,指示每次重采样的保留预测应保存多少…”final“保存最佳调谐参数的预测。”(保留“所有”预测,然后过滤到最佳调整值并不能解决问题。)

这个 train 医生说 后果 对象是“训练错误率的数据帧…”我不确定这意味着什么,但最佳行的值始终不同于根据 pred公司 .他们为什么不同?我如何让他们排成一行?

d <- data.frame(y = rnorm(50))
d$x1 <- rnorm(50, d$y)
d$x2 <- rnorm(50, d$y)
train_control <- caret::trainControl(method = "cv",
                                     number = 4,
                                     search = "random",
                                     savePredictions = "final")
m <- caret::train(x = d[, -1],
                     y = d$y,
                     method = "ranger",
                     trControl = train_control,
                     tuneLength = 3)
#> Loading required package: lattice
#> Loading required package: ggplot2
m
#> Random Forest 
#> 
#> 50 samples
#>  2 predictor
#> 
#> No pre-processing
#> Resampling: Cross-Validated (4 fold) 
#> Summary of sample sizes: 38, 36, 38, 38 
#> Resampling results across tuning parameters:
#> 
#>   min.node.size  mtry  splitrule   RMSE       Rsquared   MAE      
#>   1              2     maxstat     0.5981673  0.6724245  0.4993722
#>   3              1     extratrees  0.5861116  0.7010012  0.4938035
#>   4              2     maxstat     0.6017491  0.6661093  0.4999057
#> 
#> RMSE was used to select the optimal model using the smallest value.
#> The final values used for the model were mtry = 1, splitrule =
#>  extratrees and min.node.size = 3.
MLmetrics::RMSE(m$pred$pred, m$pred$obs)
#> [1] 0.609202
MLmetrics::R2_Score(m$pred$pred, m$pred$obs)
#> [1] 0.642394

创建日期:2018年4月9日 reprex package (v0.2.0)。

2 回复 | 直到 8 年前

missuse 8 年前

交叉验证的RMSE不是按照您所示的方式计算的,而是针对每个折叠进行计算,然后取平均值。完整示例:

set.seed(1)
d <- data.frame(y = rnorm(50))
d$x1 <- rnorm(50, d$y)
d$x2 <- rnorm(50, d$y)
train_control <- caret::trainControl(method = "cv",
                                     number = 4,
                                     search = "random",
                                     savePredictions = "final")
set.seed(1)
m <- caret::train(x = d[, -1],
                  y = d$y,
                  method = "ranger",
                  trControl = train_control,
                  tuneLength = 3)
#output
Random Forest 

50 samples
 2 predictor

No pre-processing
Resampling: Cross-Validated (4 fold) 
Summary of sample sizes: 37, 38, 37, 38 
Resampling results across tuning parameters:

  min.node.size  mtry  splitrule   RMSE       Rsquared   MAE      
   8             1     extratrees  0.6106390  0.4360609  0.4926629
  12             2     extratrees  0.6156636  0.4294237  0.4954481
  19             2     variance    0.6472539  0.3889372  0.5217369

RMSE was used to select the optimal model using the smallest value.
The final values used for the model were mtry = 1, splitrule = extratrees and min.node.size = 8.

最佳型号的RMSE为 0.6106390

现在,计算每个折叠和平均值的RMSE:

m$pred %>%
  group_by(Resample) %>%
  mutate(rmse = caret::RMSE(pred, obs)) %>%
  summarise(mean = mean(rmse)) %>%
  pull(mean) %>%
  mean
#output
0.610639

m$pred %>%
  group_by(Resample) %>%
  mutate(rmse = MLmetrics::RMSE(pred, obs)) %>%
  summarise(mean = mean(rmse)) %>%
  pull(mean) %>%
  mean
#output
0.610639

IRTFM 8 年前

我得到了不同的结果。这显然是一个随机过程。

MLmetrics::RMSE(m$pred$pred, m$pred$obs)
[1] 0.5824464
> MLmetrics::R2_Score(m$pred$pred, m$pred$obs)
[1] 0.5271595

如果希望随机过程(更准确地说是伪随机过程)是可复制的,那么在调用之前立即使用set.seed。