代码之家 › 专栏 › 技术社区 › Lin Ma

在python中使用CountVectorizer时发生内存错误

xgboost scikit-learn machine-learning python

1

Lin Ma · 技术社区 · 7 年前

todense() ,我正在使用GBDT模型,想知道是否有人对如何解决内存错误有好的想法?谢谢。

  for feature_colunm_name in feature_columns_to_use:
    X_train[feature_colunm_name] = CountVectorizer().fit_transform(X_train[feature_colunm_name]).todense()
    X_test[feature_colunm_name] = CountVectorizer().fit_transform(X_test[feature_colunm_name]).todense()
  y_train = y_train.astype('int')
  grd = GradientBoostingClassifier(n_estimators=n_estimator, max_depth=10)
  grd.fit(X_train.values, y_train.values)

详细的错误消息,

in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)
MemoryError
...

林

1 回复 | 直到 7 年前

1

Vivek Kumar 7 年前

这里有很多错误:

for feature_colunm_name in feature_columns_to_use:
    X_train[feature_colunm_name] = CountVectorizer().fit_transform(X_train[feature_colunm_name]).todense()
    X_test[feature_colunm_name] = CountVectorizer().fit_transform(X_test[feature_colunm_name]).todense()

1) 您正在尝试分配多个列(的结果) CountVectorizer 将是一个二维数组,其中列表示特征)到单个列的 feature_colunm_name '的数据帧。这是行不通的,会产生错误。

transform() fit_transform() .

比如:

cv = CountVectorizer()
X_train_cv = cv.fit_transform(X_train[feature_colunm_name])
X_test_cv = cv.transform(X_test[feature_colunm_name])

3) GradientBoostingClassifier 适用于稀疏数据。它还没有在文档中提到(似乎是文档中的一个错误)。

更新 :

# To merge sparse matrices
from scipy.sparse import hstack

result_matrix_train = None
result_matrix_test = None

for feature_colunm_name in feature_columns_to_use:
    cv = CountVectorizer()
    X_train_cv = cv.fit_transform(X_train[feature_colunm_name])

    # Merge the vector with others
    result_matrix_train = hstack((result_matrix_train, X_train_cv)) 
                          if result_matrix_train is not None else X_train_cv

    # Now transform the test data
    X_test_cv = cv.transform(X_test[feature_colunm_name])
    result_matrix_test = hstack((result_matrix_test, X_test_cv)) 
                         if result_matrix_test is not None else X_test_cv

注意:如果您还有其他列,您没有通过Countvectorizer处理这些列,因为它们已经是数字了,您希望与 result_matrix_train

result_matrix_train = hstack((result_matrix_test, X_train[other_columns].values)) 
result_matrix_test = hstack((result_matrix_test, X_test[other_columns].values))

现在用这些来训练:

...
grd.fit(result_matrix_train, y_train.values)