代码之家 › 专栏 › 技术社区 › turnip

训练多项式朴素贝叶斯模型时的记忆错误

nltk out-of-memory scikit-learn python

0

turnip · 技术社区 · 8 年前

我的代码的训练部分可以处理以下数量级的数据 10^4 但考虑到我的整个数据集由大约500000条评论组成,我想用更多的数据来训练它。我在运行100000条评论的培训师时,似乎内存不足。

我的 get_features 功能似乎是罪魁祸首。

data = get_data(limit=size)
data = clean_data(data)
all_words = [w.lower() for (comment, category) in data for w in comment]
word_features = []
for i in nltk.FreqDist(all_words).most_common(3000):
    word_features.append(i[0])
random.shuffle(data)

def get_features(comment):
    features = {}
    for word in word_features:
        features[word] = (word in set(comment))  # error here
    return features

# I can do it myself like this:
feature_set = [(get_features(comment), category) for
                (comment, category) in data]

# Or use nltk's Lazy Map implementation which arguable does the same thing:
# feature_set = nltk.classify.apply_features(get_features, data, labeled=True)

为运行此 100,000 评论耗尽了我所有的32GB内存,最终导致崩溃 Memory Error ,位于 features[word] = (word in set(comment)) 线

我能做些什么来缓解这个问题?

编辑:我显著减少了功能的数量:我现在只使用前3000个最常见的单词作为功能-这显著提高了性能(原因很明显)。我还纠正了@Marat指出的一个小错误。

1 回复 | 直到 8 年前

1

2

Marat 8 年前

免责声明:这段代码有许多潜在的缺陷,所以我希望很少有迭代能够找到根本原因。

参数不匹配:

# defined with one parameter
def get_features(comment):
    ...

# called with two
... get_features(comment, word_features), ...

次优单词查找:

# set(comment) executed on every iteration
for word in word_features:
    features[word] = (word in set(comment))

# can be transformed into something like:
word_set = set(comment)
for word in word_features:
    features[word] = word in word_set

# if typical comment length is < 30, list lookup is faster
for word in word_features:
    features[word] = word in comment

次优特征计算:

# it is cheaper to set few positives than to check all word_features
# also MUCH more memory efficient
from collections import defaultdict
...
def get_features(comment):
    features = defaultdict(bool)
    for word in comment:
        features[word] = True
    return features

次优功能存储:

# numpy array is much more efficient than a list of dicts
# .. and with pandas on top it's even nicer:
import pandas as pd
...
feature_set = pd.DataFrame(
    ({word: True for word in comment}
      for (comment, _) in data),
    columns = word_features
).fillna(False)
feature_set['category'] = [category for (_, category) in data]