代码之家 › 专栏 › 技术社区 › Istvan

如何使用scikit learn评估基于文本的模型?

nlp scikit-learn python

Istvan · 技术社区 · 6 年前

index   field1      field2            field3
1079    COMPUTER    long text....     3

我使用以下代码学习sklearn的field2到category映射:

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB

X_train, X_test, y_train, y_test = train_test_split(df['Text'], df['category_id'], random_state = 0)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf = MultinomialNB().fit(X_train_tfidf, y_train)

在我训练了模型之后,我可以用它来预测一个类别,而且效果很好。但是,我想使用测试集来评估模型。

X_test_counts = count_vect.fit_transform(X_test)
X_test_tfidf = tfidf_transformer.fit_transform(X_test_counts)
clf.score(X_test_tfidf, y_test)

它抛出以下错误:

ValueError: dimension mismatch

更新:向测试集添加类似的转换。

3 回复 | 直到 6 年前

Amir 5 年前

这个 MultinomialNB TFIDF 转换将文档编码为连续值特征。然而,在实践中,诸如tf-idf之类的分数计数也可能起作用 [reference]

要解决此问题,请将代码更改为以下内容:

count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(df['Text'].values.tolist())
X_train, X_test, y_train, y_test = train_test_split(X_train_counts, df['category_id'], random_state = 0)

clf = MultinomialNB().fit(X_train, y_train)
clf.predict(X_test)

要增强代码使用管道:

from sklearn.pipeline import Pipeline
X_train, X_test, y_train, y_test = train_test_split(df['Text'], df['category_id'], random_state = 0)
text_clf = Pipeline([('vect', CountVectorizer()),
                    ('tfidf', TfidfTransformer()),
                    ('clf', MultinomialNB())])
text_clf.fit(X_train, y_train)
text_clf.predict(X_test)

runcoderun 6 年前

X_test X_train .

ValueError: dimension mismatch

因为变压器已经安装到训练设备上了,你应该打电话给 .transform() 在测试集上:

tfidf_transformer.transform(X_test_counts)

更多信息 here .

Prayson W. Daniel 5 年前

您应该只转换测试数据。不适合转换。所以,如果你去掉测试数据上的拟合,它应该会起作用。

最好使用将进行转换的管道,然后进行训练/评分/预测。例如。

from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB

model = Pipeline(steps = [    
            ('word_vec', CountVectorizer()),
            ('word_tdf',  TfidfTransformer()),
            ('mnb',MultinomialNB()),
        ])

simple_model.fit(X_train,y_train)
simple_model.score(X_test,y_test)

这使得您的代码更简单,并且不太可能适合您的测试数据。