Doc2Vec的管道和网格搜索

我目前有以下脚本,可以帮助找到doc2vec模型的最佳模型。它的工作原理是这样的:首先根据给定的参数训练几个模型,然后对分类器进行测试。最后,它输出了最好的模型和分类器(我希望如此)。

数据

示例数据(data.csv)可在此处下载: https://pastebin.com/takYp6T8 请注意,数据的结构应使分类器的精度达到1.0。

剧本

import sys
import os
from time import time
from operator import itemgetter
import pickle
import pandas as pd
import numpy as np
from argparse import ArgumentParser

from gensim.models.doc2vec import Doc2Vec
from gensim.models import Doc2Vec
import gensim.models.doc2vec
from gensim.models import KeyedVectors
from gensim.models.doc2vec import TaggedDocument, Doc2Vec

from sklearn.base import BaseEstimator
from gensim import corpora

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report


dataset = pd.read_csv("data.csv")

class Doc2VecModel(BaseEstimator):

    def __init__(self, dm=1, size=1, window=1):
        self.d2v_model = None
        self.size = size
        self.window = window
        self.dm = dm

    def fit(self, raw_documents, y=None):
        # Initialize model
        self.d2v_model = Doc2Vec(size=self.size, window=self.window, dm=self.dm, iter=5, alpha=0.025, min_alpha=0.001)
        # Tag docs
        tagged_documents = []
        for index, row in raw_documents.iteritems():
            tag = '{}_{}'.format("type", index)
            tokens = row.split()
            tagged_documents.append(TaggedDocument(words=tokens, tags=[tag]))
        # Build vocabulary
        self.d2v_model.build_vocab(tagged_documents)
        # Train model
        self.d2v_model.train(tagged_documents, total_examples=len(tagged_documents), epochs=self.d2v_model.iter)
        return self

    def transform(self, raw_documents):
        X = []
        for index, row in raw_documents.iteritems():
            X.append(self.d2v_model.infer_vector(row))
        X = pd.DataFrame(X, index=raw_documents.index)
        return X

    def fit_transform(self, raw_documents, y=None):
        self.fit(raw_documents)
        return self.transform(raw_documents)


param_grid = {'doc2vec__window': [2, 3],
              'doc2vec__dm': [0,1],
              'doc2vec__size': [100,200],
              'logreg__C': [0.1, 1],
}

pipe_log = Pipeline([('doc2vec', Doc2VecModel()), ('log', LogisticRegression())])

log_grid = GridSearchCV(pipe_log, 
                        param_grid=param_grid,
                        scoring="accuracy",
                        verbose=3,
                        n_jobs=1)

fitted = log_grid.fit(dataset["posts"], dataset["type"])

# Best parameters
print("Best Parameters: {}\n".format(log_grid.best_params_))
print("Best accuracy: {}\n".format(log_grid.best_score_))
print("Finished.")

关于我的脚本,我确实有以下问题(我将它们结合在一起,以避免三篇文章使用相同的代码片段):

1. 目的是什么 def __init__(self, dm=1, size=1, window=1): ?我是否可以以某种方式删除此部分(尝试失败)?
2. 如何添加 RandomForest GridSearch工作流/管道的分类器(或其他分类器)?
3. 由于当前脚本只在完整数据集上训练,如何将训练/测试数据分割添加到上述代码中?