代码之家  ›  专栏  ›  技术社区  ›  amirouche

如何使用共享状态初始化python多处理工作线程池?

  •  0
  • amirouche  · 技术社区  · 8 年前

    我正在尝试并行执行 some machine learning algorithm .

    当我使用多处理时,它比不使用时慢。我的猜测是 pickle 我使用的模型的序列化减慢了整个过程。所以问题是: 如何用初始状态初始化池的worker,这样就不需要对模型的每次调用进行序列化/反序列化?

    以下是我目前的代码:

    import pickle
    from pathlib import Path
    from collections import Counter
    from multiprocessing import Pool
    
    from gensim.models.doc2vec import Doc2Vec
    
    from wikimark import html2paragraph
    from wikimark import tokenize
    
    
    def process(args):
        doc2vec, regressions, filepath = args
        with filepath.open('r') as f:
            string = f.read()
        subcategories = Counter()
        for index, paragraph in enumerate(html2paragraph(string)):
            tokens = tokenize(paragraph)
            vector = doc2vec.infer_vector(tokens)
            for subcategory, model in regressions.items():
                prediction = model.predict([vector])[0]
                subcategories[subcategory] += prediction
        # compute the mean score for each subcategory
        for subcategory, prediction in subcategories.items():
            subcategories[subcategory] = prediction / (index + 1)
        # keep only the main category
        subcategory = subcategories.most_common(1)[0]
        return (filepath, subcategory)
    
    
    def main():
        input = Path('./build')
        doc2vec = Doc2Vec.load(str(input / 'model.doc2vec.gz'))
        regressions = dict()
        for filepath in input.glob('./*/*/*.model'):
            with filepath.open('rb') as f:
                model = pickle.load(f)
            regressions[filepath.parent] = model
    
        examples = list(input.glob('../data/wikipedia/english/*'))
    
        with Pool() as pool:
            iterable = zip(
                [doc2vec] * len(examples),  # XXX!
                [regressions] * len(examples),  # XXX!
                examples
            )
            for filepath, subcategory in pool.imap_unordered(process, iterable):
                print('* {} -> {}'.format(filepath, subcategory))
    
    
    if __name__ == '__main__':
        main()
    

    标有 XXX! 指向我调用时序列化的数据 pool.imap_unodered 。至少有200MB的数据被序列化。

    如何避免序列化?

    1 回复  |  直到 8 年前
        1
  •  1
  •   amirouche    8 年前

    解决方案非常简单,只需使用全局 doc2vec 和 regressions .