代码之家 › 专栏 › 技术社区 › Kevin Smeeks

Python Multiprocessing不允许在创建进程后在主进程中打印

python-multiprocessing multiprocessing python

Kevin Smeeks · 技术社区 · 5 年前

在开始流程之前,我试图确保我正确设置了这段代码。在添加了一些打印语句后,我发现只有“外部”和“内部”正在打印,我不明白为什么其他打印语句没有执行。

import multiprocessing
from itertools import product

retailer_ids = [41, 499]  # defined retailers
product_ids = [4, 5, 10, 11, 12]  # ProductIDs to search on
NUMBER_OF_PROCESSES = 2

retailer_products = list(product(retailer_ids, product_ids))

# Start processing the retailer/product combinations
for i in range(0, len(retailer_products), NUMBER_OF_PROCESSES):
    print('outer')
    try:
        current_processes = []
        for j in range(0, NUMBER_OF_PROCESSES):
            print('inner')
            process = multiprocessing.Process(scrape_retailer_product, retailer_products[i+j])
            #process.start()
            current_processes.append(process)
        # wait for current process to finish before starting more
        print('waiting for processes to complete')
        for p in current_processes:
            p.join()

        print('completed')

    # something bad happened during process creation or a
    # a scrape process returned with an exception it could not handle
    except Exception as e:
        for p in current_processes:
            p.terminate()
            print('term')
            exit()

0 回复 | 直到 5 年前

Miguel Alorda 5 年前

问题是你正在捕捉所有异常。因此,您的代码没有将正确的参数传递给 Process 构造函数(生成 AssertionError ),但是你的 catch 语句正在默默地处理异常。

目前的例外情况是:

Traceback (most recent call last):
  File "C:\Users\MiguelAngel\Downloads\test.py", line 19, in <module>
    process = multiprocessing.Process(scrape_retailer_product, args=(retailer_products[i+j]))
  File "C:\Users\MiguelAngel\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\process.py", line 82, in __init__
    assert group is None, 'group argument must be None for now'
AssertionError: group argument must be None for now

我想是的 scrape_retailer_product 是应该在新进程中执行的函数。因此,根据 documentation ,对构造函数的调用应该是:

process = multiprocessing.Process(target=scrape_retailer_product, 
                                  args=(retailer_products[i+j],))

如果你想捕获所有多进程异常,你应该捕获 multiprocessing.ProcessError .根据 documentation ,它是所有多处理异常的基类。