代码之家 › 专栏 › 技术社区 › user241691

通过多线程和多处理进行并行化比串行化花费的时间要多得多

python-multiprocessing multiprocessing parallel-processing multithreading python

user241691 · 技术社区 · 8 年前

我正在尝试学习如何用python进行并行编程。我编写了一个简单的int-square函数,然后在串行、多线程和多进程中运行它:

import time
import multiprocessing, threading
import random


def calc_square(numbers):
    sq = 0
    for n in numbers:
        sq = n*n

def splita(list, n):
    a = [[] for i in range(n)]
    counter = 0
    for i in range(0,len(list)):
        a[counter].append(list[i])
        if len(a[counter]) == len(list)/n:
            counter = counter +1
            continue
    return a


if __name__ == "__main__":

    random.seed(1)
    arr = [random.randint(1, 11) for i in xrange(1000000)]
    print "init completed"

    start_time2 = time.time()
    calc_square(arr)
    end_time2 = time.time()

    print "serial: " + str(end_time2 - start_time2)

    newarr = splita(arr,8)
    print 'split complete'

    start_time = time.time()

    for i in range(8):
        t1 = threading.Thread(target=calc_square, args=(newarr[i],))

        t1.start()
        t1.join()

    end_time = time.time()

    print "mt: " + str(end_time - start_time)

    start_time = time.time()

    for i in range(8):
        p1 = multiprocessing.Process(target=calc_square, args=(newarr[i],))
        p1.start()
        p1.join()

    end_time = time.time()

    print "mp: " + str(end_time - start_time)

输出:

init completed
serial: 0.0640001296997
split complete
mt: 0.0599999427795
mp: 2.97099995613

然而,正如您所看到的,发生了一些奇怪的事情,mt与serial花费的时间相同,mp实际花费的时间明显更长(几乎是serial的50倍)。

我做错了什么?有人能把我推向正确的方向去学习python并行编程吗?

编辑01

查看这些注释,我发现没有返回任何内容的函数似乎毫无意义。我之所以尝试这一点,是因为之前我尝试了以下add函数:

def addi(numbers):
    sq = 0
    for n in numbers:
        sq = sq + n
    return sq

我尝试将每个部分的加法返回到一个序列号加法器,这样至少可以看到相对于纯串行实现的性能改进。然而,我不知道如何存储和使用返回的值,这就是为什么我试图找出比这更简单的方法,即分割数组并在其上运行一个简单的函数。

谢谢

1 回复 | 直到 8 年前

quamrana Ryuzaki L 8 年前

我认为 multiprocessing 创建和启动每个流程需要相当长的时间。我已更改程序,使其大小为 arr 并改变了流程的启动方式,速度略有加快:

(另请注意python 3)

import time
import multiprocessing, threading
from multiprocessing import Queue
import random

def calc_square_q(numbers,q):
    while q.empty():
        pass
    return calc_square(numbers)

if __name__ == "__main__":

    random.seed(1)   # note how big arr is now vvvvvvv
    arr = [random.randint(1, 11) for i in range(10000000)]
    print("init completed")

    # ...
    # other stuff as before
    # ...

    processes=[]
    q=Queue()
    for arrs in newarr:
        processes.append(multiprocessing.Process(target=calc_square_q, args=(arrs,q)))

    print('start processes')
    for p in processes:
        p.start()  # even tho' each process is started it waits...

    print('join processes')
    q.put(None)   # ... for q to become not empty.
    start_time = time.time()
    for p in processes:
        p.join()

    end_time = time.time()

    print("mp: " + str(end_time - start_time))

还要注意上面我如何在两个不同的循环中创建和启动流程,然后最终在第三个循环中与流程连接。

输出:

init completed
serial: 0.53214430809021
split complete
start threads
mt: 0.5551605224609375
start processes
join processes
mp: 0.2800724506378174

另一个10倍的因素是 arr公司 :

init completed
serial: 5.8455305099487305
split complete
start threads
mt: 5.411392450332642
start processes
join processes
mp: 1.9705185890197754

是的,我在python 2.7中也尝试过这一点 Threads 速度似乎较慢。