代码之家  ›  专栏  ›  技术社区  ›  JE_Muc

在dict中更新多个键值对的性能

cython dictionary performance python
  •  2
  • JE_Muc  · 技术社区  · 9 年前

    我目前正在使用python开发一个建模环境,该环境使用DICT共享连接部件的连接属性。我目前这样做的方式大约占我总程序运行时间的15-20%,这相当多有几百万次迭代。。。

    所以我发现自己正在研究如何加速更新dict中的多个值,并从dict中获取多个值。
    我的示例dict如下所示(键值对的数量预计将保持在当前300到1000的范围内,因此我将其填充到这个数量): 

    val_dict = {'a': 5.0, 'b': 18.8, 'c': -55/2}
for i in range(200):
    val_dict[str(i)] = i
    val_dict[i] = i**2

keys = ('b', 123, '89', 'c')
new_values = np.arange(10, 41, 10)
length = new_values.shape[0]

    虽然 keys new_values 以及 val_dict 始终保持不变 新的_值 每次迭代时更改

    我计时了几种方法,其中 获取多个值 从dicts看来,使用 itemgetter operator getter 在迭代开始之前,因为所需的变量是常数: 

    getter = itemgetter(*keys)
%timeit getter(val_dict)
The slowest run took 10.45 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 140 ns per loop

    我想这很好,或者有更快的吗?

    但是,当通过掩蔽将这些值分配给numpy数组时,速度会非常慢: 

    result = np.ones(25)
idx = np.array((0, 5, 8, -1))
def getter_fun(result, idx, getter, val_dict):
    result[idx] = getter(val_dict)
%timeit getter_fun(result, idx, getter, new_values)
The slowest run took 11.44 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 2.77 µs per loop

    设置多个值 我已经计时了几种方法:解包值的函数,使用给定键值对的更新的函数,使用for循环的函数,dict理解和生成器函数。 

    def unpack_putter(val_dict, keys, new_values):
    (val_dict[keys[0]],
     val_dict[keys[1]],
     val_dict[keys[2]],
     val_dict[keys[3]]) = new_values
%timeit unpack_putter(val_dict, keys, new_values)
The slowest run took 8.85 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 1.29 µs per loop

def upd_putter(val_dict, keys, new_values):
    val_dict.update({keys[0]: new_values[0],
    keys[1]: new_values[1],
    keys[2]: new_values[2],
    keys[3]: new_values[3]})
%timeit upd_putter(val_dict, keys, new_values)
The slowest run took 15.22 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 963 ns per loop

def for_putter(val_dict, keys, new_values, length):
    for i in range(length):
        val_dict[keys[i]] = new_values[i]
%timeit for_putter(val_dict, keys, new_values, length)
The slowest run took 12.31 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 1.14 µs per loop

def dictcomp_putter(val_dict, keys, new_values, length):
    val_dict.update({keys[i]: new_values[i] for i in range(length)})
%timeit dictcomp_putter(val_dict, keys, new_values, length)
The slowest run took 7.13 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 1.69 µs per loop

def gen_putter(val_dict, keys, new_values, length):
    gen = ((keys[i], new_values[i]) for i in range(length))
    val_dict.update(dict(gen))
%timeit gen_putter(val_dict, keys, new_values, length)
The slowest run took 10.03 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 2.54 µs per loop

    这个 upd_putter 钥匙 和 (在迭代过程中,它们仍然是恒定的,但所考虑的每个部分都有不同数量的密钥要更新,这必须由用户输入决定)。有趣的是,for循环对我来说似乎很好。所以我想我做错了 必须

    joblib 使for循环并行化。我也想过使用 numba ,但我必须摆脱所有的口述。。。

    希望你能帮我解决这个问题。

    为MSeifert编辑 

    tuplelist = list()
for i in range(200):
    tuplelist.append(i)
    tuplelist.append(str(i))
keys_long = tuple(tuplelist)
new_values_long = np.arange(0,400)

%timeit for_putter(val_dict, keys_long, new_values_long, 400)
10000 loops, best of 3: 73.5 µs per loop    
%timeit dictcomp_putter(val_dict, keys_long, new_values_long, 400)
10000 loops, best of 3: 96.4 µs per loop    
%timeit gen_putter(val_dict, keys_long, new_values_long, 400)
10000 loops, best of 3: 129 µs per loop
    1 回复  |  直到 9 年前
        1
  •  5
  •   MSeifert    9 年前

    现在让我们关注两件与性能无关的非常重要的事情: 可维护性 可扩展性 . 

    (val_dict[keys[0]],
 val_dict[keys[1]],
 val_dict[keys[2]],
 val_dict[keys[3]]) = new_values

    和 

    val_dict.update({keys[0]: new_values[0],
                 keys[1]: new_values[1],
                 keys[2]: new_values[2],
                 keys[3]: new_values[3]})

    硬代码(维护噩梦)是指插入的元素数量,因此这些方法的可扩展性不太好。因此,我不会把它们包括在剩下的答案中。我并不是说这些都不好——它们只是不能很好地扩展,而且很难比较只适用于特定数量条目的函数的计时。

    zip (使用 itertools.izip 如果您使用的是python-2。x) : 

    def new1(val_dict, keys, new_values, length):
    val_dict.update(zip(keys, new_values))

def new2(val_dict, keys, new_values, length):
    for key, val in zip(keys, new_values):
        val_dict[key] = val

    哪种方法是解决这个问题的“最具pythonic”的方法(至少在我看来)。

    我还更改了 new_values 因为在NumPy数组上迭代比将数组转换为列表然后在列表上迭代更糟糕,以防您对细节感兴趣,我在另一篇文章中详细阐述了这一部分 answer .

    让我们看看这些方法是如何执行的: 

    import numpy as np

def old_for(val_dict, keys, new_values, length):
    for i in range(length):
        val_dict[keys[i]] = new_values[i]

def old_update_comp(val_dict, keys, new_values, length):
    val_dict.update({keys[i]: new_values[i] for i in range(length)})

def old_update_gen(val_dict, keys, new_values, length):
    gen = ((keys[i], new_values[i]) for i in range(length))
    val_dict.update(dict(gen))

def new1(val_dict, keys, new_values, length):
    val_dict.update(zip(keys, new_values))

def new2(val_dict, keys, new_values, length):
    for key, val in zip(keys, new_values):
        val_dict[key] = val

val_dict = {'a': 1, 'b': 2, 'c': 3}
keys = ('b', 123, '89', 'c')
new_values = np.arange(10, 41, 10).tolist()
length = len(new_values)
%timeit old_for(val_dict, keys, new_values, length)
# 4.1 µs ± 183 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit old_update_comp(val_dict, keys, new_values, length)
# 9.56 µs ± 180 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit old_update_gen(val_dict, keys, new_values, length)
# 17 µs ± 332 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit new1(val_dict, keys, new_values, length)
# 5.92 µs ± 123 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit new2(val_dict, keys, new_values, length)
# 3.23 µs ± 84.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)


    val_dict = {'a': 1, 'b': 2, 'c': 3}
keys = range(1000)
new_values = range(1000)
length = len(new_values)
%timeit old_for(val_dict, keys, new_values, length)
# 1.08 ms ± 26 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit old_update_comp(val_dict, keys, new_values, length)
# 1.08 ms ± 13.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit old_update_gen(val_dict, keys, new_values, length)
# 1.44 ms ± 31.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit new1(val_dict, keys, new_values, length)
# 242 µs ± 3.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit new2(val_dict, keys, new_values, length)
# 346 µs ± 8.24 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

    因此,对于较大的输入,我的方法似乎比您的方法快得多(2-5倍)。

    cdef 或 cpdef 功能,所以我只对其他方法进行了分类: 

    %load_ext cython

%%cython

cpdef new1_cy(dict val_dict, tuple keys, new_values, Py_ssize_t length):
    val_dict.update(zip(keys, new_values.tolist()))

cpdef new2_cy(dict val_dict, tuple keys, new_values, Py_ssize_t length):
    for key, val in zip(keys, new_values.tolist()):
        val_dict[key] = val

cpdef new3_cy(dict val_dict, tuple keys, int[:] new_values, Py_ssize_t length):
    cdef Py_ssize_t i
    for i in range(length):
        val_dict[keys[i]] = new_values[i]

    这次我做了 keys 一 tuple 

    import numpy as np

val_dict = {'a': 1, 'b': 2, 'c': 3}
keys = tuple(range(4))
new_values = np.arange(4)
length = len(new_values)
%timeit new1(val_dict, keys, new_values, length)
# 7.88 µs ± 317 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit new2(val_dict, keys, new_values, length)
# 4.4 µs ± 140 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit new2_cy(val_dict, keys, new_values, length)
# 5.51 µs ± 56.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

val_dict = {'a': 1, 'b': 2, 'c': 3}
keys = tuple(range(1000))
new_values = np.arange(1000)
length = len(new_values)
%timeit new1_cy(val_dict, keys, new_values, length)
# 208 µs ± 9.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit new2_cy(val_dict, keys, new_values, length)
# 231 µs ± 13.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit new3_cy(val_dict, keys, new_values, length)
# 156 µs ± 4.13 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

    因此,如果你有一个元组和一个numpy数组,你可以通过使用普通索引和memoryview的函数实现几乎2倍的加速 new3_cy . 至少如果你有很多需要插入的键值对。

    operator.itemgetter 这可能是最好的方法。

    推荐文章
    关于   移动版
    代码之家 - 一站式码农服务社区
    沪ICP备11025650号