我一直使用熊猫的历史,因为它的灵活性,简单,因为我在R.有一些经验,我真的不认为我放弃了这么多的速度方面…但是今天为了好玩做一些a/b比较让我有点抓狂。
我会使用一些集合,但当我做任何类型的数据科学时,我只是自然而然地被熊猫吸引,因为这感觉是一个相当安全的举动。但是熊猫够快了吗?我不确定。。。我想我可能在很多情况下都很马虎。这些差别相当大:
%%timeit
file = "Ex_Files_Python_Efficiently/Exercise Files/chapter2/02_05/goldmedals.txt"
df = pd.read_table(file,header=None, names=['year','athlete','team','event'])
def clean_event(name):
return ' '.join(word for word in name.split() if word not in ['men', 'women'])
df['event'] = df.apply(lambda x: clean_event(x['event']), axis=1)
df.pivot_table(values='team', index='event', columns='athlete', aggfunc='nunique').sum().sort_values(ascending=False)
>>> 29.4 ms ± 285 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
现在使用集合:
%%timeit
file = "Ex_Files_Python_Efficiently/Exercise Files/chapter2/02_05/goldmedals.txt"
Medal = collections.namedtuple('medal',['year','athlete','team','event'])
medals = [Medal(*line.strip().split('\t')) for line in open(file, 'r')]
d = collections.defaultdict(set)
def howmany(tup):
return len(tup[1])
def clean_event(name):
return ' '.join(word for word in name.split() if word not in ['men', 'women'])
for medal in medals:
d[medal.athlete].add(clean_event(medal.event))
sorted(d.items(), key=howmany, reverse=True)
>>> 2.94 ms ± 40.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
真 的!我是不是缺了什么重要的东西?我很确定是df.apply消耗了处理能力(clean_事件不是用cython编写的,即使如此,我认为它仍然必须返回python来完成连接)。
这里有没有人经历过类似的a/b测试,或者有人能给我指一个方向,让我在这里再深入一点?随着时间的推移,我已经找到了一些零碎的东西,但是我很乐意看到一些更有经验的人偶然发现的任何智慧。。。
编辑:
已将clean_event()更改为列表理解。仍然在考虑是否应该留在pandas中使用pivot表,它添加了abt 13ms。
%%timeit
from typing import List
file = "Ex_Files_Python_Efficiently/Exercise Files/chapter2/02_05/goldmedals.txt"
df = pd.read_table(file,header=None, names=['year','athlete','team','event'])
def clean_event(names: List[str]):
return [' '.join(word for word in name.split() if word not in ['men', 'women']) for name in names]
df['event'] = clean_event(df['event'].values)
df.pivot_table(values='team', index='event', columns='athlete', aggfunc='nunique').sum().sort_values(ascending=False)
>>> 16.3 ms ± 58 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
最终编辑:
我去掉了pivot表,并使用了groupby(),这将减少一些不必要的操作。离我们很近。
%%timeit
from typing import List
file = "Ex_Files_Python_Efficiently/Exercise Files/chapter2/02_05/goldmedals.txt"
df = pd.read_table(file,header=None, names=['year','athlete','team','event'])
def clean_event(names: List[str]):
return [' '.join(word for word in name.split() if word not in ['men', 'women']) for name in names]
df['event'] = clean_event(df['event'].values)
df.groupby('athlete').nunique().sort_values('event', ascending=False)['event']
>>> 7.67 ms ± 64.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
我最后的工作如下:
https://github.com/bejoinka/playground
