代码之家 › 专栏 › 技术社区 › 2Obe

从数据帧中删除重复项的矢量化方法

pandas python-3.x python

2Obe · 技术社区 · 9 年前

frame=pd.read_excel(io=r"D:XXXX\test.xlsx")
df=pd.DataFrame(frame)


dup=[df.duplicated(subset=(i),keep=False) for i in [("id","Type"),("id","Time"),("Time","Type")]]
duplicate="duplicate"

for i in range(len(dup)):
    for j in range(len(dup[i])):
        if dup[i][j]==True:
            df.loc[j,"Attribute"]=duplicate

数据帧的形状类似于:

id  Type    Time
12  ab  12:00:00
11  cd  11:12:22
663 dfd 10:00:00

但如果一个文件中有许多行,这种方法就会变得乏味。我正在寻找一种方法,用lambda组合替换循环,或者用LC应用

感谢您的回复

1 回复 | 直到 9 年前

Rayhane Mama 9 年前

pandas.duplicated 和 pandas.apply

for i in [("id","Type"),("id","Time"),("Time","Type")]:
    df['Attribute'] = df.duplicated(subset=i,keep=False).apply(lambda x: "duplicate" if x else "not duplicate")

解决方案 numpy.where :

for i in [("id","Type"),("id","Time"),("Time","Type")]:
    df['Attribute'] = np.where(df.duplicated(subset=i,keep=False),"duplicate","not duplicate")

我使用这个数据帧作为输入:

    id   Type      Time
0   12     ab  12:00:00
1   12  abacd  11:12:22
2  663    dfd  10:00:00
3   11     ab  12:00:00
4  663    dfd  10:00:00
5   11   caad  11:12:22

    id   Type      Time      Attribute
0   12     ab  12:00:00      duplicate
1   12  abacd  11:12:22  not duplicate
2  663    dfd  10:00:00      duplicate
3   11     ab  12:00:00      duplicate
4  663    dfd  10:00:00      duplicate
5   11   caad  11:12:22  not duplicate

希望这有帮助。

推荐文章