Python NLTK:使用BigramConsolutionFinder从数据帧的文本字段显示常用短语(ngrams)的频率

这应该够了。。。

首先,设置数据集(或类似的数据集):

import pandas as pd
from nltk.collocations import *
import nltk.collocations
from nltk import ngrams
from collections import Counter

s = pd.Series(
    [
        ['coffee', 'maker', 'brewing', 'properly', '2', '420', '420', '420'],
        ['galley', 'work', 'table', 'stuck'],
        ['cloth', 'stuck'],
        ['stuck', 'coffee']
    ]
)

finder = BigramCollocationFinder.from_documents(s.values)
bigram_measures = nltk.collocations.BigramAssocMeasures()

# only bigrams that appear 1+ times
finder.apply_freq_filter(1) 

# return the 10 n-grams with the highest PMI
result = finder.nbest(bigram_measures.pmi, 10)

使用 nltk.ngrams 要重新创建ngrams列表:

ngram_list = [pair for row in s for pair in ngrams(row, 2)]

使用 collections.Counter 要计算每个ngram在整个语料库中出现的次数:

counts = Counter(ngram_list).most_common()

构建一个看起来像你想要的数据框架:

pd.DataFrame.from_records(counts, columns=['gram', 'count'])
                   gram  count
0            (420, 420)      2
1       (coffee, maker)      1
2      (maker, brewing)      1
3   (brewing, properly)      1
4         (properly, 2)      1
5              (2, 420)      1
6        (galley, work)      1
7         (work, table)      1
8        (table, stuck)      1
9        (cloth, stuck)      1
10      (stuck, coffee)      1

然后,您可以进行过滤,只查看由您的 finder.nbest 电话:

df = pd.DataFrame.from_records(counts, columns=['gram', 'count'])
df[df['gram'].isin(result)]