在Python中实现多列表理解的最有效方法

您可以简单地定义2个函数并在一个列表理解中使用它们,而不是在同一个列表上迭代3次:

cachedStopWords = stopwords.words('english')


def remove_punctuation(text):
    return ''.join(c for c in text.lower() if c not in string.punctuation)

def remove_stop_words(text):
    return ' '.join([word for word in p.split() if word not in cachedStopWords])

rowsaslist = [remove_stop_words(remove_punctuation(text)) for text in rowsaslist]

我从未使用过 stopwords 。如果返回列表,最好将其转换为 set word not in cachedStopWords 测验

最后 NLTK 包可能会帮助您处理文本。看见 @alvas' answer

我倾向于功能性方法*

' '.join(filter(lambda word: word not in cachedStopWords,
                ''.join(filter(lambda c: c not in string.punctuation,
                       map(str.lower, rowsaslist))).split())

不

# removes punctuation, filters out stop words, and lowercases

这完美地解释了一切。

*不可否认,这可能是因为我在哈斯克尔玩得越来越多!

首先,您有两个似乎要删除的黑名单:

• 标点符号
• 停止文字。

为什么标点不能是标记? 这样,您可以通过循环标记来删除标点和停止词,即。

>>> from nltk import word_tokenize
>>> from nltk.corpus import stopwords
>>> from string import punctuation
>>> blacklist = set(punctuation).union(set(stopwords.words('english')))
>>> blacklist
set([u'all', u'just', u'being', u'when', u'over', u'through', u'during', u'its', u'before', '$', u'hadn', '(', u'll', u'had', ',', u'should', u'to', u'only', u'does', u'under', u'ours', u'has', '<', '@', u'them', u'his', u'very', u'they', u'not', u'yourselves', u'now', '\\', u'nor', '`', u'd', u'did', u'shan', u'didn', u'these', u'she', u'each', u'where', '|', u'because', u'doing', u'there', u'theirs', u'some', u'we', u'him', u'up', u'are', u'further', u'ourselves', u'out', '#', "'", '+', u'weren', '/', u're', u'won', u'above', u'between', ';', '?', u't', u'be', u'hasn', u'after', u'here', u'shouldn', u'hers', '[', u'by', '_', u'both', u'about', u'couldn', u'of', u'o', u's', u'isn', '{', u'or', u'own', u'into', u'yourself', u'down', u'mightn', u'wasn', u'your', u'he', '"', u'from', u'her', '&', u'aren', '*', u'been', '.', u'few', u'too', u'wouldn', u'then', u'themselves', ':', u'was', u'until', '>', u'himself', u'on', u'with', u'but', u'mustn', u'off', u'herself', u'than', u'those', '^', u'me', u'myself', u'ma', u'this', u'whom', u'will', u'while', u'ain', u'below', u'can', u'were', u'more', u'my', '~', u'and', u've', u'do', u'is', u'in', u'am', u'it', u'doesn', u'an', u'as', u'itself', u'against', u'have', u'our', u'their', u'if', '!', u'again', '%', u'no', ')', u'that', '-', u'same', u'any', u'how', u'other', u'which', u'you', '=', u'needn', u'y', u'haven', u'who', u'what', u'most', u'such', ']', u'why', u'a', u'don', u'for', u'i', u'm', u'having', u'so', u'at', u'the', '}', u'yours', u'once'])
>>> sent = "This is a humanly readable string, that Tina Guo doesn't want to play"
>>> [word for word in word_tokenize(sent) if word not in blacklist]
['This', 'humanly', 'readable', 'string', 'Tina', 'Guo', "n't", 'want', 'play']

set().difference

>>> set(word_tokenize(sent)).difference(blacklist)
set(['humanly', 'play', 'string', 'This', 'readable', 'Guo', 'Tina', "n't", 'want'])

或者,如果不想标记字符串,可以使用 str.translate 删除标点符号肯定比在字符之间循环更有效:

>>> sent
"This is a humanly readable string, that Tina Guo doesn't want to play"
>>> sent.translate(None, punctuation)
'This is a humanly readable string that Tina Guo doesnt want to play't
>>> stoplist = stopwords.words('english')
>>> [word for word in sent.translate(None, punctuation).split() if word not in stoplist]
['This', 'humanly', 'readable', 'string', 'Tina', 'Guo', 'doesnt', 'want', 'play']

按照您目前的方式,每个列表都将 完全 在创建下一个之前创建。您可以通过从列表理解切换到生成器表达式来解决这个问题(注意 () [] ):

rowsaslist = (x.lower() for x in rows as list) 
rowsaslist = (''.join(c for c in s if c not in string.punctuation) for s in rows as list) 
rowsaslist = (' '.join([word for word in p.split() if word not in cachedStopWords]) for p in rowsaslist)