查找并拆分单词后面的某些字符

似乎你想做标记化。尝试 nltk

http://text-processing.com/demo/tokenize/

from nltk.tokenize import TreebankWordTokenizer
splits = TreebankWordTokenizer().tokenize(text)

你可以用

re.findall(r'\w+(?:\.\w+)*|[^\w\s]', s)

见 regex demo

细节

• \w+(?:\.\w+)* -1+字字符后跟0个或多个点,后跟1+字字符
• | -或者
• [^\w\s]

Python demo :

import re
rx = r"\w+(?:\.\w+)*|[^\w\s]"
s = "Mr.Smith is a professor at Harvard, and is a great guy."
print(re.findall(rx, s))

输出: ['Mr.Smith', 'is', 'a', 'professor', 'at', 'Harvard', ',', 'and', 'is', 'a', 'great', 'guy', '.']

这种方法可以进一步精确化。E、 g.仅将字母、单词、数字和下划线标记为标点符号:

re.findall(r'[+-]?\d*\.?\d+|[^\W\d_]+(?:\.[^\W\d_]+)*|[^\w\s]|_', s)

见 regex demo

([.,](?=\s)|\s) 然后过滤空字符串或空字符串:

In [16]: filter(lambda s: not re.match(r'\s*$', s) , re.split(r'([.,](?=\s)|\s)',  'Mr.Smith is a professor at Har
    ...: vard, and is a great guy.'))
Out[16]: 
['Mr.Smith',
 'is',
 'a',
 'professor',
 'at',
 'Harvard',
 ',',
 'and',
 'is',
 'a',
 'great',
 'guy.']