Imdb review编码错误

有一个词汇词典,在预处理中,我回顾了一些索引序列。
例如

“这部电影是最好的”->[2,5,10,3]

num of reviews 100
number of unique tokens : 4761
Traceback (most recent call last):
  File "preprocess.py", line 47, in <module>
    print(vocab)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 10561: ordinal not in range(128)

代码如下:

from bs4 import BeautifulSoup
reviews = []
for item in os.listdir('imdbdata/train/pos')[:100]:
    with open("imdbdata/train/pos/"+item,'r',encoding='utf-8') as f:
        sample = BeautifulSoup(f.read()).get_text()
    sample = word_tokenize(sample.lower())
    reviews.append(sample)
print("num of reviews", len(reviews))
word_freq = nltk.FreqDist(itertools.chain(*reviews))
print("number of unique tokens : %d"%(len(word_freq.items())))
vocab = word_freq.most_common(vocab_size-1)
index_to_word = [x[0] for x in vocab]
index_to_word.append(unknown_token)
word_to_index = dict((w,i) for i,w in enumerate(index_to_word))
print(vocab)

问题是,我怎么能逃脱这个 UnicodeEncodeError 在用Python处理自然语言处理问题时?尤其是在使用 open 作用