如何处理带有未终止多行注释的标记化错误(python 2.6)

import token, tokenize, StringIO

def generate_tokens(src):
    rawstr = StringIO.StringIO(unicode(src))
    tokens = tokenize.generate_tokens(rawstr.readline)
    for i, item in enumerate(tokens):
        toktype, toktext, (srow,scol), (erow,ecol), line = item
        print i, token.tok_name[toktype], toktext

s = \
"""
 def test(x):
     \"\"\" test with an unterminated docstring
"""

generate_tokens(s)

引发以下事件:

... (stripped a little)
File "/usr/lib/python2.6/tokenize.py", line 296, in generate_tokens
    raise TokenError, ("EOF in multi-line string", strstart)
tokenize.TokenError: ('EOF in multi-line string', (3, 5))

关于这种行为的一些问题:

1. 我是否应该在此处捕获并“选择性”忽略tokenize.TokenError?或 我应该停止尝试从不合规/不完整的代码生成令牌吗?如果是,我将如何检查?
2. 此错误(或类似错误)是否可能是由其他原因引起的