如何在python中通过折叠实现Unicode字符串匹配

我有一个实现增量搜索的应用程序。我有一个要匹配的unicode字符串目录,并将它们与给定的“键”字符串匹配;如果目录字符串按顺序包含键中的所有字符,则它是一个“命中”,如果键字符在目录字符串中聚集,则它的排名会更好。

无论如何,这很好用,并且与unicode完全匹配,因此“st”将匹配 圣 圣 或“r” D 圣 嗯。

例如:“Ole”应与“Ol”匹配

如何在Python中最好地实现这个unicode折叠匹配器?效率很重要,因为我必须将数千个目录字符串与短的给定键相匹配。

它不必将其转换为ascii;事实上,算法的输出字符串可以是unicode。将角色保留在中比将其剥离要好。

# -*- encoding: UTF-8 -*-

import unicodedata
from unicodedata import normalize, category

def _folditems():
    _folding_table = {
        # general non-decomposing characters
        # FIXME: This is not complete
        u"ł" : u"l",
        u"œ" : u"oe",
        u"ð" : u"d",
        u"þ" : u"th",
        u"ß" : u"ss",
        # germano-scandinavic canonical transliterations
        u"ü" : u"ue",
        u"Ã¥" : u"aa",
        u"ä" : u"ae",
        u"æ" : u"ae",
        u"ö" : u"oe",
        u"ø" : u"oe",
    }

    for c, rep in _folding_table.iteritems():
        yield (ord(c.upper()), rep.title())
        yield (ord(c), rep)

folding_table = dict(_folditems())

def tofolded(ustr):
    u"""Fold @ustr

    Return a unicode str where composed characters are replaced by
    their base, and extended latin characters are replaced by
    similar basic latin characters.

    >>> tofolded(u"Wyłącz")
    u'Wylacz'
    >>> tofolded(u"naïveté")
    u'naivete'

    Characters from other scripts are not transliterated.

    >>> tofolded(u"Ἑλλάς") == u'Ελλας'
    True

    (These doctests pass, but should they fail, they fail hard)
    """
    srcstr = normalize("NFKD", ustr.translate(folding_table))
    return u"".join(c for c in srcstr if category(c) != 'Mn')

if __name__ == '__main__':
    import doctest
    doctest.testmod()

(对于实际匹配,如果有人感兴趣的话:我预先为我的所有目录构造折叠字符串,并将折叠版本放入已经可用的目录对象别名属性中。)