代码之家  ›  专栏  ›  技术社区  ›  Michael Podgortsev

加速解析器:HTML到数据库

  •  0
  • Michael Podgortsev  · 技术社区  · 8 年前

    我需要插入所有的html标签和属性到数据库

    el.driver.get(url_page)
    txthtml = el.driver.page_source
    soup = BeautifulSoup(txthtml, "html.parser")
    body = soup.find('html')
    html_parse(body, el, url_page_id, 0, 0, 0,url_page)
    
    def html_parse(html, el, url_page_id, level, i, parent_id, url_page):
        txt = ""
        if len(html.text) > 0:
           txt = html.text.replace("\n","").replace("\t","").replace("\r","")
        ta = tag_list()
        ta.p_id = el.id
        ta.page_id = url_page_id
        ta.level = level
        ta.number = i
        ta.txt = txt
        ta.name = html.name
        ta.parent_id = parent_id
        ta.html = str(html)
        ta.save()
        insert_attr(html, el.id, url_page_id, ta.id, url_page)
        children = list(html.children)
        j = 0
        for child in children:
            if child.name is None:
                continue
            j = j + 1
            html_parse(child, el, url_page_id, level + 1, j, ta.id, url_page)
    

    当我有递归函数html_parse时

    • el-驾驶员等级
    • level-DOM中的level
    • parent_id—父项的id
    • url_页面-当前url
    • 标签列表-插入当前标签

    每个html_parse函数都运行得很快,但是完整的html解析在每个大html页面上运行大约4-5分钟。

    如何加快代码的速度?

    0 回复  |  直到 8 年前