代码之家 › 专栏 › 技术社区 › Ankur

使用beautifulsoup4查找包含文本的所有结束节点

beautifulsoup python-3.x python

Ankur · 技术社区 · 7 年前

我是巨蟒和美女的新手4

我试图提取(仅)所有标记的文本内容,这些标记要么是'div'、'p'、'li',而且只来自直接节点,而不是子节点,因此有两个选项 text=True, recursive=False

以下是我的尝试:

content = soup.find_all("b", "div", "p", text=True, recursive=False)

和

tags = ["div", "p", "li"]
content = soup.find_all(tags, text=True, recursive=False)

这两个都没有给我输出,你知道我做错了什么吗?

编辑-添加更多代码和我正在测试的示例文档, print(content) 是空的

import requests
from bs4 import BeautifulSoup

url = "https://www.crummy.com/software/BeautifulSoup/bs4/doc/#a-list"
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})

soup = BeautifulSoup(response.text, "html.parser")

tags = ["div", "p", "li"]
content = soup.find_all(tags, text=True, recursive=False)

print(content)

2 回复 | 直到 7 年前

Bitto 7 年前

从你对前一个答案的问题和评论中,我认为你正在努力寻找

最里面的标签

要么是“p”,要么是“li”,要么是“div”

应该包含一些文本

import requests
from bs4 import BeautifulSoup
from bs4 import NavigableString

url = "https://www.crummy.com/software/BeautifulSoup/bs4/doc/#a-list"
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})

soup = BeautifulSoup(response.text, "html.parser")
def end_node(tag):
    if tag.name not in ["div", "p", "li"]:
        return False
    if isinstance(tag,NavigableString): #if str return
        return False
    if not tag.text: #if no text return false
        return False
    elif len(tag.find_all(text=False)) > 0: #no other tags inside other than text
        return False
    return True #if valid it reaches here
content = soup.find_all(end_node)
print(content) #all end nodes matching our criteria

输出样本

[<p>These instructions illustrate all major features of Beautiful Soup 4,
with examples. I show you what the library is good for, how it works,
how to use it, how to make it do what you want, and what to do when it
violates your expectations.</p>, <p>The examples in this documentation should work the same way in Python
2.7 and Python 3.2.</p>, <p>This documentation has been translated into other languages by
Beautiful Soup users:</p>, <p>Here are some simple ways to navigate that data structure:</p>, <p>One common task is extracting all the URLs found within a pageâs &lt;a&gt; tags:</p>, <p>Another common task is extracting all the text from a page:</p>, <p>Does this look like what you need? If so, read on.</p>, <p>If youâre using a recent version of Debian or Ubuntu Linux, you can
install Beautiful Soup with the system package manager:</p>, <p>I use Python 2.7 and Python 3.2 to develop Beautiful Soup, but it
should work with other recent versions.</p>, <p>Beautiful Soup is packaged as Python 2 code. When you install it for
use with Python 3, itâs automatically converted to Python 3 code. If
you donât install the package, the code wonât be converted. There have
also been reports on Windows machines of the wrong version being
installed.</p>, <p>In both cases, your best bet is to completely remove the Beautiful
Soup installation from your system (including any directory created
when you unzipped the tarball) and try the installation again.</p>, <p>This table summarizes the advantages and disadvantages of each parser library:</p>, <li>Batteries included</li>, <li>Decent speed</li>, 
....
]

RoadRunner 7 年前

您可以迭代标记,然后应用 soup.find_all() 在每个标签上:

import requests
from bs4 import BeautifulSoup

url = "https://www.crummy.com/software/BeautifulSoup/bs4/doc/#a-list"
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})

soup = BeautifulSoup(response.text, features="lxml")

tags = ["div", "p", "li"]

for tag in tags:
    content = soup.find_all(tag, recursive=True)

    for x in content:
        print(x)

每个打印出来的 <div> , <p> 和 <li> HTML页上的标记。

你也可以设置 recursive=True 递归遍历文档并提取所有嵌套的子标记。如果不需要这些嵌套的子级,请保留 recursive=False .

您也可以使用 lxml 相反,这比 html.parser . 你可以看到这两者的区别 answer . 如果HTML文档非常大,这可能是有益的。