我个人认为这是罕见的
在不使用HTML解析器的情况下将正则表达式应用于完整的文档是一种简单而有效的方法
in this thread
对于以下情况无效:
In [1]: data = """
...: <meta name="twitter:image" content="https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_sns_img_A.jpg?1532940869">
...: <img style="width:100%" id="box_img1" alt="box1" src="https://smtgvs.weathernews.jp/s/topics/img/dummy.png" class="lazy" data-original="https:
...: //smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img1_A.png?1503665797">`
...: <img style="width:100%" id="box_img2" alt="box2" src="https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img2_A.jpg?1503378518
...: ">
...: """
In [2]: import re
In [3]: pattern = re.compile(r"https://smtgvs.weathernews.jp/s/topics/img/[0-9]+/.+\?[0-9]+")
In [4]: pattern.findall(data)
Out[4]:
['https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_sns_img_A.jpg?1532940869',
'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img1_A.png?1503665797',
'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img2_A.jpg?1503378518']
如果您对如何将正则表达式模式应用于
BeautifulSoup
,可能是这样的(我知道,不漂亮):
In [6]: results = soup.find_all(lambda tag: any(pattern.search(attr) for attr in tag.attrs.values()))
In [7]: [next(attr for attr in tag.attrs.values() if pattern.search(attr)) for tag in results]
Out[7]:
[u'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_sns_img_A.jpg?1532940869',
u'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img1_A.png?1503665797',
u'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img2_A.jpg?1503378518']
这里我们基本上是遍历所有元素的所有属性并检查模式匹配。然后,一旦得到所有匹配的标记,我们就在结果上迭代,并得到匹配属性的值。我真的不喜欢这样的事实,我们应用regex检查两次-当寻找标记和检查匹配标记的期望属性时。
lxml.html
lxml
支持不支持正则表达式的XPath 1.0。你可以这样做:
In [10]: from lxml.html import fromstring
In [11]: root = fromstring(data)
In [12]: root.xpath('.//@*[contains(., "smtgvs.weathernews.jp") and contains(., "?")]')
Out[12]:
['https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_sns_img_A.jpg?1532940869',
'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img1_A.png?1503665797',
'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img2_A.jpg?1503378518']
这并不是100%你所做的,而且可能会产生误报,但是如果需要的话,你可以进一步增加“字符串中的子字符串”检查。
或者,您可以获取所有元素的所有属性,并使用已有的regex进行筛选:
In [14]: [attr for attr in root.xpath("//@*") if pattern.search(attr)]
Out[14]:
['https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_sns_img_A.jpg?1532940869',
'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img1_A.png?1503665797',
'https://smtgvs.weathernews.jp/s/topics/img/201807/201807300285_box_img2_A.jpg?1503378518']
