代码之家 › 专栏 › 技术社区 › Jack Gruber

我应该如何使用BeautifulSoup在页面上的特定dt标记之间刮除dd标记中的文本?

beautifulsoup web-scraping python-3.x html python

Jack Gruber · 技术社区 · 7 年前

timeDiv = mezzrowSource.find_all("dd", class_="orange event-date")
eventDiv = mezzrowSource.find_all("dd", class_="event")
index = 0 
for time in timeDiv:
    returnValue[timeDiv[index].text] = eventDiv[index].text.strip()
    if "8" in timeDiv[index+3].text or "4:30" in timeDiv[index+3].text:
        break
    index += 1

以这种方式枚举会导致太多来自其他人的文本大部分时间都在工作,但有时会从其他日期提取事件。在这里的来源,部分问题是粘贴在下面。有什么想法吗?

  <dt class="purple">Sun, September 30th, 2018</dt>

    <dd class="orange event-date">4:30 PM to 7:00 PM</dd>
    <dd class="event"><a href="/events/4094-mezzrow-classical-salon-with-david-oei"
                         class="event-title">Mezzrow Classical Salon with David Oei</a>


    </dd>

    <dd class="orange event-date">8:00 PM to 10:30 PM</dd>
    <dd class="event"><a href="/events/4144-luke-sellick-ron-blake-adam-birnbaum"
                         class="event-title">Luke Sellick, Ron Blake &amp; Adam Birnbaum</a>


    </dd>

    <dd class="orange event-date">11:00 PM to 1:00 AM</dd>
    <dd class="event"><a href="/events/4099-ryo-sasaki-friends-after-hours"
                         class="event-title">Ryo Sasaki &amp; Friends &quot;After-hours&quot;</a>


    </dd>


  <dt class="purple">Mon, October 1st, 2018</dt>

    <dd class="orange event-date">8:00 PM to 10:30 PM</dd>
    <dd class="event"><a href="/events/4137-greg-ruggiero-murray-wall-steve-little"
                         class="event-title">Greg Ruggiero, Murray Wall &amp; Steve Little</a>


    </dd>

    <dd class="orange event-date">11:00 PM to 1:00 AM</dd>
    <dd class="event"><a href="/events/4174-pasquale-grasso-after-hours"
                         class="event-title">Pasquale Grasso &quot;After-hours&quot;</a>


    </dd>

预期输出的字典如下:{'4:30 PM to 7:00 PM':'Mezzrow Classic Salon with David Oei','8:00 PM to 10:30 PM':'Greg Ruggiero,Murray Wall&史蒂夫·利特尔(Steve Little),“晚上11:00到凌晨1:00”:“帕斯夸尔·格拉索(Pasquale Grasso)”“下班后”}

2 回复 | 直到 7 年前

Dan-Dev 7 年前

如果我理解正确,您可以使用zip():

mezzrowSource = BeautifulSoup(html , 'lxml')
timeDiv = [tag.get_text() for tag in mezzrowSource.find_all("dd", class_="orange event-date")]
eventDiv = [tag.get_text().strip() for tag in mezzrowSource.find_all("dd", class_="event")]
print(dict(zip(timeDiv, eventDiv)))

输出:

{'4:30 PM to 7:00 PM': 'Mezzrow Classical Salon with David Oei', '8:00 PM to 10:30 PM': 'Greg Ruggiero, Murray Wall & Steve Little', '11:00 PM to 1:00 AM': 'Pasquale Grasso "After-hours"'}

更新时间:

您希望从中获取数据的元素都是同级元素,即没有包含每组数据的元素,这使得按照您的需要对数据进行分组变得更加困难。唯一对您有利的是,带有日期的元素首先出现,然后是时间,然后是标题。时间和标题可以重复。所以这个方法选择了我们想要的所有元素并对它们进行迭代。在第一次迭代中,它将日期存储在一个字符串中,并创建一个包含时间和标题的元组列表。当它下次找到一个日期时,它会将日期和元组列表附加到字典中。在迭代结束时,它会将最终日期和元组列表附加到字典中。这有点混乱,但这是由于HTML中缺乏结构。

from bs4 import BeautifulSoup
import requests
import re
import pprint

url = 'https://www.mezzrow.com/'
r = requests.get(url)
soup = BeautifulSoup(r.text , 'lxml')
ds = soup.find_all(True, {'class': re.compile('purple|event|orange event_date')})
ret = {}
tmp = []
i = None
for d in ds:
    if d.attrs['class']==['purple']:
        if i is not None:
            ret[i] = tmp
            tmp = []
        i = (d.get_text())
    elif d.attrs['class']==['orange', 'event-date']:
        j =  d.get_text()
    elif d.attrs['class']==['event']:
        tmp.append ((j,d.get_text(strip=True)))
ret[i] = tmp
pp = pprint.PrettyPrinter(depth=6)
pp.pprint(ret)

输出:

{'Fri, October 12th, 2018': [('8:00 PM to 10:30 PM',
                              'Rossano Sportiello, Pasquale Grasso & Frank '
                              'Tate'),
                             ('11:00 PM to 2:00 AM',
                              'Ben Paterson "After-hours"')],
 'Fri, October 5th, 2018': [('8:00 PM to 10:30 PM',
                             'Vanessa Rubin, Brandon McCune, Kenny Davis & '
                             'Winard Harper'),
                            ('11:00 PM to 2:00 AM',
                             'Joe Davidian "After-hours"')],
 'Mon, October 1st, 2018': [('8:00 PM to 10:30 PM',
                             'Greg Ruggiero, Murray Wall & Steve Little'),
                            ('11:00 PM to 1:00 AM',
                             'Pasquale Grasso "After-hours"')],
....

然后从dict对象中选择所需的日期。