代码之家 › 专栏 › 技术社区 › hyoni

使用Python BeautifulSoup进行网页抓取

beautifulsoup web-scraping html python

2

hyoni · 技术社区 · 11 月前

我想通过Python BeautifulSoup从 website 我的数据分析项目。

我想从网站上收集的数据;

日期:027年7月6日
舞台:Berghain,全景酒吧,花园
时刻表
艺术家
标签

最终,我想将数据传输到SQL以构建此示例表

sample table

我非常困在网络抓取的第一步。

import requests
from bs4 import BeautifulSoup

url = 'https://www.berghain.berlin/en/event/77218/'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
soup.find('div', class_='running-order-set__info').find('span').contents[0]

结果:“Norman Nodge”

我只得到了第一个艺术家的名字。。不知道如何收集其他信息!

任何善良的灵魂都能帮助贫困学生做一些有趣的项目吗?

** 更新我能够用@Andrej Kesely的回答来解决这个问题。

我稍作调整以适应数据帧。

dates = []
names = []
stages = []
times = []
artists = []
notes = []

for li in soup.select('main li'):
    date = li.find_previous('p')
    dates.append(date.get_text(strip=True, separator=' ').split()[1])

    name = li.find_previous('h1')
    names.append(name.get_text(strip=True))

    stage = li.find_previous('h2')
    stages.append(stage.get_text(strip=True))

    time = li.time.text
    times.append(time)

    artist = li.select_one('.running-order-set__info span')
    artists.append(artist.contents[0] if artist.contents else 'NaN')

    notes.append(artist.span.text if artist.span else 'NaN')

bh_df = pd.DataFrame(
{'date': dates,
'party_name': names,
'stage': stages,
'start_time': times,
'artist_name': artists,
'note': notes
})

结果:

1 回复 | 直到 11 月前

1

2

Andrej Kesely 11 月前

您可以尝试:

import requests
from bs4 import BeautifulSoup

url = "https://www.berghain.berlin/en/event/77218/"

soup = BeautifulSoup(requests.get(url).content, "html.parser")

for li in soup.select("main li"):
    date = li.find_previous("p")
    name = li.find_previous("h1")
    stage = li.find_previous("h2")
    time = li.time.text
    artist = li.select_one(".running-order-set__info span")

    print("Date  ", date.get_text(strip=True, separator=" ").split()[1])
    print("Name  ", name.get_text(strip=True))
    print("Stage ", stage.get_text(strip=True))
    print("Time  ", time)
    print("Artist", artist.contents[0] if artist.contents else "-")
    print("Note  ", artist.span.text if artist.span else "-")

    print("-" * 80)

打印:

Date   06.07.2024                           
Name   Klubnacht                                                                                         
Stage  Berghain  
Time   23:59    
Artist Norman Nodge 
Note   Ostgut Ton
--------------------------------------------------------------------------------
Date   06.07.2024      
Name   Klubnacht                                                                                         
Stage  Berghain  
Time   04:30    
Artist Alienata    
Note   -    
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht                                                                                         
Stage  Berghain  
Time   08:30    
Artist UVB         
Note   Mord 
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht                                                                                         
Stage  Berghain  
Time   12:30    
Artist Matthew Cha 
Note   -    
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht                                                                                         
Stage  Berghain  
Time   16:30    
Artist Gaetano Parisio
Note   -    
--------------------------------------------------------------------------------
Date   06.07.2024         
Name   Klubnacht                                                                                         
Stage  Berghain  
Time   20:30    
Artist Justine Perry
Note   -    
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht                                                                                         
Stage  Berghain  
Time   00:30    
Artist DJ Nobu 
Note   Bitta
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht                                                                                         
Stage  Panorama Bar                            
Time   23:59
Artist Lauer 
Note   Live at Robert Johnson / Running Back
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht
Stage  Panorama Bar
Time   04:00
Artist Dam Swindle 
Note   Heist Recordings
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht
Stage  Panorama Bar
Time   08:00
Artist Kikelomo
Note   -
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht
Stage  Panorama Bar
Time   12:30
Artist -
Note   -
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht
Stage  Panorama Bar
Time   19:30
Artist Wallace
Note   -
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht
Stage  Panorama Bar
Time   00:00
Artist Cinthie 
Note   803 Crystal Grooves
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht
Stage  Garten
Time   12:00
Artist Suze IjÃ³
Note   -
--------------------------------------------------------------------------------
Date   06.07.2024
Name   Klubnacht
Stage  Garten
Time   16:00
Artist Hiroko Yamamura
Note   -
--------------------------------------------------------------------------------