代码之家 › 专栏 › 技术社区 › Jensen Holm

在非常大的字符串中查找链接时遇到问题

beautifulsoup web-scraping parsing string

Jensen Holm · 技术社区 · 1 年前

我正在为一个数据科学项目收集棒球参考资料,当我试图从一个特定的联盟收集球员数据时,遇到了一个交叉问题。jsut本赛季开始参加的联赛。当我刮掉已经打完比赛的旧联赛时,我没有任何问题。但我想在这个链接上与联盟擦肩而过: https://www.baseball-reference.com/register/league.cgi?id=c346199a 随季节而活。然而,这些链接隐藏在许多看似纯文本的内容背后。如此美丽的群像。find\u all('a',href=True)不工作。

这是我迄今为止的思考过程。

html = BeautifulSoup(requests.get('https://www.baseball-reference.com/register/league.cgi?id=c346199a').text, features = 'html.parser').find_all('div')
ind = [str(div) for div in html][0]
orig_ind = ind[ind.find('/register/team.cgi?id='):]
count = orig_ind.count('/register/team.cgi?id=')

team_links = []
for i in range(count):
  # rn finds the same one over and over
  link = orig_ind[orig_ind.find('/register/team.cgi?id='):orig_ind.find('title')].strip().replace('"', '')
  # try to remove it from orig_ind and do the next link...
  # this is the part that is not working rn
  orig_ind = orig_ind.replace(link, '')
  team_links.append('https://baseball-reference.com' + link)

其输出:

['https://baseball-reference.com/register/team.cgi?id=71fe19cd',
 'https://baseball-reference.com',
 'https://baseball-reference.com',
 'https://baseball-reference.com',
 'https://baseball-reference.com',
 'https://baseball-reference.com',
 'https://baseball-reference.com',
 'https://baseball-reference.com',
 'https://baseball-reference.com',
 'https://baseball-reference.com',
 'https://baseball-reference.com',

等等我正在尝试从此页面获取所有团队链接: https://www.baseball-reference.com/register/league.cgi?id=c346199a

然后爬行到每个页面上的播放器链接并收集一些数据。就像我说的,除了这一个,我试过的几乎每一个联赛都是如此。

任何帮助都是非常感激的。

1 回复 | 直到 1 年前

Andrej Kesely 1 年前

您在此网站上看到的表存储在HTML注释中(  )所以BeautifulSoup通常不会看到它们。要分析它们,请尝试下一个示例:

import requests
from bs4 import BeautifulSoup, Comment


soup = BeautifulSoup(
    requests.get(
        "https://www.baseball-reference.com/register/league.cgi?id=c346199a"
    ).text,
    features="html.parser",
)

s = "".join(c for c in soup.find_all(text=Comment) if "table_container" in c)
soup = BeautifulSoup(s, "html.parser")

for a in soup.select('[href*="/register/team.cgi?id="]'):
    print("{:<30} {}".format(a.text, a["href"]))

打印:

Battle Creek Bombers           /register/team.cgi?id=f3c4b615
Kenosha Kingfish               /register/team.cgi?id=71fe19cd
Kokomo Jackrabbits             /register/team.cgi?id=8f1a41fc
Rockford Rivets                /register/team.cgi?id=9f4fe2ef
Traverse City Pit Spitters     /register/team.cgi?id=7bc8d111
Kalamazoo Growlers             /register/team.cgi?id=9995d2a1
Fond du Lac Dock Spiders       /register/team.cgi?id=02911efc

...and so on.