代码之家 › 专栏 › 技术社区 › Young Pattewa

Web scrape未返回完整html

beautifulsoup web-scraping python

Young Pattewa · 技术社区 · 8 年前

我正试图刮 https://www.kaggle.com/kernels '为了返回站点上的所有标题名,但我遇到了一个问题,即用于此详细信息的容器“div data reactroot”没有被拉入到刮取的数据中。

import urllib
from bs4 import BeautifulSoup

kaggle = 'https://www.kaggle.com/kernels'
data = urllib.request.urlopen(kaggle).read()
htmlparse = BeautifulSoup(data, 'html.parser')
print(htmlparse.findAll("div", {"class" : "block-link block-link--bordered"}))

我的代码中是否有错误,或者网站上是否存在某种阻止我删除这些数据的障碍?

2 回复 | 直到 8 年前

Dan-Dev 8 年前

每次请求页面时,JavaScript都会以json格式获取所需的数据。你可以从“ https://www.kaggle.com/kernels.json?sortBy=hotness&group=everyone&pageSize=20&after=439354&language=all&outputType=all “就像这样。

import requests
import json
source = requests.get("https://www.kaggle.com/kernels.json?sortBy=hotness&group=everyone&pageSize=20&after=439354&language=all&outputType=all")
json_obj = source.json()
for a in json_obj:
    print (a["title"])

输出:

2004-2005 Landfalling Hurricanes animation
Visualization of StockData
Generating Sentences One Letter at a Time 
Decoding the Sexiest Job of 21st Century!!
Novice to Grandmaster
Analysis  on Pokemon Data
ROC Curve with k-Fold CV
Japan Bulgaria trade playground
Bootstrapping and CIs with Veteran Suicides
Replicating "Did I do that?" paper analyses with R
Social Progress Index and World Happiness Report
SVM+HOG On ColourCompositeImage
Low- level students
PyTorch Speech Recognition Challenge (WIP)  
Loans -getting Insights
Exploring Youtube Trending Statistics EDA
3 Simple Steps (LB: .9878 with new data)
Titanic: Neural Network using Keras
Feature Engineering 
Why do employees leave and what to do about it

唯一需要更改的是“after”查询字符串参数,在我的请求中是439354,但可以将其设置为0以获取第一条记录。

您还可以通过更改“pageSize”查询字符串参数来更改返回的记录量,例如: https://www.kaggle.com/kernels.json?sortBy=hotness&group=everyone&pageSize=5&after=0&language=all&outputType=all

输出:

Data ScienceTutorial for Beginners
Data visualization and investigation
Spooky NLP and Topic Modelling tutorial
20 Years Of Games Analysis
NYC Taxi EDA - Update: The fast & the curious

或urllib示例:

import urllib.request
import json
kaggle = "https://www.kaggle.com/kernels.json?sortBy=hotness&group=everyone&pageSize=5&after=0&language=all&outputType=all"
data = urllib.request.urlopen(kaggle).read()
json_obj = json.loads(data.decode("utf-8"))
for a in json_obj:
    print (a["title"])

Bohdan Kaminskyi 8 年前

像 Elis Byberi 问题在于,您试图在从后端呈现数据之前获取数据。您可以使用phantomjs在后端工作后获取页面内容。你可以找到小教程 here