我设置了一个简单的python脚本,从H&M.返回名称时没有遇到问题,但图像URL似乎只返回前几个,然后采用以下格式:“data:image.gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAAAAAAABAAEAAAIBRAA7”我分别尝试过请求和selenium与chromedriver。我错过了什么?
首次尝试(请求):
import requests
from bs4 import BeautifulSoup
# URL of the H&M men's section
url = "https://www2.hm.com/en_us/men/products/view-all.html?page=1"
# Headers to mimic a browser visit
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Referer": "https://www.google.com/",
"Connection": "keep-alive"
}
# Send a GET request to the webpage
response = requests.get(url, headers=headers)
# Check if the request was successful
if response.status_code == 200:
# Parse the HTML content
soup = BeautifulSoup(response.content, 'html.parser')
# Find all the product items
items = soup.find_all('article', class_='f0cf84')
# Iterate over the items and extract the name and image URL
for item in items:
# Extract the product name
name = item.find('a', class_='db7c79')['title']
# Extract the image URL (the 'src' attribute of the <img> tag)
img_tag = item.find('img', imagetype='PRODUCT_IMAGE')
img_url = img_tag['src'] if img_tag else 'No image'
# Print the name and image URL
print(f"Product Name: {name}")
print(f"Image URL: {img_url}\n")
else:
print(f"Failed to retrieve the page. Status code: {response.status_code}")
第二次尝试(硒)
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome()
# URL of the H&M men's section
url = "https://www2.hm.com/en_us/men/products/view-all.html?page=1"
# Open the webpage
driver.get(url)
# Get the page source and parse it with BeautifulSoup
soup = BeautifulSoup(driver.page_source, 'html.parser')
# Find all the product items
items = soup.find_all('article', class_='f0cf84')
# Iterate over the items and extract the name and image URL
for item in items:
# Extract the product name
name = item.find('a', class_='db7c79')['title']
# Extract the image URL (the 'src' attribute of the <img> tag)
img_tag = item.find('img', imagetype='PRODUCT_IMAGE')
img_url = img_tag['src'] if img_tag else 'No image'
# Print the name and image URL
print(f"Product Name: {name}")
print(f"Image URL: {img_url}\n")
# Quit the WebDriver
driver.quit()
两次的响应都是相同的:
Product Name: Baggy Jeans
Image URL: https://image.hm.com/assets/hm/9e/53/9e53035efef96606bc4b50eaf6a0eee4f08a152c.jpg?imwidth=1536
Product Name: Regular Fit Cotton Shorts
Image URL: https://image.hm.com/assets/hm/8f/d8/8fd8d52f2e2c778041410f9a2727b448053ca8b7.jpg?imwidth=1536
Product Name: Regular Fit Linen-blend Shorts
Image URL: https://image.hm.com/assets/hm/d7/54/d7546a095c04387d1ad98575588c84e0426fb4be.jpg?imwidth=1536
Product Name: Muscle Fit Cotton Shirt
Image URL: https://image.hm.com/assets/hm/c7/d4/c7d49cef60f9d196d2f5347815f416bba7d4b636.jpg?imwidth=1536
Product Name: Slim Fit Ribbed Tank Top
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Slim Fit Jacket
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: 5-pack Slim Fit T-shirts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Linen-blend Resort Shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Slim Fit Suit Pants
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Cotton Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Slim Fit Suit Pants
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Slim Fit Polo Shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Baggy Jeans
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Slim Fit Half-zip Polo Shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Slim Fit Linen Jacket
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Loose Fit Cargo Jeans
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Nylon Cargo Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Loose Fit T-shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Loose Jeans
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Swim Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Chino Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Muscle Fit Polo Shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Linen-blend Pants
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: 5-pack Short Cotton Boxer Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Cropped Cotton Chinos
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Linen-blend Shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Slim Fit Linen Suit Pants
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Linen-blend Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Swim Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Patterned Swim Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Patterned Swim Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit T-shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit T-shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Sweatshorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Regular Fit Cotton Shorts
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
Product Name: Slim Fit T-shirt
Image URL: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
