代码之家 › 专栏 › 技术社区 › Kamikaze_goldfish

Python Scrapy不输出到csv文件

scrapy python

Kamikaze_goldfish · 技术社区 · 7 年前

scrapy runspider yellowpages.py -o items.csv 但仍然没有什么出来,只有一个空白的csv文件。我在这里关注了不同的事情,还观看了youtube,试图找出我在哪里犯了错误,但仍然无法找出我做的不对。

# -*- coding: utf-8 -*-
import scrapy
import requests

search = "Plumbers"
location = "Hammond, LA"
url = "https://www.yellowpages.com/search"
q = {'search_terms': search, 'geo_location_terms': location}
page = requests.get(url, params=q)
page = page.url
items = ()


class YellowpagesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['yellowpages.com']
    start_urls = [page]

    def parse(self, response):
        self.log("I just visited: " + response.url)
        items = response.css('a[class=business-name]::attr(href)')
        for item in items:
            print(item)

3 回复 | 直到 7 年前

soldy 7 年前

没有项目的简单蜘蛛。

就你而言:

刮痧蜘蛛黄页.py-a servise=“水管工”-a location=“哈蒙德,洛杉矶”-oHammondsplumbers.csv文件

代码还可以处理任何查询。例如:

等。。。

# -*- coding: utf-8 -*-
import scrapy
from scrapy.http import Request
from scrapy.exceptions import CloseSpider


class YellowpagesSpider(scrapy.Spider):
    name = 'yellowpages'
    allowed_domains = ['yellowpages.com']
    start_urls = ['https://www.yellowpages.com/']

    # We can use any pair servise + location on our request
    def __init__(self, servise=None, location=None):
        self.servise = servise
        self.location = location

    def parse(self, response):
        # If "service " and" location " are defined 
        if self.servise and self.location:
            # Create search phrase using "service" and " location"
            search_url = 'search?search_terms={}&geo_location_terms={}'.format(self.servise, self.location)
            # Send request with url "yellowpages.com" + "search_url", then call parse_result
            yield Request(url=response.urljoin(search_url), callback=self.parse_result)
        else:
            # Else close our spider
            # You can add deffault value if you want.
            self.logger.warning('=== Please use keys -a servise="service_name" -a location="location" ===')
            raise CloseSpider()

    def parse_result(self, response):
        # all blocks without AD posts
        posts = response.xpath('//div[@class="search-results organic"]//div[@class="v-card"]')
        for post in posts:
            yield {
                'title': post.xpath('.//span[@itemprop="name"]/text()').extract_first(),
                'url': response.urljoin(post.xpath('.//a[@class="business-name"]/@href').extract_first()),
            }

        next_page = response.xpath('//a[@class="next ajax-page"]/@href').extract_first()
        # If we have next page url
        if next_page:
            # Send request with url "yellowpages.com" + "next_page", then call parse_result
            yield scrapy.Request(url=response.urljoin(next_page), callback=self.parse_result)

Yash Pokar 7 年前

for item in items:
    print(item)

for item in items:
    yield item

Woody1193 Nimmi Rashinika 7 年前

在检查您的代码时,我注意到一些问题:

首先,初始化 items items = [] .

name scrapy crawl my_crawler 哪里 name = "my_crawler" .

start_urls 应该包含字符串,而不是 Request 物体。你应该把条目改为 page 要使用的确切搜索字符串。如果您有许多搜索字符串,并且希望对它们进行迭代,我建议使用 middleware

当你试图从CSS中提取数据时,你忘记了调用 extract_all() 它将把你的选择器转换成你可以使用的字符串数据。

另外,您不应该重定向到标准输出流,因为那里有很多日志记录,这会使您的输出文件非常混乱。相反,您应该将响应提取到 items loaders .

最后,您可能缺少了您的应用程序中的适当设置 settings.py 文件。你可以找到相关的文件 here .

FEED_FORMAT = "csv"
FEED_EXPORT_FIELDS = ["Field 1", "Field 2", "Field 3"]