代码之家 › 专栏 › 技术社区 › maximodesousadias

循环的Scrapy XPATH

scrapy xpath for-loop loops python

maximodesousadias · 技术社区 · 4 年前

我正在设置我的第一个Scrapy Spider,我在使用Xpath选择器时遇到了一些困难。

我试图从中提取数据的url是:

https://socios.xubio.com/ar/contadores/

该网站共有9页,每页都有几个公司名称的“盒子”或“容器”。

我试图提取:

名称
住址
电话
电子邮件

Box's schema

如果我在刮壳 ,我正确地提取数据:

response.xpath('//div[@class="col-11"]//p//text()').extract() #Names
response.xpath('//div[@class="col-md-2 col-12 h-60"]//p//text()').extract() #Phones
response.xpath('//div[@class="col-md-4 col-12 h-60"]//p//text()').extract() #Addresses
response.xpath('//div[@class="col-md-3 col-12 h-60"]//p//text()').extract() #Emails

此外,如果我为每个框运行以下循环,它将返回10条记录(第1页中的框数):

Scrapy Shell Output

for box in response.xpath('//div[@class="w-100 padding-15 "]'):
    print(1)

但是,当我运行以下程序时,我不会刮取任何数据:

(在本例中,我使用相对Xpath)

import scrapy
from NewContacts_Crawler.items import NewContactItem

class XubioContadoresSpider(scrapy.Spider):
    name = "XubioContadores"
    start_urls = [
        'https://socios.xubio.com/ar/contadores/?pag=1',
        'https://socios.xubio.com/ar/contadores/?pag=2',
        'https://socios.xubio.com/ar/contadores/?pag=3',
        'https://socios.xubio.com/ar/contadores/?pag=4',
        'https://socios.xubio.com/ar/contadores/?pag=5',
        'https://socios.xubio.com/ar/contadores/?pag=6',
        'https://socios.xubio.com/ar/contadores/?pag=7',
        'https://socios.xubio.com/ar/contadores/?pag=8',
        'https://socios.xubio.com/ar/contadores/?pag=9',
    ]

    def parse(self, response):

        items = NewContactItem()

        for box in response.xpath('//div[@class="w-100 padding-15 "]'):

            name = box.response.xpath('.//div[@class="col-11"]//p//text()').extract_first()
            phone = box.response.xpath('.//div[@class="col-md-2 col-12 h-60"]//p//text()').extract_first()
            address = box.response.xpath('.//div[@class="col-md-4 col-12 h-60"]//p//text()').extract_first()
            email = box.response.xpath('.//div[@class="col-md-3 col-12 h-60"]//p//text()').extract_first()
            
            items['name'] = name
            items['phone'] = phone
            items['address'] = address
            items['email'] = email

            yield items

我以Scrapy的Xpath代码为例:

import scrapy


class ToScrapeSpiderXPath(scrapy.Spider):
    name = 'toscrape-xpath'
    start_urls = [
        'http://quotes.toscrape.com/',
    ]

    def parse(self, response):
        for quote in response.xpath('//div[@class="quote"]'):
            yield {
                'text': quote.xpath('./span[@class="text"]/text()').extract_first(),
                'author': quote.xpath('.//small[@class="author"]/text()').extract_first(),
                'tags': quote.xpath('.//div[@class="tags"]/a[@class="tag"]/text()').extract()
            }

        next_page_url = response.xpath('//li[@class="next"]/a/@href').extract_first()
        if next_page_url is not None:
            yield scrapy.Request(response.urljoin(next_page_url))

所以,我认为循环的两个部分可以单独工作。但是,我无法确定代码中的问题在哪里。

0 回复 | 直到 4 年前

furas 4 年前

Scrapy 在控制台/终端中运行时显示许多信息,您应该检查消息中的内容,因为我收到了错误消息

  File "<pyshell#0>", line 15, in parse
    name = box.response.xpath('.//div[@class="col-11"]//p//text()').extract_first()
AttributeError: 'NoneType' object has no attribute 'xpath'

这表明这是必须的 box.xpath 而不是 box.response.xpath

最小工作代码。

它给了我文件 CSV 共有87项。

您可以将所有代码放在一个文件中并运行 python script.py 不创造 project

import scrapy

class XubioContadoresSpider(scrapy.Spider):

    name = "XubioContadores"

    start_urls = [
        f'https://socios.xubio.com/ar/contadores/?pag={i}' for i in range(1, 10)
    ]

    def parse(self, response):
        print('url:', response.url)
        
        for box in response.xpath('//div[@class="w-100 padding-15 "]'):

            name = box.xpath('.//div[@class="col-11"]//p//text()').extract_first()
            phone = box.xpath('.//div[@class="col-md-2 col-12 h-60"]//p//text()').extract_first()
            address = box.xpath('.//div[@class="col-md-4 col-12 h-60"]//p//text()').extract_first()
            email = box.xpath('.//div[@class="col-md-3 col-12 h-60"]//p//text()').extract_first()

            item = dict()
            
            item['name'] = name
            item['phone'] = phone
            item['address'] = address
            item['email'] = email

            yield item
            
# --- run without project and save in `output.csv` ---

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    'USER_AGENT': 'Mozilla/5.0',
    'FEEDS': {'output.csv': {'format': 'csv'}},  # new in 2.1
})
c.crawl(XubioContadoresSpider)
c.start()