代码之家  ›  专栏  ›  技术社区  ›  Philipp

瘙痒+硒问题

  •  0
  • Philipp  · 技术社区  · 8 年前

    我正在尝试使用Selenium和Scrapy(见下面的代码)搜索英国一家著名零售商的网站。我得到一个 [scrapy.core.scraper] ERROR: Spider error processing

    import scrapy
    from selenium import webdriver
    from nl_scrape.items import NlScrapeItem
    import time
    
    class ProductSpider(scrapy.Spider):
        name = "product_spider"
        allowed_domains = ['newlook.com']
        start_urls = ['http://www.newlook.com/uk/womens/clothing/c/uk-womens-clothing?comp=NavigationBar%7Cmn%7Cwomens%7Cclothing#/?q=:relevance&page=1&sort=relevance&content=false']
    
    def __init__(self):
        self.driver = webdriver.Safari()
        self.driver.set_window_size(800,600)
        time.sleep(4)
    
    def parse(self, response):
        self.driver.get(response.url)
        time.sleep(4)
    
        # Collect products
        products = driver.find_elements_by_class_name('plp-item ng-scope')
    
        # Iterate over products; extract data and append individual features to NlScrapeItem
        for item in products:
    
            # Pull features
            desc = item.find_element_by_class_name('product-item__name link--nounderline ng-binding').text
            href = item.find_element_by_class_name('plp-carousel__img-link ng-scope').get_attribute('href')
    
            # Price Symbol removal and integer conversion
            priceString = item.find_element_by_class_name('price ng-binding').text
            priceInt = priceString.split('£')[1]
            price = float(priceInt)
    
            # Generate a product identifier
            identifier = href.split('/p/')[1].split('?comp')[0]
            identifier = int(identifier)
    
            # datetime
            dt = date.today()
            dt = dt.isoformat()
    
            # NlScrapeItem
            item = NlScrapeItem()
    
            # Append product to NlScrapeItem
            item['id'] = identifier
            item['href'] = href
            item['description'] = desc
            item['price'] = price
            item['firstSighted'] = dt
            item['lastSighted'] = dt
            yield item
    
        self.driver.close()
    

    2017-08-26 15:48:38[scrapy.core.scraper]错误:蜘蛛错误处理http://www.newlook.com/uk/womens/clothing/c/uk-womens-clothing?comp=NavigationBar%7Cmn%7Cwomens%7Cclothing#/?q=:relevance&页码=1&排序=相关性;内容=假>(参考:无)

    文件“/Users/username/Documents/nl\u scraping/nl\u env/lib/python3.6/site packages/twisted/internet/defer.py”,第653行,在runCallbacks中 现在的结果=回调(current.result,*args,**kw) 产品=驱动程序。按类名查找元素('plp-item ng scope') NameError:未定义名称“driver”

    1 回复  |  直到 8 年前
        1
  •  2
  •   Tarun Lalwani    8 年前

    所以你的代码有两个问题

    def parse(self, response):
        self.driver.get(response.url)
        time.sleep(4)
    
        # Collect products
        products = driver.find_elements_by_class_name('plp-item ng-scope')
    

    很方便你换了衣服 self.driver 只是 driver . 这样不行。您应该在函数顶部添加

    def parse(self, response):
        driver = self.driver
        driver.get(response.url)
        time.sleep(4)
    
        # Collect products
        products = driver.find_elements_by_class_name('plp-item ng-scope')
    

    下一步您已经使用 self.driver.close()