代码之家 › 专栏 › 技术社区 › Mrowkacala

每个项目有多个页面-使用临时链接

scrapy web-scraping web python

0

Mrowkacala · 技术社区 · 8 年前

我的水龙头看起来像这样;

class ScrapeMovies(scrapy.Spider):

    start_urls = [
        'https://www.trekearth.com/members/page1.htm?sort_by=md'
    ]

    def parse(self, response):
        for row in response.xpath('//table[@class="member-table"]//tr[position() > 1]'):

            item = loopitem()
            website = row.xpath('./td[2]//a/@href/text()').extract_first()
            item['name'] = row.xpath('./td[2]//a/text()').extract_first()

            yield item
   # This part is responsible for scraping all of the pages on a start url commented out for convinience    
#       next_page=response.xpath('//div[@class="page-nav-btm"]/ul/li[last()]/a/@href').extract_first()
#       if next_page is not None:
#            next_page=response.urljoin(next_page)
#            yield scrapy.Request(next_page, callback=self.parse)

它所做的一切都会刮表(请参见起始url)。我希望它然后转到链接(成员名称列),然后从这个链接中提取一些信息(链接是例如。 https://www.trekearth.com/members/monareng/ )并将其作为物品退回。

我该怎么做?

如果有任何不清楚的地方,请立即要求澄清。

编辑: 现在,我的代码如下所示(但仍然不起作用):

class ScrapeMovies(scrapy.Spider):
    name='final'

    start_urls = [
        'https://www.trekearth.com/members/page1.htm?sort_by=md'
    ]

    def parse(self, response):
        for row in response.xpath('//table[@class="member-table"]//tr[position() > 1]'):

            item = FinalItem()    
            website = row.xpath('./td[2]//a/@href/text()').extract_first()
            item['name'] = row.xpath('./td[2]//a/text()').extract_first()

            request = scrapy.Request(website,
            callback=self.parse_page2)
            request.meta['item'] = item
            return request

    def parse_page2(self, response): 
        item = response.meta['item']
        item['other_url'] = response.url
        item['groups'] = response.xpath('//div[@class="groups-btm"]/ul/li/text()').extract_first()
        return item

1 回复 | 直到 8 年前

1

0

Danil 8 年前

使用 meta 字段以将项目提交给下一次回调

def parse_page1(self, response):
    item = MyItem(main_url=response.url)
    request = scrapy.Request("http://www.example.com/some_page.html",
                             callback=self.parse_page2)
    request.meta['item'] = item
    return request

def parse_page2(self, response):
    item = response.meta['item']
    item['other_url'] = response.url
    return item

使用UPD处理所有行 yield 在您的循环中

    for row in response.xpath('//table[@class="member-table"]//tr[position() > 1]'):

        item = FinalItem()    
        website = row.xpath('./td[2]//a/@href/text()').extract_first()
        item['name'] = row.xpath('./td[2]//a/text()').extract_first()

        request = scrapy.Request(website,
        callback=self.parse_page2)
        request.meta['item'] = item
        yield request