scrapy只跟随一个深度的外部链接

你可以根据 CrawlSpider Rule s已实施 process_links 规则 documentation :

process_链接 link_extractor . 这主要用于过滤目的。

class BrokenLinksSpider(CrawlSpider):
name = test
start_urls = "your_url"

def parse(self, response):
    flag = response.meta.get('flag')
    if flag or flag==None:
        extractor = LinkExtractor(deny_domains="")
        links = extractor.extract_links(response)
        for link in links:
            if link.url[:8]=="your_url":
                new_request = Request(link.url, callback=self.parse,meta={'flag': True})
            else:
                new_request = Request(link.url, callback=self.parse,meta={'flag': False})
            yield new_request

这不是一个内置的解决方案,但我相信您必须自己中断递归。通过在爬行器中保留一个域数组(集)并中断或忽略,可以很容易地做到这一点。

诸如此类的事情:

from urllib.parse import urlparse

self.track = set()

...
domain = tracktraurlparse(response.url).netloc
x.add(domain)
if len(x) > MAX_RECURSION:
   x.remove(domain)
   # raise StopIteration (# if you're within a generator)
   return None

response.meta['depth'] 可以使用它,而不必设置任何标志。

class BrokenLinksSpider(CrawlSpider):
    name = test
    start_urls = "your_url"
    MAX_DEPTH = 1 # 2nd level

    def parse(self, response: Response) -> None:
        current_depth = response.meta["depth"]
        if current_depth < self.MAX_DEPTH:
            extractor = LinkExtractor(deny_domains="")
            links = extractor.extract_links(response)
            for link in links:
                yield Request(link.url, callback=self.parse)