代码之家  ›  专栏  ›  技术社区  ›  Ettore Rizza

使用rvest在循环中跟踪链接

  •  1
  • Ettore Rizza  · 技术社区  · 9 年前

    rvest 但是web上的文档和示例要么非常基本,要么非常复杂。我找不到如何使用 follow_link 函数在循环中浏览一定数量的页面。也许我根本不明白它的逻辑。。。

    library(rvest)
    
    url <-
      "https://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500"
    
    s <- html_session(url)
    
    liste <- list()
    for (i in 1:2) {
      data <-
        s %>%
        read_html() %>%
        html_nodes("#mw-whatlinkshere-list li")
    
      result <- c(liste, data)
    
      s <- s %>% 
        follow_link(xpath = "//a[text()='next 500']/@href")
    
    }
    

    我也试图避免 jump_link

    liste <- c()
    while (!is.na(url)) {
      data <-
        url %>%
        read_html() %>%
        html_nodes("#mw-whatlinkshere-list li")
    
    
      liste <- c(liste, data)
    
      url <- url %>% 
        read_html() %>% 
        html_node(xpath = "//a[text()='next 500']") %>% 
        html_attr("href") %>% 
        paste0("https://www.wikidata.org", .) 
    
      print(url)
    
    
    }
    

    欢迎提供任何建议,我们将不胜感激。

    1 回复  |  直到 9 年前
        1
  •  1
  •   RogerioJB    8 年前

    试试这个:

    library(rvest)
    
    url <- "https://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500"
    s   <- html_session(url)
    
    liste <- list()
    for (i in 1:2) {
            data <-
                    s %>%
                    read_html() %>%
                    html_nodes("#mw-whatlinkshere-list li")
    
            # There was a mistake here. You were overwriting your results
            liste <- c(liste, data) 
    
            # Here you have to pass a 'a' tag, not a 'href' value. Besides,
            # there is two 'next 500' tags. They are the same, but you have
            # to pick one.
            s <- s %>% 
                follow_link(xpath = "//a[text()='next 500'][1]") 
    }