代码之家 › 专栏 › 技术社区 › Akarsh Jain

R中的Xpath表达式给出与chrome检查器不同的结果

xml2 web-scraping xpath xml r

0

Akarsh Jain · 技术社区 · 7 年前

使用下面给出的xpath,从各个页面获取日期内容,我得到了所需的结果。但这一页特别指出 http://eventsgeneva.strikingly.com//blog/agenda-geneve-something-you-should-never-miss ,在使用chrome检查器时给出所需的结果,而在R中使用相同的xpath则不会给出任何结果。

在chrome中使用下面的xpath时。

xpath = '((//h1/parent::*/following::*|//h1/ancestor::*[position()<3]/descendant-or-self::*)[position()<150 and (string-length(text())<150 and (contains(text(), "Jan") or contains(text(), "Feb") or contains(text(), "Mar") or contains(text(), "Apr") or contains(text(), "May") or contains(text(), "Jun") or contains(text(), "Jul") or contains(text(), "Aug") or contains(text(), "Sep") or contains(text(), "Oct") or contains(text(), "Nov") or contains(text(), "Dec")))])'

我明白了,

而使用库“xml2”在R中使用相同的xpath。

library(dplyr)

library(xml2)

html_page<-read_html("http://eventsgeneva.strikingly.com//blog/agenda-geneve-something-you-should-never-miss")

html_page%>%
  xml_find_all(xpath = '((//h1/parent::*/following::*|//h1/ancestor::*[position()<3]/descendant-or-self::*)[position()<150 and (string-length(text())<150 and (contains(text(), "Jan") or contains(text(), "Feb") or contains(text(), "Mar") or contains(text(), "Apr") or contains(text(), "May") or contains(text(), "Jun") or contains(text(), "Jul") or contains(text(), "Aug") or contains(text(), "Sep") or contains(text(), "Oct") or contains(text(), "Nov") or contains(text(), "Dec")))])')
#> {xml_nodeset (0)}

有什么我错过的吗?

1 回复 | 直到 7 年前

1

hrbrmstr 7 年前

从上述内容推断:

使用 `decapitated`

library(rvest)
library(decapitated)
library(tidyverse)

doc <- decapitated::chrome_read_html("http://eventsgeneva.strikingly.com//blog/agenda-geneve-something-you-should-never-miss")

html_nodes(doc, xpath = '((//h1/parent::*/following::*|//h1/ancestor::*[position()<3]/descendant-or-self::*)[position()<150 and (string-length(text())<150 and (contains(text(), "Jan") or contains(text(), "Feb") or contains(text(), "Mar") or contains(text(), "Apr") or contains(text(), "May") or contains(text(), "Jun") or contains(text(), "Jul") or contains(text(), "Aug") or contains(text(), "Sep") or contains(text(), "Oct") or contains(text(), "Nov") or contains(text(), "Dec")))])')
## {xml_nodeset (1)}
## [1] <span class="s-blog-date">August 4, 2018</span>

请阅读自述文件和pkg文档,因为您需要Chrome(最好是一个单独的Chrome二进制文件,如软件包中所述)和一个环境变量设置,并且您必须自己调试任何设置问题。

使用 `splashr`

这个包需要 reticulate 包、Docker和Python docker 模块。因此,如果遇到问题,可以进行更多的自我调试:

library(rvest)
library(splashr)
library(tidyverse)

sp <- splashr::start_splash()

doc <- render_html(splash_local, "http://eventsgeneva.strikingly.com//blog/agenda-geneve-something-you-should-never-miss")

html_nodes(doc, xpath = '((//h1/parent::*/following::*|//h1/ancestor::*[position()<3]/descendant-or-self::*)[position()<150 and (string-length(text())<150 and (contains(text(), "Jan") or contains(text(), "Feb") or contains(text(), "Mar") or contains(text(), "Apr") or contains(text(), "May") or contains(text(), "Jun") or contains(text(), "Jul") or contains(text(), "Aug") or contains(text(), "Sep") or contains(text(), "Oct") or contains(text(), "Nov") or contains(text(), "Dec")))])')
## {xml_nodeset (1)}
## [1] <span class="s-blog-date">August 4, 2018</span>

killall_splash()

使用 `V8`

为了避免使用外部程序,您可以使用要处理页面变量并获取内容:

library(rvest)
library(V8)
library(tidyverse)

ctx <- v8()

doc <- read_html("http://eventsgeneva.strikingly.com//blog/agenda-geneve-something-you-should-never-miss")

html_nodes(doc, xpath=".//script")[[1]] %>% # get 1st <script>
  html_text() %>% # get contents of it
  str_replace(regex("^.*window\\.", multiline=TRUE), "var $S = {};\n") %>% # make the variable usable in V8
  ctx$eval() # evaluate the javascript
## [1] "[object Object]"

pg <- ctx$get("$S") # marshall it to R

str(pg, 1)
## List of 6
##  $ globalConf        :List of 26
##  $ conf              :List of 12
##  $ miniProgramAppType: NULL
##  $ blogPostData      :List of 5
##  $ siteData          :List of 5
##  $ stores            :List of 3

str(pg$blogPostData, 1)
## List of 5
##  $ blogPostMeta:List of 25
##  $ pageMeta    :List of 33
##  $ content     :List of 8
##  $ settings    :List of 2
##  $ pageMode    : NULL

str(pg$blogPostData$content, 1)
## List of 8
##  $ type            : chr "Blog.BlogData"
##  $ id              : chr "f_cc4ace2d-21ed-4b94-83a0-e83497e5afc4"
##  $ defaultValue    : NULL
##  $ showComments    : logi TRUE
##  $ showShareButtons: NULL
##  $ header          :List of 6
##  $ footer          :List of 5
##  $ sections        :'data.frame':    9 obs. of  4 variables:

内容似乎在这里:

str(pg$blogPostData$content$sections)
## 'data.frame':    9 obs. of  4 variables:
##  $ type        : chr  "Blog.Section" "Blog.Section" "Blog.Section" "Blog.Section" ...
##  $ id          : chr  "f_9ca5a1d7-ccb8-4315-9883-bcd43d271b9c" "f_4b7b30f1-387c-4cbe-aaed-ddaedea92cc1" "f_252813ac-b6cb-484b-81f5-64d7f0745c8e" "f_bd7412a4-b94b-4c5a-8cdd-a48931639dce" ...
##  $ defaultValue: logi  NA NA NA NA NA NA ...
##  $ component   :'data.frame':    9 obs. of  6 variables:
##   ..$ type        : chr  "RichText" "RichText" "RichText" "RichText" ...
##   ..$ id          : chr  "f_4e41d6f3-8449-4f66-b701-28d1bcfb08c9" "f_c27703de-8679-4916-9697-220cb8c7a74d" "f_c3c20474-99fc-434a-aff1-102d2a342450" "f_7b3e5247-39ef-42c7-b95c-f0be0b6e9728" ...
##   ..$ defaultValue: logi  FALSE NA NA NA NA NA ...
##   ..$ value       : chr  "<p style=\"text-align: justify;\">We all make our plans beforehand in order to avoid any unnecessary issues. So"| __truncated__ "<p style=\"text-align: justify;\">Take a glance at the below-listed events and plan accordingly -</p>" "<p style=\"text-align: justify;\"><u>Siestes dominicales</u> â Here you are invited to groove on the grass and "| __truncated__ "<p style=\"text-align: justify;\"><u>Sonoboat ACT</u> â Neptune is one the most popular and historic sailing bo"| __truncated__ ...
##   ..$ backupValue : logi  NA NA NA NA NA NA ...
##   ..$ version     : int  1 NA NA NA NA NA NA 1 1

value 单独或 paste0() 将它们转换成一个HTML块并进行评估。

R中的Xpath表达式给出与chrome检查器不同的结果

使用 decapitated

使用 splashr

使用 V8

使用 `decapitated`

使用 `splashr`

使用 `V8`