代码之家  ›  专栏  ›  技术社区  ›  SIM

从网页中获取标题时遇到问题

  •  10
  • SIM  · 技术社区  · 8 年前

    我已经用php编写了一个脚本来刮取 标题 可视为 垂发香波 从一个网页。当我执行下面的脚本时,我得到以下错误:

    注意:正在尝试获取C:\xampp\htdocs\runcode\testfile中非对象的属性“nodeValue”。php第16行。

    Link to that site

    我尝试过的脚本:

    <?php
        function get_content($url){
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_URL, $url);
            curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0');
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
            curl_exec($ch);
            $htmlContent = curl_exec($ch);
            curl_close($ch);
            return $htmlContent;
        }
        $link = "https://www.purplle.com/search?q=hair%20fall%20shamboo"; 
        $xml = get_content($link);
        $dom = @DOMDocument::loadHTML($xml);
        $xpath = new DOMXPath($dom);
        $title = $xpath->query('//h1[@class="br-hdng"]/span')->item(0)->nodeValue;
        echo "{$title}";
    ?>
    

    我的预期产出是:

    hair fall shamboo
    

    虽然 xpath 我在上面的脚本中使用的似乎是正确的,我在这里粘贴了html元素的相关部分,其中 title 可以找到:

    <h1 _ngcontent-c0="" class="br-hdng"><span _ngcontent-c0="" class="pr dib">hair fall shamboo<!----></span></h1>
    

    附言: 这个 标题 我希望解析动态加载的数据。由于我是php新手,我不知道我尝试的方法是否准确。如果不是,那我该怎么办?

    以下是我用两种不同的语言创建的脚本,它们工作起来很神奇。

    我成功地使用了 javascript :

    const puppeteer = require('puppeteer');
    function run () {
        return new Promise(async (resolve, reject) => {
            try {
                const browser = await puppeteer.launch();
                const page = await browser.newPage();
                await page.goto("https://www.purplle.com/search?q=hair%20fall%20shamboo");
                let urls = await page.evaluate(() => {
                let items = document.querySelector('h1.br-hdng span');
                return items.innerText;;
                })
                browser.close();
                return resolve(urls);
            } catch (e) {
                return reject(e);
            }
        })
    }
    run().then(console.log).catch(console.error);
    

    再一次,我成功地使用了 python :

    import requests_html
    
    with requests_html.HTMLSession() as session:
        r = session.get('https://www.purplle.com/search?q=hair%20fall%20shamboo')
        r.html.render()
        item = r.html.find("h1.br-hdng span",first=True).text
        print(item)
    

    怎么了 php 然后

    2 回复  |  直到 8 年前
        1
  •  7
  •   Decent Dabbler    8 年前

    很可能您的代码中存在的问题比我在本回答中提到的更多,但我看到的最突出的问题是:

    DOMDocument::loadHTML() 不是静态方法,而是实例方法(返回布尔值)。您应该首先创建 DOMDocument 然后打电话 loadHTML() 在这种情况下:

    $dom = new DOMDocument;
    $dom->loadHTML($xml);
    

    但是,由于您已使用 @ 接线员在那条线上,你没有收到关于这件事的警告。虽然错误抑制器操作符 @ 用于抑制HTML验证错误,例如,您应该研究使用 libxml_use_internal_errors() 1. 相反,因为这不会抑制一般的PHP错误。

    $dom = new DOMDocument;
    $oldSetting = libxml_use_internal_errors(true);
    $dom->loadHTML($xml);
    libxml_use_internal_errors($oldSetting);
    

    最后,请注意:
    可以直接从URL加载DOM文档(无需 cURL )与 DOMDocument::loadHTMLFile() ,如果您的PHP安装配置为允许通过配置设置加载URL allow_url_fopen 。请注意,由于安全原因,此设置通常会被禁用,因此,如果打算使用,请小心使用。


    下面是一个简单的测试用例,它应该按照预期工作:

    <?php
    
    $html = '
    <html>
    <head>
      <title>DOMDocument test-case</title>
    </head>
    <body>
      <div class="dummy-container">
        <h1 _ngcontent-c0="" class="br-hdng"><span _ngcontent-c0="" class="pr dib">hair fall shamboo<!----></span></h1>
      </div>
    </body>';
    
    $dom = new DOMDocument;
    
    $oldSetting = libxml_use_internal_errors(true);
    $dom->loadHTML( $html );
    libxml_use_internal_errors($oldSetting);
    
    $xpath = new DOMXPath( $dom );
    $title = $xpath->query( '//h1[@class="br-hdng"]/span' )->item( 0 )->nodeValue;
    echo $title;
    

    See this example interpreted online on 3v4l.org

    你应该替换文件的内容 $html 用你的 get_content() 呼叫如果不起作用,那么:

    1. 使用获取HTML时出现问题 卷曲 (是吗 var_dump( $html ); 在装载到 DOMDocument ,例如,查看检索到的内容),或。。。

    2. 也许您是在一个名称空间中工作,在这种情况下,您应该在使用前预先加上反斜杠 DOMDocument 和 DOMXPath ,即: new \DOMDocument; 和 new \DOMXPath( $dom ); .


    1.LibXML是DOMDocument用来解析XML/HTML文档的XML库。

        2
  •  5
  •   hanshenrik    7 年前

    那么php怎么了?

    php不运行javascript。大概 puppeteer 来自javascript代码的请求和来自python代码的html请求都运行javascript。

    您的问题是,此页面加载 br-hdng 标题及;使用javascript的产品,它根本不是HTML的一部分。实际上都是从 https://www.purplle.com/api/shop/itemsv3 ,带有一系列GET参数。您需要在这里进行JSON解析,而不是HTML解析:),但在访问该api之前,您需要搜索页面提供的cookie,以及 搜索字符串必须与api搜索字符串匹配 (否则api只会返回错误),请检查以下内容:

    <?php
    declare(strict_types = 0);
    header ( "Content-Type: text/plain;charset=UTF-8" );
    $ch = curl_init ();
    curl_setopt_array ( $ch, array (
            CURLOPT_ENCODING => '',
            CURLOPT_COOKIEFILE => '', // enables cookie handling without saving them anywhere. this page requires cookie handling.
            CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0', // 'libcurl/? PHP/' . PHP_VERSION, // many websites block requests without a useragent
            CURLOPT_RETURNTRANSFER => 1 
    ) );
    // we don't care what's on this page, we just need to fetch it to create a cookie session.
    $search_query = 'hair fall shamboo';
    curl_setopt ( $ch, CURLOPT_URL, 'https://www.purplle.com/search?q=' . rawurlencode ( $search_query ) );
    curL_exec ( $ch );
    $url = 'https://www.purplle.com/api/shop/itemsv3?' . http_build_query ( array (
            'list_type' => 'search',
            'custom' => '',
            'list_type_value' => $search_query,
            'page' => '1',
            'sort_by' => 'rel',
            'elite' => '0' 
    ) );
    // $url = 'https://www.purplle.com/api/shop/itemsv3?list_type=search&custom=&list_type_value=hair%20fall%20shamboo&page=1&sort_by=rel&elite=0';
    // $out = tmpfile ();
    // curl_setopt_array ( $ch, array (
    // CURLOPT_HTTPHEADER => array (
    // 'Accept: application/json, text/plain, */*',
    // 'Accept-Language: en-US,en;q=0.5',
    // 'Referer: https://www.purplle.com/search?q=hair%20fall%20shamboo',
    // // Cookie: __cfduid=d3199415b5ce18cbff2779802b1f843331544901552; csrftoken=f8f18b5deae92972f63343e13c6a460b; purpllesession=hedxkc%2FkdGye%2BYi6ebmJktUN1LeqA5rdVXu96%2F0j0yqtP2xZ8LfwpK8daXqPSkeZulO9ZvqpMYXTmY8oMD03VcG9vdKGBm30R9fU%2FQygtXBFhZvfvsu0scyaL3FqHbePp2zG45MevWU961eg82KAkCuHk0qFM8URQBRyYV5gg8TeqnTPgI3tF87H5nJ%2BmfO4pn%2BRWmIuWXvgNXAO%2F8GEaH6lJVl17QZm9c5vwi10OYeLfmSdIMy6V2Pp0ZjLTzuFw2de5jpR0zsbHHKZ0C2e548PiDl3taHIE5wuZO4HYIeXUqTpE98%2Fo3kztoU1bTlXGZgu%2FxVQ3EWLRFWQ2t57UawA%2FuERlD8vvOyFGbYHGAWVxgFTR%2FObAhFLHns5kqoj; _autm30d=null; visitorppl=NZ5tqQpGlFYWg2MrDl1302113161544901552; session_initiated=Direct; _tmpsess=1; token=desktop_5c1553b07c61c_7955_16122018; __uzma=5c1553b085a480.63440826; __uzmb=1544901552; __uzmc=632121030774; __uzmd=1544901552
    // 'Connection: keep-alive'
    // ),
    // // CURLOPT_CONNECT_TO=>array('www.purplle.com:443:dumpinput.ratma.net:80'),
    // CURLOPT_STDERR => $out,
    // CURLOPT_VERBOSE => 1
    // ) );
    // var_dump ( $url );
    curl_setopt ( $ch, CURLOPT_URL, $url );
    $json = curl_exec ( $ch );
    $data = json_decode ( $json, true );
    
    // var_dump ($json, $data );
    $title = $data ['list_title'];
    echo 'title: ' . $title . "\n";
    foreach ( $data ['items'] as $item ) {
        echo "name: ", $item ['name'], "\n";
    }
    

    输出:

    title: hair fall shamboo
    name: VLCC Hair fall Shampoo 350 ML (Buy1 Get1) & Ayurveda Hair Oil Combo (470 ml)
    name: Biotique Bio Kelp Protein Shampoo For Falling Hair (190 ml)
    name: Biotique Fresh Texture Shampoo - Bio Henna Leaf (120 ml)
    name: Good Vibes Scalp Purifying Shampoo -Neem And Aloe Vera (200 ml)
    name: Khadi Shikakai Sat Hair Cleanser Scalp Therapy (210 ml) By Swati Gramodyog
    name: Good Vibes Apple Cider Vinegar Shampoo (120 ml)
    name: Good Vibes Refreshing Shampoo - Green Apple (200 ml)
    name: Good Vibes Hydrating Shampoo -Marigold (200 ml)
    name: Alps Goodness Smoothening Shampoo - Keratin (50 ml)
    name: Alps Goodness Softening Shampoo - Coconut & Almond (50 ml)
    name: Alps Goodness Split End Control Shampoo - Coconut, Garlic & Shea Butter (50 ml)
    name: Passion Indulge Papain Shampoo & Conditioner For Soft & Shiny Hair (200 ml + 100 ml)
    name: Good Vibes Apple Cider Vinegar Shampoo (200 ml)
    name: Alps Goodness Split End Control Shampoo - Coconut, Garlic & Shea Butter (200 ml)
    name: Alps Goodness Nourishing Shampoo - Argan Oil & Olive (200 ml)
    name: Alps Goodness Moisturizing Shampoo - Ginger & Egg (200 ml)
    name: Alps Goodness Conditioning Shampoo - Pure Honey (200 ml)
    name: Alps Goodness Hydrating Shampoo - Tea Tree (200 ml)
    name: Alps Goodness Smoothening Shampoo - Keratin (200 ml)
    name: Alps Goodness Softening Shampoo - Coconut & Almond (200 ml)
    name: Good Vibes Scalp Purifying Shampoo -Neem And Aloe Vera (120 ml)
    name: Good Vibes Hydrating Shampoo - Marigold (120 ml)
    name: Alps Goodness Conditioning Shampoo - Pure Honey (50 ml)
    name: Alps Goodness Moisturizing Shampoo - Ginger & Egg (50 ml)
    
    推荐文章