代码之家  ›  专栏  ›  技术社区  ›  HJA24

从网页表中刮取值

  •  0
  • HJA24  · 技术社区  · 7 年前

    我想从以下网站检索匹配数据:

    https://understat.com/match/81

    我写了以下脚本:

    import sys
    import time 
    import os
    import io
    import csv
    
    from selenium import webdriver
    import selenium.webdriver.support.expected_conditions as ec
    import selenium.webdriver.support.ui as ui
    from selenium.common.exceptions import TimeoutException
    from selenium.webdriver.common.by import By
    
    driver = None
    cnx = None
    currentDir = sys.path[0]
    
    def scrap_understat():
       init_browser('firefox')
    
       for i in range(80, 10080):
          try:
             driver.get('https://understat.com/match/' + str(i))
             time.sleep(1)
             if try_find_Element(driver, By.CLASS_NAME, 'error-code') is not None:
                continue
    
             data = get_match_data()
             save_data(data)
    
          except Exception as ex:
             log_this(ex)
             print(str(ex))
    
       close_browser()
    
    
    def get_match_data():
       data = []
    
       teams = driver.find_elements(By.NAME, 'team')
    
       for team in teams:
           team.find_element(By.XPATH, 'following-sibling::*').click()
           time.sleep(1)
           players = driver.find_element(By.ID, 'match-rosters').find_element(By.TAG_NAME, 'tbody').find_elements(By.TAG_NAME, 'tr')
           for player in players:
               tds = player.find_elements(By.TAG_NAME, 'td')
               record = [tds[1].text, tds[2].text, tds[3].text, tds[4].text,
                      tds[5].text, tds[6].text, tds[7].text, tds[8].text, tds[9].text]
               data.append(record)
    
       return data
    
    
    def save_data(data):
    # save CSV
      csv_file_path = currentDir + '/output' + current_time + '.csv'
      file = None
      writer = None
      if not os.path.exists(csv_file_path):
         file = io.open(csv_file_path, 'w', newline='', encoding='ISO-8859-1')
         writer = csv.writer(file)
         writer.writerow(
            ['player', 'pos', 'min', 'sh', 'g', 'kp', 'a', 'xG',
             'xA'])
      else:
          file = io.open(csv_file_path, 'a', newline='', encoding='ISO-8859-1')
          writer = csv.writer(file)
      for record in data:
          writer.writerow(record)
          file.close()
    

    enter image description here

    因此,xG和xA列存在一个问题。我只想要lowerscript部分,而脚本获取 运输署 如何将脚本更改为只包含第一部分?通过检查页面元素,我看到调用了不需要的部分 子类

    第二个问题:如何将球队名称声明为变量(曼联/托特纳姆热刺) enter image description here

    2 回复  |  直到 7 年前
        1
  •  0
  •   Andersson    7 年前

    record = [tds[1].text, tds[2].text, tds[3].text, tds[4].text,
                      tds[5].text, tds[6].text, tds[7].text,
              driver.execute_script('return arguments[0].firstChild.textContent', tds[8]), 
              driver.execute_script('return arguments[0].firstChild.textContent', tds[9])]
    

    home = driver.find_element_by_xpath('//label[@for="team-home"]').text
    away = driver.find_element_by_xpath('//label[@for="team-away"]').text
    

    P.S.考虑使用 Waits time.sleep

        2
  •  0
  •   pguardiario    7 年前

    看起来您只需要删除 sup

    driver.execute_script("$('sup').remove()")