代码之家 › 专栏 › 技术社区 › Nathan123

用于刮取结果的脚本似乎不起作用?

urllib pandas python

-1

Nathan123 · 技术社区 · 8 年前

这是我发现的一个脚本,用于使用grad cafe网站获取不同项目的研究生入学结果。然而,当我运行它来查找关于“政治学”的结果时,它指出我有以下错误

Traceback (most recent call last):
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 57, in <module>
    data = get_data()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 52, in get_data
    pages = get_pages()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 45, in get_pages
    n = find_n_pages()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 41, in find_n_pages
    reg = re.search('over\s([\d]*)\spages',html)
  File "C:\Users\lakna\AppData\Local\Programs\Python\Python36-32\lib\re.py", line 182, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object

我该如何着手解决这个问题?以下是我使用的代码

def get_page(i=0, keyword="Political Science"):
    time.sleep(10)
    if i==0:
        #To change subjects, you want to change the keyword to say biostatistics,
        #test it by searching on the site to make sure you get what you want.
        url = "http://thegradcafe.com/survey/index.php?q="+keyword+"*&t=a&o=&pp=250"
    else:
        url="http://thegradcafe.com/survey/index.php?q="+keyword+"*&t=a&pp=250&o=&p="+str(i)
    response = urlopen(url)
    html = response.read().decode('utf-8')
    return
def find_n_pages():
    html = get_page()
    reg = re.search('over\s([\d]*)\spages',html)
    return int(reg.groups()[0])

def get_pages():
    n = find_n_pages()
    print ("Getting",n,"pages.")
    pages = [get_page(i) for i in range(1,n+1)]
    return pages

def get_data():
    data=[]
    pages = get_pages()
    for page in pages:
        data=get_data_from_page(page,data)
    return data

1 回复 | 直到 8 年前

Andy Hayden 8 年前

你的 get_page 函数没有返回html,它没有返回任何html。

def get_page(i=0, keyword="Political Science"):
    ...
    html = response.read().decode('utf-8')
    return  # this is equivalent to return None (or not having this line at all)

应为:

def get_page(i=0, keyword="Political Science"):
    ...
    return response.read().decode('utf-8')

因此出现错误:

In [11]: re.search("", None)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-1858f7517272> in <module>()
----> 1 re.search("", None)

/Users/andy/.miniconda3/lib/python3.6/re.py in search(pattern, string, flags)
    180     """Scan through string looking for a match to the pattern, returning
    181     a match object, or None if no match was found."""
--> 182     return _compile(pattern, flags).search(string)
    183
    184 def sub(pattern, repl, string, count=0, flags=0):

TypeError: expected string or bytes-like object