Python:urllib/urllib2/httplib

专注于 urllib2 对于这一点,它工作得相当好。别惹我 httplib ,它不是顶级API。

你注意到的是 不遵循重定向。

HTTPRedirectHandler 这将捕获并跟踪重定向。

此外,您可能希望对默认值进行子类化 HTTPRedirectHandler 捕获信息,然后作为单元测试的一部分进行检查。

cookie_handler= urllib2.HTTPCookieProcessor( self.cookies )
redirect_handler= HTTPRedirectHandler()
opener = urllib2.build_opener(redirect_handler,cookie_handler)

然后你可以用这个 opener

HTTPHandler 还可以捕获和记录各种错误代码。

以下是我对这个问题的看法。

#!/usr/bin/env python

import urllib
import urllib2


class HttpBot:
    """an HttpBot represents one browser session, with cookies."""
    def __init__(self):
        cookie_handler= urllib2.HTTPCookieProcessor()
        redirect_handler= urllib2.HTTPRedirectHandler()
        self._opener = urllib2.build_opener(redirect_handler, cookie_handler)

    def GET(self, url):
        return self._opener.open(url).read()

    def POST(self, url, parameters):
        return self._opener.open(url, urllib.urlencode(parameters)).read()


if __name__ == "__main__":
    bot = HttpBot()
    ignored_html = bot.POST('https://example.com/authenticator', {'passwd':'foo'})
    print bot.GET('https://example.com/interesting/content')
    ignored_html = bot.POST('https://example.com/deauthenticator',{})

data = urllib.urlencode(params)
url = host+page
request = urllib2.Request(url, data, headers)
response = urllib2.urlopen(request)

cookies = CookieJar()
cookies.extract_cookies(response,request)

cookie_handler= urllib2.HTTPCookieProcessor( cookies )
redirect_handler= HTTPRedirectHandler()
opener = urllib2.build_opener(redirect_handler,cookie_handler)

response = opener.open(request)

最近我不得不亲自做这件事。我只需要标准库中的类。以下是我的代码摘录:

from urllib import urlencode
from urllib2 import urlopen, Request

# encode my POST parameters for the login page
login_qs = urlencode( [("username",USERNAME), ("password",PASSWORD)] )

# extract my session id by loading a page from the site
set_cookie = urlopen(URL_BASE).headers.getheader("Set-Cookie")
sess_id = set_cookie[set_cookie.index("=")+1:set_cookie.index(";")]

# construct headers dictionary using the session id
headers = {"Cookie": "session_id="+sess_id}

# perform login and make sure it worked
if "Announcements:" not in urlopen(Request(URL_BASE+"login",headers=headers), login_qs).read():
    print "Didn't log in properly"
    exit(1)

# here's the function I used after this for loading pages
def download(page=""):
    return urlopen(Request(URL_BASE+page, headers=headers)).read()

# for example:
print download(URL_BASE + "config")

http://wwwsearch.sourceforge.net/mechanize/

尝试 twill 是用英文写的 Python python API ,例如:

from twill import get_browser
b = get_browser()

b.go("http://www.python.org/")
b.showforms()

除了您可能缺少cookie之外,表单中可能还有一些字段您没有发布到Web服务器。最好的方法是从web浏览器中捕获实际的帖子。你可以用 LiveHTTPHeaders WireShark 窥探流量并在脚本中模拟相同的行为。

Funkload 也是一个很棒的web应用程序测试工具。它包装webunit以处理浏览器模拟,然后在顶部为您提供功能和负载测试特性。