代码之家 › 专栏 › 技术社区 › Dsp guy sam

尝试使用Python请求读取NSE上市前网站时请求超时

python-requests python

-2

Dsp guy sam · 技术社区 · 1 年前

我正在尝试阅读NSE印度网站,但请求超时。我有下面的代码:

import requests

url = 'https://www.nseindia.com/market-data/pre-open-market-cm-and-emerge-market'

response = requests.get(url, headers={"Content-Type":"text"})

print(response.text)

我得到的错误:

 File "<string>", line 3, in raise_from
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1349, in getresponse
    response.begin()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 277, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 353, in _raise_timeout

缺少什么?我在MAC操作系统上运行这个

2 回复 | 直到 1 年前

Poojan 1 年前

我建议尽量尊重 robots.txt 无论何时报废。
这是一个有效的解决方案。您可以添加一些额外的标题,使其表现得像浏览器。

import requests

url_base = 'https://www.nseindia.com/market-data/pre-open-market-cm-and-emerge-market'


request_headers = {
    'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
}

res = requests.request('get', url_base, headers=request_headers)

print(res.status_code) # returns 200

web服务器有时不喜欢来自脚本/代码等来源的请求。在这种情况下,添加 User-Agent 工作,但可能不同的网站。

Shriniwas 1 年前

从我的角度来看,你正试图阅读NSE印度网站的内容。你面临请求超时是有原因的,因为网站不允许你阅读它的页面。在这里,在这篇reddit帖子上,告诉为什么你无法请求该网站。如果你想知道为什么会发生这种事,这个答案来自reddit的一篇帖子。 Reddit Link

你必须明白,网络正试图保护自己不受你的伤害。无论这是否是你的意图,从技术上讲,你都是一个恶意行为者。网站不想被刮——他们在robots.txt中这么说,你可能根本没看过——而且你无论如何都在试图刮它们。因此,你正在经历网站通过使用各种技巧来阻止你来保护自己的结果,包括挂断连接,使你的脚本停滞。他们寄希望于你对请求库不够了解,无法在自己的一端设置超时,到目前为止,他们是对的。

假设您正在阅读给定网站的内容。你缺少的是 User-Agent 在您的页眉中。

因此,您的代码看起来更像这样:

import requests

url = 'https://www.nseindia.com/market-data/pre-open-market-cm-and-emerge-market'

header = {
  "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
}
get_response = requests.get(url, headers=header)
print(get_response.status_code)
print(get_response.text)