我一直在尝试读取3GB的gzip文件。我已经用gzip提取了它,但提取后它甚至无法容纳60GB的存储空间。所以,如果我不能提取哪个是JSON,那么我就不能以字节为单位读取。我发现了很多问题——所有这些都是下载的,然后提取成字节,然后读取成字节,但在这里我甚至无法提取它们。
那么,如果可以在下载时以字节为单位进行读取呢?我试过了,但错误显示需要下载整个文件。
正在下载代码:
default_path_download = '.'
def saveanddelete(download_url, name):
with requests.get(download_url, headers=headersfordowlnoading, stream=True) as r:
content_lenght = int(r.headers['Content-Length'])
print(f"Download File Size : {round(content_lenght/1000000, 5)} MB")
r.raise_for_status()
with open(f"{default_path_download}/{name}", 'wb') as f:
total_chunk_downloaded = 0
chunk_size_to_get = 4096
for chunk in r.iter_content(chunk_size=chunk_size_to_get):
total_chunk_downloaded += len(chunk)
if chunk:
f.write(chunk)
done = int(50 * total_chunk_downloaded / content_lenght)
print(f"Downloaded : ", end='',flush=True)
print(f"{round(total_chunk_downloaded/1000000, 5)} MB [{'=' * done}{' ' * (50-done)}]\r", end='', flush=True)
break
print("\n")
print("Download Complete")
你可以在上看到整个代码
colab
如果可能的话,我想用其他方法吗?
谢谢你的帮助!
