在中为超过500000个请求运行异步函数时,Asyncio+Aiohttp内存泄漏)(python)

我提出了很多从不同url获取数据的请求。随着我的python程序的运行,我的内存占用越来越多,运行大约十分钟后,我可以达到8G的内存。我也看到过类似的问题 Asyncio + Aiohttp Memory Leak when running async function in for loop (python) 但它不起作用。我的程序是这样工作的。(``是省略某些元素)

async def add_success_callback(future, callback):
    result = await future
    callback(result)

async def fetch(session,url):
    try:
        headers = {```}
        params = {```}
        jar = aiohttp.DummyCookieJar()
        session = aiohttp.ClientSession(cookie_jar=jar)
        async with session.get(url, headers=headers, params=params, verify_ssl=False) as resp:
            return await resp.text(), await resp.read()

    except Exception:
        print(f"url: {url} error happened:")

async def fetch_all(urls):
    sem = Semaphore(100)
    connector = aiohttp.TCPConnector(limit=100)
    async with aiohttp.ClientSession() as session:
        tasks = []
        for url in urls:
            async with sem:
                task = asyncio.create_task(fetch(session, url))         
                tasks.append(task)
        await asyncio.gather(*tasks)
        datas = await asyncio.gather(*tasks, return_exceptions=True)
        return datas

def get_result(data):
    text, content = data
    soup = BeautifulSoup(text, 'html.parser')
    name = soup.find('div', class_='name').text
    chupin = soup.find('div', class_="panel-wrapper", id="出品").text
    chupindict[name] = chupin



urls=[]
chupindict = {}
for i in range(0,500000):
      url = '```'
      urls.append(url)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(fetch_all(urls))
loop.close()

这让我很困扰,但我是一名大一新生,我试着搜索我能找到的任何信息。任何帮助都将不胜感激,谢谢!

我试过了 在for循环(python)中运行异步函数时,Asyncio+Aiohttp内存泄漏 [https://github.com/encode/httpx/issues/978][https://github.com/encode/httpx/issues/978]但对于SSL解决方案,我发现它可以与httpx一起使用,我不知道是否可以在我的程序中使用。 我只是想解决记忆问题,谢谢!