如何为不泄漏内存的大型发电机编写耗电元件?

TL/DR: Memory usage with concurrent.futures.ThreadPoolExecutor in Python3

我知道问题在于主函数不返回,并且它内部创建的对象没有被垃圾收集器清除。

我的主要问题是:是否可以为返回的生成器编写一个消费者,以便清理数据?或者我应该直接调用垃圾收集器实用程序吗?

# thread pool executor like in python documentation example
def table_process(callable, total):
    with ThreadPoolExecutor(max_workers=threads) as e:
    future_map = {
        e.submit(callable, i): i
        for i in range(total)
    }

    for future in as_completed(future_map):
        if future.exception() is None:
            yield future.result()
        else:
            raise future.exception()

@argh.dispatch_command
def main():
    threads = 10
    data = pd.DataFrame(...)  # about 12K rows

    # this function routes only one slice of sources/destinations
    def _process_chunk(x:int) -> gpd.GeoDataFrame:
        # slicing is more complex, but simplified here for presentation
        # do cross-product and an http request to process the result
        result_df = _do_process(grid[x], grid)
        return result_df

    # writing to geopackage
    with fiona.open('/tmp/some_file.gpkg', 'w', driver='GPKG', schema=...) as f:
        for results_df in table_process(_process_chunk, len(data)):
            aggregated_df = results_df.groupby('...').aggregate({...})
            f.writerecords(aggregated_df)