用dask将单个16M行csv并行转换为拼花地板

我不确定是不是数据有问题。我在我的机器上做了一个玩具例子,同样的命令需要大约9秒。

import dask.dataframe as dd
import numpy as np
import pandas as pd
from dask.distributed import Client
import dask
client = Client()
# if you wish to connect to the dashboard
client

# fake df size ~2.1 GB
# takes ~180 seconds
N = int(5e6)
df = pd.DataFrame({i: np.random.rand(N) 
                   for i in range(22)})
df.to_csv("data.csv", index=False)

# the following takes ~9 seconds on my machine
dd.read_csv("data.csv").to_parquet("data_pq")