如何提高Sparklyr读取csv的性能?

我认为在csv文件中读取速度非常慢 sparklyr . 见MVE

library(sparklyr)
library(dplyr)

conf <- spark_config()
conf$spark.executor.memory <- "60GB"
conf$spark.memory.fraction <- 0.9
conf$spark.executor.cores <- 6
conf$spark.dynamicAllocation.enabled <- "false"
sc <- sparklyr::spark_connect(master = "local", config = conf)

library(data.table)

fwrite(data.table(
  id1 = sample(sprintf("id%03d",1:K), N, TRUE),      # large groups (char)
  id2 = sample(sprintf("id%03d",1:K), N, TRUE),      # large groups (char)
  id3 = sample(sprintf("id%010d",1:(N/K)), N, TRUE), # small groups (char)
  id4 = sample(K, N, TRUE),                          # large groups (int)
  id5 = sample(K, N, TRUE),                          # large groups (int)
  id6 = sample(N/K, N, TRUE),                        # small groups (int)
  v1 =  sample(5, N, TRUE),                          # int in range [1,5]
  v2 =  sample(5, N, TRUE),                          # int in range [1,5]
  v3 =  sample(round(runif(100,max=100),4), N, TRUE) # numeric e.g. 23.5749
), "a.csv")

system.time(sparklyr::spark_read_csv(sc, "a", "a.csv"))

我已经尝试增加可点燃的内存级别,但在500秒时读取速度太慢!这比 data.table::fread .

有没有配置火花使其更快?