我正在使用pyspark,我想将结果写入JSON,但是当我使用
df.write.json("s3e://somepath")
然后我得到结果json:part-00000-sdfh837fjh-6f8a-44d1-b0bb-sdjfh9236dj-c000.json
创建df的命令如下:
import json
from pyspark.sql.functions import *
from pyspark.sql.types import *
rdd = sc.parallelize([(1,2,3),(4,5,6),(7,8,9)])
df = rdd.toDF(["a","b","c"])
resultrdd = df.rdd.map(lambda x: ({"x": {"y": x.a}, "xx" + "yy": {"yy" + "yy": x.b}}))
resultdf = resultrdd.toDF()
resultdf.write.json("s3e://mybucket/testingjson") # and the resulting files are binary and not json files. why? how can i fix it??
resultrdd.collect()
resultdf.printSchema()
当我打开文件时,
s3e://mybucket/testingjson
它们是二进制的,不能用文本编辑器打开。为什么会这样?如何让df.write.json创建实际的json文件?
注:打印方案如下:
root
|-- x: map (nullable = true)
| |-- key: string
| |-- value: long (valueContainsNull = true)
|-- xxyy: map (nullable = true)
| |-- key: string
| |-- value: long (valueContainsNull = true)
注意,如果我打印我得到的数据帧(以验证JSON包含的内容):
resultdf
[{'x': {'y': 1}, 'xxyy': {'yyyy': 2}},
{'x': {'y': 4}, 'xxyy': {'yyyy': 5}},
{'x': {'y': 7}, 'xxyy': {'yyyy': 8}}]