代码之家 › 专栏 › 技术社区 › Steven

在火花中排列的字符串

pyspark apache-spark python

0

Steven · 技术社区 · 8 年前

我在pyspark中有一个带有值的字符串列的数据帧 [{"AppId":"APACON","ExtId":"141730"}] (字符串与我的列中的字符串完全相同,它是字符串,而不是数组)

我想把它转换成一个结构数组。

我可以简单地使用本机spark函数,还是必须解析字符串或使用udf?

sqlContext.createDataFrame(
    [ (1,'[{"AppId":"APACON","ExtId":"141730"}]'),
      (2,'[{"AppId":"APACON","ExtId":"141793"}]'),
    ],
    ['idx','txt']
).show()

+---+--------------------+
|idx|                 txt|
+---+--------------------+
|  1|[{"AppId":"APACON...|
|  2|[{"AppId":"APACON...|
+---+--------------------+

1 回复 | 直到 8 年前

1

0

plalanne 8 年前

火花2.1或以上

import pyspark.sql.functions as F
from pyspark.sql.types import *

df = sqlContext.createDataFrame(
    [ (1,'[{"AppId":"APACON","ExtId":"141730"}]'),
      (2,'[{"AppId":"APACON","ExtId":"141793"}]'),
    ],
    ['idx','txt']
)

您确实可以使用pyspark.sql.functions.from_json,如下所示:

schema = StructType([StructField("AppId", StringType()),
                     StructField("ExtId", StringType())])
df = df.withColumn('array',F.from_json(F.col('txt'), schema))
df.show()

+---+--------------------+---------------+
|idx|                 txt|          array|
+---+--------------------+---------------+
|  1|[{"AppId":"APACON...|[APACON,141730]|
|  2|[{"AppId":"APACON...|[APACON,141793]|
+---+--------------------+---------------+

# Use regexp_extract to ignore square brackets
df.withColumn('txt_parsed',F.regexp_extract(F.col('txt'),'[^\\[\\]]+',0))
df.show()

+---+-------------------------------------+-----------------------------------+
|idx|txt                                  |txt_parsed                         |
+---+-------------------------------------+-----------------------------------+
|1  |[{"AppId":"APACON","ExtId":"141730"}]|{"AppId":"APACON","ExtId":"141730"}|
|2  |[{"AppId":"APACON","ExtId":"141793"}]|{"AppId":"APACON","ExtId":"141793"}|
+---+-------------------------------------+-----------------------------------+

df = df.withColumn('AppId', F.get_json_object(df.txt, '$.AppId'))
df = df.withColumn('ExtId', F.get_json_object(df.txt, '$.ExtId'))
df.show()


+---+--------------------+------+------+
|idx|                 txt| AppId| ExtId|
+---+--------------------+------+------+
|  1|{"AppId":"APACON"...|APACON|141730|
|  2|{"AppId":"APACON"...|APACON|141793|
+---+--------------------+------+------+