通过一些简短的测试,pyspark数据帧的列删除函数似乎不区分大小写,例如。
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
import sys
sparkSession = SparkSession.builder.appName("my-session").getOrCreate()
dff = sparkSession.createDataFrame([(10,123), (14,456), (16,678)], ["age", "AGE"])
>>> dff.show()
+---+---+
|age|AGE|
+---+---+
| 10|123|
| 14|456|
| 16|678|
+---+---+
>>> dff.drop("AGE")
DataFrame[]
>>> dff_dropped = dff.drop("AGE")
>>> dff_dropped.show()
++
||
++
||
||
||
++
"""
What I'd like to see here is:
+---+
|age|
+---+
| 10|
| 14|
| 16|
+---+
"""
是否有方法以区分大小写的方式删除数据帧列?(我在spark中看到过一些与此相关的评论。)
JIRA discussions
drop()
临时操作(
)).
