代码之家 › 专栏 › 技术社区 › Raphael Roth

spark不检查自定义项/列类型吗?

apache-spark scala

2

Raphael Roth · 技术社区 · 8 年前

考虑以下Spark 2.1代码:

val df = Seq("Raphael").toDF("name")
df.show()    

+-------+
|   name|
+-------+
|Raphael|
+-------+

val squareUDF = udf((d:Double) => Math.pow(d,2))

df.select(squareUDF($"name")).show

+---------+
|UDF(name)|
+---------+
|     null|
+---------+

为什么我会 null ? 我期待着像 ClassCastException 因为我试图在Scala Double上映射字符串

"Raphael".asInstanceOf[Double]

java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Double

1 回复 | 直到 8 年前

1

3

Alper t. Turker 8 年前

很容易判断您是否检查了执行计划:

scala> df.select(squareUDF($"name")).explain(true)
== Parsed Logical Plan ==
'Project [UDF('name) AS UDF(name)#51]
+- AnalysisBarrier Project [value#36 AS name#38]

== Analyzed Logical Plan ==
UDF(name): double
Project [if (isnull(cast(name#38 as double))) null else UDF(cast(name#38 as double)) AS UDF(name)#51]
+- Project [value#36 AS name#38]
   +- LocalRelation [value#36]

== Optimized Logical Plan ==
LocalRelation [UDF(name)#51]

== Physical Plan ==
LocalTableScan [UDF(name)#51]

如您所见,Spark在应用UDF之前执行类型转换:

UDF(cast(name#38 as double))

SQL强制转换不会为类型兼容的强制转换引发异常。如果无法进行实际转换,则值未定义( NULL ). 如果类型不兼容:

Seq((1, ("Raphael", 42))).toDF("id", "name_number").select(squareUDF($"name_number"))
// org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(name_number)' due to data type mismatch: argument 1 requires double type, however, '`name_number`' is of struct<_1:string,_2:int> type.;;
// 
// at org.apache...

你会得到一个例外。

如果类型不兼容

其余部分包括:

if (isnull(cast(name#38 as double))) null

因为值为 null 从未调用udf。