如何只显示Spark数据框架中的相关列?

你可以尝试如下,了解更多信息, referred_question

scala> val df = Seq((1,2,null),(3,4,null),(5,6,null),(7,8,"9")).toDF("a","b","c")

scala> df.show
+---+---+----+
|  a|  b|   c|
+---+---+----+
|  1|  2|null|
|  3|  4|null|
|  5|  6|null|
|  7|  8|   9|
+---+---+----+

scala> val dfl = df.limit(3) //limiting the number of rows you need, in your case it is 20

scala> val col_names = dfl.select(dfl.columns.map(x => count(col(x)).alias(x)):_*).first.toSeq.zipWithIndex.filter(x => x._1.toString.toInt > 0).map(_._2).map(x => dfl.columns(x)).map(x => col(x)) // this will give you column names which is having not null values
col_names: Seq[org.apache.spark.sql.Column] = ArrayBuffer(a, b)

scala> dfl.select(col_names : _*).show
+---+---+
|  a|  b|
+---+---+
|  1|  2|
|  3|  4|
|  5|  6|
+---+---+

让我知道它是否适合你。

类似于Sathiyan的想法,但是在count()本身中使用columnname。

scala>  val df = Seq((1,2,null),(3,4,null),(5,6,null)).toDF("a","b","c")
df: org.apache.spark.sql.DataFrame = [a: int, b: int ... 1 more field]

scala> df.show
+---+---+----+
|  a|  b|   c|
+---+---+----+
|  1|  2|null|
|  3|  4|null|
|  5|  6|null|
+---+---+----+


scala> val notnull_cols = df.select(df.columns.map(x=>concat_ws("=",first(lit(x)),count(col(x)))):_*).first.toSeq.map(_.toString).filter(!_.contains("=0")).map( x=>col(x.split("=")(0)) )
notnull_cols: Seq[org.apache.spark.sql.Column] = ArrayBuffer(a, b)

scala> df.select(notnull_cols:_*).show
+---+---+
|  a|  b|
+---+---+
|  1|  2|
|  3|  4|
|  5|  6|
+---+---+

中间结果显示计数和列名

scala> df.select(df.columns.map(x=>concat_ws("=",first(lit(x)),count(col(x))).as(x+"_nullcount")):_*).show
+-----------+-----------+-----------+
|a_nullcount|b_nullcount|c_nullcount|
+-----------+-----------+-----------+
|        a=3|        b=3|        c=0|
+-----------+---------- -+-----------+


scala>