代码之家 › 专栏 › 技术社区 › ds_user

在apache spark中复制记录计数

spark-dataframe apache-spark-sql pyspark apache-spark scala

0

ds_user · 技术社区 · 8 年前

这是这个问题的延伸, Apache Spark group by combining types and sub types 。

val sales = Seq(
  ("Warsaw", 2016, "facebook","share",100),
  ("Warsaw", 2017, "facebook","like",200),
  ("Boston", 2015,"twitter","share",50),
  ("Boston", 2016,"facebook","share",150),
  ("Toronto", 2017,"twitter","like",50)
).toDF("city", "year","media","action","amount")

这种解决方案都很好,但是预期产出应该有条件地按不同类别计算。

所以,输出应该如下所示,

+-------+--------+-----+
| Boston|facebook|    1|
| Boston| share1 |    2|
| Boston| share2 |    2|
| Boston| twitter|    1|
|Toronto| twitter|    1|
|Toronto| like   |    1|
| Warsaw|facebook|    2|
| Warsaw|share1  |    1|
| Warsaw|share2  |    1|
| Warsaw|like    |    1|
+-------+--------+-----+

在这里,如果操作是share,我需要将其计入share1和share2。当我以编程方式计算时,我使用case语句,并在操作为share时使用case,share1=share1+1,share2=share2+1

但我如何在Scala、pyspark或sql中做到这一点?

1 回复 | 直到 8 年前

1

Ramesh Maharjan 8 年前

易于理解的 filter 和 unions 应该给你你想要的输出

val media = sales.groupBy("city", "media").count()

val action = sales.groupBy("city", "action").count().select($"city", $"action".as("media"), $"count")

val share = action.filter($"media" === "share")

  media.union(action.filter($"media" =!= "share"))
      .union(share.withColumn("media", lit("share1")))
      .union(share.withColumn("media", lit("share2")))
      .show(false)

这应该给你

+-------+--------+-----+
|city   |media   |count|
+-------+--------+-----+
|Boston |facebook|1    |
|Boston |twitter |1    |
|Toronto|twitter |1    |
|Warsaw |facebook|2    |
|Warsaw |like    |1    |
|Toronto|like    |1    |
|Boston |share1  |2    |
|Warsaw |share1  |1    |
|Boston |share2  |2    |
|Warsaw |share2  |1    |
+-------+--------+-----+