代码之家 › 专栏 › 技术社区 › COLD ICE

将键值对与键映射对连接起来

apache-spark scala

0

COLD ICE · 技术社区 · 8 年前

数据集 :

(apple,1)
(banana,4)
(orange,3)
(grape,2)
(watermelon,2)

其他数据集 是:

(apple,Map(Bob -> 1))
(banana,Map(Chris -> 1))
(orange,Map(John -> 1))
(grape,Map(Smith -> 1))
(watermelon,Map(Phil -> 1))

我的目标是结合两组都要获得:

(apple,1,Map(Bob -> 1))
(banana,4,Map(Chris -> 1))
(orange,3,Map(John -> 1))
(grape,2,Map(Smith -> 1))
(watermelon,2,Map(Phil -> 1))

密码

...  
val counts_firstDataset = words.map(word => 
(word.firstWord, 1)).reduceByKey{case (x, y) => x + y}

第二个数据集:

...
val counts_secondDataset  = secondSet.map(x => (x._1,
x._2.toList.groupBy(identity).mapValues(_.size)))

join方法 val joined_data = counts_firstDataset.join(counts_secondDataset) 但不起作用,因为连接需要[K,V]对。我该如何回避这个问题?

2 回复 | 直到 8 年前

1

Glennie Helles Sindholt 8 年前

最简单的方法就是转换为 DataFrames 然后 join

import spark.implicits._
val counts_firstDataset = words
  .map(word => (word.firstWord, 1))
  .reduceByKey{case (x, y) => x + y}
  .toDF("type", "value")

val counts_secondDataset = secondSet
  .map(x => (x._1,x._2.toList.groupBy(identity).mapValues(_.size)))
  .toDF("type_2","map")

counts_firstDataset
  .join(counts_secondDataset, 'type === 'type_2)
  .drop('type_2)

2

1

Shahbaz Shueb 8 年前

由于两个列表的第一个元素(水果名称)的顺序相同,因此可以使用拉链地图按以下方式将列表更改为元组:

counts_firstDataset.zip(counts_secondDataset)
  .map(vk => (vk._1._1, vk._1._2, vk._2._2))