替换BroadcastedNestedLoopJoin

Dataset A: App specific data that contains (among other fields): ip_address

Dataset B: Location data that contains start_ip_address_int, end_ip_address_int, latitude, longitude

A.ip_address >= B.start_ip_address_int AND A.ip_address <= B.end_ip_address_int

Dataset A => Usually contains 110,000
Dataset B => Contains 12.5 Million rows. This is “static” data. Hasn’t changed since August 2020.

当我们看到 ,似乎是 广播嵌套循环联接 被认为是非常缓慢的习惯。似乎Spark在相同条件下(如大于、小于)默认选择它。

加快此过程的最佳方法是什么?