代码之家 › 专栏 › 技术社区 › Limnic

正在侦听重新分区事件?

hazelcast-imap hazelcast

Limnic · 技术社区 · 7 年前

我正在试验hazelcast的分布图,以基本上将元素分布在同一应用程序的几个实例中。其想法是首先启动一个应用程序,然后填充地图。当然,本地密钥集将是完整的映射。

当另一个实例加入集群时,映射将被重新分区,这样两个实例都有大约一半(我假设)映射条目作为它们的本地键集。

Config hzConfig = new Config("hz");
HazelcastInstance hzInstance = HazelcastInstanceFactory.newHazelcastInstance(hzConfig);
IMap<Long, Long> test = hzInstance.getMap("test");
LocalListener listener = new LocalListener();
test.addLocalEntryListener(listener);
test.addPartitionLostListener(listener);

我以为重新分区会调用本地条目侦听器,但显然它不会调用,而且实际上也不会调用任何其他侦听器。

具体的用例是,第一个应用程序填充映射,当其他应用程序连接时,条目会分散在它们上面,对于它们本地键集中的每个条目,它们必须做些什么。与其定期检查本地密钥集,我更愿意在发生诸如从本地密钥集添加/删除之类的事件时进行检查。

但是,如果一个实例加入或离开集群,重新分区就会发生(我希望总是这样),但是我似乎听不到哪种方式会破坏目标。

使用上面的配置和下面的监听器,我有一个测试应用程序,它每隔10秒向地图中放入一个随机的长文件。

private static class LocalListener implements EntryAddedListener<Long, Long>, EntryRemovedListener<Long, Long>,
        EntryUpdatedListener<Long, Long>, EntryEvictedListener<Long, Long>, MapClearedListener, MapPartitionLostListener {
    @Override
    public void entryAdded(EntryEvent<Long, Long> event) {
        LOG.info("An entry was added to the local set: {}", event.getValue());
    }
    @Override
    public void entryRemoved(EntryEvent<Long, Long> event) {
        LOG.info("An entry was removed from the local set: {}", event.getValue());
    }
    @Override
    public void entryEvicted(EntryEvent<Long, Long> event) {
        LOG.info("An entry was evicted from the local set: {}", event.getValue());
    }
    @Override
    public void entryUpdated(EntryEvent<Long, Long> event) {
        LOG.info("An entry was updated in the local set: {}", event.getValue());
    }
    @Override
    public void mapCleared(MapEvent event) {
        LOG.info("The map was cleared: {}", event);
    }
    @Override
    public void partitionLost(MapPartitionLostEvent event) {
        LOG.info("A partition was lost: {}", event);
    }
}

第一个测试实例的输出:

15:43:47.718 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -1012665372499231549
15:43:47.858 [hz.hz.event-4] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -1012665372499231549
15:43:57.716 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -5501878816285329759
15:43:57.717 [hz.hz.event-1] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -5501878816285329759

然后我启动第二个实例,它加入集群。

实例1输出:

INFO: [172.20.20.7]:5701 [dev] [3.9.3] Re-partitioning cluster data... Migration queue size: 271
15:44:12.137 [hz.hz.event-4] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -642323604672752630
jan 10, 2019 3:44:12 PM com.hazelcast.internal.partition.impl.MigrationThread
INFO: [172.20.20.7]:5701 [dev] [3.9.3] All migration tasks have been completed, queues are empty.
15:44:17.716 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -2929992218325845758
15:44:17.718 [hz.hz.event-2] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -2929992218325845758
15:44:27.716 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -7717112084150209257
15:44:27.717 [hz.hz.event-3] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -7717112084150209257
15:44:37.716 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -3756253634059275245
15:44:37.717 [hz.hz.event-3] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -3756253634059275245
15:44:47.716 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry 9175632974694161488

实例2输出:

15:44:12.131 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -642323604672752630
15:44:22.130 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -785281121378041075
15:44:22.136 [hz.hz.event-1] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -785281121378041075
15:44:32.130 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry 3465608643988715362
15:44:32.132 [hz.hz.event-1] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: 3465608643988715362
15:44:42.131 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry 1474484225334222922
15:44:42.133 [hz.hz.event-1] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: 1474484225334222922
15:44:47.719 [hz.hz.event-1] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: 9175632974694161488
15:44:52.130 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -4535267276695561636
15:44:52.131 [hz.hz.event-2] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -4535267276695561636

然后关闭第二个实例以触发重新分区。

实例1输出:

INFO: [172.20.20.7]:5701 [dev] [3.9.3] Partition balance is ok, no need to re-partition cluster data... 
jan 10, 2019 3:45:03 PM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
INFO: [172.20.20.7]:5701 [dev] [3.9.3] Invocations:1 timeouts:0 backup-timeouts:1
15:45:07.716 [pool-1-thread-1] INFO com.example.playground.DistributedMapTests - Adding new entry -4645280647407966219
15:45:07.716 [hz.hz.event-5] INFO com.example.playground.DistributedMapTests - An entry was added to the local set: -4645280647407966219

正如所料,当应用程序是单独的时,它拥有所有的条目,当另一个实例连接时,会发生重新分区,但是,第二个实例不知道它现在在本地键集中有更多的元素,直到发生另一个Put。

另外,当第二个实例离开时,出于某种原因没有重新分区,所以我不知道它在本地密钥集中的条目发生了什么。

所以,tl;dr:我想知道如何倾听再分配事件。也许在哈泽尔卡特还有其他的选择吗?

更新 :

通过进一步的测试,尽管说分区平衡正常,但离开集群的条目确实返回到其他实例。我假设消息意味着它不需要重新分配给不同的成员,因为只剩下一个成员。

2 回复 | 直到 7 年前

sertug 7 年前

本地条目侦听器仅由用户操作触发。每当将成员添加到群集中或从群集中删除成员时,都会重新分区。如果您的集群大小仍然是>1,则会发生迁移,您可以使用 migration listeners .

o如果一个成员不礼貌地离开集群,那么剩下的成员将从自己的备份中恢复它的数据。这可能就是你看到的。

然而,我不明白为什么您需要倾听迁移事件并根据它们采取行动。它与您的业务逻辑相关吗?否则,您不必担心密钥的位置,只需使用密钥将任何业务逻辑发送到集群即可。请详细说明我是否遗漏了什么。

Limnic 7 年前

听一个再分配事件是错误的想法。当成员身份发生更改时,会重新分区。倾听成员资格的变化以及条目的变化将涵盖所有需要的事件。