代码之家 › 专栏 › 技术社区 › Alihossein shahabi

kafka confluent群集中组my_group_id的自动偏移提交失败

kafka-python apache-kafka

Alihossein shahabi · 技术社区 · 6 年前

我有一个 融合的卡夫卡 kafka-python 用于连接到 Kafka 和消费。

我的 index.py 文件:

cid = "_" + str(randint(1, 99999999))

consumer = KafkaConsumer('test', bootstrap_servers=['109.XXX.XXX.XXX:9092', '5.XXX.XXX.XXX:9092', '88.XXX.XXX.XXX:9092'], group_id='my_group_id', client_id=cid, auto_offset_reset='earliest')
for msg in consumer:
    logger.warning(msg.value)

我负责 索引.py 文件三次,所以我有一个有3个消费者的消费者组,每个消费者在一个分区中接收消息,一切都很好。

当消息在 Topic 2 使用者遇到此警告,并且只有一个使用者连接到主题。

警告信息:

[2018-07-10 12:32:38] (kafka.coordinator.consumer) WARNING : Auto offset commit failed for group my_group_id: CommitFailedError: Commit cannot be completed since the group has already
        rebalanced and assigned the partitions to another member.
        This means that the time between subsequent calls to poll()
        was longer than the configured max_poll_interval_ms, which
        typically implies that the poll loop is spending too much
        time message processing. You can address this either by
        increasing the rebalance timeout with max_poll_interval_ms,
        or by reducing the maximum size of batches returned in poll()
        with max_poll_records.