代码之家 › 专栏 › 技术社区 › xmar

Kafka流式消费群并发:仅从一个分区消费?

apache-kafka-streams amazon-ecs apache-kafka docker

xmar · 技术社区 · 7 年前

我有一个aws基础设施

3个Docker容器上的1个Kafka集群,运行在ECS上,使用EFS作为存储服务(为了简单起见)。
1个kafka streams应用程序集群,位于3个容器上。

有一个包含16个分区的源主题“事件”,即复制2。一个PAPI拓扑处理器“流处理器”产生输出到其他一些主题,并使用3个状态存储。

我可以通过kafka管理器看到数据被消耗,输出被生成到这些其他输出主题。很明显它起作用了(虽然很慢)。

但是通过 宾客/卡夫卡消费集团 ,我看得出来 只有一个分区同时被使用 随着时间的推移。在命令的分离、连续运行中,仅减少一个偏移。

首次执行:

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                          HOST            CLIENT-ID
events          6          -               4021552         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          13         5030392         5030541         149             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          2          7056462         7056462         0               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          8          671945          6046546         5374601         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          1          164123          3009191         2845068         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          12         1962842         11052506        9089664         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          5          -               4022059         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          0          -               4019992         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          4          -               5032053         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          11         5037439         5037584         145             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          15         1683056         5034689         3351633         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          7          164702          7052434         6887732         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          14         -               3011069         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          3          1927601         6044400         4116799         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          10         5031461         5031612         151             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          9          1686979         8052924         6365945         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer

第二次执行:只有分区8提高了它的偏移量。1、5或15分钟后,这个分区是唯一消耗的分区。

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                          HOST            CLIENT-ID
events          6          -               4021552         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          13         5030392         5030541         149             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          2          7056462         7056462         0               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          8          686685          6046546         5359861         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          1          164123          3009191         2845068         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          12         1962842         11052506        9089664         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          5          -               4022059         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          0          -               4019992         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          4          -               5032053         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          11         5037439         5037584         145             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          15         1683056         5034689         3351633         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          7          164702          7052434         6887732         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          14         -               3011069         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          3          1927601         6044400         4116799         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          10         5031461         5031612         151             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          9          1686979         8052924         6365945         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer

查看日志,只有一个实例同时打印日志。也就是说,如果一个在工作,另两个不在。

有什么问题吗?

Kafka&Kafka流版本1.1。

0 回复 | 直到 7 年前