I’m trying to set up 2 consumers with Rdkafka (C++) and try to have them be part of the same consumer group and consume from 2 different partitions of a topic.
However I’m struggling to understand how this is achieved: from my understanding, using a consumer group means not having to directly choose which partition each consumer consumes from, but rather I should just give the consumers the name of the topic and then they are told by Kafka DB how many partitions there are for said topic and then the partition assignment algorithm kicks in.
This is not happening though.
The code I’ve been using so far (where I explicitly tell each consumer which partitions to consume from) was akin to this:
std::vector<std::string> vTopics; std::vector<RdKafka::TopicPartition*> vTopicPartitions; vTopics.push_back("SomeTopicName"); RdKafka::TopicPartition::create("SomeTopicName", 0); mConsumer->subscribe(vTopics); mConsumer->assign(vTopicPartitions); //... mConsumer->consume(1000);
For the consumer group variant though, I’d assume that I should not be calling the "assign" method (the partitions should be assigned dynamically, right?). If I do that though, the consumer is unable to consume, it just times out.
Keep in mind that yes, I have set up the group name and the consumer unique id in the configuration.
Source: Windows Questions C++