Retrieve always latest messages from Kafka on reconnection

  apache-kafka, c++, librdkafka

I’m writing a piece of code that needs to read hundreds of messages from Kafka each few milliseconds. I’m using C++ and librdkafka. When my program stops and then restarts it does not need to recover all lost messages since it was stopped, but instead it needs to always read from latest messages sent.

As far as I know I can manage consumer offsets by playing with enable.auto.commit and auto.offset.reset. But, the latter one is only useful when there are no committed offsets while the former one instead let me manage myself the offsets to store.

Playing with these two values I found that if I set enable.auto.commit to false, without committing any offset, and auto.offset.reset to latest it seems to retrieve always the latest messages; but how clean is this solution?

My fear is that if between two consumer polls there are 2 messages sent than my consumer only takes the latest, or if no messages where sent it continually reads the same. Both are unwanted behaviour.

Another idea was to clear consumer group offsets or seeking forward, but the seek method in librdkafka seems to not work as needed and I cannot find methods to manage consumer groups..

How can I always read latest messages from Kafka using librdkafka?

Source: Windows Questions C++

LEAVE A COMMENT