How do Kafka consumers manage message offsets?

Consumers in Kafka can manage message offsets in two ways: manually or automatically.

1. Manual management: Consumers can manually submit the offset of messages by invoking the commitSync or commitAsync methods. In manual management mode, consumers have the flexibility to decide when to submit the offset and which offset to submit.

The sample code is as follows:

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        // 处理消息
    }
    consumer.commitSync();
}

1. Automatic management: Consumers can set the enable.auto.commit parameter to true, allowing Kafka to automatically manage message offsets. In automatic management mode, Kafka will periodically automatically submit message offsets.

The sample code is as follows:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("enable.auto.commit", "true");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("test-topic"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        // 处理消息
    }
}

Consumers have the option to choose between manually managing or automatically managing message offsets based on their actual needs. Manual management offers more precise control, but requires consumers to write more code to handle offset submission. Automatic management is more convenient, but may result in message duplication due to periodic offset submission.