What is the process for reading data in Kafka?
The process of reading data for Kafka is as follows:
- Kafka consumers send fetch requests to Kafka brokers in order to retrieve the latest message records.
- Once Kafka broker receives a pull request, it will check the consumer’s offset and then return the message record corresponding to that offset to the consumer.
- After receiving the message log, consumers process the message and perform corresponding business logic operations.
- Consumers send offset commit requests to the Kafka broker, requesting to submit the offset of processed messages to the Kafka broker.
- After receiving offset commit requests, the Kafka broker will save the committed offsets to a specific topic (__consumer_offsets) for tracking consumer offsets.
- When consumers send a pull request again, the Kafka broker will determine the position of the message record to return based on the offset submitted by the consumer, and return the corresponding message record to the consumer.
- Repeat the above steps to achieve continuous message consumption.