apache-kafka Consumer Groups and Offset Management What is a Consumer Group

Example

As of Kafka 0.9, the new high level KafkaConsumer client is availalbe. It exploits a new built-in Kafka protocol that allows to combine multiple consumers in a so-called Consumer Group. A Consumer Group can be describes as a single logical consumer that subscribes to a set of topics. The partions over all topics are assigend to the physical consumers within the group, such that each patition is assigned to exaclty one consumer (a single consumer can get multiple partitons assigned). The indiviual consumers belonging to the same group can run on different hosts in a distributed manner.

Consumer Groups are identified via their group.id. To make a specific client instance member of a Consumer Group, it is sufficient to assign the groups group.id to this client, via the client's configuration:

Properties props = new Properties();
props.put("group.id", "groupName");
// ...some more properties required
new KafkaConsumer<K, V>(config);

Thus, all consumers that connect to the same Kafka cluster and use the same group.id form a Consumer Group. Consumers can leave a group at any time and new consumers can join a group at any time. For both cases, a so-called rebalance is triggered and partitions get reassigned with the Consumer Group to ensure that each partition is processed by exaclty one consumer within the group.

Pay attention, that even a single KafkaConsumer forms a Consumer Group with itself as single member.