Tutorial by Examples

FlinkKafkaConsumer let's you consume data from one or more kafka topics. versions The consumer to use depends on your kafka distribution. FlinkKafkaConsumer08: uses the old SimpleConsumer API of Kafka. Offsets are handled by Flink and committed to zookeeper. FlinkKafkaConsumer09: uses the new ...

As explained in the docs, With Flink’s checkpointing enabled, the Flink Kafka Consumer will consume records from a topic and periodically checkpoint all its Kafka offsets, together with the state of other operations, in a consistent manner. In case of a job failure, Flink will restore the streami...

SimpleStringSchema: SimpleStringSchema deserializes the message as a string. In case your messages have keys, the latter will be ignored. new FlinkKafkaConsumer09<>(kafkaInputTopic, new SimpleStringSchema(), prop); JSONDeserializationSchema JSONDeserializationSchema deserializes json-form...

In kafka, each consumer from the same consumer group gets assigned one or more partitions. Note that it is not possible for two consumers to consume from the same partition. The number of flink consumers depends on the flink parallelism (defaults to 1). There are three possible cases: kafka pa...