kafka_docker

Kafka Setup with Docker

Setup

Download and start Docker containers:

docker-compose up

For more details check Confluent Kafka.

CMD interface

List topics:

docker exec broker kafka-topics --list --bootstrap-server localhost:29092

Create two new topics users and pageviews:

docker exec broker kafka-topics --bootstrap-server localhost:29092 --create --replication-factor 1 --partitions 2 --topic users
docker exec broker kafka-topics --bootstrap-server localhost:29092 --create --replication-factor 1 --partitions 2 --topic pageviews

Check topic details

docker exec broker kafka-topics --bootstrap-server localhost:29092 --describe --topic users,pageviews

Create a consumer in a new terminal

docker exec broker kafka-console-consumer --bootstrap-server localhost:29092 --topic users

Create a producer in a new terminal

docker exec -it broker kafka-console-producer --bootstrap-server localhost:29092 --topic users

and type

> first message
> second message

You should be able to see the above messages at the consumer terminal.

The data generated by Kafka is stored in the folder /var/lib/kafka/data.

Examples with different programming languages

• Python example
• Java example