Gangmax Blog

Python Kafka code example

From here and here.

This is a code example that how to use “kafka-python” package to write Kafka producer/consumer.

First, start Kafka and create topics.

1
2
3
4
5
6
7
8
9
10
11
# 1. Start ZooKeeper.
bin/zookeeper-server-start.sh config/zookeeper.properties &
# 2. Start Kafka.
bin/kafka-server-start.sh config/server.properties &
# 3. Create 2 topics.
bin/kafka-topics.sh --create --zookeeper localhost:2181 \
--replication-factor 1 --partitions 1 --topic fast-messages
bin/kafka-topics.sh --create --zookeeper localhost:2181 \
--replication-factor 1 --partitions 1 --topic summary-markers
# 4. List the topics.
bin/kafka-topics.sh --list --zookeeper localhost:2181

The Python Kafka code example.

producer.py
1
2
3
4
5
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
for i in range(1000):
    producer.send('fast-messages', key=str.encode('key_{}'.format(i)),
                  value=b'some_message_bytes')
consumer.py
1
2
3
4
from kafka import KafkaConsumer
consumer = KafkaConsumer('fast-messages', bootstrap_servers='localhost:9092')
for message in consumer:
    print(message)

The code is much shorter and simpler than written in Java for the same purpose.

Comments