Kinesis Vs Kafka Vs SQS

Kinesis	Kafka
Messaging system	Messaging system
Stream/Shard	Topic/Partition
Proprietary	Open Source
No operational overload	Heavy operational overload
Stores upto 7 days (default 1 day)	Can store data indefinitely
Kinesis Client Library	Various Kafka clients
	Log compaction
Kinesis	SQS
Messaging system	Messaging system
AWS	AWS
No operational overload	No operational overload
State tracking by client	State tracking by SQS
Message can be read many times	Processed message is deleted
Unified log/stream processing	Balancing tasks among workers

How to Configure Zookeeper and Kafka?

How to Configure Kafka?

# Download Kafka

• https://kafka.apache.org/downloads.html
• Download from Binary Downloads

# Kafka ENV  

• export KAFKA_HOME=$HOME/Workspace/prabhath/personal/kafka_2.12-2.5.0 
• export PATH=$KAFKA_HOME/bin:$PATH

Zookeeper config:

• bin/zookeeper-server-start.sh
• bin/zookeeper-server-stop.sh
• config/zookeeper.properties --> Default port: 2181, dataDir: /tmp/zookeeper

Kafka Config:

• bin/kafka-server-start.sh
• bin/kafka-server-stop.sh
• config/server.properties --> Default port: 9092

1) Start zookeeper

• zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties

2) Start Kafka server

• kafka-server-start.sh $KAFKA_HOME/config/server.properties

3) Create a Kafka topic

• kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic first_kafka_topic
• kafka-topics.sh --list --zookeeper localhost:2181 consumer_offsets
• It lists first_kafka_topic

4) Start Kafka Producer

• kafka-console-producer.sh --broker-list localhost:9092 --topic first_kafka_topic
• <start typing data>

5) Start Kafka Consumer

• kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_kafka_topic --from-beginning