PySpark Streaming Vs Structured Streaming

Spark Streaming	Structured Streaming
Spark 1.X	Introduced in Spark 2.X
Separate library in Spark to process continuously flowing Streaming data	Built on Spark SQL library
Uses DStreams API powered by Spark RDDs. It works on micro batches (each batch represent RDDs)	This model is based on Dataframe and Dataset APIs. No batch concept here.
DStrams provide us data divided into chunks as RDDs received from source of Streaming to be processed and outputs batches of processed data	Here we keep adding stream data to DataFrame (Unbounded table)
Not easy to apply	We can easily apply SQL query or scala operations on streaming data
	Result of Unbounded table/dataframe is based on mode of your operations Complete, Append, Update
RDD	Dataframe/Dataset are more optimized & less time consuming, easy to understand. Apply aggregations
No such option called event-time, only works with timestamp when the data is received. Based on the ingestion timestamp, Spark Streaming puts the data in a batch even if the event is generated early and belonged to the earlier batch, which may result in less accurate information as it is equal to the data loss	(Windowing) With event-time handling of late data, Structured Streaming outweighs Spark Streaming.

How to Configure Zookeeper and Kafka?

How to Configure Kafka?

# Download Kafka

• https://kafka.apache.org/downloads.html
• Download from Binary Downloads

# Kafka ENV  

• export KAFKA_HOME=$HOME/Workspace/prabhath/personal/kafka_2.12-2.5.0 
• export PATH=$KAFKA_HOME/bin:$PATH

Zookeeper config:

• bin/zookeeper-server-start.sh
• bin/zookeeper-server-stop.sh
• config/zookeeper.properties --> Default port: 2181, dataDir: /tmp/zookeeper

Kafka Config:

• bin/kafka-server-start.sh
• bin/kafka-server-stop.sh
• config/server.properties --> Default port: 9092

1) Start zookeeper

• zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties

2) Start Kafka server

• kafka-server-start.sh $KAFKA_HOME/config/server.properties

3) Create a Kafka topic

• kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic first_kafka_topic
• kafka-topics.sh --list --zookeeper localhost:2181 consumer_offsets
• It lists first_kafka_topic

4) Start Kafka Producer

• kafka-console-producer.sh --broker-list localhost:9092 --topic first_kafka_topic
• <start typing data>

5) Start Kafka Consumer

• kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_kafka_topic --from-beginning