AWS Kinesis

AWS Kinesis (3 types)

• Kinesis Streams
• Kinesis Firehose
• Kinesis Analytics

• Kinesis Streams
• Ingest and process streaming data with "custom applications"
• Producers put data into streams
• Consumers consume data and process them (fleet of instances)
• Shard
• It is a basic throughout unit of a stream
• Streaming data is in the form on Shards
• Default retention period is 1 day
• You can keep data till 7 days (168 hrs)
• Kinesis Firehose
• Capture, transform & load streaming data
• Deliver real time data to "AWS destinations" like S3, RedShift, ElasticSearch, Splunk etc.,
• You don't need to write applications or manage resources
• Using Lambda, you can transform data before delivering data
• You can compress/encrypt data there by saving storage cost and increasing security
• Scales automatically
• Kinesis Analytics
• Run SQL queries on Kinesis Streaming data
• Analyze and process streaming data from Kinesis Streaming/Firehose

Kinesis Limitations

• No of shards
• Upto 200
• Upto 500 in some regions
• No hard limit (AWS can lift off if u request)
• Data retention
• 1 day be default
• Upto 7 days
• Hard limit (AWS cannot lift off)
• Write limitations
• Shard throughput - upto 1 MB
• No of transactions - 1000 records/sec
• Read limitations
• One shard can return < 2 MB/sec
• No of transactions: 5 records/shard/sec