Spark Intro

• Spark
• Apache is unified analytics engine and large-scale data processing
• Latest version 2.4.5 Feb 2020
• Speed
• Apache Spark achieves high performance for both batch and streaming using state of the art DAG scheduler, query optimizer and physical execution engine
• Runs 100X times faster than Hadoop
• Ease of use
• Write applications quickly in Java, Scala, Python, R and SQL
• Generality
• Spark SQL
• Spark Streaming
• MLib
• GraphX
• Runs every where
• Spark runs on Hadoop, Apache Mesos, Kubernetes, standalone, or in the cloud. It can access diverse data sources.
• You can run Spark using its standalone cluster mode, on EC2, on Hadoop YARN, on Mesos, or on Kubernetes. 
• Access data in HDFS, Alluxio, Apache Cassandra, Apache HBase, Apache Hive, and hundreds of other data sources.

Deploy docker container on google cloud - Docker + Google Cloud + Kubernetes

Ref:

https://www.youtube.com/watch?v=xJL-guFRZHo&list=LLdQeOBlmn5GgEL9YH_68KPA&index=2&t=1s

Prerequisites:

• Setup Google Cloud (gcloud in your system)
• Have your container ready in Dockerhub

Steps:

• gcloud container clusters create kubecluster

#This below port 80 - should match the DockerFile (EXPOSE 80)

• kubectl run kubecluster --image=prabhathkota/test-docker:tag1 --port=80 --image-pull-policy=IfNotPresent

O/P:

deployment.apps "kubecluster" created


#Create a service object that exposes the deployment

• kubectl expose deployment kubecluster --type="LoadBalancer"

O/P:
service "kubecluster" exposed

• kubectl get services kubecluster

O/P:
NAME         TYPE             CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubecluster LoadBalancer 10.23.246.XXX 35.244.47.XXX 80:31607/TCP 3m

#Test
curl http://35.244.47.XXX:80

• Cleanup

kubectl delete services kubecluster
kubectl delete deployment kubecluster
gcloud container clusters delete kubecluster