Python __init__ vs __call__

######################################
# __init__ vs __call__
# x = Foo(1, 2, 3) # __init__
# x = Foo()
# x(1, 2, 3) # __call__
######################################


class Foo:
    def __init__(self, a, b, c):
        self.a = a
        self.b = b
        self.c = c
        print('In Foo __init__ {}, {}, {}'.format(self.a, self.b, self.c))


class Bar:
    def __init__(self, a, b, c):
        self.a = a
        self.b = b
        self.c = c

    def __call__(self):
        print('In Bar __call__ {}, {}, {}' .format(self.a, self.b, self.c))


if __name__ == '__main__':
    print('--------------------------------------------------')
    f = Foo(100, 200, 300)
    print(f.a)
    #f()    #'Foo' object is not callable
    print('--------------------------------------------------')
    b = Bar(10, 20, 30)
    print(b.a)
    b()
    print('--------------------------------------------------')


"""
Output:

--------------------------------------------------
In Foo __init__ 100, 200, 300
100
--------------------------------------------------
10
In Bar __call__ 10, 20, 30
--------------------------------------------------
"""

Python Decorators

#########################
# A decorator is a design pattern in Python that allows a user to add new functionality to an existing object without modifying its structure.
# Decorators are usually called before the definition of a function you want to decorate.
#########################


# Decorated function
def escape_unicode(f):
    def wrap(*args, **kwargs):
        text = f(*args, **kwargs)
        return ascii(text)
    return wrap


def display_text(text):
    return ascii(text)


@escape_unicode
def display_text_ascii(text):
    return text


if __name__ == '__main__':
    # print('మీరు ఎలా ఉన్నారు?')
    # print(ascii('మీరు ఎలా ఉన్నారు?'))
    print(display_text('మీరు ఎలా ఉన్నారు?'))
    print(display_text_ascii('మీరు ఎలా ఉన్నారు?'))


"""
Output:

'\u0c2e\u0c40\u0c30\u0c41 \u0c0e\u0c32\u0c3e \u0c09\u0c28\u0c4d\u0c28\u0c3e\u0c30\u0c41?'
'\u0c2e\u0c40\u0c30\u0c41 \u0c0e\u0c32\u0c3e \u0c09\u0c28\u0c4d\u0c28\u0c3e\u0c30\u0c41?'
"""

Docker for Beginner

Docker Vs Virtual Machine

A container runs natively on Linux and shares the kernel of the host machine with other containers. It runs a discrete process, taking no more memory than any other executable, making it lightweight.

By contrast, a virtual machine (VM) runs a full-blown “guest” operating system with virtual access to host resources through a hypervisor. In general, VMs provide an environment with more resources than most applications need.

Docker commands:

docker info   

docker --version

docker-compose —version

docker-machine —version

#Execute Docker image

docker run hello-world

#list of running running containers

docker ps

#list docker images

docker image ls 

#list Docker containers

docker container ls

docker container ls --all

docker images

docker ps -l

You create a Dockerfle

###################

From this Dockerfile, you will create/build an image

docker build  --tag=friendlyhello  .

When you run a docker image, it will create a container for you (when u kill the running image, the running container is gone)

#Mapping container port to host/local_machine port

docker run -p 4000:80 friendlyhello

#4000 is the host port (locally) - http://localhost:4000/   (thats why we use 4000 port locally)

#80 is the container port

docker run -d -p 4000:80 friendlyhello   #run as a background daemon

docker run -d -p 4000:80 prabhathkota/test-docker:tag1

docker container ls

docker container stop 1fa4ab2cf395

docker stats <container_id>

docker logs <container_id>

docker cp <container_id>:/path/to/useful/file /local-path

Get inside the container:

####################

docker exec -it <containerid> bash

#curl http://localhost:4000

Docker hub - is like GitHub repository (to share)

####################################

Share your image

docker login

docker tag friendlyhello prabhathkota/test-docker:tag1

docker image ls

Remove image

#############

docker rmi -f 8b810fbdcf2d   #(forcefully remove image)

Publish the image:

##############

docker push prabhathkota/test-docker:tag1

Run image from remote repository

##########################

docker run -p 4000:80 prabhathkota/test-docker:tag1

docker build -t friendlyhello .  # Create image using this directory's Dockerfile

Run local repository

################

docker run -p 4000:80 friendlyhello  # Run "friendlyname" mapping port 4000 to 80

docker run -d -p 4000:80 friendlyhello         # Same thing, but in detached mode

Docker commands:

################

docker container ls                                # List all running containers

docker container ls -a             # List all containers, even those not running

docker container stop <hash>           # Gracefully stop the specified container

docker container kill <hash>         # Force shutdown of the specified container

docker container rm <hash>        # Remove specified container from this machine

docker container rm $(docker container ls -a -q)         # Remove all containers

docker image ls -a                             # List all images on this machine

docker image rm <image id>            # Remove specified image from this machine

docker image rm $(docker image ls -a -q)   # Remove all images from this machine

docker login             # Log in this CLI session using your Docker credentials

docker tag <image> username/repository:tag  # Tag <image> for upload to registry

docker push username/repository:tag            # Upload tagged image to registry

docker run username/repository:tag                   # Run image from a registry

Deploy docker container on google cloud - Docker + Google Cloud + Kubernetes

Ref:

https://www.youtube.com/watch?v=xJL-guFRZHo&list=LLdQeOBlmn5GgEL9YH_68KPA&index=2&t=1s

Prerequisites:

• Setup Google Cloud (gcloud in your system)
• Have your container ready in Dockerhub

Steps:

• gcloud container clusters create kubecluster

#This below port 80 - should match the DockerFile (EXPOSE 80)

• kubectl run kubecluster --image=prabhathkota/test-docker:tag1 --port=80 --image-pull-policy=IfNotPresent

O/P:

deployment.apps "kubecluster" created


#Create a service object that exposes the deployment

• kubectl expose deployment kubecluster --type="LoadBalancer"

O/P:
service "kubecluster" exposed

• kubectl get services kubecluster

O/P:
NAME         TYPE             CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubecluster LoadBalancer 10.23.246.XXX 35.244.47.XXX 80:31607/TCP 3m

#Test
curl http://35.244.47.XXX:80

• Cleanup

kubectl delete services kubecluster
kubectl delete deployment kubecluster
gcloud container clusters delete kubecluster

Run a docker in Amazon EC2

I created a docker sample in docker hub & tried to deploy in EC2 instance.

It’s super easy & simple.

yum install docker -y

sudo service docker start

docker run -p 4000:80 prabhathkota/test-docker:tag1

Test: 

http://ec2-18-209-XXX-XXX.compute-1.amazonaws.com:4000

Output:

Hello World!

Hostname: f156f8eeb177

Visits: cannot connect to Redis, counter disabled

Python Pickle/UnPickle Serialize/DeSerialize

#Python pickle module is used for serializing and de-serializing a Python objects.
#Any object in Python can be pickled so that it can be saved on disk.
#Pickling “serializes” the object first before writing it to file.
#Pickling will convert a python object into a character stream.


#Serializing - pickle
import pickle
emp = {"employees":[
    {"name":"Shyam", "email":"shyam@mail.com"},
    {"name":"Bob", "email":"bob32@mail.com"},
    {"name":"Jai", "email":"jai87@mail.com"}
]}
pickling_on = open("Emp.pickle","wb")
pickle.dump(emp, pickling_on)
pickling_on.close()


#De-Serializing - unpickle
pickle_off = open("Emp.pickle","rb")
emp = pickle.load(pickle_off)
print(emp)


#Output
{'employees': [{'name': 'Shyam', 'email': 'shyam@mail.com'}, {'name': 'Bob', 'email': 'bob32@mail.com'}, {'name': 'Jai', 'email': 'jai87@mail.com'}]}

Python Monkey Patching

Monkey Patching:

• Monkey Patching refers to dynamic (run-time) modifications of a class or module
• A MonkeyPatch is a piece of Python code which extends or modifies other code at runtime
• The unittest.mock library makes use of monkey patching to replace part of your code under test by mock objects. 
• It provides functionality for writing clever unittests

### test1.py 

class A: 

    def func(self): 

        print("func() is being called")

### test2.py

from test1 import A

def monkey_func(self): 

     print("monkey_func() is being called...")

   

# Replacing address of "func" with "monkey_func" 

A.func = monkey_func

if __name__ == '__main__':

    obj = A()

  

    # calling function "func" whose address got replaced with "monkey_func()" 

    obj.func() 

Output:

monkey_func() is being called...

Switching Java Versions on MacOS

Switching Java Versions on MacOS 

Add these in ~/.profile

alias java9="export JAVA_HOME=`/usr/libexec/java_home -v 9`; java -version"

alias java8="export JAVA_HOME=`/usr/libexec/java_home -v 1.8`; java -version"

alias java7="export JAVA_HOME=`/usr/libexec/java_home -v 1.7`; java -version"

Git commands

# Git Config

git init

git config user.email test@test.com

git config user.name 'Prabhath Kota'

git clone https://github.com/prabhathkota/Django-user-registration.git

Checkout Branch:

• git checkout -b test_branch
• git branch
• git push --set-upstream origin test_branch

git status -s #check if any modified files

modified:   common_utils.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
../api/log/
misc/test_rl.py

# stash without a name
git stash

# stash 
git stash save <name_of_stash>
stash@{0}: On develop: new_changes

# -u if you want to stash untracked files as well
# git stash save -u <name_of_stash>  # Dont use this

#git stash list
stash@{0}: On develop: new_changes

git status -s

# stash apply
git stash apply stash@{0}

# Takes the latest and apply
git stash pop

# Clear stash
git stash clear

# gitignore

Python Virtual Enviroment

Install python3 on top of python2 (using virtual environment)

python3 -m venv < python3env>

or

pip3 install virtualenv

virtualenv -p /usr/local/bin/python3 python3env

Activate:

    source python3env/bin/activate

    

    #Install modules within the virtual ENV

    pip install requests

    #pip Install requirements.txt, python3

    pip3 install -r requirements.txt

Deactivate:

python3env>> deactivate

java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

Error:

java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

Solution:
Both spark driver and executor need mysql driver on class path so specify

• spark.driver.extraClassPath = /usr/share/java/mysql-connector-java.jar 
• spark.executor.extraClassPath = /usr/share/java/mysql-connector-java.jar

#Initialize SparkSession and SparkContext

from pyspark.sql import SparkSession

from pyspark import SparkContext

#Create a Spark Session

SpSession = SparkSession \

    .builder \

    .master("local[2]") \

    .appName("prabhath") \

    .config("spark.executor.memory", "1g") \

    .config("spark.cores.max","2") \

    .config("spark.driver.extraClassPath", "/usr/share/java/mysql-connector-java.jar") \

    .config("spark.executor.extraClassPath", "/usr/share/java/mysql-connector-java.jar") \

    .config("spark.sql.warehouse.dir", "/Users/jlyang/Spark/spark-warehouse")\

    .getOrCreate()

#Get the Spark Context from Spark Session

SpContext = SpSession.sparkContext

demoDf = SpSession.read.format("jdbc").options(

    url="jdbc:mysql://localhost:3306/testpraba1",

    driver = "com.mysql.jdbc.Driver",

    dbtable = "users_userprofile",

    user="root",

    password="XXXXX").load()

demoDf.show()

ERROR MasterWebUI: Failed to bind MasterWebUI

Problem: 
While starting start-master.sh, getting ERROR MasterWebUI: Failed to bind MasterWebUI

Solution:
check SPARK_LOCAL_IP is set to correct IP Address
export SPARK_LOCAL_IP=192.168.254.122 

Error:
Spark Command: /opt/java/jdk1.8.0_201/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host 127.0.0.1 --port 7077 --webui-port 8080
========================================
19/03/12 14:03:50 ERROR MasterWebUI: Failed to bind MasterWebUI
java.net.BindException: Cannot assign requested address: Service 'MasterUI' failed after 16 retries (starting from 8080)! Consider explicitly setting the appropriate port for the service 'MasterUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.spark_project.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:351)
    at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:319)
    at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
    at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
    at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$newConnector$1(JettyUtils.scala:353)
    at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$httpConnect$1(JettyUtils.scala:380)
    at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:383)
    at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:383)
    at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2269)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
    at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2261)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:383)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:132)
    at org.apache.spark.deploy.master.Master.onStart(Master.scala:140)
    at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:122)
    at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
    at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
    at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:221)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

python reduce using lambda

#Using reduce get average MGP-CITY (miles per Gallon) from sample car data
from functools import reduce

car_data = ['MAKE,FUELTYPE,ASPIRE,4,BODY,DRIVE,CYLINDERS,HP,RPM,MPG-CITY,MPG-HWY,PRICE',\
 'mercedes-benz,gas,std,2,convertible,RWD,eight,155,4750,16,18,35056', \
 'jaguar,gas,std,4,sedan,RWD,six,176,4750,15,19,35550', \
 'jaguar,gas,std,2,sedan,RWD,twelve,262,5000,13,17,36000', \
 'bmw,gas,std,4,sedan,RWD,six,182,5400,15,20,36880', \
 'porsche,gas,std,2,convertible,RWD,six,207,5900,17,25,37028', \
 'mercedes-benz,gas,std,4,sedan,RWD,eight,184,4500,14,16,40960', \
 'bmw,gas,std,2,sedan,RWD,six,182,5400,16,22,41315', \
 'mercedes-benz,gas,std,2,hardtop,RWD,eight,184,4500,14,16,45400']

#Use a function to perform reduce
def getMPGCity( autoStr) :
    if isinstance(autoStr, int) :
        return autoStr
    attList=autoStr.split(",")
    if attList[9].isdigit() : #this will ignore header
        return int(attList[9])
    else:
        return 0

total_mgp = reduce(lambda x,y : getMPGCity(x) + getMPGCity(y), car_data)
print 'Total MPG: {}'.format(total_mgp)

data_length_without_header = len(car_data)-1.0  # account for header line
print 'Total lines without header: {}'.format(data_length_without_header)

print 'Average MPG: {} '.format(total_mgp/data_length_without_header)

Output:
Total MPG: 120
Total lines without header: 8.0
Average MPG: 15.0

python reduce

from functools import reduce

input_list = [10, 20, 30, 5, 7, 1, 34, 45, 566]

#Find smallest no
print reduce(lambda x,y: x if x < y else y, input_list)  #1

#Find largest no
print reduce(lambda x,y: x if x > y else y, input_list)  #566

python change conf variables

conf.py
x=10

a.py
import conf
print 'Inside A: %d ' % (conf.x)
conf.x=20  #Here overwriting conf.x
print 'Inside A after changing value to 20 : %d ' % (conf.x)

b.py
import a
import conf
print 'Inside B: %d' % (conf.x)

Output:
python a.py
Inside A: 10
Inside A after changing value to 20 : 20

python b.py
Inside A: 10


Inside A after changing value to 20 : 20
Inside B: 20
#Here you get 20 as output, since you imported a 


When you import using 2 ways:
1) from conf import x 
#this will create local variable scope

2) import conf
conf.x  #refers to original location


a.py
from conf import x   #This creates variable in local scope
print 'Inside A: %d ' % (x)
x=20
print 'Inside A after changing value to 20 : %d ' % (x)

Output: python a.py
Inside A: 10
Inside A after changing value to 20 : 20

python b.py
Inside A: 10
Inside A after changing value to 20 : 20
Inside B: 10

Conclusion:
Always safer to use 
from conf import x
instead of 
import x

python ConfigParser

#file.ini
#[HEADER]
#name = Matt

import ConfigParser
config = ConfigParser.ConfigParser()
config.read('file.ini')
name = config.get('HEADER', 'NAME') #Matt
print name
name = 'Will'
config.set('HEADER', 'NAME', name)
print config.get('HEADER', 'NAME') #Will

# Writing our configuration file to 'example.cfg'
with open('file.ini', 'w') as configfile:
    config.write(configfile)

python open very big file, python memory

Python open very big file: 

#Say you are looping through a big 2 TB file

logfile = open("huge_log_file.txt","r")
info_lines = [(line,len(line)) for line in logfile if line.startswith("INFO")]
#Here it will get a huge list - costs RAM, this list could contain 2 TB of content

logfile = open("huge_log_file.txt","r")
info_lines = ((line,len(line)) for line in logfile if line.startswith("INFO"))
#Here it will get generator object - memory efficient

Opening a file (read mode) does NOT implicitly read nor load its contents into memory.  
Even when you do so using Python's context management protocol (the with keyword).

e.g.,

with open('huge_log_file.txt', 'r') as f:
   for each_line in f:
     do_something_each_line(each_line)

Then your peak memory utilization shouldn't be much larger than the longest line of the file

If you really are reading the full content of the file into a data structure like list, then it's no wonder that your RAM usage peaks like that. 
It's not that python puts the full contents of the file in RAM, but that you do.
e.g.,

#Here it will load all into memory as you are storing/dumping into a list called info_lines
info_lines = [(line,len(line)) for line in logfile if line.startswith("INFO")] 


#Memory efficinet - since it return generator
info_lines = ((line,len(line)) for line in logfile if line.startswith("INFO"))



file = '/tmp/huge_log_file.txt'

#This is fine
with open(file, 'r') as fh:
   for each in fh:
       print each

#this is a blunder, this will crash, it loads every thing into memory
#with open(file, 'r') as fh:
#    lines = fh.readlines()
#    print len(lines)

What is the difference between __init__ and __call__?

# __init__ method is used when the class is called to initialize the instance
# while the __call__ method is called when the instance is called

class Foo:
    def __init__(self, a, b, c):
        print 'inside Foo __init__'
    def __call__(self, a, b, c):
        print 'inside Foo __call__'

class Bar:
    def __init__(self):
        print 'inside Bar __init__'
    def __call__(self, a, b, c):
        print 'inside Bar __call__'


f = Foo(1, 2, 3) # __init__
b = Bar()
b(1, 2, 3) # __call__

Output:

inside Foo __init__

inside Bar __init__

inside Bar __call__

__all__ in Python

__all__ in a module

e.g. module.py:

__all__ = ['foo', 'Bar']
means that when you import * from the module, only those names in the __all__ are imported:

from module import *   
# imports only foo and Bar (through wild card)

test1.py

__all__ = ['var1', 'func1']

var1 = 100

def func1():
    return 'func1'

def func2():
    return 'func2'


test2.py
from test import var1, func1, func2
print var1
print func1()
print func2()

Output:
100
func1
func2


test3.py
from test import *

print var1
print func1()
print func2() #NameError: name 'func2' is not defined

Output:
100
func1
NameError: name 'func2' is not defined

Python __name__, __main__

test1.py
if __name__ == '__main__':
   print 'inside test1 main...'
else:
   print 'test1 imported: ' + __name__


test2.py
import test1

if __name__ == '__main__':
   print 'inside test2 main...'
else:
   print 'test2 imported: ' + __name__

test3.py
import test2

if __name__ == '__main__':
   print 'inside test3 main...'
else:
   print 'test3 imported: ' + __name__

python test1.py

Output:

inside test1 main...

python test2.py

Output:

test1 imported: test1

inside test2 main...

python test3.py    ##(from test1 -> test2 -> test3)

Output:

test1 imported: test1

test2 imported: test2

inside test3 main...

Python multi threading Vs multi processing

Python multi threading Vs multi processing

Ref:
https://medium.com/@nbosco/multithreading-vs-multiprocessing-in-python-c7dc88b50b5b

#Threading
#The Python threading module uses threads instead of processes. Threads run in the same unique memory heap.
#Whereas Processes run in separate memory heaps. This, makes sharing information harder with processes and object instances.
#One problem arises because threads use the same memory heap, multiple threads can write to the same location in the memory heap which is why the global interpreter lock(GIL) in CPython was created as a mutex to prevent it from happening.

#Multiprocessing

#The multiprocessing library uses separate memory space, multiple CPU cores, bypasses GIL limitations in CPython, child processes are killable(ex. function calls in program) and is much easier to use.

#Some caveats of the module are a larger memory footprint and IPC’s a little more complicated with more overhead.

import threading

#Threading
#The Python threading module uses threads instead of processes. Threads run in the same unique memory heap.
#Whereas Processes run in separate memory heaps. This, makes sharing information harder with processes and object instances.
#One problem arises because threads use the same memory heap, multiple threads can write to the same location in the memory heap which is why the global interpreter lock(GIL) in CPython was created as a mutex to prevent it from happening.

def calc_square(number):
    print('Square:' , number * number)

def calc_quad(number):
    print('Quad:' , number * number * number * number)

if __name__ == "__main__":
    number = 7
    thread1 = threading.Thread(target=calc_square, args=(number,))
    thread2 = threading.Thread(target=calc_quad, args=(number,))
    # Will execute both in parallel
    thread1.start()
    thread2.start()
    # Joins threads back to the parent process, which is this
    # program
    thread1.join()
    thread2.join()
    # This program reduces the time of execution by running tasks in parallel

import multiprocessing

#The multiprocessing library uses separate memory space, multiple CPU cores, bypasses GIL limitations in CPython, child processes are killable(ex. function calls in program) and is much easier to use.

#Some caveats of the module are a larger memory footprint and IPC’s a little more complicated with more overhead.

def calc_square(number):

    print('Square:' , number * number)

    result = number * number

    print(result)

def calc_quad(number):

    print('Quad:' , number * number * number * number)

if __name__ == "__main__":

    number = 7

    result = None

    p1 = multiprocessing.Process(target=calc_square, args=(number,))

    p2 = multiprocessing.Process(target=calc_quad, args=(number,))

    p1.start()

    p2.start()

    p1.join()

    p2.join()

    # Wont print because processes run using their own memory location

    print(result)

Python read from XLS/XLSX

import xlrd

loc = ("/tmp/Students.xlsx")

wb = xlrd.open_workbook(loc)

sheet1 = wb.sheet_by_index(0)
sheet2 = wb.sheet_by_index(1)

#Sheet1
for i in range(1, sheet1.nrows):
each_list = sheet1.row_values(i)
print each_list

#Sheet2
for i in range(1, sheet1.nrows):
each_list = sheet2.row_values(i)
print each_list

Python Find second smallest number

Python Find second smallest number

def get_second_smallest(numbers):
    num1, num2 = float('inf'), float('inf')
    print num1, num2
    print '#####'
    for x in numbers:
        if x <= num1:
            num1, num2 = x, num1
            print num1, '-------', num2
        elif x < num2:
            num2 = x
            print num1, '-------', num2
        else:
            print 'pass... ' + str(x)
    return num2

out = get_second_smallest([2, 3, 4, 5, 1, 1.3, 1.5])
print '\nOutput: ' + str(out)

Output:
2 ------- inf
2 ------- 3
pass... 4
pass... 5
1 ------- 2
1 ------- 1.3
pass... 1.5

Output: 1.3

AWS Create/change key pair for EC2 instance

AWS Create/change key pair for EC2 instance

Under EC2 -> Network & Security -> Key Pairs -> Create with name: Mirror
It will give you Mirror.pem

PuyttyGen:
PuttyGen -> Load this PEM -> Generate
Download public key as Mirror.pub & private key as Mirror.ppk

Login in instance:
vim ~/.ssh/authorized_keys
Open Mirror.pub, copy whole key as single line, without new line  characters (after Comment: and before END SSH2 PUBLIC KEY)

ssh-rsa <copied text from above> <Mirror(without pem extension)>

e.g.,
ssh-rsa  AAAAB3NzaC1yc2EAAA...urR2A5IUkqscHRU1Nc7TFz363UFJW6XMYae1116PO4  Mirror

Setup Mirror under Putty like below:
Host Name -> EC2 Host Name
Connection -> SSH -> Auth -> Add Mirror.pek path & save

Now login into EC2 as root

(it should not ask any password)

How to install pyspark in centos

Install spark ref:
http://devopspy.com/python/apache-spark-pyspark-centos-rhel/

cd /opt
wget http://www-eu.apache.org/dist/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz
ln -s spark-2.4.0-bin-hadoop2.7 spark
check /etc/hosts

How to set path?
export SPARK_HOME = /opt/spark
export PATH = $PATH:/opt/spark
export export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$SPARK_HOME/python/lib/pyspark.zip:$PYTHONPATH
export PATH = $SPARK_HOME/python:$PATH

How to start master?
./sbin/start-master.sh

    1) If you get error like blow:
"hostname: Unknown host" start-master.sh
set the hostname properly
hostname test.com
hostname -f #should give you some output

    2) If you get error like below:
Getting "Unsupported major.minor version 52.0" exception while using  
        Spark Web Application framework
check Java version of jar files (/opt/spark/jars) and your installed java

How to start spark master?
cd /opt/spark
./sbin/start-master.sh
This internally runs command like below:
Spark Command: /opt/java/jdk1.8.0_201/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host test.com --port 7077 --webui-port 8080

How to access from web?
test.com:8080 (port: 8080)

How to start spark shell?
cd /opt/spark
./bin/pyspark
FYI, This internally runs command like below:
/opt/java/jdk1.8.0_201/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.SparkSubmit --name PySparkShell pyspark-shell

How to access the spark process in ps commands?
ps -ef | grep spark
      e.g.,root     13770     1  0 14:18 pts/0    00:00:10 /opt/java/jdk1.8.0_201/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host test.com --port 7077 --webui-port 8080

PIP modules to install
pip install py4j

How to access in web?

http://localhost:8080