python how to overcome GIL

Ref: https://realpython.com/python-gil/

# single_threaded.py
import time
from threading import Thread

COUNT = 50000000

def countdown(n):
    while n>0:
        n -= 1

start = time.time()
countdown(COUNT)
end = time.time()

print('Time taken in seconds -', end - start)
#Time taken in seconds - 6.20024037361145


# multi_threaded.py
import time
from threading import Thread

COUNT = 50000000

def countdown(n):
    while n>0:
        n -= 1

t1 = Thread(target=countdown, args=(COUNT//2,))
t2 = Thread(target=countdown, args=(COUNT//2,))

start = time.time()
t1.start()
t2.start()
t1.join()
t2.join()
end = time.time()

print('Time taken in seconds -', end - start)
# Time taken in seconds - 6.924342632293701
# No improvement here due to GIL & due to sharing of lock & releases


# multi_processing.py
# Uses cores instead of threads
from multiprocessing import Pool
import time

COUNT = 50000000
def countdown(n):
    while n>0:
        n -= 1

if __name__ == '__main__':
    pool = Pool(processes=2)
    start = time.time()
    r1 = pool.apply_async(countdown, [COUNT//2])
    r2 = pool.apply_async(countdown, [COUNT//2])
    pool.close()
    pool.join()
    end = time.time()
    print('Time taken in seconds -', end - start)
    #Time taken in seconds - 4.060242414474487

Python GIL, lock, mutex, semaphore

Lock
#####
A lock allows only one thread to enter the part that's locked and the lock is not shared with any other processes.

Mutex
#######
Mutex as the name means mutual exclusion.
A mutex is used to guard shared data such as a list, dict.
A mutex allows only a single thread to access a resource or critical section.
A mutex is the same as a lock but it can be system wide (shared by multiple processes).
Same thread can acquire and release lock
Owned by thread

Semaphore
##########
Semaphore, is used for limiting access to a collection of resources.
A semaphore does the same as a mutex but allows x number of threads to enter, this can be used
E.g., to limit the number of cpu, io or ram intensive tasks running at the same time.
      Pool of DB connections to be handed out to requesting threads
Different threads can call acquire and release on the semaphore
No ownership

GIL
####
Ref: https://realpython.com/python-gil/

The Python interpreter can only execute a single thread at a time
If your machine has one or hundred processors, the Python interpreter is only able to run a single thread at a time using a single processor.
Two threads on a machine with two available processors can't be executed in parallel each running on a single CPU.
This lock is known as the Global Interpreter Lock using CPython (to avoid deadlocks)
Python implementations which overcome the GIL altogether. E.g., Jython, IronPython and pypy-stm.

Python uses reference counting for memory management. (unlike garbage collection in other languages)
It means that objects created in Python have a reference count variable that keeps track of the number of references that point to the object. When this count reaches zero, the memory occupied by the object is released.

This reference count variable can be kept safe by adding locks to all data structures that are shared across threads so that they are not modified inconsistently.

But adding a lock to each object or groups of objects means multiple locks will exist which can cause another problem—Deadlocks (deadlocks can only happen if there is more than one lock). Another side effect would be decreased performance caused by the repeated acquisition and release of locks.

The GIL is a single lock on the interpreter itself which adds a rule that execution of any Python bytecode requires acquiring the interpreter lock. This prevents deadlocks (as there is only one lock) and doesn’t introduce much performance overhead. But it effectively makes any CPU-bound Python program single-threaded.

Python Disadvantages

Python Disadvantages

• Its an interpreted language, not as fast as a compiled language
• Slower than C & C++. Since Python is a high level language unlike C, C++, its not close to hardware 
• Not good for Gaming, Mobile dev, Desktop UI applications
• Not good for memory intensive, due to flexibility of data types, python memory consumption is high
• Python's database access layer is found to be bit underdeveloped and primitive (JDBC/ODBC)
• GIL
• Global interpreter lock: can’t run more than one thread using in one interpreter 
• Creating, managing and tearing down processes (multi-processing, processes are heavier than threads) is more expensive than doing the same for threads. Furthermore, inter-process communication is relatively slower than inter-thread communication. 
• Both these drawbacks may not make Python a practical technology choice for super-critical or time-sensitive use-cases.
• Python community are working to remove the GIL from CPython. One such attempt is known as the Gilectomy.
• GIL exists only in the original Python implementation that is CPython.
• Python has multiple interpreter implementations. CPython, Jython, IronPython and PyPy, written in C, Java, C# and Python respectively, are the most popular ones.