python open very big file, python memory

Python open very big file: 

#Say you are looping through a big 2 TB file

logfile = open("huge_log_file.txt","r")
info_lines = [(line,len(line)) for line in logfile if line.startswith("INFO")]
#Here it will get a huge list - costs RAM, this list could contain 2 TB of content

logfile = open("huge_log_file.txt","r")
info_lines = ((line,len(line)) for line in logfile if line.startswith("INFO"))
#Here it will get generator object - memory efficient

Opening a file (read mode) does NOT implicitly read nor load its contents into memory.  
Even when you do so using Python's context management protocol (the with keyword).

e.g.,

with open('huge_log_file.txt', 'r') as f:
   for each_line in f:
     do_something_each_line(each_line)

Then your peak memory utilization shouldn't be much larger than the longest line of the file

If you really are reading the full content of the file into a data structure like list, then it's no wonder that your RAM usage peaks like that. 
It's not that python puts the full contents of the file in RAM, but that you do.
e.g.,

#Here it will load all into memory as you are storing/dumping into a list called info_lines
info_lines = [(line,len(line)) for line in logfile if line.startswith("INFO")] 


#Memory efficinet - since it return generator
info_lines = ((line,len(line)) for line in logfile if line.startswith("INFO"))



file = '/tmp/huge_log_file.txt'

#This is fine
with open(file, 'r') as fh:
   for each in fh:
       print each

#this is a blunder, this will crash, it loads every thing into memory
#with open(file, 'r') as fh:
#    lines = fh.readlines()
#    print len(lines)

Compare List Vs List Comprehension, timeit

import timeit

# timeit.timeit(stmt, setup, timer, number)
# stmt   - which is the statement you want to measure it defaults to 'pass'.
# setup  - which is the code that you run before running the stmt; it defaults to 'pass'. We generally use this to import the required modules for our code.
# number - number of executions you like to run the stmt.

# List Comprehension

lst = [3, 2, 41, 3, 34, 99]

print(lst)

# If Condition alone - keep after the For loop

print([number for number in lst if number % 2 != 0])

# If-Else Condition - keep before For loop

print([number if number % 2 != 0 else -1 for number in lst])

#List Vs List Comprehension
#Ref: https://stackoverflow.com/questions/16341775/what-is-the-advantage-of-a-list-comprehension-over-a-for-loop
#List comprehensions are more compact and faster than an explicit for loop building a list:
#This is because calling .append() on a list causes the list object to grow (in chunks) to make space for new elements individually, while the list comprehension gathers all elements first before creating the list to fit the elements in one go:

def for_loop_test():
    result = []
    for elem in iter_var:
        result.append(elem)
    return result

def list_comprehension_test():
    return [elem for elem in iter_var]

if __name__ == '__main__':
iter_var = range(1000)
print timeit.timeit('f()', 'from __main__ import for_loop_test as f', number=10000)
print timeit.timeit('f()', 'from __main__ import list_comprehension_test as f', number=10000)


"""
Output:
1.41242463295
0.474811508482
"""'

Python compare lists

#Comapre lists

student1_scores = [61, 73, 84, 90, 85, 45]
student2_scores = [40, 37, 45, 87, 99, 54]

#set - will change the order
print list(set(student1_scores)) #[73, 45, 84, 85, 90, 61]
print list(set(student2_scores)) #[99, 37, 40, 45, 54, 87]

#Commom elements in two lists
print list(set(student1_scores) & set(student2_scores)) #[45]

#Commom elements in two lists
print list(set(student1_scores).intersection(set(student2_scores))) #[45]

#Union of two lists
print list(set(student1_scores).union(set(student2_scores))) 
#[99, 37, 40, 73, 45, 84, 85, 54, 87, 90, 61]

#Commom elements in two lists - using list comprehension
print [i for i in student1_scores if i in student2_scores] #[45]

#compare common elements with for the same subject (same index)
student1_scores = [61, 73, 84, 90, 85, 45]
student2_scores = [40, 37, 45, 90, 99, 54]

print [i for i, j in zip(student1_scores, student2_scores) if i == j] #[90]

Python - Filter Vs Map Vs Reduce Vs List Comprehension

Today we discuss about the following topics :
1) Filter
2) Map
3) List Comprehension

Lets discuss with few examples:

Filter :
filter(function, sequence) returns a sequence consisting of those items from the sequence for which function(item) is true

Map :
map(function, sequence) calls function(item) for each of the sequence's items and returns a list of the return values

List Comprehension :
List comprehension in Python provides a clear and concise syntax for creating lists from other lists


filter_test.py

#filter(function, sequence) returns a sequence consisting of those items from the sequence for which function(item) is true
def isPythonFile(list_1):
    if list_1.find(".py") == -1:
        return False
    else:
        return True
 
list_1 = ["1.py","2.pl", "3.zip", "4.py","5.php" ]
py_files = filter(isPythonFile, list_1)   #Function is called for each element of list
 
for item in py_files:
    print("Each item in Filetr :" , item)  

Output:

Each item in Filetr : 1.py
Each item in Filetr : 4.py

filter_lambda_test.py

foo = [2, 18, 9, 22, 17, 24, 8, 12, 27]
print("Before Lambda :", list(foo))
print("After  Lambda :", list(filter(lambda x: x % 3 == 0, foo)))

Output:

Before Lambda : [2, 18, 9, 22, 17, 24, 8, 12, 27]
After  Lambda : [18, 9, 24, 12, 27]  

map_test.py

print("Before Map :", list(foo))
print("After  Map :", list(map(lambda x: x * 2 + 10, foo)))  

Output:

Before Map : [2, 18, 9, 22, 17, 24, 8, 12, 27]
After  Map : [14, 46, 28, 54, 44, 58, 26, 34, 64]  

smallest_no_using_reduce_test.py
#how to use if else in lambda

from functools import reduce

ll = [10, 12, 45, 2, 100]
out = reduce(lambda x, y: x  if x < y  else y,  ll)
print(out)

Output:
2

largest_no_using_reduce_test.py
#how to use if else in lambda

from functools import reduce

ll = [10, 12, 45, 2, 100]
out = reduce(lambda x, y:  x  if x > y  else y,  ll)
print(out)

Output:
100

list_comprehension_test1.py

input_arr = [2, 3, 4]
output_arr = [2*i for i in input_arr if i > 2]
print("List Comprehension Test 1 :", output_arr)  

Output:

List Comprehension Test 1 : [6, 8]

list_comprehension_test2.py

input_arr = ['Mother Teresa', 'Abraham Lincoln', 'Nelson Mandela']
output_arr = ['Dear...' + i  for i in input_arr if len(i) > 5]
print("List Comprehension Test 2 :",output_arr)  

Output:

List Comprehension Test 2 : ['Dear...Mother Teresa', 'Dear...Abraham Lincoln', 'Dear...Nelson Mandela']  

list_comprehension_test3.py  (if-else inside list comprehension)

input_arr = [20, 30, 40, 33, 55]
output_arr = [2*i if i%2 == 0 else i for i in input_arr]
print("List Comprehension Test 3 :", output_arr)  

Output:

List Comprehension Test 3 :  [40, 60, 80, 33, 55]

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton