Python Delete old files

Many a times, we come across deleting old files like logs, deprecated files from file system

We come across scenarios like 

To delete logs older than x days

To delete txt files older than x days

Let us discuss how we can achieve deleting old files using python script

What does the script do?

It will take the path from where you need to delete the txt files

The below program checks the time stamp of the files and deletes the txt files older than 7 days.

You can change the file extension, folder path and use it.

folder_path = "C:\Files_To_Read"

file_ends_with = ".txt"

how_many_days_old_logs_to_remove = 7

How to call the script?

python delete_old_logs.py

delete_old_logs.py

import os, time, sys

folder_path = "C:\Files_To_Read"
file_ends_with = ".txt"
how_many_days_old_logs_to_remove = 7

now = time.time()
only_files = []

for file in os.listdir(folder_path):
    file_full_path = os.path.join(folder_path,file)
    if os.path.isfile(file_full_path) and file.endswith(file_ends_with):
        #Delete files older than x days
        if os.stat(file_full_path).st_mtime < now - how_many_days_old_logs_to_remove * 86400: 
             os.remove(file_full_path)
             print "\n File Removed : " , file_full_path
 

Python - Filter Vs Map Vs Reduce Vs List Comprehension

Today we discuss about the following topics :
1) Filter
2) Map
3) List Comprehension

Lets discuss with few examples:

Filter :
filter(function, sequence) returns a sequence consisting of those items from the sequence for which function(item) is true

Map :
map(function, sequence) calls function(item) for each of the sequence's items and returns a list of the return values

List Comprehension :
List comprehension in Python provides a clear and concise syntax for creating lists from other lists


filter_test.py

#filter(function, sequence) returns a sequence consisting of those items from the sequence for which function(item) is true
def isPythonFile(list_1):
    if list_1.find(".py") == -1:
        return False
    else:
        return True
 
list_1 = ["1.py","2.pl", "3.zip", "4.py","5.php" ]
py_files = filter(isPythonFile, list_1)   #Function is called for each element of list
 
for item in py_files:
    print("Each item in Filetr :" , item)  

Output:

Each item in Filetr : 1.py
Each item in Filetr : 4.py

filter_lambda_test.py

foo = [2, 18, 9, 22, 17, 24, 8, 12, 27]
print("Before Lambda :", list(foo))
print("After  Lambda :", list(filter(lambda x: x % 3 == 0, foo)))

Output:

Before Lambda : [2, 18, 9, 22, 17, 24, 8, 12, 27]
After  Lambda : [18, 9, 24, 12, 27]  

map_test.py

print("Before Map :", list(foo))
print("After  Map :", list(map(lambda x: x * 2 + 10, foo)))  

Output:

Before Map : [2, 18, 9, 22, 17, 24, 8, 12, 27]
After  Map : [14, 46, 28, 54, 44, 58, 26, 34, 64]  

smallest_no_using_reduce_test.py
#how to use if else in lambda

from functools import reduce

ll = [10, 12, 45, 2, 100]
out = reduce(lambda x, y: x  if x < y  else y,  ll)
print(out)

Output:
2

largest_no_using_reduce_test.py
#how to use if else in lambda

from functools import reduce

ll = [10, 12, 45, 2, 100]
out = reduce(lambda x, y:  x  if x > y  else y,  ll)
print(out)

Output:
100

list_comprehension_test1.py

input_arr = [2, 3, 4]
output_arr = [2*i for i in input_arr if i > 2]
print("List Comprehension Test 1 :", output_arr)  

Output:

List Comprehension Test 1 : [6, 8]

list_comprehension_test2.py

input_arr = ['Mother Teresa', 'Abraham Lincoln', 'Nelson Mandela']
output_arr = ['Dear...' + i  for i in input_arr if len(i) > 5]
print("List Comprehension Test 2 :",output_arr)  

Output:

List Comprehension Test 2 : ['Dear...Mother Teresa', 'Dear...Abraham Lincoln', 'Dear...Nelson Mandela']  

list_comprehension_test3.py  (if-else inside list comprehension)

input_arr = [20, 30, 40, 33, 55]
output_arr = [2*i if i%2 == 0 else i for i in input_arr]
print("List Comprehension Test 3 :", output_arr)  

Output:

List Comprehension Test 3 :  [40, 60, 80, 33, 55]

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Python User Defined Exceptions

Today we discuss about raising exceptions manually (Python User Defined Exceptions).

1) We already know we have built in exceptions in Python like ValueError, IOError, FileNotFoundError, ZeroDivisionError

2) But sometimes we might have to write custom exceptions on our own to catch few scenarios.

Lets discuss how to implement user defined exception with an example.

In the below example :

nonEmptyValException                                  #User Defined Exception

ValueError, ZeroDivisionError, EOFError   #Built-in Exceptions


exceptions_raise_manual.py

import sys

class nonEmptyValException(Exception):
    '''A user-defined exception class.'''
    def __init__(self, num):
        Exception.__init__(self)
        self.num = num
      
try:
    a = int(input('Enter some integer for Dividend --> '))
    b = int(input('Enter some integer for Divisor --> '))
    
    if not a:
       raise nonEmptyValException(a)
       
    if not b:
       raise nonEmptyValException(b)
    
    c = a/b;    

    print ("Final Value is : ", c) 
    
except  ValueError:
    print('ValueError: Please enter only Integer Value')   
except ZeroDivisionError:
    print('ZeroDivisionError: The input divisor is %d, was expecting a Non-Zero number' % (b))   
except EOFError:
    print('\nWhy did you do an EOF on me?')
except nonEmptyValException as error:
    print('nonEmptyValException: The input is %d, was expecting a integer number' % (error.num))  

Please refer to other Python Exceptions Concepts :
All Python Exceptions Related Topics
Exceptions in Python

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Python Regular Expressions (re.compile Vs re.match)

Today we discuss about the following topics :
1) Pre-compiled Regular Expressions - Uses & Advantages (General Concept)
2) Explain the difference between Python re.compile Vs re.match with an example.

Let's dive into the topic & continue the fun :

1) Pre-compiled Regular Expressions - Uses & Advantages (General Concept)
This is a general topic irrespective of any language (Perl or Python or any)
The extra mile of using a pre-compiled regular expression is :

     a) When you have to execute the same regular expression pattern over millions of lines (say reading lines a file), then pre-compiled regular expressions are very handy by drastically reducing the time of execution.

     b) Since the regex pattern is pre-compiled in advance, no need to process the regex pattern each and every time while reading the lines from the file (suppose millions of lines in the file)

    c) Here, we assume, the regex pattern which you have to use, should be constant, if has to vary every time then pre-compiled regex will not be that useful.

2) Explain the difference between Python re.compile Vs re.match with an example.
Let us explain the explain with a lucid example.
Here we are trying to run a regex pattern in loop of 100000 times with both Python re.compile(pre-complied) & re.match
Please check the time difference between in the output file.

E.g., regex_compile_vs_match.py

import re
import time

input_str       = "Mother Teresa"
count_compile   = 0
count_match     = 0

time_val_1 = time.time()

compiled_regex = re.compile('(\w+)\s+(\w+)')

for i in range(1000000):
    if compiled_regex.match(input_str):
        count_compile += 1

print ("Count Compile Value : ", count_compile)        
print("Time taken by Using Compile : ",  time.time() - time_val_1)

time_val_2 = time.time()

for i in range(1000000):
    if re.match('(\w+)\s+(\w+)', input_str):
        count_match += 1

print ("Count Match Value : ", count_compile)          
print("Time taken by Using Match   : ",  time.time() - time_val_2)  

Output:

Count Compile Value : 1000000
Time taken by Using Compile :  1.340999994277954
Count Math Value : 1000000
Time taken by Using Match   :  2.994999885559082  

You might wish to read Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

How to remove duplicate lines from a file in Python

Today, we will discuss removing duplicate lines from a file in Python

Lets discuss in two ways as mentioned below :

1) Removing duplicate lines and print the lines in order (When Order is important)

        - Using normal way

2) Removing duplicate lines and print the lines in any order (When Order is NOT important) 

        - Using SET concept in Python. SET concept in python does not consider Order.

Note:
Both the scripts read duplicated content from file_with_duplicates.txt, 
Read the above file and remove duplicate lines and finally
Print to file_without_duplicates.txt 

Input: file_with_duplicates.txt

Mother Teresa
Winston Churchill
Abraham Lincoln
Mahatma Gandhi
Winston Churchill
Mother Teresa
Abraham Lincoln  

1)

infile = open('file_with_duplicates.txt', 'r')

outfile = open('file_without_duplicates.txt', 'w')



lines_seen = set()

for line in infile:

    if line not in lines_seen:

        outfile.write(line)

        lines_seen.add(line)

outfile.close()

1) remove_duplicate_lines_from_file_with_order.py
- Using Normal Way - it take cares of order of the lines

#!/usr/bin/python

try:
    input_file = open("file_with_duplicates.txt", "r")
    output_file = open("file_without_duplicates.txt", "w")

    unique = []
    for line in input_file:
        line = line.strip()
        if line not in unique:
            unique.append(line)
    input_file.close()
    
    for i in range(0, len(unique)-1):
        unique[i] += "\n"

    output_file.writelines(unique)
    output_file.close
    
except FileNotFoundError:
    print('\n File NOT Found Error')
    sys.exit
except IOError:
    print('\n IO Error')
    sys.exit  

Output: file_without_duplicates.txt

Mother Teresa
Winston Churchill
Abraham Lincoln
Mahatma Gandhi  

2) remove_duplicate_lines_from_file_without_order.py
- Using SET concept - it does not consider the order of the lines

#!/usr/bin/python

try:
    input_file = open("file_with_duplicates.txt", "r")
    output_file = open("file_without_duplicates.txt","w")

    #The main drawback of using sets is, the order of the lines may not be same as in input file
    uniquelines = set(input_file.read().split("\n"))
    output_file.write("".join([line + "\n" for line in uniquelines]))
    
    input_file.close()
    output_file.close()
    
except FileNotFoundError:
    print('\n File NOT Found Error')
    sys.exit
except IOError:
    print('\n IO Error')
    sys.exit      

Output: file_without_duplicates.txt

Abraham Lincoln
Winston Churchill
Mother Teresa
Mahatma Gandhi  

You may also wish to read Object Oriented Concepts in Python as mentioned :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in perl

Check an element exists in an array/list in Python

inspiring_ppl = ["Mahatma Gandhi", "Mother Teresa", "Paul Coelho", "Bill Gates", "Steve Jobs"]


#raw_input(), print for Python 2.X Version 
#input(), print() for Python 3.X Version 

another_person = input("Enter person Name : ")

if another_person in inspiring_ppl:
    inspiring_ppl.remove(another_person)
else:
    inspiring_ppl.append(another_person)

print(inspiring_ppl)

Print File Content in Python

file = open("read.txt","r")
text = file.readlines()   #Reads the file content
file.close()

for line in text:
    print line        #Python 2.X Versions
    #print(line)      #Python 3.X Versions

Print File in Reverse Order in Python

file = open("read.txt","r")
text = file.readlines()   #Reads the file content
file.close()

text.reverse()    #It reverses the content

for line in text:
    print line        #Python 2.X Versions
    #print(line)    #Python 3.X Versions