Unix Sort, Uniq, Combine Files

We will discuss about 
1) Combining contents of two files in Unix 
2) Sort & Get the Unique Lines (delete duplicate lines)
3) How to get only the duplicate lines
4) How to get only the unique lines

Suppose we have two files a.txt and b.txt as mentioned below :

a.txt

10
20
30
40
50

b.txt

10
20
30
40
50
60
70
80

1) Combining contents of two files in Unix 
Combining contents of both a.txt & b.txt into c.txt
]# cat a.txt b.txt > c.txt 

10
20
30
40
50
10
20
30
40
50
60
70
80

2) Sort and Get the Unique Lines (delete duplicate lines)
]# cat c.txt | sort | uniq

10
20
30
40
50
60
70
80

3) How to get only the duplicate lines
]# cat c.txt | sort | uniq -u

60
70
80

4) How to get only the unique lines

10
20
30
40
50

5) How to get sort by ignore case
]# cat requirements.txt | sort --ignore-case > requirements_bak.txt 

Please refer to other topics on Unix like :
Unix Grep Examples
Search a Directory in Unix
Unix For Loop
pushd & popd in Unix
Find Size of Directory
Word Count

Please refer to other topics on AWK like :
Awk Examples
Print First Two Columns of File
Print Last Two Columns of File

Please refer to other topics on Dict like :
Dict in Python
Dict keys and values in Python

Please refer to other topics on List like :
List in Python
Append to list in Python
Delete the last name from the list in Python
Remove an element from List in Python
Check an element exists in an list in Python
Python Filter Vs Map Vs List Comprehension

Please refer to other topics on File Concepts like :
Print File Content in Python
Print File in Reverse Order in Python

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Python List

List : 
A Python list contains an ordered collection of objects

E.g.,

abc = [1, 2, 3, 4, 5]

Python index of list starts from 0 (left most)
abc[0], abc[1], abc[2], abc[3], abc[4]

Python also supports negative indices from the right most, starts with -1
abc[-5], abc[-4], abc[-3], abc[-2], abc[-1]

abc[-1] is 5
abc[0] is 1



Please refer to other topics on List like :
Append to list in Python
Delete the last name from the list in Python
Remove an element from List in Python
Check an element exists in an list in Python
Python Filter Vs Map Vs List Comprehension

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Replace string in all files in a directory in perl

We will discuss briefly about the script (To replace string in all files in a directory in perl)

Please define your directory path in the following :
my $input_dir  = "/usr/src/perl_test/input/";
my $output_dir = "/usr/src/perl_test/output/";
my $log_dir    = "/usr/src/perl_test/log/";

Define the string to replace and string to replace with as mentioned below
my $str_to_replace = "Mother Teresa":
my $str_replace_with = "MOTHER TERESA";

If you want to replace a particular string in all the files in the same directory path, then give the same path for both $input_dir & $output_dir.
E.g.,
my $input_dir  = "/usr/src/perl_test/same_path/";
my $output_dir = "/usr/src/perl_test/same_path/";

Log Directory :
It creates the log file(log_<timestamp>.log) in the path mentioned in $log_dir

Important Note : 
Please take a backup before running the script for safer side in case if you are replacing the strings in the same folder


conversion_script.pl

#!/usr/intel/bin/perl

use strict;
use warnings;
use Data::Dumper;

my $input_dir  = "/usr/src/perl_test/input/";
my $output_dir = "/usr/src/perl_test/output/";
my $log_dir    = "/usr/src/perl_test/log/";

my $str_to_replace = "Mother Teresa":
my $str_replace_with = "MOTHER TERESA";

my ($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst);

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
 $sec  = sprintf("%02d",$sec);
 $min  = sprintf("%02d",$min);
 $hour = sprintf("%02d",$hour);
 $mday = sprintf("%02d", $mday);
 $mon  = sprintf("%02d", $mon+1);
 $year = sprintf("%04d", $year+1900);

my $output_tag = $mday .'_'. $mon .'_'. $year .'_'. $hour .'_'. $min;

opendir(IN_DIR, $input_dir)   or die $!;
opendir(OUT_DIR, $output_dir) or die $!;
opendir(LOG_DIR, $log_dir)    or die $!;

open (LOG_FILE, "> $log_dirlog_$output_tag.log");

print LOG_FILE "------------- START ------------ \n";

#print LOG_FILE "Started Copying Files from Source: $srcdir   to Destination: $dest \n";
#my $cmd = "cp -R $srcdir/* $dest/";
#`$cmd`; #or die "Chk the command : " . $cmd;
#print LOG_FILE "Finished Copying Files from Source: $srcdir   to Destination: $dest \n";

my (%files_changed, %files_not_changed, @invalid_list_files);

while (my $file = readdir(IN_DIR)) {

    # Ignore if it is not file 
    unless (-f "$input_dir$file") {
       print LOG_FILE "-W- Not a File, Ignoring : " . $file . "\n";
       next;
    }

    print LOG_FILE "\n\nFile Name : " . $file . "\n";

    open(IN_FILE, "<$input_dir$file") || warn "Cant open file for reading: " . "$input_dir$file";
    my @lines = ;
    close(IN_FILE);

    my @newlines;
    foreach my $each_line (@lines) {
        #chomp $each_line;
        if($each_line =~ /$str_to_replace/) {
            print LOG_FILE "Converting String From : " . $each_line;
            $each_line =~ s/$str_to_replace/$str_replace_with/ig;
            print LOG_FILE "Converting String To   : " . $each_line;
            push(@newlines, $each_line);
            $files_changed{$file} = 1;
        } else {
            push(@newlines, $each_line);
            $files_not_changed{$file} = 1;
        }   
    }

    open(OUT_FILE, ">$output_dir$file") || warn "-W- Cant open file for writing: " . "$output_dir$file";
    print OUT_FILE @newlines;
    close(OUT_FILE);
}

my @f_changed     = keys %files_changed;
my @f_not_changed = keys %files_not_changed;

print LOG_FILE "\n\nOutput Summary as below ------------------------- : ";

print LOG_FILE "\n List of all the Files changed     : " . Dumper(\@f_changed);
print LOG_FILE "\n List of all the Files NOT changed : " . Dumper(\@f_not_changed);

print LOG_FILE "\n\n Total No.of files changed   : " . scalar(@f_changed);
print LOG_FILE "\n Total No.of files NOT changed : " . scalar(@f_not_changed);

close (LOG_FILE);
closedir(IN_DIR);
closedir(OUT_DIR);
closedir(LOG_DIR);

1;  

Input Directory: /usr/src/perl_test/input/

1) mother_teresa_intro.txt
Mother Teresa born on August 26, 1910
Full name of Mother Teresa is "Agnes Gonxha Bojaxhiu"
Mother Teresa founded the Missionaries of Charity, which in 2012 consisted of over 4,500 sisters and is active in 133 countries.

2) mother_teresa_awards.txt
In 1962, Mother Teresa was awarded the Ramon Magsaysay Award
In 1979, Mother Teresa was awarded the Nobel Peace Prize, for work undertaken in the struggle to overcome poverty and distress
In 1980, Mother Teresa was awarded the Bharat Ratna Prize  

Output Directory: /usr/src/perl_test/output/

1) mother_teresa_intro.txt
MOTHER TERESA born on August 26, 1910
Full name of MOTHER TERESA is "Agnes Gonxha Bojaxhiu"
MOTHER TERESA founded the Missionaries of Charity, which in 2012 consisted of over 4,500 sisters and is active in 133 countries.


2) mother_teresa_awards.txt
In 1962, MOTHER TERESA was awarded the Ramon Magsaysay Award
In 1979, MOTHER TERESA was awarded the Nobel Peace Prize, for work undertaken in the struggle to overcome poverty and distress
In 1980, MOTHER TERESA was awarded the Bharat Ratna Prize  

Log Directory: /usr/src/perl_test/log/
log_<time_stamp>.log

Compare Numbers/Strings in Perl

Today we discuss about comparing numbers/strings in Perl.


Compare Numbers:

Syntax : 
$a == $b (Equal check for numbers)
$a != $b (Not Equal check for numbers)

compare_numbers.pl
#!/usr/bin/perl
use strict;
use warnings;

my $num1 = 100;
my $num2 = 200;

if ($num1 == $num2) {
    print "Equal\n";
} else {
    print "Not Equal\n";
}

O/P:
Not Equal

Compare Strings:

Syntax : 
$a eq $b (Equal check for strings )
$a ne $b (Not Equal check for strings)

compare_strings.pl
#!/usr/bin/perl
use strict;
use warnings;

my $string1 = 'one';
my $string2 = 'two';

if ($string1 eq $string2) {
    print "Equal\n";
} else {
    print "Not Equal\n";
}

O/P:
Not Equal

Create Log File in Perl

Today we discuss about how to create a log file in Perl, Logging is always part and parcel in developing code.

Here in the below mentioned example : 
1) We are trying to read the lines from the file in the path 'C:\Perl\data.txt'
2) While reading the lines, in case of any errors, we are capturing (means we are writing to log file) to log file 'C:\Perl\log.txt'
3) You can either use write/append mode to log file based on your requirement.
4) Write (">") mode : You write the content to a new log file
5) Append (">>") mode : if Log File already exists, you append the content to the existing file

log_test.pl

#!/usr/bin/perl

use strict;
use warnings;

my $log_file_path = 'C:\Perl\log.txt';
my $read_file_path = 'C:\Perl\data.txt';

#use the ">"  symbol to write to new file (Write mode)
#use the ">>" to append to the file (Append mode)
open (LOG_FILE, ">$log_file_path"); 

open (READ_FILE, "$read_file_path"); 
#"READ_FILE" is File Handle

unless (-e $read_file_path) {
    print LOG_FILE "\n Read File Doesn't Exist!!! : "  . $read_file_path;
    exit;
} 

#use the ">"  symbol to write to new file (Write mode)
#use the ">>" to append to the file (Append mode)
while my $each_line (<READ_FILE>) {
     print "\n Each Line: :" . $each_line;
}

print LOG_FILE "\nTesting Writing into Perl file";
print LOG_FILE "\nWriting more content 111";
print LOG_FILE "\nWriting more content 222";

close (READ_FILE); 
close (LOG_FILE);

O/P:
log.txt (In case file data.txt is present in the path)
Testing Writing into Perl file
Writing more content 111

Writing more content 222

log.txt (In case file data.txt is not present in the path)
Read File Doesn't Exist!!! : log.txt

Chop Vs Chomp

Now, we discuss about Chop & Chomp functionality in Perl

Chop:

It removes the last character of the string completely (even if the last character is new line or a character)

Chomp:

It only removes the last character if it is a newline. 

Chomp is more useful when we are reading the files to trim the new line characters

Chop: It removes the last character of the string completely

E.g., chop_test.pl

#!/usr/bin/perl

use strict;

use warnings;

$name = "Mother Teresa";

print "Name before chop: " . $name;  

chop($name); 

print "Name after chop: " . $name;  

O/P:

Name before chop : Mother Teresa

Name after chop    : Mother Teres

Chomp: It only removes the last character if it is a newline. 

E.g., chomp_test.pl

#!/usr/bin/perl

use strict;

use warnings;

$name = "Mother Teresa\n";

print "Name before chomp: " . $name;  

chomp($name); 

print "Name after chomp: " . $name;  

O/P:

Name after chomp: Mother Teresa

Name after chomp: Mother Teresa

Write to File in Perl

Now, we discuss about writing content to a file

The following code snippet explains
1) Write (">") mode : You write the content to a new file
2) Append (">>") mode : File already exists, you append the content to the existing file
3) You can either define Write or Append mode based on your requirement
4) Define a file read handle (FILE_WRITE), it's can be any user defined name
5) Write the contents to file

file_write.pl

#!/usr/bin/perl

use strict;
use warnings;

my $filename = 'C:\Perl\data_write.txt';

#use the ">"  symbol to write to new file (Write mode)
#use the ">>" to append to the file (Append mode)
open (FILE_WRITE, ">$filename");
#"FILE_WRITE" is File Handle

print FILE_WRITE "\nTesting Writing into Perl file";
print FILE_WRITE "\nWriting more content 111";
print FILE_WRITE "\nWriting more content 222";
close (FILE_WRITE); 

#use the ">"  symbol to write to new file
#use the ">>" to append to the file


Output: 
data_write.txt
Testing Writing into Perl file
Writing more content 111
Writing more content 222

Reading File in Perl

Let's discuss about reading a file in Perl with an example.

The following code snippet 
1) Checks whether the file (data_read.txt) we are going to read is present or not 
2) Define a file read handle (FILE_READ), it's can be any user defined name
3) Loop through the file handle and read line line from the file
2) Chomp each & every line (to remove the new-line characters (e.g., \n)
4) Print the contents of the file line by line

E.g., 
data_read.txt
Mother Teresa
Nelson Mandela
Abraham Lincoln


file_read.pl

#!/usr/bin/perl

use strict;
use warnings;

my $filename = 'C:\Perl\data_read.txt';

#Check if file is present in the path or not
unless (-e $filename) {
    print "\n File Doesn't Exist!" . $filename;
    exit;
} 

#"FILE_READ" is File Handle
open (FILE_READ, $filename); 

while (my $each_line = <FILE_READ>) {
   
   #"chomp" removes the new lines characters like "\n" at the end of each line 
   chomp $each_line;   
   
   print $each_line . "\n";
}

close (FILE_READ); 



Output:
Mother Teresa
Nelson Mandela
Abraham Lincoln

For Vs Foreach in Perl

Today we discuss about the difference between For & Foreach with simple examples:

FOR Syntax:
Syntax :for (initialization; test; re-initialization) BLOCK


#!/usr/bin/perl
use strict;
use warnings;

print "\nStart For Loop"; 

for ($count = 1; $count <= 5; $count++) {
  print "\nEach Element : " . $count;
}
print "\nEnd For Loop"; 


O/P:
Start For Loop
Each Element : 1
Each Element : 2
Each Element : 3
Each Element : 4
Each Element : 5
End For Loop


Foreach
The foreach keyword is actually a synonym for the "for" keyword, so we can use "foreach" (more readability or brevity)


#!/usr/bin/perl
use strict;
use warnings;

@array = ("Mother Teresa", "Abraham Lincoln", "Nelson Mandela");
foreach my $each (@array) {
  print "\nEach Element : " . $each;
}


O/P:
Each Element : Mother Teresa
Each Element : Abraham Lincoln
Each Element : Nelson Mandela

For Loop in Perl

FOR Loop Syntax in Perl:

Syntax: 
for (initialization; test; re-initialization) BLOCK


#!/usr/bin/perl

use strict;
use warnings;

print "\nStart For Loop"; 

for ($count = 1; $count <= 5; $count++) {
  print "\nEach Element : " . $count;
}
print "\nEnd For Loop"; 

O/P:

Start For Loop
Each Element : 1
Each Element : 2
Each Element : 3
Each Element : 4
Each Element : 5
End For Loop

Python - Filter Vs Map Vs Reduce Vs List Comprehension

Today we discuss about the following topics :
1) Filter
2) Map
3) List Comprehension

Lets discuss with few examples:

Filter :
filter(function, sequence) returns a sequence consisting of those items from the sequence for which function(item) is true

Map :
map(function, sequence) calls function(item) for each of the sequence's items and returns a list of the return values

List Comprehension :
List comprehension in Python provides a clear and concise syntax for creating lists from other lists


filter_test.py

#filter(function, sequence) returns a sequence consisting of those items from the sequence for which function(item) is true
def isPythonFile(list_1):
    if list_1.find(".py") == -1:
        return False
    else:
        return True
 
list_1 = ["1.py","2.pl", "3.zip", "4.py","5.php" ]
py_files = filter(isPythonFile, list_1)   #Function is called for each element of list
 
for item in py_files:
    print("Each item in Filetr :" , item)  

Output:

Each item in Filetr : 1.py
Each item in Filetr : 4.py

filter_lambda_test.py

foo = [2, 18, 9, 22, 17, 24, 8, 12, 27]
print("Before Lambda :", list(foo))
print("After  Lambda :", list(filter(lambda x: x % 3 == 0, foo)))

Output:

Before Lambda : [2, 18, 9, 22, 17, 24, 8, 12, 27]
After  Lambda : [18, 9, 24, 12, 27]  

map_test.py

print("Before Map :", list(foo))
print("After  Map :", list(map(lambda x: x * 2 + 10, foo)))  

Output:

Before Map : [2, 18, 9, 22, 17, 24, 8, 12, 27]
After  Map : [14, 46, 28, 54, 44, 58, 26, 34, 64]  

smallest_no_using_reduce_test.py
#how to use if else in lambda

from functools import reduce

ll = [10, 12, 45, 2, 100]
out = reduce(lambda x, y: x  if x < y  else y,  ll)
print(out)

Output:
2

largest_no_using_reduce_test.py
#how to use if else in lambda

from functools import reduce

ll = [10, 12, 45, 2, 100]
out = reduce(lambda x, y:  x  if x > y  else y,  ll)
print(out)

Output:
100

list_comprehension_test1.py

input_arr = [2, 3, 4]
output_arr = [2*i for i in input_arr if i > 2]
print("List Comprehension Test 1 :", output_arr)  

Output:

List Comprehension Test 1 : [6, 8]

list_comprehension_test2.py

input_arr = ['Mother Teresa', 'Abraham Lincoln', 'Nelson Mandela']
output_arr = ['Dear...' + i  for i in input_arr if len(i) > 5]
print("List Comprehension Test 2 :",output_arr)  

Output:

List Comprehension Test 2 : ['Dear...Mother Teresa', 'Dear...Abraham Lincoln', 'Dear...Nelson Mandela']  

list_comprehension_test3.py  (if-else inside list comprehension)

input_arr = [20, 30, 40, 33, 55]
output_arr = [2*i if i%2 == 0 else i for i in input_arr]
print("List Comprehension Test 3 :", output_arr)  

Output:

List Comprehension Test 3 :  [40, 60, 80, 33, 55]

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Python User Defined Exceptions

Today we discuss about raising exceptions manually (Python User Defined Exceptions).

1) We already know we have built in exceptions in Python like ValueError, IOError, FileNotFoundError, ZeroDivisionError

2) But sometimes we might have to write custom exceptions on our own to catch few scenarios.

Lets discuss how to implement user defined exception with an example.

In the below example :

nonEmptyValException                                  #User Defined Exception

ValueError, ZeroDivisionError, EOFError   #Built-in Exceptions


exceptions_raise_manual.py

import sys

class nonEmptyValException(Exception):
    '''A user-defined exception class.'''
    def __init__(self, num):
        Exception.__init__(self)
        self.num = num
      
try:
    a = int(input('Enter some integer for Dividend --> '))
    b = int(input('Enter some integer for Divisor --> '))
    
    if not a:
       raise nonEmptyValException(a)
       
    if not b:
       raise nonEmptyValException(b)
    
    c = a/b;    

    print ("Final Value is : ", c) 
    
except  ValueError:
    print('ValueError: Please enter only Integer Value')   
except ZeroDivisionError:
    print('ZeroDivisionError: The input divisor is %d, was expecting a Non-Zero number' % (b))   
except EOFError:
    print('\nWhy did you do an EOF on me?')
except nonEmptyValException as error:
    print('nonEmptyValException: The input is %d, was expecting a integer number' % (error.num))  

Please refer to other Python Exceptions Concepts :
All Python Exceptions Related Topics
Exceptions in Python

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Python Regular Expressions (re.compile Vs re.match)

Today we discuss about the following topics :
1) Pre-compiled Regular Expressions - Uses & Advantages (General Concept)
2) Explain the difference between Python re.compile Vs re.match with an example.

Let's dive into the topic & continue the fun :

1) Pre-compiled Regular Expressions - Uses & Advantages (General Concept)
This is a general topic irrespective of any language (Perl or Python or any)
The extra mile of using a pre-compiled regular expression is :

     a) When you have to execute the same regular expression pattern over millions of lines (say reading lines a file), then pre-compiled regular expressions are very handy by drastically reducing the time of execution.

     b) Since the regex pattern is pre-compiled in advance, no need to process the regex pattern each and every time while reading the lines from the file (suppose millions of lines in the file)

    c) Here, we assume, the regex pattern which you have to use, should be constant, if has to vary every time then pre-compiled regex will not be that useful.

2) Explain the difference between Python re.compile Vs re.match with an example.
Let us explain the explain with a lucid example.
Here we are trying to run a regex pattern in loop of 100000 times with both Python re.compile(pre-complied) & re.match
Please check the time difference between in the output file.

E.g., regex_compile_vs_match.py

import re
import time

input_str       = "Mother Teresa"
count_compile   = 0
count_match     = 0

time_val_1 = time.time()

compiled_regex = re.compile('(\w+)\s+(\w+)')

for i in range(1000000):
    if compiled_regex.match(input_str):
        count_compile += 1

print ("Count Compile Value : ", count_compile)        
print("Time taken by Using Compile : ",  time.time() - time_val_1)

time_val_2 = time.time()

for i in range(1000000):
    if re.match('(\w+)\s+(\w+)', input_str):
        count_match += 1

print ("Count Match Value : ", count_compile)          
print("Time taken by Using Match   : ",  time.time() - time_val_2)  

Output:

Count Compile Value : 1000000
Time taken by Using Compile :  1.340999994277954
Count Math Value : 1000000
Time taken by Using Match   :  2.994999885559082  

You might wish to read Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

How to remove duplicate lines from a file in Python

Today, we will discuss removing duplicate lines from a file in Python

Lets discuss in two ways as mentioned below :

1) Removing duplicate lines and print the lines in order (When Order is important)

        - Using normal way

2) Removing duplicate lines and print the lines in any order (When Order is NOT important) 

        - Using SET concept in Python. SET concept in python does not consider Order.

Note:
Both the scripts read duplicated content from file_with_duplicates.txt, 
Read the above file and remove duplicate lines and finally
Print to file_without_duplicates.txt 

Input: file_with_duplicates.txt

Mother Teresa
Winston Churchill
Abraham Lincoln
Mahatma Gandhi
Winston Churchill
Mother Teresa
Abraham Lincoln  

1)

infile = open('file_with_duplicates.txt', 'r')

outfile = open('file_without_duplicates.txt', 'w')



lines_seen = set()

for line in infile:

    if line not in lines_seen:

        outfile.write(line)

        lines_seen.add(line)

outfile.close()

1) remove_duplicate_lines_from_file_with_order.py
- Using Normal Way - it take cares of order of the lines

#!/usr/bin/python

try:
    input_file = open("file_with_duplicates.txt", "r")
    output_file = open("file_without_duplicates.txt", "w")

    unique = []
    for line in input_file:
        line = line.strip()
        if line not in unique:
            unique.append(line)
    input_file.close()
    
    for i in range(0, len(unique)-1):
        unique[i] += "\n"

    output_file.writelines(unique)
    output_file.close
    
except FileNotFoundError:
    print('\n File NOT Found Error')
    sys.exit
except IOError:
    print('\n IO Error')
    sys.exit  

Output: file_without_duplicates.txt

Mother Teresa
Winston Churchill
Abraham Lincoln
Mahatma Gandhi  

2) remove_duplicate_lines_from_file_without_order.py
- Using SET concept - it does not consider the order of the lines

#!/usr/bin/python

try:
    input_file = open("file_with_duplicates.txt", "r")
    output_file = open("file_without_duplicates.txt","w")

    #The main drawback of using sets is, the order of the lines may not be same as in input file
    uniquelines = set(input_file.read().split("\n"))
    output_file.write("".join([line + "\n" for line in uniquelines]))
    
    input_file.close()
    output_file.close()
    
except FileNotFoundError:
    print('\n File NOT Found Error')
    sys.exit
except IOError:
    print('\n IO Error')
    sys.exit      

Output: file_without_duplicates.txt

Abraham Lincoln
Winston Churchill
Mother Teresa
Mahatma Gandhi  

You may also wish to read Object Oriented Concepts in Python as mentioned :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in perl

How to remove duplicate lines from a file in perl

Today, we discuss about the following :
1) How to open the directory & read all the files
2) In each file, remove duplicate lines from a file and keep it in same path/file or different path/file
4) How to create time-stamp in Perl
3) How to create a log file and log the changes

Mention the corresponding path where you want to copy

e.g., If you want to remove duplicated line and copy in the same file
my $input_dir  = "/nfs/fm/disks/my_files";
my $output_dir = "/nfs/fm/disks/my_files";
my $log_dir    = "/nfs/fm/disks/my_files";

e.g., If you want to copy the output of the files in different locations
my $input_dir  = "/nfs/fm/disks/input_dir";
my $output_dir = "/nfs/fm/disks/output_dir";
my $log_dir    = "/nfs/fm/disks/log_fir";

remove_duplicates.pl

#!/usr/bin/perl

use strict;
use warnings;

my $input_dir  = "/nfs/fm/disks/my_files";
my $output_dir = "/nfs/fm/disks/my_files"; 
my $log_dir    = "/nfs/fm/disks/my_files";

my ($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst);

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$sec  = sprintf("%02d",$sec);
$min  = sprintf("%02d",$min);
$hour = sprintf("%02d",$hour);
$mday = sprintf("%02d", $mday);
$mon  = sprintf("%02d", $mon+1);
$year = sprintf("%04d", $year+1900);

my $output_tag = $mday .'_'. $mon .'_'. $year .'_'. $hour .'_'. $min;

opendir(IN_DIR, $input_dir) or die $!; 
opendir(OUT_DIR, $output_dir) or die $!; 
opendir(LOG_DIR, $log_dir) or die $!; 

open (LOG_FILE, "> $log_dir/remove_duplicate_lines.log");

print LOG_FILE "------------- START ------------ \n";

my (@invalid_list_files, %file_without_dup );

while (my $file = readdir(IN_DIR)) {

    next if $file =~ /^\./;

    print LOG_FILE "\n Folder Path : " , "$input_dir/$file";    

    unless (-f "$input_dir/$file") {
        print LOG_FILE "\n Not a File : " . $file;
        next;
    }
        
    print LOG_FILE "\n File Name : " . $file  . "\n";

    #Reading the File 
    open(IN_FILE, "<$file") ||  die "\n Cant open file for reading: " . $file;
    my @lines = 
    close(IN_FILE);

    my @newlines;
    foreach my $each_line (@lines) {
        if (not defined $file_without_dup{$each_line}) {
            push(@newlines, $each_line);
        }
        $file_without_dup{$each_line} = 1;
    }

    #Writing into Output Files
    open(OUT_FILE, ">$output_dir/$file") || die "-W- Cant open file for writing: " . "$output_dir/$file";
    print OUT_FILE @newlines;
    close(OUT_FILE);
    
}
close (LOG_FILE);
closedir(IN_DIR);
closedir(OUT_DIR);
closedir(LOG_DIR);

1;

You may also wish to read Object Oriented Concepts in Python as mentioned :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python

Exceptions in Python

Today, we discuss about the following :
1) Try block in Python
2) finally block
3) Exceptions
-> In the following example, we can catch exceptions occurred in the code.

-> Keep the code in try block, when an exception is there, it is being caught by except

-> Finally block will be called at the last after the end of execution of the block.

-> In this example, there are few well know exceptions like FileNotFoundError, IOError, EOFError, ValueError


read.txt (Input File)

Abraham Lincoln
Mother Teresa
Paul Coelho

exceptions_test.py

  
import time
import sys

try:    
    f = open('read.txt', 'r') 
    while True: # our usual file-reading idiom
        line = f.readline()
        if len(line) == 0:
            break
        time.sleep(0.5)  #1/2 sec
        print(line)
except FileNotFoundError:
    print('\n File NOT Found Error')
    sys.exit
except IOError:
    print('\n IO Error')
    sys.exit
except EOFError:
    print('\nWhy did you do an EOF on me?')
    sys.exit
except ValueError:
    print("\nValue Error.")
    sys.exit
finally:
    f.close()
    print('Cleaning up...closed the file')
                                                    
                                           

You may also wish to read Object Oriented Concepts in Python as mentioned :
Python Class and Object Example
Inheritance in Python
Packages in Python

Packages in Python

Today, We discuss about the following :
1) how to create and use a Package in Python with a simple example.
2) Use of __init__.py
3) How to import classes/function and how to use them

Lets take a ride.

Create a folder structure as mentioned below (Where the package name is Package_Sample):
Package_Sample/__init__.py
Package_Sample/A.py
Package_Sample/B.py
Package_Sample/test.py
Package_Sample/sub_folder/C.py
Package_Sample/sub_folder/D.py

Use of __init__.py
In this file, we define the path of all the files & functions to be imported required for the project.
Once you define here, all these classes/functions, you can access directly, we discuss about this in a short while

e.g.,
from A import *
from B import *
from sub_folder.C import *
from sub_folder.D import *

How to use classes defined in __init__.py
Once you have defined in __init__.py
You can directly access functions of C & D (which are in the path Package_Sample/sub_folder/)
Main advantage is that you no need to specify the whole path (sub_folder) again and again.

e.g.,
from Package_Sample import A
from Package_Sample import B
from Package_Sample import C
from Package_Sample import D



Package_Sample/__init__.py

'''
Once you have defined in __init__.py
You can directly access functions of C & D (which are in the path Package_Sample/sub_folder/)
Main advantage is that you no need to specify the whole path (sub_folder) again and again.
'''

from A import *
from B import *
from sub_folder.C import *
from sub_folder.D import *
 

Package_Sample/A.py

class A:
    def __init__(self, type, height, price, age):
        self.type = type
        self.height = height
        self.price = price
        self.age = age
        
    def print_details(self):
        print("\nThis A Type is  :" + self.type + " type ")
        print("This A Height is  :" + self.height + " foot ")
        print("This A Price is   :" , str(self.price) + " dollars ")
        print("This A Age is     :" , str(self.age) + " years")
	 

Package_Sample/B.py

class B:
    def __init__(self, type, height, price, age):
        self.type = type
        self.height = height
        self.price = price
        self.age = age
        
    def print_details(self):
        print("\nThis B Type is    :" + self.type + " type ")
        print("This B Height is    :" + self.height + " foot ")
        print("This B Price is     :" , str(self.price) + " dollars ")
        print("This B Age is       :" , str(self.age) + " years")
        
        	 

Package_Sample/test.py

#Testing __init__.py
from Package_Sample import A 
from Package_Sample import B
from Package_Sample import C
from Package_Sample import D

cfcpiano_A = A("AAAA", "100 Cms", 11000, 10)   #Constructor or Creating Object for Class A
cfcpiano_A.print_details()

cfcpiano_B = B("BBBB", "200 Cms", 22000, 20)   #Constructor or Creating Object for Class B
cfcpiano_B.print_details()

cfcpiano_C = C("CCCC", "300 Cms", 33000, 30)   #Constructor or Creating Object for Class C
cfcpiano_C.print_details()

cfcpiano_D = D("DDDD", "400 Cms", 44000, 40)   #Constructor or Creating Object for Class D
cfcpiano_D.print_details()


	 

Package_Sample/sub_folder/C.py

   
class C:
    def __init__(self, type, height, price, age):
        self.type = type
        self.height = height
        self.price = price
        self.age = age
        
    def print_details(self):
        print("\nThis C Type is  :" + self.type + " type ")
        print("This C Height is  :" + self.height + " foot ")
        print("This C Price is   :" , str(self.price) + " dollars ")
        print("This C Age is     :" , str(self.age) + " years")

    	 

Package_Sample/sub_folder/D.py

class D:
    def __init__(self, type, height, price, age):
        self.type = type
        self.height = height
        self.price = price
        self.age = age
        
    def print_details(self):
        print("\nThis D Type is  :" + self.type + " type ")
        print("This D Height is  :" + self.height + " foot ")
        print("This D Price is   :" , str(self.price) + " dollars ")
        print("This D Age is     :" , str(self.age) + " years")
        	 

How to Run :
Package_Sample/]# python test.py

You may also wish to read Object Oriented Concepts in Python as mentioned :
Python Class and Object Example
Inheritance in Python

Python Class and Object Example

Lets discuss the basic Object-Oriented concept of Class & Object in Python with a simple example.

Lets create a Person Class and create objects for the Person class.


Python_Class_Object.py (Copy the below snippet and run the program)

class Person:
    '''Represents a person.'''
    count = 0

    def __init__(self, name):
        '''Initializing data.'''
        self.name = name
        print('\n Initializing %s' % self.name)

        # To count the people
        Person.count += 1

    def displayName(self):
        ''' Saying Hello'''
        print('My name is %s.' % self.name)
    
    def count_people(self):
        '''Prints the count.'''
        if Person.count == 1:
            print('I am the only person here.')
        else:
            print('We have %d persons here.' % Person.count)

Winston = Person('Winston Churchil')
Winston.displayName()
Winston.count_people()

Abraham = Person('Abraham Lincoln')
Abraham.displayName()
Abraham.count_people()
	 

You may also wish to read Object Oriented Concepts in Python as mentioned :
Inheritance in Python

Inheritance in Python

Lets discuss about Inheritance in Python using an example... Lets take a ride...

Here we consider Vehicle as Parent Class where as Car & Truck are sub-classes which inherit from the Parent class(Vehicle).

class Vehicle:
    '''Represents any Vehicle.'''
    def __init__(self, name, model):
        self.name = name
        self.model = model
        print('Initialized Vehicle: %s' % self.name)
    
    def details(self):
        '''Call my details.'''
        print('Name:"%s" model:"%s"' % (self.name, self.model))

class Car(Vehicle):
    '''Represents a Car.'''
    def __init__(self, name, model, price):
        Vehicle.__init__(self, name, model)
        self.price = price
        print('Initialized Car: %s' % self.name)

    def details(self):
        Vehicle.details(self)
        print('price: "%d"' % self.price)

class Truck(Vehicle):
    '''Represents a Truck.'''
    def __init__(self, name, model, price):
        Vehicle.__init__(self, name, model)
        self.price = price
        print('Initialized Truck: %s' % self.name)
    
    def details(self):
        Vehicle.details(self)
        print('price: "%d"' % self.price)

c = Car('Cooper', 100, 30000)
t = Truck('Jeep', 200, 50000)

print() # prints a blank line

vehicles = [c, t]
for member in vehicles:
    print() # prints a blank line
    member.details() # works for both Cars and Trucks