Perl Remove Special Characters From File

While reading some kind of log files as mentioned below, we need to get rid of these special characters.

Because of these special characters, it makes the Developers job tough :
To parse the content of a file
To convert the special characters from the file

Let us explain how can we get rid of these special characters with a simple example

test.log

^[[1;31mTest 1^[[0m
^[[1;31mTest 2^[[0m
^[[1;31mTest 3^[[0m
^[[1;31mTest 4^[[0m
^[[1;31mTest 5^[[0m  

In the above test.log file, first of all, what is displayed as ^[ is not ^ and [
But it is the ASCII ESC character, produced by Esc or Ctrl[ (the ^ notation means the Ctrl key).

We can use the following regular expression :

s/\e\[[\d;]*[a-zA-Z]//g;

Note: 
\e represents escape character in the above regular expression (substituting instead of ^[ )
We can shorten from [a-zA-Z] to just [mK], based on the requirement
You can make use of the above regular expression while parsing the file as well (line by line)

In case if you want to backup file (test.log.bak) instead of changing in the original file (test.log) then use the following :
perl -pi.bak -e 's/\e\[[\d;]*[a-zA-Z]//g' test.log

The following will remove the special chars in test.log
perl -pi -e 's/\e\[[\d;]*[a-zA-Z]//g' test.log

Output after removing

Test 1
Test 2
Test 3
Test 4
Test 5
  

Python Regular Expressions (re.compile Vs re.match)

Today we discuss about the following topics :
1) Pre-compiled Regular Expressions - Uses & Advantages (General Concept)
2) Explain the difference between Python re.compile Vs re.match with an example.

Let's dive into the topic & continue the fun :

1) Pre-compiled Regular Expressions - Uses & Advantages (General Concept)
This is a general topic irrespective of any language (Perl or Python or any)
The extra mile of using a pre-compiled regular expression is :

     a) When you have to execute the same regular expression pattern over millions of lines (say reading lines a file), then pre-compiled regular expressions are very handy by drastically reducing the time of execution.

     b) Since the regex pattern is pre-compiled in advance, no need to process the regex pattern each and every time while reading the lines from the file (suppose millions of lines in the file)

    c) Here, we assume, the regex pattern which you have to use, should be constant, if has to vary every time then pre-compiled regex will not be that useful.

2) Explain the difference between Python re.compile Vs re.match with an example.
Let us explain the explain with a lucid example.
Here we are trying to run a regex pattern in loop of 100000 times with both Python re.compile(pre-complied) & re.match
Please check the time difference between in the output file.

E.g., regex_compile_vs_match.py

import re
import time

input_str       = "Mother Teresa"
count_compile   = 0
count_match     = 0

time_val_1 = time.time()

compiled_regex = re.compile('(\w+)\s+(\w+)')

for i in range(1000000):
    if compiled_regex.match(input_str):
        count_compile += 1

print ("Count Compile Value : ", count_compile)        
print("Time taken by Using Compile : ",  time.time() - time_val_1)

time_val_2 = time.time()

for i in range(1000000):
    if re.match('(\w+)\s+(\w+)', input_str):
        count_match += 1

print ("Count Match Value : ", count_compile)          
print("Time taken by Using Match   : ",  time.time() - time_val_2)  

Output:

Count Compile Value : 1000000
Time taken by Using Compile :  1.340999994277954
Count Math Value : 1000000
Time taken by Using Match   :  2.994999885559082  

You might wish to read Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Regular Expressions Modifers

/i    => ignore case
/g    => global match
/s  => single line mode    
/m   => multi line mode
/x  => free-spacing mode
/o  => One-time pattern compilation

$text = "foo\nfoot\nroot";

/s => singile-line mode trates the whole as a single line including \n as well, it has only one start (^) and end ($)
/m => multi-line mode treats the string $text as 3 lines with each line starting with ^ and $

/s, /m Example:

$text = "foo\nfoot\nroot";

$text =~ /^foo/g;           # matches only the first foo

$text =~ /^foo/gm;          # matches both foo

$text =~ /f.*t/g;           # matches only foot

$text =~ /f.*t/gs;          # matches foo\nfoot\nroot

$text =~ /f.*?t/gs;         # matches foo\nfoot
    here \s is the modifier, so it treats the whole as only one string (it won't bother about \n)
    .* is greedy operator
    .*? restricts the greediness till the first occurance

$text =~ /^foot.*root$/g;   # doesn't match
    its understandable

$text =~ /^foot.*root$/gm;  # doesn't match
    here \m is the modifier, so it treats the string as
    foo
    foot
    root
no where it has the foot.*root, so it didn't match

$text =~ /^foot.*root$/gs;  # doesn't match
    here \s is the modifier, so it treats the whole as only one string (it won't bother about \n)
    the string is not starting with foot

$text =~ /^foot.*root$/gms; # matches foot\nroot
    Carefully observe here we have both modifiers \m and \s
    foo    (using \m it splitted)
    foot\nroot (using \s it matched the string as required)       

   
/o modifier (One time compilation) - Compiled regular expression
When using a regular expression containing an interpolated Perl variable that you are confident will not change during the execution of the program, a standard speed-optimization technique is to add the /o modifier to the regex pattern.
This compiles the regular expression once, for the entire lifetime of the script, rather than every time the pattern is executed   
   
e.g.,

@list = qw/prabhath 100 lakshmi 200 500/;
my $pattern = '^\d+$';  #Only digit validation
                        #This will compile only once, if you are confident that the regex will not change, you can go for it
foreach my $each (@list) {
    if ($each=~/$pattern/o) {
        print "\n Only Digits Match : " . $each;
    }   
}

Output:
Only Digits Match : 100
Only Digits Match : 200
Only Digits Match : 500


/x modifier - Free Spacing Mode


m/\w+:(\s+\w+)\s*\d+/;       # A word, colon, space, word, space, digits.

m/\w+: (\s+ \w+) \s* \d+/x;  # A word, colon, space, word, space, digits.

m{
    \w+:                     # Match a word and a colon.

    (                        # (begin group)
         \s+                 # Match one or more spaces.
         \w+                 # Match another word.
    )                        # (end group)
    \s*                      # Match zero or more spaces.
    \d+                      # Match some digits
}x;
   

qr//  - Compiling a pattern

    $string = "people of this town";   
   
    $pattern = '^peo';
    $re = qr/$pattern/;

    if($string =~ /$re/) {
        print "Matched Pattern, string starts with p";
    } else {
        print "String does'nt start with p";
    }

Result:
    Matched Pattern, string starts with p

Non Capturing Paranthesis (?:) in Regular Expressions

From my earlier blog on capturing, you might have understood about the capturing concept in regular expressions.

Let us now look the Non-capturing concept in regular expressions with a simple example

To achieve non-capturing parenthesis use "?:" 

E.g.  $input = +456.987c
        $input = ~/([-+]?[0-9]+(?:\.[0-9]*)?)([cf])$/     
   
     Where  $1 = ([-+]?[0-9]+(\.[0-9]*)?) #which matches '456.987'
                  $2 = ([cf])                                #which matches 'c'

Carefully observe, here
$2 is not (?:\.[0-9]*) but instead it is ([cf]), why because (?:\.[0-9]*) is starting with ?: which means it is not being captured, so $2 becomes ([cf])
                     
Advantages of Non-Capturing:
    1) Avoids unnecessary Capturing (Matches only what ever required)
    2) Efficiency Enhancement



Please refer to my earlier blog on capturing

I will be discussing about "Look Around" concepts of regular expressions in my coming blog.

Don't hesitate to comment if you have any doubts or queries on this topic.

Thanks for your valuable time and have a great day :-)

Capturing in Regular Expressions

Parenthesis inside regex will be grouped as well as captured.
Let me explain the grouping concept with an example where in you want to validate a decimal part

Eg: Validation for decimal part as given in the below mentioned example

        $input = +345.34f
        $input = ~/([-+]?[0-9]+(\.[0-9]*)?)([cf])$/
        where
                  $1 = /([-+]?[0-9]+(\.[0-9]*)?)    # 345.34
                  $2 = (\.[0-9]*)                            # .34
                  $3 = ([cf])                                  #  f

Notations used in the above example :
1) $1, $2, $3 are grouping data, one can capture the specific data and use it later in the program with $1, $2, $3 etc.,
2) *, + are greedy operators, they match more than what they required, that's why they are called as Greedy Operators
3) ? is a limiting operator which limits the mapping to what ever required
4) [] is a character class
5) $2 lies within $1 even then they both are distinct
6) $ represents the end of the string

Advantages of capturing:
 a) You can capture the data and use the data later in the program
 b) Very useful when we want to grep or capture tricky regex

Disadvantages of unnecessary Capturing:
 a) Waste of memory.
 b) Performance problem.
 c) Regex will be costlier


I hope you have really liked today's topic.

Please share your views with me in case of any any doubts/suggestions. Feel free, don't hesitate :-)

I will be explaining about the Non-Capturing Parenthesis in my next blog. Keep watching my blog.

Thanks for your valuable time. Have a good day :-)

Regular expressions or Regex or Regexp

Regular Expressions are some patterns describing some amount of text.

Real World Application of Regular Expressions:
Suppose you have a file that contains comma separated employee details like
First name, Last name, Cell no, Salary, address, Email etc.,
1) You want to extract only the Employee name and email-id from that huge list of data.

2) You are looping through the employee details, you just want to pick all the employees info whose salary is greater than 5000$
3) You want to replace all the occurrences of a misspelled employee name
e.g., Grorge to George

And lot more............



Advantages of Regular Expressions:
1) Regular expressions is a part and parcel in Perl.
2) It makes the life easy for a developer or programmer to search/replace a pattern in a line.
3) It saves a lot of time of the developer or programmer


General Topics on Regular Expressions:
1) Search & Replace
2) Start and End of String or Match
3) Word Boundaries etc.,


Advanced Topics on Regular Expressions:
1) Capturing
2) Non-Capturing PAranthesis
3) Look Around Concepts
4) Look Ahead
5) Look Behind
6) Back Tracking
7) Atomic Grouping
8) Possessive Quantifiers etc.,

We will discuss these topics one by one in brief in the coming posts.
Thanks for your valuable time, Have a good day :-)

substitute for n'th occurance

#!F:\Perl\bin\perl -w
use strict;

# Substitue 3rd occurance of 'perl' with 'PERL'
my $text = 'perl is good, perl is better, perl is best';
print "\n INPUT Text:", $text;

my $nth_occurance = 3;
my $count=0;

$text =~ s{(perl)}{
++$count == $nth_occurance ? 'PERL' : $1
}ige;

print "\n OUTPUT Text:", $text,"\n";

Make first letter of every word in a string to Upper case

#!F:\Perl\bin\perl -w
use strict;

my $text = 'india is a great country';
print "\n Before:", $text; # india is a great country
$text =~ s/(\w+)/\u$1/g; # \u option is used
print "\n After :", $text,"\n"; # India Is A Great Country