AWS Route 53 Configuration

Using CLI53 Tool - Export from GoDaddy & Import Zone File to Route53

cli53 is a Python script for managing Amazon Route 53.

It's freaking awesome as it can import, export and help you debug your Route 53 setup.

Ref:

http://amix.dk/blog/post/19712

http://blog.domenech.org/2014/12/how-to-transfer-your-dns-configuration-to-aws-route-53.html

1) Godadd'y DNS Manager you can export your domain's setup:

export file <todoist.com.zone>

2) pip install cli53

3) Simply add $ORIGIN yourdomain.com. at the top of the Godaddy's export file

from Step 1

$ORIGIN yourdomain.com.

e.g., $ORIGIN todoist.com.

4) Import data with cli53

- Create a hosted Zone : cli53 create todoist.com

- Import your Godaddy export : cli53 import todoist.com --file <todoist.com.zone>

5)  Use the excellent "Interstate53" service to recheck that everything is setup correctly

     using a web interface.

      Or

      just use "cli53 rrlist command"

     e.g, cli53 rrlist todoist.com

6) Debug and re-check everything

For GoDaddy:

dig -t any @PDNS01.DOMAINCONTROL.COM todoist.com

For Route53:

dig -t any @ns-158.awsdns-19.com todoist.com

7) Update your domain's nameservers

- The last step is to update your domain's nameservers to point to Route 53

8) Transfer your domain from GoDaddy

- Ref:  https://www.namecheap.com/support/knowledgebase/article.aspx/876/83/how-to-transfer-a-domain-from-godaddy

1. How to unlock your domain with GoDaddy:

- Login to your GoDaddy account

- Next to Domains click on Manage

- Select the domain(s) to unlock and click Lock

- Select Off radio-button to have the domain(s) unlocked and click Save

2. How to obtain EPP/Authorization code from GoDaddy:

- Login to your GoDaddy account

- Next to Domains click on Launch

- Click on the domain you need EPP code

- Click Email my code in the Authorization Code field - send

3. You would also need to disable privacy protection service for the domain (if it's enabled).

- Go to the Domains By Proxy website and login to your account (the login details are not the same as for your GoDaddy account. Look for the email sent by support@domainsbyproxy.com for the login details)

- Select a check box next to the domain name(s) you need to disable privacy protection for

- Click Cancel Selected, OR click on the cancellation icon next to the domain name

- In "Confirm" window click OK

4. How to accept the transfers at GoDaddy.

- Log in to your Account Manager

- Next to Domains, click Manage,

- From the Domains menu, select Transfers

- Click on Pending Transfers Out, select the domain name(s) you are transferring from GoDaddy and click on Accept/ Decline above

- Select Accept and click OK. The request will be processed within 15 minutes.

Using Route53 Console UI - Export from GoDaddy & Import Zone File to Route53

• Get a zone file from the DNS service provider that is currently servicing the domain. The process and terminology vary from one service provider to another. Refer to your provider's interface and documentation for information about exporting or saving your records in a zone file or a BIND file.

If the process isn't obvious, try asking your current DNS provider's customer support for your records list or zone file information.

• Sign in to the AWS Management Console and open the Amazon Route 53 console at https://console.aws.amazon.com/route53/.
• On the Hosted Zones page, create a new hosted zone:

• Click Create Hosted Zone.
• Enter the name of your domain and, optionally, a comment. Note that the comment can't be edited later.
• Click Create.

• On the Hosted Zones page, double-click the name of your new hosted zone.
• Click Import Zone File.
• In the Import Zone File pane, paste the contents of your zone file into the Zone File text box.
• Click Import.

Django Example Project - Student Application

About Django Student Application

This student application achieves a reporting tool for subject and marks of the student

Based on the subject filtering, this application generates a brief report on student marks (Beautiful Pie Chart)

Download Application

https://github.com/prabhathkota/Django-Student-Application

Prerequisites

Python

Django

MySQL

HighCharts

Jqury

JavaScript Modules Used

Highcharts

 http://code.highcharts.com/modules/exporting.js

  http://code.highcharts.com/highcharts.js

Jquery

http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js

Steps to install Student Application

django-admin.py startproject StudentWebsite

python manage.py startapp StudentApp

python manage.py validate

python manage.py sqlall StudentApp

MySQL Database Details

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.mysql',

        'NAME': 'studentsdb',                     

        'USER': 'XXXX',

        'PASSWORD': 'XXXX',

        'HOST': '127.0.0.1',                     

        'PORT': '3306',                

    }

}

Insert MySQL sample data

Note: You can either add the data from Admin page (one by one) or Load/Import sample "studentapp.sql" into database

https://github.com/prabhathkota/Django-Student-Application/blob/master/studentapp.sql

Database Models Used:

Student

Subject

StudentMarks

How to start Django Development Server

Django develeopment server works on default port 8000

python manage.py runserver 8000

Access Home Page

http://127.0.0.1:8000/home/

How Application Looks

Select the desired subject

After selecting the subject as Physics - Generates the report as above

models.py

#-*- encoding=UTF-8 -*-
from django.db import models
from django.core.exceptions import ValidationError
import re

class Student(models.Model):
    firstname = models.CharField(max_length=30, null=False, blank=False)
    lastname = models.CharField(max_length=30, null=False, blank=False)

    def clean(self):
        if re.match(r'^\s+$', self.firstname):
            raise ValidationError('Firstname should contain some characters, only spaces not allowed')
        if re.match(r'^\s+$', self.lastname):
            raise ValidationError('Lastname should contain some characters, only spaces not allowed') 
 
    def __unicode__(self):
        return u'%s %s' % (self.firstname, self.lastname)

    class Meta:
        unique_together = (("firstname", "lastname"),)
        ordering = ['firstname']

class Subject(models.Model):
    name = models.CharField(max_length=10, unique=True, null=False, blank=False)
    
    def clean(self):
        if re.match(r'^\s+$', self.name):
            raise ValidationError('Subject should contain some characters, only spaces not allowed')
    
    def __unicode__(self):
        return u'%s' % (self.name)

class StudentMarks(models.Model):
    student = models.ForeignKey(Student)
    subject = models.ForeignKey(Subject)
    marks   = models.IntegerField(max_length=3, null=False, blank=False)

    class Meta:
        unique_together = (("student", "subject"),)  

urls.py

from django.conf.urls import patterns, include, url

# Uncomment the next two lines to enable the admin:
from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    # Examples:
    # url(r'^$', 'StudentWebsite.views.home', name='home'),
    # url(r'^StudentWebsite/', include('StudentWebsite.foo.urls')),

    # Uncomment the admin/doc line below to enable admin documentation:
    url(r'^admin/doc/', include('django.contrib.admindocs.urls')),

    # Uncomment the next line to enable the admin:
    url(r'^admin/', include(admin.site.urls)),
)

urlpatterns += patterns('StudentWebsite.StudentApp.views',
 #url(r'^$', 'home'),
 url(r'^home/$', 'subjectView', name="home"),
    url(r'^getMarksDetailsView/(?P\w+)/$', 'getMarksDetailsView', name="getMarksDetailsView"),
)

views.py

#-*- encoding=UTF-8 -*-
from django.http import HttpResponse, HttpResponseRedirect, Http404
#from django.http import *
from django.shortcuts import render, render_to_response
from django.core.urlresolvers import reverse
from StudentWebsite.StudentApp.models import Subject, Student, StudentMarks
from django.db.models import Avg, Count

def subjectView(request):
    subjectList = Subject.objects.values('pk', 'name')
    side_bar = {'Home':'home', 'Admin' : 'admin:index'}
    return render(request, 'StudentsMain.html', {'subjects' : subjectList, 'side_bar' : side_bar})

def getMarksDetailsView(request, subjectName):
    subjectStudentDetails = StudentMarks.objects.filter(subject__name=subjectName).values('student__firstname', 'student__lastname', 'subject__name', 'marks')
    avgSubjectMarks = Subject.objects.filter(name=subjectName).annotate(average_rating=Avg('studentmarks__marks'))[0].average_rating
    side_bar = {'Home':'home', 'Admin' : 'admin:index'}
    ranges = [ '0-40', '41-60', '61-80', '81-90', '91-100']
    marksRange = {}
    for eachRange in ranges:
        count = StudentMarks.objects.filter(subject__name=subjectName).filter(marks__range=(eachRange.split('-'))).count()
        marksRange[eachRange] = count 
    return render(request, 'StudentSubjectDetails.html', {'subjectStudentDetails' : subjectStudentDetails, 'avgSubjectMarks' : avgSubjectMarks, 'side_bar' : side_bar, 'marksRange' : marksRange, 'subjectName' : subjectName.title() })

admin.py

from django.contrib import admin
from StudentWebsite.StudentApp.models import Student, Subject, StudentMarks

#This is just for dispaly order of columns for Author
class StudentAdmin(admin.ModelAdmin):
    list_display = ('firstname', 'lastname')
    search_fields = ('firstname', 'lastname')
    ordering = ('-firstname',)

class SubjectAdmin(admin.ModelAdmin):
    list_display = ('name',)
    list_filter = ('name',)
    ordering = ('-name',)
    fields = ('name',)

class StudentMarksAdmin(admin.ModelAdmin):
    list_display = ('student', 'subject', 'marks')
    list_filter = ('student', 'subject', 'marks')
    ordering = ('-student',)

admin.site.register(Student, StudentAdmin) #StudentAdmin is just for display in Admin page
admin.site.register(Subject, SubjectAdmin) #SubjectAdmin is just for display in Admin page
admin.site.register(StudentMarks, StudentMarksAdmin)

Django Tutorial - Part I

Reference: 
Django Project / Django Book

Django web framework / Python web development with Django

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
The tage line for Django is "The web framework for perfectionists with deadlines"
Django makes it easier to build Web apps more quickly and with less code, mainly used for web development

Main components of The Django framework

Object-relational mapper
Define your data models entirely in Python. You get a rich, dynamic database-access API for free, but you can still write SQL if required.
If your web application is data extensive, better write your SQL instead of object-relational mapper

Automatic admin interface
It comes with Admin interface, you need to enable it in the web framework
There by saving extra effort in creating the admin interface from the scratch

Elegant URL design
Very Flexible URL design using loose coupling
URL definitions and the view functions they call are loosely coupled; that is, the decision of what the URL should be for a given function, and the implementation of the function itself, reside in two separate places. This lets you switch out one piece without affecting the other.
Map the URLs and view methods, if you want to change the URL pattern later, you can do it very easily by changing urls.py
It makes use of regular expression for mapping the URL patterns
It generates dynamic URLs as well

Template system
Use Django's powerful, extensible and designer-friendly template language to separate design, content and Python code.

Cache system
Hook into memcached or other cache frameworks for super performance — caching is as granular as you need.

Internationalization
Django has full support for multi-language applications, letting you specify translation strings and providing hooks for language-specific functionality.

How to Install Django

Download Django (https://www.djangoproject.com/download/) and unzip

django-admin.py should be on your system path if you installed Django via its setup.py utility.
on Windows, you need to update your PATH environment variable.

python setup.py install
python -c 'import sys, pprint; pprint.pprint(sys.path)' #this locate site-packages direcotry

How to check Django Installed

>>> import django
>>> django.VERSION
(1, 4, 2, 'final', 0)

How to create a project

django-admin.py startproject <proj_name>
E.g., django-admin.py startproject mysite

Django Project Structure

The startproject command creates a directory containing five files:
mysite/
    manage.py
    mysite/
        __init__.py
        settings.py
        urls.py
        wsgi.py

How to access Django from shell

python manage.py shell

Django Development Server

Django comes with its own development server

Difference between Django Development Server Vs Web Development Server like Apache is as mentioned below :

Advantage:
Comes as part of Django itself
You can develop your site rapidly without having to deal with configuring your production server (e.g., Apache) until you are ready for production
Django Development server watches your code and automatically reloads it. making it easy for you to change code without restarting server

Disadvantage:
Not suitable for production environment
You have to deploy your code to Apache when moving to production

Django Development Server suitable in development environment
Apache web development server suitable in production environment

How to start Django Development Server

Django develeopment server works on default port 8000
python manage.py runserver 8000

How to test home page on Web Page

Using Django Development Server
http://127.0.0.1:8000/

Using Apache Server (If you configure Apache)
http://127.0.0.1/mysite/

How to configure Apache with Django

In Apache httpd.conf
WSGScriptAlias /mysite "C:/django_workspace/mysite/mysite/wsgi.py"


Thanks for reading this article. I will discuss more examples in the coming sessions, stay tuned :-)

UIAutomator Tutorial - Part II

In continuation to earlier post on UIAutomator Introduction / UIAutomator Basics lets discuss more UIAutomator Android with few examples

Reference: UIAutomator Documentation / Android Developer Docs

UIAutomator API in brief

It provides six different classes, these classes with interfaces and exceptions allows to capture and manipulate UI components on Android Device.

Following are the classes :
   UiDevice
   UiObject
   UiScrollable
   UiSelector
   UiCollection
   UiConfigurator

UiDevice
  Provides access to state information about the device. We can  also use this class to simulate user actions on the device, such  as pressing the d-pad hardware button or pressing the Home and Menu buttons.
UiObject
 Represents a user interface (UI) element.
UiScrollable
 Provides support for searching for items in a scrollable UI .
UiSelector
 Represents a query for one or more target UI elements on a device screen.
UiCollection
  Used to enumerate  screen elements for the purpose of counting, or targeting a sub elements by a child's text or description.
UiConfigurator
  Allows  to set key parameters for running uiautomator tests

We need to import following classes

import android.widget.LinearLayout;

import com.android.uiautomator.core.UiObject;

import com.android.uiautomator.core.UiObjectNotFoundException;

import com.android.uiautomator.core.UiScrollable;

import com.android.uiautomator.core.UiSelector;

import com.android.uiautomator.testrunner.UiAutomatorTestCase;

import com.android.uiautomator.core.UiDevice;

UiObject/UISelector

Click on Text

new UiObject(new UiSelector().text(text)).click();

Click on Button

We need to use Button Class name for this

new UiObject(new UiSelector().text(btnText).className("android.widget.Button")).click();

Long Click

new UiObject(new UiSelector().text(text)).longClick();

Press Back

getUiDevice().pressBack();

Go Back to Home

getUiDevice().pressHome();

Click and Wait

new UiObject(new UiSelector().description(text)).clickAndWaitForNewWindow();

Check if text exists

new UiObject(new UiSelector().text(value)).exists();

Wait until 

new UiObject(new UiSelector().className(android.widget.ProgressBar.class.getName())).waitUntilGone(1000);

UIDevice

Example : Swipe Down Notification

import com.android.uiautomator.core.UiDevice;

import com.android.uiautomator.core.UiObjectNotFoundException;

public static void swipeDownNotificationBar() throws UiObjectNotFoundException {

UiDevice deviceInstance = UiDevice.getInstance();

int dHeight = deviceInstance.getDisplayHeight();

int dWidth = deviceInstance.getDisplayWidth();

int xScrollPosition = dWidth / 2;

int yScrollStop = dHeight / 2;

UiDevice.getInstance().swipe(xScrollPosition, 0, xScrollPosition, yScrollStop, 100);

}

UiScrollable

Example: 

1) Scroll and Click event

import com.android.uiautomator.core.UiObject;

import com.android.uiautomator.core.UiScrollable;

new UiScrollable(new UiSelector().scrollable(false))

.scrollIntoView(new UiSelector().text(text));

new UiObject(new UiSelector().text(text)).click();

2) Scroll Back

new UiScrollable(new UiSelector().scrollable(true)).scrollBackward();

How to create a project in Eclipse

Create a Java project in Eclipse

After Creating Java Project, add external jar files which are android.jar and uiautomator.jar

android.and uiautomator.jar are present in

“D:\adt-bundle-windows-x86\adt-bundle-windows\sdk\platforms\android-4.2”

Create Java File

Create Test.java in project / eclipse work space, say it is in D:\UIAutomator

public void clickText(String text) throws UiObjectNotFoundException {

    new UiObject(new UiSelector().text(text)).click();

}

How to build UIAutomator project

You need Apache Ant software for building Java project

Syntax:

android create uitest-project -n "project-name" -t 1 -p "project-location"

"-t" referes to Android target (in case of different versions of Android are present, you can define which target to build this JAR)

E.g.,

cd D:\UIAutomator

D:

android create uitest-project -n UIAutomatorTest -t 1 -p D:\UIAutomator

ant build

cd bin

adb push UIAutomatorTest.jar /data/local/tmp

cd ..

The above commands create a JAR file in the bin folder

Then push the JAR file to /data/local/tmp folder of Android phone

How to call UIAutomator methods from Adb Shell

Go to command line and execute the below command

adb shell uiautomator runtest UIAutomatorTest.jar -c com.test.uiautomator.Test#clickText -e text "Phone"

Here Test refers to Test.java class

ClickText refers to the method inside Test.java

-e <Name> <Value> pairs for passing to test classes

If you haven't go through earlier post, Please go through

UIAutomator Introduction / UIAutomator Basics

UIAutomator Tutorial - Part I

Reference: http://developer.android.com/tools/help/uiautomator

What is UIAutomator / UIAutomator Introduction

• UIAutomator  is Introduced  by Google and it is available from Jellybean Versions of Android.
• The UIAutomator is a testing framework provided by Google's Android 

Use of UIAutomator

• To test Android Applications efficiently by creating automated functional UI test cases that can be run against applications on one or more devices.
• Works on all the Android applications, including Google’s installed apps such as Settings, Contacts, Phone, etc.

Skills Required for working on UIAutomator

Core Java
Software: ANT, Android SDK, Eclipse

How UIAutomator works

• Java project is created using android.jar and uiautomator.jar. 
• uiautomator.jar provided by Android,  has in built methods which are used to perform UI actions  on Android device.
• Create the Android test project based on your requirement; build a Jar file for the Java project is created and push the JAR file to /data/local/tmp location of Android device. 
• Using adb shell, one can access the methods inside the Jar

How to build UIAutomator Project

cd D:\UIAutomator
D:
android create uitest-project -n UIAutomatorTest -t 1 -p D:\UIAutomator
ant build
cd bin
adb push UIAutomatorTest.jar /data/local/tmp

The above commands create a JAR file in the bin folder
Then push the JAR file to /data/local/tmp folder of Android phone

How to invoke a method in the JAR file

   adb shell uiautomator runtest UIAutomatorTest.jar -c com.test.uiautomator.Bluetooth#btPair

Debug UIAutomator

• Using android DDMS tool (present in Eclipse), we can analyze properties of Android screen (Activity)
• Using android DDMS tool, we can get the screenshot of the android screen and access the layout of the android screen, there implement the logic of what action we want to achieve

Advantages / Pros

• Free ware, developed by Google, trust-able
• Simple, uses class object OOPS methodology, less time to implement
• Handling asynchronous events like Toasts,dialog and alerts occurring  during tests is easy. (JAR approach)

Disadvantages / Cons

• Works only from Android version 4.1. 

UIAutomator API

• It provides six different classes, these classes with interfaces and exceptions allows to capture and manipulate UI components on Android Device.

Following are the classes : 
   UiDevice 
   UiObject 
   UiScrollable 
   UiSelector 
   UiCollection 
   UiConfigurator 

UiDevice 

  Provides access to state information about the device. We can  also use this class to simulate user actions on the device, such  as pressing the d-pad hardware button or pressing the Home and Menu buttons. 

UiObject 

 Represents a user interface (UI) element. 

UiScrollable 

 Provides support for searching for items in a scrollable UI . 

UiSelector 

 Represents a query for one or more target UI elements on a device screen. 

UiCollection 

  Used to enumerate  screen elements for the purpose of counting, or targeting a sub elements by a child's text or description. 

UiConfigurator 

  Allows  to set key parameters for running uiautomator tests  



Thanks for reading. I will discuss more samples/examples on UIAutomator in detail in the next post

python serial

The following example illustrates python serial communication / python read serial port 

What you need to have
Connect the device you want to read the SERIAL logs to the system
To know the port is working, cross check by opening the port and check the logs in TeraTerm
If TeraTerm already using the port, you cannot read the logs from SERIAL port, so make sure before executing the script, stop the TeraTerm

What the script will do
We use python module 'serial'
Create a serial object by passing port, baudrate, bytesize, parity etc.,
To continuously read the logs, use Python Thread concept

from serial import *
from threading import Thread

text_received = ''

def receiveSerialDataFromPort(ser):
    global text_received
    read_buffer = ''

    while True:
        read_buffer += ser.read(ser.inWaiting())
        if '\n' in read_buffer:
            text_received, read_buffer = read_buffer.split('\n')[-2:]
            print text_received

if __name__ ==  '__main__':
    ser = Serial(
        port='COM23',
        baudrate=230400,
        bytesize=EIGHTBITS,
        parity=PARITY_NONE,
        stopbits=STOPBITS_ONE,
        interCharTimeout=None
    )
    
    Thread(target=receiveSerialDataFromPort, args=(ser,)).start()
  

python find

Python find helps us to find a search pattern in a given text
Python method find() determines if a sub-string presents in a given string

Syntax:
str.find(str, beg=0 end=len(string))

For Example:
stra = "python find substring"
strb = "substring"

stra.find(strb)
stra.find(strb, 10) #This will search the required sub-string from index 10
stra.find(strb, 20) #This will search the required sub-string from index 20

print stra.find(strb)             #12
print stra.find(strb, 10)       #12
print stra.find(strb, 16, 30)  #-1
print stra.find(strb, 20, 30)  #-1

Return Value
find() returns -1 if there is no match, else it will return the index of the match

Example:

def getText():
    return "Python find Return the lowest index in s where the substring sub is found such that sub is wholly contained in s[start:end]. Return -1 on failure. Defaults for start and end and interpretation of negative values is the same as for slices";

input_text = getText()

print input_text.find("Defaults for start")
print "\n"

if (input_text and input_text.find("Defaults for start") != -1):
    print "\n Found"
else:
    print "\n Not Found"
  

python check process running

We make use of python subprocess module

• To write/execute the command in the command line
• To write to stdin and read the output of stdout

Methods Used
1) checkProcessRunning
        To check whether process in running or not

2) writeToCMD
        To execute the command in the command line and read the output of it

Pass the required arguments
checkProcessRunning(cmd = "adb shell ps", processName = "com.android.phone")

cmd -> Command to execute
processName -> Text to search from the command line output

How to Run
python python_check_process_running.py

import subprocess

def checkProcessRunning(cmd, processName):
    print "\n Process to check : " + cmd
    result = writeToCMD(cmd)
    if (result[0].rstrip().find(processName) != -1):
        print processName + " is present/running"
        return True
    else:
        print processName + " is not present/running"
        return False
          
def writeToCMD(cmd):
    proc = None       
    proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell="True")
    stdout_value = proc.communicate()
    if stdout_value:
        return stdout_value

checkProcessRunning(cmd = "adb shell ps", processName = "com.android.phone")
  

Python Delete old files

Many a times, we come across deleting old files like logs, deprecated files from file system

We come across scenarios like 

To delete logs older than x days

To delete txt files older than x days

Let us discuss how we can achieve deleting old files using python script

What does the script do?

It will take the path from where you need to delete the txt files

The below program checks the time stamp of the files and deletes the txt files older than 7 days.

You can change the file extension, folder path and use it.

folder_path = "C:\Files_To_Read"

file_ends_with = ".txt"

how_many_days_old_logs_to_remove = 7

How to call the script?

python delete_old_logs.py

delete_old_logs.py

import os, time, sys

folder_path = "C:\Files_To_Read"
file_ends_with = ".txt"
how_many_days_old_logs_to_remove = 7

now = time.time()
only_files = []

for file in os.listdir(folder_path):
    file_full_path = os.path.join(folder_path,file)
    if os.path.isfile(file_full_path) and file.endswith(file_ends_with):
        #Delete files older than x days
        if os.stat(file_full_path).st_mtime < now - how_many_days_old_logs_to_remove * 86400: 
             os.remove(file_full_path)
             print "\n File Removed : " , file_full_path
 

perl extract urls from file

As a developer, we need to extract URLs from an input file quite often in our day to day work.
Let us discuss how to extract URLs from an input Text or HTML file using a perl snippet.
Let us automate extracting URLs

What does the script do?
The following script will take input file as command line argument.
It will read the file line by line, extract the HTTP urls and push the results into an array.
Finally prints the array object.

How to call the script?
perl get_urls_from_input_file.pl "C:\Files_To_Read\blog_html.txt"


get_urls_from_input_file.pl

use strict;
use warnings;
use warnings;
use Data::Dumper;

my %substitute = ( '%20' => '', '%3A' => '' , '%24' => '');
my @result_urls;

#perl get_urls_from_input_file.pl 'C:\Files_To_Read\blog_html.txt'

my $infile  = $ARGV[0];

open(my $fh, '<', $infile) or die "Could not open logfile: $!";
while ( my $each_line = <$fh> ) {
    chomp $each_line;

 if ($each_line =~ /['"](http:\/\/.*?)['"]/) {
  my $grep_url = $1;
  push(@result_urls, $1);
 }
}
close $fh;

print "\n" . Dumper(\@result_urls);

1;

Perl Config File

We can make use of Config::Simple module for this.
This library supports parsing, updating and creating configuration files.

Main Features of Config::Simple are as mentioned below:
1) It allows to read config file in different formats/styles like INI-FILE format and HTML format.
2) It allows to read config file in the form of objects and access the variables from the object.
3) It allows to fetch all the variables at into a hash/hashref using "vars" method.

Let's discuss how can we read/modify/write config files easily in perl as mentioned below :
Reading config file in INI-FILE (ini) style
Reading config file in HTTP-LIKE style
Creating config file in INI style

1) Reading/Updating config file in INI-FILE (ini) style

If the configuration file has different blocks, then this style is very useful
Let's explain with the below mentioned example

db_ini.cfg

[mysql]
host=DBI:mysql:host
login=mysql_user
password=mysql_pass
db_name=test
RaiseError=1
PrintError=1

[oracle]
host=DBI:oracle:host
login=oracle_user
password=oracle_pass
db_name=oracle_db
RaiseError=1
PrintError=1
  

Script

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;
use Config::Simple;

my $cfg = new Config::Simple('db_ini.cfg');

#Get Values from Config File
print "\n MySql DB Name     : " . $cfg->param("mysql.db_name");
print "\n MySql DB Password : " . $cfg->param("mysql.password");

print "\n\n Oracle DB Name     : " . $cfg->param("oracle.db_name");
print "\n Oracle DB Password : " . $cfg->param("oracle.password");
 
#Set/Update Values Config File
$cfg->param("mysql.db_name", "new_mysql_db_name");
$cfg->param("mysql.password", "new_mysql_password");

print "\n\n MySql DB Name     : " . $cfg->param("mysql.db_name");
print "\n MySql DB Password : " . $cfg->param("mysql.password");

$cfg->param("oracle.db_name", "new_orcl_db_name");
$cfg->param("oracle.password", "new_orcl_password");

print "\n\n Oracle DB Name     : " . $cfg->param("oracle.db_name");
print "\n Oracle DB Password : " . $cfg->param("oracle.password");

#Adding a new Variable to Config File
$cfg->param("mysql.new_var", "mysql_adding_variable");

print "\n\n MySql New Var     : " . $cfg->param("mysql.new_var");

$cfg->param("oracle.new_var", "oracle_adding_variable");

print "\n\n Oracle New Var     : " . $cfg->param("oracle.new_var");

print "\n\n Deleting Mysql New Var 'new_var' ... ";
$cfg->delete('mysql.new_var'); # deletes 'new_var' from [mysql] block

print "\n\n Deleting Oracle New Var 'new_var' ... ";
$cfg->delete('oracle.new_var'); # deletes 'new_var' from [oracle] block

#Config Vars
#Config::Simple also supports vars() method, which, depending on the context used, returns all the values either as hash or hashref
my %Config = $cfg->vars();
print "\n\n Config Hash Obj : " . Dumper(\%Config);

my $config_ref = $cfg->vars();
print "\n\n Config Hash Ref : " . Dumper($config_ref);
  

db_ini.cfg Output

 MySql DB Name     : test
 MySql DB Password : mysql_pass

 Oracle DB Name     : oracle_db
 Oracle DB Password : oracle_pass

 MySql DB Name     : new_mysql_db_name
 MySql DB Password : new_mysql_password

 Oracle DB Name     : new_orcl_db_name
 Oracle DB Password : new_orcl_password

 MySql New Var     : mysql_adding_variable

 Oracle New Var     : oracle_adding_variable

 Deleting Mysql New Var 'new_var' ... 
 
 Deleting Oracle New Var 'new_var' ... 
 
 Config Hash Obj : $VAR1 = {
          'mysql.PrintError' => '1',
          'mysql.db_name' => 'new_mysql_db_name',
          'oracle.password' => 'new_orcl_password',
          'oracle.host' => 'DBI:oracle:host',
          'mysql.host' => 'DBI:mysql:host',
          'mysql.password' => 'new_mysql_password',
          'oracle.PrintError' => '1',
          'oracle.login' => 'oracle_user',
          'mysql.RaiseError' => '1',
          'oracle.db_name' => 'new_orcl_db_name',
          'oracle.RaiseError' => '1',
          'mysql.login' => 'mysql_user'
        };


 Config Hash Ref : $VAR1 = {
          'mysql.PrintError' => '1',
          'oracle.password' => 'new_orcl_password',
          'mysql.db_name' => 'new_mysql_db_name',
          'oracle.host' => 'DBI:oracle:host',
          'mysql.host' => 'DBI:mysql:host',
          'oracle.PrintError' => '1',
          'mysql.password' => 'new_mysql_password',
          'oracle.login' => 'oracle_user',
          'mysql.RaiseError' => '1',
          'oracle.db_name' => 'new_orcl_db_name',
          'oracle.RaiseError' => '1',
          'mysql.login' => 'mysql_user'
        };
  

2) Reading/Updating config file in HTTP-LIKE style

When we just key and value pairs, this simple HTTP-Like style works.
Let's explain with the below mentioned example

db_http.cfg

host:'DBI:mysql:host'
login:user
password:secret
db_name:test
RaiseError:1
PrintError:1
  

Script

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;
use Config::Simple;

my $cfg = new Config::Simple('db_http.cfg');

#Get Values from Config File
print "\n DB Name     : " . $cfg->param("db_name");
print "\n DB Password : " . $cfg->param("password");
 
#Set/Update Values Config File
$cfg->param("db_name", "new_db_name");
$cfg->param("password", "new_password");

print "\n\n DB Name     : " . $cfg->param("db_name");
print "\n DB Password : " . $cfg->param("password");

#Adding a new Variable to Config File
$cfg->param("new_var", "adding_variable");

print "\n\n New Var     : " . $cfg->param("new_var");

print "\n\n Deleting New Var 'new_var' ... ";
$cfg->delete('new_var'); # deletes 'new_var'

#Config Vars
#Config::Simple also supports vars() method, which, depending on the context used, returns all the values either as hash or hashref
my %Config = $cfg->vars();
print "\n\n Config Hash Obj : " . Dumper(\%Config);

my $config_ref = $cfg->vars();
print "\n\n Config Hash Ref : " . Dumper($config_ref);

db_http.cfg Output

 DB Name     : test
 DB Password : secret

 DB Name     : new_db_name
 DB Password : new_password

 New Var     : adding_variable

 Deleting New Var 'new_var' ... 
 
 Config Hash Obj : $VAR1 = {
          'db_name' => 'new_db_name',
          'password' => 'new_password',
          'RaiseError' => '1',
          'PrintError' => '1',
          'login' => 'user',
          'host' => 'DBI:mysql:host'
        };

 Config Hash Ref : $VAR1 = {
          'db_name' => 'new_db_name',
          'password' => 'new_password',
          'RaiseError' => '1',
          'host' => 'DBI:mysql:host',
          'login' => 'user',
          'PrintError' => '1'
        };
  

3) Creating config file in INI style

Creating a config file is explained as mentioned below.

$cfg = new Config::Simple(syntax=>'ini');
$cfg->write("test.cfg");

Supported "syntax" keywords are "ini", "simple" or "http"

write()
Alternatively, you can pass a name to either write() or save() to indicate the name of the file to create instead of modifying existing configuration file

Since we are trying to create a new config file, we need to pass the file name.

Script

$cfg = new Config::Simple(syntax=>'ini');

$cfg->param("mysql.dsn", "DBI:mysql:db;host=localhost.com");
$cfg->param("mysql.user", "username");
$cfg->param("mysql.pass", 'secret');
$cfg->write("test.cfg");
  

test.cfg Output

[mysql]
dsn:mysql:db;host=localhost.com
user:username
pass:secret
  

Perl Remove Special Characters From File

While reading some kind of log files as mentioned below, we need to get rid of these special characters.

Because of these special characters, it makes the Developers job tough :
To parse the content of a file
To convert the special characters from the file

Let us explain how can we get rid of these special characters with a simple example

test.log

^[[1;31mTest 1^[[0m
^[[1;31mTest 2^[[0m
^[[1;31mTest 3^[[0m
^[[1;31mTest 4^[[0m
^[[1;31mTest 5^[[0m  

In the above test.log file, first of all, what is displayed as ^[ is not ^ and [
But it is the ASCII ESC character, produced by Esc or Ctrl[ (the ^ notation means the Ctrl key).

We can use the following regular expression :

s/\e\[[\d;]*[a-zA-Z]//g;

Note: 
\e represents escape character in the above regular expression (substituting instead of ^[ )
We can shorten from [a-zA-Z] to just [mK], based on the requirement
You can make use of the above regular expression while parsing the file as well (line by line)

In case if you want to backup file (test.log.bak) instead of changing in the original file (test.log) then use the following :
perl -pi.bak -e 's/\e\[[\d;]*[a-zA-Z]//g' test.log

The following will remove the special chars in test.log
perl -pi -e 's/\e\[[\d;]*[a-zA-Z]//g' test.log

Output after removing

Test 1
Test 2
Test 3
Test 4
Test 5
  

Unit Testing in Perl

We can write test scripts in Perl using Test::Simple and Test::More in easy way.

Test::More is an yet another framework for writing test scripts

Lets explain Test::More module with examples.

ok() vs is() 

1) Similar to ok(), is() and isnt() compare their two arguments with eq and ne respectively and use the result of that to determine if the test succeeded or failed. 

is($hsh{a}, 200, 'Test 200');
isnt($hsh{a}, 200, 'Test 200');

are similar to these:

ok($hsh{a} eq 200, 'Test 200');
ok($hsh{a} ne 200, 'Test 200');

2) They produce better diagnostics on failure. 
ok() cannot know what you are testing for (beyond the name)
but 
is() and isnt() know what the test was and why it failed. 

In case of test fail, ok() shows fail like below :
ok($hsh{a} == 200, 'Test 200');
# not ok 5 - Fail
#   Failed test 'Fail'
#   at main.pl line 21.

In case of test fail, is() or isnt() shows fail like below :
is($hsh{a}, 200, 'Test 200');
# not ok 5 - Fail
#   Failed test 'Fail'
#   at main.pl line 20.
#          got: '100'
#     expected: '200'

like()
like() allows to use regular exporessions in writing test cases
The following example, we are trying to check $str contains string 'freedom' or not

like($str, qr/freedom/i, 'String contains Freedom - ignore case');
ok($str =~ /freedom/i, 'String contains Freedom - ignore case');


cmp_ok()
cmp_ok() also useful in those cases where you are giving explicit conditions like >=, <=, ==, != etc.,

cmp_ok($scalar, "<=", 200, '10 is less than or equal to 200 - pass');
cmp_ok($scalar, "==", 10, '10 is equal to 10 - pass');

can_ok()

Syntax:
can_ok($module, @methods);
can_ok($object, @methods);

can_ok checks if a mthod exists in an object, here $obj is the object of "support" class
can_ok also checks if a method exists in an class, here "support" is the class
can_ok is good at taking mutiples methods to test at one shot

E.g.,
can_ok($obj, "add");
can_ok("support", "multiply");
can_ok("support", qw(test add multiply));

Lets explain the above methods with an example :
In the following example, we use support.pm for testing object, class and methods

support.pm

#!/usr/bin/perl

package support;

use strict;
use warnings;
use Data::Dumper;

sub new {
    my($class) = shift;
        
    my($self) = {};

    return(bless($self, $class));
}


sub multiply {
  my $a = shift;
  my $b = shift;
 
  return ($a*$b);
}  

sub add {
  my $a = shift;
  my $b = shift;
 
  return ($a+$b);
}  

sub test {
    print "\n test mehthod inside support package"; 
}  

1;  

main.pl

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;
use Test::More tests=>20;
use support;

my $obj = new support;

#You usually want to test if the module you're testing loads ok, rather than just vomiting if its load fails. It's recommended that you run use_ok() inside a BEGIN block so its functions are exported at compile-time and prototypes are properly honored.
BEGIN { use_ok("CGI"); }

#print "\n Obj : " . Dumper($obj);


#Testing Packages and Objects -----------------------------------------------
#Testing an object belongs to a class or not
ok (defined($obj) && ref $obj eq 'support', 'support obj new worked');

#can_ok checks if a mthod exists in an object, here $obj is the object of "support" class
#can_ok also checks if a method exists in an class, here "support" is the class
#can_ok is good at taking mutiples methods to test at one shot
can_ok($obj, "add");
can_ok("support", "multiply");
can_ok("support", qw(test add multiply));


#isa_ok checks if an object belongs to a class or not
isa_ok($obj, 'support');


#Testing Values -----------------------------------------------
my $scalar = 10;
my %hsh    = (a=>100, b=>200, c=>300);
my $str    = 'Freedom lies in being bold';


#is() vs ok()
#is() is recommended over ok()
#ok() function doesn't provide good diagnostic output.
#ok() cannot know what you are testing for (beyond the name), but is() and isnt() know what the test was and why it failed.

ok($scalar == 10, 'Pass');
is($scalar, 10, 'Pass');

is($hsh{a}, 100, 'Pass');
ok($hsh{a} == 100, 'Pass');

#is($hsh{a}, 200, 'Fail');
# not ok 5 - Fail
#   Failed test 'Fail'
#   at main.pl line 20.
#          got: '100'
#     expected: '200'

#ok($hsh{a} == 200, 'Fail');
# not ok 5 - Fail
#   Failed test 'Fail'
#   at main.pl line 21.

isnt($scalar, 20, 'Pass');

#like() allows to use regular exporessions in writing test cases
like($str, qr/freedom/i, 'String contains Freedom - ignore case');
ok($str =~ /freedom/i, 'String contains Freedom - ignore case');

#It's also useful in those cases where you are comparing numbers and applying conditions
cmp_ok($scalar, "<=", 200, '10 is less than or equal to 200 - pass');
cmp_ok($scalar, "==", 10, '10 is equal to 10 - pass');

Output:

1..20
ok 1 - use CGI;
ok 2 - support obj new worked
ok 3 - support->can('add')
ok 4 - support->can('multiply')
ok 5 - support->can(...)
ok 6 - The object isa support
ok 7 - Pass
ok 8 - Pass
ok 9 - Pass
ok 10 - Pass
ok 11 - Pass
ok 12 - String contains Freedom - ignore case
ok 13 - String contains Freedom - ignore case
ok 14 - 10 is less than or equal to 200 - pass
ok 15 - 10 is equal to 10 - pass
# Looks like you planned 20 tests but ran 15.
  

Perl Debug

Perl Debug is simple tool to easily debug perl scripts.

Few people like me find it difficult using Debug, but to be frank this is quite simple and straight forward. It makes the life of programmer easy to trace the bugs in a fast mode.

Without this, programmer needs to write unnecessary Print statements inside code and check where exactly things going wrong.

If you invoke Perl with the -d switch, your script runs under the Perl source debugger.

This works like an interactive Perl environment, prompting for debugger commands that let you examine source code, set breakpoints, dynamically pass the values of variables, etc.

This is so convenient that you often fire up the debugger all by itself just to test out Perl constructs interactively.

Debug Interactively:
Andrew E. Page, written an useful CPAN module Devel::Ptkdb, using this we can debug interactively.

This makes life much easier, no need to use <debug> prompt instead one can use the interactive mode to set the break points and pass the values to arguments dynamically.


How to Debug Perl Script Interactively using above module
perl -d:ptkdb myscript.pl  (for graphical representation of debugging)

Normal Debugger
perl debugger works using -d switch
Once you execute script using -d switch, it will change to debug prompt as mentioned below :
DB<1> ....
DB<2> ....

Some commonly used commands in Debug mode:

1) "s"  (Stepping through line by line execution) 
Keep pressing "s" will execute line by line

2) "l"  (List line) 
e.g.,
l 20                   #Lists line 20
l 20-25             #Lists 20 to 25 lines of script
l <subroutine>   #Lists a sub-routine

3) "b" (Setting break point on subroutine)
    "b" <subroutine>  
     #This will allow you to set a break point (set a break point on first line of subroutine)

4) "c" <subrutine>    
#This will take you directly to that break point (set one-time bkpt at subname and continue)

5) "q"                         
#It will quit  the debugger