sed examples

Ref: https://www.geeksforgeeks.org/sed-command-in-linux-unix-with-examples/

sed 's/unix/linux/' test.txt   # Replacing or substituting string (sed command replaces the first occurrence of the pattern in each line and it)

sed 's/unix/linux/2' test.txt  # Replacing the nth occurrence of a pattern in a line

sed 's/unix/linux/g' test.txt  # Replace all occurances in a line as well

sed 's/unix/linux/3g' test.txt # Replace nth till last (3rd, 4th, 5th etc.,)

echo "Welcome To The Geek Stuff" | sed 's/\(\b[A-Z]\)/\(\1\)/g'

sed '3 s/unix/linux/' test.txt # restrict the sed command to replace the string on a specific line number.

sed 's/unix/linux/p' test.txt  # The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.

sed -n 's/unix/linux/p' test.txt # print only replaced lines, n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.

sed '1,3 s/unix/linux/' test.txt # Replace range of lines

sed '5d' test1.txt  # delete 5th line

sed '$d' test1.txt  # delete last line

sed '3,6d' test1.txt # delete from range x to y

sed '12,$d' test1.txt # delete from nth line till last

sed '/abc/d' test1.txt # delete line which matches pattern

# Replace & update same file (-i means --in-place)

sed -i 's/ABC/ABCDEF/g' sed_test.txt

# Replace & update same file (-i means --in-place), -ibak takes original file as backup (sed_test.txtbak)

sed -ibak 's/ABC/ABCDEF/g' sed_test.txt

Find Unix Distribution Version

• cat /proc/version
• Linux version 4.14.219-164.354.amzn2.x86_64 (mockbuild@ip-10-0-1-103) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-12) (GCC)) #1 SMP Mon Feb 22 21:18:39 UTC 2021 
• rpm -E %{rhel}
• 7

How to evaluate time taken for urls using curl

• Configure ~/.curlrc
• Add file ~/.curlrc & add the below
• -w "dnslookup: %{time_namelookup} | connect: %{time_connect} | appconnect: %{time_appconnect} | pretransfer: %{time_pretransfer} | starttransfer: %{time_starttransfer} | total: %{time_total} | size: %{size_download}\n"
•  How to call urls using Curl?
• curl -so /dev/null "https://www.site.com/test/video.mp4"
• dnslookup: 0.013 | connect: 0.335 | appconnect: 1.090 | pretransfer: 1.090 | starttransfer: 1.666 | total: 24.770 | size: 128750670
• curl -so /dev/null "https://cdn.site.com/test/video.mp4"
• dnslookup: 0.013 | connect: 0.051 | appconnect: 0.311 | pretransfer: 0.311 | starttransfer: 1.287 | total: 11.556 | size: 128750670

We can observe 2nd url (CDN) takes almost half the time as the first once.

Unix Cut Comamnd

CUT command has lot of advantages in grep/cut the fields/chars from the files easily in no time

We can use CUT command in the following ways:
1) Print Characters by Position (chars)
2) Print Characters by Range (range a-b)
3) Print the fields using the delimiter

Now, we discuss the above mentioned briefly. Let's have fun with some simple examples.
In all the below examples, we use same.txt as input as mentioned below :

sample.txt
Andy,100,Ney york,USA
Arjun,200,New Delhi,India
Venkatesh,300,Chennai,India
Andy,400,Boston,USA
John,500,Chicago,USA


1) Using Cut Print Characters by Position

This command prints the fourth character in each line of the file

]# cut -c4 sample.txt
y
u
k
y
n

This command prints the fourth and sixth character in each line.

]# cut -c4,6 sample.txt
y1
u,
kt
y4
n5

2) Using Cut Print Characters by Range

This command prints the characters from fourth position to the eighth position in each line

]# cut -c4-8 sample.txt
y,100
un,20
kates
y,400
n,500

This command prints the first four characters in a line

]# cut -c-4 sample.txt
Andy
Arju
Venk
Andy
John

This command prints the characters from fourth position to the end

]# cut -c4- sample.txt
y,100,Ney york,USA
un,200,New Delhi,India
katesh,300,Chennai,India
y,400,Boston,USA
n,500,Chicago,USA

This command omits the start and end positions, then the cut command prints the entire line.

]# cut -c- sample.txt
Andy,100,Ney york,USA
Arjun,200,New Delhi,India
Venkatesh,300,Chennai,India
Andy,400,Boston,USA
John,500,Chicago,USA

3) Print the fields using the delimiter
-d  option in cut command can be used to specify the delimiter 
-f  option is used to specify the field position

This command prints the first field of comma separated line

]# cut -d',' -f1 sample.txt
Andy
Arjun
Venkatesh
Andy
John

This command prints the first and second field in each comma separated line

]# cut -d',' -f1,2 sample.txt
Andy,100
Arjun,200
Venkatesh,300
Andy,400
John,500

This command prints the first, second and third field (range 1-3) in each comma separated line

]# cut -d',' -f1-3 sample.txt
Andy,100,Ney york
Arjun,200,New Delhi
Venkatesh,300,Chennai
Andy,400,Boston
John,500,Chicago

This command prints the first two fields in each comma separated line

]# cut -d',' -f-2 sample.txt
Andy,100
Arjun,200
Venkatesh,300
Andy,400
John,500

This command prints from first field to last field in each comma separated line

]# cut -d',' -f1- sample.txt
Andy,100,Ney york,USA
Arjun,200,New Delhi,India
Venkatesh,300,Chennai,India
Andy,400,Boston,USA
John,500,Chicago,USA 

Please refer to other topics on Unix like :
Unix Delete Duplicated Lines in a File
Unix Unique Lines in a File
Unix Grep Examples
Search a Directory in Unix
Unix For Loop
pushd & popd in Unix
Find Size of Directory
Word Count

Please refer to other topics on AWK like :
Awk Examples
Print First Two Columns of File
Print Last Two Columns of File

Please refer to other topics on Dict like :
Dict in Python
Dict keys and values in Python

Please refer to other topics on List like :
List in Python
Append to list in Python
Delete the last name from the list in Python
Remove an element from List in Python
Check an element exists in an list in Python
Python Filter Vs Map Vs List Comprehension

Please refer to other topics on File Concepts like :
Print File Content in Python
Print File in Reverse Order in Python

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Unix Sort, Uniq, Combine Files

We will discuss about 
1) Combining contents of two files in Unix 
2) Sort & Get the Unique Lines (delete duplicate lines)
3) How to get only the duplicate lines
4) How to get only the unique lines

Suppose we have two files a.txt and b.txt as mentioned below :

a.txt

10
20
30
40
50

b.txt

10
20
30
40
50
60
70
80

1) Combining contents of two files in Unix 
Combining contents of both a.txt & b.txt into c.txt
]# cat a.txt b.txt > c.txt 

10
20
30
40
50
10
20
30
40
50
60
70
80

2) Sort and Get the Unique Lines (delete duplicate lines)
]# cat c.txt | sort | uniq

10
20
30
40
50
60
70
80

3) How to get only the duplicate lines
]# cat c.txt | sort | uniq -u

60
70
80

4) How to get only the unique lines

10
20
30
40
50

5) How to get sort by ignore case
]# cat requirements.txt | sort --ignore-case > requirements_bak.txt 

Please refer to other topics on Unix like :
Unix Grep Examples
Search a Directory in Unix
Unix For Loop
pushd & popd in Unix
Find Size of Directory
Word Count

Please refer to other topics on AWK like :
Awk Examples
Print First Two Columns of File
Print Last Two Columns of File

Please refer to other topics on Dict like :
Dict in Python
Dict keys and values in Python

Please refer to other topics on List like :
List in Python
Append to list in Python
Delete the last name from the list in Python
Remove an element from List in Python
Check an element exists in an list in Python
Python Filter Vs Map Vs List Comprehension

Please refer to other topics on File Concepts like :
Print File Content in Python
Print File in Reverse Order in Python

Please refer to Regular Expressions Concepts :
Brief on Regular Expressions
Greedy Operators in Regular Expressions in Perl
Modifiers in Regular Expressions in Perl
Capturing concept in Regular Expressions in Perl
Capture Pre Match ,Post Match, Exact match in Regular Expressions in Perl
Non Capturing Paranthesis in Regular Expressions in Perl
Substitute nth occurance in Regular Expressions in Perl
All Topics in Regular Expressions in Perl

You might also wish to read other topics like :
Python Class and Object Example
Inheritance in Python
Packages in Python
Exceptions in Python
How to remove duplicate lines from a file in Perl
How to remove duplicate lines from a file in Pyhton

Word Count - Unix

>> wc test.txt

15  43 282 test.txt

ie.,

15 lines
43 words
282 chars


>>wc -l test.txt    #Lines
15 test.txt


>> wc -c test.txt    #Words
282 test.txt


>> wc -w test.txt    #Characters
43 test.txt

How to print first two columns of a file - AWK

1) It will print first line alone

>>awk 'NR==1' test.txt #1st line from the top
>>awk 'NR==7' test.txt #7th line from the top

2) How to print first line/lines of the file

>>head -8 test.txt

How to print last two columns of a file - AWK

1) How to print last two columns of a file?

NF => prints the last line

>>cat test.txt
prabhath   kota   Jan
lakshmi     muvvala   Feb
PRABHATH    KOTA   Mar
LAKSHMI       MUVVALA   Apr

>>awk '{print $(NF-1),"\t",$NF}' test.txt #RIGHT

output:
kota              Jan
muvvala         Feb
KOTA           Mar
MUVVALA  Apr

>>awk '{print $NF-1,"\t",$NF}' test.txt #WRONG


2) How to print last two columns of a command?

>> ls
-rw-r----- 1 kotaprax users 203 Nov 22 12:51 a.txt
-rw-r----- 1 kotaprax users 282 Nov 23 10:18 b.txt
-rw-r----- 1 kotaprax users  17 Nov 16 15:25 c.txt

>>ls -l | awk '{print $(NF-1),"\t",$NF}'

output:
12:51 a.txt
10:18 b.txt
15:25 c.txt


3) How to print the last line/lines of the file

>>tail -f test.txt
>>tail -4 test.txt #Prints Last 4 lines
>>tail -1 test.txt #Prints Last line of the file