Spark 1 vs Spark 2

 

Spark 1.x	Spark 2.x
Spark Context is the entry point	Spark Session is the entry point
We need to create separately sql context, hive context if we have only SparkContext.	Spark Session is enough
Spark 1.x uses compilers which uses of several function calls and CPU cycles, because of which so much unnecessary work spent on CPU cycles.	Spark 2.x uses performance enhanced Tungsten engine
1X	10X times faster than Spark 1.X
Spark Streaming (uses RDD batch concept)	Structured Streaming (uses DataFrames/DataSet APIs)
	Unified Dataset and DataFrame APIs (Dataset has more type safety, not available in Python). Now Dataframe is just an alias for Dataset of Row
	Many machine learning algorithms like Gaussian Mixture Model, MaxAbsScaler, Bisecting K-Means clustering feature transformer are added to DataFrame based API and many ML algorithms added to PySpark and SparkR also.
RDD based API is going into maintenance mode	DataFrame based has become the primary API now

 

Backup Apache log files using logrotate

/etc/logrotate.d/httpd

If you want to backup apache logs for all instances (you can copy this logrotate script/snippet in /etc/logrotate.d/httpd)

Prerequisite: setup s3cmd

/var/log/httpd/*log {
    daily
    dateext
    #dateext dateformat -%Y-%m-%d-%s
    missingok
    notifempty
    #size 3M
    sharedscripts
    delaycompress
    rotate 4
    create
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true

        BUCKET=logs-backup
        INSTANCE_ID=`curl --silent http://169.254.169.254/latest/meta-data/instance-id`
        /usr/local/bin/s3cmd -c /root/.s3cfg -m text/plain sync /var/log/httpd/access_log* s3://${BUCKET}/system_logs/httpd/${INSTANCE_ID}/
        /usr/local/bin/s3cmd -c /root/.s3cfg -m text/plain sync /var/log/httpd/error_log* s3://${BUCKET}/system_logs/httpd/${INSTANCE_ID}/

    endscript
}

How to Detect User Idle Time or Inactivity in Acess logs

How to Detect User Idle Time or Inactivity in Acess logs
Requirement:

• Read access log
• Find the top most idle time(s) between the requests

Script

import itertools
import datetime
import logging

fo = open("access_log_time", "r+")
print "Name of the file: ", fo.name

lst = fo.readlines()
print len(lst)

def diff_date(x, y):
diff=0
try:
x = x.strip()
y = y.strip()
d1 = datetime.datetime.strptime(x, '%d/%b/%Y:%H:%M:%S')
d2 = datetime.datetime.strptime(y, '%d/%b/%Y:%H:%M:%S')
diff = (d2 - d1).total_seconds()
print '-------'
print diff
print x
print y
except Exception, e:
logging.error(e)
return int(diff)

#zip Vs izip
#zip computes all the list at once, izip computes the elements only when requested.
#One important difference is that 'zip' returns an actual list, 'izip' returns an 'izip #object', which is not a list and does not support list-specific features

res= [diff_date(x,y) for x, y in itertools.izip (lst, lst[1:])]
print sorted(res, reverse=True)
#print res

View number of requests by time from Apache access log

View number of requests by time from Apache access log

• Overall requests in an hour
• grep "18/Apr/2018:11" /var/log/httpd/access_log | wc -l
• Overall requests in a minute
• grep "18/Apr/2018:11:05" /var/log/httpd/access_log | wc -l
• Overall requests in a minute
• grep "18/Apr/2018:11:05:10" /var/log/httpd/access_log | wc -l
• Overall requests by sec in an hour (group by sec)
• grep "18/Apr/2018:11" /var/log/httpd/access_log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'

How to run multiple sites on one Apache

Assuming we have two sites to configure in Apache
www.example1.com
www.example2.com

/etc/httpd/conf.d/example1_http.conf
<VirtualHost *>
        ServerAdmin webmaster@example1.com
        ServerName  www.example1.com
        ServerAlias example1.com

        # Indexes + Directory Root.
        DirectoryIndex index.html
        DocumentRoot /home/www/www.example1.com/htdocs/
.....
</VirtualHost>

/etc/httpd/conf.d/example2_http.conf
<VirtualHost *>
        ServerAdmin webmaster@example2.com
        ServerName  www.example2.com
        ServerAlias example2.com

        # Indexes + Directory Root.
        DirectoryIndex index.html
        DocumentRoot /home/www/www.example2.com/htdocs/
.....
</VirtualHost>

Configure Hosts File in Windows:

The Hosts file in Windows is located at the following location:
C:\Windows\System32\drivers\etc

Suppose your system IP is: 192.55.44.55

Add the following to the Hosts File:
192.55.44.55 example1.com
192.55.44.55 example2.com
192.55.44.55 www.example1.com
192.55.44.55 www.example2.com

Restart Apache

Now your Apache runs the two multiple sites

You can access example1.com and example2.com respectively

Access Default Site when Server Name/Alias mismatches

When you have multiple sites (multiple conf files) in conf.d, you can create a default conf file

If Apache finds difficulty in finding the site, it defaults to
The following defaults to www.example1.com in case of any conflicts

/etc/httpd/conf.d/aaa_http.conf
<VirtualHost *>
        ServerAdmin webmaster@example1.com
        ServerName  www.example1.com
        ServerAlias example1.com

        # Indexes + Directory Root.
        DirectoryIndex index.html
        DocumentRoot /home/www/www.example1.com/htdocs/
.....
</VirtualHost>

How to simulate the conflict state where Apache not able to find the correct site conf file

For two different sites, give the same ServerName
/etc/httpd/conf.d/test1_http.conf
          ServerName  www.test.com
/etc/httpd/conf.d/test2_http.conf
          ServerName  www.test.com

If you try to access www.test.com, Apache gets CONFUSED to pick which sites conf file (whether to choose test1_http.conf or test2_http.conf)

So Apache picks the top conf file (as per naming order) defined above aaa_http.conf
So you are re-directed to www.example1.com (as defined in aaa_http.conf)