How to Detect User Idle Time or Inactivity in Acess logs

How to Detect User Idle Time or Inactivity in Acess logs
Requirement:

• Read access log
• Find the top most idle time(s) between the requests

Script

import itertools
import datetime
import logging

fo = open("access_log_time", "r+")
print "Name of the file: ", fo.name

lst = fo.readlines()
print len(lst)

def diff_date(x, y):
diff=0
try:
x = x.strip()
y = y.strip()
d1 = datetime.datetime.strptime(x, '%d/%b/%Y:%H:%M:%S')
d2 = datetime.datetime.strptime(y, '%d/%b/%Y:%H:%M:%S')
diff = (d2 - d1).total_seconds()
print '-------'
print diff
print x
print y
except Exception, e:
logging.error(e)
return int(diff)

#zip Vs izip
#zip computes all the list at once, izip computes the elements only when requested.
#One important difference is that 'zip' returns an actual list, 'izip' returns an 'izip #object', which is not a list and does not support list-specific features

res= [diff_date(x,y) for x, y in itertools.izip (lst, lst[1:])]
print sorted(res, reverse=True)
#print res

View number of requests by time from Apache access log

View number of requests by time from Apache access log

• Overall requests in an hour
• grep "18/Apr/2018:11" /var/log/httpd/access_log | wc -l
• Overall requests in a minute
• grep "18/Apr/2018:11:05" /var/log/httpd/access_log | wc -l
• Overall requests in a minute
• grep "18/Apr/2018:11:05:10" /var/log/httpd/access_log | wc -l
• Overall requests by sec in an hour (group by sec)
• grep "18/Apr/2018:11" /var/log/httpd/access_log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'

Find Sum of RSS memory in ps command in linux/unix

Find Sum of RSS memory in ps command in linux/unix

• In KB
• ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum}'
• In MB
• ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum/1024}'
• In GB
• ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum/1024/1024}'

How to find free memory available

• cat /proc/meminfo
• using free command
• In KB
• free
• In MB
• free -m
• In GB
• free -g 

I observed

• Sum of RSS memory in ps less than memory actually used
• Total used memory a lot higher than sum of RSS
• Reason
• The Linux kernel will use available memory for disk caching

AWS ELB

Classic ELB
    Supported Protocols
        HTTP, HTTPS (Secure HTTP), SSL (Secure TCP) and TCP protocols
    TCP Ports
        [EC2-VPC] 1-65535
        [EC2-Classic] 25, 80, 443, 465, 587, 1024-65535
    IPv6 support
        Each Classic Load Balancer has an associated IPv4, IPv6, and dualstack (both IPv4 and IPv6) DNS name.
        IPv6 is not supported in VPC. You can use an Application Load Balancer for native IPv6 support in VPC.
    Cross-Zone Load Balancing
        This option distributes traffic evenly across all your back-end instances in all Availability Zones.
        This reduces to maintain equivalent no of instances in each enabled AZ (But it is recommended to maintain to same no of instances in each AZ for higher fault tolerance)
        This option is enabled by default in AWS console
        This option is disabled by default in AWS API/CLI
    Can I privately access Elastic Load Balancing APIs from my Amazon Virtual Private Cloud (VPC) without using public IPs?
        Yes, you can privately access Elastic Load Balancing APIs from your Amazon Virtual Private Cloud (VPC) by creating VPC Endpoints
            

Application ELB
    Supported Protocols
        HTTP, HTTPS (Secure HTTP)
    TCP ports
        1-65535
    Can I convert my Classic Load Balancer to an Application Load Balancer (and vice versa)? - No
    Can I migrate to Application Load Balancer from Classic Load Balancer? - Yes
    Can I use an Application Load Balancer as a Layer-4 load balancer?
        No. If you need Layer-4 features, you should use Network Load Balancer.
    Is IPv6 supported with an Application Load Balancer? - Yes
    Can I associate multiple certificates for the same domain to a secure listener?
        Yes, you can associate multiple certificates for the same domain to a secure listener. For example, you can associate
        (a) ECDSA and RSA certificates
        (b) Certificates with different key sizes (e.g. 2K and 4K) for SSL/TLS certificates
        (c) Single-Domain, Multi-Domain (SAN) and Wildcard certificates



Network ELB
    Can I create a TCP (Layer 4) listener for my Network Load Balancer?
        Yes. Network Load Balancers support only TCP (Layer 4) listeners.
    Network Load Balancer Vs TCP listener on a Classic Load Balancer?
        Network Load Balancer preserves the source IP of the client which in the Classic Load Balancer is not preserved.
        Customers can use proxy protocol with Classic Load Balancer to get the source IP.
        Network Load Balancer automatically provides a static IP per Availability Zone to the load balancer and also enables assigning an Elastic IP to the load balancer per Availability Zone. This is not supported with Classic Load Balancer.
        Classic Load Balancer provides SSL termination that is not available with Network Load Balancer.

AWS EBS

EBS Encryption support
EBS Encryption is supported in all EBS volume types
But not all EC2 instances support encryption

EBS Volume/Snapshot - Encryption keys
KMS  - AWS Key management service
CMKs - Customer master keys
When you encrypt first EBS volume, AWS KMS creates default CMS key
After that, each newly encrypted volume is encrypted with a unique/separate AES256 bits encryption key

Sharing EBS Snapshots
Only account owner can create volume from snapshots
Encrypted snapshots cannot be shared, only un-encrypted are meant to be shared
But an Encrypted snapshot can be shared to a selected AWS account id, by making them private (with cross-account permissions)
Account A - with Key 1, has shared encrypted snapshot with Account B, Account B owner needs to create a copy of snapshot with his own key(Key 2) & create volumes 
AWS will not allow you to share snapshots encrypted using default CMK key ****
Snapshots are stored in S3 (unknown location to us)
S3 SSE protects snapshot data in transit

Copy Snapshots
Copy snapshot to another to encrypt or to another region
user defined tags are NOT copied from original
5 copy requests per account in parallel

Instance Backed AMI vs EBS Backed AMI
When you create EC2 AMI, AWS automatically EBS volume(s) & Intance root volume snapshots are created
When snapshots attached to AMI, you cannot delete snapshots, you need to de-register AMI & delete snapshots
1) Instance Backed AMI
  You need to register AMI (in Amazon market place)
You need to specify S3 bucket for storing snapshots
2) EBS Backed AMI
Registration of AMI happens automatically
You don't need to specify S3 bucket
When creating EMI, stop instance to ensure data consistancy and integrity
EBS Snapshots are point in time
How to ensure EBS data consistancy (stop EC2 instance is best)
1) Pause I/O operations on EBS from EC2
2) Unmount EBS volumes, create snapshot & mount again

RAID in EBS (Redundant array of independent disks)
Increase I/O performance/throughput of EC2
using EBS optimized EC2 instances
use RAID array of EBS volumes
RAID array is collection of multiple EBS volumes
Make sure EC2 max bandwidth >= Total I/O of EBS volume (or RAID)
RAID is not meant for root/boot volumes of EC2
RAID 0 - stripping
Fastest of all RAID types
Distributing data to be written over array of disks in parallel (without redundancy) - faster
If data fails in one volume, whole array of EBS volumes gets corrupted
RAID 1 -  Mirroring
Cares about redundancy but not faster
Not able achieve I/O performance/throughput
RAID 10 - It has benefits of both RAID0 & RAID1


EBS Practice scenarios

• non-encrypt to encrypt 
• copy ebs to another AZ
• make ebs volume public/private
• Encryption types
• customer specific encryption
• play in CMK & other encrypt patterns - customer managed keys
• copy snapshot to another region
• copy snapshot to another AWS account 
• attach to EC2 & mount
• increase/decrease EBS volume size
• Root volume encrypt (work around)
• Do I need to turn off EC2 while taking backup?

How to encrypt an existing ebs volume of an EC2 instance

How to encrypt an existing ebs volume of an EC2 instance ?
 

• 1)
• Take snapshot of existing ebs volume
• copy snapshot of above to new encrypted snapshot (using encryption)
• create volume from the encrypted snapshot (in the same us-east-1a/us-east-1b/us-east-1c availability zone as of EC2)
• stop ec2 instance
• detach existing unencrypted ebs volume
• attach new encrypted ebs volume to the ec2 instance
• start the instance again

• 2)
• Assume you have an non-encrypted EBS volume attached to EC2 instance
• Create an EBS volume with encrypt option
• Attach encrypted EBS volume to EC2 (in addition to the existing non-encrypted EBS volume)
• Now EC2, 2 EBS volumes are under a single AZ say us-east-1a
• 1st EBS volume mounted to /opt/ebs1 -> non-encrypted EBS volume
• 2nd EBS volume mounted to /opt/ebs2 -> Encrypted EBS volume
• Now copy content from non-encrypted EBS volume to Encrypted EBS volume
• Once done, detach non-encrypted EBS volume from EC2 instance

Note:

• EBS volumes are limited to a specific availability zone
  Snapshots are limited to a specific region
  EC2 and EBS volumes attached to the EC2 instance must be in the same AZ
  However you can copy snapshots across regions

How to make an Amazon EBS Volume Available for Use - mount

How to make an Amazon EBS Volume Available for Use
Please find the steps to mount EBS volume to Ec2 instance

• Attach to EC2 intance
• Login to EC2 instance
• lsblk - gives list of volumes
• NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
• xvda 202:0 0 8G 0 disk
• xvda1 202:1 0 8G 0 part /
• xvdf 202:80 0 10G 0 disk
• sudo file -s /dev/xvdf #for new volumes (it shows as data)
• /dev/xvdf: data
• sudo file -s /dev/xvda1 #for existing
• /dev/xvda1: Linux rev 1.0 ext4 filesystem data, UUID=XXXXX-XXXX-XXXXX-XXXX-XXXXXX (needs journal recovery) (extents) (large files) (huge files)
• cd ~
• mkdir ebsvolume
• sudo mkfs -t ext4 /dev/xvdf
• sudo mount /dev/xvdf ebsvolume
• df -h
• lsblk
• NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
• xvda 202:0 0 8G 0 disk
• └─xvda1 202:1 0 8G 0 part /
• xvdf 202:80 0 10G 0 disk /root/ebsvolume

Note:

EC2 instance, EBS volumes must be in a same AZ

python list vs tuple

t1 = [1,2,3,4]
t2 = t1
t2.append(5)
print 'T1: ' + str(t1)
print 'T2: ' + str(t2)

print '------------------'

t1 = (1,2,3,4)
t2 = t1
t2 = t1 + (5,)
print 'T1: ' + str(t1)
print 'T2: ' + str(t2)


Output:
T1: [1, 2, 3, 4, 5]
T2: [1, 2, 3, 4, 5]
------------------
T1: (1, 2, 3, 4)
T2: (1, 2, 3, 4, 5)


Reason:
Tuples are immutable and not supposed to be changed
Lists are mutable and are supposed to be changed

get last billing date of previous month

relm.py

import datetime

from dateutil import relativedelta

import sys

try:

   input_date = sys.argv[1]

   input_date = datetime.datetime.strptime(input_date, "%Y-%m-%d").date()

   #This gives always last date of the month - relativedelta.relativedelta(day=31)

   prev_month = input_date - relativedelta.relativedelta(months=1) - relativedelta.relativedelta(day=31)

   print prev_month

except Exception, e:

   print e

Output:

python relm.py 2018-02-27 #2018-01-31

python relm.py 2018-12-28 #2018-11-30

python relm.py 2018-01-28 #2017-12-31

python relm.py 2018-02-31  #day is out of range for month