A photo of Mitesh Shah

Mitesh Shah

Linux Expert | Automation Enthusiast | Security Consultant

Email Skype Github Twitter Resume Hire Me Keybase LinkedIn Stackoverflow


Overview

Import NGINX/WordPress/EasyEngine(ee) Logs on Logstash

NOTE! We assume that you already setup/configure ELK Stack.

  • To Import NGINX Logs on Logstash, We have to create configuration file.
# Create NGINX/EasyEngine Patterns
# http://grokdebug.herokuapp.com/
$ mkdir /etc/logstash/patterns
$ cat /etc/logstash/patterns/nginx
NGINX_ACCESS %{IPORHOST:visitor_ip} (?:-|(%{WORD}.%{WORD})) %{WORD:nginx_cache_status} \[%{HTTPDATE:timestamp}\] %{HOST:nginx_host} "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} %{QS:ignore} %{QS:referrer}
NGINX_ERROR %{DATE} %{TIME} %{GREEDYDATA:error} limiting requests, excess: %{GREEDYDATA:limit} client: %{IPORHOST:visitor_ip}, server: %{HOST:nginx_host}, request: "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}", %{GREEDYDATA:msg}
# Create NGINX Access Log configuration file
$ vim /etc/logstash/conf.d/nginx.conf
input {
  file {
    type => "nginx"
    start_position => "beginning"
    path => [ "/var/log/nginx/*.log" ]
  }
}
filter {
  if [type] == "nginx" {
    grok {
	patterns_dir => "/etc/logstash/patterns"
	match => { "message" => "%{NGINX_ACCESS}" }
	remove_tag => ["_grokparsefailure"]
	add_tag => ["nginx_access"]
    }
    grok {
	patterns_dir => "/etc/logstash/patterns"
	match => { "message" => "%{NGINX_ERROR}" }
	remove_tag => ["_grokparsefailure"]
	add_tag => ["nginx_error"]
    }
    geoip {
      source => "visitor_ip"
    }
  }
}

Fix NGINX Logs Permission

  • Let’s make NGINX logs are readable by Logstash
# Temp Fix
$ chmod 644 /var/log/*.log

# Permeant Fix
$ cat /etc/logrotate.d/nginx
/var/log/nginx/*.log {
	size 10M
	missingok
	rotate 52
	compress
	delaycompress
	notifempty
	create 0644 www-data adm
	sharedscripts
	prerotate
		if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
			run-parts /etc/logrotate.d/httpd-prerotate; \
		fi \
	endscript
	postrotate
		invoke-rc.d nginx rotate >/dev/null 2>&1
	endscript
}

Restart Logstash Service

$ sudo service logstash restart

Configure Kibana

  • Open http://192.168.0.1:5601/
  • Click on Settings > Objects > Import

  • Import Dashboard/Visualizations

Let’s Monitor NGINX/WordPress/EasyEngine Logs on Kibana

  • Open http://192.168.0.1:5601/#/dashboard/NGINX?_g=()

NGINX WordPress EasyEngine Logs on ELK Stack





Post Navigation