Overview
Import NGINX/WordPress/EasyEngine(ee) Logs on Logstash
NOTE! We assume that you already setup/configure ELK Stack.
- To Import NGINX Logs on Logstash, We have to create configuration file.
# Create NGINX/EasyEngine Patterns
# http://grokdebug.herokuapp.com/
$ mkdir /etc/logstash/patterns
$ cat /etc/logstash/patterns/nginx
NGINX_ACCESS %{IPORHOST:visitor_ip} (?:-|(%{WORD}.%{WORD})) %{WORD:nginx_cache_status} \[%{HTTPDATE:timestamp}\] %{HOST:nginx_host} "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} %{QS:ignore} %{QS:referrer}
NGINX_ERROR %{DATE} %{TIME} %{GREEDYDATA:error} limiting requests, excess: %{GREEDYDATA:limit} client: %{IPORHOST:visitor_ip}, server: %{HOST:nginx_host}, request: "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}", %{GREEDYDATA:msg}
# Create NGINX Access Log configuration file
$ vim /etc/logstash/conf.d/nginx.conf
input {
file {
type => "nginx"
start_position => "beginning"
path => [ "/var/log/nginx/*.log" ]
}
}
filter {
if [type] == "nginx" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{NGINX_ACCESS}" }
remove_tag => ["_grokparsefailure"]
add_tag => ["nginx_access"]
}
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{NGINX_ERROR}" }
remove_tag => ["_grokparsefailure"]
add_tag => ["nginx_error"]
}
geoip {
source => "visitor_ip"
}
}
}
Fix NGINX Logs Permission
- Let’s make NGINX logs are readable by Logstash
# Temp Fix
$ chmod 644 /var/log/*.log
# Permeant Fix
$ cat /etc/logrotate.d/nginx
/var/log/nginx/*.log {
size 10M
missingok
rotate 52
compress
delaycompress
notifempty
create 0644 www-data adm
sharedscripts
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi \
endscript
postrotate
invoke-rc.d nginx rotate >/dev/null 2>&1
endscript
}
Restart Logstash Service
$ sudo service logstash restart
Configure Kibana
- Open http://192.168.0.1:5601/
-
Click on Settings > Objects > Import
- Import Dashboard/Visualizations
Let’s Monitor NGINX/WordPress/EasyEngine Logs on Kibana
- Open http://192.168.0.1:5601/#/dashboard/NGINX?_g=()