Logstash configuration examples
You'll find some Logstash configuration example below. To use some of these examples you need to add the Logstash grok expressions.
Contents
Configuration file(s)
Logstash can have many configuration files.
It is recommended to have 1 file per log indice.
Depending on your taste you can choose between the following setup:
- 1 indice per log file ==> 1 Logstash configuration file per log file
- 1 indice for all ==> only 1 Logstash configuration, then you rely on tags
Anyway, configuration file(s) must be in /etc/logstash/conf.d/*.conf
Basic config file
A configuration file must always have 3 sections: input, filter, output.
example:
vim /etc/logstash/conf.d/logstash.conf
The following example will process a bunch of log4j files.
## List of complete inputs | filters | output available on the official website:
## http://logstash.net/docs/latest/index
## Configuration syntax: http://logstash.net/docs/latest/configuration
###### Data sources to process #####
input {
file {
path => [ "/home/qa1/catalina.base/logs/vehco/*.log" ]
type => "vehco-qa1"
}
}
filter {
# REMINDERS:
# >> you can check on Kibana the field name to use for each filter.
# >> you can find the list of GROK pattern over here: https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns
# All lines that does not start with %{TIMESTAMP} or ' ' + %{TIMESTAMP} belong to the previous event
multiline {
pattern => "(([\s]+)20[0-9]{2}-)|20[0-9]{2}-"
negate => true
what => "previous"
}
# QA1
if [type] == "vehco-qa1" {
grok {
patterns_dir => ["/etc/logstash/grok"]
match => [ "message", "%{LOG4J}" ]
add_tag => "vehco-log-qa1"
}
# Something wrong occurred !!!
if "_grokparsefailure" in [tags] {
grok {
patterns_dir => "/etc/logstash/grok"
match=>[ "message","(?<content>(.|\r|\n)*)" ]
add_tag => "vehco-log-qa1-grok_error"
}
}
}
}
output {
elasticsearch {
cluster => "VEHCO"
protocol => "http"
# port => ""
host => "172.16.50.223"
node_name => "vehco-qa"
index => "vehco-qa-%{+YYYY.MM.dd}"
}
}
[!] Note 1:
Grok will normally break on rule match == it will stop processing after the 1st pattern that matches and return success.
[!] Note 2:
You can set generic blob expression as INPUT filters.
Multi-lines
Some logs can be on N lines. They are called multi-lines. Multi-line filter must always be before any GROK filter!
Handle spaces
A new event must NOT start with a space.
# All lines starting with a space belong to the previous event
multiline {
pattern => "^\s"
negate => false
what => "previous"
}
Java exceptions
This will make all exceptions belong to the previous event.
# All exceptions belong to the previous event
multiline {
pattern => "(([^\s]+)Exception.+)|(at:.+)"
negate => false
what => "previous"
}
LOG4J trick
If you only expect Log4j logs then you know that each line that does NOT start with a %{TIMESTAMP} is NOT a new event.
# All lines that does not start with %{TIMESTAMP} or ' ' + %{TIMESTAMP} belong to the previous event
multiline {
pattern => "(([\s]+)20[0-9]{2}-)|20[0-9]{2}-"
negate => true
what => "previous"
}
Grok failure
If your Grok expression is wrong the line will be tagged as '_grokparsefailure' .
Since you know how to detect error, you can attempt to apply an alternate filter on the log.
filter {
# myApplication
if [type] == "myApp" {
grok {
...
}
# Something wrong occurred !!! :O Do something else instead!
if "_grokparsefailure" in [tags] {
grok {
patterns_dir => "/etc/logstash/grok"
match=>[
"message","(?<content>(.|\r|\n)*)"
]
}
}
}
}
Common Logstash configurations
Apache2
Requirements:
- Make sure your logs are in "/var/log/apache2" or adjust the paths
- Make sure your using the COMBINED logs (default in Apache 2.4+)
Logstash configuration extract:
input {
file {
path => [ "/var/log/apache2/access.log", "/var/log/apache2/ssl_access.log", "/var/log/apache2/other_vhosts_access.log" ]
type => "apache-access"
}
file {
path => "/var/log/apache2/error.log"
type => "apache-error"
}
}
filter {
# ------------------------ Parse services logs into fields ---------------------------
# APACHE 2
if [type] == "apache-access" {
# To process log data (message's content) using some regex or precompiled GROK pattern
grok {
match => [ "message", "%{COMBINEDAPACHELOG}"]
}
# To extract log's time according to a date pattern
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
}
# Extraction browser information, if available.
if [agent] != "" {
useragent {
source => "agent"
}
}
if [clientip] != "" {
geoip {
source => "clientip"
target => "apache_clientip"
add_tag => [ "geoip" ]
}
}
}
if [type] == "apache-error" {
grok {
match => [ "message", "%{APACHEERRORLOG}"]
# Directory where to find the custom patterns
patterns_dir => ["/etc/logstash/grok"]
}
if [clientip] != "" {
geoip {
source => "clientip"
target => "apache_clientip"
add_tag => [ "geoip" ]
}
}
}
}
output {
...
}
IpTables
Requirements:
- Make sure you are logging dropped packets into a dedicated file. See Firewall log dropped
Logstash configuration extract:
input {
file {
path => "/var/log/iptables.log"
type => "iptables"
}
}
filter {
# IPTABLES
if [type] == "iptables" {
grok {
match => [
"message", "%{IPTABLES_IP}",
"message", "%{IPTABLES_ICMP}",
"message", "%{IPTABLES_GENERIC}"
]
patterns_dir => ["/etc/logstash/grok"]
}
# Something wrong occurred !!! :O
if "_grokparsefailure" in [tags] {
grok {
patterns_dir => "/etc/logstash/grok"
match=>["message", "%{IPTABLES_ERROR}" ]
add_tag => "iptables-grok_error"
}
}
# Default 'geoip' == src_ip. That means it's easy to display the DROPPED INPUT :)
if [src_ip] != "" {
geoip {
source => "src_ip"
add_tag => [ "geoip" ]
target => "src_geoip"
}
}
if [dst_ip] != "" {
geoip {
source => "dst_ip"
add_tag => [ "geoip" ]
target => "dst_geoip"
}
}
}
}
output {
...
}
Fail2ban
Logstash configuration extract:
input {
file {
path => "/var/log/fail2ban.log"
type => "fail2ban"
}
}
filter {
# Fail2ban
if [type] == "fail2ban" {
grok {
match => ["message", "%{FAIL2BAN}"]
patterns_dir => ["/etc/logstash/grok"]
}
if [ban_ip] != "" {
geoip {
source => "ban_ip"
add_tag => [ "geoip" ]
target => "ban_geoip"
}
}
}
}
output {
...
}
Syslog
Logstash configuration extract:
input {
file {
path => [ "/var/log/syslog", "/var/log/auth.log", "/var/log/mail.info" ]
type => "syslog"
}
}
filter {
# SYSLOG
if [type] == "syslog" {
grok {
match => ["message", "%{SYSLOGBASE}"]
}
}
}
output {
...
}
Tomcat
... To be done ...
Log4J
input {
file {
path => [ "/home/beta3/catalina.base/logs/vehco/*.log" ]
type => "myApp"
}
}
filter {
# All lines that does not start with %{TIMESTAMP} or ' ' + %{TIMESTAMP} belong to the previous event
multiline {
pattern => "(([\s]+)20[0-9]{2}-)|20[0-9]{2}-"
negate => true
what => "previous"
}
# myApplication
if [type] == "myApp" {
grok {
patterns_dir => ["/etc/logstash/grok"]
match => [
"message", "%{LOG4J}"
]
add_tag => "myApp-log"
}
# Something wrong occurred !!! :O
if "_grokparsefailure" in [tags] {
grok {
patterns_dir => "/etc/logstash/grok"
match=>[
"message","(?<content>(.|\r|\n)*)"
]
}
}
}
}
output {
...
}
VEHCO specific patterns
Now that you have some specific GROK patterns, you need to update your Logstash configuration.
input {
file {
path => [ "/var/log/vehco/*.log" ]
type => "vehco-rtd"
}
}
filter {
# VEHCO-RTD
if [type] == "vehco-rtd" {
grok {
patterns_dir => ["/etc/logstash/grok"]
match => [
"message", "%{RTD_TERMINAL}",
"message", "%{RTD_AUTH_START}",
"message", "%{RTD_AUTH_DONE}"
]
}
}
}
output {
...
}
