Logstash configuration examples

You'll find some Logstash configuration example below. To use these example you need to add the Logstash grok expressions I talked about earlier.

Configuration file(s)

Logstash can have many configuration files.

It is recommended to have 1 file per log indice.

Depending on your taste you can choose between the following setup:

• 1 indice per log file ==> 1 Logstash configuration file per log file
• 1 indice for all ==> only 1 Logstash configuration, then you rely on tags

Anyway, configuration file(s) must be in /etc/logstash/conf.d/*.conf

Basic config file

A configuration file must always have 3 sections: input, filter, output.

example:

vim /etc/logstash/conf.d/logstash.conf

The following example will process a bunch of log4j files.

## List of complete inputs | filters | output available on the official website: 
## http://logstash.net/docs/latest/index
## Configuration syntax: http://logstash.net/docs/latest/configuration
 
###### Data sources to process #####
input {
	file {
	    path => [ "/home/qa1/catalina.base/logs/vehco/*.log" ]
	    type => "vehco-qa1"
	}
}

filter {
	# REMINDERS:
	#  >> you can check on Kibana the field name to use for each filter.
	#  >> you can find the list of GROK pattern over here: https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns

	# All lines that does not start with %{TIMESTAMP} or ' ' + %{TIMESTAMP} belong to the previous event
	multiline {
		pattern => "(([\s]+)20[0-9]{2}-)|20[0-9]{2}-"
		negate => true
		what => "previous"
	}	

	# QA1
	if [type] == "vehco-qa1" {
		grok {
			patterns_dir => ["/etc/logstash/grok"]
			match => [ "message", "%{LOG4J}" ]
			add_tag => "vehco-log-qa1"
		}
		# Something wrong occurred !!! 
		if "_grokparsefailure" in [tags] {
			grok {
				 patterns_dir => "/etc/logstash/grok"
				 match=>[ "message","(?<content>(.|\r|\n)*)" ]
				 add_tag => "vehco-log-qa1-grok_error"
	                     }
		}
	}
}
 
output {
	elasticsearch {
		cluster => "VEHCO"
		protocol => "http"
		# port => ""
		host => "172.16.50.223"
		node_name => "vehco-qa"
		index => "vehco-qa-%{+YYYY.MM.dd}"
	}
}

[!] Note 1:

Grok will normally break on rule match == it will stop processing after the 1st pattern that matches and return success.

[!] Note 2:

You can set generic blob expression as INPUT filters.

Multi-lines

Some logs can be on N lines. They are called multi-lines. Multi-line filter must always be before any GROK filter!

Handle spaces

A new event must NOT start with a space.

# All lines starting with a space belong to the previous event
multiline {
       pattern => "^\s"
       negate => false
       what => "previous"
}

Java exceptions

This will make all exceptions belong to the previous event.

# All exceptions belong to the previous event
multiline {
       pattern => "(([^\s]+)Exception.+)|(at:.+)"
       negate => false
       what => "previous"
}

LOG4J trick

If you only expect Log4j logs then you know that each line that does NOT start with a %{TIMESTAMP} is NOT a new event.

# All lines that does not start with %{TIMESTAMP} or ' ' + %{TIMESTAMP} belong to the previous event
multiline {
       pattern => "(([\s]+)20[0-9]{2}-)|20[0-9]{2}-"
       negate => true
       what => "previous"
}

Grok failure

If your Grok expression is wrong the line will be tagged as '_grokparsefailure' .

Since you know how to detect error, you can attempt to apply an alternate filter on the log.

filter {
	# myApplication
	if [type] == "myApp" {
		grok {
                      ...
		}
		# Something wrong occurred !!! :O Do something else instead!
		if "_grokparsefailure" in [tags] {
			grok {
				 patterns_dir => "/etc/logstash/grok"
				 match=>[
					"message","(?<content>(.|\r|\n)*)"
					]
				 }
		}
	}
}

Common Logstash configurations

Apache2

Requirements:

• Make sure your logs are in "/var/log/apache2" or adjust the paths
• Make sure your using the COMBINED logs (default in Apache 2.4+)

Logstash configuration extract:

input {
	file {
	    path => [ "/var/log/apache2/access.log", "/var/log/apache2/ssl_access.log", "/var/log/apache2/other_vhosts_access.log" ]
	    type => "apache-access"
	}
	file {
	    path => "/var/log/apache2/error.log"
	    type => "apache-error"
	}
}

filter {
	# ------------------------ Parse services logs into fields ---------------------------
	# APACHE 2
	if [type] == "apache-access" {
		# To process log data (message's content) using some regex or precompiled GROK pattern
		grok {
			match => [ "message", "%{COMBINEDAPACHELOG}"]
		}
		# To extract log's time according to a date pattern
		date {
			match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
		}
		# Extraction browser information, if available.
		if [agent] != "" {
			useragent {
				source => "agent"
			}
		}
		if [clientip] != "" {
			geoip {
				source => "clientip"        
                                target => "apache_clientip"
                                add_tag => [ "geoip" ]
                        }
                }
	}

	if [type] == "apache-error" {
		grok {
			match => [ "message", "%{APACHEERRORLOG}"]
			# Directory where to find the custom patterns
			patterns_dir => ["/etc/logstash/grok"]
		}
                if [clientip] != "" {
			geoip {
				source => "clientip"        
                                target => "apache_clientip"
                                add_tag => [ "geoip" ]
			}
		}	
	}
}

output { 
   ...
}

IpTables

Requirements:

• Make sure you are logging dropped packets into a dedicated file. See Firewall log dropped

Logstash configuration extract:

input {
	file {
	    path => "/var/log/iptables.log"
	    type => "iptables"
	}
}


filter {
    # IPTABLES
    if [type] == "iptables" {
        grok {
            match => [
                        "message", "%{IPTABLES_IP}",
                        "message", "%{IPTABLES_ICMP}",
                        "message", "%{IPTABLES_GENERIC}"
                     ]
            patterns_dir => ["/etc/logstash/grok"]
        }
        # Something wrong occurred !!! :O
        if "_grokparsefailure" in [tags] {
            grok {
                     patterns_dir => "/etc/logstash/grok"
                     match=>["message", "%{IPTABLES_ERROR}" ]
                     add_tag => "iptables-grok_error"
                 }
        }
        # Default 'geoip' == src_ip. That means it's easy to display the DROPPED INPUT :)
        if [src_ip] != "" {
            geoip {
                source => "src_ip"
                add_tag => [ "geoip" ]
                target => "src_geoip"
            }
        }
        if [dst_ip] != "" {
            geoip {
                source => "dst_ip"
                add_tag => [ "geoip" ]
                target => "dst_geoip"
            }
        }
    }
}

output { 
   ...
}

Fail2ban

Logstash configuration extract:

input {
	file {
	    path => "/var/log/fail2ban.log"
	    type => "fail2ban"
	}
}


filter {
	# Fail2ban
	if [type] == "fail2ban" {
		grok {
			match => ["message", "%{FAIL2BAN}"]
			patterns_dir => ["/etc/logstash/grok"]
		}
                if [ban_ip] != "" {
			geoip {
			        source => "ban_ip"
                                add_tag => [ "geoip" ]
			        target => "ban_geoip"
			}
		}	
	}
}

output { 
   ...
}

Syslog

Logstash configuration extract:

input {
	file {
	    path => [ "/var/log/syslog", "/var/log/auth.log", "/var/log/mail.info" ]
	    type => "syslog"
	}
}


filter {
	# SYSLOG
	if [type] == "syslog" {
		grok {
			match => ["message", "%{SYSLOGBASE}"]
		}
	}
}

output { 
   ...
}

Tomcat

... To be done ...

Log4J

input {
	file {
	    path => [ "/home/beta3/catalina.base/logs/vehco/*.log" ]
	    type => "myApp"
	}
}

filter {	
	# All lines that does not start with %{TIMESTAMP} or ' ' + %{TIMESTAMP} belong to the previous event
	multiline {
		pattern => "(([\s]+)20[0-9]{2}-)|20[0-9]{2}-"
		negate => true
		what => "previous"
	}
	
	# myApplication
	if [type] == "myApp" {
		grok {
			patterns_dir => ["/etc/logstash/grok"]
			match => [
				"message", "%{LOG4J}"
				]
			add_tag => "myApp-log"
		}
		# Something wrong occurred !!! :O
		if "_grokparsefailure" in [tags] {
			grok {
				 patterns_dir => "/etc/logstash/grok"
				 match=>[
					"message","(?<content>(.|\r|\n)*)"
					]
				 }
		}
	}
}

output { 
   ...
}

VEHCO specific patterns

Now that you have some specific GROK patterns, you need to update your Logstash configuration.

input {
	file {
	    path => [ "/var/log/vehco/*.log" ]
	    type => "vehco-rtd"
	}
}


filter {
	# VEHCO-RTD
	if [type] == "vehco-rtd" {
		grok {
			patterns_dir => ["/etc/logstash/grok"]
			match => [
                                    "message", "%{RTD_TERMINAL}",
                                    "message", "%{RTD_AUTH_START}",
                                    "message", "%{RTD_AUTH_DONE}"
                                 ]
		}
	}
}

output { 
   ...
}