ElasticSearch

ElasticSearch is the central point of the ELK architecture. This is where data will be aggregated and persisted.

Requirements

To install and use ELK you need:

• JAVA 1.7.55+ (Java 8 is recommended)

java -version

Java version must be > 1.7.0_55

• Firewall rule

Open the TCP ports 9200 + 9300, allow multicast too.

See FW input && FW output

• Apache2 server

See Apache2 setup

• Python

apt-get install python3
apt-get install python-pip

ElasticSearch

Installation

Source: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html

• Add ELK repository: see Sources#ELK
• Install application

apt-get install elasticsearch

>> Binaries in /usr/share/elasticsearch

>> Configuration in /etc/elasticsearch

>> Logs in /var/log/elasticsearch

• Register application as a service

cd /etc/init.d
update-rc.d elasticsearch defaults 95 10

Configuration

Edit the configuration file:

vim /etc/elasticsearch/elasticsearch.yml

Set your CLUSTER and NODE name + allow Kibana access.

cluster.name: VEHCO         # line 33      
node.name: "VEHCO_MASTER"   # line 40

                            ### [...] At the end
http.cors.enabled: true
http.cors.allow-origin: https://smartcards.vehco.com     # http://192.168.1.203

!! You need to adjust your IP || hostname according to your needs.

Extensions (plugins)

You need to install some extensions (plugins) to get the full power of ElasticSearch.

The following plugins are the ones recommended by the ElasticSearch team.

cd /usr/share/elasticsearch/bin
./plugin -install karmi/elasticsearch-paramedic
./plugin -install mobz/elasticsearch-head
./plugin -install royrusso/elasticsearch-HQ

More information about each plugin:

• Paramedic = Simple ElasticSearc overview and real-time data, see: https://github.com/karmi/elasticsearch-paramedic
• Head = Quick cluster overview, see: http://mobz.github.io/elasticsearch-head
• HQ = Overall manager - quite powerful, see http://www.elastichq.org/

You can access the plugins using the /_plugin/ URL:

• http://localhost:9200/_plugin/paramedic/index.html
• http://localhost:9200/_plugin/head/
• http://localhost:9200/_plugin/HQ/

You can search for more plugins on Google or the official ElasticSearch web-site.

Checkout http://www.elasticsearch.org/download to get a list of plugins available per official developer.

ElasticSearch tooling

Curator

Curator allows you to remove the old indices.

Installation

pip install elasticsearch-curator

Usage

# Display VEHCO- indices
curator show --show-indices --prefix vehco-
# Remove indices that are more than 10 days old

References

• Curator: https://github.com/elasticsearch/curator/

Start ElasticSearch

service elasticsearch start 

## OR ##
/etc/init.d/elasticsearch start

Delete indices

Reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-index.html

curl -XDELETE 'http://localhost:9200/smartcard-monitoring-2014*/'

replace smartcard-monitoring-2014* by your own expression.

References

• Very good webinar from the ElasticSearch team: http://www.elasticsearch.org/webinars/introduction-to-logstash/?watch=1
• Official ElasticSearch documentation: http://www.elasticsearch.org