Difference between revisions of "ElasticSearch"
 
(9 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
[[Category:Linux]]
 
[[Category:Linux]]
 +
 +
[[File:Icon elastic search.png|32px|caption|ElasticSearch]] ''ElasticSearch'' is the central point of the ELK architecture. This is where data will be aggregated and persisted.
  
  
Line 6: Line 8:
 
To install and use ELK you need:  
 
To install and use ELK you need:  
  
−
* '''JAVA 1.7.55+'''  
+
* '''JAVA 1.7.55+''' (Java 8 is recommended)
  
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
Line 17: Line 19:
 
* '''Firewall rule'''
 
* '''Firewall rule'''
  
−
Open the port 9200. See [[Firewall INPUT filters#ElasticSearch|FW input]] && [[Firewall OUTPUT filters#IT_ports|FW output]]
+
Open the TCP ports 9200 + 9300, allow multicast too.  
 +
 
 +
See [[Firewall INPUT filters#ElasticSearch|FW input]] && [[Firewall OUTPUT filters#IT_ports|FW output]]
 +
 
 +
 
 +
* '''Apache2 server'''
  
 +
See [[Template:Menu content web#Web_server|Apache2 setup]]
 +
 +
 +
* Python
 +
 +
<syntaxhighlight lang="bash">
 +
apt-get install python3
 +
apt-get install python-pip
 +
</syntaxhighlight>
  
  
Line 36: Line 52:
 
apt-get install elasticsearch
 
apt-get install elasticsearch
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
>> Binaries in ''/usr/share/elasticsearch''
 +
 +
>> Configuration in ''/etc/elasticsearch''
 +
 +
>> Logs in ''/var/log/elasticsearch''
 +
  
  
Line 55: Line 78:
  
  
−
Set your CLUSTER and NODE name.
+
Set your CLUSTER and NODE name + allow Kibana access.
  
 
<syntaxhighlight lang="yaml">
 
<syntaxhighlight lang="yaml">
−
cluster.name: VEHCO        # line 33
+
cluster.name: VEHCO        # line 33    
 
node.name: "VEHCO_MASTER"  # line 40
 
node.name: "VEHCO_MASTER"  # line 40
 +
 +
                            ### [...] At the end
 +
http.cors.enabled: true
 +
http.cors.allow-origin: https://smartcards.vehco.com    # http://192.168.1.203
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
!! You need to adjust your IP || hostname according to your needs.
  
  
 
==Extensions (plugins)==
 
==Extensions (plugins)==
  
−
You need to install some extensions (plugins) to get the full power of ElasticSearch. The following plugin list is the one recommended by ElasticSearch team.  
+
You need to install some extensions (plugins) to get the full power of ElasticSearch.  
 +
 
 +
The following plugins are the ones recommended by the ElasticSearch team.  
  
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
−
cd /usr/share/elasticsearch/
+
cd /usr/share/elasticsearch/bin
 
./plugin -install karmi/elasticsearch-paramedic
 
./plugin -install karmi/elasticsearch-paramedic
 
./plugin -install mobz/elasticsearch-head
 
./plugin -install mobz/elasticsearch-head
Line 90: Line 121:
  
 
Checkout http://www.elasticsearch.org/download to get a list of plugins available per official developer.
 
Checkout http://www.elasticsearch.org/download to get a list of plugins available per official developer.
 +
 +
 +
==ElasticSearch tooling==
 +
 +
===Curator===
 +
 +
Curator allows you to remove the old indices.
 +
 +
 +
'''Installation'''
 +
 +
<syntaxhighlight lang="bash">
 +
pip install elasticsearch-curator
 +
</syntaxhighlight>
 +
 +
 +
 +
'''Usage'''
 +
 +
<syntaxhighlight lang="bash">
 +
# Display VEHCO- indices
 +
curator show --show-indices --prefix vehco-
 +
# Remove indices that are more than 10 days old
 +
</syntaxhighlight>
 +
 +
 +
'''References'''
 +
 +
* Curator: https://github.com/elasticsearch/curator/
 +
  
  
Line 103: Line 164:
  
  
 +
=Delete indices=
 +
 +
Reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-index.html
  
−
=References=
 
  
−
* Very good webinar from the ElasticSearch team: http://www.elasticsearch.org/webinars/introduction-to-logstash/?watch=1
+
<syntaxhighlight lang="bash">
−
* Official ElasticSearch documentation: http://www.elasticsearch.org
+
curl -XDELETE 'http://localhost:9200/smartcard-monitoring-2014*/'
 +
</syntaxhighlight>
  
 +
replace ''smartcard-monitoring-2014*'' by your own expression.
  
  
  
−
TO BE CONTINUED...
 
  
−
user & group "elasticsearch"
+
=References=
  
−
>> CONF_DIR=/etc/elasticsearch
+
* Very good webinar from the ElasticSearch team: http://www.elasticsearch.org/webinars/introduction-to-logstash/?watch=1
−
>> LOG_DIR=/var/log/elasticsearch
+
* Official ElasticSearch documentation: http://www.elasticsearch.org
−
>> DATA=/var/lib/elasticsearch
 

Latest revision as of 15:54, 5 February 2015


ElasticSearch ElasticSearch is the central point of the ELK architecture. This is where data will be aggregated and persisted.


Requirements

To install and use ELK you need:

  • JAVA 1.7.55+ (Java 8 is recommended)
java -version

Java version must be > 1.7.0_55


  • Firewall rule

Open the TCP ports 9200 + 9300, allow multicast too.

See FW input && FW output


  • Apache2 server

See Apache2 setup


  • Python
apt-get install python3
apt-get install python-pip


ElasticSearch

Installation

Source: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html


  • Add ELK repository: see Sources#ELK
  • Install application
apt-get install elasticsearch

>> Binaries in /usr/share/elasticsearch

>> Configuration in /etc/elasticsearch

>> Logs in /var/log/elasticsearch


  • Register application as a service
cd /etc/init.d
update-rc.d elasticsearch defaults 95 10


Configuration

Edit the configuration file: 

vim /etc/elasticsearch/elasticsearch.yml


Set your CLUSTER and NODE name + allow Kibana access. 

cluster.name: VEHCO         # line 33      
node.name: "VEHCO_MASTER"   # line 40

                            ### [...] At the end
http.cors.enabled: true
http.cors.allow-origin: https://smartcards.vehco.com     # http://192.168.1.203

!! You need to adjust your IP || hostname according to your needs.


Extensions (plugins)

You need to install some extensions (plugins) to get the full power of ElasticSearch.

The following plugins are the ones recommended by the ElasticSearch team. 

cd /usr/share/elasticsearch/bin
./plugin -install karmi/elasticsearch-paramedic
./plugin -install mobz/elasticsearch-head
./plugin -install royrusso/elasticsearch-HQ


More information about each plugin:


You can access the plugins using the /_plugin/ URL:


You can search for more plugins on Google or the official ElasticSearch web-site.

Checkout http://www.elasticsearch.org/download to get a list of plugins available per official developer.


ElasticSearch tooling

Curator

Curator allows you to remove the old indices.


Installation 

pip install elasticsearch-curator


Usage 

# Display VEHCO- indices
curator show --show-indices --prefix vehco-
# Remove indices that are more than 10 days old


References



Start ElasticSearch

service elasticsearch start 

## OR ##
/etc/init.d/elasticsearch start


Delete indices

Reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-index.html 


curl -XDELETE 'http://localhost:9200/smartcard-monitoring-2014*/'

replace smartcard-monitoring-2014* by your own expression.



References

Retrieved from "http://www.daxiongmao.eu/wiki/index.php?title=ElasticSearch&oldid=2075"