Difference between revisions of "ElasticSearch"
| (2 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
[[Category:Linux]] | [[Category:Linux]] | ||
| + | |||
| + | [[File:Icon elastic search.png|32px|caption|ElasticSearch]] ''ElasticSearch'' is the central point of the ELK architecture. This is where data will be aggregated and persisted. | ||
| Line 6: | Line 8: | ||
To install and use ELK you need: | To install and use ELK you need: | ||
| − | * '''JAVA 1.7.55+''' | + | * '''JAVA 1.7.55+''' (Java 8 is recommended) |
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
| Line 17: | Line 19: | ||
* '''Firewall rule''' | * '''Firewall rule''' | ||
| − | Open the ports 9200 + 9300, allow multicast too. See [[Firewall INPUT filters#ElasticSearch|FW input]] && [[Firewall OUTPUT filters#IT_ports|FW output]] | + | Open the TCP ports 9200 + 9300, allow multicast too. |
| + | |||
| + | See [[Firewall INPUT filters#ElasticSearch|FW input]] && [[Firewall OUTPUT filters#IT_ports|FW output]] | ||
| Line 77: | Line 81: | ||
<syntaxhighlight lang="yaml"> | <syntaxhighlight lang="yaml"> | ||
| − | cluster.name: VEHCO # line 33 | + | cluster.name: VEHCO # line 33 |
| − | node.name: "VEHCO_MASTER" # line 40 | + | node.name: "VEHCO_MASTER" # line 40 |
### [...] At the end | ### [...] At the end | ||
http.cors.enabled: true | http.cors.enabled: true | ||
| − | http.cors.allow-origin: https:// | + | http.cors.allow-origin: https://smartcards.vehco.com # http://192.168.1.203 |
</syntaxhighlight> | </syntaxhighlight> | ||
| Line 90: | Line 94: | ||
==Extensions (plugins)== | ==Extensions (plugins)== | ||
| − | You need to install some extensions (plugins) to get the full power of ElasticSearch. The following | + | You need to install some extensions (plugins) to get the full power of ElasticSearch. |
| + | |||
| + | The following plugins are the ones recommended by the ElasticSearch team. | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
| Line 118: | Line 124: | ||
==ElasticSearch tooling== | ==ElasticSearch tooling== | ||
| − | |||
===Curator=== | ===Curator=== | ||
| Line 157: | Line 162: | ||
/etc/init.d/elasticsearch start | /etc/init.d/elasticsearch start | ||
</syntaxhighlight> | </syntaxhighlight> | ||
| + | |||
| + | |||
| + | =Delete indices= | ||
| + | |||
| + | Reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-index.html | ||
| + | |||
| + | |||
| + | <syntaxhighlight lang="bash"> | ||
| + | curl -XDELETE 'http://localhost:9200/smartcard-monitoring-2014*/' | ||
| + | </syntaxhighlight> | ||
| + | |||
| + | replace ''smartcard-monitoring-2014*'' by your own expression. | ||
| + | |||
Latest revision as of 15:54, 5 February 2015
ElasticSearch is the central point of the ELK architecture. This is where data will be aggregated and persisted.
Contents
Requirements
To install and use ELK you need:
- JAVA 1.7.55+ (Java 8 is recommended)
java -versionJava version must be > 1.7.0_55
- Firewall rule
Open the TCP ports 9200 + 9300, allow multicast too.
- Apache2 server
See Apache2 setup
- Python
apt-get install python3
apt-get install python-pip
ElasticSearch
Installation
Source: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html
- Add ELK repository: see Sources#ELK
- Install application
apt-get install elasticsearch>> Binaries in /usr/share/elasticsearch
>> Configuration in /etc/elasticsearch
>> Logs in /var/log/elasticsearch
- Register application as a service
cd /etc/init.d
update-rc.d elasticsearch defaults 95 10
Configuration
Edit the configuration file:
vim /etc/elasticsearch/elasticsearch.yml
Set your CLUSTER and NODE name + allow Kibana access.
cluster.name: VEHCO # line 33
node.name: "VEHCO_MASTER" # line 40
### [...] At the end
http.cors.enabled: true
http.cors.allow-origin: https://smartcards.vehco.com # http://192.168.1.203!! You need to adjust your IP || hostname according to your needs.
Extensions (plugins)
You need to install some extensions (plugins) to get the full power of ElasticSearch.
The following plugins are the ones recommended by the ElasticSearch team.
cd /usr/share/elasticsearch/bin
./plugin -install karmi/elasticsearch-paramedic
./plugin -install mobz/elasticsearch-head
./plugin -install royrusso/elasticsearch-HQ
More information about each plugin:
- Paramedic = Simple ElasticSearc overview and real-time data, see: https://github.com/karmi/elasticsearch-paramedic
- Head = Quick cluster overview, see: http://mobz.github.io/elasticsearch-head
- HQ = Overall manager - quite powerful, see http://www.elastichq.org/
You can access the plugins using the /_plugin/ URL:
- http://localhost:9200/_plugin/paramedic/index.html
- http://localhost:9200/_plugin/head/
- http://localhost:9200/_plugin/HQ/
You can search for more plugins on Google or the official ElasticSearch web-site.
Checkout http://www.elasticsearch.org/download to get a list of plugins available per official developer.
ElasticSearch tooling
Curator
Curator allows you to remove the old indices.
Installation
pip install elasticsearch-curator
Usage
# Display VEHCO- indices
curator show --show-indices --prefix vehco-
# Remove indices that are more than 10 days old
References
Start ElasticSearch
service elasticsearch start
## OR ##
/etc/init.d/elasticsearch start
Delete indices
Reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-index.html
curl -XDELETE 'http://localhost:9200/smartcard-monitoring-2014*/'replace smartcard-monitoring-2014* by your own expression.
References
- Very good webinar from the ElasticSearch team: http://www.elasticsearch.org/webinars/introduction-to-logstash/?watch=1
- Official ElasticSearch documentation: http://www.elasticsearch.org
