Category Archives: Elasticsearch

[Elasticsearch] The health status means…

Posted on 2015 年 02 月 24 日 by cowman.chiang

When we list the indexes, we will get the status information. It shows the health of index in the elasticsearch cluster. green All shards are allocated. yellow The primary shard is allocated but replicas are not. red The specific shard … Continue reading →

Posted in Elasticsearch | Leave a comment

[Elasticsearch] List all indexes and optimize it.

Posted on 2015 年 02 月 24 日 by cowman.chiang

When we deleted the indexes, the data were just marked by deleted. We can list all indexes to show status.

sudo curl 'localhost:9200/_cat/indices?v'

1 2	sudo curl 'localhost:9200/_cat/indices?v'

It will show the health, index name, pri, rep, docs.count, docs.deleted, store.size, pri.store.size.

health index               pri rep docs.count docs.deleted store.size pri.store.size
yellow kibana-int            5   1          5            0     66.7kb         66.7kb
yellow logstash-2015.02.13   5   1     402030      3366353      1.2gb          1.2gb
yellow logstash-2015.02.18   5   1      86843      3393979      1.1gb          1.1gb
yellow logstash-2015.02.15   5   1     218402      3063776      1.2gb          1.2gb
yellow logstash-2015.02.03   5   1     178621      2072000    756.3mb        756.3mb
yellow logstash-2015.02.06   5   1     466185      2479658    963.4mb        963.4mb
yellow logstash-2015.02.24   5   1     364473            0    166.5mb        166.5mb
yellow logstash-2015.02.02   5   1     313488      1838855    710.1mb        710.1mb
yellow logstash-2015.02.07   5   1     371602      3397300      1.2gb          1.2gb
yellow logstash-2015.02.21   5   1    3583288            0      1.2gb          1.2gb
yellow logstash-2015.02.22   5   1    3554707            0      1.1gb          1.1gb
yellow logstash-2015.02.23   5   1    3680625            0      1.2gb          1.2gb

health index pri rep docs.count docs.deleted store.size pri.store.size

yellow kibana-int 5 1 5 0 66.7kb 66.7kb

yellow logstash-2015.02.13 5 1 402030 3366353 1.2gb 1.2gb

yellow logstash-2015.02.18 5 1 86843 3393979 1.1gb 1.1gb

yellow logstash-2015.02.15 5 1 218402 3063776 1.2gb 1.2gb

yellow logstash-2015.02.03 5 1 178621 2072000 756.3mb 756.3mb

yellow logstash-2015.02.06 5 1 466185 2479658 963.4mb 963.4mb

yellow logstash-2015.02.24 5 1 364473 0 166.5mb 166.5mb

yellow logstash-2015.02.02 5 1 313488 1838855 710.1mb 710.1mb

yellow logstash-2015.02.07 5 1 371602 3397300 1.2gb 1.2gb

yellow logstash-2015.02.21 5 1 3583288 0 1.2gb 1.2gb

yellow logstash-2015.02.22 5 1 3554707 0 1.1gb 1.1gb

yellow logstash-2015.02.23 5 1 3680625 0 1.2gb 1.2gb

If we want to clear … Continue reading →

Posted in Elasticsearch | Leave a comment

[Elasticsearch] Delete index by search syntax

Posted on 2015 年 01 月 27 日 by cowman.chiang

check the search syntax

curl -XGET 'http://{ElasticSearch_Server_IP}:{ElasticSearch_Server_Port}/{Index}/{Type}/_search?q={Search Syntax}

1	curl -XGET 'http://{ElasticSearch_Server_IP}:{ElasticSearch_Server_Port}/{Index}/{Type}/_search?q={Search Syntax}

ex:

curl -XGET 'http://localhost:9200/logstash-2015.01.26/syslog/_search?q="Rhttpproxy"'

1	curl -XGET 'http://localhost:9200/logstash-2015.01.26/syslog/_search?q="Rhttpproxy"'

Delete index by search syntax

curl -XDELETE 'http://{ElasticSearch_Server_IP}:{ElasticSearch_Server_Port}/{Index}/{Type}/_query?q={Search Syntax}

1	curl -XDELETE 'http://{ElasticSearch_Server_IP}:{ElasticSearch_Server_Port}/{Index}/{Type}/_query?q={Search Syntax}

ex:

curl -XDELETE 'http://localhost:9200/logstash-2015.01.26/syslog/_query?q="Rhttpproxy"'

1	curl -XDELETE 'http://localhost:9200/logstash-2015.01.26/syslog/_query?q="Rhttpproxy"'

Posted in Elasticsearch | Leave a comment

[Apache] Proxypass

Posted on 2015 年 01 月 27 日 by cowman.chiang

Because the default web port of Elasticsearch is 9200. And I want to change it with 80. Maybe I could use the proxypass function. enable the proxy, and restart apache

a2enmod proxy
a2enmod proxy_http
service apache2 restart

a2enmod proxy

a2enmod proxy_http

service apache2 restart

add following into the /etc/apache2/sites-enabled/000-default.conf

<VirtualHost *:80>
       ProxyPreserveHost On
       ProxyVia full

       <proxy>
         Order deny,allow
         Allow from all
       </proxy>

       ProxyPass        /kibana9200  http://localhost:9200
</VirtualHost>

ProxyPreserveHost On

ProxyVia full

<proxy>

Order deny,allow

Allow from all

</proxy>

ProxyPass /kibana9200 http://localhost:9200

</VirtualHost>

edit the kibana … Continue reading →

Posted in Apache, Elasticsearch, Kibana | Leave a comment

淺談 Elasticsearch 的定義

Posted on 2014 年 12 月 30 日 by cowman.chiang

[摘錄自Elasticsearch Server, 2nd Edition第12頁] The basics of Elasticsearch Elasticsearch is an open source search server project started by Shay Banon and published in February 2010. During this time, the project has grown into a major player in the field of … Continue reading →

Posted in Elasticsearch | Leave a comment

elasticsearch.yml 檔案說明

Posted on 2014 年 12 月 30 日 by cowman.chiang

##################### Elasticsearch Configuration Example #####################

# This file contains an overview of various configuration settings,
# targeted at operations staff. Application developers should
# consult the guide at <http://elasticsearch.org/guide>.

##################### Elasticsearch Configuration Example #####################

# This file contains an overview of various configuration settings,

# targeted at operations staff. Application developers should

# consult the guide at <http://elasticsearch.org/guide>.

說明此檔案包含多個設定的部份，可以至 Elasticsearch Guide 觀看說明

#
# The installation procedure is covered at
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

# The installation procedure is covered at

# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.

安裝相關的部份可參考 Elasticsearch 安裝說明

#
# Elasticsearch comes with reasonable defaults for most settings,
# so you can try it out without bothering with configuration.

# Elasticsearch comes with reasonable defaults for most settings,

# so you can try it out without bothering with configuration.

Elasticsearch預設會套用許多設定，所已可以不用更改設定就可先進行測試

#
# Most of the time, these defaults are just fine for running a production
# cluster. If you're fine-tuning your cluster, or wondering about the
# effect of certain configuration option, please _do ask_ on the
# mailing list or IRC channel [http://elasticsearch.org/community].

# Most of the time, these defaults are just fine for running a production

# cluster. If you're fine-tuning your cluster, or wondering about the

# effect of certain configuration option, please _do ask_ on the

# mailing list or IRC channel [http://elasticsearch.org/community].

預設情況下，絕大部分都能夠支應一個生產情況下的Cluster，假如想要試著調整設定，或是擔心更改設定會影響整各服務，可以寫信給開發團隊或是至 Elasticsearch 社群詢問

# Any element in the configuration can be replaced with environment variables
# by placing them in ${...} notation. For example:
#
# node.rack: ${RACK_ENV_VAR}

# Any element in the configuration can be replaced with environment variables

# by placing them in ${...} notation. For example:

# node.rack: ${RACK_ENV_VAR}

任何設定檔的元素皆可以使用環境變數

# For information on supported formats and syntax for the config file, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>

1 2	# For information on supported formats and syntax for the config file, see # <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>

針對設定檔內部所支援的格式可以參考 Elasticsearch 安裝設定說明

################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
 cluster.name: elasticsearch

################################### Cluster ###################################

# Cluster name identifies your cluster for auto-discovery. If you're running

# multiple clusters on the same network, make sure you're using unique names.

cluster.name: elasticsearch

設定cluster的名稱，後續將使用multicast進行詢問server在哪裡，當server收到multicast後就會跟node進行聯繫

#################################### Node #####################################

# Node names are generated dynamically on startup, so you're relieved
# from configuring them manually. You can tie this node to a specific name:
#
 node.name: "Cowb"

#################################### Node #####################################

# Node names are generated dynamically on startup, so you're relieved

# from configuring them manually. You can tie this node to a specific name:

node.name: "Cowb"

對本node進行命名

# Every node can be configured to allow or deny being eligible as the master,
# and to allow or deny to store the data.
#
# Allow this node to be eligible as a master node (enabled by default):
#
# node.master: true
#
# Allow this node to store data (enabled by default):
#
# node.data: true

# Every node can be configured to allow or deny being eligible as the master,

# and to allow or deny to store the data.

# Allow this node to be eligible as a master node (enabled by default):

# node.master: true

# Allow this node to store data (enabled by default):

# node.data: true

在此先對master node跟data node進行解釋 [摘錄自Elasticsearch Server, 2nd Edition第324頁] The master node is the one … Continue reading →

Posted in Elasticsearch | Leave a comment

Logstash + Redis + Nginx + Kibana + ElasticSearch

Posted on 2014 年 12 月 26 日 by cowman.chiang

環境: Ubuntu 14.04 x64 server 安裝Java環境

sudo apt-get install openjdk-7-jdk

1	sudo apt-get install openjdk-7-jdk

下載Logstash壓縮檔

cd /tmp
curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.0.tar.gz
tar zxvf logstash-1.4.0.tar.gz -C /opt/logstash

cd /tmp

curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.0.tar.gz

tar zxvf logstash-1.4.0.tar.gz -C /opt/logstash

基本測試

cd /opt/logstash
bin/logstash -e 'input { stdin { } } output { stdout {} }
# 試著打幾個字看看

cd /opt/logstash

bin/logstash -e 'input { stdin { } } output { stdout {} }

# 試著打幾個字看看

安裝Elasticsearch

cd /tmp
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb
sudo dpkg -i elasticsearch-1.1.1.deb

cd /tmp

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb

sudo dpkg -i elasticsearch-1.1.1.deb

設定Elasticsearch 編輯/etc/elasticsearch/elasticsearch.yml 取消cluster.name、node.name的注解並改成自己想要的重新啟動Elasticsearch

sudo service elastic search restart

1	sudo service elastic search restart

安裝Redis

sudo apt-get install redis-server

1	sudo apt-get install redis-server

設定Redis 編輯/etc/redis/redis.conf，使用

bind 0.0.0.0
取代
bind 127.0.0.1

bind 0.0.0.0

取代

bind 127.0.0.1

重新啟動Redis

sudo service redis-server restart

1	sudo service redis-server restart

下載Kibana

cd /tmp
wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz
mkdir -p /var/www/kibana
tar xvfz kibana-3.0.0.tar.gz -C /var/www/kibana

cd /tmp

wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz

mkdir -p /var/www/kibana

tar xvfz kibana-3.0.0.tar.gz -C /var/www/kibana

編輯Kibana設定檔編輯/var/www/kubana/config.js，使用

elasticsearch: "http://本機IP:9200"
取代
elasticsearch: "http://"+window.location.hostname+":9200"

elasticsearch: "http://本機IP:9200"

取代

elasticsearch: "http://"+window.location.hostname+":9200"

設定Logstash Server設定檔編輯/etc/logstash/server.conf

input {
  redis {
    host => "本機IP"
    type => "redis"
    data_type => "list"
    key => "logstash"
  }
}
output {
stdout { }
  elasticsearch {
    cluster => "elasticsearch"
  }
}

input {

redis {

host => "本機IP"

type => "redis"

data_type => "list"

key => "logstash"

}

output {

stdout { }

elasticsearch {

cluster => "elasticsearch"

}

測試Logstash能否以Server啟動

/opt/logstash/bin/logstash --verbose -f /etc/logstash/server.conf

1	/opt/logstash/bin/logstash --verbose -f /etc/logstash/server.conf

安裝Nginx

sudo apt-get install nginx

1	sudo apt-get install nginx

設定Nginx 編輯/etc/nginx/sites-enabled/default，使用 … Continue reading →

Posted in Elasticsearch, Kibana, Logstash, Redis | Leave a comment

Category Archives: Elasticsearch

[Elasticsearch] The health status means…

[Elasticsearch] List all indexes and optimize it.

[Elasticsearch] Delete index by search syntax

[Apache] Proxypass

淺談 Elasticsearch 的定義

elasticsearch.yml 檔案說明

Logstash + Redis + Nginx + Kibana + ElasticSearch

Categories