标签：elasticsearch logstash kibana kafka
在经过了近半个月的ELK环境的搭建配置，我做了我个人的工作总结，和大家分享一下。
一、命令总结
1.1、Es服务端口查看
# netstat -nlpt | grep -E"9200|9300"
1.2、Es插件安装和移除
# ./bin/plugin install file:///home/apps/license-2.3.3.zip
# ./bin/plugin install file:///home/apps/marvel-agent-2.3.3.zip
es移除插件
/usr/local/elasticsearch # ./bin/pluginremove plugins/marvel-agent
/usr/local/elasticsearch # ./bin/pluginremove plugins/license
1.3、Kibana插件安装和移除
/usr/local/kibana # ./bin/kibana plugin--install marvel --url file:///home/apps/marvel-2.3.3.tar.gz
移除插件
./bin/kibana plugin --remove marvel
1.4、logstash检查文件是否有错误
/usr/local/logstash/etc # ../bin/logstash-f logstash.conf --configtest --verbose
（显示“Configuration OK”说明没有问题）
1.5、启动logstash
/usr/local/logstash/etc # ../bin/logstash-f logstash.conf
1.6、验证es服务
#curl -XGET ES-NODE-IP:9200
1.7、查看es集群状态
# curl -XGET ES-NODE-IP:9200/_cluster/health?pretty=true
1.8、添加和删除es索引，nginx-logs为索引名
curl -XPUT http://ES-NODE-IP:9200/nginx-logs/
删除es索引
curl -XDELETE http://ES-NODE-IP:9200/nginx-logs/
1.9验证head插件
http://ES-NODE-IP:9200/_plugin/head/
1.10、创建elasticsearch账户
# groupadd esuser
# useradd -d /home/esuser -m esuser
# passwd esuser
1.11查看kafka服务启动状态
# jps
9536 Main
15200 Jps
14647 Kafka
8760 Elasticsearch
21177 -- process information unavailable
14316 QuorumPeerMain
5791 QuorumPeerMain
1．12、查看kafka端口状态
# netstat -nlpt | grep -E"2181|3888"
tcp 0 0 :::2181 :::* LISTEN 14316/java
tcp 0 0 192.168.1.105:3888 :::* LISTEN 5791/java
1.13、创建、删除、列出主题（topic）
创建主题（topic）
/usr/local/kafka # bin/kafka-topics.sh--create --zookeeper localhost:2181 --replication-factor 1 --partitions 1--topic test
Created topic "test".
删除主题（topic）
/usr/local/kafka # ./bin/kafka-topics.sh--delete --zookeeper 182.180.50.211:2181 --topic nginx-messages
列出所有的topic
/usr/local/kafka # bin/kafka-topics.sh--list --zookeeper localhost:2181
Test
显示topic的详细信息
/usr/local/kafka # bin/kafka-topics.sh --describe--zookeeper localhost:218
1.14、kafka创建生产者和消费者
创建生产者
/usr/local/kafka #bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
创建消费者
/usr/local/kafka # bin/kafka-console-consumer.sh --zookeeperlocalhost:2181 --topic test --from-beginning
This is a message
如果向上边的可以收到来自生产者的消息，就说明基于kafka的zookeeper环境配置好了。
1.15、测试日志文件的数据传输
# cp /var/log/messages /home
# >/var/log/messages
# echo "hello kibana aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa">> /var/log/messages
1.16、重新加载nginx配置文件
# ./nginx -s reload
二、 配置汇总
2.1、Java环境配置
# vi /etc/profile
export PATH=$PATH:/soft_ins/mysql/bin
JAVA_HOME=/usr/local/jdk1.8.0_101
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar
export PATH JAVA_HOME CLASSPATH
# source /etc/profile
# java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build25.101-b13, mixed mode)
2.2、es配置
# vi config/elasticsearch.yml
cluster.name: es_cluster
node.name: node3
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
network.host: 192.168.1.103
http.port: 9200
discovery.zen.ping.unicast.hosts:["192.168.1.103","192.168.1.104","192.168.1.105"]
说明：
discovery.zen.ping.unicast.hosts:["192.168.1.103","192.168.1.104","192.168.1.105"]是es集群的说明，如果是单各es节点的话就没必要配置这个。
2.3、kibana配置
# vi /usr/local/kibana/config/kibana.yml
server.port: 5601
host: "192.168.1.103"
elasticsearch_url: http://192.168.1.103:9200，192.168.1.104:9200,192.168.1.105:9200
说明：
elasticsearch_url: http://192.168.1.103:9200，192.168.1.104:9200,192.168.1.105:9200也是只想的es集群，如果是单台则只需配置单个es节点。
2.4、kafka和zookeeper非分离的配置
单个kafka&zookeeper节点
/usr/local/kafka # vi config/zookeeper.properties
dataDir=/usr/local/kafka/tmp/zookeeper
/usr/local/kafka # viconfig/server.properties
log.dirs=/usr/local/kafka/tmp/kafka-logs
Zookeepe集群配置
# vi config/zookeeper.properties
dataDir=/usr/local/kafka/tmp/zookeeper
initLimit=5
syncLimit=2
server.2=192.168.1.101:2888:3888
server.3=192.168.1.102:2888:3888
server.4=192.168.1.103:2888:3888
kafka集群：
/usr/local/kafka # viconfig/server.properties
broker.id=2
prot=9092
host.name=192.168.1.101
log.dirs=/usr/local/kafka/tmp/kafka-logs
num.partitions=16
zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181
注意其他两个节点broker.id分别是3和4，,host.name也按实际的配置
启动zookeeper服务
/usr/local/kafka #./bin/zookeeper-server-start.sh config/zookeeper.properties
启动kafka服务
/usr/local/kafka #./bin/kafka-server-start.sh config/server.properties
2.5、kafka和zookeeper分离的配置
Zookeeper的配置：
生成zookeeper配置文件
# cd zookeeper/conf
# cp zoo_sample.cfg zoo.cfg
编辑配置文件
# vi zoo.cfg
dataDir=/usr/local/zookeeper/tmp/zookeeper
server.1=192.168.1.101:2888:3888
server.2=192.168.1.102:2888:3888
server.3=192.168.1.103:2888:3888
# cd ..
# mkdir -p tmp/zookeeper
# echo "1" >tmp/zookeeper/myid
配置node2和node3的zookeeper
依照node1的配置配置node2和node3，注意下面的参数三个节点各有差异
Node2：
# echo "2" >tmp/zookeeper/myid
Node3：
# echo "3" >tmp/zookeeper/myid
其他配置都一样
依次启动三个节点的服务
# ./bin/zkServer.sh start conf/zoo.cfg
Kafka的配置：
配置node1的kafka
# cd ../kafka
# vi config/server.properties
broker.id=0
port=9092
host.name=x-shcs-creditcard-v01
log.dirs=/usr/local/kafka/tmp/kafka-logs
num.partitions=2
zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181
配置Node2和node3的kafka
依照node1的配置配置node2和node3，注意下面的参数三个节点各有差异
Node2:
broker.id=1
host.name=node2
node3：
broker.id=:2
host.name=node3
说明：
host.name是节点的主机名
依次启动三个节点的kafka
# ./bin/kafka-server-start config/server.properties
三、问题汇总
3.1、marvel插件安装出错
node3:/usr/local/elasticsearch# ./bin/plugin install file:///home/apps/license-2.3.3.zip
-> Installing fromfile:/home/apps/license-2.3.3.zip...
Trying file:/home/apps/license-2.3.3.zip...
Downloading .DONE
Verifying file:/home/apps/license-2.3.3.zipchecksums if available ...
NOTE: Unable to verify checksum fordownloaded plugin (unable to find .sha1 or .md5 file to verify)
ERROR: Plugin [license] is incompatiblewith Elasticsearch [2.1.1]. Was designed for version [2.3.3]
插件版本和ES版本不兼容，换为2.3.3版本的插件就没有问题了。
3.2、启动es服务出错
/usr/local/elasticsearch>./bin/elasticsearch
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException:/usr/local/elasticsearch/logs/es_cluster.log (Permission denied)
文件权限的问题，es服务启动需要切换到非root用户，而该用户没有访问/usr/local/elasticsearch/logs/es_cluster.log的权限。
修改文件权限就好了
# chown -R esuser:esuser/usr/local/elasticsearch/data/
# chown -R esuser:esuser/usr/local/elasticsearch/logs
3.3、kibana版本不兼容
# ./bin/kibana
{"name":"Kibana","hostname":"node3","pid":20969,"level":50,"err":{"message":"unknownerror","name":"Error","stack":"Error:unknown error\n at respond(/home/apps/kibana-4.1.4-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:237:15)\n at checkRespForFailure(/home/apps/kibana-4.1.4-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n at HttpConnector.<anonymous>(/home/apps/kibana-4.1.4-linux-x64/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n at IncomingMessage.bound(/home/apps/kibana-4.1.4-linux-x64/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n at IncomingMessage.emit(events.js:117:20)\n at_stream_readable.js:944:16\n atprocess._tickCallback (node.js:442:13)"},"msg":"","time":"2016-08-30T07:07:57.923Z","v":0}
换为更高的kibana版本就没问题了
3.4、nginx重载配置问价出错
/usr/local/nginx/sbin # ./nginx -s reload
nginx: [error] open()"/usr/local/nginx/logs/nginx.pid" failed (2: No such file ordirectory)
/usr/local/nginx/sbin # ls ../logs/
access.log error.log
/usr/local/nginx/sbin # ./nginx -c/usr/local/nginx/conf/nginx.conf
/usr/local/nginx/sbin # ls ../logs/
access.log error.log nginx.pid
3.5、启动kafka出错有一个broker没起来，提示Failed to acquire lock on file .lock in/usr/local/kafka/tmp/kafka-logs.
关闭kafka服务，查看还有哪个节点上的kafka还有进程，杀死改进城之后再次启动就没问题了。
3.6、kafka删除topic出错，删除不了（还未解决）
/usr/local/kafka # ./bin/kafka-topics.sh--delete --zookeeper 192.168.1.101:2181 --topic nginx-messages
Topic nginx-messages is marked for deletion.
Note: This will have no impact ifdelete.topic.enable is not set to true
按网上说的修改
配置文件在kafka\config目录
# vi server.properties
delete.topic.enable=true
但是修改之后还是删除不了topic。
四、 Logstash配置文件
4.1、配置一
input {
stdin {
type => "stdin-type"
}
file {
type => "syslog-ng"
#Wildcards work, here :)
path => [ "/var/log/*.log", "/var/log/messages","/var/log/syslog" ]
}
}
output {
stdout { }
elasticsearch{
hosts=>["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]
}
配置一的架构是被收集日志的机器直接通过本机的logstash把日志传送给es集群
4.2、配置二
input {
file {
type => "system-message"
path => "/var/log/messages"
start_position => "beginning"
}
}
output {
#stdout { codec => rubydebug }
kafka {
bootstrap_servers =>"192.168.1.103:9092"
topic_id => "system-messages"
compression_type => "snappy"
}
}
配置二的架构是被收集日志的机器直接通过本机的logstash把日志传送给es单个节点
4.3、配置三
input {
kafka {
zk_connect =>"192.168.1.103:9092"
topic_id => "System-Messages"
codec => plain
reset_beginning => false
consumer_threads => 5
decorate_events => true
}
}
output {
elasticsearch {
hosts => "192.168.1.103:9092"
index =>"test-System-Messages-%{+YYYY-MM}"
}
}
配置三的架构是被kafka节点收集到日志之后传送给es节点
4.4、配置四
# vi/usr/local/logstash/etc/logstash_shipper.conf
input {
file {
type => "system-message"
path => "/var/log/messages"
start_position => "beginning"
}
}
output {
stdout { codec => rubydebug }
kafka {
bootstrap_servers =>"192.168.1.101:9092,192.168.1.102:9092,192.168.1.103:9092"
topic_id => "messages"
compression_type => "snappy"
}
logstash消费端
/usr/local/logstash # vietc/logstash_indexer.conf
input {
kafka {
zk_connect => "192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181"
topic_id => "system-message"
codec => plain
reset_beginning => false
consumer_threads => 5
decorate_events => true
}
}
output {
elasticsearch {
hosts => "192.168.1.105:9200"
index =>"test-system-messages-%{+YYYY-MM}"
}
}
配置四的架构是被收集日志的机器直接通过本机的logstash把日志传送给kafka集群，之后kafka集群再把日志传送给es节点。
本文出自 “xiaoxiaozhou” 博客，请务必保留此出处http://xiaoxiaozhou.blog.51cto.com/4681537/1855188
标签：elasticsearch logstash kibana kafka
原文地址：http://xiaoxiaozhou.blog.51cto.com/4681537/1855188
