码迷,mamicode.com
首页 > 其他好文 > 详细

统一日志ELK部署配置(2)——kafka

时间:2018-06-13 17:21:28      阅读:206      评论:0      收藏:0      [点我收藏+]

标签:日志收集

前提:
你服务器上已经安装并配置了java运行环境;

一、zookeeper安装
1、从zookeeper官网:http://zookeeper.apache.org/ 下载;我这里下载的是zookeeper-3.4.9.tar.gz;
2、解压到你安装目录:tar-zxvf zookeeper-3.4.9.tar.gz

二、zookeeper配置
1、修改zoo.cfg(直接复制zoo_sample.cfg);
2、修改配置参数(集群模式):
tickTime=2000
dataDir=/mnt/zookeeper-3.4.9/data/
clientPort=2181
initLimit=5
syncLimit=2
server.1=broker1:2888:3888
server.2=broker2:2888:3888
server.3=broker3:2888:3888

3、修改zkEnv.sh,指定配置文件(这里为conf/zoo.cfg)
4、启动:
    bin/zkServer.sh start  启动成功

三、kafka安装
1、从官网下载你需要的kafka:http://kafka.apache.org/downloads
2、接下到你安装目录:tar -zxvf kafka_2.11-1.1.0.tgz
四、kafka配置:
broker.id=1
delete.topic.enable=true

############################# Socket Server Settings #############################
listeners=PLAINTEXT://kafka1_ip:9092
#advertised.listeners=PLAINTEXT://your.host.name:9092
port=9092
host.name=kafka1_ip

num.network.threads=4

#= The number of threads doing disk I/O
num.io.threads=8

#= The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

#= The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

#= The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600

############################# Log Basics #############################
#= A comma seperated list of directories under which to store log files
log.dirs=/mnt/kafka_2.12-0.10.2.1/kafka-logs

num.partitions=3

#= The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
#= This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Log Flush Policy #############################

#= The number of messages to accept before forcing a flush of data to disk
log.flush.interval.messages=1000

#= The maximum amount of time a message can sit in a log before we force a flush
log.flush.interval.ms=1000

############################# Log Retention Policy #############################

#= The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=24

#= The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

#= The interval at which log segments are checked to see if they can be deleted according
#= to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################
zookeeper.connect=zk_ip1:2181,zk_ip2:2181,zk_ip3:2181

#= Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

max.message.bytes=52428700
replica.fetch.max.bytes=52428700

五、启动kafka:
bin/kafka-server-start.sh config/server.properties &

统一日志ELK部署配置(2)——kafka

标签:日志收集

原文地址:http://blog.51cto.com/jfzhang/2128967

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!