kafka的三种部署模式

时间：2014-11-18 11:48:28 阅读：363 评论：0 收藏：0 [点我收藏+]

/*************
*kafka 0.8.1.1的安装部署
*blog:www.r66r.net
*qq:26571864

**************/

相关部署视频地址：http://edu.51cto.com/course/course_id-2374.html

bubuko.com,布布扣

kafka的部署模式为3种模式
1)单broker模式

2)单机多broker模式 (伪集群)

3)多机多broker模式 (真正的集群模式)

第一种模式安装

1.在hadoopdn2机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面

2.修改 /opt/hadoop/kafka/kafka_2.9.2-0.8.1.1/config 下面的server.properties 配置文件
broker.id=0 默认不用修改
修改
log.dirs=/opt/hadoop/kafka/kafka-logs
log.flush.interval.messages=10000 默认不用修改
log.flush.interval.ms=1000        默认不用修改
zookeeper.connect=hadoopdn2:2181

3.启动kafka的broker

> bin/kafka-server-start.sh config/server.properties

正常启动如下：
[2014-11-18 10:36:32,196] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.version=2.6.32-220.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.dir=/opt/hadoop/kafka/kafka_2.9.2-0.8.1.1 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,197] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@c2f8b5a (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,231] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,238] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,262] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x349c07dcd7a0002, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,266] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-11-18 10:36:32,415] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)
[2014-11-18 10:36:32,422] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-11-18 10:36:32,502] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2014-11-18 10:36:32,503] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)
[2014-11-18 10:36:32,634] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2014-11-18 10:36:32,716] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2014-11-18 10:36:32,887] INFO Registered broker 0 at path /brokers/ids/0 with address JobTracker:9092. (kafka.utils.ZkUtils$)
[2014-11-18 10:36:32,941] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2014-11-18 10:36:33,034] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

4.创建topics
> bin/kafka-topics.sh --create --zookeeper hadoopdn2:2181 --replication-factor 1 --partitions 1 --topic test

查看队列列表
> bin/kafka-topics.sh --list --zookeeper hadoopdn2:2181

查看队列明细
> bin/kafka-topics.sh --describe --zookeeper hadoopdn2:2181 --topic test

Topic[队列]:test   PartitionCount[分区数量]:1   ReplicationFactor:1   Configs:
   Topic: test   Partition: 0   Leader: 0   Replicas: 0   Isr: 0
第一行是所有Partition的总结。后面的行是每个partition一行。

查看帮助文档
> bin/kafka-topics.sh --help 查看与topics相关的指令

第二种模式部署：

1.为第二个broker创建server的配置文件
> cp server.properties server1.properties

2.修改server1.properties

broker.id=1
port=9093
log.dirs=/opt/hadoop/kafka/kafka-logs-server1
zookeeper.connect=hadoopdn2:2181

3.启动kafka的broker

> nohup bin/kafka-server-start.sh config/server1.properties &

4.通过zookeeper的客户端可以查看当前的broker

[zk: hadoopdn2:2181(CONNECTED) 7] ls /
[zookeeper, admin, consumers, config, controller, brokers, controller_epoch]
[zk: hadoopdn2:2181(CONNECTED) 8] ls /brokers
[topics, ids]
[zk: hadoopdn2:2181(CONNECTED) 9] ls /brokers/ids
[1, 0]

5.查看队列情况

$ bin/kafka-topics.sh --describe test --zookeeper hadoopdn2:2181
Topic:test   PartitionCount:1   ReplicationFactor:1   Configs:
   Topic: test   Partition: 0   Leader: 0   Replicas: 0   Isr: 0

6.修改test队列的参数
$ bin/kafka-topics.sh   --zookeeper hadoopdn2:2181 --partitions 3 --topic test --alter

$ bin/kafka-topics.sh --describe test --zookeeper hadoopdn2:2181
Topic:test   PartitionCount:3   ReplicationFactor:1   Configs:
   Topic: test   Partition: 0   Leader: 0   Replicas: 0[在broker0上面]   Isr: 0
   Topic: test   Partition: 1   Leader: 1   Replicas: 1[在broker1上面]   Isr: 1
   Topic: test   Partition: 2   Leader: 0   Replicas: 0[在broker0上面]   Isr: 0


第三种部署方式：

1.在hadoopdn3机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面

2.修改 /opt/hadoop/kafka/kafka_2.9.2-0.8.1.1/config 下面的server.properties 配置文件
broker.id=2 必须修改保证每个broker的ID唯一
修改
log.dirs=/opt/hadoop/kafka/kafka-logs
log.flush.interval.messages=10000 默认不用修改
log.flush.interval.ms=1000        默认不用修改
zookeeper.connect=hadoopdn2:2181

3.通过zookeeper的客户端查看

[zk: hadoopdn2:2181(CONNECTED) 10] ls /brokers/ids
[2, 1, 0]

broker的id为2的已经注册到zookeeper上面了

到此为止，kafka的部署模式已经完整。

kafka的三种部署模式

标签：zookeeper

原文地址：http://blog.csdn.net/0flying0/article/details/41244597

踩

(0)

评论一句话评论（0）

分享档案

更多>

2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)

周排行