CentOS 7环境下Kafka的安装和基本使用
基础环境
Windows 10 X64
VMware-workstation-full-12.0.0-2985596
CentOS Linux release 7.4.1708
CentOS 7环境下安装Kafka
安装JDK
- 在CentOS上将JDK下载到本地.jdk-8u152-linux-x64.tar.gz
- 解压并将文件移动到
/opt/java目录下
sudo tar -vxzf jdk-8u152-linux-x64.tar.gz
sudo mv jdk1.8.0_152 /opt/java
- 配置java环境变量
vim /etc/profile
#在profile文件的最后面添加
export JAVA_HOME=/opt/jdk1.8.0_152
export PATH=$JAVA_HOME/bin:$PATH
#添加成功后通过java -version来查看jdk是否安装成功
java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
安装Zookeeper
- 通过wget下载zookeeper到本机
sudo wdget https://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz
- 解压并将文件移动到
/opt/zookeeper目录下
# 解压zookeeper
sudo tar -zxvf zookeeper-3.3.6.tar.gz
# 将zookeeper移动到/opt/zookeeper目录下
sudo mv zookeeper-3.3.6 /opt/zookeeper
- 编辑zookeeper的配置
# 复制一份zoo_sample.cfg文件并改名为zoo.cfg
sudo cp /opt/zookeeper/zoo_sample.cfg zoo.cfg
# 编辑zoo.cfg 文件
sudo vim /opt/zookeeper/zoo.cfg
#主要修改dataDir和server.1=127.0.0.1:2888:3888这2处
# the directory where the snapshot is stored.
dataDir=/opt/zookeeper/data
# the port at which the clients will connect
clientPort=2181
server.1=127.0.0.1:2888:3888
- 配置zookeeper环境变量
sudo vim /etc/profile
#添加如下内容
ZOOKEEPER_HOME=/opt/zookeeper
PATH=$PATH:ZOOKEEPER_HOME/bin
export ZOOKEEPER_HOME
- 启动zookeeper
zkServer.sh start
# 如果上面命令报错,则cd /opt/zookeeper/bin 目录
./zkServer.sh start
安装kafka
- 通过wget下载kafka到本机
sudo wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/1.0.0/kafka_2.12-1.0.0.tgz
- 解压并将文件移动到
/opt/kafka目录下
# 解压
sudo tar -zxvf kafka_2.12-1.0.0.tgz
# 移动
sudo mv kafka_2.12-1 /opt/kafka
- 编辑kafka的配置
#创建日志存放目录
cd /opt/kafka
mkdir -p log/kafka
#修改配置文件/opt/kafka/config/server.properties
sudo vim /opt/kafka/config/server.properties
#主要修改下面几项内容如下：
broker.id=181
delete.topic.enable=true
listeners = PLAINTEXT://127.0.0.1:9092
log.dirs=/opt/kafka/log/kafka
zookeeper.connect=127.0.0.1:2181
- 配置kafka环境变量
sudo vim /etc/profile
#添加如下内容：
KAFKA_HOME=/opt/kafka
PATH=$PATH:$KAFKA_HOME/bin
export PATH KAFKA_HOME
- 启动kafka
kafka-server-start.sh /opt/kafka/config/server.properties
CentOS 7环境下Kafka的基本使用
- 创建一个Topic (test)
kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test
- Producer向Topic发送消息
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
[yang@localhost bin]$ ./kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>producer send one message
>producer send two message
>
- Consumer读取Topic的消息
kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic test --from-beginning
[yang@localhost bin]$ ./kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 -topic test --from-beginning
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
[2017-11-19 20:32:13,047] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,085] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,100] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,265] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
producer send one message
producer send two message
