flume+kafka+hdfs详解

时间：2017-11-03 18:47:29 阅读：294 评论：0 收藏：0 [点我收藏+]

flume架构图

单节点flume配置

flume-1.4.0 启动flume

bin/flume-ng agent --conf ./conf  -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console -n agent

-n表示配置文件中agent的名字

[java] view plain copy

flume-1.4.0 + kafka-0.7.2+hdfs flume配置

[xml] view plain copy

agent.sources = r1
agent.sinks = s_kafka s_hdfs
agent.channels = c_kafka c_hdfs
agent.sources.r1.channels = c_kafka c_hdfs
agent.sources.r1.type = exec
#下面这个脚本tail某个日志
agent.sources.r1.command = tail -F /home/flume/loginfo
agent.channels.c_kafka.type = memory
agent.channels.c_hdfs.type = memory
agent.sinks.s_kafka.type = com.sink.FirstkafkaSink
agent.sinks.s_kafka.channel = c_kafka
#kafka需要连接zk，写入broker数据
agent.sinks.s_kafka.zkconnect = localhost:2181
agent.sinks.s_kafka.topic = test
agent.sinks.s_kafka.serializer.class = kafka.serializer.StringEncoder
agent.sinks.s_kafka.metadata.broker.list = localhost:9092 #配置文件server.properties
agent.sinks.s_kafka.custom.encoding = UTF-8
agent.sinks.s_hdfs.type = hdfs
agent.sinks.s_hdfs.channel = c_hdfs
#默认端口8020
agent.sinks.s_hdfs.hdfs.path = hdfs://localhost:9000/root/source
agent.sinks.s_hdfs.hdfs.filePrefix = events-
agent.sinks.s_hdfs.hdfs.fileType = DataStream
agent.sinks.s_hdfs.hdfs.writeFormat = Text
agent.sinks.s_hdfs.hdfs.rollCount = 30 #达到某一数值记录生成文件
agent.sinks.s_hdfs.hdfs.rollSize = 0
agent.sinks.s_hdfs.hdfs.rollInterval = 0
agent.sinks.s_hdfs.hdfs.useLocalTimeStamp = true
agent.sinks.s_hdfs.hdfs.idleTimeout = 51
agent.sinks.s_hdfs.hdfs.threadsPoolSize = 2

Flume内置channel,source,sink汇总

原文地址：http://www.cnblogs.com/zzmmyy/p/7779050.html

踩

(0)

评论一句话评论（0）

分享档案

更多>

周排行