码迷,mamicode.com
首页 > 其他好文 > 详细

使用kafka作为生产者生产数据到hdfs(单节点)

时间:2018-05-29 00:28:17      阅读:132      评论:0      收藏:0      [点我收藏+]

标签:write   group   path   ann   top   .data   使用   channels   use   

关键:查看kafka官网的userguide

agent.sources = kafkaSource
agent.channels = memoryChannel
agent.sinks = hdfsSink

agent.sources.kafkaSource.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.kafkaSource.zookeeperConnect = 192.168.57.11:2181
agent.sources.kafkaSource.topic = test_pan
agent.sources.kafkaSource.groupId = test-consumer-group
agent.sources.kafkaSource.kafka.consumer.timeout.ms = 100

agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity=100
agent.channels.memoryChannel.transactionCapacity=100

agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.hdfs.path = hdfs://beicai/test/pan
agent.sinks.hdfsSink.hdfs.writeFormat = Text
agent.sinks.hdfsSink.hdfs.fileType = DataStream


agent.sinks.hdfsSink.hdfs.rollSize = 1024
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.rollInterval = 60

agent.sinks.hdfsSink.hdfs.filePrefix=test
agent.sinks.hdfsSink.hdfs.fileSuffix=.data

agent.sinks.hdfsSink.hdfs.inUserPrefix=_
agent.sinks.hdfsSink.hdfs.inUserSuffix=
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.writeFormat = TEXT
agent.sinks.hdfsSink.hdfs.rollInterval = 1
agent.sinks.sink1.hdfs.filePrefix =A

agent.sources.kafkaSource.channels = memoryChannel
agent.sinks.hdfsSink.channel = memoryChannel

使用kafka作为生产者生产数据到hdfs(单节点)

标签:write   group   path   ann   top   .data   使用   channels   use   

原文地址:https://www.cnblogs.com/pingzizhuanshu/p/9102602.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!