码迷,mamicode.com
首页 > 其他好文 > 详细

flume 日志导入elasticsearch

时间:2014-10-09 17:03:48      阅读:290      评论:0      收藏:0      [点我收藏+]

标签:flume   日志导入   elasticsearch   

Flume配置

flume生成的数据结构

<span style="font-size:18px;">"_index" : "logstash-2013.01.07",
"_type" : "tms_jboss_syslog",
"_id" : "a_M9X_0YSpmE7A_bEzIFiw",
"_score" : 1.0, "_source" : {"@source":"file://localhost.localdomain/tmp/logstash_test.log","@tags":[],"@fields":{},"@timestamp":"2013-01-07T10:53:50.941Z","@source_host":"localhost.localdomain","@source_path":"/tmp/logstash_test.log","@message":"[2013-01-05 11:02:19,969] packBoxNumber eq 00004000000044043412 createdOffice eq VIP_BJ:;null","@type":"tms_jboss_syslog"}</span>

flume配置文件

agent.sources = tail

agent.channels = memoryChannel

agent.channels.memoryChannel.type = memory

agent.sources.tail.channels = memoryChannel

agent.sources.tail.type = exec

agent.sources.tail.command = tail -F /home/hadoop/flume/conf/es_log/es_log.log

agent.sources.tail.interceptors=i1 i2 i3

agent.sources.tail.interceptors.i1.type=regex_extractor

agent.sources.tail.interceptors.i1.regex = (\\w.*):(\\w.*):(\\w.*)\\s

agent.sources.tail.interceptors.i1.serializers = s1 s2 s3

agent.sources.tail.interceptors.i1.serializers.s1.name = source

agent.sources.tail.interceptors.i1.serializers.s2.name = type

agent.sources.tail.interceptors.i1.serializers.s3.name = src_path

agent.sources.tail.interceptors.i2.type=org.apache.flume.interceptor.TimestampInterceptor$Builder

agent.sources.tail.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builder

agent.sources.tail.interceptors.i3.hostHeader = host

agent.sinks = elasticsearch

agent.sinks.elasticsearch.channel = memoryChannel

agent.sinks.elasticsearch.type=org.apache.flume.sink.elasticsearch.ElasticSearchSink

agent.sinks.elasticsearch.batchSize=100

agent.sinks.elasticsearch.hostNames=127.0.0.1:9300
agent.sinks.k1.indexType = bar_type
agent.sinks.elasticsearch.indexName=logstash
agent.sinks.elasticsearch.clusterName=elasticsearch
agent.sinks.elasticsearch.serializer=org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer
启动:

 ../bin/flume-ng agent -c . -f es_log.conf -n agent  -Dflume.root.logger=INFO,console

测试数据

website:weblog:login_page weblog data1
website:weblog:profile_page weblog data2
website:weblog:transaction_page weblog data3
website:weblog:docs_page weblog data4
syslog:syslog:sysloggroup syslog data1
syslog:syslog:sysloggroup syslog data2
syslog:syslog:sysloggroup syslog data3
syslog:syslog:sysloggroup syslog data4
syslog:syslog:sysloggroup syslog data5
syslog:syslog:sysloggroup syslog data6
bubuko.com,布布扣

之后就可以在es集群上看到通过flume导入的数据了

这时候编辑log文件时候会被flume读入es集群中并实时生成索引  如下图所示:

bubuko.com,布布扣

这样就可以通过es对日志的实时检索了

flume 日志导入elasticsearch

标签:flume   日志导入   elasticsearch   

原文地址:http://blog.csdn.net/sunflower_cao/article/details/39929931

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!