码迷,mamicode.com
首页 > Web开发 > 详细

07_Flume_regex interceptor实践

时间:2017-12-03 18:09:32      阅读:314      评论:0      收藏:0      [点我收藏+]

标签:alt   验证   exclude   观察   action   bin   hand   art   分组   

 实践一:regex filter interceptor

1、目标场景

技术分享图片

regex filter interceptor的作用:

1)将event body的内容和配置中指定的正则表达式进行匹配
2)如果内容匹配,则将该event丢弃
3)如果内容不匹配,则将该event放行

 

2、Flume Agent配置文件

# 01 define agent name, source/sink/channel 
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# 02 source,http,jsonhandler
a1.sources.r1.type = http
a1.sources.r1.bind = master
a1.sources.r1.port = 6666
a1.sources.r1.handler = org.apache.flume.source.http.JSONHandler

# 03 regex filter interceptor, match event body for filter
a1.sources.r1.interceptors = i1  
a1.sources.r1.interceptors.i1.type = regex_filter  
a1.sources.r1.interceptors.i1.regex = ^[0-9]*$ 
# filter matched event 
a1.sources.r1.interceptors.i1.excludeEvents = true  

# 04 logger sink
a1.sinks.k1.type = logger

# 05 channel,memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 06 bind source,sink to channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

3、验证regex filter interceptor

1) 通过curl -X POST -d ‘json数据‘ 发送带有不同body的HTTP请求,其中有1个满足regex

技术分享图片

 

2)观察终端打印出的event,body为1234的event被过滤, 并没有出现

技术分享图片

 

 4、regex filter interceptor的官方文档

技术分享图片

 

 

实践二:regex extractor interceptor

1、目标场景

技术分享图片

regex extractor interceptor的作用:
1)将event body的内容和配置中指定的正则表达式进行匹配
2)如果内容匹配,将配合配置文件中给定的key, 组成key:value添加到event的header中
3)event body中的内容不会变化

 

2、Flume Agent的配置文件

# 01 define agent name, source/sink/channel 
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# 02 source,http,jsonhandler
a1.sources.r1.type = http
a1.sources.r1.bind = master
a1.sources.r1.port = 6666
a1.sources.r1.handler = org.apache.flume.source.http.JSONHandler

# 03 regex extractor interceptor,match event body to extract character and digital
a1.sources.r1.interceptors = i1  
a1.sources.r1.interceptors.i1.type = regex_extractor
a1.sources.r1.interceptors.i1.regex = (^[a-zA-Z]*)\\s([0-9]*$)  # regex匹配并进行分组,匹配结果将有两个部分, 注意\s空白字符要进行转义
# specify key for 2 matched part
a1.sources.r1.interceptors.i1.serializers = s1 s2
# key name
a1.sources.r1.interceptors.i1.serializers.s1.name = word
a1.sources.r1.interceptors.i1.serializers.s2.name = digital 

# 04 logger sink
a1.sinks.k1.type = logger

# 05 channel,memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 06 bind source,sink to channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

3、验证regex extractor interceptor

1) 通过curl -X POST -d ‘json数据‘的方式发送HTTP请求,body中的内容为"shayzhang 1234", 其中shayzhang,1234将被正则表达式匹配

技术分享图片

 

2) 观察logger打印到终端的event,header中将增加两部分 word:shayzhang, digital:1234

技术分享图片

 

07_Flume_regex interceptor实践

标签:alt   验证   exclude   观察   action   bin   hand   art   分组   

原文地址:http://www.cnblogs.com/shay-zhangjin/p/7966452.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!