标签:get tar ons col xxxxx 保持数据 href 字段 access
S3日志拉取<source>
@type s3
aws_key_id XXXXXXXXXXXXXXXXXXXX
aws_sec_key XXXXXXXXXXXXXXXXXXXX
s3_bucket nx-rc-rancher-newelb-inner
s3_region cn-northwest-1
tagelb-inner
<sqs>
queue_name fluentd-s3 #这里只需要配置SQS的名字
</sqs>
</source>
<source>
@type s3
aws_key_id XXXXXXXXXXXXXXXXXXXX
aws_sec_key XXXXXXXXXXXXXXXXXXXX
s3_bucket nx-rc-rancher-newelb-com
s3_region cn-northwest-1
<sqs>
queue_name fluentd-s3
</sqs>
</source>
<source>
@type s3
aws_key_id XXXXXXXXXXXXXXXXXXXX
aws_sec_key XXXXXXXXXXXXXXXXXXXX
s3_bucket nx-rc-www-newelb-com
s3_region cn-northwest-1
<sqs>
queue_name fluentd-s3
</sqs>
</source>
因为fluentd是通过流的形式进行信息的过滤和处理,而且没有if语句,只能通过重写tag来筛选不同的日志
官方文档
https://github.com/fluent/fluent-plugin-rewrite-tag-filter
<match elb-inner>
@type rewrite_tag_filter
<rule>
key message
pattern /.yufuid.net:80/
tag elb-inner.net #匹配的就重写标签
</rule>
</match>
<filter elb-inner.net>
@type parser #对日志就行分割和命名处理,fluentd的正则表达式可能需要自己写
key_name message
reserve_data yes
<parse>
@type regexp
expression /(?<elb_http_method>[^ ]+) (?<access_timestamp>[^ ]+) (?<elb_name>[^ ]+) (?<client_ip>[^ ]+):(?<client_Port>[^ ]+) (?<target_ip_port>[^ ]+) (?<request_processing_time>[^ ]+) (?<target_processing_time>[^ ]+) (?<response_processing_time>[^ ]+) (?<elb_status_code>[^ ]+) (?<target_status_code>[^ ]+) (?<received_bytes>[^ ]+) (?<send_bytes>[^ ]+) (?<request>"[^\"]+") (?<client_info>"[^"]+") (?<ssl_cipher>[^ ]+) (?<ssl_protocol>[^ ]+) (?<target_group_arn>[^ ]+) (?<trace_ip>[^ ]+) (?<domainname>[^ ]+) (?<chose_cert_arn>[^ ]+) (?<matched_rule_priority>[^ ]+) (?<elb_name>[^ ]+) (?<request_creation_time>[^ ]+)/
</parse>
</filter>
<filter elb-com>
@type parser
key_name message
reserve_data yes
<parse>
@type regexp
expression /(?<elb_http_method>[^ ]+) (?<access_timestamp>[^ ]+) (?<elb_name>[^ ]+) (?<client_ip>[^ ]+):(?<client_Port>[^ ]+) (?<target_ip_port>[^ ]+) (?<request_processing_time>[^ ]+) (?<target_processing_time>[^ ]+) (?<response_processing_time>[^ ]+) (?<elb_status_code>[^ ]+) (?<target_status_code>[^ ]+) (?<received_bytes>[^ ]+) (?<send_bytes>[^ ]+) (?<request>"[^\"]+") (?<client_info>"[^"]+") (?<ssl_cipher>[^ ]+) (?<ssl_protocol>[^ ]+) (?<target_group_arn>[^ ]+) (?<trace_ip>[^ ]+) (?<domainname>[^ ]+) (?<chose_cert_arn>[^ ]+) (?<matched_rule_priority>[^ ]+) (?<elb_name>[^ ]+) (?<request_creation_time>[^ ]+)/
</parse>
</filter>
<filter elb-www>
@type parser
key_name message
reserve_data yes
<parse>
@type regexp
expression /(?<elb_http_method>[^ ]+) (?<access_timestamp>[^ ]+) (?<elb_name>[^ ]+) (?<client_ip>[^ ]+):(?<client_Port>[^ ]+) (?<target_ip_port>[^ ]+) (?<request_processing_time>[^ ]+) (?<target_processing_time>[^ ]+) (?<response_processing_time>[^ ]+) (?<elb_status_code>[^ ]+) (?<target_status_code>[^ ]+) (?<received_bytes>[^ ]+) (?<send_bytes>[^ ]+) (?<request>"[^\"]+") (?<client_info>"[^"]+") (?<ssl_cipher>[^ ]+) (?<ssl_protocol>[^ ]+) (?<target_group_arn>[^ ]+) (?<trace_ip>[^ ]+) (?<domainname>[^ ]+) (?<chose_cert_arn>[^ ]+) (?<matched_rule_priority>[^ ]+) (?<elb_name>[^ ]+) (?<request_creation_time>[^ ]+)/
</parse>
</filter>
<match elb-www>
@type elasticsearch
host elasticsearchlog-lb.elasticsearch-log
index_name fluentd-www-elb
type_name fluentd-www-elb
flush_interval 2s #这里的刷新是把buffer的数据及时发送到ES,保持数据的实时性
include_timestamp true #这里需配置时间字段,便于kibana的时间跟踪
ssl_verify false
</match>
<match elb-com>
@type elasticsearch
host elasticsearchlog-lb.elasticsearch-log
index_name fluentd-elb
type_name fluentd-elb
flush_interval 2s
include_timestamp true
ssl_verify false
</match>
<match elb-inner.net>
@type elasticsearch
host elasticsearchlog-lb.elasticsearch-log
index_name fluentd-elb
type_name fluentd-elb
flush_interval 2s
include_timestamp true
ssl_verify false
</match>
标签:get tar ons col xxxxx 保持数据 href 字段 access
原文地址:http://blog.51cto.com/11078047/2316910