码迷,mamicode.com
首页 > 其他好文 > 详细

filebeat日志采集

时间:2019-05-23 15:46:34      阅读:137      评论:0      收藏:0      [点我收藏+]

标签:local   host   port   put   服务   sys   类型   日志文件   word   

架构一: filebeat -> logstash1 -> redis -> logstash2 -> elastash(集群) -> kibana 这里就不写安装程序的步骤了相信大家都没有难度: (软件安装可自行设计) 230,安装filebeat, logstash1 ,elastash 232,安装logstash2, redis, elastash ,kibana 注意:filebeat文件很注重文件格式 1,配置filebeat文件: [root@localhost filebeat]# cat /etc/filebeat/filebeat.yml filebeat: prospectors: # - #每个日志文件的开始 # paths: #定义路径 # - /var/www/logs/access.log #绝对路径 # input_type: log #日志类型为log # document_type: api4-nginx-accesslog # 此名称要与logstash定义的名称相对应,logstash要使用此名称做type判断使用 - paths: - /opt/apps/huhu/logs/ase.log input_type: log document_type: "ase-ase-log" encoding: utf-8 tail_files: true #每次最后一行 multiline.pattern: ‘^\[‘ #分割符 multiline.negate: true multiline.match: after #最后合并 #tags: ["ase-ase"] - paths: #收集json格式日志 - /var/log/nginx/access.log input_type: log document_type: "nginx-access-log" tail_files: true json.keys_under_root: true json.overwrite_keys: true registry_file: /var/lib/filebeat/registry output: #输出到230 logstash: hosts: ["192.168.0.230:5044"] shipper: logging: to_files: true files: path: /tmp/mybeat 2.配置230:logstash-->input-redis [root@web1 conf.d]# pwd /etc/logstash/conf.d [root@web1 conf.d]# cat nginx-ase-input.conf input { beats { port => 5044 codec => "json" }} output { if [type] == "nginx-access-log" { redis { #nginx日志写到redis信息 data_type => "list" key => "nginx-accesslog" host => "192.168.0.232" port => "6379" db => "4" password => "123456" }} if [type] == "ase-ase-log" { redis { #写到els日志写到redis信息 data_type => "list" key => "ase-log" host => "192.168.0.232" port => "6379" db => "4" password => "123456" }} } 3.redis写到elstach里,232服务器配置:logstash-->output-->resid->elstash [root@localhost conf.d]# pwd /etc/logstash/conf.d [root@localhost conf.d]# cat nginx-ase-output.conf input { redis { type => "nginx-access-log" data_type => "list" key => "nginx-accesslog" host => "192.168.0.232" port => "6379" db => "4" password => "123456" codec => "json" } redis { type => "ase-ase-log" data_type => "list" key => "ase-log" host => "192.168.0.232" port => "6379" db => "4" password => "123456" } } output { if [type] == "nginx-access-log" { elasticsearch { hosts => ["192.168.0.232:9200"] index => "nginx-accesslog-%{+YYYY.MM.dd}" }} if [type] == "ase-ase-log" { elasticsearch { hosts => ["192.168.0.232:9200"] index => "ase-log-%{+YYYY.MM.dd}" }} } 4,在232上配置elsaticsearch--->kibana 在kibana上找到ELS的索引即可。 架构二: filebeat -> redis -> logstash --> elsasctic --> kibana #缺点filebeat写进redis有限制,占时还没找到多个写入。 1.feilebeat配置: [root@localhost yes_yml]# cat filebeat.yml filebeat: prospectors: # - #每个日志文件的开始 # paths: #定义路径 # - /var/www/logs/access.log #绝对路径 # input_type: log #日志类型为log # document_type: api4-nginx-accesslog # 此名称要与logstash定义的名称相对应,logstash要使用此名称做type判断使用 - paths: - /opt/apps/qpq/logs/qpq.log input_type: log document_type: "qpq-qpq-log" encoding: utf-8 tail_files: true multiline.pattern: ‘^\[‘ multiline.negate: true multiline.match: after #tags: ["qpq-qpq-log"] registry_file: /var/lib/filebeat/registry output: redis: host: "192.168.0.232" port: 6379 db: 3 password: "123456" timeout: 5 reconnect_interval: 1 index: "pqp-pqp-log" shipper: logging: to_files: true files: path: /tmp/mybeat 2.由232redis-->els--kibana [root@localhost yes_yml]# cat systemlog.conf input { redis { type => "qpq-qpq-log" data_type => "list" key => "qpq-pqp-log" host => "192.168.0.232" port => "6379" db => "3" password => "123456" }} output { if [type] == "qpq-qpq-log"{ elasticsearch { hosts => ["192.168.0.232:9200"] index => "qpq-qpq-log-%{+YYYY.MM.dd}" } } } 3.在232上配置elsaticsearch--->kibana 在kibana上找到ELS的索引即可

filebeat日志采集

标签:local   host   port   put   服务   sys   类型   日志文件   word   

原文地址:https://blog.51cto.com/13399294/2399021

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!