码迷,mamicode.com
首页 > 其他好文 > 详细

通过filebeat的modules搜集nginx日志

时间:2020-07-10 14:57:44      阅读:123      评论:0      收藏:0      [点我收藏+]

标签:mic   启动   version   启用   source   module   index   http   hosts   

前提安装好ES和Kibana

NGINX日志格式如下

log_format main $remote_addr - $remote_user [$time_local] "$request" 
                    $status $body_bytes_sent "$http_referer" 
                    "$http_user_agent" "$http_x_forwarded_for" 
                    "$request_body" "$upstream_addr" "$uri" "$upstream_response_time" "$upstream_http_name" "$upstream_http_host" "$request_time" ;

 

1.下载并安装filebeat

rpm -ivh filebeat-7.5.1-x86_64.rpm

 

2.启用nginx modules

filebeat modules enable nginx

 

3.配置NGINX访问日志路径

/etc/filebeat/modules.d/nginx.yml 

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/*.access.log"]

 

4.配置ES地址并修改index名称

/etc/filebeat/filebeat.yml

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
setup.ilm.enabled: false
setup.template.name: "filebeat-nginx"
setup.template.pattern: "filebeat-nginx-*"

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.160.34:9200"]
  index: "filebeat-nginx-%{[agent.version]}-%{+yyyy.MM.dd}"

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "pass"

 

5.修改日志的默认匹配规则

/usr/share/filebeat/module/nginx/access/ingest/default.json

添加upstream_addr、upstream_url、upstream_response_time、request_time等字段

可以通过Grok Debugger检查匹配规则是否正确:http://grokdebug.herokuapp.com/

"grok": {
    "field": "message",
    "patterns": [
        "\"?(?:%{IP_LIST:nginx.access.remote_ip_list}|%{DATA:source.address}) - %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" \"%{URIHOST:nginx.access.upstream_addr}\" \"%{DATA:nginx.access.upstream_url}\" \"%{NUMBER:nginx.access.upstream_response_time:float}\" \"-\" \"-\" \"%{NUMBER:nginx.access.request_time:float}\""
    ],
    "pattern_definitions": {
        "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
    },
    "ignore_missing": true
}

 

6.修改fields字段

/etc/filebeat/fields.yml

添加upstream_addr、upstream_url、upstream_response_time、request_time等字段

- name: agent
  type: alias
  path: user_agent.original
  migration: true
- name: upstream_response_time
  type: alias
  path: upstream_response_time
  migration: true
- name: upstream_addr
  type: alias
  path: upstream_addr
  migration: true
- name: upstream_url
  type: alias
  path: upstream_url
  migration: true
- name: request_time
  type: alias
  path: request_time
  migration: true

 

 删除原有的pipeline,防止kibana不显示自定义的字段

技术图片

 

 

重新启动filebeat,查看自定义的字段是否出现

技术图片

 

 

字段出来以后有黄色小三角,刷新一下index 的字段缓存就好了

技术图片

 

 

 

参考:https://www.iamle.com/archives/2610.html

https://elasticsearch.cn/question/4580

https://cloud.tencent.com/developer/article/1643602

https://elasticsearch.cn/question/7894

通过filebeat的modules搜集nginx日志

标签:mic   启动   version   启用   source   module   index   http   hosts   

原文地址:https://www.cnblogs.com/wsl222000/p/13278892.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!