标签:管理 expr agent 重启 条件 layout 比较 UNC line
微信:tangy8080
电子邮箱:914661180@qq.com
更新时间:2019-06-13 11:02:14 星期四
欢迎您订阅和分享我的订阅号,订阅号内会不定期分享一些我自己学习过程中的编写的文章
如您在阅读过程中发现文章错误,可添加我的微信 tangy8080 进行反馈.感谢您的支持。
由于我们对fluentd-elasticsearch的定制比较多,所以我们选择使用源码来安装fluentd-elasticsearch
#克隆源码库
cd /usr/local/src/
git clone https://github.com/kiwigrid/helm-charts
cd helm-charts/charts/fluentd-elasticsearch/
#为所有输出添加hostname,physics.hostname,tag字段,方便查找日志
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml
#在containers.input.conf节点下添加如下配置
<filter **>
@id filter_hostname
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
physics.hostname "#{ENV['K8S_NODE_NAME']}"
tag ${tag}
</record>
</filter>
#执行安装
cd /usr/local/src/helm-charts/charts/fluentd-elasticsearch
helm install --name fluentd-elasticsearch --set elasticsearch.host=elasticsearch-client,hostLogDir.dockerContainers=/data/k8s/docker/data/containers .
helm del --purge fluentd-elasticsearch
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/values.yaml
#nginx日志不在/var/log目录下,在文件中最后添加Nginx日志文件挂载
extraVolumes:
- name: nginxlog
hostPath:
path: /usr/local/nginx/logs
extraVolumeMounts:
- name: nginxlog
mountPath: /var/log/nginx
readOnly: true
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml
# service.honeysuckle-log-consumer
<source>
@id honeysuckle-log-consumer.log
@type tail
<parse>
@type regexp
expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>.*) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>[\s\S]*)$/
time_key time
</parse>
path /var/log/businesslogs/honeysuckle-log-consumer/Log.txt
pos_file /var/log/businesslogs/honeysuckle-log-consumer/Log.txt.pos
tag service.honeysuckle-log-consumer
</source>
# service.honeysuckle-configmanager-service
<source>
@id honeysuckle-configmanager-service.log
@type tail
<parse>
@type regexp
expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>.*) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>[\s\S]*)$/
time_key time
</parse>
path /var/log/businesslogs/honeysuckle-configmanager-service/Log.txt
pos_file /var/log/businesslogs/honeysuckle-configmanager-service/Log.txt.pos
tag service.honeysuckle-configmanager-service
</source>
# Nginx Access Log Source
<source>
@id nginx.accesslog
@type tail
path /var/log/nginx/access.log
pos_file /var/log/access.log.pos
format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$/
time_format %d/%b/%Y:%H:%M:%S %z
tag nginx
</source>
该项目用于模拟一个服务,它每隔3秒向 /app/App_Data/Logs/Log.txt 写入一条日志
该项目采用Log4net写入日志,应用本身支持日志轮转,最新的日志都在Log.txt中
log4net.config的配置如下:
<?xml version="1.0" encoding="utf-8" ?>
<log4net>
<!--文本文件appender-->
<appender name="RollingFileAppender" type="log4net.Appender.RollingFileAppender" >
<file value="App_Data/Logs/Log.txt" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="1024KB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
</layout>
</appender>
<root>
<appender-ref ref="RollingFileAppender" />
<level value="ALL" />
</root>
</log4net>
日志输出的样例类似于:
2019-06-17 07:29:52,038 [5] INFO honeysuckle.log.consumer.HoneysuckleWebModule [(null)] - electronicinvoice_log_queue.BasicConsume 已创建.
2019-06-17 07:31:51,510 [WorkPool-Session#1:Connection(21956434-0138-45f3-be65-848ca544cad3,amqp://honeysuckle.site:5672)] ERROR honeysuckle.log.consumer.HoneysuckleWebModule [(null)] - Error in EventingBasicConsumer.Received
Elasticsearch.Net.UnexpectedElasticsearchClientException: The operation was canceled. ---> System.OperationCanceledException: The operation was canceled.
at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Nest.LowLevelDispatch.BulkDispatch[TResponse](IRequest`1 p, SerializableData`1 body)
at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 responseGenerator, Func`3 dispatch)
at honeysuckle.log.consumer.HoneysuckleWebModule.<>c__DisplayClass10_0.<CreateNewChannel>b__0(Object sender, BasicDeliverEventArgs e) in /src/src/honeysuckle.log.consumer/HoneysuckleWebModule.cs:line 141
基于日志输出格式,我们采用正则表达式插件进行日志解析(表达式在下面的source中可以看到)
可以使用fluentular工具进行表达式的正确性测试
该项目托管在:
http://admin@gitblit.honeysuckle.site/r/public/helloworld.git
如有测试需要.欢迎clone
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml
# service.helloworld.log Log Source
<source>
@id helloworld.log
@type tail
<parse>
@type regexp
expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>\[\d+\]) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>.*)$/
time_key time
</parse>
path /var/log/businesslogs/helloworld/Log.txt
pos_file /var/log/businesslogs/helloworld/Log.txt.pos
tag service.helloworld
</source>
待所有组件全部runing之后,可以通过curl查看fluentd是否采集到了数据
curl 'http://10.254.193.78:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2019.06.11 _v7q6b4DSt6mLJ_GLDyFUQ 5 1 4112 0 4.6mb 2.2mb
yellow open logstash-2019.06.12 mz3sF15LTbqQwZiXMubdgg 5 1 29394 0 27.9mb 15.2mb
green open logstash-2019.06.13 rtjvfGVWSM-5rLvq72jxlA 5 1 971 0 2.4mb 1.1mb
检查点一:日志是否根据tag正确分类
检查点二:日志否则附加了hostname和physics.hostname属性
随着业务的需要,我们后期可能会加入其他的服务.这个时候我们可以在配置文件(configmaps.yaml)中添加好采集规则之后,进行升级
cd /usr/local/src/helm-charts/charts/fluentd-elasticsearch
helm upgrade fluentd-elasticsearch .
rm /var/log/kernel.pos/worker0/storage.json
https://github.com/helm/charts/tree/master/stable/kibana#configuration
标签:管理 expr agent 重启 条件 layout 比较 UNC line
原文地址:https://www.cnblogs.com/gytangyao/p/11407229.html