标签:nofile iba nes ebe 文件 uid document guide glibc
Elasticstack官网:https://www.elastic.co 本文档仅限搭建过程参考,使用相关的文档,不在本文档讨论范围之内。 一切依据的核心即是Elasticstack官网。 查看支持的操作系统: Elasticstack各版本软件支持的系统等:https://www.elastic.co/support/matrix#show_os 必备软件: Elasticsearch:elasticsearch-5.3.0.tar.gz Logstash:logstash-5.3.0.tar.gz Kibana:kibana-5.3.0-linux-x86_64.tar.gz Beats:filebeat-5.3.0-linux-x86_64.tar.gz X-pack:x-pack-5.3.0.zip jdk1.8:jdk-8u121-linux-i586.tar 搭建流程: Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html 修改时区(如必要): cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime 安装JDK1.8,先安装yum install -y glibc.i686 root权限下 sysctl -w vm.max_map_count=262144 /etc/security/limits.conf 增加 * soft nproc 2048 (第一列为Linux账户名) * hard nproc 4096(第一列为Linux账户) * soft nofile 65536 * hard nofile 131072 /etc/security/limits.d/90-nproc.conf 修改 * soft nproc 2048 sudo sysctl -p 生产环境需要的配置 Elasticsearch通过name判断所属的集群 ● path.data and path.logs ○ 不要和安装目录放一起,防止升级的时候数据丢失 ○ 可以配置多个目录 ● cluster.name ○ 各节点通过cluster.name加入集群 ● node.name ○ 保证重启机器名称不变 ○ 可以设置成机器名例如node.name:${HOSTNAME} ● bootstrap.memory_lock ○ bootstrap.memory_lock: true(遇到问题,见博客园博客) ○ /etc/sysctl.conf 增加vm.swappiness=0 ● network.host ○ 一旦配置标志进入生产环境 ● discovery.zen.ping.unicast.hosts ○ 自动入集群 ● discovery.zen.minimum_master_nodes ○ 避免脑裂 生产环境需要设置bootstrap.memory_lock: true 节点如下配置,承担的角色为负载均衡 node.master: false node.data: false node.ingest: false 配置文件elasticsearch.yml参考: cluster.name: rokid-test node.name: node-41 node.master: true node.data: true path.data: /home/zhangzhenghai/elk/data/elasticsearch/data path.logs: /home/zhangzhenghai/elk/data/elasticsearch/logs bootstrap.memory_lock: true bootstrap.system_call_filter: false network.host: test41 http.port: 9200 discovery.zen.ping.unicast.hosts: - test41 - test42 - test43 discovery.zen.minimum_master_nodes: 2 action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history* 启动方式:bin/elasticsearch -d -p pid 停止方式:kill `cat pid` 重启方式:kill `cat pid` bin/elasticsearch -d -p pid Logstash: 参考https://www.elastic.co/guide/en/logstash/current/index.html grok正则表达式参考:https://grokdebug.herokuapp.com/ 配置文件: input { beats { port => "5043" } } filter { grok { match => { "message" => "%{HOSTNAME:hostabc} %{DAY:zhouji} %{WORD:month} %{MONTHDAY:jihao} %{TIME:shijian} %{TZ:biaozhun} %{YEAR:nian} %{TIMESTAMP_ISO8601:shijianquan} \[%{WORD:zhonglei}\] %{WORD:caozuo} %{NOTSPACE:info}" } } } output { file { codec => line {format => "%{message}"} path => "/home/zhangzhenghai/elk/data/logstash/%{hostabc}.log" } elasticsearch { hosts => ["test41:9200","test42:9200","test43:9200"] index => "%{hostabc}" document_type => "%{hostabc}" flush_size => 100 idle_flush_time => 10 user => "elastic" password => "baoshan" } } 上面的正则表达式对应的原始日志参考: test-41.dev.rokid-inc.com Tue Apr 25 14:54:36 CST 2017 2017-04-16 23:37:44,282 [DEBUG] add service:com.rokid.open.nlp.facade.NLPService 启动方式(自动加载配置文件,修改配置文件不用重启Logstash): bin/logstash -f config/firtst.conf --config.reload.automatic Kibana: 配置文件参考: server.port: 5601 server.host: "test43" elasticsearch.url: "http://test43:9200" elasticsearch.preserveHost: true kibana.index: ".kibana" kibana.defaultAppId: "discover" elasticsearch.username: "elastic" elasticsearch.password: "xxx" 启动方式: bin/kibana X-Pack bin/elasticsearch-plugin install x-pack 或者线下安装方式 bin/elasticsearch-plugin install file:///path/to/file/x-pack-5.3.1.zip bin/kibana-plugin install file:///path/to/file/x-pack-5.3.1.zip 查看是否安装成功: bin/kibana-plugin list bin/elasticsearch-plugin list (如是elastic集群,需要将所有节点安装x-pack后重启,默认用户名密码elastic/changeme才生效) elasticsearch.yml配置如下信息 action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history* 卸载x-pack bin/elasticsearch-plugin remove x-pack Beats(可直接写入Elasticseatch): filebeat-5.3.0-linux-x86_64.tar.gz 配置文件参考: filebeat.prospectors: - input_type: log paths: - /home/zhangzhenghai/elk/data/filebeat/testdate.log document_type: typetest42 output.elasticsearch: hosts: ["test41:9200","test42:9200","test43:9200"] index: "indextest42" username: "elastic" password: "xxx" output.logstash: hosts: ["test39:5043"] 启动方式: ./filebeat -e -c filebeat.yml -d "publish" 可能短期内将会部署到线上,随着部署的进行,逐步更新完善本文档。 (发现每当需要文档的时候,经常嫌弃写得少;当需要自己写文档的时候,总感觉没得写?OMG)
如更新忘记更新此文档,请参考链接:
http://note.youdao.com/noteshare?id=d9b4d5c0f5991c63c5b8ae965722f619
标签:nofile iba nes ebe 文件 uid document guide glibc
原文地址:http://www.cnblogs.com/zhzhang/p/6762239.html