用ElasticSearch,LogStash,Kibana搭建实时日志收集系统

介绍

• 这套系统，logstash负责收集处理日志文件内容存储到elasticsearch搜索引擎数据库中。kibana负责查询elasticsearch并在web中展示。
• logstash收集进程收获日志文件内容后，先输出到redis中缓存，另一logstash处理进程从redis中读出并转存到elasticsearch中，以解决读快写慢速度不一致问题。
• 官方在线文档：https://www.elastic.co/guide/index.html

一、安装jdk7

• ElasticSearch,LogStash均是java程序，所以需要jdk环境。

• 下载：jdk-7u71-linux-x64.rpm
  http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

• rpm -ivh jdk-7u71-linux-x64.rpm

• 配置JDK
  编辑/etc/profile文件
  在文件中找到export PATH USER LOGNAME 等内容，添加以下内容：

  JAVA_HOME=/usr/java/jdk1.7.0_71
  JRE_HOME=/usr/java/jdk1.7.0_71/jre
  PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
  CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
  export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL JAVA_HOME JRE_HOME CLASSPATH

• 检查JDK环境
  使用source /etc/profile命令，使环境变量立即生效。
  查看当前安装的JDK版本，命令：java -version
  检查环境变量，echo $PATH

二、安装elasticsearch

• elasticsearch是一个搜索引擎，负责存储日志内容。

• 下载安装
  wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.4.4.tar.gz
  tar zxvf elasticsearch-1.4.4.tar.gz

• 修改config/elasticsearch.yml配置文件

bootstrap.mlockall: true

index.number_of_shards: 1
index.number_of_replicas: 0

#index.translog.flush_threshold_ops: 100000
#index.refresh_interval: -1
index.translog.flush_threshold_ops: 5000
index.refresh_interval: 1  

# Security 允许所有http请求
http.cors.enabled: true
http.cors.allow-origin: "/.*/"    

• 修改bin/elasticsearch文件

# 使jvm使用os，max-open-files
es_parms="-Delasticsearch -Des.max-open-files=ture"

# Start up the service
# 修改OS打开最大文件数
ulimit -n 1000000
ulimit -l unlimited
launch_service "$pidfile" "$daemonized" "$properties"

• 运行
  ./bin/elasticsearch -d
  ./logs下为日志文件

• 检查节点状态
  curl -XGET ‘http://localhost:9200/_nodes?os=true&process=true&pretty=true’

  {
    "cluster_name" : "elasticsearch",
    "nodes" : {
      "7PEaZbvxToCL2O2KuMGRYQ" : {
        "name" : "Gertrude Yorkes",
        "transport_address" : "inet[/172.16.18.116:9300]",
        "host" : "casimbak",
        "ip" : "172.16.18.116",
        "version" : "1.4.4",
        "build" : "c88f77f",
        "http_address" : "inet[/172.16.18.116:9200]",
        "settings" : {
          "index": {
              "number_of_replicas": "0",
              "translog": {
                  "flush_threshold_ops": "5000"
              },
              "number_of_shards": "1",
              "refresh_interval": "1"
          },      
          "path" : {
            "logs" : "/home/jfy/soft/elasticsearch-1.4.4/logs",
            "home" : "/home/jfy/soft/elasticsearch-1.4.4"
          },
          "cluster" : {
            "name" : "elasticsearch"
          },
          "bootstrap" : {
            "mlockall" : "true"
          },
          "client" : {
            "type" : "node"
          },
          "http" : {
            "cors" : {
              "enabled" : "true",
              "allow-origin" : "/.*/"
            }
          },
          "foreground" : "yes",
          "name" : "Gertrude Yorkes",
          "max-open-files" : "ture"
        },
        "process" : {
          "refresh_interval_in_millis" : 1000,
          "id" : 13896,
          "max_file_descriptors" : 1000000,
          "mlockall" : true
        },
  
        ...
  
      }
    }
  }

• 表明ElasticSearch已运行，状态与配置相符

          "index": {
              "number_of_replicas": "0",
              "translog": {
                  "flush_threshold_ops": "5000"
              },
              "number_of_shards": "1",
              "refresh_interval": "1"
          }, 
  
        "process" : {
          "refresh_interval_in_millis" : 1000,
          "id" : 13896,
          "max_file_descriptors" : 1000000,
          "mlockall" : true
        },

• 安装head插件监控elasticsearch状态
  elasticsearch/bin/plugin -install mobz/elasticsearch-head
  http://172.16.18.116:9200/_plugin/head/

三、安装logstash

• logstash一个日志收集处理过滤程序。

• LogStash分为日志收集端进程和日志处理端进程，收集端负责收集多个日志文件实时的将日志内容输出到redis队列缓存，处理端负责将redis队列缓存中的内容输出到ElasticSarch中存储。收集端进程运行在产生日志文件的服务器上，处理端进程运行在redis,elasticsearch同一服务器上。

• 下载
  wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz

• redis安装配置略，但要注意监控redis队列长度，如果长时间堆集说明elasticsearch出问题了
  每2S检查一下redis中数据列表长度，100次
  redis-cli -r 100 -i 2 llen logstash:redis

• 配置Logstash日志收集进程
  vi ./lib/logstash/config/shipper.conf

input {
    #file {
    #    type => "mysql_log"
    #    path => "/usr/local/mysql/data/localhost.log"
    #    codec => plain{
    #        charset => "GBK"
    #    }
    #}
    file {
        type => "hostapd_log"
        path => "/root/hostapd/hostapd.log"
        sincedb_path => "/home/jfy/soft/logstash-1.4.2/sincedb_hostapd.access"
        #start_position => "beginning"
        #http://logstash.net/docs/1.4.2/codecs/plain
        codec => plain{
            charset => "GBK"
        }
    }
    file {
        type => "hkt_log"
        path => "/usr1/app/log/bsapp.tr"
        sincedb_path => "/home/jfy/soft/logstash-1.4.2/sincedb_hkt.access"
        start_position => "beginning"
        codec => plain{
            charset => "GBK"
        }
    }
#   stdin {
#       type => "hostapd_log"
#   }
}

#filter {
#    grep {
#        match => [ "@message", "mysql|GET|error" ]
#    }
#}

output {
    redis {
        host => ‘172.16.18.116‘
        data_type => ‘list‘
        key => ‘logstash:redis‘
#        codec => plain{
#            charset => "UTF-8"
#        }
    }
#    elasticsearch {
#      #embedded => true
#      host => "172.16.18.116"
#    }
}

• 运行收集端进程
  ./bin/logstash agent -f ./lib/logstash/config/shipper.conf

• 配置Logstash日志处理进程
  vi ./lib/logstash/config/indexer.conf

  input {
    redis {
      host => ‘127.0.0.1‘
      data_type => ‘list‘
      key => ‘logstash:redis‘
      #threads => 10
      #batch_count => 1000
    }
  }
  
  output {
    elasticsearch {
      #embedded => true
      host => localhost
      #workers => 10
    }
  }

• 运行处理端进程
  ./bin/logstash agent -f ./lib/logstash/config/indexer.conf
  处理端从redis读出缓存的日志内容，输出到ElasticSarch中存储

四、安装kibana

• kibana是elasticsearch搜索引擎的web展示界面，一套在webserver下的js脚本,可以定制复杂的查询过滤条件检索elasticsearch，并以多种方式（表格，图表）展示。

• 下载
  wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.tar.gz
  解压后将kibana目录放到webserver能访问到的地方

• 配置
  修改kibana/config.js:

如果kibana与elasticsearch不在同一机器则修改：
elasticsearch: "http://192.168.91.128:9200",
#这里实际上是浏览器直接访问该地址连接elasticsearch

否则默认，一定不要修改

如果出现connection failed，则修改elasticsearch/config/elasticsearch.yml，增加：

http.cors.enabled: true  
http.cors.allow-origin: "/.*/"

具体含义参见：
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html

• 访问kibana
  http://172.16.18.114:6090/kibana/index.html#/dashboard/file/logstash.json

• 配置kibana界面
  可以filtering中配置需要访问的日志type,如_type=voip_log,对应着上面logstash中shipper.conf的type
  也可以点击右上角的save将当前界面的配置保存到elasticsearch中，默认是保存在kibana-int索引中