标签：logstash elasticsearch kibana log
Logstash+Elasticsearch+Kibana+S3+Nginx
build our private log query system
vi /etc/elasticsearch/elasticsearch.ymlAdd the following line somewhere in the file, to disable dynamic scripts script.disable_dynamic: true
You will also want to restrict outside access to your Elasticsearch instance, so outsiders can‘t read your data or shutdown your Elasticseach cluster through the HTTP API. Find the line that specifies network.host and uncomment it so it looks like this:
network.host: localhost
Then disable multicast by finding the discovery.zen.ping.multicast.enabled item and uncommenting so it looks like this: discovery.zen.ping.multicast.enabled: false
chkconfig --add elasticsearch
service elasticsearch restart
cd ~; curl -O https://download.elasticsearch.org/kibana/kibana/kibana-3.0.1.tar.gztar xvf kibana-3.0.1.tar.gzvi ~/kibana-3.0.1/config.js ( replace the port number (9200 by default) with 80 ) elasticsearch: "http://"+window.location.hostname+":80"
mkdir -p /usr/share/nginx/kibana3
cp -R ~/kibana-3.0.1/* /usr/share/nginx/kibana3/
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum -y install nginx
cd ~; curl -OL https://github.com/elasticsearch/kibana/raw/master/sample/nginx.conf
vi nginx.conf ( Find and change the values of the server_name to your FQDN (or localhost if you aren‘t using a domain name) and root to where we installed Kibana, so they look like the following entries )
server_name FQDN;
root /usr/share/nginx/kibana3;
cp ~/nginx.conf /etc/nginx/conf.d/default.conf
htpasswd -c /etc/nginx/conf.d/kibana.myhost.org.htpasswd linuxblind
Password: linuxblind
chkconfig --levels 235 nginx on
service nginx restart
vi /etc/yum.repos.d/logstash.repo (Add the following repository configuration)
[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
yum -y install logstash-1.4.2
vi /etc/logstash/conf.d/logstash.conf ( see below)
service logstash start
vi /etc/yum.repos.d/logstash.repo (Add the following repository configuration)
yum -y install java-1.7.0-openjdk
cd /etc/pki/tls; openssl req -new -newkey rsa:4096 -days 3650 -nodes -x509 -subj "/C=CN/ST=Beijing/L=Beijing/O=IT/CN=logstash.linuxblind.com" -keyout logstash.linuxblind.com.key -out logstash.linuxblind.com.certLogstash agent
[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
yum -y install logstash-1.4.2
service logstash start
=============== Logstash Collect maillog (sendmail) =====================
Step1:
Copy SSL certificate from logstash-indexer to logstash-agent
scp /etc/pki/tls/certs/logstash.linuxblind.com.cert user@logstash_agent_IP:/tmp
Step2:
In logstash-agent
Install and Configure logstash-forwarder
Change content like below:
cd ~; curl -O http://packages.elasticsearch.org/logstashforwarder/centos/logstash-forwarder-0.3.1-1.x86_64.rpm
rpm -ivh ~/logstash-forwarder-0.3.1-1.x86_64.rpm
cd /etc/init.d/; curl -o logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_init
chmod +x logstash-forwarder
curl -o /etc/sysconfig/logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_sysconfig
vi /etc/sysconfig/logstash-forwarder
LOGSTASH_FORWARDER_OPTIONS="-config /etc/logstash-forwarder -spool-size 100"
cp /tmp/logstash.linuxblind.com.cert /etc/pki/tls/certs/
vi /etc/logstash-forwarder
{
"network": {
"servers": [ "logstash.linuxblind.com:5000" ],
"timeout": 15,
"ssl certificate": "/etc/pki/tls/certs/logstash.linuxblind.com.cert",
"ssl key": "/etc/pki/tls/private/logstash.linuxblind.com.key",
"ssl ca": "/etc/pki/tls/certs/logstash.linuxblind.com.cert"
},
"files": [
{
"paths": [
"/var/log/message"
],
"fields": { "type": "syslog" }
},
{
"paths": [
"/var/log/maillog"
],
"fields": { "type": "maillog" }
}
]
}
chkconfig logstash-forwarder on
/etc/init.d/logstash-forwarder start
crtl+c # exit
Step3:
In logstash-indexer
Modified /opt/logstash/patterns/grok-patterns and append following content
# Match for sendmail
#EMAIL [.a-zA-Z0-9_-]+@%{IPORHOST}
LOGIN [.a-zA-Z0-9_-]+
EMAIL %{LOGIN}@%{IPORHOST}
DSN [0-9][.][0-9][.][0-9]
QID [A-za-z0-9]{14}
# Match a relay that gives us a QID in the return status.
SENDMAIL_TO_1 %{SYSLOGBASE} %{QID:qid}: to=<%{EMAIL:to}>, (%{WORD}=%{DATA},)+ relay=%{IPORHOST:relay} \[%{IP}\], dsn=%{DSN:dsn}, stat=%{DATA:status} \(%{QID:qid} %{GREEDYDATA:status_message}\)
# Match a relay that does NOT give us a QID in the return status.
SENDMAIL_TO_2 %{SYSLOGBASE} %{QID:qid}: to=<%{EMAIL:to}>, (%{WORD}=%{DATA},)+ relay=%{IPORHOST:relay} \[%{IP}\], dsn=%{DSN:dsn}, stat=%{DATA:status} \(%{GREEDYDATA:status_message}\)
# Match a message with no relay IP address or status message.
SENDMAIL_TO_3 %{SYSLOGBASE} %{QID:qid}: to=<%{EMAIL:to}>, (%{WORD}=%{DATA},)+ relay=%{IPORHOST:relay}, dsn=%{DSN:dsn}, stat=%{GREEDYDATA:status}
# Match a message with no relay info at all.
SENDMAIL_TO_4 %{SYSLOGBASE} %{QID:qid}: to=<%{EMAIL:to}>, (%{WORD}=%{DATA},)+ stat=%{GREEDYDATA:status}
### TODO - match multiple recipients in To: field.
SENDMAIL_TO_5 %{SYSLOGBASE} %{QID:qid}: to=(<%{EMAIL:to}>,)+ (%{WORD}=%{DATA},)+ %{GREEDYDATA:status}
SENDMAIL_TO (%{SENDMAIL_TO_1}|%{SENDMAIL_TO_2}|%{SENDMAIL_TO_3}|%{SENDMAIL_TO_4})
SENDMAIL_FROM %{SYSLOGBASE} %{QID:qid}: from=<%{EMAIL:from}>, (%{WORD}=%{DATA},)+ relay=%{IPORHOST:relay} \[%{IP}\]
SENDMAIL_OTHER_1 %{SYSLOGBASE} %{QID:qid}: %{GREEDYDATA:message}
SENDMAIL_OTHER_2 %{SYSLOGBASE} STARTTLS=(client|server), relay=(\[)?%{IPORHOST:relay}(\])?%{GREEDYDATA:message}
SENDMAIL_OTHER_3 %{SYSLOGBASE} STARTTLS: %{GREEDYDATA:message}
SENDMAIL_OTHER_4 %{SYSLOGBASE} ruleset=tls_server, arg1=SOFTWARE, relay=%{IPORHOST:relay}, %{GREEDYDATA:message}
SENDMAIL_OTHER_5 %{SYSLOGBASE} STARTTLS=client, error: %{GREEDYDATA:message}
SENDMAIL_RELAY %{SYSLOGBASE} ruleset=check_relay, arg1=(\[)?%{IPORHOST}(\])?, arg2=%{IP:ip}, relay=(\[)?%{IPORHOST:relay}(\])??%{GREEDYDATA:message}
SENDMAIL_AUTH_1 %{SYSLOGBASE} AUTH=server, relay=%{IPORHOST:relay} \[%{IP}\]( \(may be forged\))?, authid=%{LOGIN:user}(@%{IPORHOST})?, %{GREEDYDATA:message}
SENDMAIL_AUTH_2 %{SYSLOGBASE} AUTH=server, relay=\[%{IP}\], authid=%{LOGIN:user}(@%{IPORHOST})?, %{GREEDYDATA:message}
SENDMAIL_AUTH (%{SENDMAIL_AUTH_1}|%{SENDMAIL_AUTH_2})
SENDMAIL_OTHER (%{SENDMAIL_OTHER_1}|%{SENDMAIL_OTHER_2}|%{SENDMAIL_OTHER_3}|%{SENDMAIL_OTHER_4}|%{SENDMAIL_OTHER_5})
SENDMAIL (%{SENDMAIL_TO}|%{SENDMAIL_FROM}|%{SENDMAIL_OTHER}|%{SENDMAIL_AUTH}|%{SENDMAIL_RELAY})
Modified /etc/logstash/conf.d/10-syslog.conf and Add following content in right place.
if [type] == "maillog" {
grok {
match => [ "message", "%{SENDMAIL_TO_1}",
"message", "%{SENDMAIL_TO_2}",
"message", "%{SENDMAIL_TO_3}",
"message", "%{SENDMAIL_TO_4}",
"message", "%{SENDMAIL_TO_5}",
"message", "%{SENDMAIL_TO}",
"message", "%{SENDMAIL_FROM}",
"message", "%{SENDMAIL_OTHER_1}",
"message", "%{SENDMAIL_OTHER_2}",
"message", "%{SENDMAIL_OTHER_3}",
"message", "%{SENDMAIL_OTHER_4}",
"message", "%{SENDMAIL_OTHER_5}",
"message", "%{SENDMAIL_RELAY}",
"message", "%{SENDMAIL_AUTH_1}",
"message", "%{SENDMAIL_AUTH_2}",
"message", "%{SENDMAIL_AUTH}",
"message", "%{SENDMAIL_OTHER}",
"message", "%{SENDMAIL}"
]
}
}
service logstash restart
============= Configuration For Logstash with Logstash-forwarder =============
On Logstash master
cat 01-lumberjack-input.conf
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash.linuxblind.com.cert"
ssl_key => "/etc/pki/tls/private//logstash.linuxblind.com.key"
}
}
cat 10-syslog.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
if [type] == "maillog" {
grok {
match => [ "message", "%{SENDMAIL_TO_1}",
"message", "%{SENDMAIL_TO_2}",
"message", "%{SENDMAIL_TO_3}",
"message", "%{SENDMAIL_TO_4}",
"message", "%{SENDMAIL_TO_5}",
"message", "%{SENDMAIL_TO}",
"message", "%{SENDMAIL_FROM}",
"message", "%{SENDMAIL_OTHER_1}",
"message", "%{SENDMAIL_OTHER_2}",
"message", "%{SENDMAIL_OTHER_3}",
"message", "%{SENDMAIL_OTHER_4}",
"message", "%{SENDMAIL_OTHER_5}",
"message", "%{SENDMAIL_RELAY}",
"message", "%{SENDMAIL_AUTH_1}",
"message", "%{SENDMAIL_AUTH_2}",
"message", "%{SENDMAIL_AUTH}",
"message", "%{SENDMAIL_OTHER}",
"message", "%{SENDMAIL}"
]
}
}
if [type] == "apache" {
grok {
pattern => "%{COMBINEDAPACHELOG}"
}
}
}
cat 30-lumberjack-output.conf
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
On Logstash agent
cat /etc/logstash-forwarder
{
"network": {
"servers": [ "logstash.linuxblind.com:5000" ],
"timeout": 15,
"ssl certificate": "/etc/pki/tls/certs/logstash.linuxblind.com.cert",
"ssl key": "/etc/pki/tls/private/logstash.linuxblind.com.key",
"ssl ca": "/etc/pki/tls/certs/logstash.linuxblind.com.cert"
},
"files": [
{
"paths": [
"/var/log/message"
],
"fields": { "type": "syslog" }
},
{
"paths": [
"/var/log/maillog"
],
"fields": { "type": "maillog" }
}
]
}
================================================================
How to package rpm for logstash-forwarder(32bit) and then Install logstash-forwarder on Amazon Linux (32bit) - The first time, you build environment need release is 32bit (i686/i386)
make rpm (choose this)
make deb
yum -y install gcc ruby-devel golang git rpm-build
gem install fpm
git clone git://github.com/elasticsearch/logstash-forwarder.git
cd logstash-forwarder
go build
Make packages, either:
rpm -ivh logstash-forwarder-0.3.1-1.i686.rpm
wget https://raw.githubusercontent.com/jamtur01/logstashbook-code/master/code/4/logstash_forwarder_redhat_init --no-check-certificate
mv logstash_forwarder_redhat_init logstash-forwarder
chmod +x logstash-forwarder
cp -f logstash-forwarder /etc/init.d/logstash-forwarder
The Second Time
Change content like below:
cd ~;
rpm -ivh logstash-forwarder-0.3.1-1.i686.rpm # you can download from below
curl -o /etc/pki/tls/certs/logstash..crt http://logstash.linuxblind.com/logstash/logstash-forwarder.crt
curl -o /etc/init.d/logstash-forwarder http://logstash.linuxblind.com/logstash/logstash-forwarder
chkconfig logstash-forwarder on
curl -o /etc/logstash-forwarder http://logstash.linuxblind.com/logstash/logstash-forwarder.conf
curl -o /etc/sysconfig/logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_sysconfig
vi /etc/sysconfig/logstash-forwarder
LOGSTASH_FORWARDER_OPTIONS="-config /etc/logstash-forwarder -spool-size 100"
Modified /etc/logstash-forwarder to match log paths
cd /var/log/; /etc/init.d/logstash-forwarder start
作者注：写的比较粗，有点不堪入目，有时间再仔细整理下, 另外上面你下载不了，可以找我索取。
QQ：1037447289
本文出自 “linuxblind开放空间” 博客，请务必保留此出处http://linuxblind.blog.51cto.com/7616603/1558694
Logstash+Elasticsearch+Kibana+Nginx set up our private log query system
标签：logstash elasticsearch kibana log
原文地址：http://linuxblind.blog.51cto.com/7616603/1558694
