标签：监控系统
告警系统
需求：使用shell定制各种个性化告警工具，但需要统一化管理、规范化管理。
思路：指定一个脚本包，包含主程序、子程序、配置文件、邮件引擎、输出日志等。
主程序：作为整个脚本的入口，是整个系统的命脉。
配置文件：是一个控制中心，用它来开关各个子程序，指定各个相关联的日志文件。
子程序：这个才是真正的监控脚本，用来监控各个指标。
邮件引擎：是由一个Php程序来实现，它可以定义发邮件的服务器、发邮件人以及收邮件人。
输出日志：整个监控系统要有日志输出。
要求：我们的机器角色多种多样，但是所有机器上都要部署同样的监控系统，也就说所有机器不管什么角色，整个
程序框架都是一致的，不同的地方在于根据不同的角色，定制不同的配置文件。
程序架构：
（ 主目录 mon ）
___________________________|__________________________________
| | | | |
bin conf shares mail log
| | | | |
[main.sh] [mon.conf] [load.sh 502.sh] [mail.php mail.sh] [mon.log err.log]
bin下是主程序
conf下是配置文件
shares下是各个监控脚本
mail下是邮件引擎
log下是日志
一、监控系统主脚本
告警系统，文档参考地址 http://www.apelearn.com/bbs/thread-8106-1-1.html
1、搭建系统架构目录
[root@wy ~]# cd /usr/local/sbin
[root@wy sbin]# mkdir mon
[root@wy sbin]# cd mon
[root@wy mon]# mkdir bin conf mail shares log
[root@wy mon]# ls -l
总用量 20
drwxr-xr-x 2 root root 4096 12月 5 02:45 bin
drwxr-xr-x 2 root root 4096 12月 5 02:45 conf
drwxr-xr-x 2 root root 4096 12月 5 02:45 log
drwxr-xr-x 2 root root 4096 12月 5 02:45 mail
drwxr-xr-x 2 root root 4096 12月 5 02:45 shares
2、主脚本
[root@wy mon]# cd bin
[root@wy bin]# vim main.sh
#!/bin/bash
# 标记是否发送邮件的开关
export send=1
# 过滤出ip地址，用于在邮件中显示具体哪台机器告警
export addr=`/sbin/ifconfig |grep -A1 ‘eth0‘ |grep ‘addr:‘ |awk ‘{print $2}‘ |awk -F: ‘{print $2}‘`
# 只需要最后一级目录名
last_dir=`pwd|awk -F‘/‘ ‘{print $NF}‘`
# 下面的判断目的是，保证执行脚本的时候，我们在bin目录里，不然监控脚本、邮件和日志很有可能找不到
if [ $last_dir == "bin" ] || [ $last_dir == "bin/" ]; then
conf_file="../conf/mon.conf"
else
echo "you shoud cd bin dir"
exit
fi
# 指定log文件输出
exec 1>>../log/mon.log 2>>../log/err.log
# 执行监控负载脚本
echo "`date +"%F %T"` load average"
/bin/bash ../shares/load.sh
# 先检查配置文件中是否需要监控502
if grep -q ‘to_mon_502=1‘ $conf_file; then
# 过滤出日志路径
export log=`grep ‘logfile=‘ $conf_file |awk -F ‘=‘ ‘{print $2}‘ |sed ‘s/ //g‘`
# 执行脚本
/bin/bash ../shares/502.sh
fi
解释说明：
"../conf/mon.conf" 因不确定mon路径，所以尽量使用相对路径
export 便于子shell识别该变量
../log/ 规范化管理，将所有日志写入同一个目录的不同文件下
sed ‘s/ //g‘ 去空格
3、mon配置文件
[root@wy bin]# vim ../conf/mon.conf
## to config the options if to monitor
## cdb 主要定义mysql的服务器地址、端口以及user、password
to_mon_cdb=0 ##0 or 1, default 0,0 not monitor, 1 monitor
cdb_ip=10.20.3.13
cdb_port=3315
cdb_user=username
cdb_pass=passwd
## httpd 如果是1则监控，为0不监控
to_mon_httpd=0
## php 如果是1则监控，为0不监控
to_mon_php_socket=0
## http_code_502 需要定义访问日志的路径
to_mon_502=1
logfile=/data/log/xxx.xxx.com/access.log
## request_count 定义日志路径以及域名
to_mon_request_count=0
req_log=/data/log/www.discuz.net/access.log
domainname=www.discuz.net
二、监控子脚本load.sh 502.sh
4、监控负载脚本load.sh
[root@wy bin]# vim ../shares/load.sh
#!/bin/bash
load=`uptime |awk -F‘load average:‘ ‘{print $2}‘ |cut -d, -f1 |sed ‘s/ //g‘ |cut -d. -f1`
if [ "$load" -gt 20 ] && [ "$send" -eq 1 ]; then
echo "$addr `date +%T` load is $load" >../log/load.tmp
/bin/bash ../mail/mail.sh $addr\_load $load ../log/load.tmp
fi
echo "`date +%T` load is $load"
5、监控502脚本502.sh
[root@wy bin]# vim ../shares/502.sh
#!/bin/bash
# 一分钟前的小时和分钟
d=`date -d "-1 min" +%H:%M`
# 统一502个数
c_502=`grep :$d: $log |grep ‘ 502 ‘|wc -l`
if [ $c_502 -gt 10 ] && [ $send == 1 ]; then
echo "$addr $d 502 count is $c_502">../log/502.tmp
# 执行发邮件脚本
/bin/bash ../mail/mail.sh $addr\_502 $c_502 ../log/502.tmp
fi
# 记录日志
echo "`date +%T` 502 $c_502"
三、邮件脚本
6、邮件脚本
[root@wy bin]# vim ../mail/mail.php
<?php
class Smtp
{
/* Public Variables */
var $smtp_port;
var $time_out;
var $host_name;
var $log_file;
var $relay_host;
var $debug;
var $auth;
var $user;
var $pass;
/* Private Variables */
var $sock;
/* Constractor */
function Smtp($relay_host = "", $smtp_port = 25,$auth = false,$user,$pass)
{
$this->debug = FALSE;
$this->smtp_port = $smtp_port;
$this->relay_host = $relay_host;
$this->time_out = 30; //is used in fsockopen()
#
$this->auth = $auth;//auth
$this->user = $user;
$this->pass = $pass;
#
$this->host_name = "localhost"; //is used in HELO command
$this->log_file = "";
$this->sock = FALSE;
}
/* Main Function */
function sendmail($to, $from, $subject = "", $body = "", $mailtype, $cc = "", $bcc = "", $additional_headers = "")
{
$mail_from = $this->get_address($this->strip_comment($from));
$body = ereg_replace("(^|(\r\n))(\.)", "\1.\3", $body);
$header = "MIME-Version:1.0\r\n";
if($mailtype=="HTML"){
$header .= "Content-Type:text/html\r\n";
}
$header .= "To: ".$to."\r\n";
if ($cc != "") {
$header .= "Cc: ".$cc."\r\n";
}
$header .= "From: $from<".$from.">\r\n";
$header .= "Subject: ".$subject."\r\n";
$header .= $additional_headers;
$header .= "Date: ".date("r")."\r\n";
$header .= "X-Mailer:By Redhat (PHP/".phpversion().")\r\n";
list($msec, $sec) = explode(" ", microtime());
$header .= "Message-ID: <".date("YmdHis", $sec).".".($msec*1000000).".".$mail_from.">\r\n";
$TO = explode(",", $this->strip_comment($to));
if ($cc != "") {
$TO = array_merge($TO, explode(",", $this->strip_comment($cc)));
}
if ($bcc != "") {
$TO = array_merge($TO, explode(",", $this->strip_comment($bcc)));
}
$sent = TRUE;
foreach ($TO as $rcpt_to) {
$rcpt_to = $this->get_address($rcpt_to);
if (!$this->smtp_sockopen($rcpt_to)) {
$this->log_write("Error: Cannot send email to ".$rcpt_to."\n");
$sent = FALSE;
continue;
}
if ($this->smtp_send($this->host_name, $mail_from, $rcpt_to, $header, $body)) {
$this->log_write("E-mail has been sent to <".$rcpt_to.">\n");
} else {
$this->log_write("Error: Cannot send email to <".$rcpt_to.">\n");
$sent = FALSE;
}
fclose($this->sock);
$this->log_write("Disconnected from remote host\n");
}
return $sent;
}
/* Private Functions */
function smtp_send($helo, $from, $to, $header, $body = "")
{
if (!$this->smtp_putcmd("HELO", $helo)) {
return $this->smtp_error("sending HELO command");
}
#auth
if($this->auth){
if (!$this->smtp_putcmd("AUTH LOGIN", base64_encode($this->user))) {
return $this->smtp_error("sending HELO command");
}
if (!$this->smtp_putcmd("", base64_encode($this->pass))) {
return $this->smtp_error("sending HELO command");
}
}
#
if (!$this->smtp_putcmd("MAIL", "FROM:<".$from.">")) {
return $this->smtp_error("sending MAIL FROM command");
}
if (!$this->smtp_putcmd("RCPT", "TO:<".$to.">")) {
return $this->smtp_error("sending RCPT TO command");
}
if (!$this->smtp_putcmd("DATA")) {
return $this->smtp_error("sending DATA command");
}
if (!$this->smtp_message($header, $body)) {
return $this->smtp_error("sending message");
}
if (!$this->smtp_eom()) {
return $this->smtp_error("sending <CR><LF>.<CR><LF> [EOM]");
}
if (!$this->smtp_putcmd("QUIT")) {
return $this->smtp_error("sending QUIT command");
}
return TRUE;
}
function smtp_sockopen($address)
{
if ($this->relay_host == "") {
return $this->smtp_sockopen_mx($address);
} else {
return $this->smtp_sockopen_relay();
}
}
function smtp_sockopen_relay()
{
$this->log_write("Trying to ".$this->relay_host.":".$this->smtp_port."\n");
$this->sock = @fsockopen($this->relay_host, $this->smtp_port, $errno, $errstr, $this->time_out);
if (!($this->sock && $this->smtp_ok())) {
$this->log_write("Error: Cannot connenct to relay host ".$this->relay_host."\n");
$this->log_write("Error: ".$errstr." (".$errno.")\n");
return FALSE;
}
$this->log_write("Connected to relay host ".$this->relay_host."\n");
return TRUE;
}
function smtp_sockopen_mx($address)
{
$domain = ereg_replace("^.+@([^@]+)$", "\1", $address);
if (!@getmxrr($domain, $MXHOSTS)) {
$this->log_write("Error: Cannot resolve MX \"".$domain."\"\n");
return FALSE;
}
foreach ($MXHOSTS as $host) {
$this->log_write("Trying to ".$host.":".$this->smtp_port."\n");
$this->sock = @fsockopen($host, $this->smtp_port, $errno, $errstr, $this->time_out);
if (!($this->sock && $this->smtp_ok())) {
$this->log_write("Warning: Cannot connect to mx host ".$host."\n");
$this->log_write("Error: ".$errstr." (".$errno.")\n");
continue;
}
$this->log_write("Connected to mx host ".$host."\n");
return TRUE;
}
$this->log_write("Error: Cannot connect to any mx hosts (".implode(", ", $MXHOSTS).")\n");
return FALSE;
}
function smtp_message($header, $body)
{
fputs($this->sock, $header."\r\n".$body);
$this->smtp_debug("> ".str_replace("\r\n", "\n"."> ", $header."\n> ".$body."\n> "));
return TRUE;
}
function smtp_eom()
{
fputs($this->sock, "\r\n.\r\n");
$this->smtp_debug(". [EOM]\n");
return $this->smtp_ok();
}
function smtp_ok()
{
$response = str_replace("\r\n", "", fgets($this->sock, 512));
$this->smtp_debug($response."\n");
if (!ereg("^[23]", $response)) {
fputs($this->sock, "QUIT\r\n");
fgets($this->sock, 512);
$this->log_write("Error: Remote host returned \"".$response."\"\n");
return FALSE;
}
return TRUE;
}
function smtp_putcmd($cmd, $arg = "")
{
if ($arg != "") {
if($cmd=="") $cmd = $arg;
else $cmd = $cmd." ".$arg;
}
fputs($this->sock, $cmd."\r\n");
$this->smtp_debug("> ".$cmd."\n");
return $this->smtp_ok();
}
function smtp_error($string)
{
$this->log_write("Error: Error occurred while ".$string.".\n");
return FALSE;
}
function log_write($message)
{
$this->smtp_debug($message);
if ($this->log_file == "") {
return TRUE;
}
$message = date("M d H:i:s ").get_current_user()."[".getmypid()."]: ".$message;
if (!@file_exists($this->log_file) || !($fp = @fopen($this->log_file, "a"))) {
$this->smtp_debug("Warning: Cannot open log file \"".$this->log_file."\"\n");
return FALSE;;
}
flock($fp, LOCK_EX);
fputs($fp, $message);
fclose($fp);
return TRUE;
}
function strip_comment($address)
{
$comment = "\([^()]*\)";
while (ereg($comment, $address)) {
$address = ereg_replace($comment, "", $address);
}
return $address;
}
function get_address($address)
{
$address = ereg_replace("([ \t\r\n])+", "", $address);
$address = ereg_replace("^.*<(.+)>.*$", "\1", $address);
return $address;
}
function smtp_debug($message)
{
if ($this->debug) {
echo $message;
}
}
}
$file = $argv[2];
$smtpserver = "smtp.qq.com";
$smtpserverport = "25";
$smtpusermail = "1198658@qq.com";
$smtpemailto = "lishi@139.com";
$smtpuser = "1198658";
$smtppass = "1212lss";
$mailsubject = $argv[1];
$mailbody = file_get_contents($file);
$mailtype = "HTML";
$smtp = new smtp($smtpserver,$smtpserverport,true,$smtpuser,$smtppass);
$smtp->sendmail($smtpemailto, $smtpusermail, $mailsubject, $mailbody, $mailtype);
?>
解释说明：
$smtpserver = "smtp.qq.com"; //SMTP服务器
$smtpserverport = "25"; //SMTP服务器端口
$smtpusermail = "1198658@qq.com"; //SMTP服务器的用户邮箱
$smtpemailto = "lishi@139.com"; //发送给谁
$smtpuser = "1198658"; //SMTP服务器的用户帐号
$smtppass = "1212lss"; //SMTP服务器的用户密码(独立密码)
$mailsubject = $argv[1]; //邮件主题
$mailbody = file_get_contents($file); //邮件内容
$mailtype = "HTML"; //邮件格式（HTML/TXT）,TXT为文本邮件
$smtp = new smtp($smtpserver,$smtpserverport,true,$smtpuser,$smtppass);
//这里面的一个true是表示使用身份验证,否则不使用身份验证.
//$smtp->debug = TRUE;//是否显示发送的调试信息
$smtp->sendmail($smtpemailto, $smtpusermail, $mailsubject, $mailbody, $mailtype);
?>
要想发邮件的话，首先要有php支持，若你没有安装过lamp或者lnmp，则需要运行yum install -y php 安装。
然后运行mail.php"邮箱主题写在这里" "/tmp/test.txt"其中，/tmp/test.txt内容为邮件内容。
开启SMTP服务（调用远程SMTP）
打开qq邮箱
安装php环境
[root@wy bin]# yum install -y php
7、mail.sh
[root@wy bin]# vim ../mail/mail.sh
#!/bin/bash
log=$1
t_s=`date +%s`
t_s2=`date -d "2 hours ago" +%s`
# 文件不存在，写入时间并创建文件
[ ! -f /tmp/$log ] && echo $t_s2 > /tmp/$log
# 从文件中读取出时间
t_s2=`tail -1 /tmp/$log|awk ‘{print $1}‘`
echo $t_s >> /tmp/$log
v=$[$t_s-$t_s2]
echo $v
if [ $v -gt 3600 ]; then
/dir/to/php ../mail/mail.php "$1 $2" "$3"
echo "0" > /tmp/$log.txt
else
[ ! -f /tmp/$log.txt ] && echo "0" > /tmp/$log.txt
nu=`cat /tmp/$log.txt`
nu2=$[$nu+1]
echo $nu2>/tmp/$log.txt
if [ $nu2 -gt 10 ]; then
/dir/to/php ../mail/mail.php "trouble continue 10 min $1 $2 " "$3"
echo "0" > /tmp/$log.txt
fi
fi
四、邮件脚本收敛讲解
第一次的时候
/tmp/$log是不存在的，t_s2是两小时前的时间就写进了/tmp/$log；
$v的值大于或等于7200；
然后发邮件，并把0写入/tmp/$log.txt里（作为告警收敛的标记）
第二次的时候
/tmp/$log已经存在了，这时t_s2里是第一次的两小时前的时间；
$v的值小于等于60；
这时/tmp/$log.txt已经存在，则把/tmp/$log.txt里的值取出来加1，再写进去；
如果这个值大于10才会发邮件；
这样就反复执行，每次$v的值都是60；直到/tmp/$log.txt里的值为11，这时大于10了，就发送邮件，
这时也把/tmp/$log.txt清空为0，这也就相当于完成了一个轮回。
五、监控系统整体回顾
8、加入任务计划
[root@wy bin]# crontab -e
* * * * * cd /usr/local/sbin/mon/bin; /bin/bash main.sh
总结
mon程序实现本机监控
优点：
1、不会因为客户端服务端通信出现故障导致告警收不到，每台机器都有自己的监控告警体系，永远都在执行，不会因为某一台机器挂掉导致其他机器监控不顺利。
2、初创公司，服务器比较少的情况下，不愿意搭建大型监控系统的情况下，临时使用的解决方案。
--------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
监控磁盘脚本
假设：
需要监控的盘符有/、boot、db1
思路：监控数字
首先过滤出数字
[root@wy mon]# vim shares/disk.sh
#!/bin/bash
rm -f ../log/disk.tmp
for r in `df -h |awk -F ‘[ %]+‘ ‘{print $5}‘ |grep -v ‘Use‘`
do
if [ "$r" -gt 90 ] && [ "$send" -eq "1" ]; then
echo "$addr `date +%T` disk useage is $r" >>../log/disk.tmp
fi
done
if [ -f ../log/disk.tmp ]; then
df -h >> ../log/disk.tmp
/bin/bash ../mail/mail.sh $addr\_disk $r ../log/disk.tmp
echo "`date +%T` disk useage not ok"
else
echo "`date +%T` disk useage is ok"
fi
本文出自 “linux” 博客，转载请与作者联系！
标签：监控系统
原文地址：http://warm51fun.blog.51cto.com/3884274/1891466
