一、思路说明
使用shell可以定制各种个性化的告警工具,实现统一规范化管理。我们可以指定一个脚本包,包含主程序、子程序、配置文件、邮件引擎、输出日志等。虽然机器角色多种多样,但是所有机器上都部署同样的监控系统,也就说所有机器不管什么角色,整个程序框架都是一致的,不同的地方在于根据不同的角色,定制不同的配置文件。
1)主程序:作为整个脚本的入口,是整个系统的命脉。
2)子程序:这个才是真正的监控脚本,用来监控各个指标。
3)配置文件:是一个控制中心,用它来开关各个子程序,指定各个相关联的日志文件。
4)邮件引擎:是由一个php程序来实现,它可以定义发邮件的服务器、发邮件人以及收邮件人。
5)输出日志:整个监控系统要有日志输出。
二、程序架构
bin下是主程序,conf下是配置文件,shares下是各个监控脚本,mail下是邮件引擎,log下是日志。
三、编写shell
1、新建目录架构
[root@centos sbin]# mkdir mon
[root@centos sbin]# cd mon/
[root@centos mon]# mkdir bin conf shares mail log
[root@centos mon]# ls
bin conf log mail shares
2、主程序
主程序脚本main.sh,每台机器都要拷贝,具体的配置按需修改。
[root@centos mon]# cd bin/
[root@centos bin]# vim main.sh
#!/bin/bash # 是否发送邮件的开关 export send=1 # 过滤ip地址 export addr=`/sbin/ifconfig |grep -A1 ‘eth0‘ |grep addr: |awk ‘{print $2}‘|awk -F: ‘{print $2}‘` dir=`pwd` # 只需要最后一级目录名 last_dir=`echo $dir|awk -F‘/‘ ‘{print $NF}‘` # 下面的判断目的是,保证执行脚本的时候,我们在bin目录里,不然监控脚本、邮件和日志很有可能找不到 if [ $last_dir == "bin" ] || [ $last_dir == "bin/" ]; then conf_file="../conf/mon.conf" else echo "you shoud cd bin dir" exit fi exec 1>>../log/mon.log 2>>../log/err.log echo "`date +"%F %T"` load average" /bin/bash ../shares/load.sh #先检查配置文件中是否需要监控502 if grep -q ‘to_mon_502=1‘ $conf_file; then export log=`grep ‘logfile=‘ $conf_file |awk -F ‘=‘ ‘{print $2}‘ |sed ‘s/ //g‘` /bin/bash ../shares/502.sh fi |
3、配置文件
新建配置文件mon.conf,配置文件主要是定义各个参数
[root@centos bin]# vim ../conf/mon.conf
## to config the options if to monitor ## cdb 主要定义mysql的服务器地址、端口以及user、password to_mon_cdb=0 ##0 or 1, default 0,0 not monitor, 1 monitor cdb_ip=10.20.3.13 cdb_port=3315 cdb_user=username cdb_pass=passwd ## httpd 如果是1则监控,为0不监控 to_mon_httpd=0 ## php 如果是1则监控,为0不监控 to_mon_php_socket=0 ## http_code_502 需要定义访问日志的路径 to_mon_502=1 logfile=/data/log/xxx.xxx.com/access.log ## request_count 定义日志路径以及域名 to_mon_request_count=0 req_log=/data/log/www.discuz.net/access.log domainname=www.discuz.net |
4、子程序
share是存放各种子程序的目录,通过主程序调用这里的脚本
1)load.sh
[root@centos bin]# vim ../shares/load.sh
#! /bin/bash load=`uptime |awk -F ‘average:‘ ‘{print $2}‘|cut -d‘,‘ -f1|sed ‘s/ //g‘ |cut -d. -f1` if [ $load -gt 20 ] && [ $send -eq "1" ] then echo "$addr `date +%T` load is $load" >../log/load.tmp /bin/bash ../mail/mail.sh $addr\_load $load ../log/load.tmp fi echo "`date +%T` load is $load" |
2)502.sh
[root@centos bin]# vim ../shares/502.sh
#! /bin/bash d=`date -d "-1 min" +%H:%M` c_502=`grep :$d: $log |grep ‘ 502 ‘|wc -l` if [ $c_502 -gt 10 ] && [ $send == 1 ]; then echo "$addr $d 502 count is $c_502">../log/502.tmp /bin/bash ../mail/mail.sh $addr\_502 $c_502 ../log/502.tmp fi echo "`date +%T` 502 $c_502" |
3)disk.sh
[root@centos bin]# vim ../shares/disk.sh
#! /bin/bash rm -f ../log/disk.tmp for r in `df -h |awk -F ‘[ %]+‘ ‘{print $5}‘|grep -v Use` do if [ $r -gt 90 ] && [ $send -eq "1" ] then echo "$addr `date +%T` disk useage is $r" >>../log/disk.tmp fi if [ -f ../log/disk.tmp ] then df -h >> ../log/disk.tmp /bin/bash ../mail/mail.sh $addr\_disk $r ../log/disk.tmp echo "`date +%T` disk useage is nook" else echo "`date +%T` disk useage is ok" fi |
5、邮件引擎
利用php来做一个邮件引擎mail.php,php作用就是远程调用smtp服务器来做认证。
1)mail.php
[root@centos bin]# vim ../mail/mail.php
<?php class Smtp { /* Public Variables */ var $smtp_port; var $time_out; var $host_name; var $log_file; var $relay_host; var $debug; var $auth; var $user; var $pass; /* Private Variables */ var $sock; /* Constractor */ function Smtp($relay_host = "", $smtp_port = 25,$auth = false,$user,$pass) { $this->debug = FALSE; $this->smtp_port = $smtp_port; $this->relay_host = $relay_host; $this->time_out = 30; //is used in fsockopen() # $this->auth = $auth;//auth $this->user = $user; $this->pass = $pass; # $this->host_name = "localhost"; //is used in HELO command $this->log_file = ""; $this->sock = FALSE; } /* Main Function */ function sendmail($to, $from, $subject = "", $body = "", $mailtype, $cc = "", $bcc = "", $additional_headers = "") { $mail_from = $this->get_address($this->strip_comment($from)); $body = ereg_replace("(^|(\r\n))(\.)", "\1.\3", $body); $header = "MIME-Version:1.0\r\n"; if($mailtype=="HTML"){ $header .= "Content-Type:text/html\r\n"; } $header .= "To: ".$to."\r\n"; if ($cc != "") { $header .= "Cc: ".$cc."\r\n"; } $header .= "From: $from<".$from.">\r\n"; $header .= "Subject: ".$subject."\r\n"; $header .= $additional_headers; $header .= "Date: ".date("r")."\r\n"; $header .= "X-Mailer:By Redhat (PHP/".phpversion().")\r\n"; list($msec, $sec) = explode(" ", microtime()); $header .= "Message-ID: <".date("YmdHis", $sec).".".($msec*1000000).".".$mail_from.">\r\n"; $TO = explode(",", $this->strip_comment($to)); if ($cc != "") { $TO = array_merge($TO, explode(",", $this->strip_comment($cc))); } if ($bcc != "") { $TO = array_merge($TO, explode(",", $this->strip_comment($bcc))); } $sent = TRUE; foreach ($TO as $rcpt_to) { $rcpt_to = $this->get_address($rcpt_to); if (!$this->smtp_sockopen($rcpt_to)) { $this->log_write("Error: Cannot send email to ".$rcpt_to."\n"); $sent = FALSE; continue; } if ($this->smtp_send($this->host_name, $mail_from, $rcpt_to, $header, $body)) { $this->log_write("E-mail has been sent to <".$rcpt_to.">\n"); } else { $this->log_write("Error: Cannot send email to <".$rcpt_to.">\n"); $sent = FALSE; } fclose($this->sock); $this->log_write("Disconnected from remote host\n"); } return $sent; } /* Private Functions */ function smtp_send($helo, $from, $to, $header, $body = "") { if (!$this->smtp_putcmd("HELO", $helo)) { return $this->smtp_error("sending HELO command"); } #auth if($this->auth){ if (!$this->smtp_putcmd("AUTH LOGIN", base64_encode($this->user))) { return $this->smtp_error("sending HELO command"); } if (!$this->smtp_putcmd("", base64_encode($this->pass))) { return $this->smtp_error("sending HELO command"); } } # if (!$this->smtp_putcmd("MAIL", "FROM:<".$from.">")) { return $this->smtp_error("sending MAIL FROM command"); } if (!$this->smtp_putcmd("RCPT", "TO:<".$to.">")) { return $this->smtp_error("sending RCPT TO command"); } if (!$this->smtp_putcmd("DATA")) { return $this->smtp_error("sending DATA command"); } if (!$this->smtp_message($header, $body)) { return $this->smtp_error("sending message"); } if (!$this->smtp_eom()) { return $this->smtp_error("sending <CR><LF>.<CR><LF> [EOM]"); } if (!$this->smtp_putcmd("QUIT")) { return $this->smtp_error("sending QUIT command"); } return TRUE; } function smtp_sockopen($address) { if ($this->relay_host == "") { return $this->smtp_sockopen_mx($address); } else { return $this->smtp_sockopen_relay(); } } function smtp_sockopen_relay() { $this->log_write("Trying to ".$this->relay_host.":".$this->smtp_port."\n"); $this->sock = @fsockopen($this->relay_host, $this->smtp_port, $errno, $errstr, $this->time_out); if (!($this->sock && $this->smtp_ok())) { $this->log_write("Error: Cannot connenct to relay host ".$this->relay_host."\n"); $this->log_write("Error: ".$errstr." (".$errno.")\n"); return FALSE; } $this->log_write("Connected to relay host ".$this->relay_host."\n"); return TRUE; } function smtp_sockopen_mx($address) { $domain = ereg_replace("^.+@([^@]+)$", "\1", $address); if (!@getmxrr($domain, $MXHOSTS)) { $this->log_write("Error: Cannot resolve MX \"".$domain."\"\n"); return FALSE; } foreach ($MXHOSTS as $host) { $this->log_write("Trying to ".$host.":".$this->smtp_port."\n"); $this->sock = @fsockopen($host, $this->smtp_port, $errno, $errstr, $this->time_out); if (!($this->sock && $this->smtp_ok())) { $this->log_write("Warning: Cannot connect to mx host ".$host."\n"); $this->log_write("Error: ".$errstr." (".$errno.")\n"); continue; } $this->log_write("Connected to mx host ".$host."\n"); return TRUE; } $this->log_write("Error: Cannot connect to any mx hosts (".implode(", ", $MXHOSTS).")\n"); return FALSE; } function smtp_message($header, $body) { fputs($this->sock, $header."\r\n".$body); $this->smtp_debug("> ".str_replace("\r\n", "\n"."> ", $header."\n> ".$body."\n> ")); return TRUE; } function smtp_eom() { fputs($this->sock, "\r\n.\r\n"); $this->smtp_debug(". [EOM]\n"); return $this->smtp_ok(); } function smtp_ok() { $response = str_replace("\r\n", "", fgets($this->sock, 512)); $this->smtp_debug($response."\n"); if (!ereg("^[23]", $response)) { fputs($this->sock, "QUIT\r\n"); fgets($this->sock, 512); $this->log_write("Error: Remote host returned \"".$response."\"\n"); return FALSE; } return TRUE; } function smtp_putcmd($cmd, $arg = "") { if ($arg != "") { if($cmd=="") $cmd = $arg; else $cmd = $cmd." ".$arg; } fputs($this->sock, $cmd."\r\n"); $this->smtp_debug("> ".$cmd."\n"); return $this->smtp_ok(); } function smtp_error($string) { $this->log_write("Error: Error occurred while ".$string.".\n"); return FALSE; } function log_write($message) { $this->smtp_debug($message); if ($this->log_file == "") { return TRUE; } $message = date("M d H:i:s ").get_current_user()."[".getmypid()."]: ".$message; if (!@file_exists($this->log_file) || !($fp = @fopen($this->log_file, "a"))) { $this->smtp_debug("Warning: Cannot open log file \"".$this->log_file."\"\n"); return FALSE;; } flock($fp, LOCK_EX); fputs($fp, $message); fclose($fp); return TRUE; } function strip_comment($address) { $comment = "\([^()]*\)"; while (ereg($comment, $address)) { $address = ereg_replace($comment, "", $address); } return $address; } function get_address($address) { $address = ereg_replace("([ \t\r\n])+", "", $address); $address = ereg_replace("^.*<(.+)>.*$", "\1", $address); return $address; } function smtp_debug($message) { if ($this->debug) { echo $message; } } } $file = $argv[2]; $smtpserver = "smtp.qq.com"; //SMTP服务器 $smtpserverport = "25"; //SMTP服务器端口 $smtpusermail = "1198658@qq.com"; //SMTP服务器的用户邮箱 $smtpemailto = "lishi@139.com"; //发送给谁 $smtpuser = "1198658"; //SMTP服务器的用户帐号 $smtppass = "1212lss"; //SMTP服务器的用户密码 $mailsubject = $argv[1]; //邮件主题 $mailbody = file_get_contents($file); //邮件内容 $mailtype = "HTML"; //邮件格式(HTML/TXT),TXT为文本邮件 $smtp = new smtp($smtpserver,$smtpserverport,true,$smtpuser,$smtppass); //这里面的一个true是表示使用身份验证,否则不使用身份验证 $smtp->debug = TRUE; //是否显示发送的调试信息$smtp->sendmail($smtpemailto, $smtpusermail, $mailsubject, $mailbody, $mailtype); ?> |
要想发邮件的话,首先要有php支持,若你没有安装过lamp或者lnmp,则需要运行yum install -y php 安装。
然后运行:
php mail.php "邮箱主题" "/tmp/test.txt"
其中,/tmp/test.txt 内容为邮件内容。
2)mail.sh
[root@centos bin]# vim ../mail/mail.sh
#! /bin/bash log=$1 t_s=`date +%s` t_s2=`date -d "2 hours ago" +%s` if [ ! -f /tmp/$log ] then echo $t_s2 > /tmp/$log fi t_s2=`tail -1 /tmp/$log|awk ‘{print $1}‘` echo $t_s>>/tmp/$log v=$[$t_s-$t_s2] echo $v if [ $v -gt 3600 ] then /dir/to/php ../mail/mail.php "$1 $2" "$3" echo "0" > /tmp/$log.txt else if [ ! -f /tmp/$log.txt ] then echo "0" > /tmp/$log.txt fi nu=`cat /tmp/$log.txt` nu2=$[$nu+1] echo $nu2>/tmp/$log.txt if [ $nu2 -gt 10 ] then /dir/to/php ../mail/mail.php "trouble continue 10 min $1 $2 " "$3" echo "0" > /tmp/$log.txt fi fi |
注意:/dir/to/php是你php的路径,我这里是/usr/local/php。/tmp/$log.txt文件记录的是告警次数,第一次发生告警,则发送一次邮件,若每分钟都检测出到告警,暂不发送邮件;当告警次数达到了10次再发送一次邮件,然后告警次数再置0,这样一直轮回,可以做到邮件告警收敛。
本文出自 “M四月天” 博客,请务必保留此出处http://msiyuetian.blog.51cto.com/8637744/1711488
原文地址:http://msiyuetian.blog.51cto.com/8637744/1711488