码迷,mamicode.com
首页 > 其他好文 > 详细

zabbix监控nginx性能状态

时间:2017-02-05 01:08:39      阅读:886      评论:0      收藏:0      [点我收藏+]

标签:zabbix   nginx web   

  nginx在生产环境中的应用越来越广泛,所以需要对nginx的性能状态做一些监控,来发现出来出现的问题。nginx处理流程图具体如下:

技术分享

注释:Accepts(接受)、Handled(已处理)、Requests(请求数)是一直在增加的计数器。Active(活跃)、Waiting(等待)、Reading(读)、Writing(写)随着请求量而增减

名称

描述

指标类型

Accepts(接受)

NGINX 所接受的客户端连接数

资源: 功能

Handled(已处理)

成功的客户端连接数

资源: 功能

Active(活跃)

当前活跃的客户端连接数

资源: 功能

Dropped(已丢弃,计算得出)

丢弃的连接数(接受 – 已处理)

工作:错误*

Requests(请求数)

客户端请求数

工作:吞吐量

NGINX worker 进程接受 OS 的连接请求时 Accepts 计数器增加,而Handled 是当实际的请求得到连接时(通过建立一个新的连接或重新使用一个空闲的)。这两个计数器的值通常都是相同的,如果它们有差别则表明连接被Dropped, 往往这是由于资源限制,比如已经达到 NGINX 的worker_connections的限制。


首先nginx需要配置nginx_status  具体步骤是:在zabbix agentd客户端上,查看nginx是否加载了with-http_stub_status_module。因为zabbix监控nginx是根据nginx的Stub Status模块,抓取Status模块所提供的数据。假如以前没开启,现在想启用StubStatus 模块,在编译nginx 的时候要加上参数with-http_stub_status_module,执行./configure && make就可以了,不用make install,一般情况下都是安装的。

(一)配置nginx 

1,查看nginx_status是否开启,查看已开启。

[root@iZ237lzm354Z scripts]# /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.4.7
built by gcc 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) 
TLS SNI support enabled
configure arguments: --with-http_stub_status_module --with-http_ssl_module --with-pcre 
--with-http_realip_module --with-http_image_filter_module
[root@iZ237lzm354Z scripts]#

2,nginx_status开启的步骤:

[root@iZ237lzm354Z scripts]# vim /usr/local/nginx/conf/nginx.conf
 server {
                        listen       80 ;
                        server_name  www.guojinbao.com; 
           rewrite ^/invitejoin/(.*)\.htm[l]?$  /register.shtml?$1 last; 
                 index index.jsp index.html;
                 root /opt/home;
            location = /nginx-status {
                              stub_status on;
                              access_log  off;
                              allow 127.0.0.1;
                              allow 10.253.12.34; 
                             ####zabbix服务器端的IP地址一般为内网IP
                               }

3,测试并启动nginx

[root@iZ237lzm354Z scripts]# /usr/local/nginx/sbin/nginx -t

nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok

nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

[root@iZ237lzm354Z scripts]# /usr/local/nginx/sbin/nginx -s reload

4,用curl来进行测试:

[root@iZ237lzm354Z scripts]# curl www.guojinbao.com/nginx-status
Active connections: 979 
server accepts handled requests
 756072922 756072922 1136799890 
Reading: 0 Writing: 4 Waiting: 975

备注:

Active connections –对后端发起的活动连接数 
server accepts handled requests — 总共处理了756072922个连接 , 成功创建 756072922次握手, 总共处理了1136799890个请求
Reading — nginx读取客户端的连接(header)数.
Writing —  nginx返回给客户端的header数
waiting — nginx 请求处理完成,正在等待下一请求指令的连接。(开启 keep-alive 的情况下,这个值等于 active – (reading+writing), 意思就是 Nginx 已经处理完正在等候下一次请求指令的驻留连接。在访问效率很高,请求很快被处理完毕的情况下,Waiting 数比较多是正常的。如果 reading + writing 数较多,则说明并发访问量很大,正在处理过程中


(二)配置zabbix_agentd

1,编写脚步来获取nginx的相关信息

[root@ittestserver1 opt]# vim /usr/local/zabbix/scripts/nginx-check_performance.sh 

#!/bin/bash
##################################
# Zabbix monitoring script
#
# nginx:
# - anything available via nginx stub-status module
#
##################################
# Contact:
# vincent.viallet@gmail.com
# Zabbix requested parameter
ZBX_REQ_DATA="$1"
ZBX_REQ_DATA_URL="$2"
# Nginx defaults
NGINX_STATUS_DEFAULT_URL="www.guojinbao.com/nginx-status"    #(这里写网站的域名)
WGET_BIN="/usr/bin/wget"
#
# Error handling:
# - need to be displayable in Zabbix (avoid NOT_SUPPORTED)
# - items need to be of type "float" (allow negative + float)
#
ERROR_NO_ACCESS_FILE="-0.9900"
ERROR_NO_ACCESS="-0.9901"
ERROR_WRONG_PARAM="-0.9902"
ERROR_DATA="-0.9903" # either can not connect / bad host / bad port
# Handle host and port if non-default
if [ ! -z "$ZBX_REQ_DATA_URL" ]; then
 URL="$ZBX_REQ_DATA_URL"
else
 URL="$NGINX_STATUS_DEFAULT_URL"
fi
# save the nginx stats in a variable for future parsing
NGINX_STATS=$($WGET_BIN -q $URL -O - 2> /dev/null)
# error during retrieve
if [ $? -ne 0 -o -z "$NGINX_STATS" ]; then
 echo $ERROR_DATA
 exit 1
fi
#
# Extract data from nginx stats
#
case $ZBX_REQ_DATA in
 active_connections) echo "$NGINX_STATS" | head -1 | cut -f3 -d‘ ‘;;
 accepted_connections) echo "$NGINX_STATS" | grep -Ev ‘[a-zA-Z]‘ | cut -f2 -d‘ ‘;;
 handled_connections) echo "$NGINX_STATS" | grep -Ev ‘[a-zA-Z]‘ | cut -f3 -d‘ ‘;;
 handled_requests) echo "$NGINX_STATS" | grep -Ev ‘[a-zA-Z]‘ | cut -f4 -d‘ ‘;;
 reading) echo "$NGINX_STATS" | tail -1 | cut -f2 -d‘ ‘;;
 writing) echo "$NGINX_STATS" | tail -1 | cut -f4 -d‘ ‘;;
 waiting) echo "$NGINX_STATS" | tail -1 | cut -f6 -d‘ ‘;;
 *) echo $ERROR_WRONG_PARAM; exit 1;;
esac
exit 0


[root@ittestserver1 opt]# chmod +x /usr/local/zabbix/scripts/nginx-check_performance.sh
-rw-r--r-x1 root root 1645 2月 4 14:26/usr/local/zabbix/scripts/nginx-check_performance.sh


2,配置zabbix_agentd.conf。启用UserParameter,并配置相关的参数。

[root@ittestserver1 opt]# vim /usr/local/zabbix/etc/zabbix_agentd.conf
####### USER-DEFINED MONITORED PARAMETERS #######
### Option: UnsafeUserParameters
#       Allow all characters to be passed in arguments to user-defined parameters.
#       The following characters are not allowed:
#       \ ‘ " ` * ? [ ] { } ~ $ ! & ; ( ) < > | # @
#       Additionally, newline characters are not allowed.
#       0 - do not allow
#       1 - allow
#
# Mandatory: no
# Range: 0-1
# Default:
# UnsafeUserParameters=0
UnsafeUserParameters=1
### Option: UserParameter
#       User-defined parameter to monitor. There can be several user-defined parameters.
#       Format: UserParameter=<key>,<shell command>
#       See ‘zabbix_agentd‘ directory for examples.
#
# Mandatory: no
# Default:
# UserParameter=
UserParameter=nginx[*],/usr/local/zabbix/scripts/nginx-check_performance.sh "$1"


3,重启zabbix_agentd客户端

[root@zabbix ~]# /etc/init.d/zabbix_agentd restart
Shutting down zabbix_agentd:                               [  OK  ]
Starting zabbix_agentd:                                    [  OK  ]
[root@zabbix ~]#

4,在zabbix服务端(server)进行测试。

[root@zabbix ~]# zabbix_get -s 10.253.17.20 -p 10050 -k "nginx[reading]"
0
[root@zabbix ~]#


(三)在网页上配置nginx模板的相关监控

1,登录zabbix界面,依次点击:配置(configuration)---模板(template)---导入(import)

技术分享

2,给主机添加模板:选择主机---nginx服务器主机---模板---选择(刚刚导入的nginx性能状态的模板)---添加---更新

技术分享


3,查看nginx监控的最新数据:监控中---图形---选择相应的监控类型。

技术分享









Active:当前活跃的连接数。

Accepts:接受的请求数

Handled:处理的请求数(正常服务器响应,这两项应该是可以相等的)

Requests:客户端处理的请求数。(吞吐量)

Reading:当接收到请求时,连接离开 Waiting 状态,并且该请求本身使 Reading 状态计数增加。在这种状态下 NGINX 会读取客户端请求首部。请求首部是比较小的,因此这通常是一个快速的操作。


Writing:请求被读取之后,其使 Writing 状态计数增加,并保持在该状态,直到响应返回给客户端。这意味着,该请求在 Writing 状态时, 一方面 NGINX 等待来自上游系统的结果(系统放在 NGINX “后面”),另外一方面,NGINX 也在同时响应。请求往往会在 Writing 状态花费大量的时间。


Waiting:活跃的连接也可以处于 Waiting 子状态,如果有在此刻没有活跃请求的话。新连接可以绕过这个状态并直接变为到 Reading 状态,最常见的是在使用“accept filter(接受过滤器)” 和 “deferred accept(延迟接受)”时,在这种情况下,NGINX 不会接收 worker 进程的通知,直到它具有足够的数据才开始响应。如果连接设置为 keep-alive ,那么它在发送响应后将处于等待状态



备注:nginx模板脚本

<?xml version="1.0" encoding="UTF-8"?>
<zabbix_export>
    <version>2.0</version>
    <date>2013-03-26T04:17:58Z</date>
    <groups>
        <group>
            <name>Templates</name>
        </group>
    </groups>
    <templates>
        <template>
            <template>Template_Nginx</template>
            <name>Template_Nginx</name>
            <groups>
                <group>
                    <name>Templates</name>
                </group>
            </groups>
            <applications>
                <application>
                    <name>Nginx</name>
                </application>
            </applications>
            <items>
                <item>
                    <name>Nginx $1</name>
                    <type>0</type>
                    <snmp_community/>
                    <multiplier>0</multiplier>
                    <snmp_oid/>
                    <key>nginx[waiting,{$NGINX_STATUS_URL}]</key>
                    <delay>60</delay>
                    <history>30</history>
                    <trends>365</trends>
                    <status>0</status>
                    <value_type>0</value_type>
                    <allowed_hosts/>
                    <units/>
                    <delta>0</delta>
                    <snmpv3_securityname/>
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>
                    <snmpv3_authpassphrase/>
                    <snmpv3_privpassphrase/>
                    <formula>1</formula>
                    <delay_flex/>
                    <params/>
                    <ipmi_sensor/>
                    <data_type>0</data_type>
                    <authtype>0</authtype>
                    <username/>
                    <password/>
                    <publickey/>
                    <privatekey/>
                    <port/>
                    <description/>
                    <inventory_link>0</inventory_link>
                    <applications>
                        <application>
                            <name>Nginx</name>
                        </application>
                    </applications>
                    <valuemap/>
                </item>
                <item>
                    <name>Nginx $1</name>
                    <type>0</type>
                    <snmp_community/>
                    <multiplier>0</multiplier>
                    <snmp_oid/>
                    <key>nginx[writing,{$NGINX_STATUS_URL}]</key>
                    <delay>60</delay>
                    <history>30</history>
                    <trends>365</trends>
                    <status>0</status>
                    <value_type>0</value_type>
                    <allowed_hosts/>
                    <units/>
                    <delta>0</delta>
                    <snmpv3_securityname/>
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>
                    <snmpv3_authpassphrase/>
                    <snmpv3_privpassphrase/>
                    <formula>1</formula>
                    <delay_flex/>
                    <params/>
                    <ipmi_sensor/>
                    <data_type>0</data_type>
                    <authtype>0</authtype>
                    <username/>
                    <password/>
                    <publickey/>
                    <privatekey/>
                    <port/>
                    <description/>
                    <inventory_link>0</inventory_link>
                    <applications>
                        <application>
                            <name>Nginx</name>
                        </application>
                    </applications>
                    <valuemap/>
                </item>
                <item>
                    <name>Nginx $1</name>
                    <type>0</type>
                    <snmp_community/>
                    <multiplier>0</multiplier>
                    <snmp_oid/>
                    <key>nginx[active_connections,{$NGINX_STATUS_URL}]</key>
                    <delay>60</delay>
                    <history>30</history>
                    <trends>365</trends>
                    <status>0</status>
                    <value_type>0</value_type>
                    <allowed_hosts/>
                    <units/>
                    <delta>0</delta>
                    <snmpv3_securityname/>
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>
                    <snmpv3_authpassphrase/>
                    <snmpv3_privpassphrase/>
                    <formula>1</formula>
                    <delay_flex/>
                    <params/>
                    <ipmi_sensor/>
                    <data_type>0</data_type>
                    <authtype>0</authtype>
                    <username/>
                    <password/>
                    <publickey/>
                    <privatekey/>
                    <port/>
                    <description/>
                    <inventory_link>0</inventory_link>
                    <applications>
                        <application>
                            <name>Nginx</name>
                        </application>
                    </applications>
                    <valuemap/>
                </item>
                <item>
                    <name>Nginx $1</name>
                    <type>0</type>
                    <snmp_community/>
                    <multiplier>0</multiplier>
                    <snmp_oid/>
                    <key>nginx[reading,{$NGINX_STATUS_URL}]</key>
                    <delay>60</delay>
                    <history>30</history>
                    <trends>365</trends>
                    <status>0</status>
                    <value_type>0</value_type>
                    <allowed_hosts/>
                    <units/>
                    <delta>0</delta>
                    <snmpv3_securityname/>
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>
                    <snmpv3_authpassphrase/>
                    <snmpv3_privpassphrase/>
                    <formula>1</formula>
                    <delay_flex/>
                    <params/>
                    <ipmi_sensor/>
                    <data_type>0</data_type>
                    <authtype>0</authtype>
                    <username/>
                    <password/>
                    <publickey/>
                    <privatekey/>
                    <port/>
                    <description/>
                    <inventory_link>0</inventory_link>
                    <applications>
                        <application>
                            <name>Nginx</name>
                        </application>
                    </applications>
                    <valuemap/>
                </item>
                <item>
                    <name>Nginx $1/sec</name>
                    <type>0</type>
                    <snmp_community/>
                    <multiplier>0</multiplier>
                    <snmp_oid/>
                    <key>nginx[handled_requests,{$NGINX_STATUS_URL}]</key>
                    <delay>60</delay>
                    <history>30</history>
                    <trends>365</trends>
                    <status>0</status>
                    <value_type>0</value_type>
                    <allowed_hosts/>
                    <units/>
                    <delta>1</delta>
                    <snmpv3_securityname/>
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>
                    <snmpv3_authpassphrase/>
                    <snmpv3_privpassphrase/>
                    <formula>1</formula>
                    <delay_flex/>
                    <params/>
                    <ipmi_sensor/>
                    <data_type>0</data_type>
                    <authtype>0</authtype>
                    <username/>
                    <password/>
                    <publickey/>
                    <privatekey/>
                    <port/>
                    <description/>
                    <inventory_link>0</inventory_link>
                    <applications>
                        <application>
                            <name>Nginx</name>
                        </application>
                    </applications>
                    <valuemap/>
                </item>
                <item>
                    <name>Nginx $1/sec</name>
                    <type>0</type>
                    <snmp_community/>
                    <multiplier>0</multiplier>
                    <snmp_oid/>
                    <key>nginx[accepted_connections,{$NGINX_STATUS_URL}]</key>
                    <delay>60</delay>
                    <history>30</history>
                    <trends>365</trends>
                    <status>0</status>
                    <value_type>0</value_type>
                    <allowed_hosts/>
                    <units/>
                    <delta>1</delta>
                    <snmpv3_securityname/>
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>
                    <snmpv3_authpassphrase/>
                    <snmpv3_privpassphrase/>
                    <formula>1</formula>
                    <delay_flex/>
                    <params/>
                    <ipmi_sensor/>
                    <data_type>0</data_type>
                    <authtype>0</authtype>
                    <username/>
                    <password/>
                    <publickey/>
                    <privatekey/>
                    <port/>
                    <description/>
                    <inventory_link>0</inventory_link>
                    <applications>
                        <application>
                            <name>Nginx</name>
                        </application>
                    </applications>
                    <valuemap/>
                </item>
                <item>
                    <name>Nginx $1/sec</name>
                    <type>0</type>
                    <snmp_community/>
                    <multiplier>0</multiplier>
                    <snmp_oid/>
                    <key>nginx[handled_connections,{$NGINX_STATUS_URL}]</key>
                    <delay>60</delay>
                    <history>30</history>
                    <trends>365</trends>
                    <status>0</status>
                    <value_type>0</value_type>
                    <allowed_hosts/>
                    <units/>
                    <delta>1</delta>
                    <snmpv3_securityname/>
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>
                    <snmpv3_authpassphrase/>
                    <snmpv3_privpassphrase/>
                    <formula>1</formula>
                    <delay_flex/>
                    <params/>
                    <ipmi_sensor/>
                    <data_type>0</data_type>
                    <authtype>0</authtype>
                    <username/>
                    <password/>
                    <publickey/>
                    <privatekey/>
                    <port/>
                    <description/>
                    <inventory_link>0</inventory_link>
                    <applications>
                        <application>
                            <name>Nginx</name>
                        </application>
                    </applications>
                    <valuemap/>
                </item>
                <item>
                    <name>Number of $1 process</name>
                    <type>0</type>
                    <snmp_community>public</snmp_community>
                    <multiplier>0</multiplier>
                    <snmp_oid>interfaces.ifTable.ifEntry.ifInOctets.1</snmp_oid>
                    <key>proc.num[nginx]</key>
                    <delay>300</delay>
                    <history>30</history>
                    <trends>365</trends>
                    <status>0</status>
                    <value_type>3</value_type>
                    <allowed_hosts/>
                    <units/>
                    <delta>0</delta>
                    <snmpv3_securityname/>
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>
                    <snmpv3_authpassphrase/>
                    <snmpv3_privpassphrase/>
                    <formula>1</formula>
                    <delay_flex/>
                    <params/>
                    <ipmi_sensor/>
                    <data_type>0</data_type>
                    <authtype>0</authtype>
                    <username/>
                    <password/>
                    <publickey/>
                    <privatekey/>
                    <port/>
                    <description/>
                    <inventory_link>0</inventory_link>
                    <applications/>
                    <valuemap/>
                </item>
            </items>
            <discovery_rules/>
            <macros>
                <macro>
                    <macro>{$NGINX_STATUS_URL}</macro>
                    <value>http://127.0.0.1:10061/nginx_status</value>
                </macro>
            </macros>
            <templates/>
            <screens/>
        </template>
    </templates>
    <triggers>
        <trigger>
            <expression>{Template_Nginx:proc.num[nginx].last(0)}=0</expression>
            <name>Nginx is not running on {HOSTNAME}</name>
            <url/>
            <status>0</status>
            <priority>4</priority>
            <description>Nginx is not running.&#13;
      &#13;
      It has been stopped / shutdown or has crashed. &#13;
      Check on the server for more details:&#13;
        - w / last&#13;
        - dmesg logs&#13;
        - /var/log/messages&#13;
        - nginx error logs</description>
            <type>0</type>
            <dependencies/>
        </trigger>
    </triggers>
    <graphs>
        <graph>
            <name>Nginx - Connections and Requests status</name>
            <width>900</width>
            <height>200</height>
            <yaxismin>0.0000</yaxismin>
            <yaxismax>100.0000</yaxismax>
            <show_work_period>0</show_work_period>
            <show_triggers>0</show_triggers>
            <type>0</type>
            <show_legend>1</show_legend>
            <show_3d>0</show_3d>
            <percent_left>0.0000</percent_left>
            <percent_right>0.0000</percent_right>
            <ymin_type_1>1</ymin_type_1>
            <ymax_type_1>0</ymax_type_1>
            <ymin_item_1>0</ymin_item_1>
            <ymax_item_1>0</ymax_item_1>
            <graph_items>
                <graph_item>
                    <sortorder>0</sortorder>
                    <drawtype>1</drawtype>
                    <color>FF9999</color>
                    <yaxisside>0</yaxisside>
                    <calc_fnc>4</calc_fnc>
                    <type>0</type>
                    <item>
                        <host>Template_Nginx</host>
                        <key>nginx[accepted_connections,{$NGINX_STATUS_URL}]</key>
                    </item>
                </graph_item>
                <graph_item>
                    <sortorder>1</sortorder>
                    <drawtype>2</drawtype>
                    <color>990000</color>
                    <yaxisside>0</yaxisside>
                    <calc_fnc>4</calc_fnc>
                    <type>0</type>
                    <item>
                        <host>Template_Nginx</host>
                        <key>nginx[handled_connections,{$NGINX_STATUS_URL}]</key>
                    </item>
                </graph_item>
                <graph_item>
                    <sortorder>2</sortorder>
                    <drawtype>0</drawtype>
                    <color>009900</color>
                    <yaxisside>0</yaxisside>
                    <calc_fnc>4</calc_fnc>
                    <type>0</type>
                    <item>
                        <host>Template_Nginx</host>
                        <key>nginx[handled_requests,{$NGINX_STATUS_URL}]</key>
                    </item>
                </graph_item>
            </graph_items>
        </graph>
        <graph>
            <name>Nginx - Threads status</name>
            <width>900</width>
            <height>200</height>
            <yaxismin>0.0000</yaxismin>
            <yaxismax>100.0000</yaxismax>
            <show_work_period>0</show_work_period>
            <show_triggers>0</show_triggers>
            <type>1</type>
            <show_legend>1</show_legend>
            <show_3d>0</show_3d>
            <percent_left>0.0000</percent_left>
            <percent_right>0.0000</percent_right>
            <ymin_type_1>1</ymin_type_1>
            <ymax_type_1>0</ymax_type_1>
            <ymin_item_1>0</ymin_item_1>
            <ymax_item_1>0</ymax_item_1>
            <graph_items>
                <graph_item>
                    <sortorder>0</sortorder>
                    <drawtype>1</drawtype>
                    <color>990000</color>
                    <yaxisside>0</yaxisside>
                    <calc_fnc>4</calc_fnc>
                    <type>0</type>
                    <item>
                        <host>Template_Nginx</host>
                        <key>nginx[writing,{$NGINX_STATUS_URL}]</key>
                    </item>
                </graph_item>
                <graph_item>
                    <sortorder>1</sortorder>
                    <drawtype>1</drawtype>
                    <color>999900</color>
                    <yaxisside>0</yaxisside>
                    <calc_fnc>4</calc_fnc>
                    <type>0</type>
                    <item>
                        <host>Template_Nginx</host>
                        <key>nginx[reading,{$NGINX_STATUS_URL}]</key>
                    </item>
                </graph_item>
                <graph_item>
                    <sortorder>2</sortorder>
                    <drawtype>1</drawtype>
                    <color>009900</color>
                    <yaxisside>0</yaxisside>
                    <calc_fnc>4</calc_fnc>
                    <type>0</type>
                    <item>
                        <host>Template_Nginx</host>
                        <key>nginx[waiting,{$NGINX_STATUS_URL}]</key>
                    </item>
                </graph_item>
            </graph_items>
        </graph>
    </graphs>
</zabbix_export>


本文出自 “清风明月” 博客,请务必保留此出处http://liqingbiao.blog.51cto.com/3044896/1894902

zabbix监控nginx性能状态

标签:zabbix   nginx web   

原文地址:http://liqingbiao.blog.51cto.com/3044896/1894902

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!