码迷,mamicode.com
首页 > 其他好文 > 详细

lvs+keepalived+nginx实现高性能负载均衡集群(转)

时间:2019-01-22 19:48:10      阅读:133      评论:0      收藏:0      [点我收藏+]

标签:overruns   efs   ldl   alived   rom   mask   enc   mtu   after   

一、为什么要使用负载均衡技术?

1、系统高可用性

2、  系统可扩展性

3、  负载均衡能力

LVS+keepalived能很好的实现以上的要求,LVS提供负载均衡,keepalived提供健康检查,故障转移,提高系统的可用性!采用这样的架构以后很容易对现有系统进行扩展,只要在后端添加或者减少realserver,只要更改lvs的配置文件,并能实现无缝配置变更!

二、LVS+Keepalived介绍

1、  LVS

LVS是一个开源的软件,可以实现LINUX平台下的简单负载均衡。LVS是Linux Virtual Server的缩写,意思是Linux虚拟服务器。目前有三种IP负载均衡技术(VS/NAT、VS/TUN和VS/DR);八种调度算法(rr,wrr,lc,wlc,lblc,lblcr,dh,sh)。

2、  keepalived

Keepalived 是运行在lvs 之上,它的主要功能是实现真实机的故障隔离及负载均衡器间的失败切换,提高系统的可用性

三、环境:

四台服务器,系统全为CentOS6.8:

192.168.2.203 master lvs+keepalived

192.168.2.202 backup lvs+keepalived

192.168.2.204 web1(nginx)

192.168.2.205 web2 (nginx)

vip:192.168.2.13

其中nginx已预装好,这里不再写搭建过程

 

四、搭建并配置

1、分别在backup lvs和master lvs上安装lvs

技术分享图片
 1 root@bogon src]# yum -y install ipvsadm
 2 已加载插件:fastestmirror
 3 设置安装进程
 4 Determining fastest mirrors
 5 epel/metalink                                                                                   | 5.4 kB     00:00     
 6  * base: mirror.lzu.edu.cn
 7 ... ...
 8 已安装:
 9   ipvsadm.x86_64 0:1.26-4.el6                                                                                          
10 
11 作为依赖被安装:
12   libnl.x86_64 0:1.1.4-2.el6                                                                                           
13 
14 完毕!
技术分享图片

2、把ipvsadm模块加载进系统

技术分享图片
1 [root@bogon src]# ipvsadm
2 IP Virtual Server version 1.2.1 (size=4096)
3 Prot LocalAddress:Port Scheduler Flags
4   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
5 [root@bogon src]# lsmod | grep ip_vs
6 ip_vs                 126897  0 
7 libcrc32c               1246  1 ip_vs
8 ipv6                  336282  270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6
技术分享图片

3、分别在backup lvs和master lvs上安装keepalived(keepalived官网:http://www.keepalived.org/)

技术分享图片
[root@bogon src]# tar zxf keepalived-1.2.24.tar.gz 
[root@bogon src]# cd keepalived-1.2.24
[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/
报错:
configure: error: 
  !!! OpenSSL is not properly installed on your system. !!!
  !!! Can not include OpenSSL headers files.            !!!
技术分享图片

系统缺少openssl-devel包所致

安装openssl-devel

root@bogon keepalived-1.2.24]# yum -y install openssl-devel

再次编辑安装

技术分享图片
[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/
Keepalived configuration
------------------------
Keepalived version       : 1.2.24
Compiler                 : gcc
Preprocessor flags       : -I/lib/modules/2.6.32-642.3.1.el6.x86_64//include
Compiler flags           : -Wall -Wunused -Wstrict-prototypes
Linker flags             : 
Extra Lib                : -ldl -lssl -lcrypto 
Use IPVS Framework       : Yes
IPVS use libnl           : No
IPVS syncd attributes    : No
IPVS 64 bit stats        : No
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
Use VRRP authentication  : Yes
With ip rules/routes     : Yes
SNMP keepalived support  : No
SNMP checker support     : No
SNMP RFCv2 support       : No
SNMP RFCv3 support       : No
SHA1 support             : No
Use Debug flags          : No
Stacktrace support       : No
Memory alloc check       : No
libnl version            : None
Use IPv4 devconf         : No
Use libiptc              : No
Use libipset             : No
Build genhash            : Yes
Build documentation      : No

[root@bogon keepalived-1.2.24]# make && make install
技术分享图片
[root@bogon keepalived-1.2.24]# ln -s /usr/local/sbin/keepalived /sbin/
[root@bogon keepalived-1.2.24]# chkconfig --add keepalived
[root@bogon keepalived-1.2.24]# chkconfig --level 35 keepalived on

4、配置keepalived

lvs-master的配置文件如下

技术分享图片
[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {      #全局配置部分
#   notification_email {      #email 通知,基本不用此处所以注释掉
#     acassen@firewall.loc
#     failover@firewall.loc
#     sysadmin@firewall.loc
#  }
#   notification_email_from Alexandre.Cassen@firewall.loc
#   smtp_server 192.168.200.1
#   smtp_connect_timeout 30
   router_id LVS_DEVEL    # 设置lvs的id,在一个网络内应该是唯一的
   vrrp_skip_check_adv_addr  
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {    #vrrp实例定义部分
    state MASTER     #设置lvs的状态,报错MASTER和BACKUP两种,必须大写
    interface eth1    #设置对外服务的接口
    virtual_router_id 60  #设置虚拟路由标示,这个标示是一个数字,同一个vrrp实例使用唯一标示
    priority 100    #定义优先级,数字越大优先级越高,在一个vrrp——instance下,master的优先级必须大于backup
    advert_int 1    #设定master与backup负载均衡器之间同步检查的时间间隔,单位是秒
    authentication {  #设置验证类型和密码
        auth_type PASS  #主要有PASS和AH两种
        auth_pass 1111  #验证密码,同一个vrrp_instance下MASTER和BACKUP密码必须相同
    }
    virtual_ipaddress {  #设置虚拟ip地址,可以设置多个,每行一个
        192.168.2.13
    }
}

virtual_server 192.168.2.13 80 {  #设置虚拟服务器,需要指定虚拟ip和服务端口
    delay_loop 3    #健康检查时间间隔
    lb_algo rr     #负载均衡调度算法
    lb_kind DR    #负载均衡转发规则
    persistence_timeout 50    #设置会话保持时间,对动态网页非常有用
    protocol TCP    #指定转发协议类型,有TCP和UDP两种

    real_server 192.168.2.204 80 {  #配置服务器节点1,需要指定real server的真实IP地址和端口
        weight 1    #设置权重,数字越大权重越高
    TCP_CHECK {     #realserver的状态监测设置部分单位秒
            connect_timeout 3    #超时时间
            nb_get_retry 3      #重试次数
            delay_before_retry 3    #重试间隔
         connect_port 80      #监测端口
    }
}
    real_server 192.168.2.205 80 {
    weight 1
    TCP_CHECK {
    connect_timeout 3
    nb_get_retry 3
    delay_before_retry 3
    connect_port 80
        }
    } 

}
技术分享图片

 

LVS-backup的配置文件如下

技术分享图片
[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
#   notification_email {
#     acassen@firewall.loc
#     failover@firewall.loc
#     sysadmin@firewall.loc
#  }
#   notification_email_from Alexandre.Cassen@firewall.loc
#   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 60
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.2.13
    }
}

virtual_server 192.168.2.13 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    persistence_timeout 3
    protocol TCP

    real_server 192.168.2.204 80 {
        weight 1
    TCP_CHECK { 
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
         connect_port 80    
    }
}
    real_server 192.168.2.205 80 {
    weight 1
    TCP_CHECK {
    connect_timeout 3
    nb_get_retry 3
    delay_before_retry 3
    connect_port 80
        }
    } 

}
技术分享图片

 

5、realserver的配置

两台web服务器都要执行下面脚本

技术分享图片
[root@bogon www]# cat /etc/rc.d/init.d/realserver.sh 
#!/bin/bash
# description: Config realserver lo and apply noarp
 
SNS_VIP=192.168.2.13
 
/etc/rc.d/init.d/functions
 
case "$1" in
start)
       ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
       /sbin/route add -host $SNS_VIP dev lo:0
       echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
       sysctl -p >/dev/null 2>&1
       echo "RealServer Start OK"
 
       ;;
stop)
       ifconfig lo:0 down
       route del $SNS_VIP >/dev/null 2>&1
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
       echo "RealServer Stoped"
       ;;
*)
       echo "Usage: $0 {start|stop}"
       exit 1
esac
 
exit 0
技术分享图片
技术分享图片
[root@bogon www]# /etc/rc.d/init.d/realserver.sh start
/etc/rc.d/init.d/realserver.sh: line 6: /etc/rc.d/init.d/functions: 权限不够
RealServer Start OK
[root@bogon www]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:41:71:DF  
          inet addr:192.168.12.129  Bcast:192.168.12.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe41:71df/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:728 errors:0 dropped:0 overruns:0 frame:0
          TX packets:98 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:137311 (134.0 KiB)  TX bytes:7369 (7.1 KiB)

eth1      Link encap:Ethernet  HWaddr 00:0C:29:41:71:E9  
          inet addr:192.168.2.204  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe41:71e9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:119838 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31612 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:23411786 (22.3 MiB)  TX bytes:2119106 (2.0 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:182 (182.0 b)  TX bytes:182 (182.0 b)

lo:0      Link encap:Local Loopback  
          inet addr:192.168.2.13  Mask:255.255.255.255
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
技术分享图片

6、启动keepalived并进行测试

[root@bogon keepalived-1.2.24]# service keepalived start
正在启动 keepalived:                                      [确定]

lvs-master

技术分享图片
[root@bogon keepalived-1.2.24]# tail -f /var/log/messages 
Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:46 bogon Keepalived_healthcheckers[6596]: Netlink reflector reports IP 192.168.2.13 added
Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
技术分享图片
技术分享图片
[root@bogon keepalived-1.2.24]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.2.13:80 rr persistent 3
  -> 192.168.2.204:80             Route   1      0          0         
  -> 192.168.2.205:80             Route   1      0          0      
技术分享图片

访问curl http://192.168.2.13/test.txt

[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2

关掉web2再次测试

[root@www etc]# curl http://192.168.2.13/test.txt
it is web1
[root@www etc]# curl http://192.168.2.13/test.txt
it is web1

查看lvs-master

技术分享图片
[root@bogon keepalived-1.2.24]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.2.13:80 rr persistent 3
  -> 192.168.2.204:80             Route   1      0          2         
[root@bogon keepalived-1.2.24]# tail -f /var/log/messages 
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.
Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.
技术分享图片

已经自动把web2剔除

打开web2再次查看

技术分享图片
[root@bogon keepalived-1.2.24]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.2.13:80 rr persistent 3
  -> 192.168.2.204:80             Route   1      0          0         
  -> 192.168.2.205:80             Route   1      0          0         
[root@bogon keepalived-1.2.24]# tail -f /var/log/messages 
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.
Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.
Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 success.
Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Adding service [192.168.2.205]:80 to VS [192.168.2.13]:80
Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.
技术分享图片

恢复后已自动添加进来

关掉lvs master的keepalived

[root@bogon keepalived-1.2.24]# service keepalived stop
停止 keepalived:                                          [确定]

访问web并查看lvs backup

[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
技术分享图片
[root@lys2 src]# tail -f /var/log/messages
Oct 23 19:03:26 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Transition to MASTER STATE
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Entering MASTER STATE
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) setting protocol VIPs.
Oct 23 19:03:27 lys2 Keepalived_healthcheckers[13123]: Netlink reflector reports IP 192.168.2.13 added
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
技术分享图片
技术分享图片
[root@lys2 src]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:89:0f:e3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.144.101/24 brd 192.168.144.255 scope global eth0
    inet6 fe80::20c:29ff:fe89:fe3/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:89:0f:ed brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.202/24 brd 192.168.2.255 scope global eth1
    inet 192.168.2.13/32 scope global eth1
    inet6 fe80::20c:29ff:fe89:fed/64 scope link 
       valid_lft forever preferred_lft forever
技术分享图片

 

可以看到lvs backup已自动切换成master状态并自动绑定了vip

查看lvs master vip

技术分享图片
[root@bogon keepalived-1.2.24]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:55:4d:7a brd ff:ff:ff:ff:ff:ff
    inet 192.168.12.128/24 brd 192.168.12.255 scope global eth0
    inet6 fe80::20c:29ff:fe55:4d7a/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:55:4d:84 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.203/24 brd 192.168.2.255 scope global eth1
    inet6 fe80::20c:29ff:fe55:4d84/64 scope link 
       valid_lft forever preferred_lft forever
技术分享图片

已自动解除vip

 

到此全部结束

 

lvs+keepalived+nginx实现高性能负载均衡集群(转)

标签:overruns   efs   ldl   alived   rom   mask   enc   mtu   after   

原文地址:https://www.cnblogs.com/lyz108/p/10305509.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!