标签：lvs nginx 负债均衡 tomcat优化 nginx静动分离
最近公司重整架构,前端使用LVS做负债均衡,虽然之前也研究过集群,看过很多LVS原理和架构,但毕竟没有真正操作过，以下案例是在虚拟机中完成实验，记录一下，方便以后在服务器中实施搭建。
架构图如下：
前提介绍：本案例采用Centos7+Keepalived1.3.5+Tomcat9+Mysql5.6+Redis3.2.8+Rabbitmq3.6.10
集群实现的功能有：
1）：实现单点访问，利用keepalived的vip实现对不同的后端服务器进行访问；
2）：健康检查，利用keepalived的多种检查机制TCP、URL、HTTP_GET等；
3）：利用nginx实现负债均衡和静动分离；
4）：tomcat链接mysql做集群的web服务器，结合nginx静动分离，实现高可用集群；
5）：rabbitmq实现消息队列；
6）：redis实现数据库缓存服务器；
7）：mysql数据存储。（这里没做集群，业务不大）
一：集群搭建
前端LVS采用俩主互为备模式，Nginx做后端web服务器负债均衡，Mysql做数据存储使用主从模式切换。
服务器环境如下
nginx+LVS_master 192.168.23.14
nginx+LVS_backup 192.168.23.15
VIP 192.168.23.20
Tomcat1 192.168.23.16
2 192.168.23.17
3 192.168.23.18
rabbitmq 192.168.23.19
redis 192.168.23.21
Mysql 192.168.23.22
安装说明：前端俩台keepalived+nginx做前段服务器每个服务器各俩块网卡，有路由器的情况下，只需要路由器提供的外网ip即可，使用路由器映射功能，这样集群就变为全部是后端服务器，提高服务器的安全性。
1：安装前调试以及优化
安装centos7选择默认最小化
停止firewalld运行
systemctl stop firewalld.service
systemctl disable firewalld.service
内核优化vi/etc/sysctl.conf
##关闭ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
##路由转发
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
防火墙安装iptables
yum -y installiptables-services iptables
修改selinux
vi/etc/selinux/config 修改selinux为disabled
重启reboot
2：安装master和backup
安装前准备
yum -y installopenssl openssl-devel popt popt-devel gcc* wget vim ipvsadm pcre pcre-devel
是否运行ipvsadm
查看内核模块是否存在lsmod|grep ip_vs
建立data目录
makdir /data
下载wgethttp://www.keepalived.org/software/keepalived-1.3.5.tar.gz
tar zxfkeepalived-1.3.5.tar.gz -C /data
./configure--prefix=/usr/local/keepalived-1.3.5 && make && make install
复制keepalived文件和脚本
cp/usr/local/keepalived-1.3.5/sbin/keepalived /usr/sbin/
cp/usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp/usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp/data/keepalived-1.3.5/./keepalived/etc/init.d/keepalived /etc/init.d/
chmod 755/etc/init.d/keepalived
mkdir/etc/keepalived/
设置外网的VIP
ifconfig ens33:0 ip 192.168.23.20 netmask 255.255.255.0 up
重启网卡
systemctl restart network.service
设置开启启动
chkconfig --add keepalived
chkconfig --level 345 keepalived on
设置/etc/keepalived/文件
! Configuration File for keepalived
global_defs {
router_id LVS1
}
vrrp_scriptchk_http_port { #编写nginx脚本在etc/keepalived/目录，下面会给出脚本
script "/etc/keepalived/nginx.sh" #监控服务脚步
interval 2 #检测时间间隔
weight 2
}
vrrp_instance VI_1 {
state MASTER #master服务器名称
interface ens33
virtual_router_id 51
priority 100 #优先级越大越高
advert_int 1
authentication {
auth_type PASS
auth_pass lmnginx #与backup的认证密码
}
virtual_ipaddress {
192.168.23.20
}
}
virtual_server 192.168.23.20 80 {
delay_loop 1
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.23.14 80 {
weight 1
TCP_CHECK {
connect_timeout 3
retry 2
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.23.15 80 {
weight 1
TCP_CHECK {
connect_timeout 3
retry 3
delay_before_retry 3
connect_port 80
}
}
}
keepalived配置文件backup设置
注：backup和master配置一致，这里不在重复介绍，主要介绍配置文件的差异
router_id LVS2
state BACKUP
interface ens37 （改为vip所在的网卡设置名）
priority 99
nginx监控脚本
#!/bin/sh
nginx=`ps -C nginx --no-header |wc -l`
if [ $nginx -eq 0 ];then
killall keepalived
/usr/local/keepalived/sbin/keepalived -D
else
exit 0
fi
keepalived监控脚本。
#!/bin/bash
master=`ps -C keepalived --no-header |wc -l`
ip=`ifconfig ens33 |grep "inet" |awk -F " " ‘{print $2}‘`
error=/etc/keepalived/error.txt
cb=`df -hT |grep "root" |awk -F " " ‘{print $6}‘`
free=`free -m |grep Mem |awk -F " " ‘{print $3}‘`
cpu=`uptime |awk -F "," ‘{print $3,$4,$5}‘`
date=`date +%F-%r`
while true
do
if [ $master -eq 0 ]; then
echo -e "$date $ip master down\n已用磁盘空间$cb\n已用cpu格式为:1分钟,5分钟,15分钟.大于0.7负债率请联系管理员$cpu\n已用内存空间$free m" >$error
echo "####################################keepalived报错信息如下 #######################################" >>$error
/etc/init.d/keepalived status >>$error
cat $error |/bin/mail -s master mynamelim@163.com >/dev/null 2>&1
else
echo "$ip $date Check keepalived correctly" >$error
fi
sleep 10
done
加入计划任务，每分钟执行一次脚本
附：mail设置
安装mail
yum -y install mailx
设置mail文件
vim /etc/mail.rc
set from=***@163.com（你的邮箱名，建议和账号一致） smtp=smtp.163.com （邮箱的stmp服务器）
set smtp-auth-user=***@163.com（你的邮箱账号） smtp-auth-password=password（你的邮箱密码，这个密码是stmp认证用户的密码，不懂可以百度） smtp-auth=login （确定登陆）
3：nginx编译安装
安装依赖包
yum install -y vim wget lsof gcc gcc-c++ bzip2 net-tools bind-utilszlib-devel openssl-devel libxml2-devel libxslt-devel perl-devel perl-ExtUtils-Embed
wget
创建nginx用户
groupadd nginx
useradd -s /sbin/nologin -g nginx -M nginx
安装Pcre
下载 wget http://www.pcre.org/pcre-8.41.tar.gz
cd /data
tar zxf pcre.tar.gz
./configure --prefix=/usr/local/pcre &&make && make install
安装openssl
下载 wget https://www.openssl.org/source/openssl-1.1.0f.tar.gz
cd /data
tar zxf openssl.tar.gz
./config --prefix=/usr/local/openssl &&make && make install
安装zlib
下载 wget www.zlib.net/zlib-1.2.11.tar.gz
cd /data
tar zxf zlib.tar.gz
./configure --prefix=/usr/local/zlib &&make && make install
创建目录
mkdir -p/var/tmp/nginx/{client,proxy,fastcgi,uwsgi,scgi}
mkdir /var/run/nginx
mkdir /var/log/nginx
mkdir /etc/nginx
安装nginx
下载 wget http://nginx.org/download/nginx-1.13.3.tar.gz
./configure --prefix=/usr/local/nginx--sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf--error-log-path=/var/log/nginx/error.log--http-log-path=/var/log/nginx/access.log --lock-path=/var/lock/nginx.lock--user=nginx --group=nginx --with-http_ssl_module --with-http_dav_module--with-http_flv_module --with-http_realip_module --with-http_addition_module--with-http_xslt_module --with-http_stub_status_module --with-http_sub_module--with-http_random_index_module --with-http_degradation_module--with-http_secure_link_module --with-http_gzip_static_module--with-http_perl_module --with-http_v2_module --with-threads --with-stream--with-stream_ssl_module --with-pcre=/data/pcre-8.41--with-zlib=/data/zlib-1.2.11 --with-openssl=/data/openssl-1.1.0f--with-file-aio --with-mail --with-mail_ssl_module--http-client-body-temp-path=/var/tmp/nginx/client_body--http-proxy-temp-path=/var/tmp/nginx/proxy--http-fastcgi-temp-path=/var/tmp/nginx/fastcgi --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi--http-scgi-temp-path=/var/tmp/nginx/scgi --with-ld-opt="-Wl,-E"
make && make install
配置nginx
实现静动分离和负债均衡
user nginx;
worker_processes auto;
events {
worker_connections 65535;
use epoll;
multi_accept on;
}
http {
include mime.types;
include conf.d/*;
default_type application/octet-stream;
log_format main ‘$remote_addr - $remote_user [$time_local] "$request" ‘
‘$status $body_bytes_sent "$http_referer" ‘
‘"$http_user_agent" "$http_x_forwarded_for"‘;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
charset UTF-8;
keepalive_timeout 65;
gzip on;
gzip_proxied any;
gzip_min_length 1000;
gzip_comp_level 4;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
upstream www.limeng.com {
server 192.168.23.16;
server 192.168.23.17;
server 192.168.23.18;
}
server {
listen 80;
server_name www.limeng.com;
location / {
index index;
}
location ~ .*\.(js|css|ico|png|jpg|eot|svg|ttf|woff) {
index index.html index.htm;
proxy_pass http://www.limeng.com;
proxy_cache cache_one;
proxy_cache_valid 200 302 304 5d;
proxy_cache_valid any 5d;
proxy_cache_key ‘$host:$server_port$request_uri‘
add_header X-Cache ‘$upstream_cache_status from $host‘
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
expires 7d;
}
location ~ .*$ {
index index.jsp;
proxy_pass http://www.limeng.com;
}
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
注：俩台nginx配置一样
nginx启动（脚本自行谷歌）
chmod 755 /etc/init.d/nginx
启动nginx
/etc/init.d/nginx start
Nginx安装完成！
4：后端web服务器Tomcat安装
jdk安装
安装前准备：
yum -y install gcc*wget vim
检查命令
java -version
java -version 有就卸载rpm -e --nodeps jdk
解压安装
wget http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1502180263_4da80e17ef2af74cdbd6c02a779d1db6
tar -zxvf jdk-8u144-linux-x64.tar.gz
移动目录
mv ./jdk1.8.0_121/usr/local/java/
最后再删除安装文件：
rm -rf jdk-8u144-linux-x64.gz
配置环境变量
vi /etc/profile加入以下变量
exportJAVA_HOME=/usr/local/java/
exportCLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
exportPATH=$PATH:$JAVA_HOME/bin
加载变量
source /etc/profile
tomcat安装
解压
wget http://mirror.bit.edu.cn/apache/tomcat/tomcat-9/v9.0.0.M22/bin/apache-tomcat-9.0.0.M18.tar.gz
tar -zxvf apache-tomcat-9.0.0.M18.tar.gz
解压完整后删除安装包
rm -rf apache-tomcat-9.0.0.M18.tar.gz
移动目录
mkdir /usr/local/tomcat/
mv ./apache-tomcat-9.0.0.M18//usr/local/tomcat/
首先我们来配置一下tomcat的环境变量和内存设置，进入tomcat的bin目录后通过vim命令打开catalina.sh文件，并在其中加入如下配置：
JAVA_OPTS="-Dfile.encoding=UTF-8 -server -Xms1024m -Xmx1024m -Xmn512m
-XX:SurvivorRatio=10
-XX:MaxTenuringThreshold=15
-XX:NewRatio=2
-XX:+DisableExplicitGC"
export TOMCAT_HOME=/tmp/tomcat/apache-tomcat-9.0.0.M18
export CATALINA_HOME=/tmp/tomcat/apache-tomcat-9.0.0.M18
export JRE_HOME=/tmp/java/jdk1.8.0_121
export JAVA_HOME=/tmp/java/jdk1.8.0_121
server.xml配置优化
maxThreads="1000"
minSpareThreads="100"
maxSpareThreads="1000"
acceptCount="1000"
URIEncoding="utf-8"
server="Neo App Srv 1.0"
connectionTimeout="20000"
redirectPort="8443" />
启动
./startup.sh
5：mysql安装配置
安装前准备
yum -y install makegcc cmake bison-devel ncurses-devel libaio libaio-devel per-Data-Dumpernet-tools gcc* wget vim autoconf
查看是否有mysql的rpm包
rpm -qa |grep mysql
rpm -e --nodeps mysql
查看是否有mariadb-libs的rpm包
rpm -qa |grep mariadb-libs
yum -y install mariadb-libs
解压文件
wget https://www.mysql.com/downloads/mysql-5.6.23.tar.gz
tar -zxf /data/mysql-5.6.23.tar.gz
编译安装
cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql-DMYSQL_DATADIR=/usr/local/mysql/data -DSYSCONFDIR=/etc-DWITH_MYISAM_STORAGE_ENGINE=1 -DWITH_INNOBASE_STORAGE_ENGINE=1-DWITH_MEMORY_STORAGE_ENGINE=1 -DWITH_READLINE=1 -DMYSQL_UNIX_ADDR=/var/lib/mysql/mysql.sock-DMYSQL_TCP_PORT=3306 -DENABLED_LOCAL_INFILE=1-DWITH_PARTITION_STORAGE_ENGINE=1 -DEXTRA_CHARSETS=all -DDEFAULT_CHARSET=utf8-DDEFAULT_COLLATION=utf8_general_ci
make &&make install
建立用户和组
groupadd mysql
useradd mysql -s/sbin/nologin -g mysql
给权限
chown -Rmysql:mysql /usr/local/mysql/
mysql下的初始化
/usr/local/mysql/scripts/mysql_install_db--user=mysql --basedir=/usr/local/mysql/
拷贝配置文件my.cnf
cd support-files/
cp my-default.cnf /etc/my.cnf
配置mysql
[client]
port=3306
socket=/var/lib/mysql/mysql.sock
[mysql]
no-auto-rehash
[mysqld]
user=mysql
port=3306
basedir=/usr/local/mysql
datadir=/var/lib/mysql
open_files_limit=10240
back_log=600
max_connections = 3000
#max_connect_errors = 6000
external-locking = FALSE
max_allowed_packet = 32M
thread_cache_size = 300
query_cache_size = 64M
query_cache_limit = 4M
key_buffer_size = 2048M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
innodb_lock_wait_timeout = 120
skip-name-resolve
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd
[mysqld_safe]
log-error=/var/log/mysql_error.log
pid-file=/var/run/mysql/mysql.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
添加变量
echo "exportPATH=$PATH:/usr/local/mysql/bin" >> /etc/profile
source /etc/profile
添加开机启动
cp/usr/local/mysql/./support-files/mysql.server /etc/rc.d/init.d/mysqld
chmod +x/etc/init.d/mysqld
chkconfig --addmysqld
chkconfig mysqld on
启动mysql
systemctl startmysqld.service
查看端口
netstat -anpt |grepmysql
编辑etc下的my.cnf文件
设置mysql密码
mysqladmin -uroot password
7：redis安装
解压安装
wget https://redis.io/download/redis-3.2.8.tar.gz
tar -zxf /data/redis-3.2.8.tar.gz
make
make install/usr/local/redis
执行完，进入目录/usr/local/redis
建立文件夹并拷贝
mkdir /etc/redis
cp -rf/usr/local/redis/redis.conf /etc/redis/6379.conf
编辑配置文件vim/etc/redis/6379.conf
daemonize yes
bind 192.168.23.21 127.0.0.1 (链接ip和本地）
logfile "/var/logs/redis.log"(自定义)
databases 16 (默认数据库数量）
dbfilename dump.rdb （默认存储名）
dir/tmp/redis/data/redisdata （默认数据存放目录）
requirepassyour password (自己设置redis的密码)
启动redis
redis_server /usr/local/redis/redis.conf
8：rabbitmq安装
安装前准备
yum -y install makegcc gcc-c++ kernel-devel m4 ncurses-devel openssl-devel
解压 zlang
tar -zxf /data/otp_src_19.3.tar.gz
cd otp_src_19.3
编译安装
./configure--prefix=/usr/local/erlang --with-ssl -enable-threads -enable-smmp-support-enable-kernel-poll --enable-hipe --without-javac && make &&make install
解压rtabbitmq
下载 wget http://www.rabbitmq.com/download.html/rabbitmq-server-generic-unix-3.6.10.tar.xz
tar -xf rabbitmq-server-generic-unix-3.6.10.tar.xz
移动位置
mv rabbitmq_server-3.6.10/ /usr/local/rabbitmq-3.6.10/
cd rabbitmq-3.6.10/sbin
启动rabbitmq
./rabbitmq-plugins enabler abbitmq_management
./rabbitmq-server -detached（后台启动）
停止
./rabbitmqctl stop
查看程序和端口运行状态
ps aux |grep rabbitmq
netstat -anpt |grep 15672
添加账户并设置为管理员
./rabbitmqctl add_user admin password
./rabbitmq set_tagsadmin administrator
删除默认用户guest
./rabbitmqctl delete_user guest
查看设置用户是否存在
./rabbitmqctl list_users
二：测试集群
在界面打开浏览器测试，访问192.168.23.20，如下：
期间不断的刷新，会出现不同的后端web页面。
关闭master测试，访问正常，backup接替工作
systemctl stop keepalived
邮件报警
本文出自 “麻花的博客” 博客，请务必保留此出处http://mahua.blog.51cto.com/11572858/1954544
标签：lvs nginx 负债均衡 tomcat优化 nginx静动分离
原文地址：http://mahua.blog.51cto.com/11572858/1954544
