LAMP+haproxy+varnish实现网站访问的动静分离及静态资源缓存

标签：lamp   varnish   haproxy   缓存   动静分离   

实验目标：
1.    LAMP节点提供用户动态请求访问，数据库单独有数据库节点提供；
2.    LAMP动态网站有两台服务器，提供负载均衡；
3.    静态网站服务器节点提供用户的静态资源请求访问；存在两台静态web服务器，其网站静态资源在静态服务器上存放；
4.    用户的静态请求访问后缓存在varnish服务器上，实现访问加速
5.    前端的haproxy提供反向代理功能，将用户的动态资源请求发送给后端LAMP节点，静态资源请求发往后端静态web服务器；
6.    该架构考虑还不健全，如静态内容的一致性，数据库的单点故障，只是一个不成熟的架构实现简单的动静分离及缓存服务器实现；

实现过程：

一. LAMP构建
================ ha.stu31.com : 172.16.31.10 ================
vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.31.10 ha.stu31.com ha
172.16.31.11 node1.stu31.com node1
172.16.31.12 node2.stu31.com node2
172.16.31.13 node3.stu31.com node3
172.16.31.14 node4.stu31.com node4
172.16.31.15 mysql.stu31.com mysql
172.16.31.16 node5.stu31.com node5
172.16.31.17 node6.stu31.com node6
=======
ssh-keygen -t rsa -P "" -f "/root/.ssh/id_rsa"
for i in {1..6}; do ssh-copy-id -i .ssh/id_rsa.pub node$i; done
for i in {1..6}; do scp /etc/hosts node$i:/etc/hosts; done
for i in {1..6}; do ssh node$i "ntpdate -u 202.112.10.60"; done
for i in {1..6}; do ssh node$i date; done
for i in {1..6}; do ssh node$i "mount /dev/cdrom /mnt/cdrom"; done
for i in {1..2}; do ssh node$i "yum -y install httpd php  mysql  php-mysql"; done
for i in {1..2}; do ssh node$i "service httpd start"; done
vim index.php
<?php
       phpinfo();
?>
///////////
for i in {1..2}; do scp index.php root@node$i:/var/www/html; done


====================== windows ======================
浏览器访问 http://172.16.31.11/index.php
浏览器访问 http://172.16.31.12/index.php



============== mysql.stu31.com : 172.16.31.15 ================
mount /dev/cdrom /mnt/cdrom/
yum -y install mysql-server mysql
service mysqld start
mysql
mysql> create schema bbsdb;
mysql> grant all on bbsdb.* to ‘bbsadmin‘@‘172.16.%.%‘identified by ‘oracle‘;
mysql> flush privileges;
mysql> quit


============== node1.stu31.com : 172.16.31.11 ================
wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip
unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
cd /var/www/html/upload
chmod -R go+w config/ data uc_*


====================== windows ======================
浏览器访问 http://172.16.31.11/upload/install
1. 检查安装环境：
2. 设置运行环境：点击选择全新安装
3. 安装数据库
填写数据库信息
数据库服务器: 172.16.31.15
数据库名: bbsdb
数据库用户名: bbsadmin
数据库密码: oracle
数据表前缀: pre_
系统邮箱: admin@stu31.com
填写管理员信息
管理员账号: admin
管理员密码: admin
重复密码: admin
管理员Email: admin@stu31.com
设置好数据库后点击安装即可


============== node2.stu31.com : 172.16.31.12 ===============

1. cd /var/www/html; scp -r upload/ readme/ utility/ node2:/var/www/html/
2. 重新解压软件包到node2，重新安装一遍，删除数据库重新安装(因为数据库中无重要内容)。



访问测试
====================== windows ======================
http://172.16.31.11/upload/forum.php
http://172.16.31.12/upload/forum.php
能正常访问证明LAMP平台构建成功



二. 静态web服务器构建

============== node5.stu31.com : 172.16.31.16 ===============
scp node1:/root/Discuz_X3.2_SC_UTF8.zip ./
(wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip)
unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
service httpd start


============== node6.stu31.com : 172.16.31.17 ===============
scp node1:/root/Discuz_X3.2_SC_UTF8.zip ./
(wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip)
unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
service httpd start


访问测试
====================== windows ======================
http://172.16.31.16/upload/forum.php
http://172.16.31.17/upload/forum.php
只能看到静态界面



三. Varnish缓存服务器构建

============== node3.stu31.com : 172.16.31.13 ===============

wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-3.0.6-1.el6.x86_64.rpm
wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-docs-3.0.6-1.el6.x86_64.rpm
wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-libs-3.0.6-1.el6.x86_64.rpm
yum -y install *.rpm

vim /etc/sysconfig/varnish
NFILES=131072
MEMLOCK=82000
#NPROCS="unlimited"            #加注释
DAEMON_COREFILE_LIMIT="unlimited"        #去注释
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_PORT=80              #修改
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_MIN_THREADS=1            #修改
VARNISH_MAX_THREADS=1000
VARNISH_THREAD_TIMEOUT=120
VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
VARNISH_STORAGE_SIZE=1G
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
VARNISH_TTL=120
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -t ${VARNISH_TTL} \
             -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_T
             -u varnish -g varnish \
             -S ${VARNISH_SECRET_FILE} \
             -s ${VARNISH_STORAGE}"
///////////////////////   ( grep -v ^# /etc/sysconfig/varnish |sed ‘/^$/d‘ )

cp /etc/varnish/default.vcl{,.bak}
vim /etc/varnish/default.vcl
:set nu
:15,$s@^#@@g

backend default {
  .host = "127.0.0.1";
  .port = "80";
}
#
# Below is a commented-out copy of the default VCL logic.  If you
# redefine any of these subroutines, the built-in logic will be
# appended to your code.
 sub vcl_recv {
     if (req.restarts == 0) {
        if (req.http.x-forwarded-for) {
            set req.http.X-Forwarded-For =
                req.http.X-Forwarded-For + ", " + client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
     }
     if (req.request != "GET" &&
       req.request != "HEAD" &&
       req.request != "PUT" &&
       req.request != "POST" &&
       req.request != "TRACE" &&
       req.request != "OPTIONS" &&
       req.request != "DELETE") {
         /* Non-RFC2616 or CONNECT which is weird. */
         return (pipe);
     }
     if (req.request != "GET" && req.request != "HEAD") {
         /* We only deal with GET and HEAD by default */
         return (pass);
     }
     if (req.http.Authorization || req.http.Cookie) {
         /* Not cacheable by default */
         return (pass);
     }
     return (lookup);
 }

 sub vcl_pipe {   
     # Note that only the first request to the backend will have
     # X-Forwarded-For set.  If you use X-Forwarded-For and want to
     # have it set for all requests, make sure to have:
     # set bereq.http.connection = "close";
     # here.  It is not set by default as it might break some broken web
     # applications, like IIS with NTLM authentication.
     return (pipe);
 }

 sub vcl_pass {
     return (pass);
 }

 sub vcl_hash {
     hash_data(req.url);
     if (req.http.host) {
         hash_data(req.http.host);
     } else {
         hash_data(server.ip);
     }
     return (hash);
 }

 sub vcl_hit {
     return (deliver);
 }

 sub vcl_miss {
     return (fetch);
 }

sub vcl_fetch {
     if (beresp.ttl <= 0s ||
         beresp.http.Set-Cookie ||
         beresp.http.Vary == "*") {
                /*
                 * Mark as "Hit-For-Pass" for the next 2 minutes
                 */
                set beresp.ttl = 120 s;
                return (hit_for_pass);
     }
     return (deliver);
 }

 sub vcl_deliver {
     return (deliver);
 }

 sub vcl_error {
     set obj.http.Content-Type = "text/html; charset=utf-8";
     set obj.http.Retry-After = "5";
     synthetic {"
 <?xml version="1.0" encoding="utf-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html>
   <head>
     <title>"} + obj.status + " " + obj.response + {"</title>
   </head>
   <body>
     <h1>Error "} + obj.status + " " + obj.response + {"</h1>
     <p>"} + obj.response + {"</p>
     <h3>Guru Meditation:</h3>
     <p>XID: "} + req.xid + {"</p>
     <hr>
     <p>Varnish cache server</p>
   </body>
 </html>
 "};
     return (deliver);
 }

 sub vcl_init {
        return (ok);
 }

 sub vcl_fini {
        return (ok);
 }
///////////////////

vim /etc/varnish/web.vcl
:set paste
######定义ACL
acl purgers {                    #定义acl，实现IP地址过滤
   "127.0.0.1";
   "172.16.0.0"/16;
}
######定义健康状态检测
probe dynamic {                  #设置动态网站服务器健康状态检测
   .url = "/index.html";
   .interval = 5s;
   .timeout = 1s;
    .expected_response= 200;
}           #这里设置了两个健康状态检测主要是为了区分动、静网站
probe static {                   #设置动态网站服务器健康状态检测
   .url = "/index.html";       #定义检测的页面
   .interval = 5s;              #探测请求的发送周期，默认为5秒
   .timeout = 1s;               #每次探测请求的过期时间
   .expected_response = 200;
}
######定义后端服务器
backend app1 {                  #定义一个后端服务器
   .host = "172.16.31.11";     #服务器地址
   .port = "80";              #服务器监听端口
   .probe = dynamic;           #健康状态检测
}
backend app2 {
   .host = "172.16.31.12";
    .port = "80";
   .probe = dynamic;
}
backend web1 {         
   .host = "172.16.31.16";
   .port = "80";
   .probe = static;
}
backend web2 {
   .host = "172.16.31.17";
   .port = "80";
   .probe = static;
}
######定义后端动态服务器组，实现负载均衡效果
director apps random {          #定义一个后端服务器组，实现负载均衡效果
    {
        .backend = app1;       #调用前面已定义过的后端主机
    .weight = 2;           #设置权重
    }
    {
    .backend = app2;
    .weight = 2;
    }
}
######定义后端静态服务器组，实现负载均衡效果
director webs random {
    {
       .backend = web1;
       .weight = 2 ;
    }
    {
       .backend = web2;
       .weight = 2 ;
    }
}
######定义vcl_recv函数，实现请求到达并成功接收后调用此函数中定义的规则
sub vcl_recv {
######定义动、静分离,以".php"或".php?后面跟所有文件"结尾的请求都发送到动态服务器，其他请求都发送到静态服务器
   if (req.url ~ "\.php(\?\.*|$)") {
   set req.backend = apps;
    }else {
   set req.backend = webs;
    }
   return(lookup);
######定义允许清除缓存的IP地址,调用的是前面定义的ACL
   if (req.request == "PURGE") {
       if (!client.ip ~ purgers) {
       error 405 "Method not allowed";
    }
       return(lookup);
    }
######重新定义http请求首部，让后端服务器可以记录请求客户端的真实IP地址
       if (req.restarts == 0) {
           if (req.http.x-forwarded-for) {
               set req.http.X-Forwarded-For =
               req.http.X-Forwarded-For +", " + client.ip;
           } else {
                 set req.http.X-Forwarded-For =client.ip;
           }
        }
######除了定义的请求方法外，其他请求都到后端服务器
   if (req.request != "GET" &&
       req.request != "HEAD" &&
       req.request != "PUT" &&
       req.request != "POST" &&
       req.request != "TRACE" &&
       req.request != "OPTIONS" &&
       req.request != "DELETE") {
       return (pipe);
    }
   if (req.request != "GET" && req.request !="HEAD") {
       return (pass);
    }
######定义不缓存认证与Cookie信息
   if (req.http.Authorization || req.http.Cookie) {
       return (pass);
    }
######定义压缩功能
   if (req.http.Accept-Enconding) {
      if (req.url ~ "\.(jpg|jpeg|gif|bmp|png|flv|gz|tgz|tbz|mp3)$"){
          remove req.http.Accept-Encoding;
      remove req.http.Cookie;
      } else if (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
      } else if (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
      } else { remove req.http.Accept-Encoding;
      }
    }
######定义指定格式结尾的文件去除Cookie信息
   if (req.request == "GET" && req.url ~"\.(jpeg|jpg|gif|png|bmp|swf)$") {
   unset req.http.cookie;
    }
######定义防盗链设置
   if (req.http.referer ~ "http://.*") {
       if (!(req.http.referer ~ "http://.*\.baidu\.com" ||req.http.referer ~ "http://.*\.google\.com.*")) {
              set req.http.host ="www.stu31.com";
         set req.url = "http://172.16.31.10/error.html";
    }
    }
}
######定义vcl_hash函数
sub vcl_hash {
    hash_data(req.url);
   if (req.http.host) {
       hash_data(req.http.host);
    }else {
       hash_data(server.ip);
    }
   return(hash);
}
######定义vcl_hit函数
sub vcl_hit {
   if (req.request == "PURGE") { #语法方法为"PURGE"
      purge;                     #清除缓存
      error 200 "Purged.";      #返回错误状态码为"200"
    }
   return(deliver);
}
######定义vcl_miss函数
sub vcl_miss {
   if (req.request == "PURGE") {
   purge;
   error 404 "Not In Cache.";
    }
   return(fetch);
}
######定义vcl_psss函数
sub vcl_pass {
   if (req.request == "PURGE") {
      error 502 "Purged On A Passed Object.";
    }
   return(pass);
}
######定义vcl_fetch函数
sub vcl_fetch {
######定义缓存，如果匹配到已定义文件结尾的缓存1天，其他则缓存1小时
   if (req.request == "GET" && req.url ~"\.(html|jpg|png|bmp|jpeg|gif|js|ico|swf|css)$") {
      set beresp.ttl = 1d;
      set beresp.http.expires = beresp.ttl;
    }else {
      set beresp.ttl = 1h;
    }
   return(deliver);
}
######定义在http首部中，如果请求命中显示"HIT",未命中则显示"MISS",通过F12可查看缓存命中状态及varnish服务器IP
sub vcl_deliver {
   if (obj.hits > 0) {
      set resp.http.X-Cache = "Hit from " + server.ip;
    }else {
      set resp.http.X-Cache = "MISS";
    }
}
//////////////////////////// end for /etc/varnish/web.vcl

/etc/rc.d/init.d/varnish  start
netstat -tnlp | grep varnish

varnishadm -S/etc/varnish/secret -T 127.0.0.1:6082
varnish> vcl.load cache web.vcl
varnish> vcl.use cache
varnish> vcl.list
varnish> quit


============== node4.stu31.com : 172.16.31.14 ===============
scp node3:/root/*.rpm ./
yum -y install *.rpm
scp node3:/etc/sysconfig/varnish /etc/sysconfig/
scp node3:/etc/varnish/*.vcl /etc/varnish

/etc/rc.d/init.d/varnish  start
netstat -tnlp | grep varnish

varnishadm -S/etc/varnish/secret -T 127.0.0.1:6082
varnish> vcl.load cache web.vcl
varnish> vcl.use cache        #立刻生效, 重启varnish失效
varnish> vcl.list
varnish> quit





如果我们希望后端的web服务器记录客户端访问的真实IP地址，我们需要配置httpd的配置文件中的日志格式：
node1, node2, node5, node6四个节点都要
vim /etc/httpd/conf/httpd.conf   
LogFormat "%{X-Forwarded-For}i %l %u%t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""combined
///////////
service httpd reload
echo "ok" > /var/www/html/index.html    #此页面用于varnish做健康检测





四. HAProxy反向代理构建

=============== ha.stu31.com : 172.16.31.10 ================
mount /dev/cdrom /mnt/cdrom
yum -y install haproxy

vim /etc/rsyslog.conf
# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
local2.*                                                /var/log/haproxy.log     #添加此行
//////////////
service rsyslog restart

cd /etc/haproxy/
cp haproxy.cfg{,.bak}
vim haproxy.cfg
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

:set paste

listen stats        #添加此块
    mode http
    bind 0.0.0.0:1080
    stats enable
    stats hide-version
    stats uri     /haproxyadmin?stats
    stats realm   Haproxy\ Statistics
    stats auth    admin:admin
    stats admin if TRUE

frontend http-in        #修改很多内容
    bind *:80
    mode http
    log global
    option httpclose
    option logasap
    option dontlognull
    capture request  header Host len 20
    capture request  header Referer len 60
    acl url_static       path_beg       -i /upload/static/image/
    acl url_static       path_end       -i .html .jpeg .gif .png .jpg
    acl url_dynamic      path_end       -i .php .css .js .jsp
    use_backend static_servers         if url_static
    use_backend dynamic_servers        if url_dynamic
    default_backend static_servers

backend static_servers
    balance roundrobin
    server staticsrv1 172.16.31.13:80 check maxconn 3000
    server staticsrv2 172.16.31.14:80 check maxconn 3000

backend dynamic_servers
    balance source
    server dynamicsrv1 172.16.31.11:80 check maxconn 3000
    server dynamicsrv2 172.16.31.12:80 check maxconn 3000
//////////////////

service haproxy start




五. 测试访问

====================== windows =====================
访问论坛
http://172.16.31.10/upload/forum.php

访问haproxy的状态页
http://172.16.31.10:1080/haproxyadmin?stats
admin
admin




window浏览器F12检查缓存命中情况

varnish上的日志
[root@node3 ~]# varnishlog

原始服务器的日志
[root@node1 ~]# tail /var/log/httpd/access_log

LAMP+haproxy+varnish实现网站访问的动静分离及静态资源缓存

标签：lamp   varnish   haproxy   缓存   动静分离   

原文地址：http://8327626.blog.51cto.com/8317626/1707489