码迷,mamicode.com
首页 > 系统相关 > 详细

Linux 缓存服务varnish

时间:2018-06-06 23:47:06      阅读:274      评论:0      收藏:0      [点我收藏+]

标签:linux varnish

Varnish是一款高性能的开源HTTP加速器,Squid服务也是,两者的关系就像apache跟nginx,Varnish更年轻轻量,Squid更成熟稳重。
Varnish4.0工作工作流程如下图
技术分享图片
各个点说明
vcl_recv:接收请求
vcl_pass, 跳过缓存
vcl_hit, 缓存命中
vcl_miss, 缓存丢失
vcl_pipe, 非http
vcl_purge, 手动清楚缓存为过期
vcl_synth, 手动清楚缓存过期同步
vcl_deliver 发送给客户端
每个节点上的变量作用域如下图
技术分享图片

内建变量:

1、req.*:request,表示由客户端发来的请求报文相关;
req.http.User-Agent, req.http.Referer, ...
2、bereq.*:由varnish发往BE主机的httpd请求相关;
3、 beresp.*:由BE主机响应给varnish的响应报文相关;
4、 resp.*:由varnish响应给client相关;
5、obj.*:存储在缓存空间中的缓存对象的属性;只读;

常用变量:

bereq.http.HEADERS
bereq.request:请求方法;
bereq.url:请求的url;
bereq.proto:请求的协议版本;
bereq.backend:指明要调用的后端主机;
req.http.Cookie:客户端的请求报文中Cookie首部的值;
req.http.User-Agent ~ "chrome"
//-------------------------------------
beresp.http.HEADERS
beresp.status:响应的状态码;
reresp.proto:协议版本;
beresp.backend.name:BE主机的主机名;
beresp.ttl:BE主机响应的内容的余下的可缓存时长;
//-------------------------------------
obj.hits:此对象从缓存中命中的次数;
obj.ttl:对象的ttl值
//-------------------------------------
server.ip
server.hostname
//-------------------------------------
client.ip


Varnish的特长体现在缓存命中vcl_hit,以及缓存清理vcl_purge方便。

简单使用说明

1、 安装

[root@node1 ~]# yum install varnish

2、 查看

[root@node1 ~]# rpm -ql varnish
/etc/varnish   #配置文件目录
/etc/varnish/default.vcl      #配置各Child/Cache线程的缓存策略;
/etc/varnish/varnish.params   # 配置varnish服务进程的工作特性,例如监听的地址和端口,缓存机制;
/run/varnish.pid

/usr/bin/varnishadm  #客户端脚本
/usr/bin/varnishtest  #测试工具程序:

#Shared Memory Log交互工具:
/usr/bin/varnishhist
/usr/bin/varnishlog
/usr/bin/varnishncsa
/usr/bin/varnishstat
/usr/bin/varnishtop

/usr/sbin/varnishd  #主程序
/usr/sbin/varnish_reload_vcl  #编译配置文件

#启动脚本
/usr/lib/systemd/system/varnish.service
/usr/lib/systemd/system/varnishlog.service     #持久化日志  二选一
/usr/lib/systemd/system/varnishncsa.service    #持久化日志  二选一

3、配置文件

[root@node1 varnish]# vim /etc/varnish/varnish.params
RELOAD_VCL=1   #启动程序后能否reload配置文件
VARNISH_VCL_CONF=/etc/varnish/default.vcl   #
VARNISH_LISTEN_PORT=80 #默认6081 端口   如果直接当web服务器用改80

# Admin interface listen address and port  管理端口地址
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret  #密钥

#VARNISH_STORAGE="malloc,256M"  #缓存
VARNISH_STORAGE="file,/www/data/varnish,1g"

VARNISH_USER=varnish
VARNISH_GROUP=varnish
[root@node1 data]# mkdir varnish/cache
[root@node1 data]# chown -R varnish.varnish varnish

4、启动

[root@node1 varnish]# systemctl start varnish
[root@node1 varnish]# ss -lntup|grep varnish
tcp    LISTEN     0      128       *:80                    *:*                   users:(("varnishd",pid=1765,fd=7))
tcp    LISTEN     0      10     127.0.0.1:6082                  *:*                   users:(("varnishd",pid=1764,fd=6))
tcp    LISTEN     0      128      :::80                   :::*                   users:(("varnishd",pid=1765,fd=8))

5、缓存策略后台配置

[root@node1 varnish]# vim /etc/varnish/default.vcl
backend default {
    .host = "192.168.1.202";
    .port = "80";
}

6、重载

Loading vcl from /etc/varnish/default.vcl
Current running config name is
Using new config name reload_2018-06-06T09:17:17
VCL compiled.
VCL ‘reload_2018-06-06T09:17:17‘ now active
available       0 boot
active          0 reload_2018-06-06T09:17:17

Done

7、admin管理

[root@node1 varnish]# varnishadm -h
varnishadm: invalid option -- ‘h‘
usage: varnishadm [-n ident] [-t timeout] [-S secretfile] -T [address]:port command [...]
    -n is mutually exlusive with -S and -T

[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
200
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,3.10.0-693.el7.x86_64,x86_64,-sfile,-smalloc,-hcritbit
varnish-4.0.5 revision 07eff4c29

Type ‘help‘ for command list.
Type ‘quit‘ to close CLI session.

varnish> vcl.list  #配置文件版本
200
available       0 boot
active          0 reload_2018-06-06T09:17:17

#切换vcl配置  每次配置文件加载都会保存下来,在这里可以切换
vcl.use boot
200
VCL ‘boot‘ now active
vcl.use reload_2018-06-06T09:17:17
200
VCL ‘reload_2018-06-06T09:17:17‘ now active

#查看默认配置选项
vcl.show -v reload_2018-06-06T09:17:17

#线程数量
param.show thread_pools
200
thread_pools
        Value is: 2 [pools] (default)
        Default is: 2
        Minimum is: 1

param.show thread_pool_max   #min
200
thread_pool_max
        Value is: 5000 [threads] (default)
        Default is: 5000
        Minimum is: 100

#设置参数
varnish> param.set thread_pools 4
200

#存储查看
varnish> storage.list
200
Storage devices:
    storage.Transient = malloc
    storage.s0 = file

#后端查看
varnish> backend.list
200
Backend name                   Refs   Admin      Probe
default(127.0.0.1,,8080)       1      probe      Healthy (no probe)
default(192.168.1.202,,80)     1      probe      Healthy (no probe)

#--------------------------------------------------------------------------
#非交互式
[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 status
Child in state running

规则编写

默认规则查看,很重要,因为varnish先读取客户端配置,在读取默认规则

[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 vcl.list
available       0 boot
available       0 reload_2018-06-06T09:17:17
available       0 reload_2018-06-06T10:43:58
available       0 reload_2018-06-06T11:08:20
available       0 reload_2018-06-06T12:52:47
available       0 reload_2018-06-06T13:04:19
active          0 reload_2018-06-06T13:06:08

[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 vcl.show -v reload_2018-06-06T13:06:08

1、不检查缓存

sub vcl_recv {
        if (req.url ~ "(?i)^/(login|admin)") {
                return(pass);
        }
}

技术分享图片

2、合成错误码

sub vcl_recv {
    if (req.http.User-Agent ~ "(?i)curl"){
        return(synth(403));
    }

}

marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.200/admin.php
<!DOCTYPE html>
<html>
  <head>
    <title>403 Forbidden</title>
  </head>
  <body>
    <h1>Error 403 Forbidden</h1>
    <p>Forbidden</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 196684</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

3、默认vcl_recv策略

sub vcl_recv {

       if (req.method != "GET" &&
       req.method != "HEAD" &&
       req.method != "PUT" &&
       req.method != "POST" &&
       req.method != "TRACE" &&
       req.method != "OPTIONS" &&
       req.method != "DELETE") {
                        /* Non-RFC2616 or CONNECT which is weird. */
                        return (pipe);
       }

        if (req.method != "GET" && req.method != "HEAD") {
            /* We only deal with GET and HEAD by default */
            return (pass);
        }
        if (req.http.Authorization || req.http.Cookie) {
            /* Not cacheable by default */
            return (pass);
        }
            return (hash);

}

4、指定资源去除cookie做缓存

sub vcl_recv {
    if (req.url ~ "(?i)\.(jpg|jpge)$" ){
                unset req.http.Cookie;
                return(hash);
    }
}
sub vcl_backend_response {
    if (beresp.http.cache-control !~ "s-maxage") {
        if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif|css|js)$") {
            unset beresp.http.Set-Cookie;
            set beresp.ttl = 3600s;
        }
    }
}  

技术分享图片

5、客户端ip

 vcl_recv: 如果多级代理需要加 ,默认是有X-Forwarded-For
 if (req.http.X-Forwarded-For){
              set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip;
        }else {
            set req.http.X-Forwarded-For = client.ip;
        }

技术分享图片

6、缓存修剪(实用)

#白名单设置  掩码写在外面
acl purgers{
#       "192.168.1.104";
        "127.0.0.1"/8;
}
sub vcl_recv {
    if(req.method == "PURGE"){
        if (!client.ip ~ purgers){
                return(synth(403));
        }
        return(purge);
    }
}

#默认规则里面的行为    
sub vcl_purge {
    return (synth(200, "Purged"));
}    

技术分享图片
技术分享图片
技术分享图片
以上三次请求,第一次删除缓存,第二次没有命中(添加缓存),第三次命中缓存

在交互模式下(用于临时清理)   这种方式很有用
ban req.url ~ ^/javascript   #清空所有/javascript 开头的
ban req.url ~ ^/            #清空所有

#配置文件清空
if (req.method == "BAN") {
    ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
    # Throw a synthetic page so the request won‘t go to the backend.
    return(synth(200, "Ban added"));
}

7、绑定多个后台

import directors;
#默认是backend default
backend websrv1 {
    .host = "192.168.1.201";
    .port = "80";
}

backend websrv2 {
    .host = "192.168.1.202";
    .port = "80";
}
sub vcl_init {
    #轮询
    new websrvs = directors.round_robin();
    websrvs.add_backend(websrv1);
    websrvs.add_backend(websrv2);

    #权重
    new websrvs3 = directors.random();
    websrvs3.add_backend(websrv1,1);
    websrvs3.add_backend(websrv2,2);

}

#可以区分出来 动态页面跟静态页面
sub vcl_recv {
       if (req.url ~ "(?i)\.php$") {
                set req.backend_hint = websrv1;
        } else {
                set req.backend_hint = websrvs.backend();
        }
}    

#---------------------------------------------------------
#会话绑定
sub vcl_init {
    new h = directors.hash();
    h.add_backend(one, 1);   // backend ‘one‘ with weight ‘1‘
    h.add_backend(two, 1);   // backend ‘two‘ with weight ‘1‘
}

sub vcl_recv {
    // pick a backend based on the cookie header of the client
    set req.backend_hint = h.backend(req.http.cookie);
}

8、健康状态监测

probe www_probe {
    .url = "/index.html";
    .timeout = 1s;
    .interval = 4s;
    .window = 5;   #探测5次 成功3次算在线
    .threshold = 3;
    .expected_response:期望的响应码,默认为200;
}

backend websrv1 {
    .host = "192.168.1.201";
    .port = "80";
    .probe = www_probe;
}

backend websrv2 {
    .host = "192.168.1.202";
    .port = "80";
    .probe = www_probe;
}

[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 backend.list
Backend name                   Refs   Admin      Probe
default(192.168.1.201,,80)     9      probe      Healthy (no probe)
websrv1(192.168.1.201,,80)     5      probe      Healthy 5/5
websrv2(192.168.1.202,,80)     5      probe      Healthy 5/5

在交互模式下手动设置下线
backend.set_health websrv Sick
backend.set_health websrv Healthy

服务进程配置

DAEMON_OPTS="-p thread_pool=3  thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"

thread_pool 2 线程池 小于等于cpu
thread_pool_min 100 每个线程池最小几个线程 默认100
thread_pool_max 5000 最大几个线程  默认5000
thread_pool_timeout 300.000 空闲线程的最长时常  超过thread_pool_min个数后 空闲的超过这个时间就会被杀死
thread_queue_limit 20  线程池队列长度默认20  可以增加
thread_pool_add_delay 0.000 [seconds]  增加不延迟
thread_pool_destroy_delay  1 second 杀死延迟一秒
thread_pool_fail_delay 0.200 [seconds] 创建失败时候,间隔多少时间在重新创建

client:
send_timeout 600.000   发送响应包给客户端
timeout_idle 5     保持连接超时空闲时常  可调增加
timeout_req  2      接收客户端请求报文首部  超过2秒放弃
cli_timeout    admin 管理工具请求超时时常  
backend BE_NAME {
    ...
    .connect_timeout = 3.5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
}

log

1、varnishstat - Varnish Cache statistics

    -1
    -1 -f FILED_NAME 
    -l:可用于-f选项指定的字段名称列表;

    MAIN.cache_hit 
    MAIN.cache_miss

    # varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss
    # varnishstat -l -f MAIN -f MEMPOOL

[root@node1 varnish]# varnishstat -1 #一次显示所有

[root@node1 varnish]# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss
MAIN.cache_hit              63         0.00 Cache hits
MAIN.cache_miss             18         0.00 Cache misses

2、varnishtop - Varnish log entry ranking

    -1     Instead of a continously updated display, print the statistics once and exit.
    -i taglist,可以同时使用多个-i选项,也可以一个选项跟上多个标签;
    -I <[taglist:]regex>
    -x taglist:排除列表
    -X  <[taglist:]regex>

[root@node1 varnish]# varnishtop -i RespStatus  #压测 状态码

3、日志文件,一般不开启,一级代理采用

[root@node1 varnish]# systemctl start varnishncsa
[root@node1 varnish]# tail /var/log/varnish/varnishncsa.log
192.168.1.104 - - [06/Jun/2018:20:54:42 +0800] "GET http://192.168.1.200/robots.txt HTTP/1.1" 404 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"

Linux 缓存服务varnish

标签:linux varnish

原文地址:http://blog.51cto.com/marvin89/2125731

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!