标签：没有 工作 varnish缓存 文件中 enabled centos lse clear erro
数据缓存：例如MySQL到web应用服务器之间的缓存服务器缓存的资源是数据缓存
过期时间验证缓存是否失效颗粒度太大，如果页面刚刚缓存应用服务器发生了变化，结果客户端拿到的就是过期数据；从而加入了条件式验证缓存的失效性，每次客户端请求到达缓存服务器，缓存服务器都要拿本地的数据和应用服务器的数据比较时间戳，如果时间戳发生了变化则缓存新的数据；这样虽然粒度小了，但是还是会有问题，如果应用服务器在同一秒页面数据变化了三次，而缓存服务器拿到的是第一份数据，这样还是会发生数据失效的问题；从而又引入了Etag（扩展标记）来标记唯一的页面数据。此时虽然解决了数据失效性的问题，但是每次客户端的请求都要去后端服务器做比较，对缓存和应用服务器都是不小的压力，我们不得不采取折中的解决方案就是“过期时间验证+条件式验证”，将不经常变动的页面做过期时间验证，变动频繁的采用条件式验证。
请求报文用于通知缓存服务如何使用缓存响应请求：
cache-request-directive =
"no-cache" 不能使用缓存系统中的缓存响应我，必须先去应用服务器做缓存验证
"no-store" 不能使用缓存系统中的缓存响应我，必须去应用服务器请求响应我
"max-age" "=" delta-seconds
"max-stale" [ "=" delta-seconds ]
"min-fresh" "=" delta-seconds
"no-transform"
"only-if-cached"
cache-extension响应报文用于通知缓存服务器如何存储上级服务器响应的内容：
cache-response-directive =
"public" 所有缓存系统都可以缓存
"private" [ "=" <"> 1#field-name <"> ] 仅能够被私有缓存所缓存
"no-cache" [ "=" <"> 1#field-name <"> ]，可缓存，但响应给客户端之前需要revalidation，即必须发出条件式请求进行缓存有效性验正
"no-store" ，不允许存储响应内容于缓存中
"no-transform" 不能转换格式
"must-revalidate" 必须重新验证
"proxy-revalidate"
"max-age" "=" delta-seconds 私有缓存最大缓存时长
"s-maxage" "=" delta-seconds 公共缓存最大缓存时长
cache-extensionWeb Page Cache解决方案：squid和varnish，它们的关系就像Apache和Nginx
Varnish cache，或称Varnish，是一套高性能的反向网站缓存服务器（reverse proxy server）
varnish官方站点： http://www.varnish-cache.org/
varnish拥有俩套配置文件；一套配置文件用于varnish自身进程的参数配置，另一套用于定义缓存规则；定义缓存规则需要使用灵活的语言来定义，这就是VCL（varnish语言）；应用时需要将VCL编写的规则送给VCC编译后才能运行，所以安装varnish需要依赖gcc编译器。
varnish的安装：yum install varnish -y，依赖epel源，目前CentOS7的epel源提供的版本是v4.0.5
varnish的程序环境：
/etc/varnish/varnish.params： 配置varnish服务进程的工作特性，例如监听的地址和端口，缓存机制/etc/varnish/default.vcl：配置各Child/Cache线程的缓存策略/usr/sbin/varnishd：主程序/usr/bin/varnishadm：命令行工具/usr/bin/varnishhist：/usr/bin/varnishlog：查看内存中的日志/usr/bin/varnishncsa：以NCSA格式查看日志/usr/bin/varnishstat：查看缓存日志状态信息/usr/bin/varnishtop：以rank方式查看日志/usr/bin/varnishtest：测试工具程序/usr/sbin/varnish_reload_vcl：VCL配置文件重载程序/usr/lib/systemd/system/varnish.service：varnish服务/usr/lib/systemd/system/varnishlog.service：日志持久的服务/usr/lib/systemd/system/varnishncsa.service：日志持久的服务# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss #显示指定参数的当前统计数据# varnishstat -l -f MAIN -f MEMPOOL #列出指定配置段的每个参数的意义# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 #登录管理程序help [<command>] 获取帮助
ping [<timestamp>] 测试服务器
auth <response>
quit 退出cli
banner
status 显示状态
start 启动
stop 停止
vcl.load <configname> <filename> 加载VCL配置文件
vcl.inline <configname> <quoted_VCLstring>
vcl.use <configname> 激活VCL配置文件
vcl.discard <configname> 删除VCL配置
vcl.list 列出VCL配置
param.show [-l] [<param>] 列出当前运行的参数
param.set <param> <value> 运行参数临时调整
panic.show
panic.clear
storage.list 列出数据存储信息
vcl.show [-v] <configname> 列出VCL详细配置
backend.list [<backend_expression>] 列出后端服务器
backend.set_health <backend_expression> <state>
ban <field> <operator> <arg> [&& <field> <oper> <arg>]...
ban.list默认配置文件：
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl #指定加载VCL配置文件
VARNISH_LISTEN_ADDRESS=192.168.1.5 #服务监听的地址
VARNISH_LISTEN_PORT=6081 #默认监听端口
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 #管理服务监听的地址
VARNISH_ADMIN_LISTEN_PORT=6082 #管理服务监听的端口
VARNISH_SECRET_FILE=/etc/varnish/secret #连接秘钥
VARNISH_STORAGE="malloc,256M" #用内存提供保存缓存,大小为256M
VARNISH_USER=varnish #用户身份
VARNISH_GROUP=varnish #组身份
DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300" #指定进程的运行参数Varnish配置语言（VCL）是一种特定于域的语言，用于描述Varnish Cache的请求处理和文档缓存策略。加载新配置时，由Manager进程创建的VCC进程将VCL代码转换为C.此C代码通常由gcc共享对象编译。然后将共享对象加载到cacher进程中。
VCL有多个状态引擎，状态之间存在相关性，但状态引擎彼此间互相隔离；每个状态引擎可使用return(x)指明关联至哪个下一级引擎；每个状态引擎对应于vcl文件中的一个配置段，即为subroutine
俩个特殊的引擎：
vcl_init：在处理任何请求之前要执行的vcl代码：主要用于初始化VMODs；
vcl_fini：所有的请求都已经结束，在vcl配置被丢弃时调用；主要用于清理VMODs；默认VCL配置也叫做隐式规则，在配置文件中无法看到，即使我们修改了配置文件，默认配置规则也是在最后做处理。
varnish> vcl.show -v boot #在客户端cli工具中查看
sub vcl_recv {
if (req.method == "PRI") { #如果客户端的请求方法是PRI，不支持SPDY或HTTP/2.0
return (synth(405)); #则构建一个405的包响应给客户端
}
if (req.method != "GET" && #如果客户端的请求方法不是GET
req.method != "HEAD" && #并且不是HEAD
req.method != "PUT" && #并且不是PUT
req.method != "POST" && #并且不是...
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
return (pipe); #即，不是标准HTTP请求方法的交给pipe（管道）
}
if (req.method != "GET" && req.method != "HEAD") { #请求方法不是GET和HEAD的
return (pass); #交给pass处理，也就是除了GAT和HEAD方法其他的无法缓存
}
if (req.http.Authorization || req.http.Cookie) { #http的请求首部包含Authorization（认证）或Cookie，即个人专有信息
return (pass); #交给pass处理，因为这些带有个人信息的数据无法缓存
}
return (hash); #以上的规则都没有做处理的请求交给hash做处理，剩下的是可以查询缓存的请求了
}
sub vcl_pipe
sub vcl_pass
sub vcl_hash
sub vcl_purge
sub vcl_hit
sub vcl_miss
sub vcl_deliver
sub vcl_synth
sub vcl_backend_fetch
sub vcl_backend_response
sub vcl_backend_error
sub vcl_init
sub vcl_fini示例1：obj.hits是内建变量，用于保存某缓存项的从缓存中命中的次数
# vim /etc/varnish/varnish.params
VARNISH_LISTEN_PORT=80
# vim /etc/varnish/default.vcl
backend default {
.host = "192.168.0.9";
.port = "80";
}
sub vcl_deliver {
if (obj.hits>0) {
set resp.http.X-Cache = "HIT via" + " " + server.ip;
} else {
set resp.http.X-Cache = "MISS from " + server.ip;
}
}
# systemctl restart varnish #谨慎重启varnish服务，会导致之前的缓存失效
# for i in {1..5}; do curl -I -s 192.168.0.8 |grep "X-Cache"; done #在客户端测试，第一次Miss
X-Cache: MISS from 192.168.0.8
X-Cache: HIT via 192.168.0.8
X-Cache: HIT via 192.168.0.8
X-Cache: HIT via 192.168.0.8
X-Cache: HIT via 192.168.0.8常用变量：
示例2：强制对某类资源的请求不检查缓存
# vim /etc/varnish/default.vcl
sub vcl_recv {
if (req.url ~ "(?i)^/(login|admin)") { #"?i"表示忽略大小写，匹配到url中带有login或admin的不查询缓存
return(pass);
}
}
# varnish_reload_vcl
# for i in {1..5}; do curl -I -s http://192.168.0.8/login |grep "X-Cache"; done #客户端测试
X-Cache: MISS from 192.168.0.8 #全部Miss
X-Cache: MISS from 192.168.0.8
X-Cache: MISS from 192.168.0.8
X-Cache: MISS from 192.168.0.8
X-Cache: MISS from 192.168.0.8
# for i in {1..5}; do curl -I -s http://192.168.0.8/admin |grep "X-Cache"; done
X-Cache: MISS from 192.168.0.8
X-Cache: MISS from 192.168.0.8
X-Cache: MISS from 192.168.0.8
X-Cache: MISS from 192.168.0.8
X-Cache: MISS from 192.168.0.8
# for i in {1..5}; do curl -I -s http://192.168.0.8/ |grep "X-Cache"; done #其他网页正常查询缓存
X-Cache: MISS from 192.168.0.8
X-Cache: HIT via 192.168.0.8
X-Cache: HIT via 192.168.0.8
X-Cache: HIT via 192.168.0.8
X-Cache: HIT via 192.168.0.8示例3：对于特定类型的资源，例如公开的图片等，取消其私有标识，并强行设定其可以由varnish缓存的时长
sub vcl_backend_response {
if (beresp.http.cache-control !~ "s-maxage") {
if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif|css|js)$") {
unset beresp.http.Set-Cookie;
set beresp.ttl = 3600s;
}
}
}示例4：在报文首部添加真正的客户端IP，使得后端server可以记录真正客户端来源
[root@varnish ~]# vim /etc/varnish/default.vcl
sub vcl_recv {
if (req.restarts == 0) { #匹配没有被重写的URL请求，即第一次请求
if (req.http.X-Forwarded-For) { #变量存在并且有值则为真
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip; #将真正的client.ip添加到此变量中，用","隔开
} else {
set req.http.X-Forwarded-For = client.ip; #如果变量不存在或值为空，则直接将client.ip赋值与
}
}
}
[root@varnish ~]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
varnish> vcl.load conf1 /etc/varnish/default.vcl
varnish> vcl.use conf1
varnish> vcl.list
available 0 boot
available 0 reload_2018-07-14T09:55:58
active 0 conf1 #当前正在使用的配置
[root@web1 ~]# vim /etc/httpd/conf/httpd.conf
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
[root@web1 ~]# systemctl restart httpd
[root@client ~]# for i in {1..5}; do curl -I -s http://192.168.0.8/login |grep "X-Cache"; done #在客户端测试
[root@web1 ~]# tail /var/log/httpd/access_log
192.168.0.8 - - [14/Jul/2018:09:56:49 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.8 - - [14/Jul/2018:09:56:49 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.8 - - [14/Jul/2018:09:56:49 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.8 - - [14/Jul/2018:09:56:49 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.8 - - [14/Jul/2018:09:56:49 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.7 - - [14/Jul/2018:10:25:11 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.7 - - [14/Jul/2018:10:25:11 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.7 - - [14/Jul/2018:10:25:11 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.7 - - [14/Jul/2018:10:25:11 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0"
192.168.0.7 - - [14/Jul/2018:10:25:11 +0800] "HEAD /login HTTP/1.1" 301 - "-" "curl/7.29.0" #拿到了真正客户端IP，而不是之前的varnish服务器的IP示例5：访问控制，拒绝curl客户端的访问
sub vcl_recv {
if(req.http.User-Agent ~ "curl") {
return(synth(403));
}
}1) 能执行purge操作
sub vcl_purge {
return (synth(200,"Purged"));
}2) 何时执行purge操作
sub vcl_recv {
if (req.method == "PURGE") {
return(purge);
}
...
}示例6：清除指定缓存
[root@varnish ~]# vim /etc/varnish/default.vcl
acl purgers {
"127.0.0.0"/8;
"192.168.0.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purgers) {
return(synth(405,"Purging not allowed for " + client.ip));
}
return(purge);
}
}
varnish> vcl.load conf3 /etc/varnish/default.vcl
varnish> vcl.use conf3
[root@client ~]# curl -I http://192.168.0.8/
X-Cache: HIT via 192.168.0.8
[root@client ~]# curl -I -X "PURGE" http://192.168.0.8/
[root@client ~]# curl -I http://192.168.0.8/
X-Cache: MISS from 192.168.0.81）varnishadm： ban <field> <operator> <arg>
varnish> ban req.url ~ (?i)^/javascripts2）在配置文件中定义，使用ban()函数
sub vcl_recv {
if (req.method == "BAN") {
ban("req.http.host == " + req.http.host + " && req.url == " + req.url); #将规则拼接起来传递给ban函数
return(synth(200, "Ban added"));
}
}
# curl -I -X "BAN" http://192.168.0.8/javascripts/backend default {
.host = "172.16.0.9";
.port = "80";
}
backend appsrv {
.host = "172.16.0.10";
.port = "80";
}
sub vcl_recv {
if (req.url ~ "(?i)\.php$") {
set req.backend_hint = appsrv;
} else {
set req.backend_hint = default;
}
}import directors;
backend srv1 {
.host = "192.168.0.9";
.port = "80";
}
backend srv2 {
.host = "192.168.0.10";
.port = "80";
}
sub vcl_init {
new websrvs = directors.round_robin(); #round_robin()调度算法，不支持加权
websrvs.add_backend(srv1);
websrvs.add_backend(srv2);
}
sub vcl_recv {
set req.backend_hint = websrvs.backend();
}sub vcl_init {
new h = directors.hash();
h.add_backend(one, 1);
h.add_backend(two, 1);
}
sub vcl_recv {
set req.backend_hint = h.backend(req.http.cookie);
}sub vcl_init {
new websrvs = directors.random();
websrvs.add_backend(srv1, 1);
websrvs.add_backend(srv2, 2);
}.probe：定义健康状态检测方法；
.url：检测时要请求的URL，默认为”/";
.request：发出的具体请求；
.request =
"GET /.healthtest.html HTTP/1.1"
"Host: www.dongfei.tech"
"Connection: close"
.window：基于最近的多少次检查来判断其健康状态；
.threshold：最近.window中定义的这么次检查中至有.threshhold定义的次数是成功的；
.interval：检测频度；
.timeout：超时时长；
.expected_response：期望的响应码，默认为200；
import directors;
probe http_chk {
.url = "/index.html";
.interval = 2s;
.timeout = 2s;
.window = 10; #最近10次检查
.threshold = 7; #有7次成功则为健康主机
}
backend srv1 {
.host = "192.168.0.9";
.port = "80";
.probe = http_chk;
}
backend srv2 {
.host = "192.168.0.10";
.port = "80";
.probe = http_chk;
}
sub vcl_init {
new websrvs = directors.random();
websrvs.add_backend(srv1, 1);
websrvs.add_backend(srv2, 2);
}
sub vcl_recv {
set req.backend_hint = websrvs.backend();
}varnish> backend.list #查看后端主机健康状态信息
Backend name Refs Admin Probe
srv1(192.168.0.9,,80) 3 probe Healthy 10/10
srv2(192.168.0.10,,80) 3 probe Healthy 10/10
varnish> backend.set_health srv1 sick|healthy|auto #手动标记主机状态 down|up|probe设置后端的主机属性：
backend BE_NAME {
...
.connect_timeout = 0.5s; #连接超时时间
.first_byte_timeout = 20s; #第一个字节20s不响应则为超时
.between_bytes_timeout = 5s; #第一个字节和第二个字节间隔超时时间
.max_connections = 50; #最大连接数
}最大并发连接数 = thread_pools * thread_pool_max
virnish的日志默认存储在80M的内存空间中，如果日志记录超出了则覆盖前边的日志，服务器重启后丢失；需要更改配置使其永久保存到磁盘
# varnishstat -1 -f MAIN #指定查看MAIN段的信息# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss #显示指定参数的当前统计数据
MAIN.cache_hit 47 0.00 Cache hits
MAIN.cache_miss 89 0.01 Cache misses# varnishtop -1 -i ReqHeader #显示指定的排序信息
165.00 ReqHeader Accept: */*
165.00 ReqHeader Host: 192.168.0.8
165.00 ReqHeader User-Agent: curl/7.29.0
165.00 ReqHeader X-Forwarded-For: 192.168.0.7将日志永久保存到：/var/log/varnish/varnish.log
# systemctl start varnishlog.service以Apache/NCSA日志格式显示
# varnishncsa
192.168.0.7 - - [14/Jul/2018:12:34:23 +0800] "GET http://192.168.0.8/javascripts/test1.html HTTP/1.1" 200 11 "-" "curl/7.29.0"# tree ansible-role-varnish/
ansible-role-varnish/
├── files
│?? ├── default.vcl
│?? ├── secret
│?? └── varnish.params
├── handlers
│?? └── main.yml
├── tasks
│?? ├── copy.yml
│?? ├── main.yml
│?? ├── setup-varnish.yml
│?? └── start.yml
└── templates# find ansible-role-varnish/ -name *.yml -exec ls {} \; -exec cat {} \;
ansible-role-varnish/handlers/main.yml
- name: restart varnish
service: name=varnish state=restarted
- name: reload vcl
command: varnish_reload_vcl
ansible-role-varnish/tasks/start.yml
- name: start service
service: name=varnish state=started
ansible-role-varnish/tasks/copy.yml
- name: copy configure file
copy: src=varnish.params dest=/etc/varnish/varnish.params
notify: restart varnish
- name: copy secret file
copy: src=secret dest=/etc/varnish/secret
notify: restart varnish
- name: copy default.vcl file
copy: src=default.vcl dest=/etc/varnish/default.vcl
notify: reload vcl
ansible-role-varnish/tasks/main.yml
- include: setup-varnish.yml
- include: copy.yml
- include: start.yml
ansible-role-varnish/tasks/setup-varnish.yml
- name: install yum-utils
yum: name={{ item }} state=present
with_items:
- yum-utils
- pygpgme
- name: Add epel repo
yum_repository:
name: alibaba
description: epel
baseurl: https://mirrors.aliyun.com/epel/7Server/x86_64/
repo_gpgcheck: no
gpgcheck: no
enabled: yes
- name: install varnish
yum: name=varnish state=present# find ansible-role-varnish/files/* -exec ls {} \; -exec cat {} \;
ansible-role-varnish/files/default.vcl
#-------------------------------------------------
vcl 4.0;
import directors;
backend default {
.host = "127.0.0.1";
.port = "8080";
}
probe http_chk {
.url = "/index.html";
.interval = 2s;
.timeout = 2s;
.window = 10;
.threshold = 7;
}
backend srv1 {
.host = "192.168.0.9";
.port = "80";
.probe = http_chk;
}
backend srv2 {
.host = "192.168.0.10";
.port = "80";
.probe = http_chk;
}
sub vcl_init {
new websrvs = directors.random();
websrvs.add_backend(srv1, 1);
websrvs.add_backend(srv2, 1);
}
sub vcl_recv {
set req.backend_hint = websrvs.backend();
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if(req.http.User-Agent ~ "curl") {
return(synth(403));
}
}
sub vcl_backend_response {
if (beresp.http.cache-control !~ "s-maxage") {
if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif|css|js)$") {
unset beresp.http.Set-Cookie;
set beresp.ttl = 3600s;
}
}
}
sub vcl_deliver {
if (obj.hits>0) {
set resp.http.X-Cache = "HIT via" + " " + server.ip;
} else {
set resp.http.X-Cache = "MISS from " + server.ip;
}
}
#-------------------------------------------------
ansible-role-varnish/files/secret
7e40f334-d2e7-4edb-aecb-559519e456f9
ansible-role-varnish/files/varnish.params
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_ADDRESS=0.0.0.0
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=0.0.0.0
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_STORAGE="malloc,256M"
VARNISH_USER=varnish
VARNISH_GROUP=varnish
#DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"感谢阅读！
标签：没有 工作 varnish缓存 文件中 enabled centos lse clear erro
原文地址：https://www.cnblogs.com/L-dongf/p/9310144.html
