码迷,mamicode.com
首页 > 其他好文 > 详细

redis-cluster集群

时间:2018-08-24 20:46:25      阅读:257      评论:0      收藏:0      [点我收藏+]

标签:head   UNC   local   class   pts   fail   failover   min   res   

技术分享图片

 Redis集群特性

redis cluster集群方式支持主从自动切换 
redis cluster集群只有一个库,单例或者主从的话有多个库 
redis cluster集群去中心化,只要通过其中一个端口连接即可 
redis cluster集群,只有一个db库,不支持多库

单实例的并发QBS每秒11万次请求 读写差距不大 大约1W左右
单实例QBS不要超过5W 超过五万扩容集群 一般3W左右 7W是极限

redis自带集群搭建,一般使用三主三从来构建(最少六台机器)。

主机名 IP 端口 用途
redis-master 192.168.50.167 7000 redis-master01
    7001  redis-master02
    7002 redis-master03
redis-slave 192.168.50.168 8000 redis-slave01
    8001 redis-slave02
    8002 redis-slave03

实验初始环境要求

systemctl stop firewalld    #关防火墙和selinux
systemctl disable firewalld
setenforce 0
sestatus

redis的基础编译部署和调优

yum -y install wget gcc gcc-c++ make tar openssl openssl-devel cmake
tar xf redis-4.0.10.tar.gz -C /usr/src/
cd /usr/src/redis-4.0.10/
make
make MALLOC=jemalloc
make PREFIX=/usr/local/redis install
cd /usr/local/redis/
mkdir -p /usr/local/redis/conf
cp /usr/src/redis-4.0.10/redis.conf /usr/local/redis/conf/    #redis配置文件
cp /usr/src/redis-4.0.10/sentinel.conf /usr/local/redis/conf/  #redis哨兵配置文件
cp /usr/src/redis-4.0.10/src/redis-trib.rb /usr/local/redis/bin/   #redis-cluster的集群创建工具
ln -s /usr/local/redis/bin/* /usr/local/bin/
cp conf/redis.conf{,.bak}
egrep -v "^$|^#" conf/redis.conf.bak > conf/redis.conf
echo "* - nofile 10240" >> /etc/security/limits.conf 
echo "net.core.somaxconn = 10240" >> /etc/sysctl.conf 
echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf 
sysctl -p
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled‘ >> /etc/rc.local
echo ‘echo never > /sys/kernel/mm/transparent_hugepage/defrag‘ >> /etc/rc.local

 redis主从集群多实例配置

#在redis-master上操作
mkdir -p /data/redis-cluster
cd /data/redis-cluster
mkdir -p 7000 7001 7002
#在redis-slave上操作
mkdir -p /data/redis-cluster
cd /data/redis-cluster
mkdir -p 8000 8001 8002

修改redis-master的redis.conf配置文件,模板如下所示:

#redis-master不开任何持久化配置

vim
/data/redis-cluster/7000/redis.conf
bind
0.0.0.0        #连接为所有 protected-mode yes
port
7000          #端口7000 tcp-backlog 1024 timeout 0 tcp-keepalive 0 daemonize yes        #开启后端运行 supervised no pidfile /data/redis-cluster/7000/redis.pid      #PID文件存放位置 loglevel notice logfile "/data/redis-cluster/7000/redis.log"     #日志文件存放位置 databases 16 always-show-logo yes
#save 900 1          #关闭rdb快照
#save 300 10
#save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data/redis-cluster/7000/               #文件存放地址
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no
appendonly no                    #关闭aof持久化
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble no
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
cluster-enabled yes              #开启reids-cluster
cluster-config-file nodes.cong
cluster-node-timeout 5000


#洁净板: bind 0.0.0.0 protected-mode yes port 7000 tcp-backlog 1024 timeout 0 tcp-keepalive 0 daemonize yes supervised no pidfile /data/redis-cluster/7000/redis.pid loglevel notice logfile "/data/redis-cluster/7000/redis.log" databases 16 always-show-logo yes stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir /data/redis-cluster/7000/ slave-serve-stale-data yes slave-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-disable-tcp-nodelay no slave-priority 100 lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no slave-lazy-flush no appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes aof-use-rdb-preamble no lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 aof-rewrite-incremental-fsync yes cluster-enabled yes cluster-config-file nodes.cong cluster-node-timeout 5000

#master2和3配置文件 只修改端口号 :%s#7000#7001#g   %s#7000#7002#g

cp /data/redis-cluster/7000/redis.conf /data/redis-cluster/7001/
cp /data/redis-cluster/7000/redis.conf /data/redis-cluster/7002/

从的配置文件也用这些,如果需要开启主从复制

scp /data/redis-cluster/7000/redis.conf 192.168.50.168:/data/redis-cluster/8000/
scp /data/redis-cluster/7000/redis.conf 192.168.50.168:/data/redis-cluster/8001/
scp /data/redis-cluster/7000/redis.conf 192.168.50.168:/data/redis-cluster/8002/
#修改端口号

 

启动redis-master多实例

redis-server /data/redis-cluster/7001/redis.conf 
redis-server /data/redis-cluster/7002/redis.conf 
redis-server /data/redis-cluster/7000/redis.conf 

编译安装高ruby版本,Ruby版本需要大于等于2.2.2(yum安装的不符合)

#redis-master和redis-slave都进行如下操作
wget --no-check-certificate https://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.7.tar.gz
tar xf ruby-2.2.7.tar.gz -C /usr/src/
cd /usr/src/ruby-2.2.7/
./configure && make && make install

ruby
--version ruby 2.2.7p470 (2017-03-28 revision 58194) [x86_64-linux] #在线安装ruby的redis扩展 /usr/local/bin/gem install redis
Fetching: redis
-4.0.1.gem (100%) Successfully installed redis-4.0.1 Parsing documentation for redis-4.0.1 Installing ri documentation for redis-4.0.1 Done installing documentation for redis after 0 seconds 1 gem installed

使用redis自带redis-trib.rb工具创建集群

#在redis-master上操作
redis-trib.rb create 192.168.50.167:7000 192.168.50.167:7001 192.168.50.167:7002

>>> Creating cluster >>> Performing hash slots allocation on 3 nodes... Using 3 masters: 192.168.50.167:7000 192.168.50.167:7001 192.168.50.167:7002 M: 3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000 slots:0-5460 (5461 slots) master M: 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001 slots:5461-10922 (5462 slots) master M: 7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002 slots:10923-16383 (5461 slots) master Can I set the above configuration? (type yes to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join.. >>> Performing Cluster Check (using node 192.168.50.167:7000) M: 3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000 slots:0-5460 (5461 slots) master 0 additional replica(s) M: 7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002 slots:10923-16383 (5461 slots) master 0 additional replica(s) M: 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001 slots:5461-10922 (5462 slots) master 0 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. #两个OK表示成功

查看集群的信息

redis-cli -p 7000 cluster nodes
625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001@17001 master - 0 1535108491650 2 connected 5461-10922
3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000@17000 myself,master - 0 1535108491000 5 connected 0-5460
7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002@17002 master - 0 1535108490543 3 connected 10923-16383

ls /data/redis-cluster/7000/          

dump.rdb  nodes.cong  redis.conf  redis.log  redis.pid

cat /data/redis-cluster/7000/
nodes.cong          #信息保存在nodes.cong里
625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001@17001 master - 0 1535108491650 2 connected 5461-10922

3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000@17000 myself,master - 0 1535108491000 5 connected 0-5460
7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002@17002 master - 0 1535108490543 3 connected 10923-16383

使用redis-cli去操作集群,需要加入-c参数

redis-cli -c -p 7000        #随便进入一个redis端口就可操作整个redis

 cluster集群的重建

#删除cluster集群配置文件
rm -rf /data/redis-cluster/7000/nodes.cong
rm -rf /data/redis-cluster/7001/nodes.cong
rm -rf /data/redis-cluster/7002/nodes.cong

#删除cluster集群配置文件 redis
-cli -p 7000 shutdown redis-cli -p 7001 shutdown redis-cli -p 7002 shutdown
#启动redis
-server redis-server /data/redis-cluster/7000/redis.conf redis-server /data/redis-cluster/7001/redis.conf redis-server /data/redis-cluster/7002/redis.conf
#重新创建redis
-cluster集群
redis-trib.rb create 192.168.50.167:7000 192.168.50.167:7001 192.168.50.167:7002

 

ps -ef | grep cluster | grep -v grep
root      31952      1  0 22:32 ?        00:00:00 redis-server 0.0.0.0:7000 [cluster]
root      31957      1  0 22:32 ?        00:00:00 redis-server 0.0.0.0:7001 [cluster]
root      31962      1  0 22:32 ?        00:00:00 redis-server 0.0.0.0:7002 [cluster]

Redis Cluster集群的故障自动切换

启动redis-slave上所有的从库

redis-server /data/redis-cluster/8000/redis.conf 
redis-server /data/redis-cluster/8001/redis.conf 
redis-server /data/redis-cluster/8002/redis.conf 

netstat
-antup | grep redis tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 51680/redis-server tcp 0 0 0.0.0.0:8001 0.0.0.0:* LISTEN 51685/redis-server tcp 0 0 0.0.0.0:8002 0.0.0.0:* LISTEN 51690/redis-server tcp 0 0 0.0.0.0:18000 0.0.0.0:* LISTEN 51680/redis-server tcp 0 0 0.0.0.0:18001 0.0.0.0:* LISTEN 51685/redis-server tcp 0 0 0.0.0.0:18002 0.0.0.0:* LISTEN 51690/redis-server

 redis-cluster集群从库的添加

redis-trib.rb add-node --slave 192.168.50.168:8000 192.168.50.167:7000          #添加第一个从8000对应的主为7000
>>> Adding node 192.168.50.168:8000 to cluster 192.168.50.167:7000
>>> Performing Cluster Check (using node 192.168.50.167:7000)
M: 3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: 7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
M: 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 192.168.50.167:7000
>>> Send CLUSTER MEET to node 192.168.50.168:8000 to make it join the cluster.
Waiting for the cluster to join....
>>> Configure node as replica of 192.168.50.167:7000.
[OK] New node added correctly.

redis-trib.rb add-node --slave 192.168.50.168:8001 192.168.50.167:7001            #添加第二个从8001对应的7001
>>> Adding node 192.168.50.168:8001 to cluster 192.168.50.167:7001
>>> Performing Cluster Check (using node 192.168.50.167:7001)
M: 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
S: 7a0f3052b2474b9080f0df1a3df562461ac94d30 192.168.50.168:8000
   slots: (0 slots) slave
   replicates 3f5ff40741abf362e2dad0b26c1ef817e98b3428
M: 7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
M: 3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 192.168.50.167:7001
>>> Send CLUSTER MEET to node 192.168.50.168:8001 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.50.167:7001.
[OK] New node added correctly.

redis-trib.rb add-node --slave 192.168.50.168:8002 192.168.50.167:7002                #第三个添加的从8002对应的主7002
>>> Adding node 192.168.50.168:8002 to cluster 192.168.50.167:7002
>>> Performing Cluster Check (using node 192.168.50.167:7002)
M: 7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
S: 45f40c58756d690a6a687dca283a0274e4d1d0f3 192.168.50.168:8001
   slots: (0 slots) slave
   replicates 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb
M: 3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 7a0f3052b2474b9080f0df1a3df562461ac94d30 192.168.50.168:8000
   slots: (0 slots) slave
   replicates 3f5ff40741abf362e2dad0b26c1ef817e98b3428
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 192.168.50.167:7002
>>> Send CLUSTER MEET to node 192.168.50.168:8002 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.50.167:7002.
[OK] New node added correctly.

查看集群所有节点的信息

redis-cli -p 7000 cluster nodes
45f40c58756d690a6a687dca283a0274e4d1d0f3 192.168.50.168:8001@18001 slave 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 0 1535109423735 2 connected
625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001@17001 master - 0 1535109423535 2 connected 5461-10922
3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000@17000 myself,master - 0 1535109424000 5 connected 0-5460
eaa15a35898137b02087581f2e910d18e0d7cecf 192.168.50.168:8002@18002 slave 7a5ed824f23ca900917bec79c14507377c61506e 0 1535109424747 3 connected
7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002@17002 master - 0 1535109424239 3 connected 10923-16383
7a0f3052b2474b9080f0df1a3df562461ac94d30 192.168.50.168:8000@18000 slave 3f5ff40741abf362e2dad0b26c1ef817e98b3428 0 1535109422730 5 connected

redis的cluster集群,不论是主库还是从库都可以进行set和get。因此,在使用中我们就没必要都去主库了。

查看主从cluster集群key的分布情况

redis-cli -h 192.168.50.168 -c -p 8002 info Keyspace
# Keyspace
db0:keys=6,expires=0,avg_ttl=0

redis集群的主从自动切换,主库挂掉后,从自动变为主

手动切换主从命令cluster failover

redis-cli -p 7000 shutdown      #手动宕掉主1
redis-cli -p 7001 cluster nodes    #查看状态
eaa15a35898137b02087581f2e910d18e0d7cecf 192.168.50.168:8002@18002 slave 7a5ed824f23ca900917bec79c14507377c61506e 0 1535102657000 3 connected
45f40c58756d690a6a687dca283a0274e4d1d0f3 192.168.50.168:8001@18001 slave 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 0 1535102656525 2 connected
3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000@17000 master,fail - 1535102650269 1535102648861 1 disconnected    #7000挂掉了
7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002@17002 master - 0 1535102656000 3 connected 10923-16383
625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001@17001 myself,master - 0 1535102656000 2 connected 5461-10922
7a0f3052b2474b9080f0df1a3df562461ac94d30 192.168.50.168:8000@18000 master - 0 1535102657000 4 connected 0-5460      #8000成为了master
#重新启动7000端口的server,再次查看
redis-server /data/redis-cluster/7000/redis.conf
redis-cli -p 7001 cluster nodes
eaa15a35898137b02087581f2e910d18e0d7cecf 192.168.50.168:8002@18002 slave 7a5ed824f23ca900917bec79c14507377c61506e 0 1535102793234 3 connected
45f40c58756d690a6a687dca283a0274e4d1d0f3 192.168.50.168:8001@18001 slave 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 0 1535102792225 2 connected
3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000@17000 slave 7a0f3052b2474b9080f0df1a3df562461ac94d30 0 1535102791718 4 connected
7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002@17002 master - 0 1535102792728 3 connected 10923-16383
625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001@17001 myself,master - 0 1535102792000 2 connected 5461-10922
7a0f3052b2474b9080f0df1a3df562461ac94d30 192.168.50.168:8000@18000 master - 0 1535102792000 4 connected 0-5460
#手动将redis-server 7000端口重新切换成主库
redis-cli -p 7000 cluster failover
redis-cli -p 7001 cluster nodes
eaa15a35898137b02087581f2e910d18e0d7cecf 192.168.50.168:8002@18002 slave 7a5ed824f23ca900917bec79c14507377c61506e 0 1535102934425 3 connected
45f40c58756d690a6a687dca283a0274e4d1d0f3 192.168.50.168:8001@18001 slave 625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 0 1535102934526 2 connected
3f5ff40741abf362e2dad0b26c1ef817e98b3428 192.168.50.167:7000@17000 master - 0 1535102934000 5 connected 0-5460
7a5ed824f23ca900917bec79c14507377c61506e 192.168.50.167:7002@17002 master - 0 1535102933418 3 connected 10923-16383
625e0520c9dd87f0eaf28549daa0ee3f8f39ecbb 192.168.50.167:7001@17001 myself,master - 0 1535102934000 2 connected 5461-10922
7a0f3052b2474b9080f0df1a3df562461ac94d30 192.168.50.168:8000@18000 slave 3f5ff40741abf362e2dad0b26c1ef817e98b3428 0 1535102933000 5 connected

使用Python操作Redis Cluster集群

 

yum -y install epel-release
yum -y install python2-pip
pip install redis-py-cluster
操作集群的代码
pwd
/server/scripts
cat redis_cluster.py 
# -*- coding:utf-8 -*-
from rediscluster import StrictRedisCluster
redis_nodes = [
        {host:192.168.200.231,port:7000},
        {host:192.168.200.231,port:7001},
        {host:192.168.200.231,port:7002},
        {host:192.168.200.232,port:8000},
        {host:192.168.200.232,port:8001},
        {host:192.168.200.232,port:8002}
        ]
redis_conn = StrictRedisCluster(startup_nodes=redis_nodes)
redis_conn.set(key_test,values_test)
print(redis_conn.get(key_test))
[root@redis-master scripts]# python redis_cluster.py 
values_test
[root@redis-master scripts]# redis-cli -c -p 7002 get key_test
"values_test"
特别提示: 
若其中一个节点挂了,不影响功能的使用

分析Redis的所有key和key的大小

pip安装rdbtools分析工具

[root@redis-master ~]# pip install rdbtools
[root@redis-master ~]# which rdb
/usr/bin/rdb

分析key及key的大小

运维需求,根据dump.rdb文件分析key和key的大小
rdb -c memory /data/redis-cluster/7000/dump.rdb > /root/memory.csv
cat /root/memory.csv | head

 

 

 

 

redis-cluster集群

标签:head   UNC   local   class   pts   fail   failover   min   res   

原文地址:https://www.cnblogs.com/ywrj/p/9531800.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!