标签：bbb RoCE ESS 技术分享 文件 sha 不同 设定 数据
redis集群介绍多个redis节点实现网络互连，数据共享
所有节点都是一主一从(或一主多从)，其中从不提供服务，仅作为备份备用，保证redis数据的安全性
不支持同时处理多个键，redis集群写入数据是向集群里随机写入的(比如raid5磁盘阵列格式)，因为redis集群需要把键平均分布在各个节点上，并发量很高的情况下同时创建键值会降低性能并会导致不可预估的行为
redis集群支持在线增加、删除节点，在扩充redis的情况下，是非常容易扩容集群服务数量的
客户端可以连接任意一个主节点进行读写
redis群集场景设定
使用两台机器，分别开启三个redis服务(基于不同端口)
服务器ip为192.168.1.223和192.168.1.234
192.168.1.234上开启7000、7002、7002三个端口，设置角色全部为主
192.168.1.223上开启7001、7003、7005三个端口，设置角色全部为从
两台机器都要编译安装redis，编辑三个不同的redis.conf，分别设置不同端口号、dir等参数，还需要增加cluster相关参数，启动为六个redis服务
首先在192.168.1.234机器上拷贝三个不同的redis配置文件，修改配置文件的端口、存储数据文件的路径等，并创建配置文件中指定存储数据文件的目录
在192.168.1.234机器上创建三个基于不同端口的配置文件，如：
[root@nfs1 /]# cat /etc/redis_7000.conf
port 7000
bind 192.168.1.234
daemonize yes
pidfile /var/run/redis_7000.pid
dir /data/redis_data/7000
cluster-enabled yes
cluster-config-file nodes_7000.conf
cluster-node-timeout 10100
appendonly yes
[root@nfs1 /]# cat /etc/redis_7002.conf
port 7002
bind 192.168.1.234
daemonize yes
pidfile /var/run/redis_7002.pid
dir /data/redis_data/7002
cluster-enabled yes
cluster-config-file nodes_7002.conf
cluster-node-timeout 10100
appendonly yes
[root@nfs1 /]# cat /etc/redis_7004.conf
port 7004
bind 192.168.1.234
daemonize yes
pidfile /var/run/redis_7004.pid
dir /data/redis_data/7004
cluster-enabled yes
cluster-config-file nodes_7004.conf
cluster-node-timeout 10100
appendonly yes启动这三个基于不同端口的redis服务，并查看其启动端口信息
[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7000.conf
91981:C 14 Nov 2018 18:31:38.224 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
91981:C 14 Nov 2018 18:31:38.224 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91981, just started
91981:C 14 Nov 2018 18:31:38.224 # Configuration loaded
[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7002.conf
91986:C 14 Nov 2018 18:31:46.752 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
91986:C 14 Nov 2018 18:31:46.752 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91986, just started
91986:C 14 Nov 2018 18:31:46.752 # Configuration loaded
[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7004.conf
91991:C 14 Nov 2018 18:31:50.830 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
91991:C 14 Nov 2018 18:31:50.830 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91991, just started
91991:C 14 Nov 2018 18:31:50.830 # Configuration loaded
[root@nfs1 /]# netstat -ntlp |grep redis
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:7000 ? ? ?0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91982/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:7002 ? ? ?0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91987/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:7004 ? ? ?0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91992/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:17000 ? ? 0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91982/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:17002 ? ? 0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91987/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:17004 ? ? 0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91992/redis-server
192.168.1.223的redis也是按照这种方式启动多个redis服务的
[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7001.conf
26608:C 15 Nov 2018 00:00:10.297 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26608:C 15 Nov 2018 00:00:10.297 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26608, just started
26608:C 15 Nov 2018 00:00:10.297 # Configuration loaded
[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7003.conf
26622:C 15 Nov 2018 00:00:22.984 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26622:C 15 Nov 2018 00:00:22.985 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26622, just started
26622:C 15 Nov 2018 00:00:22.985 # Configuration loaded
[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7005.conf
26627:C 15 Nov 2018 00:00:27.128 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26627:C 15 Nov 2018 00:00:27.128 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26627, just started
26627:C 15 Nov 2018 00:00:27.128 # Configuration loaded
[root@Huaching /]# netstat -antp |grep redis
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:7003 ? ? ? ? ?0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26623/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:7005 ? ? ? ? ?0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26628/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:17001 ? ? ? ? 0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26609/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:17003 ? ? ? ? 0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26623/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:17005 ? ? ? ? 0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26628/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:7001 ? ? ? ? ?0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26609/redis-server ?启动多个redis服务后接下来安装ruby中间件实现群集功能
安装ruby只需要在一台服务器上进行安装运行
安装Development的软件组，安装过程省略
[root@nfs1 redis_data]# yum -y groupinstall "Development Tools"安装需要支持的库，安装过程省略
[root@nfs1 redis_data]# yum install gdbm-devel linbdb4-devel libffi-devel libyaml libyaml-devel ncurses-devel openssl_devel readline-devel tcl-deve创建编译rpm包的目录，下载ruby的源码包文件，将源码包下载只刚刚创建的目录下
如果下载有rpm编译器，则执行编译器来生成rpm包进行安装，如果没有就使用./configure编译安装
[root@nfs1 ~]# mkdir -p /root/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
[root@nfs1 ~]# rpmbuild -bb /root/rpmbuild/SPECS/ruby22x.spec
[root@nfs1 ~]# yum -y localinstall rpmbuild/RPMS/x86_64/ruby-2.4.5-1.e17.centos.x86_64.rpm由于这里我无法找到ruby的rpm编译器，所以无法编译成rpm包。这里依旧使用解压源码包，使用./configure来编译安装
[root@nfs1 ruby-2.4.5]# ./configure
[root@nfs1 ruby-2.4.5]# make
-----------------------------编译内容
Files: 890
Classes: 1312 ( 549 undocumented)
Modules: 275 ( 110 undocumented)
Constants: 2143 ( 538 undocumented)
Attributes: 1053 ( 247 undocumented)
Methods: 9837 (2106 undocumented)
Total: 14620 (3550 undocumented)
? 75.72% documented
Elapsed: 86.3s
[root@nfs1 ruby-2.4.5]# make install
-----------------------------编译内容
installing bundle gems: ? ? ? /usr/local/lib/ruby/gems/2.4.0 (build_info, cache, doc, extensions, gems, specifications)
? ? ? ? ? ? ? ? ? ? ? ? ? ? minitest 5.10.1
? ? ? ? ? ? ? ? ? ? ? ? ? ? xmlrpc 0.2.1
? ? ? ? ? ? ? ? ? ? ? ? ? ? test-unit 3.2.3
? ? ? ? ? ? ? ? ? ? ? ? ? ? net-telnet 0.1.1
? ? ? ? ? ? ? ? ? ? ? ? ? ? rake 12.0.0
? ? ? ? ? ? ? ? ? ? ? ? ? ? did_you_mean 1.1.0
? ? ? ? ? ? ? ? ? ? ? ? ? ? power_assert 0.4.1
installing rdoc: ? ? ? ? ? ? /usr/local/share/ri/2.4.0/system
installing capi-docs: ? ? ? ? /usr/local/share/doc/ruby编译安装完成后可以查看下ruby的版本号，验证是否编译安装正确
[root@nfs1 ruby-2.4.5]# ruby -v
ruby 2.4.5p335 (2018-10-18 revision 65137) [x86_64-linux]
安装redis的ruby集群的相关命令
[root@nfs1 ruby-2.4.5]# gem install redis
Fetching: redis-4.0.3.gem (100%)
Successfully installed redis-4.0.3
Parsing documentation for redis-4.0.3
Installing ri documentation for redis-4.0.3
Done installing documentation for redis after 1 seconds
1 gem installed拷贝redis的ruby命令到/usr/local/bin目录下，新版本舍弃了redis-trib.rb的这个命令，如果安装新版本，可以不执行此步操作
[root@nfs1 ruby-2.4.5]# cp /usr/local/src/redis-5.0.0/src/redis-trib.rb /usr/local/bin/在新版本的redis中，使用redis-trib.rb来创建redis的群集会提示版本命令不可用，因为redis-trib.rb命令被舍弃不再使用。具体信息如下：
WARNING: redis-trib.rb is not longer available!
You should use redis-cli instead.
All commands and features belonging to redis-trib.rb have been moved
to redis-cli.
In order to use them you should call redis-cli with the --cluster
option followed by the subcommand name, arguments and options.
Use the following syntax:
redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
Example:
redis-cli --cluster create 192.168.1.234:700 192.168.1.234:7002 192.168.1.234:7004 192.168.1.200:7001 192.168.1.200:7003 192.168.1.200:7005 --cluster-replicas 1
To get help about all subcommands, type:
redis-cli --cluster help从上面提示可以了解到redis-trub.rb命令已经被redis-cli代替使用
那么我们现在使用redis-cli来创建群集
格式为redis-cli --cluster create ?ip:prot ?ip:port ...... --cluster-replicas 1
这里我创建的群集如下，在确认好群集信息后，会提示你是否按照这个主从指定的方式保存
[root@nfs1 ruby-2.4.5]# redis-cli --cluster create 192.168.1.234:7000 192.168.1.234:7002 192.168.1.234:7004 192.168.1.200:7001 192.168.1.200:7003 192.168.1.200:7005 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.200:7003 to 192.168.1.234:7000 ? ? ? ? ? ? ? ? 这里指定了主从关系：从redis-->主redis
Adding replica 192.168.1.234:7004 to 192.168.1.200:7001
Adding replica 192.168.1.200:7005 to 192.168.1.234:7002
M: 3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000 ? ? ? ? 输出信息中的M为master ? S则是slave
? slots:[0-5460] (5461 slots) master
M: d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002 ? ? ? ?
? slots:[10923-16383] (5461 slots) master
S: 2a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004
? replicates 3e8c289a64638da55b0e812f924aaa242efbc315
M: 3e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001
? slots:[5461-10922] (5462 slots) master
S: 9882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003
? replicates 3af55accd31b6a057945c4b9e9378c7932d5f1f8
S: a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005
? replicates d8418ab100f8f32539fc60a7c8ead5927591f52a
Can I set the above configuration? (type ‘yes‘ to accept): yes在上面确认交互的地方确认保存后会输出两个ok信息，可以判断redis群集创建成功
Can I set the above configuration? (type ‘yes‘ to accept): yes
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.redis集群创建好后，登录redis时需要指定登入ip和端口，创建几个key值查看redis存储的redis信息
-c ?登入redis集群模式?
-h 指定集群中的一个redis
[root@nfs1 ruby-2.4.5]# redis-cli -c -h 192.168.1.234 -p 7000
192.168.1.234:7000> set key1 123
-> Redirected to slot [9189] located at 192.168.1.200:7001
OK
192.168.1.200:7001> set key3 abc
-> Redirected to slot [935] located at 192.168.1.234:7000
OK
192.168.1.234:7000> set key2 ABC
OK
192.168.1.234:7000> set key4 ABC
-> Redirected to slot [13120] located at 192.168.1.234:7002
OK
192.168.1.234:7002> set key5 ABC
-> Redirected to slot [9057] located at 192.168.1.200:7001
OK
192.168.1.200:7001> set key6 ABC
-> Redirected to slot [4866] located at 192.168.1.234:7000
OK查看群集状态信息，redis-cli代替了redis-trib.rb，所以这里使用新的命令格式。可以看出群集的redis信息状态
[root@nfs1 ruby-2.4.5]# redis-cli --cluster check 192.168.1.234:7000
192.168.1.234:7000 (3af55acc...) -> 4 keys | 5461 slots | 1 slaves.
192.168.1.234:7002 (d8418ab1...) -> 1 keys | 5461 slots | 1 slaves.
192.168.1.200:7001 (3e8c289a...) -> 2 keys | 5462 slots | 1 slaves.
[OK] 7 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.234:7000)
M: 3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000
? slots:[0-5460] (5461 slots) master
? 1 additional replica(s)
S: a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005
? slots: (0 slots) slave
? replicates d8418ab100f8f32539fc60a7c8ead5927591f52a
M: d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002
? slots:[10923-16383] (5461 slots) master
? 1 additional replica(s)
M: 3e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001
? slots:[5461-10922] (5462 slots) master
? 1 additional replica(s)
S: 2a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004
? slots: (0 slots) slave
? replicates 3e8c289a64638da55b0e812f924aaa242efbc315
S: 9882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003
? slots: (0 slots) slave
? replicates 3af55accd31b6a057945c4b9e9378c7932d5f1f8
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.cluster nodes ? 列出集群节点
[root@nfs1 ruby-2.4.5]# redis-cli -c -h 192.168.1.234 -p 7000 ? ? ? ? 登入任意一个redis中
192.168.1.234:7000> CLUSTER NODES
a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005@17005 slave d8418ab100f8f32539fc60a7c8ead5927591f52a 0 1542270025000 6 connected
d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002@17002 master - 0 1542270024896 2 connected 10923-16383
3e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001@17001 master - 0 1542270027918 4 connected 5461-10922
2a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004@17004 slave 3e8c289a64638da55b0e812f924aaa242efbc315 0 1542270026906 4 connected
3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000@17000 myself,master - 0 1542270024000 1 connected 0-5460
9882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003@17003 slave 3af55accd31b6a057945c4b9e9378c7932d5f1f8 0 1542270028928 5 connectedcluster info ? 查看集群信息
192.168.1.234:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1594
cluster_stats_messages_pong_sent:1603
cluster_stats_messages_sent:3197
cluster_stats_messages_ping_received:1598
cluster_stats_messages_pong_received:1594
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:3197所有的新添加的redis节点，都是为master角色
cluster meet ? ip ?port
在192.168.1.200上新启动一个redis，监听端口为7007。登入群集中执行redis节点添加命令
[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7000
192.168.1.234:7000> cluster meet 192.168.1.200 7007
OK
192.168.1.234:7000> cluster nodes ? ? ? ? ? ? ? ? //第一条就是新添加的节点信息
cff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542270675527 0 connected
a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005@17005 slave d8418ab100f8f32539fc60a7c8ead5927591f52a 0 1542270677550 6 connected
d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002@17002 master - 0 1542270678560 2 connected 10923-16383
----------------省略再添加一个新的redis节点，并将新节点指定为从节点
cluster reolicate ?master node info ? ? ? ? ? ?//登入需要设定为从节点的redis，执行命令指定主节点
指定从节点需要登入到那个redis节点当中操作，而不是在集群中指定，在登入节点中指定本节点的master
再次添加一个redis节点
192.168.1.234:7000> cluster meet 192.168.1.234 7006
OK
cff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542270675527 0 connected
24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 master - 0 1542270869918 7 connected退出节点登入端口为7006的节点，
192.168.1.234:7000> exit
[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7006
192.168.1.234:7006> CLUSTER REPLICATE cff62edb40a4df11d8390bc28d254abca16188ba
OK再次查看群集节点状态信息，可以看到端口7006的redis已经成为7007redis的slave节点了
192.168.1.234:7006> cluster nodes
cff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542271290173 0 connected
24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 myself,slave cff62edb40a4df11d8390bc28d254abca16188ba 0 1542271292000 7 connected移除一个redis节点
cluster forget? ? node_id
在移除某个redis节点之前，首先不能在登入该节点当中，否则不能正常移除该节点，当前处于7006的redis当中，移除节点提示信息：
24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 myself,slave cff62edb40a4df11d8390bc28d254abca16188ba 0 1542271292000 7 connected
192.168.1.234:7006> cluster forget 24f1c355bc92a0ab15a6077b1bbf4bd510dfa132
(error) ERR I tried hard but I can‘t forget myself...必须登入其他节点移除端口为7006的redis节点
[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7000移除7006节点之前查看所有节点信息
192.168.1.234:7000> CLUSTER NODES
拷贝7006节点的node_id，移除7006节点
192.168.1.234:7000> cluster forget 24f1c355bc92a0ab15a6077b1bbf4bd510dfa132
OK移除7006节点后
192.168.1.234:7000> CLUSTER NODES
保存redis集群的配置文件
cluster saveconfig ?
保存配置文件，会在当前登入的redis数据存储路径下增加redis配置内容
192.168.1.234:7000> cluster saveconfig
OK
192.168.1.234:7000> exit
标签：bbb RoCE ESS 技术分享 文件 sha 不同 设定 数据
原文地址：http://blog.51cto.com/8844414/2317509
