豆瓣Redis解决方案Codis安装使用

1.安装

1.1 Golang环境

Golang的安装非常简单，因为官网被墙，可以从国内镜像如studygolang.com下载。

[root@vm root]$ tar -C /usr/local -zxf go1.4.2.linux-amd64.tar.gz
[root@vm root]$ vim /etc/profile
export GOROOT=/usr/local/go
export PATH=$GOROOT/bin:$PATH
export GOPATH=/home/user/go

[root@vm root]$ source /etc/profile
[root@vm root]$ go version

注意：操作系统的软件仓库如apt-get或yum安装的Golang版本可能比较老，要卸载后重新手动安装。以我的Linux Mint为例，安装时apt-get install golang指定的其实是个meta，卸载时要指定apt-get remove golang-go才能卸载干净。

1.2 Codis源码包

GitHub上的官方安装指导是这样写的：

[root@vm root]$ go get -d github.com/wandoulabs/codis
[root@vm root]$ cd $GOPATH/src/github.com/wandoulabs/codis
[root@vm root]$ ./bootstrap.sh
[root@vm root]$ make gotest

这种方式会从GitHub上master下载最新的代码，但实践发现有时master代码可能会有问题。所以保险起见，还是下载稳定的release包吧。

1.3 编译安装

下载stable包后，我这里用的是最新的2.0.0版本包。手动创建出go的下载目录，将源码包解压后放进去，这样就手动完成了通过go get下载源码一步，避免用master分支代码的不稳定性。

[root@vm root] mkdir -p /home/root/go/src/github.com/wandoulabs/codis
[root@vm root] tar -xzvf ...

[root@vm root] cd codis
[root@vm codis] find . -name "configure" | xargs chmod +x
[root@vm codis] find . -name "*.sh" | xargs chmod +x

[root@vm codis] ./bootstrap.sh
    CC hyperloglog.o
    CC latency.o
    CC sparkline.o
    CC slots.o
    CC redis-cli.o
    CC redis-benchmark.o
    CC redis-check-dump.o
    CC redis-check-aof.o
    LINK redis-benchmark
    LINK redis-check-dump
    LINK redis-check-aof
    LINK redis-server
    INSTALL redis-sentinel
    LINK redis-cli

Hint: It‘s a good idea to run ‘make test‘ ;)

make[2]: Leaving directory `/home/root/go/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src‘
make[1]: Leaving directory `/home/root/go/src/github.com/wandoulabs/codis/extern/redis-2.8.21‘
ln -s codis-server-2.8.21 bin/codis-server
go test ./pkg/... ./cmd/... -race
?       github.com/wandoulabs/codis/pkg/env     [no test files]
ok      github.com/wandoulabs/codis/pkg/models  5.057s
?       github.com/wandoulabs/codis/pkg/proxy/group     [no test files]
ok      github.com/wandoulabs/codis/pkg/proxy/redis     3.061s
ok      github.com/wandoulabs/codis/pkg/proxy/router    17.916s
?       github.com/wandoulabs/codis/pkg/proxy/router/topology   [no test files]
?       github.com/wandoulabs/codis/pkg/utils   [no test files]
?       github.com/wandoulabs/codis/pkg/utils/assert    [no test files]
?       github.com/wandoulabs/codis/pkg/utils/atomic2   [no test files]
ok      github.com/wandoulabs/codis/pkg/utils/bytesize  0.025s
?       github.com/wandoulabs/codis/pkg/utils/errors    [no test files]
?       github.com/wandoulabs/codis/pkg/utils/log       [no test files]
?       github.com/wandoulabs/codis/pkg/utils/trace     [no test files]
?       github.com/wandoulabs/codis/cmd/cconfig [no test files]
?       github.com/wandoulabs/codis/cmd/proxy   [no test files]

2.启动

之前参照官网上的“流程”一节，被“坑”苦了…鼓捣了老半天也不行，要么就是Dashboard端口不对了，要么就是Slot初始化不了了，都要放弃了。发现正确的步骤在codis/sample/usage.md里，作者都写好了启动脚本。

[root@vm sample] ./start_dashboard.sh
[root@vm sample] ./start_redis.sh
[root@vm sample] ./add_group.sh
[root@vm sample] ./initslot.sh
[root@vm sample] ./start_proxy.sh
[root@vm sample] ./set_proxy_online.sh

首先启动Dashboard，然后启动四个Redis实例，为这四个Redis实例创建四个Group，初始化Slot，启动一个Proxy实例，将Proxy置为Online状态。其中像新建Group、置为Online这种操作其实在Dashboard上都能做，但作者造福大家都给准备好了。一气呵成啊！刚才还自己瞎鼓捣了半天…

3.观察ZooKeeper

Codis使用ZooKeeper做协调，下面就看一下ZooKeeper里存的到底都是些什么。

Codis会根据我们在config.ini中配置的product_name作为ZooKeeper中的路径。

[zk: localhost:2181(CONNECTED) 34] ls /zk/codis/db_test
[fence, slots, servers, proxy, migrate_tasks, dashboard, LOCK, actions, ActionResponse]

3.1 Slot信息

前面initslots初始化的所有Slot都保存在slots路径下，每个Slot是一个结点。

[zk: localhost:2181(CONNECTED) 35] ls /zk/codis/db_test/slots
[slot_240, slot_241, slot_242, slot_244, slot_243, slot_246, slot_245, slot_248, slot_247, slot_249, slot_804, slot_803, slot_802, slot_801, slot_808, slot_807, slot_806, slot_805, slot_800, slot_230, slot_231, 
                        ...     ...     ...
 slot_832, slot_286, slot_672, slot_675, slot_674, slot_670, slot_671, slot_4, slot_3, slot_2, slot_1, slot_8, slot_7, slot_6, slot_5, slot_9, slot_0, slot_600, slot_602, slot_601, slot_608, slot_607, slot_609, slot_604, slot_603, slot_606, slot_605, slot_205, slot_206, slot_203, slot_204, slot_201, slot_202, slot_200, slot_209, slot_207, slot_208]

随便选取一个Slot结点，用get命令能够查看结点上附着的数据：

[zk: localhost:2181(CONNECTED) 8] get /zk/codis/db_test/slots/slot_208
{"product_name":"test","id":208,"group_id":5,"state":{"status":"online","migrate_status":{"from":-1,"to":-1},"last_op_ts":"0"}}

3.2 Proxy信息

[zk: localhost:2181(CONNECTED) 40] ls /zk/codis/db_test/proxy    
[proxy_1]
[zk: localhost:2181(CONNECTED) 12] get /zk/codis/db_test/proxy/proxy_1  
{"id":"proxy_1","addr":"BC-VM-edce4ac67d304079868c0bb265337bd4:19000","last_event":"","last_event_ts":0,"state":"online","description":"","debug_var_addr":"BC-VM-edce4ac67d304079868c0bb265337bd4:11000","pid":2946,"start_at":"2015-07-03 16:13:53.277706979 +0800 CST"}

3.3 Server信息

Server的信息保存的只是Redis实例的IP地址和端口，所以并没有以JSON的形式附着在路径结点上，而是直接作为路径的一部分保存了。

[zk: localhost:2181(CONNECTED) 43] ls /zk/codis/db_test/servers
[group_1, group_2, group_3, group_4]
[zk: localhost:2181(CONNECTED) 44] ls /zk/codis/db_test/servers/group_1
[localhost:6380]
[zk: localhost:2181(CONNECTED) 45] ls /zk/codis/db_test/servers/group_2
[localhost:6381]
[zk: localhost:2181(CONNECTED) 46] ls /zk/codis/db_test/servers/group_3
[localhost:6382]
[zk: localhost:2181(CONNECTED) 47] ls /zk/codis/db_test/servers/group_4
[localhost:6383]

3.4 其他信息

ZooKeeper中还保存了很多其他信息，这里就不一一解释了。

[zk: localhost:2181(CONNECTED) 53] ls /zk/codis/db_test/fence
[BC-VM-edce4ac67d304079868c0bb265337bd4:19000]

[zk: localhost:2181(CONNECTED) 24] get /zk/codis/db_test/dashboard
{"addr": "localhost:18087", "pid": 2871}

[zk: localhost:2181(CONNECTED) 33] get /zk/codis/db_test/actions/0000003698
{"type":"slot_changed","desc":"","target":{"product_name":"test","id":941,"group_id":5,"state":{"status":"online","migrate_status":{"from":-1,"to":-1},"last_op_ts":"0"}},"ts":"1435914788","receivers":["{\"id\":\"proxy_1\",\"addr\":\"BC-VM-edce4ac67d304079868c0bb265337bd4:19000\",\"last_event\":\"\",\"last_event_ts\":0,\"state\":\"online\",\"description\":\"\",\"debug_var_addr\":\"BC-VM-edce4ac67d304079868c0bb265337bd4:11000\",\"pid\":2946,\"start_at\":\"2015-07-03 16:13:53.277706979 +0800 CST\"}"]}

4.测试

4.1 简单测试

因为Codis兼容大部分Redis操作，所以这里用原版Redis客户端连接到Proxy，测试一下保存和查询。

[root@vm redis-2.8.19] src/redis-cli -p 19000
127.0.0.1:19000> set person1 cdai
OK
127.0.0.1:19000> set student2 hank
OK
127.0.0.1:19000> set teacher3 carter
OK
127.0.0.1:19000> set manager4 rachel
OK
127.0.0.1:19000> get person1
"cdai"
127.0.0.1:19000> get student2
"hank"
127.0.0.1:19000> get teacher3
"carter"
127.0.0.1:19000> get manager4
"rachel"

4.2 自动Rebalance

新添加一个Redis实例6379，并在Dashboard中为其新建Group 5，将新实例添加进去。

[root@vm sample] cp redis_conf/6380.conf redis_conf/6379.conf
[root@vm redis_conf] diff redis_conf/6379.conf redis_conf/6380.conf 
41c41
< pidfile /var/run/redis_6379.pid
---
> pidfile /var/run/redis.pid
45c45
< port 6379
---
> port 6380
177c177
< dbfilename 6379.rdb
---
> dbfilename 6380.rdb

[root@vm sample] nohup ../bin/codis-server ./redis_conf/6379.conf &> ./log/redis_6379.log &

重新balance之前先看一下目前四个key的分布情况：

• 6380保存manager4和student2
• 6381什么都没保存
• 6382保存teacher3
• 6383保存person1

[root@vm redis-2.8.19] src/redis-cli
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> quit
[root@vm redis-2.8.19] src/redis-cli -p 6380
127.0.0.1:6380> keys *
1) "manager4"
2) "student2"
127.0.0.1:6380> quit
[root@vm redis-2.8.19] src/redis-cli -p 6381
127.0.0.1:6381> keys *
(empty list or set)
127.0.0.1:6381> quit
[root@vm redis-2.8.19] src/redis-cli -p 6382
127.0.0.1:6382> keys *
1) "teacher3"
127.0.0.1:6382> quit
[root@vm redis-2.8.19] src/redis-cli -p 6383
127.0.0.1:6383> keys *
1) "person1"
127.0.0.1:6383> quit

现在点”Auto Balance”看刚才保存的四个key会发生什么。

在Migration过程中，ZooKeeper中的内容也有变化。/zk/codis/db_test/migrate_tasks保存了正在migrating的Slot：

[zk: localhost:2181(CONNECTED) 61] ls /zk/codis/db_test/migrate_tasks
[0000000079, 0000000078, 0000000082, 0000000083, 0000000080, 0000000081, 0000000086, 0000000087, 0000000084, 0000000085, 0000000118, 0000000119, 0000000116, 0000000117, 0000000114, 0000000115, 0000000112, 0000000113, 0000000110, 0000000111, 0000000120, 0000000200, 0000000201, 0000000202, 0000000107, 0000000203, 0000000108, 0000000109, 0000000103, 0000000104, 0000000105, 0000000106, 0000000100, 0000000101, 0000000102, 0000000152, 0000000153, 0000000150, 0000000151, 0000000146, 0000000145, 0000000144, 0000000143, 0000000149, 0000000148, 0000000147, 0000000160, 0000000161, 0000000162, 0000000163, 0000000164, 0000000155, 0000000154, 0000000157, 0000000156, 0000000159, 0000000158, 0000000130, 0000000131, 0000000128, 0000000127, 0000000126, 0000000125, 0000000124, 0000000123, 0000000122, 0000000121, 0000000129, 0000000140, 0000000141, 0000000142, 0000000137, 0000000136, 0000000139, 0000000138, 0000000133, 0000000132, 0000000135, 0000000134, 0000000193, 0000000192, 0000000191, 0000000190, 0000000197, 0000000196, 0000000195, 0000000194, 0000000189, 0000000187, 0000000188, 0000000198, 0000000199, 0000000175, 0000000174, 0000000173, 0000000172, 0000000171, 0000000170, 0000000169, 0000000099, 0000000167, 0000000168, 0000000165, 0000000166, 0000000098, 0000000184, 0000000097, 0000000183, 0000000096, 0000000186, 0000000095, 0000000185, 0000000094, 0000000180, 0000000093, 0000000092, 0000000182, 0000000091, 0000000181, 0000000090, 0000000176, 0000000177, 0000000088, 0000000178, 0000000089, 0000000179]

迁移后的key分配是：

[root@vm redis-2.8.19] src/redis-cli
127.0.0.1:6379> keys *
1) "student2"
127.0.0.1:6379> quit
[root@vm redis-2.8.19] src/redis-cli -p 6380
127.0.0.1:6380> keys *
1) "manager4"
127.0.0.1:6380> quit
[root@vm redis-2.8.19] src/redis-cli -p 6381
127.0.0.1:6381> keys *
(empty list or set)
127.0.0.1:6381> quit
[root@vm redis-2.8.19] src/redis-cli -p 6382
127.0.0.1:6382> keys *
1) "teacher3"
127.0.0.1:6382> quit
[root@vm redis-2.8.19] src/redis-cli -p 6383
127.0.0.1:6383> keys *
1) "person1"
127.0.0.1:6383> quit