标签：heartbeat nfs drbd 自动挂载 故障切换 客户端
继续之前的操作，来完成heartbeat+drbd+nfs高可用的实现。
heartbeat+drbd还是使用原来的环境即可，然而需要添加一台nfs客户端，客户端的信息如下：
| 主机名 | IP地址 | 角色 |
| server136.contoso.com | 192.168.49.136 | nfs客户端 |
一、环境准备
1）注意：因为我的客户端是192.168.49.0/24端，所以这里需要把VIP改为这个网段，所以先修改两个节点的haresources文件，将VIP修改为192.168.49.100。可以根据实际情况来看是否需要这一步操作。
[root@heartbeat01 ~]# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.com IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4
[root@heartbeat02 ~]# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.com IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4
2）两个drbd节点要安装nfs（以下操作需要在两个节点上操作）
yum -y install rpcbind nfs-utils vi /etc/exports /data 192.168.49.0/255.255.255.0(rw,no_root_squash,no_all_squash,sync) /etc/init.d/rpcbind start chkconfig rpcbind on /etc/init.d/nfs start chkconfig nfs off chmod -R 777 /data
3)客户端上安装nfs
yum -y install rpcbind nfs-utils /etc/init.d/rpcbind start chkconfig rpcbind on
4)客户端上测试nfs挂载
showmount -e 192.168.49.100 mkdir /data mount -t nfs 192.168.49.100:/data /data echo "mount -t nfs 192.168.49.100:/data /data" >>/etc/rc.local
[root@server136 mnt]# showmount -e 172.16.49.100
Export list for 172.16.49.100:
/data 192.168.49.0/255.255.255.0
[root@server136 ~]# mount -t nfs 192.168.49.100:/data /data
[root@server136 ~]# cd /data
[root@server136 data]# mkdir nfsclient
[root@server136 data]# touch client.txt
[root@server136 data]# ls
10.txt 20.txt 30.txt 40.txt 50.txt 60.txt 70.txt 80.txt 90.txt client.txt
11.txt 21.txt 31.txt 41.txt 51.txt 61.txt 71.txt 81.txt 91.txt lost+found
12.txt 22.txt 32.txt 42.txt 52.txt 62.txt 72.txt 82.txt 92.txt nfsclient
13.txt 23.txt 33.txt 43.txt 53.txt 63.txt 73.txt 83.txt 93.txt
14.txt 24.txt 34.txt 44.txt 54.txt 64.txt 74.txt 84.txt 94.txt
15.txt 25.txt 35.txt 45.txt 55.txt 65.txt 75.txt 85.txt 95.txt
16.txt 26.txt 36.txt 46.txt 56.txt 66.txt 76.txt 86.txt 96.txt
17.txt 27.txt 37.txt 47.txt 57.txt 67.txt 77.txt 87.txt 97.txt
18.txt 28.txt 38.txt 48.txt 58.txt 68.txt 78.txt 88.txt 98.txt
19.txt 29.txt 39.txt 49.txt 59.txt 69.txt 79.txt 89.txt 99.txt
1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt
二、修改heartbeat资源配置
1）添加nfs服务
因为两个节点都没有设置nfs开机启动，而是让heartbeat接管nfs服务，在主节点获取VIP之后，然后启动nfs服务，所以需要将nfs服务加入heartbeat资源中
2）添加脚本文件
添加一个脚本文件killnfsd，目的是在发生故障切换时，能保障客户端挂载nfs正常。因为如果没有该脚本，在NFS服务切换后，客户端需要重新mount一下nfs共享目录，才可以正常使用，否则会出现“stale NFS file handle”的错误。
vi /etc/ha.d/resource.d/killnfsd #!/bin/bash # killall -9 nfsd /etc/init.d/nfs restart exit 0 chmod +x /etc/ha.d/resource.d/killnfsd
3）编辑/etc/ha.d/haresources文件
vi /etc/ha.d/haresources
[root@heartbeat01 ~]# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.com IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 nfs killnfsd
[root@heartbeat02 ~]# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.com IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 nfs killnfsd
三、配置nfs共享并手动挂载测试
1、正常启动heartbeat
依次启动heartbeat01和heartbeat02上的heartbeat服务，然后分别到两个节点上查看：
heartbeat01：
[root@heartbeat01 ~]# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
[root@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:4 nr:35740 dw:35744 dr:709 al:1 bm:9 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat01 ~]# ls /data
10.txt 20.txt 30.txt 40.txt 50.txt 60.txt 70.txt 80.txt 90.txt client.txt
11.txt 21.txt 31.txt 41.txt 51.txt 61.txt 71.txt 81.txt 91.txt lost+found
12.txt 22.txt 32.txt 42.txt 52.txt 62.txt 72.txt 82.txt 92.txt nfsclient
13.txt 23.txt 33.txt 43.txt 53.txt 63.txt 73.txt 83.txt 93.txt
14.txt 24.txt 34.txt 44.txt 54.txt 64.txt 74.txt 84.txt 94.txt
15.txt 25.txt 35.txt 45.txt 55.txt 65.txt 75.txt 85.txt 95.txt
16.txt 26.txt 36.txt 46.txt 56.txt 66.txt 76.txt 86.txt 96.txt
17.txt 27.txt 37.txt 47.txt 57.txt 67.txt 77.txt 87.txt 97.txt
18.txt 28.txt 38.txt 48.txt 58.txt 68.txt 78.txt 88.txt 98.txt
19.txt 29.txt 39.txt 49.txt 59.txt 69.txt 79.txt 89.txt 99.txt
1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt
[root@heartbeat01 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 2528) is running...
nfsd (pid 2544 2543 2542 2541 2540 2539 2538 2537) is running...
rpc.rquotad (pid 2523) is running...
heartbeat02：
[root@heartbeat02 ~]# ip a |grep 49.100
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:12288 nr:4 dw:4 dr:12288 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat02 ~]# ls /data
[root@heartbeat02 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
2、在客户端上手动挂载nfs共享目录
[root@server136 ~]# showmount -e 192.168.49.100
Export list for 192.168.49.100:
/data 192.168.49.0/255.255.255.0
[root@server136 ~]# mount -t nfs 192.168.49.100:/data /data
[root@server136 ~]# ls /data
10.txt 20.txt 30.txt 40.txt 50.txt 60.txt 70.txt 80.txt 90.txt client.txt
11.txt 21.txt 31.txt 41.txt 51.txt 61.txt 71.txt 81.txt 91.txt lost+found
12.txt 22.txt 32.txt 42.txt 52.txt 62.txt 72.txt 82.txt 92.txt nfsclient
13.txt 23.txt 33.txt 43.txt 53.txt 63.txt 73.txt 83.txt 93.txt
14.txt 24.txt 34.txt 44.txt 54.txt 64.txt 74.txt 84.txt 94.txt
15.txt 25.txt 35.txt 45.txt 55.txt 65.txt 75.txt 85.txt 95.txt
16.txt 26.txt 36.txt 46.txt 56.txt 66.txt 76.txt 86.txt 96.txt
17.txt 27.txt 37.txt 47.txt 57.txt 67.txt 77.txt 87.txt 97.txt
18.txt 28.txt 38.txt 48.txt 58.txt 68.txt 78.txt 88.txt 98.txt
19.txt 29.txt 39.txt 49.txt 59.txt 69.txt 79.txt 89.txt 99.txt
1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt
[root@server136 data]# touch test1.txt
此时，可以正常写入。
这里我添加一个脚本，来执行一个持续写入的任务，用来测试在故障切换过程中，向NFS共享目录/data中写入数据是否受影响。脚本内容如下：
#!/bin/bash
#
while true;do
for i in {1..1000}
do
touch /data/action$i.txt
sleep 2
done
done然后后台执行该脚本：
[root@server136 scripts]# sh wrnfs.sh & [1] 5272
四、故障切换测试
1、手工停止heartbeat01上的heartbeat服务，观察一下两个节点以及nfs客户单的状态：
1）heartbeat01:
[root@heartbeat01 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services: Done.
2）heartbeat02:
[root@heartbeat02 ~]# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:12292 nr:1600 dw:1604 dr:13001 al:1 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat02 ~]# ls /data
10.txt 21.txt 32.txt 43.txt 54.txt 65.txt 76.txt 87.txt 98.txt action5.txt
11.txt 22.txt 33.txt 44.txt 55.txt 66.txt 77.txt 88.txt 99.txt action6.txt
12.txt 23.txt 34.txt 45.txt 56.txt 67.txt 78.txt 89.txt 9.txt action7.txt
13.txt 24.txt 35.txt 46.txt 57.txt 68.txt 79.txt 8.txt action10.txt action8.txt
14.txt 25.txt 36.txt 47.txt 58.txt 69.txt 7.txt 90.txt action11.txt action9.txt
15.txt 26.txt 37.txt 48.txt 59.txt 6.txt 80.txt 91.txt action12.txt client.txt
16.txt 27.txt 38.txt 49.txt 5.txt 70.txt 81.txt 92.txt action13.txt lost+found
17.txt 28.txt 39.txt 4.txt 60.txt 71.txt 82.txt 93.txt action14.txt nfsclient
18.txt 29.txt 3.txt 50.txt 61.txt 72.txt 83.txt 94.txt action1.txt test1.txt
19.txt 2.txt 40.txt 51.txt 62.txt 73.txt 84.txt 95.txt action2.txt
1.txt 30.txt 41.txt 52.txt 63.txt 74.txt 85.txt 96.txt action3.txt
20.txt 31.txt 42.txt 53.txt 64.txt 75.txt 86.txt 97.txt action4.txt
[root@heartbeat02 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 2571) is running...
nfsd (pid 2587 2586 2585 2584 2583 2582 2581 2580) is running...
rpc.rquotad (pid 2566) is running...
3）nfs客户端
[root@server136 ~]# ls /data
10.txt 24.txt 38.txt 51.txt 65.txt 79.txt 92.txt action16.txt action7.txt
11.txt 25.txt 39.txt 52.txt 66.txt 7.txt 93.txt action17.txt action8.txt
12.txt 26.txt 3.txt 53.txt 67.txt 80.txt 94.txt action18.txt action9.txt
13.txt 27.txt 40.txt 54.txt 68.txt 81.txt 95.txt action19.txt client.txt
14.txt 28.txt 41.txt 55.txt 69.txt 82.txt 96.txt action1.txt lost+found
15.txt 29.txt 42.txt 56.txt 6.txt 83.txt 97.txt action20.txt nfsclient
16.txt 2.txt 43.txt 57.txt 70.txt 84.txt 98.txt action21.txt test1.txt
17.txt 30.txt 44.txt 58.txt 71.txt 85.txt 99.txt action22.txt
18.txt 31.txt 45.txt 59.txt 72.txt 86.txt 9.txt action23.txt
19.txt 32.txt 46.txt 5.txt 73.txt 87.txt action10.txt action24.txt
1.txt 33.txt 47.txt 60.txt 74.txt 88.txt action11.txt action2.txt
20.txt 34.txt 48.txt 61.txt 75.txt 89.txt action12.txt action3.txt
21.txt 35.txt 49.txt 62.txt 76.txt 8.txt action13.txt action4.txt
22.txt 36.txt 4.txt 63.txt 77.txt 90.txt action14.txt action5.txt
23.txt 37.txt 50.txt 64.txt 78.txt 91.txt action15.txt action6.txt
2、然后再启动heartbeat01的heartbeat服务，再次查看故障切换的情况
1）heartbeat01
[root@heartbeat01 ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO: Resource is stopped
Done.
[root@heartbeat01 ~]# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
[root@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1604 nr:38732 dw:40336 dr:1766 al:3 bm:9 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat01 ~]# ls /data
10.txt 39.txt 67.txt 95.txt action21.txt action4.txt action78.txt
11.txt 3.txt 68.txt 96.txt action22.txt action50.txt action79.txt
12.txt 40.txt 69.txt 97.txt action23.txt action51.txt action7.txt
13.txt 41.txt 6.txt 98.txt action24.txt action52.txt action80.txt
14.txt 42.txt 70.txt 99.txt action25.txt action53.txt action81.txt
15.txt 43.txt 71.txt 9.txt action26.txt action54.txt action82.txt
16.txt 44.txt 72.txt action100.txt action27.txt action55.txt action83.txt
17.txt 45.txt 73.txt action101.txt action28.txt action56.txt action84.txt
18.txt 46.txt 74.txt action102.txt action29.txt action57.txt action85.txt
19.txt 47.txt 75.txt action103.txt action2.txt action58.txt action86.txt
1.txt 48.txt 76.txt action104.txt action30.txt action59.txt action87.txt
20.txt 49.txt 77.txt action105.txt action31.txt action5.txt action88.txt
21.txt 4.txt 78.txt action106.txt action32.txt action60.txt action89.txt
22.txt 50.txt 79.txt action107.txt action33.txt action61.txt action8.txt
23.txt 51.txt 7.txt action108.txt action34.txt action62.txt action90.txt
24.txt 52.txt 80.txt action109.txt action35.txt action63.txt action91.txt
25.txt 53.txt 81.txt action10.txt action36.txt action64.txt action92.txt
26.txt 54.txt 82.txt action110.txt action37.txt action65.txt action93.txt
27.txt 55.txt 83.txt action111.txt action38.txt action66.txt action94.txt
28.txt 56.txt 84.txt action112.txt action39.txt action67.txt action95.txt
29.txt 57.txt 85.txt action11.txt action3.txt action68.txt action96.txt
2.txt 58.txt 86.txt action12.txt action40.txt action69.txt action97.txt
30.txt 59.txt 87.txt action13.txt action41.txt action6.txt action98.txt
31.txt 5.txt 88.txt action14.txt action42.txt action70.txt action99.txt
32.txt 60.txt 89.txt action15.txt action43.txt action71.txt action9.txt
33.txt 61.txt 8.txt action16.txt action44.txt action72.txt client.txt
34.txt 62.txt 90.txt action17.txt action45.txt action73.txt lost+found
35.txt 63.txt 91.txt action18.txt action46.txt action74.txt nfsclient
36.txt 64.txt 92.txt action19.txt action47.txt action75.txt test1.txt
37.txt 65.txt 93.txt action1.txt action48.txt action76.txt
38.txt 66.txt 94.txt action20.txt action49.txt action77.txt
[root@heartbeat01 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 4151) is running...
nfsd (pid 4168 4167 4166 4165 4164 4163 4162 4161) is running...
rpc.rquotad (pid 4146) is running...
2）heartbeat02
[root@heartbeat02 ~]# ip a |grep 49.100
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:15280 nr:1624 dw:4616 dr:13321 al:3 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat02 ~]# ls /data
[root@heartbeat02 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
3）nfs客户端
[root@server136 ~]# ls /data
10.txt 44.txt 78.txt action111.txt action145.txt action39.txt action72.txt
11.txt 45.txt 79.txt action112.txt action146.txt action3.txt action73.txt
12.txt 46.txt 7.txt action113.txt action147.txt action40.txt action74.txt
13.txt 47.txt 80.txt action114.txt action148.txt action41.txt action75.txt
14.txt 48.txt 81.txt action115.txt action149.txt action42.txt action76.txt
15.txt 49.txt 82.txt action116.txt action14.txt action43.txt action77.txt
16.txt 4.txt 83.txt action117.txt action150.txt action44.txt action78.txt
17.txt 50.txt 84.txt action118.txt action151.txt action45.txt action79.txt
18.txt 51.txt 85.txt action119.txt action152.txt action46.txt action7.txt
19.txt 52.txt 86.txt action11.txt action153.txt action47.txt action80.txt
1.txt 53.txt 87.txt action120.txt action154.txt action48.txt action81.txt
20.txt 54.txt 88.txt action121.txt action15.txt action49.txt action82.txt
21.txt 55.txt 89.txt action122.txt action16.txt action4.txt action83.txt
22.txt 56.txt 8.txt action123.txt action17.txt action50.txt action84.txt
23.txt 57.txt 90.txt action124.txt action18.txt action51.txt action85.txt
24.txt 58.txt 91.txt action125.txt action19.txt action52.txt action86.txt
25.txt 59.txt 92.txt action126.txt action1.txt action53.txt action87.txt
26.txt 5.txt 93.txt action127.txt action20.txt action54.txt action88.txt
27.txt 60.txt 94.txt action128.txt action21.txt action55.txt action89.txt
28.txt 61.txt 95.txt action129.txt action22.txt action56.txt action8.txt
29.txt 62.txt 96.txt action12.txt action23.txt action57.txt action90.txt
2.txt 63.txt 97.txt action130.txt action24.txt action58.txt action91.txt
30.txt 64.txt 98.txt action131.txt action25.txt action59.txt action92.txt
31.txt 65.txt 99.txt action132.txt action26.txt action5.txt action93.txt
32.txt 66.txt 9.txt action133.txt action27.txt action60.txt action94.txt
33.txt 67.txt action100.txt action134.txt action28.txt action61.txt action95.txt
34.txt 68.txt action101.txt action135.txt action29.txt action62.txt action96.txt
35.txt 69.txt action102.txt action136.txt action2.txt action63.txt action97.txt
36.txt 6.txt action103.txt action137.txt action30.txt action64.txt action98.txt
37.txt 70.txt action104.txt action138.txt action31.txt action65.txt action99.txt
38.txt 71.txt action105.txt action139.txt action32.txt action66.txt action9.txt
39.txt 72.txt action106.txt action13.txt action33.txt action67.txt client.txt
3.txt 73.txt action107.txt action140.txt action34.txt action68.txt lost+found
40.txt 74.txt action108.txt action141.txt action35.txt action69.txt nfsclient
41.txt 75.txt action109.txt action142.txt action36.txt action6.txt test1.txt
42.txt 76.txt action10.txt action143.txt action37.txt action70.txt
43.txt 77.txt action110.txt action144.txt action38.txt action71.txt
3、手动重启heartbeat01，查看故障切换的情况
1）heartbeat01
[root@heartbeat01 ~]# reboot
[root@heartbeat01 ~]#
Broadcast message from root@heartbeat01.contoso.com
(/dev/pts/0) at 4:23 ...
The system is going down for reboot NOW!
2）heartbeat02
[root@heartbeat02 ~]# ip a|grep 49.100
inet 192.168.49.100/24 scope global eth1
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:15284 nr:3924 dw:6920 dr:14074 al:3 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat02 ~]# ls /data
10.txt 49.txt 87.txt action125.txt action163.txt action32.txt action70.txt
11.txt 4.txt 88.txt action126.txt action164.txt action33.txt action71.txt
12.txt 50.txt 89.txt action127.txt action165.txt action34.txt action72.txt
13.txt 51.txt 8.txt action128.txt action166.txt action35.txt action73.txt
14.txt 52.txt 90.txt action129.txt action167.txt action36.txt action74.txt
15.txt 53.txt 91.txt action12.txt action168.txt action37.txt action75.txt
16.txt 54.txt 92.txt action130.txt action169.txt action38.txt action76.txt
17.txt 55.txt 93.txt action131.txt action16.txt action39.txt action77.txt
18.txt 56.txt 94.txt action132.txt action170.txt action3.txt action78.txt
19.txt 57.txt 95.txt action133.txt action171.txt action40.txt action79.txt
1.txt 58.txt 96.txt action134.txt action172.txt action41.txt action7.txt
20.txt 59.txt 97.txt action135.txt action173.txt action42.txt action80.txt
21.txt 5.txt 98.txt action136.txt action174.txt action43.txt action81.txt
22.txt 60.txt 99.txt action137.txt action175.txt action44.txt action82.txt
23.txt 61.txt 9.txt action138.txt action176.txt action45.txt action83.txt
24.txt 62.txt action100.txt action139.txt action177.txt action46.txt action84.txt
25.txt 63.txt action101.txt action13.txt action178.txt action47.txt action85.txt
26.txt 64.txt action102.txt action140.txt action179.txt action48.txt action86.txt
27.txt 65.txt action103.txt action141.txt action17.txt action49.txt action87.txt
28.txt 66.txt action104.txt action142.txt action180.txt action4.txt action88.txt
29.txt 67.txt action105.txt action143.txt action181.txt action50.txt action89.txt
2.txt 68.txt action106.txt action144.txt action182.txt action51.txt action8.txt
30.txt 69.txt action107.txt action145.txt action183.txt action52.txt action90.txt
31.txt 6.txt action108.txt action146.txt action184.txt action53.txt action91.txt
32.txt 70.txt action109.txt action147.txt action185.txt action54.txt action92.txt
33.txt 71.txt action10.txt action148.txt action186.txt action55.txt action93.txt
34.txt 72.txt action110.txt action149.txt action18.txt action56.txt action94.txt
35.txt 73.txt action111.txt action14.txt action19.txt action57.txt action95.txt
36.txt 74.txt action112.txt action150.txt action1.txt action58.txt action96.txt
37.txt 75.txt action113.txt action151.txt action20.txt action59.txt action97.txt
38.txt 76.txt action114.txt action152.txt action21.txt action5.txt action98.txt
39.txt 77.txt action115.txt action153.txt action22.txt action60.txt action99.txt
3.txt 78.txt action116.txt action154.txt action23.txt action61.txt action9.txt
40.txt 79.txt action117.txt action155.txt action24.txt action62.txt client.txt
41.txt 7.txt action118.txt action156.txt action25.txt action63.txt lost+found
42.txt 80.txt action119.txt action157.txt action26.txt action64.txt nfsclient
43.txt 81.txt action11.txt action158.txt action27.txt action65.txt test1.txt
44.txt 82.txt action120.txt action159.txt action28.txt action66.txt
45.txt 83.txt action121.txt action15.txt action29.txt action67.txt
46.txt 84.txt action122.txt action160.txt action2.txt action68.txt
47.txt 85.txt action123.txt action161.txt action30.txt action69.txt
48.txt 86.txt action124.txt action162.txt action31.txt action6.txt
[root@heartbeat02 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 4160) is running...
nfsd (pid 4176 4175 4174 4173 4172 4171 4170 4169) is running...
rpc.rquotad (pid 4155) is running...
3)nfs客户端
[root@server136 ~]# ls /data
10.txt 50.txt 90.txt action130.txt action170.txt action2.txt action6.txt
11.txt 51.txt 91.txt action131.txt action171.txt action30.txt action70.txt
12.txt 52.txt 92.txt action132.txt action172.txt action31.txt action71.txt
13.txt 53.txt 93.txt action133.txt action173.txt action32.txt action72.txt
14.txt 54.txt 94.txt action134.txt action174.txt action33.txt action73.txt
15.txt 55.txt 95.txt action135.txt action175.txt action34.txt action74.txt
16.txt 56.txt 96.txt action136.txt action176.txt action35.txt action75.txt
17.txt 57.txt 97.txt action137.txt action177.txt action36.txt action76.txt
18.txt 58.txt 98.txt action138.txt action178.txt action37.txt action77.txt
19.txt 59.txt 99.txt action139.txt action179.txt action38.txt action78.txt
1.txt 5.txt 9.txt action13.txt action17.txt action39.txt action79.txt
20.txt 60.txt action100.txt action140.txt action180.txt action3.txt action7.txt
21.txt 61.txt action101.txt action141.txt action181.txt action40.txt action80.txt
22.txt 62.txt action102.txt action142.txt action182.txt action41.txt action81.txt
23.txt 63.txt action103.txt action143.txt action183.txt action42.txt action82.txt
24.txt 64.txt action104.txt action144.txt action184.txt action43.txt action83.txt
25.txt 65.txt action105.txt action145.txt action185.txt action44.txt action84.txt
26.txt 66.txt action106.txt action146.txt action186.txt action45.txt action85.txt
27.txt 67.txt action107.txt action147.txt action187.txt action46.txt action86.txt
28.txt 68.txt action108.txt action148.txt action188.txt action47.txt action87.txt
29.txt 69.txt action109.txt action149.txt action189.txt action48.txt action88.txt
2.txt 6.txt action10.txt action14.txt action18.txt action49.txt action89.txt
30.txt 70.txt action110.txt action150.txt action190.txt action4.txt action8.txt
31.txt 71.txt action111.txt action151.txt action191.txt action50.txt action90.txt
32.txt 72.txt action112.txt action152.txt action192.txt action51.txt action91.txt
33.txt 73.txt action113.txt action153.txt action193.txt action52.txt action92.txt
34.txt 74.txt action114.txt action154.txt action194.txt action53.txt action93.txt
35.txt 75.txt action115.txt action155.txt action195.txt action54.txt action94.txt
36.txt 76.txt action116.txt action156.txt action196.txt action55.txt action95.txt
37.txt 77.txt action117.txt action157.txt action197.txt action56.txt action96.txt
38.txt 78.txt action118.txt action158.txt action198.txt action57.txt action97.txt
39.txt 79.txt action119.txt action159.txt action199.txt action58.txt action98.txt
3.txt 7.txt action11.txt action15.txt action19.txt action59.txt action99.txt
40.txt 80.txt action120.txt action160.txt action1.txt action5.txt action9.txt
41.txt 81.txt action121.txt action161.txt action20.txt action60.txt client.txt
42.txt 82.txt action122.txt action162.txt action21.txt action61.txt lost+found
43.txt 83.txt action123.txt action163.txt action22.txt action62.txt nfsclient
44.txt 84.txt action124.txt action164.txt action23.txt action63.txt test1.txt
45.txt 85.txt action125.txt action165.txt action24.txt action64.txt
46.txt 86.txt action126.txt action166.txt action25.txt action65.txt
47.txt 87.txt action127.txt action167.txt action26.txt action66.txt
48.txt 88.txt action128.txt action168.txt action27.txt action67.txt
49.txt 89.txt action129.txt action169.txt action28.txt action68.txt
4.txt 8.txt action12.txt action16.txt action29.txt action69.txt
4、将heartbeat01上的eth1（心跳网卡）down掉，查看故障切换状态
[root@heartbeat01 ~]# ifdown eth1
[root@heartbeat01 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:B3:83:3F
inet addr:192.168.49.133 Bcast:192.168.49.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb3:833f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4180 errors:0 dropped:0 overruns:0 frame:0
TX packets:3877 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3130356 (2.9 MiB) TX bytes:2748373 (2.6 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:28 errors:0 dropped:0 overruns:0 frame:0
TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1960 (1.9 KiB) TX bytes:1960 (1.9 KiB)
1）heartbeat01
[root@heartbeat01 ~]# ip a|grep 49.100
inet 192.168.49.100/24 scope global eth1
[root@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:2068 nr:2612 dw:4680 dr:1105 al:3 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat01 ~]# ls /data
10.txt 77.txt action143.txt action209.txt action275.txt action40.txt
11.txt 78.txt action144.txt action20.txt action276.txt action41.txt
12.txt 79.txt action145.txt action210.txt action277.txt action42.txt
13.txt 7.txt action146.txt action211.txt action278.txt action43.txt
14.txt 80.txt action147.txt action212.txt action279.txt action44.txt
15.txt 81.txt action148.txt action213.txt action27.txt action45.txt
16.txt 82.txt action149.txt action214.txt action280.txt action46.txt
17.txt 83.txt action14.txt action215.txt action281.txt action47.txt
18.txt 84.txt action150.txt action216.txt action282.txt action48.txt
19.txt 85.txt action151.txt action217.txt action283.txt action49.txt
1.txt 86.txt action152.txt action218.txt action284.txt action4.txt
20.txt 87.txt action153.txt action219.txt action285.txt action50.txt
21.txt 88.txt action154.txt action21.txt action286.txt action51.txt
22.txt 89.txt action155.txt action220.txt action287.txt action52.txt
23.txt 8.txt action156.txt action221.txt action288.txt action53.txt
24.txt 90.txt action157.txt action222.txt action289.txt action54.txt
25.txt 91.txt action158.txt action223.txt action28.txt action55.txt
26.txt 92.txt action159.txt action224.txt action290.txt action56.txt
27.txt 93.txt action15.txt action225.txt action291.txt action57.txt
28.txt 94.txt action160.txt action226.txt action292.txt action58.txt
29.txt 95.txt action161.txt action227.txt action293.txt action59.txt
2.txt 96.txt action162.txt action228.txt action294.txt action5.txt
30.txt 97.txt action163.txt action229.txt action295.txt action60.txt
31.txt 98.txt action164.txt action22.txt action296.txt action61.txt
32.txt 99.txt action165.txt action230.txt action297.txt action62.txt
33.txt 9.txt action166.txt action231.txt action298.txt action63.txt
34.txt action100.txt action167.txt action232.txt action299.txt action64.txt
35.txt action101.txt action168.txt action233.txt action29.txt action65.txt
36.txt action102.txt action169.txt action234.txt action2.txt action66.txt
37.txt action103.txt action16.txt action235.txt action300.txt action67.txt
38.txt action104.txt action170.txt action236.txt action301.txt action68.txt
39.txt action105.txt action171.txt action237.txt action302.txt action69.txt
3.txt action106.txt action172.txt action238.txt action303.txt action6.txt
40.txt action107.txt action173.txt action239.txt action304.txt action70.txt
41.txt action108.txt action174.txt action23.txt action305.txt action71.txt
42.txt action109.txt action175.txt action240.txt action306.txt action72.txt
43.txt action10.txt action176.txt action241.txt action307.txt action73.txt
44.txt action110.txt action177.txt action242.txt action308.txt action74.txt
45.txt action111.txt action178.txt action243.txt action309.txt action75.txt
46.txt action112.txt action179.txt action244.txt action30.txt action76.txt
47.txt action113.txt action17.txt action245.txt action310.txt action77.txt
48.txt action114.txt action180.txt action246.txt action311.txt action78.txt
49.txt action115.txt action181.txt action247.txt action312.txt action79.txt
4.txt action116.txt action182.txt action248.txt action313.txt action7.txt
50.txt action117.txt action183.txt action249.txt action314.txt action80.txt
51.txt action118.txt action184.txt action24.txt action315.txt action81.txt
52.txt action119.txt action185.txt action250.txt action316.txt action82.txt
53.txt action11.txt action186.txt action251.txt action317.txt action83.txt
54.txt action120.txt action187.txt action252.txt action318.txt action84.txt
55.txt action121.txt action188.txt action253.txt action319.txt action85.txt
56.txt action122.txt action189.txt action254.txt action31.txt action86.txt
57.txt action123.txt action18.txt action255.txt action320.txt action87.txt
58.txt action124.txt action190.txt action256.txt action321.txt action88.txt
59.txt action125.txt action191.txt action257.txt action322.txt action89.txt
5.txt action126.txt action192.txt action258.txt action323.txt action8.txt
60.txt action127.txt action193.txt action259.txt action324.txt action90.txt
61.txt action128.txt action194.txt action25.txt action325.txt action91.txt
62.txt action129.txt action195.txt action260.txt action326.txt action92.txt
63.txt action12.txt action196.txt action261.txt action327.txt action93.txt
64.txt action130.txt action197.txt action262.txt action328.txt action94.txt
65.txt action131.txt action198.txt action263.txt action329.txt action95.txt
66.txt action132.txt action199.txt action264.txt action32.txt action96.txt
67.txt action133.txt action19.txt action265.txt action330.txt action97.txt
68.txt action134.txt action1.txt action266.txt action331.txt action98.txt
69.txt action135.txt action200.txt action267.txt action332.txt action99.txt
6.txt action136.txt action201.txt action268.txt action33.txt action9.txt
70.txt action137.txt action202.txt action269.txt action34.txt client.txt
71.txt action138.txt action203.txt action26.txt action35.txt lost+found
72.txt action139.txt action204.txt action270.txt action36.txt nfsclient
73.txt action13.txt action205.txt action271.txt action37.txt test1.txt
74.txt action140.txt action206.txt action272.txt action38.txt
75.txt action141.txt action207.txt action273.txt action39.txt
76.txt action142.txt action208.txt action274.txt action3.txt
[root@heartbeat01 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 3192) is running...
nfsd (pid 3208 3207 3206 3205 3204 3203 3202 3201) is running...
rpc.rquotad (pid 3187) is running...
[root@heartbeat01 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:B3:83:3F
inet addr:192.168.49.133 Bcast:192.168.49.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb3:833f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6166 errors:0 dropped:0 overruns:0 frame:0
TX packets:7300 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3365517 (3.2 MiB) TX bytes:6246399 (5.9 MiB)
eth1 Link encap:Ethernet HWaddr 00:0C:29:B3:83:49
inet addr:192.168.49.100 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb3:8349/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:447 errors:0 dropped:0 overruns:0 frame:0
TX packets:183 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:119694 (116.8 KiB) TX bytes:43324 (42.3 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:44 errors:0 dropped:0 overruns:0 frame:0
TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3080 (3.0 KiB) TX bytes:3080 (3.0 KiB)
2）heartbeat02
[root@heartbeat02 ~]# ip a|grep 49.100
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:17896 nr:10696 dw:16304 dr:14402 al:3 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat02 ~]# ls /data
[root@heartbeat02 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
3）nfs客户端
[root@server136 scripts]# ls /data
10.txt action102.txt action194.txt action285.txt action376.txt action467.txt
11.txt action103.txt action195.txt action286.txt action377.txt action468.txt
12.txt action104.txt action196.txt action287.txt action378.txt action469.txt
13.txt action105.txt action197.txt action288.txt action379.txt action46.txt
14.txt action106.txt action198.txt action289.txt action37.txt action470.txt
15.txt action107.txt action199.txt action28.txt action380.txt action471.txt
16.txt action108.txt action19.txt action290.txt action381.txt action472.txt
17.txt action109.txt action1.txt action291.txt action382.txt action473.txt
18.txt action10.txt action200.txt action292.txt action383.txt action474.txt
19.txt action110.txt action201.txt action293.txt action384.txt action475.txt
1.txt action111.txt action202.txt action294.txt action385.txt action476.txt
20.txt action112.txt action203.txt action295.txt action386.txt action477.txt
21.txt action113.txt action204.txt action296.txt action387.txt action478.txt
22.txt action114.txt action205.txt action297.txt action388.txt action479.txt
23.txt action115.txt action206.txt action298.txt action389.txt action47.txt
24.txt action116.txt action207.txt action299.txt action38.txt action480.txt
25.txt action117.txt action208.txt action29.txt action390.txt action481.txt
26.txt action118.txt action209.txt action2.txt action391.txt action482.txt
27.txt action119.txt action20.txt action300.txt action392.txt action483.txt
28.txt action11.txt action210.txt action301.txt action393.txt action484.txt
29.txt action120.txt action211.txt action302.txt action394.txt action485.txt
2.txt action121.txt action212.txt action303.txt action395.txt action486.txt
30.txt action122.txt action213.txt action304.txt action396.txt action487.txt
31.txt action123.txt action214.txt action305.txt action397.txt action488.txt
32.txt action124.txt action215.txt action306.txt action398.txt action489.txt
33.txt action125.txt action216.txt action307.txt action399.txt action48.txt
34.txt action126.txt action217.txt action308.txt action39.txt action490.txt
35.txt action127.txt action218.txt action309.txt action3.txt action491.txt
36.txt action128.txt action219.txt action30.txt action400.txt action492.txt
37.txt action129.txt action21.txt action310.txt action401.txt action493.txt
38.txt action12.txt action220.txt action311.txt action402.txt action494.txt
39.txt action130.txt action221.txt action312.txt action403.txt action495.txt
3.txt action131.txt action222.txt action313.txt action404.txt action496.txt
40.txt action132.txt action223.txt action314.txt action405.txt action497.txt
41.txt action133.txt action224.txt action315.txt action406.txt action498.txt
42.txt action134.txt action225.txt action316.txt action407.txt action499.txt
43.txt action135.txt action226.txt action317.txt action408.txt action49.txt
44.txt action136.txt action227.txt action318.txt action409.txt action4.txt
45.txt action137.txt action228.txt action319.txt action40.txt action500.txt
46.txt action138.txt action229.txt action31.txt action410.txt action501.txt
47.txt action139.txt action22.txt action320.txt action411.txt action502.txt
48.txt action13.txt action230.txt action321.txt action412.txt action503.txt
49.txt action140.txt action231.txt action322.txt action413.txt action50.txt
4.txt action141.txt action232.txt action323.txt action414.txt action51.txt
50.txt action142.txt action233.txt action324.txt action415.txt action52.txt
51.txt action143.txt action234.txt action325.txt action416.txt action53.txt
52.txt action144.txt action235.txt action326.txt action417.txt action54.txt
53.txt action145.txt action236.txt action327.txt action418.txt action55.txt
54.txt action146.txt action237.txt action328.txt action419.txt action56.txt
55.txt action147.txt action238.txt action329.txt action41.txt action57.txt
56.txt action148.txt action239.txt action32.txt action420.txt action58.txt
57.txt action149.txt action23.txt action330.txt action421.txt action59.txt
58.txt action14.txt action240.txt action331.txt action422.txt action5.txt
59.txt action150.txt action241.txt action332.txt action423.txt action60.txt
5.txt action151.txt action242.txt action333.txt action424.txt action61.txt
60.txt action152.txt action243.txt action334.txt action425.txt action62.txt
61.txt action153.txt action244.txt action335.txt action426.txt action63.txt
62.txt action154.txt action245.txt action336.txt action427.txt action64.txt
63.txt action155.txt action246.txt action337.txt action428.txt action65.txt
64.txt action156.txt action247.txt action338.txt action429.txt action66.txt
65.txt action157.txt action248.txt action339.txt action42.txt action67.txt
66.txt action158.txt action249.txt action33.txt action430.txt action68.txt
67.txt action159.txt action24.txt action340.txt action431.txt action69.txt
68.txt action15.txt action250.txt action341.txt action432.txt action6.txt
69.txt action160.txt action251.txt action342.txt action433.txt action70.txt
6.txt action161.txt action252.txt action343.txt action434.txt action71.txt
70.txt action162.txt action253.txt action344.txt action435.txt action72.txt
71.txt action163.txt action254.txt action345.txt action436.txt action73.txt
72.txt action164.txt action255.txt action346.txt action437.txt action74.txt
73.txt action165.txt action256.txt action347.txt action438.txt action75.txt
74.txt action166.txt action257.txt action348.txt action439.txt action76.txt
75.txt action167.txt action258.txt action349.txt action43.txt action77.txt
76.txt action168.txt action259.txt action34.txt action440.txt action78.txt
77.txt action169.txt action25.txt action350.txt action441.txt action79.txt
78.txt action16.txt action260.txt action351.txt action442.txt action7.txt
79.txt action170.txt action261.txt action352.txt action443.txt action80.txt
7.txt action171.txt action262.txt action353.txt action444.txt action81.txt
80.txt action172.txt action263.txt action354.txt action445.txt action82.txt
81.txt action173.txt action264.txt action355.txt action446.txt action83.txt
82.txt action174.txt action265.txt action356.txt action447.txt action84.txt
83.txt action175.txt action266.txt action357.txt action448.txt action85.txt
84.txt action176.txt action267.txt action358.txt action449.txt action86.txt
85.txt action177.txt action268.txt action359.txt action44.txt action87.txt
86.txt action178.txt action269.txt action35.txt action450.txt action88.txt
87.txt action179.txt action26.txt action360.txt action451.txt action89.txt
88.txt action17.txt action270.txt action361.txt action452.txt action8.txt
89.txt action180.txt action271.txt action362.txt action453.txt action90.txt
8.txt action181.txt action272.txt action363.txt action454.txt action91.txt
90.txt action182.txt action273.txt action364.txt action455.txt action92.txt
91.txt action183.txt action274.txt action365.txt action456.txt action93.txt
92.txt action184.txt action275.txt action366.txt action457.txt action94.txt
93.txt action185.txt action276.txt action367.txt action458.txt action95.txt
94.txt action186.txt action277.txt action368.txt action459.txt action96.txt
95.txt action187.txt action278.txt action369.txt action45.txt action97.txt
96.txt action188.txt action279.txt action36.txt action460.txt action98.txt
97.txt action189.txt action27.txt action370.txt action461.txt action99.txt
98.txt action18.txt action280.txt action371.txt action462.txt action9.txt
99.txt action190.txt action281.txt action372.txt action463.txt client.txt
9.txt action191.txt action282.txt action373.txt action464.txt lost+found
action100.txt action192.txt action283.txt action374.txt action465.txt nfsclient
action101.txt action193.txt action284.txt action375.txt action466.txt test1.txt
4）查看heartbeat01上的日志
[root@heartbeat01 ~]# tail -f /var/log/ha-log
Sep 26 05:01:00 heartbeat01.contoso.com heartbeat: [1559]: ERROR: glib: mcast_write: Unable to send HBcomm packet eth1 225.0.0.1:694 len=225 [-1]: Invalid argument
Sep 26 05:01:00 heartbeat01.contoso.com heartbeat: [1559]: ERROR: write_child: write failure on mcast eth1.: Invalid argument
Sep 26 05:01:02 heartbeat01.contoso.com heartbeat: [1559]: ERROR: glib: mcast_write: Unable to send HBcomm packet eth1 225.0.0.1:694 len=225 [-1]: Invalid argument
Sep 26 05:01:02 heartbeat01.contoso.com heartbeat: [1559]: ERROR: write_child: write failure on mcast eth1.: Invalid argument
Sep 26 05:01:04 heartbeat01.contoso.com heartbeat: [1559]: ERROR: glib: mcast_write: Unable to send HBcomm packet eth1 225.0.0.1:694 len=225 [-1]: Invalid argument
Sep 26 05:01:04 heartbeat01.contoso.com heartbeat: [1559]: ERROR: write_child: write failure on mcast eth1.: Invalid argument
Sep 26 05:01:06 heartbeat01.contoso.com heartbeat: [1559]: ERROR: glib: mcast_write: Unable to send HBcomm packet eth1 225.0.0.1:694 len=225 [-1]: Invalid argument
Sep 26 05:01:06 heartbeat01.contoso.com heartbeat: [1559]: ERROR: write_child: write failure on mcast eth1.: Invalid argument
Sep 26 05:01:06 heartbeat01.contoso.com heartbeat: [1559]: WARN: Temporarily Suppressing write error messages
Sep 26 05:01:06 heartbeat01.contoso.com heartbeat: [1559]: WARN: Is a cable unplugged on mcast eth1?
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: [1555]: WARN: node heartbeat02.contoso.com: is dead
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: [1555]: WARN: No STONITH device configured.
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: [1555]: WARN: Shared disks are not protected.
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: [1555]: info: Resources being acquired from heartbeat02.contoso.com.
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: [1555]: info: Link heartbeat02.contoso.com:eth1 dead.
Sep 26 05:01:17 heartbeat01.contoso.com ipfail: [1580]: info: Status update: Node heartbeat02.contoso.com now has status dead
harc(default)[2675]: 2016/09/26_05:01:17 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[2712]: 2016/09/26_05:01:17 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[2712]: 2016/09/26_05:01:17 info: mach_down takeover complete for node heartbeat02.contoso.com.
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: [1555]: info: mach_down takeover complete.
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.49.100)[2729]: 2016/09/26_05:01:17 INFO: Resource is stopped
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: [2676]: info: Local Resource acquisition completed.
harc(default)[2844]: 2016/09/26_05:01:17 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default)[2844]: 2016/09/26_05:01:17 received ip-request-resp IPaddr::192.168.49.100/24/eth1 OK yes
ResourceManager(default)[2867]: 2016/09/26_05:01:17 info: Acquiring resource group: heartbeat01.contoso.com IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 nfs killnfsd
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.49.100)[2895]: 2016/09/26_05:01:17 INFO: Resource is stopped
ResourceManager(default)[2867]: 2016/09/26_05:01:17 info: Running /etc/ha.d/resource.d/IPaddr 192.168.49.100/24/eth1 start
IPaddr(IPaddr_192.168.49.100)[3017]: 2016/09/26_05:01:17 INFO: Adding inet address 192.168.49.100/24 to device eth1
IPaddr(IPaddr_192.168.49.100)[3017]: 2016/09/26_05:01:17 INFO: Bringing device eth1 up
IPaddr(IPaddr_192.168.49.100)[3017]: 2016/09/26_05:01:17 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.49.100 eth1 192.168.49.100 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.49.100)[2991]: 2016/09/26_05:01:17 INFO: Success
Sep 26 05:01:17 heartbeat01.contoso.com ipfail: [1580]: info: NS: We are dead. :<
Sep 26 05:01:17 heartbeat01.contoso.com ipfail: [1580]: info: Link Status update: Link heartbeat02.contoso.com/eth1 now has status dead
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[3121]: 2016/09/26_05:01:17 INFO: Running OK
Sep 26 05:01:18 heartbeat01.contoso.com ipfail: [1580]: info: We are dead. :<
Sep 26 05:01:18 heartbeat01.contoso.com ipfail: [1580]: info: Asking other side for ping node count.
注意：上面将heartbeat01的网卡eth1停掉，之所以没有发生脑裂是因为我在/etc/drbd.d/test.res中配置文件中的节点地址是192.168.49.0/24段，也就是eth0的网段，所以没有发生裂脑。
5、下面将该配置修改为172.16.49.0/24段，再进行测试：
on heartbeat01.contoso.com {
device /dev/drbd0;
disk /dev/sdb1;
address 172.16.49.133:7788;
flexible-meta-disk internal;
}
on heartbeat02.contoso.com {
device /dev/drbd0;
disk /dev/sdb1;
address 172.16.49.134:7788;
meta-disk internal;
1）heartbeat01
[root@heartbeat01 ~]# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
[root@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:2324 nr:0 dw:9888 dr:3707 al:4 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:40
[root@heartbeat01 ~]# ls -l /data|wc -l
942
[root@heartbeat01 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 9672) is running...
nfsd (pid 9688 9687 9686 9685 9684 9683 9682 9681) is running...
rpc.rquotad (pid 9667) is running...
2)heartbeat02
[root@heartbeat02 ~]# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:0 nr:2324 dw:2420 dr:3093 al:3 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:88
[root@heartbeat02 ~]# ls -l /data |wc -l
942
[root@heartbeat02 ~]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 6280) is running...
nfsd (pid 6296 6295 6294 6293 6292 6291 6290 6289) is running...
rpc.rquotad (pid 6275) is running...
3)nfs客户端
[root@server136 scripts]# ls -l /data|wc -l
968
[root@heartbeat01 ~]# ls -l /data|wc -l
993
[root@heartbeat02 ~]# ls -l /data |wc -l
942
此时，客户端虽然仍在进行数据写入，但是数据是写入到heartbeat01中，而且此时两个节点都有192.168.49.100这个IP地址，drbd中每个节点对端的状态都是Unknown状态，这就是典型的脑裂状况。
[root@heartbeat01 ~]# ifup eth1
Determining if ip address 172.16.49.133 is already in use for device eth1...
将heartbeat01的eth1启动后，发现heartbeat的裂脑状态消除，然而，drbd两个节点的状态依然不对，如下：
[root@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:2324 nr:0 dw:16888 dr:5012 al:4 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:6552
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----
ns:0 nr:2324 dw:2436 dr:3093 al:3 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:88
五、发生脑裂的处理
1）在secondary主机上执行如下命令：
drbdadm secondary test drbdadm disconnect test drbdadm -- --discard-my-data connect test
2）在Primary主机上执行如下命令：
drbdadm disconnect all drbdadm connect test
[root@heartbeat02 ~]# drbdadm disconnect test
[root@heartbeat02 ~]# drbdadm -- --discard-my-data connect test
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----
ns:0 nr:0 dw:2436 dr:3093 al:3 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:88
[root@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:2324 nr:0 dw:16888 dr:5012 al:4 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:6552
[root@heartbeat01 ~]# drbdadm disconnect all
[root@heartbeat01 ~]# drbdadm connect test
[root@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:7312 nr:0 dw:17644 dr:12312 al:4 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:7336 dw:9772 dr:3093 al:3 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
此时，两台drbd主机的状态已经恢复到一致状态，脑裂的问题解决完毕。
本文出自 “IT小二郎” 博客，请务必保留此出处http://jerry12356.blog.51cto.com/4308715/1856726
标签：heartbeat nfs drbd 自动挂载 故障切换 客户端
原文地址：http://jerry12356.blog.51cto.com/4308715/1856726
