基于 MHA 的MySQL高可用-CentOS7（实例）

时间：2019-11-04 13:31:58 阅读：65 评论：0 收藏：0 [点我收藏+]

标签：global 功能 rtb between arch routes sam yum utils

环境部署

角色	IP地址	主机名	ServerID	数据库类型
Primary Master	192.168.200.111	server01	1	写
Secondary Master	192.168.200.112	server02	2	写
Slave1	192.168.200.113	server03	3	读
Slave2	192.168.200.114	server04	4	读
Manager	192.168.200.115	server05	-	监控复制组

配置所有主机名称

master1 主机:

hostname server01

bash

master2 主机:

hostname server02

bash

slave1 主机:

hostname server03

bash

slave2 主机:

hostname server04

bash

manager 主机:

hostname server05

bash

配置所有主机名映射

[root@server05 ~]# vim /etc/hosts

192.168.200.111 server01

192.168.200.112 server02

192.168.200.113 server03

192.168.200.114 server04

192.168.200.115 server05

scp /etc/hosts 192.168.200.111:/etc/

scp /etc/hosts 192.168.200.112:/etc/

scp /etc/hosts 192.168.200.113:/etc/

scp /etc/hosts 192.168.200.114:/etc/

所有主机关闭防火墙和安全机制

iptables -F

systemctl stop firewalld

setenforce 0

下载 mha-manager 和 mha-node

http://downloads.mariadb.com/MHA/

安装 MHA node

所有主机安装 MHA node 及相关 perl 依赖包

安装 epel 源:

rpm -ivh epel-release-latest-7.noarch.rpm

yum -y install perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker

rpm -q perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder

perl-ExtUtils-MakeMaker

perl-DBD-MySQL-4.023-6.el7.x86_64

perl-DBI-1.627-4.el7.x86_64

perl-CPAN-1.9800-292.el7.noarch

perl-ExtUtils-CBuilder-0.28.2.6-292.el7.noarch

perl-ExtUtils-MakeMaker-6.68-3.el7.noarch

注意：安装后建议检查一下所需软件包是否全部安装

所有主机上安装 MHA Node

tar xf mha4mysql-node-0.56.tar.gz

cd mha4mysql-node-0.56/

perl Makefile.PL

make && make install

MHA Node 安装完后会在 /usr/local/bin 生成以下脚本

ls -l /usr/local/bin/

总用量 40

-r-xr-xr-x 1 root root 16346 12 月 29 15:07 apply_diff_relay_logs

-r-xr-xr-x 1 root root 4807 12 月 29 15:07 filter_mysqlbinlog

-r-xr-xr-x 1 root root 7401 12 月 29 15:07 purge_relay_logs

-r-xr-xr-x 1 root root 7395 12 月 29 15:07 save_binary_logs

安装 MHA Manger

注意：安装 MHA Manger 之前也需要安装 MHA Node

首先安装 MHA Manger 依赖的 perl 模块（我这里使用 yum 安装）

yum -y install perl perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes

yum -y install perl-Config-Tiny-2.14-7.el7.noarch.rpm

rpm -q perl cpan perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes perl-Config-Tiny

perl-5.16.3-292.el7.x86_64

perl-Log-Dispatch-2.41-1.el7.1.noarch

perl-Parallel-ForkManager-1.18-2.el7.noarch

perl-DBD-MySQL-4.023-6.el7.x86_64

perl-DBI-1.627-4.el7.x86_64

perl-Time-HiRes-1.9725-3.el7.x86_64

perl-Config-Tiny-2.14-7.el7.noarch

注意：之前时候 perl-Config-Tiny.noarch 没有安装成功,后来用 cpan(cpan install Config::Tiny )

安装 MHA Manger 软件包

tar xf mha4mysql-manager-0.56.tar.gz

cd mha4mysql-manager-0.56/

perl Makefile.PL

make && make install

安装完成后会有以下脚本文件

ls -l /usr/local/bin/

总用量 76

-r-xr-xr-x 1 root root 16346 12 月 29 15:08 apply_diff_relay_logs

-r-xr-xr-x 1 root root 4807 12 月 29 15:08 filter_mysqlbinlog

-r-xr-xr-x 1 root root 1995 12 月 29 15:37 masterha_check_repl

-r-xr-xr-x 1 root root 1779 12 月 29 15:37 masterha_check_ssh

-r-xr-xr-x 1 root root 1865 12 月 29 15:37 masterha_check_status

-r-xr-xr-x 1 root root 3201 12 月 29 15:37 masterha_conf_host

-r-xr-xr-x 1 root root 2517 12 月 29 15:37 masterha_manager

-r-xr-xr-x 1 root root 2165 12 月 29 15:37 masterha_master_monitor

-r-xr-xr-x 1 root root 2373 12 月 29 15:37 masterha_master_switch

-r-xr-xr-x 1 root root 3879 12 月 29 15:37 masterha_secondary_check

-r-xr-xr-x 1 root root 1739 12 月 29 15:37 masterha_stop

-r-xr-xr-x 1 root root 7401 12 月 29 15:08 purge_relay_logs

-r-xr-xr-x 1 root root 7395 12 月 29 15:08 save_binary_logs

配置 SSH 密钥对验证

服务器之间需要实现密钥对验证。关于配置密钥对验证可看下面步骤。但是有一点需要注意：不能禁止 password 登陆，否则会出现错误

1. 服务器先生成一个密钥对

2. 把自己的公钥传给对方

Server05(192.168.200.115)上:

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

注意：Server05 需要连接每个主机测试，因为第一次连接的时候需要输入 yes，影响后期故障切换时，对于每个主机的 SSH 控制。

ssh server01

yes
ssh server02

yes

ssh server03

yes

ssh server04

yes

Primary Master(192.168.200.111):

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

Secondary Master(192.168.200.112):

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

slave1(192.168.200.113):

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

slave2(192.168.200.114):

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

安装 mysql

111-114 主机上的操作：

yum -y install mariadb mariadb-server mariadb-devel

systemctl start mariadb

netstat -lnpt | grep :3306

设置数据库初始密码（后续操作中使用）

mysqladmin -u root password 123456

搭建主从复制环境

注意：binlog-do-db 和 replicate-ignore-db 设置必须相同。 MHA 在启动时候会检测过滤规则，如果过滤规则不同，MHA 将不启动监控和故障转移功能。

修改 mysql 主机的配置文件

Primary Master(192.168.200.111):

vim /etc/my.cnf

[mysqld]

server-id = 1

log-bin=master-bin

log-slave-updates=true

relay_log_purge=0

systemctl restart mariadb

Secondary Master(192.168.200.112):

vim /etc/my.cnf

[mysqld]

server-id=2

log-bin=master-bin

log-slave-updates=true

relay_log_purge=0

systemctl restart mariadb

slave1(192.168.200.113):

vim /etc/my.cnf

[mysqld]

server-id=3

log-bin=mysql-bin

relay-log=slave-relay-bin

log-slave-updates=true

relay_log_purge=0

systemctl restart mariadb

slave2(192.168.200.114):

vim /etc/my.cnf

[mysqld]

server-id=4

log-bin=mysql-bin

relay-log=slave-relay-bin

log-slave-updates=true

relay_log_purge=0

systemctl restart mariadb

在 Primary Master(192.168.200.111)上对旧数据进行备份

mysqldump --master-data=2 --single-transaction -R --triggers -A > all.sql

参数解释：

--master-data=2 备份时刻记录 master 的 Binlog 位置和 Position

--single-transaction 获取一致性快照

-R 备份存储过程和函数

-triggres 备份触发器

-A 备份所有的库

mysql 服务器创建复制授权用户

grant replication slave on *.* to ‘repl‘@‘192.168.200.%‘ identified by ‘123456‘;

flush privileges;

查看主库备份时的 binlog 名称和位置

MariaDB [(none)]> show master status;

+-------------------+----------+--------------+------------------+

+-------------------+----------+--------------+------------------+

| master-bin.000001 | 474 | | |

+-------------------+----------+--------------+------------------+

1 row in set (0.00 sec)

把数据备份复制其他主机

scp all.sql 192.168.200.112:/tmp/

scp all.sql 192.168.200.113:/tmp/

scp all.sql 192.168.200.114:/tmp/

导入备份数据到 112-114 主机上执行复制相关命令

mysql -uroot -p123456 < /tmp/all.sql

stop slave;

CHANGE MASTER TO

MASTER_HOST=‘192.168.200.111‘,

MASTER_USER=‘repl‘,

MASTER_PASSWORD=‘123456‘,

MASTER_LOG_FILE=‘master-bin.000001‘,

MASTER_LOG_POS=474;

start slave;

show slave status\G

# 检查 IO 和 SQL 线程是否为：yes

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

主从同步故障处理

Slave_IO_Running: No

Slave_SQL_Running: Yes

-----------------------------------忽略部分信息-----------------------------------

Last_IO_Errno: 1236

Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: ‘Could not find first log file name in binary log index file‘

-----------------------------------忽略部分信息-----------------------------------

处理方式：

stop slave;

reset slave;

set global sql_slave_skip_counter =1;

start slave;

三台 slave 服务器设置 read_only 状态

从库对外只提供读服务，只所以没有写进 mysql 配置文件，是因为随时 server02 会提升为 master

[root@server02 ~]# mysql -uroot -p123456 -e ‘set global read_only=1‘

[root@server03 ~]# mysql -uroot -p123456 -e ‘set global read_only=1‘

[root@server04 ~]# mysql -uroot -p123456 -e ‘set global read_only=1‘

创建监控用户（111-114 主机上的操作）：

grant all privileges on *.* to ‘root‘@‘192.168.200.%‘ identified by ‘123456‘;

flush privileges;

为自己的主机名授权：

grant all privileges on *.* to ‘root‘@‘server04‘ identified by ‘123456‘;

flush privileges;

到这里整个 mysql 主从集群环境已经搭建完毕。

配置 MHA 环境

创建 MHA 的工作目录及相关配置文件

Server05(192.168.200.115)：在软件包解压后的目录里面有样例配置文件

mkdir /etc/masterha

cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha

修改 app1.cnf 配置文件

/usr/local/bin/master_ip_failover 脚本需要根据自己环境修改 ip 和网卡名称等。

vim /etc/masterha/app1.cnf

[server default]

#设置 manager 的工作日志

manager_workdir=/var/log/masterha/app1

#设置 manager 的日志,这两条都是默认存在的

manager_log=/var/log/masterha/app1/manager.log

#设置 master 默认保存 binlog 的位置,以便 MHA 可以找到 master 日志

master_binlog_dir=/var/lib/mysql

#设置自动 failover 时候的切换脚本

master_ip_failover_script= /usr/local/bin/master_ip_failover

#设置 mysql 中 root 用户的密码

password=123456

user=root

#ping 包的时间间隔

ping_interval=1

#设置远端 mysql 在发生切换时保存 binlog 的具体位置

remote_workdir=/tmp

#设置复制用户的密码和用户名

repl_password=123456

repl_user=repl

[server1]

hostname=server01

port=3306

[server2]

hostname=server02

candidate_master=1

port=3306

check_repl_delay=0

[server3]

hostname=server03

port=3306

[server4]

hostname=server04

port=3306

配置故障转移脚本

[root@server05 ~]# vim /usr/local/bin/master_ip_failover

#!/usr/bin/env perl

use strict;

use warnings FATAL => ‘all‘;

use Getopt::Long;

my (

$command, $ssh_user, $orig_master_host, $orig_master_ip,

$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,

);

my $vip = ‘192.168.200.100‘; # 写入 VIP

my $key = "1"; #非 keepalived 方式切换脚本使用的

my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip";

my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令

$ssh_user = "root";

GetOptions(

‘command=s‘ => \$command,

‘ssh_user=s‘ => \$ssh_user,

‘orig_master_host=s‘ => \$orig_master_host,

‘orig_master_ip=s‘ => \$orig_master_ip,

‘orig_master_port=i‘ => \$orig_master_port,

‘new_master_host=s‘ => \$new_master_host,

‘new_master_ip=s‘ => \$new_master_ip,

‘new_master_port=i‘ => \$new_master_port,

);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

# $orig_master_host, $orig_master_ip, $orig_master_port are passed.

# If you manage master ip address at global catalog database,

# invalidate orig_master_ip here.

my $exit_code = 1;

#eval {

# print "Disabling the VIP on old master: $orig_master_host \n";

# &stop_vip();

# $exit_code = 0;

#};

eval {

print "Disabling the VIP on old master: $orig_master_host \n";

#my $ping=`ping -c 1 10.0.0.13 | grep "packet loss" | awk -F‘,‘ ‘{print $3}‘ | awk ‘{print $1}‘`;

#if ( $ping le "90.0%"&& $ping gt "0.0%" ){

#$exit_code = 0;

#else {

&stop_vip();

# updating global catalog, etc

$exit_code = 0;

};

if ($@) {

warn "Got Error: $@\n";

exit $exit_code;

}

exit $exit_code;

}

elsif ( $command eq "start" ) {

# all arguments are passed.

# If you manage master ip address at global catalog database,

# activate new_master_ip here.

# You can also grant write access (create user, set read_only=0, etc) here.

my $exit_code = 10;

eval {

print "Enabling the VIP - $vip on the new master - $new_master_host \n";

&start_vip();

$exit_code = 0;

};

if ($@) {

warn $@;

exit $exit_code;

}

exit $exit_code;

}

elsif ( $command eq "status" ) {

print "Checking the Status of the script.. OK \n";

`ssh $ssh_user\@$orig_master_ip \" $ssh_start_vip \"`;

exit 0;

}

else {

&usage();

exit 1;

}

# A simple system call that enable the VIP on the new master

sub start_vip() {

`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;

}

# A simple system call that disable the VIP on the old_master

sub stop_vip() {

`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;

}

sub usage {

"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port -- new_master_host=host --new_master_ip=ip --new_master_port=port\n"; }

[root@server05 ~]# chmod +x /usr/local/bin/master_ip_failover

设置从库 relay log 的清除方式(112-114)：

mysql -uroot -p123456 -e ‘set global relay_log_purge=0;‘

注意：

MHA 在故障切换的过程中，从库的恢复过程依赖于 relay log 的相关信息，所以这里要将 relay log 的自动清除设置为 OFF，采用手动清除 relay log 的方式。在默认情况下，从服务器上的中继日志会在 SQL 线程执行完毕后被自动删除。但是在 MHA 环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动清除功能。定期清除中继日志需要考虑到复制延时的问题。在 ext3 的文件系统下，删除大的文件需要一定的时间，会导致严重的复制延时。为了避免复制延时，需要暂时为中继日志创建硬链接，因为在 linux 系统中通过硬链接删除大文件速度会很快。（在 mysql 数据库中，删除大表时，通常也采用建立硬链接的方式）

配置从库（112-114）relay_log 清除脚本加入计划任务

MHA 节点中包含了 pure_relay_logs 命令工具，它可以为中继日志创建硬链接，执行 SET GLOBAL relay_log_purge=1,等待几秒钟以便 SQL 线程切换到新的中继日志，再执行 SET GLOBAL relay_log_purge=0。

vim purge_relay_log.sh

#!/bin/bash

user=root

passwd=123456 #注意：数据库要有密码,填自己所设置的密码就可以，前面设置过 port=3306

log_dir=‘/tmp‘

work_dir=‘/tmp‘

purge=‘/usr/local/bin/purge_relay_logs‘

if [ ! -d $log_dir ]

then

mkdir $log_dir -p

$purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1

chmod +x purge_relay_log.sh

crontab -e

0 4 * * * /bin/bash /root/purge_relay_log.sh

pure_relay_logs 脚本参数如下所示：

--user mysql 用户名

--password mysql 密码

--port 端口号

--workdir 指定创建 relay log 的硬链接的位置，默认是/var/tmp，由于系统不同分区创建硬链接文件会失败，故需要执行硬链接具体位置，成功执行脚本后，硬链接的中继日志文件被删除 --disable_relay_log_purge 默认情况下，如果 relay_log_purge=1，脚本会什么都不清理，自动退出，通过设定这个参数，当 relay_log_purge=1 的情况下会将 relay_log_purge 设置为 0。清理 relay log 之后，最后将参数设置为 OFF。

手动清除中继日志

purge_relay_logs --user=root --password=123456 --disable_relay_log_purge --port=3306 --workdir=/tmp

2017-08-31 21:33:52: purge_relay_logs script started. Found relay_log.info: /usr/local/mysql/data/relay-log.info Removing hard linked relay log files slave-relay-bin* under /tmp.. done. Current relay log file: /usr/local/mysql/data/slave-relay-bin.000002 Archiving unused relay log files (up to /usr/local/mysql/data/slave-relay-bin.000001) ... Creating hard link for /usr/local/mysql/data/slave-relay-bin.000001 under /tmp/slave-relay-bin.000001 .. ok. Creating hard links for unused relay log files completed. Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL thread can delete older relay log files (if i t keeps up); SET GLOBAL relay_log_purge=0; .. ok. Removing hard linked relay log files slave-relay-bin* under /tmp.. done. 2017-08-31 21:33:56: All relay log purging operations succeeded.

检查 MHA ssh 通信状态

[root@server05 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf

Sat Dec 29 16:03:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sat Dec 29 16:03:57 2018 - [info] Reading application default configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:03:57 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:03:57 2018 - [info] Starting SSH connection tests..

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:03:59 2018 - [debug] ok.

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:57 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:03:59 2018 - [debug] ok.

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [info] All SSH connection tests passed successfully.

最后会返回 successfully 表示没有问题

检查整个集群的状态

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Sat Dec 29 16:04:53 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sat Dec 29 16:04:53 2018 - [info] Reading application default configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:04:53 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:04:53 2018 - [info] MHA::MasterMonitor version 0.56. Creating directory /var/log/masterha/app1.. done.

Sat Dec 29 16:04:55 2018 - [info] Dead Servers:

Sat Dec 29 16:04:55 2018 - [info] Alive Servers:

Sat Dec 29 16:04:55 2018 - [info] server01(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] server02(192.168.200.112:3306)

Sat Dec 29 16:04:55 2018 - [info] server03(192.168.200.113:3306)

Sat Dec 29 16:04:55 2018 - [info] server04(192.168.200.114:3306)

Sat Dec 29 16:04:55 2018 - [info] Alive Slaves:

Sat Dec 29 16:04:55 2018 - [info] server02(192.168.200.112:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:04:55 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] Primary candidate for the new Master (candidate_master is set)

Sat Dec 29 16:04:55 2018 - [info] server03(192.168.200.113:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:04:55 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] server04(192.168.200.114:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:04:55 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] Current Alive Master: server01(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] Checking slave configurations..

Sat Dec 29 16:04:55 2018 - [info] Checking replication filtering settings..

Sat Dec 29 16:04:55 2018 - [info] binlog_do_db= , binlog_ignore_db=

Sat Dec 29 16:04:55 2018 - [info] Replication filtering check ok.

Sat Dec 29 16:04:55 2018 - [info] Starting SSH connection tests..

Sat Dec 29 16:05:00 2018 - [info] All SSH connection tests passed successfully.

Sat Dec 29 16:05:00 2018 - [info] Checking MHA Node version..

Sat Dec 29 16:05:02 2018 - [info] Version check ok.

Sat Dec 29 16:05:02 2018 - [info] Checking SSH publickey authentication settings on the current master..

Sat Dec 29 16:05:03 2018 - [info] HealthCheck: SSH to server01 is reachable.

Sat Dec 29 16:05:04 2018 - [info] Master MHA Node version is 0.56.

Sat Dec 29 16:05:04 2018 - [info] Checking recovery script configurations on the current master..

Sat Dec 29 16:05:04 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test -- manager_version=0.56 --start_file=master-bin.000001

Sat Dec 29 16:05:04 2018 - [info] Connecting to root@server01(server01).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /var/lib/mysql, up to master-bin.000001

Sat Dec 29 16:05:04 2018 - [info] Master setting check done.

Sat Dec 29 16:05:04 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Sat Dec 29 16:05:04 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=server02 --slave_ip=192.168.200.112 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Sat Dec 29 16:05:04 2018 - [info] Connecting to root@192.168.200.112(server02:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.

Sat Dec 29 16:05:05 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=server03 --slave_ip=192.168.200.113 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Sat Dec 29 16:05:05 2018 - [info] Connecting to root@192.168.200.113(server03:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to slave-relay-bin.000002 Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.

Sat Dec 29 16:05:07 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=server04 --slave_ip=192.168.200.114 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Sat Dec 29 16:05:07 2018 - [info] Connecting to root@192.168.200.114(server04:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to slave-relay-bin.000002 Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.

Sat Dec 29 16:05:08 2018 - [info] Slaves settings check done.

Sat Dec 29 16:05:09 2018 - [info]

server01 (current master)

+--server02

+--server03

+--server04

Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server02.. Sat Dec 29 16:05:09 2018 - [info] ok.

Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server03.. Sat Dec 29 16:05:09 2018 - [info] ok.

Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server04.. Sat Dec 29 16:05:09 2018 - [info] ok.

Sat Dec 29 16:05:09 2018 - [info] Checking master_ip_failover_script status:

Sat Dec 29 16:05:09 2018 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.111 --orig_m aster_port=3306

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===

Checking the Status of the script.. OK

bash: /etc/init.d/keepalived: 没有那个文件或目录

Sat Dec 29 16:05:09 2018 - [info] OK.

Sat Dec 29 16:05:09 2018 - [warning] shutdown_script is not defined.

Sat Dec 29 16:05:09 2018 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

返回 OK 表示也没有问题

VIP 配置管理

Master vip 配置有两种方式,一种是通过 keepalived 或者 heartbeat 类似的软件的方式管理 VIP 的浮动,另一种为通过命令方式管理。

通过命令方式管理 VIP 地址：

打开在前面编辑过的文件/etc/masterha/app1.cnf，检查如下行是否正确，再检查集群状态。

[root@server05 ~]# grep -n ‘master_ip_failover_script‘ /etc/masterha/app1.cnf

9:master_ip_failover_script= /usr/local/bin/master_ip_failover

Primary Master(192.168.200.111)

[root@server01 ~]# ip a | grep ens32

2: ens32: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32 inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1

Server05(192.168.200.115)修改故障转移脚本

[root@server05 ~]# head -13 /usr/local/bin/master_ip_failover

#!/usr/bin/env perl

use strict;

use warnings FATAL => ‘all‘;

use Getopt::Long;

my (

$command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port,

);

my $vip = ‘192.168.200.100‘; # 写入 VIP

my $key = "1"; #非 keepalived 方式切换脚本使用的

my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip"; #若是使用 keepalived

my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令

/usr/local/bin/master_ip_failover 文件的内容意思是当主库发生故障时，会触发 MHA 切换，MHA manager 会停掉主库上的 ens32:1 接口，触发虚拟 ip 漂移到备选从库，从而完成切换。

Server05(192.168.200.115) 检查 manager 状态

masterha_check_status --conf=/etc/masterha/app1.cnf app1

is stopped(2:NOT_RUNNING).

注意：如果正常会显示"PING_OK"，否则会显示"NOT_RUNNING"，代表 MHA 监控没有开启。

Server05(192.168.200.115) 开启 manager 监控

[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dea d_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1 &

[1] 65837

启动参数介绍：

--remove_dead_master_conf 该参数代表当发生主从切换后，老的主库的 ip 将会从配置文件中移除。

--manger_log 日志存放位置

--ignore_last_failover 在缺省情况下，如果 MHA 检测到连续发生宕机，且两次宕机间隔不足 8 小时的话，则不会进行 Failover，之所以这样限制是为了避免 ping-pong 效应。该参数代表忽略上次 MHA 触发切换产生的文件，默认情况下，MHA 发生切换后会在日志目录，也就是上面我设置的/data 产生 app1.failover.complete 文件，下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件，为了方便，这里设置为--ignore_last_failover。

Server05(192.168.200.115)查看 Server05 监控是否正常：

[root@monitor ~]# masterha_check_status --conf=/etc/masterha/app1.cnf

app1 (pid:65837) is running(0:PING_OK), master:server01

可以看见已经在监控了

Server05(192.168.200.115)查看启动日志

[root@server05 ~]# cat /var/log/masterha/app1/manager.log

Sat Dec 29 16:09:50 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sat Dec 29 16:09:50 2018 - [info] Reading application default configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:09:50 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:09:50 2018 - [info] MHA::MasterMonitor version 0.56. Sat Dec 29 16:09:51 2018 - [info] Dead Servers:

Sat Dec 29 16:09:51 2018 - [info] Alive Servers:

Sat Dec 29 16:09:51 2018 - [info] server01(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] server02(192.168.200.112:3306)

Sat Dec 29 16:09:51 2018 - [info] server03(192.168.200.113:3306)

Sat Dec 29 16:09:51 2018 - [info] server04(192.168.200.114:3306)

Sat Dec 29 16:09:51 2018 - [info] Alive Slaves:

Sat Dec 29 16:09:51 2018 - [info] server02(192.168.200.112:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:09:51 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] Primary candidate for the new Master (candidate_master is set)

Sat Dec 29 16:09:51 2018 - [info] server03(192.168.200.113:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:09:51 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] server04(192.168.200.114:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:09:51 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] Current Alive Master: server01(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] Checking slave configurations..

Sat Dec 29 16:09:51 2018 - [info] Checking replication filtering settings..

Sat Dec 29 16:09:51 2018 - [info] binlog_do_db= , binlog_ignore_db=

Sat Dec 29 16:09:51 2018 - [info] Replication filtering check ok.

Sat Dec 29 16:09:51 2018 - [info] Starting SSH connection tests..

Sat Dec 29 16:09:58 2018 - [info] All SSH connection tests passed successfully.

Sat Dec 29 16:09:58 2018 - [info] Checking MHA Node version..

Sat Dec 29 16:10:01 2018 - [info] Version check ok.

Sat Dec 29 16:10:01 2018 - [info] Checking SSH publickey authentication settings on the current master..

Sat Dec 29 16:10:02 2018 - [info] HealthCheck: SSH to server01 is reachable.

Sat Dec 29 16:10:03 2018 - [info] Master MHA Node version is 0.56.

Sat Dec 29 16:10:03 2018 - [info] Checking recovery script configurations on the current master..

Sat Dec 29 16:10:03 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test -- manager_version=0.56 --start_file=master-bin.000001

Sat Dec 29 16:10:03 2018 - [info] Connecting to root@server01(server01).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /var/lib/mysql, up to master-bin.000001

Sat Dec 29 16:10:03 2018 - [info] Master setting check done.

Sat Dec 29 16:10:03 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Sat Dec 29 16:10:03 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=server02 --slave_ip=192.168.200.112 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Sat Dec 29 16:10:03 2018 - [info] Connecting to root@192.168.200.112(server02:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.

Sat Dec 29 16:10:04 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=server03 --slave_ip=192.168.200.113 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Sat Dec 29 16:10:04 2018 - [info] Connecting to root@192.168.200.113(server03:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to slave-relay-bin.000002 Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.

Sat Dec 29 16:10:05 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=server04 --slave_ip=192.168.200.114 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Sat Dec 29 16:10:05 2018 - [info] Connecting to root@192.168.200.114(server04:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to slave-relay-bin.000002 Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.

Sat Dec 29 16:10:06 2018 - [info] Slaves settings check done.

Sat Dec 29 16:10:06 2018 - [info]

server01 (current master)

+--server02

+--server03

+--server04

Sat Dec 29 16:10:06 2018 - [info] Checking master_ip_failover_script status:

Sat Dec 29 16:10:06 2018 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.111 --orig_m aster_port=3306

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK

bash: /etc/init.d/keepalived: 没有那个文件或目录

Sat Dec 29 16:10:07 2018 - [info] OK.

Sat Dec 29 16:10:07 2018 - [warning] shutdown_script is not defined.

Sat Dec 29 16:10:07 2018 - [info] Set master ping interval 1 seconds.

Sat Dec 29 16:10:07 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.

Sat Dec 29 16:10:07 2018 - [info] Starting ping health check on server01(192.168.200.111:3306)..

Thu Aug 31 21:55:23 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn‘t respond..

注意：其中"Ping(SELECT) succeeded, waiting until MySQL doesn‘t respond.."说明整个系统已经开始监控了。

关闭 MHA manager 监控，忽略操作

masterha_stop --conf=/etc/masterha/app1.cnf

发现已经将 VIP：192.168.200.100 绑定在网卡 ens32。

[root@server01 ~]# ip a | grep ens32

Primary Master(192.168.200.111) 模拟主库故障

[root@server01 ~]# systemctl stop mariadb

[root@server01 ~]# netstat -lnpt | grep :3306

[root@server01 ~]# ip a | grep ens32

2: ens32: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32

slave1(192.168.200.113)状态：

MariaDB [(none)]> show slave status\G

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.112

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000001

Read_Master_Log_Pos: 1372

Relay_Log_File: slave-relay-bin.000002

Relay_Log_Pos: 530

Relay_Master_Log_File: master-bin.000001

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

slave2(192.168.200.114)状态：

MariaDB [(none)]> show slave status\G

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.112

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000001

Read_Master_Log_Pos: 1372

Relay_Log_File: slave-relay-bin.000002

Relay_Log_Pos: 530

Relay_Master_Log_File: master-bin.000001

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Server05(192.168.200.115) 监控已经自动关闭：

[root@server05 ~]# (回车)

[1]+ 完成 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1

Server05(192.168.200.115) 查看监控配置文件已经发生了变化（server01 的配置已被删除）：

[root@server05 ~]# cat /etc/masterha/app1.cnf

[server default]

manager_log=/var/log/masterha/app1/manager.log

manager_workdir=/var/log/masterha/app1

master_binlog_dir=/var/lib/mysql

master_ip_failover_script=/usr/local/bin/master_ip_failover

password=123456 ping_interval=1

remote_workdir=/tmp

repl_password=123456

repl_user=repl

user=root

[server2]

candidate_master=1

check_repl_delay=0

hostname=server02

port=3306

[server3]

hostname=server03

port=3306

[server4]

hostname=server04

port=3306

Server05(192.168.200.115) 故障切换过程中的日志文件内容如下：

[root@server05 ~]# tail -f /var/log/masterha/app1/manager.log

Selected server02 as a new master.

server02: OK: Applying all logs succeeded.

server02: OK: Activated master IP address.

server04: This host has the latest relay log events.

server03: This host has the latest relay log events.

Generating relay diff files from the latest slave succeeded.

server04: OK: Applying all logs succeeded. Slave started, replicating from server02.

server03: OK: Applying all logs succeeded. Slave started, replicating from server02.

server02: Resetting slave info succeeded.

Master failover to server02(192.168.200.112:3306) completed successfully.

故障主库修复及 VIP 切回测试

Primary Master(192.168.200.111)：

[root@server01 ~]# systemctl start mariadb

[root@server01 ~]# netstat -lnpt | grep :3306

tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 6131/mysqld

Primary Master(192.168.200.111) 指向新的主库

[root@server01 ~]# mysql -u root -p123456

stop slave;

CHANGE MASTER TO

MASTER_HOST=‘192.168.200.112‘,

MASTER_USER=‘repl‘,

MASTER_PASSWORD=‘123456‘;

start slave;

show slave status\G

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.112

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000001

Read_Master_Log_Pos: 1372

Relay_Log_File: mariadb-relay-bin.000002

Relay_Log_Pos: 1208

Relay_Master_Log_File: master-bin.000001

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Server05(192.168.200.115) 修改监控配置文件添加 server1 配置：

[root@server05 ~]# vim /etc/masterha/app1.cnf

[server01]

hostname=server01

port=3306

Server05(192.168.200.115) 检查集群状态：

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

-----------------------------------忽略部分信息-----------------------------------

Thu Aug 31 22:20:30 2017 - [info] Alive Servers:

Thu Aug 31 22:20:30 2017 - [info] server01(192.168.200.111:3306)

Thu Aug 31 22:20:30 2017 - [info] server02(192.168.200.112:3306)

Thu Aug 31 22:20:30 2017 - [info] server03(192.168.200.113:3306)

Thu Aug 31 22:20:30 2017 - [info] server04(192.168.200.114:3306)

-----------------------------------忽略部分信息-----------------------------------

server02 (current master)

+--server01

+--server03

+--server04

-----------------------------------忽略部分信息-----------------------------------

MySQL Replication Health is OK.

Server05(192.168.200.115) 开启监控

[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2 >&1 &

[1] 68199

基于 MHA 的MySQL高可用-CentOS7（实例）

标签：global 功能 rtb between arch routes sam yum utils

原文地址：https://www.cnblogs.com/2567xl/p/11720158.html

踩

(0)

评论一句话评论（0）

分享档案

更多>

2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)

周排行