标签:second 5.4 Owner 1.7 accept oss 协议 chain 理解
第1章 回顾是一款开源的 多功能 快速的全量和增量备份软件
a 等价于本地复制命令(cp)
b 等价于远程复制命令(scp)
c 等价于数据删除命令(rm)
d 等价于数据查看命令(ls)
1. 可以实现排除同步数据
2. 可以实现加密传输数据(借助ssh协议隧道)
3. 可以实现数据传输认证机制
4. 可以实现socket守护进程方式,传输数据(重点)
5. 可以保持数据传输属性信息不变(-p)
6. 可以传输普通文件,也可以传输特殊文件(链接文件 或者 设备文件)
7. 可以数据增量传输
基于一种特殊的算法“quick check”,实现增量复制传输数据
1. 先实现两台主机之间网络互通
2. 对要传输的数据信息,进行属性信息比对(比如 文件大小 时间信息 属主 属组 权限属性等)
3. 只传输变化的数据信息
1. 实现本地数据传输备份
2. 实现远程数据传输备份
3. 实现守护进程方式数据传输备份(重点)
第一步:检查软件是否安装
第二步:编写rsync软件配置文件
第三步:创建备份目录的管理用户
第四步:创建认证时所需要密码文件(对密码文件权限进行修改为600权限)
第五步:创建数据备份目录,修改备份目录属主和属组权限为管理用户(rsync)
第六步:启动rsync服务(rsync --daemon)
进行数据传输测试
说明:如果想实现免交互方式传输数据,客户端要完成以下操作步骤
第一步:创建密码文件,并进行修改权限为600
echo "wuhuang123" >>/etc/rsync.password
chmod 600 /etc/rsync.password
[root@nfs01 ~]# echo "wuhuang123" >>/etc/rsync.password [root@nfs01 ~]# chmod 600 /etc/rsync.password [root@nfs01 ~]# ll /etc/rsync.password -rw------- 1 root root 10 Jan 23 14:45 /etc/rsync.password |
第二步:进行免交互传输数据测试
rsync -avz /etc/hosts rsync_backup@backup::backup --password-file=/etc/rsync.password
[root@nfs01 ~]# rsync -avz /etc/hosts rsync_backup@backup::backup --password-file=/etc/rsync.password sending incremental file list hosts
sent 201 bytes received 27 bytes 456.00 bytes/sec total size is 372 speedup is 1.63 [root@backup ~]# ll /backup/ total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts |
1.7 rsync守护进程方式传输数据原理过程
在备份服务器上部署rsync软件服务,采用推的方式(push:即从本地机器拷贝文件到远程rsync服务器中)进行数据备份传输。
采用定时备份:备份的数据信息大部分是内部人员产生的数据
采用实时备份:备份的数据信息大部分是外部人员产生的数据
命令参数 | 参数 |
-v, --verbose | 详细模式输出,传输时的信息。 |
-z,--compress | 传输时进行压缩提供传输效率,--compress-level=NUM 可按级别压缩,局域网可以不用压缩 |
-a, --archive 重要 | 归档模式,表示以递归方式传输文件,并保持所有文件属性,等于-rlptgoD。 |
-r, --recursive 归类于-a参数 | 对子目录以递归模式处理,即目录下的所有目录都同样传输,注意是小写 |
-t, --times 归类于-a参数 | 保持文件时间信息。 |
-o, --owner 归类于-a参数 | 保持文件属主信息。 |
-p, --perms 归类于-a参数 | 保持文件权限。 |
-g, --group 归类于-a参数 | 保持文件属组信息。 |
-P --progress | 等同于 --partial,显示备份过程。 |
-D, --devices 归类于-a参数 | 保持设备文件信息。 |
-l, --links 归类于-a参数 | 保留软链结。(小写字母) |
-e, --rsh=command | 指定使用rsh、ssh方式进行数据同步 |
--exclude=PATTETN | 用来指定排除那些不希望传输的文件(和tar参数一样) |
--exclude-from=FILE | 文件名所在的目录文件,即可以实现排除多个文件(和tar参数一样) |
--bwlimit=RATE | limit I/O bandwidth: KBytes per second limit socket I/O bandwidth限速功能 案例:某DBA做数据同步,带宽占满,导致用户无法访问网站 |
--delete | 让目标目录SRC和源目录数据DST一致,即无差异同步数据 |
保持同步目录及文件属性: 这里的-avzP 相当于 -vzrtopgDIP(还多了DI功能),生产环境常用的参数选项为 -avzP 或 -vzrtopgP 如果是放入脚本中,也可以把 -v 和 -P去掉。这里的 --progress 可以用 -P代替。 |
例:-e, --rsh=command 指定使用rsh、ssh方式进行数据同步
[root@nfs01 ~]# rsync -avz -e "ssh -p22" /wuhuang/a 172.16.1.41:/backup The authenticity of host '172.16.1.41 (172.16.1.41)' can't be established. RSA key fingerprint is 57:3f:64:68:95:4d:99:54:01:33:ab:47:a0:72:da:bf. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.16.1.41' (RSA) to the list of known hosts. root@172.16.1.41's password: sending incremental file list a/ a/1 a/2 a/3
sent 171 bytes received 73 bytes 28.71 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# ll /backup/ total 8 drwxr-xr-x 2 root root 4096 Jan 23 17:54 a |
第4章 rsync服务常见错误
【客户端的错误】 No route to host 【错误演示过程】 [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup rsync: failed to connect to 172.16.1.41: No route to host (113) rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6] 【异常问题解决】 关闭rsync服务端的防火墙服务(iptables) 也可做防火墙设置,此处不多讲 [root@backup mnt]# /etc/init.d/iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ] [root@backup mnt]# /etc/init.d/iptables status iptables: Firewall is not running. |
客户端的错误现象: [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::/backup ERROR: The remote path must start with a module name not a / rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: rsync命令语法理解错误,::/backup是错误的语法,应该为::backup(rsync模块) |
客户端的错误现象: [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: auth failed on module backup rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: 1. 密码真的输入错误,用户名真的错误 2. secrets file = /etc/rsync.password指定的密码文件和实际密码文件名称不一致 3. /etc/rsync.password文件权限不是600 4. rsync_backup:123456密码配置文件后面注意不要有空格 5. rsync客户端密码文件中只输入密码信息即可,不要输入虚拟认证用户名称 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup @ERROR: Unknown module 'backup' rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: 1. /etc/rsyncd.conf配置文件模块名称书写错误 2. 模块对应的目录创建有误 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: sending incremental file list hosts rsync: mkstemp ".hosts.5z3AOA" (in backup) failed: Permission denied (13)
sent 196 bytes received 27 bytes 63.71 bytes/sec total size is 349 speedup is 1.57 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6] 异常问题解决: 1. 共享目录的属主和属组不正确,不是rsync 2. 共享目录的权限不正确,不是755 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: chdir failed rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: 1. 备份存储目录没有建立 2. 建立的备份存储目录和配置文件定义不一致 说明:如果没有备份存储目录 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: invalid uid rsync rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: rsync服务对应rsync虚拟用户不存在了 |
password file must not be other-accessible
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password password file must not be other-accessible continuing without password file Password: sending incremental file list
sent 26 bytes received 8 bytes 5.23 bytes/sec total size is 349 speedup is 10.26 异常问题解决: rsync客户端的秘钥文件也必须是600权限 |
4.9 rsync客户端连接慢问题
错误日志输出 2017/03/08 20:14:43 [3422] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors 2017/03/08 20:14:43 [3422] name lookup failed for 172.16.1.31: Name or service not known 2017/03/08 20:14:43 [3422] connect from UNKNOWN (172.16.1.31) 2017/03/08 20:14:43 [3422] rsync to backup/ from rsync_backup@unknown (172.16.1.31) 2017/03/08 20:14:43 [3422] receiving file list 2017/03/08 20:14:43 [3422] sent 76 bytes received 83 bytes total size 349 正确日志输出 2017/03/08 20:16:45 [3443] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors 2017/03/08 20:16:45 [3443] connect from nfs02 (172.16.1.31) 2017/03/08 20:16:45 [3443] rsync to backup/ from rsync_backup@nfs02 (172.16.1.31) 2017/03/08 20:16:45 [3443] receiving file list 2017/03/08 20:16:45 [3443] sent 76 bytes received 83 bytes total size 349 客户端连接慢原因:主机名未被解析 异常问题解决: 查看日志进行分析,配置/etc/hosts文件,写入对应主机名与IP的映射关系 |
4.10 rsync服务没有正确启动
Connection refused (111)
[root@wuhuang-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup rsync: failed to connect to 172.16.1.41: Connection refused (111) rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6] 解决 rsync服务没开启 [root@wuhuang-muban ~]# rsync --daemon [root@wuhuang-muban ~]# ss -lntup |grep rsync tcp LISTEN 0 5 :::873 :::* users:(("rsync",1434,5)) tcp LISTEN 0 5 *:873 *:* users:(("rsync",1434,4)) [root@wuhuang-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: sending incremental file list hosts
sent 196 bytes received 27 bytes 49.56 bytes/sec total size is 349 speedup is 1.57 |
第5章 rsync扩展应用说明
1. 让rsync守护进程开机自启动
echo "rsync --daemon" >>/etc/rc.local
利用xinetd服务启动rsync守护进程服务流程
rpm -qa|grep xinetd
yum install -y xinetd
[root@backup ~]# rpm -qa|grep xinetd [root@backup ~]# yum install xinetd -y |
5.1.2 第二个里程碑:配置xinetd文件,让rsync服务允许被超级守护进程管理
vim /etc/xinetd.d/rsync
disable = yes ===> disable = no
[root@backup ~]# vim /etc/xinetd.d/rsync # default: off # description: The rsync server is a good addition to an ftp server, as it \ # allows crc checksumming etc. service rsync { disable =no flags = IPv6 socket_type = stream wait = no user = root server = /usr/bin/rsync server_args = --daemon log_on_failure += USERID } |
5.1.3 第三个里程碑:确保rsync守护进程服务关闭了
[root@backup ~]# netstat -lntup|grep rsync tcp 0 0 0.0.0.0:873 0.0.0.0:* LISTEN 1294/rsync tcp 0 0 :::873 :::* LISTEN 1294/rsync [root@backup ~]# killall rsync [root@backup ~]# killall rsync rsync: no process killed |
5.1.4 第四个里程碑:启动xinetd服务
[root@backup ~]# /etc/init.d/xinetd start Starting xinetd: [ OK ] [root@backup ~]# netstat -lntup|grep 873 tcp 0 0 :::873 :::* LISTEN 1353/xinetd |
5.2 守护进程多模块功能配置
[root@backup ~]# cat /etc/rsyncd.conf #rsync_config #created by HQ at 2017 ##rsyncd.conf start##
uid = rsync gid = rsync use chroot = no max connections = 200 timeout = 300 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore errors read only = false list = false hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32 auth users = rsync_backup secrets file = /etc/rsync.password [backup] comment = "backup dir by wuhuang" path = /backup [devbackup] comment = "backup dir by wuhuang" path = /devbackup [sabackup] comment = "backup dir by wuhuang" path = /sabackup |
[root@backup ~]# mkdir /{backup,devbackup,sabackup}/ -p [root@backup ~]# chown -R rsync.rsync /{backup,devbackup,sabackup}/ [root@backup ~]# ll /backup/ -d drwxr-xr-x 3 rsync rsync 4096 Jan 23 21:08 /backup/ [root@backup ~]# ll /devbackup/ -d drwxr-xr-x 2 rsync rsync 4096 Jan 23 23:17 /devbackup/ [root@backup ~]# ll /sabackup/ -d drwxr-xr-x 2 rsync rsync 4096 Jan 23 23:17 /sabackup/ |
[root@backup ~]# /etc/init.d/xinetd restart Stopping xinetd: [ OK ] Starting xinetd: [ OK ] |
1. 如果是由rsync --daemon启动的,重启时使用 killall rsync && rsync --daemon
2. 如果是由 /etc/init.d/xinetd start启动的,重启时使用 /etc/init.d/xinetd restart
[root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::devbackup --password-file=/etc/rsync.passwordsending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::sabackup --password-file=/etc/rsync.password sending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@backup ~]# ll /backup/ /devbackup/ /sabackup/ /backup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts /devbackup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts /sabackup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts |
创建一个测试环境:nfs服务器(客户端)上操作
[root@nfs01 ~]# tree /wuhuang/ /wuhuang/ ├── a │ ├── 1 │ ├── 2 │ └── 3 ├── b │ ├── 1 │ ├── 2 │ └── 3 ├── c │ ├── 1 │ ├── 2 │ └── 3 └── d ├── 1 ├── 2 └── 3 4 directories, 12 files
|
实现数据同步传输排除方法
1. --exclude 参数指定排除的文件或目录信息
2. --exclude-from 参数指定排除的多个数据信息文件
--exclude实现排除需求:不要a和b,只要c目录所有数据,d目录中只要1和3文件
[root@nfs01 ~]# rsync -avz /wuhuang/ --exclude=a --exclude=b --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ c/ c/1 c/2 c/3 d/ d/1 d/3
sent 292 bytes received 114 bytes 270.67 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# ll -R /backup/ /backup/: total 8 drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 c drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 d
/backup/c: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 2 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3
/backup/d: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 |
以上命令精简化:
rsync -avz /wuhuang/ --exclude={a..b} --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password 或 rsync -avz /wuhuang/ --exclude={a,b} --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password 5.3.2 --exclude-from=FILE |
--exclude-from实现排除需求:不要a和b 只要c目录所有数据 d目录中只要1和3文件
[root@nfs01 wuhuang]# pwd /wuhuang [root@nfs01 wuhuang]# vim exclude.txt [root@nfs01 wuhuang]# cat exclude.txt a b d/2 [root@nfs01 wuhuang]# rsync -avz /wuhuang/ --exclude-from=/wuhuang/exclude.txt rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ exclude.txt c/ c/1 c/2 c/3 d/ d/1 d/3
sent 368 bytes received 133 bytes 1002.00 bytes/sec total size is 8 speedup is 0.02 [root@backup ~]# ll -R /backup/ /backup/: total 12 drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 c drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 d -rw-r--r-- 1 rsync rsync 8 Jan 23 23:54 exclude.txt
/backup/c: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 2 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 /backup/d: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 |
说明:如果在排除同步过程中,出现了指定的exclude.txt文件,两种方法解决:
01. 将exclude.txt文件从同步数据目录中移出
02. 在exclude.txt文件中写入排除自身文件信息
[root@backup ~]# ll /backup/ total 0 [root@nfs01 ~]# rsync -av /wuhuang/a rsync_backup@172.16.1.41::backup/01/ --password-file=/etc/rsync.password sending incremental file list created directory 01 a/ a/1 a/2 a/3
sent 176 bytes received 69 bytes 163.33 bytes/sec total size is 0 speedup is 0.00 [root@nfs01 ~]# rsync -avz /wuhuang/b rsync_backup@172.16.1.41::backup/02/ --password-file=/etc/rsync.password sending incremental file list created directory 02 b/ b/1 b/2 b/3
sent 167 bytes received 69 bytes 472.00 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# tree /backup/ /backup/ ├── 01 │ └── a │ ├── 1 │ ├── 2 │ └── 3 └── 02 └── b ├── 1 ├── 2 └── 3
4 directories, 6 files
|
[root@nfs01 ~]# rsync -avz /wuhuang/c rsync_backup@172.16.1.41::backup/03/w/h --password-file=/etc/rsync.password sending incremental file list rsync: mkdir "03/w/h" (in backup) failed: No such file or directory (2) rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6] rsync: connection unexpectedly closed (5 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6] |
[root@backup ~]# cat /etc/rsyncd.conf #rsync_config #created by HQ at 2017 ##rsyncd.conf start##
uid = rsync gid = rsync use chroot = no max connections = 200 timeout = 300 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore errors read only = false list = false hosts allow = 172.16.1.0/24 hosts deny = 0.0.0.0/32 auth users = rsync_backup secrets file = /etc/rsync.password [backup] comment = "backup dir by wuhuang" path = /backup [devbackup] comment = "backup dir by wuhuang" path = /devbackup [sabackup] comment = "backup dir by wuhuang" path = /sabackup |
第三种情况测试
[root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@10.0.0.41::backup --password-file=/etc/rsync.password
sending incremental file list
hosts
sent 205 bytes received 27 bytes 154.67 bytes/sec
total size is 372 speedup is 1.60 10.0.0.41网段可以实现传输
无差异同步通俗的讲:
1. 我有的数据,你也有;我没有的数据,你也不能有(删除)
2. 存储与备份服务数据完全一致(一模一样)
我有的数据,你也有:实践过程
rsync客户端操作命令:
[root@nfs01 ~]# rsync -avz --delete /wuhuang/ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password
sending incremental file list
./
exclude.txt
a/
a/1
a/2
a/3
b/
b/1
b/2
b/3
c/
c/1
c/2
c/3
d/
d/1
d/2
d/3
sent 705 bytes received 274 bytes 1958.00 bytes/sec
total size is 8 speedup is 0.01
rsync服务端检查:
[root@backup ~]# tree /backup/
/backup/
├── a
│ ├── 1
│ ├── 2
│ └── 3
├── b
│ ├── 1
│ ├── 2
│ └── 3
├── c
│ ├── 1
│ ├── 2
│ └── 3
├── d
│ ├── 1
│ ├── 2
│ └── 3
└── exclude.txt
4 directories, 13 files
我没有的数据,你也不能有(删除):实践过程
[root@nfs01 ~]# cd /wuhuang/
[root@nfs01 wuhuang]# rm -rf a
[root@nfs01 wuhuang]# ll
total 16
drwxr-xr-x 2 root root 4096 Jan 23 17:54 b
drwxr-xr-x 2 root root 4096 Jan 23 17:54 c
drwxr-xr-x 2 root root 4096 Jan 23 17:54 d
-rw-r--r-- 1 root root 8 Jan 23 23:54 exclude.txt
[root@nfs01 wuhuang]# rsync -avz --delete /wuhuang/ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password
sending incremental file list
./
deleting a/3
deleting a/2
deleting a/1
deleting a/
sent 177 bytes received 14 bytes 382.00 bytes/sec
total size is 8 speedup is 0.04
rsync服务端检查:
[root@backup ~]# tree /backup/
/backup/
├── b
│ ├── 1
│ ├── 2
│ └── 3
├── c
│ ├── 1
│ ├── 2
│ └── 3
├── d
│ ├── 1
│ ├── 2
│ └── 3
└── exclude.txt
3 directories, 10 files
标签:second 5.4 Owner 1.7 accept oss 协议 chain 理解
原文地址:http://blog.51cto.com/12805107/2064544