标签：【ceph】storage cluster quick start --> create a cluster
【ceph】Storage Cluster Quick Start --> Create a Cluster
配置mirro
root@admin-node:/home/my-cluster# cat /etc/apt/sources.list deb http://mirrors.163.com/debian/ wheezy main non-free contrib deb http://mirrors.163.com/debian/ wheezy-updates main non-free contrib deb http://mirrors.163.com/debian/ wheezy-backports main non-free contrib deb-src http://mirrors.163.com/debian/ wheezy main non-free contrib deb-src http://mirrors.163.com/debian/ wheezy-updates main non-free contrib deb-src http://mirrors.163.com/debian/ wheezy-backports main non-free contrib deb http://mirrors.163.com/debian-security/ wheezy/updates main non-free contrib deb-src http://mirrors.163.com/debian-security/ wheezy/updates main non-free contrib deb http://mirrors.ustc.edu.cn/ceph/debian/ wheezy main
配置hosts文件，ssh无密登录
root@admin-node:/home/my-cluster# cat /etc/hosts 172.16.66.132 admin-node 172.16.66.133 node1 172.16.66.134 node2 172.16.66.135 node3 root@admin-node:/home/my-cluster# ssh-keygen -t rsa -P ‘‘ root@admin-node:/home# ssh-copy-id admin-node root@admin-node:/home# ssh-copy-id 172.16.66.133 root@admin-node:/home# ssh-copy-id 172.16.66.134 root@admin-node:/home# ssh-copy-id 172.16.66.135 [无密登录测试] root@admin-node:/home# ssh admin-node root@admin-node:/home# ssh node1 root@admin-node:/home# ssh node2 root@admin-node:/home# ssh node3
========================================================
如果在任何时候你遇到了麻烦，你想重新开始，执行以下来清除配置：
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys
如果你想把已经安装的ceph程序也清除掉
ceph-deploy purge {ceph-node} [{ceph-node}]
========================================================
创建一个集群
root@admin-node:/home# mkdir my-cluster root@admin-node:/home# cd /home/my-cluster root@ceph:/home/my-cluster$ ceph-deploy new mon [ceph_deploy.conf][DEBUG ] found configuration file at: /home/osd/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.28): /usr/bin/ceph-deploy new mon [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] func : <function new at 0x28eab90> [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x294f098> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] ssh_copykey : True [ceph_deploy.cli][INFO ] mon : [‘mon‘] [ceph_deploy.cli][INFO ] public_network : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] cluster_network : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] fsid : None [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [mon][DEBUG ] connected to host: ceph [mon][INFO ] Running command: ssh -CT -o BatchMode=yes mon [mon][DEBUG ] connection detected need for sudo sudo: unable to resolve host ceph-osd1 [mon][DEBUG ] connected to host: mon [mon][DEBUG ] detect platform information from remote host [mon][DEBUG ] detect machine type [mon][DEBUG ] find the location of an executable [mon][INFO ] Running command: sudo /bin/ip link show [mon][INFO ] Running command: sudo /bin/ip addr show [mon][DEBUG ] IP addresses found: [‘172.16.66.133‘] [ceph_deploy.new][DEBUG ] Resolving host mon [ceph_deploy.new][DEBUG ] Monitor mon at 172.16.66.133 [ceph_deploy.new][DEBUG ] Monitor initial members are [‘mon‘] [ceph_deploy.new][DEBUG ] Monitor addrs are [‘172.16.66.133‘] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf... root@ceph:/home/my-cluster$ echo $? 0
有2个osd节点，写到配置文件里面
root@admin-node:/home/my-cluster# vim ceph.conf osd pool default size = 2
admin节点：
为所有节点安装部署ceph,
从国外获取安装包，如果你的网络质量不好，易报错。
root@admin-node:/home/my-cluster# ceph-deploy install admin-node node1 node2 node3 [localhost][DEBUG ] Reading package lists... [localhost][DEBUG ] Building dependency tree... [localhost][DEBUG ] Reading state information... [localhost][DEBUG ] ca-certificates is already the newest version. [localhost][DEBUG ] The following packages were automatically installed and are no longer required: [localhost][DEBUG ] cryptsetup-bin gdisk libaio1 libboost-program-options1.49.0 [localhost][DEBUG ] libboost-system1.49.0 libboost-thread1.49.0 libcephfs1 libcryptsetup4 [localhost][DEBUG ] libcurl3-gnutls libfcgi0ldbl libgoogle-perftools4 libicu48 libjs-jquery [localhost][DEBUG ] libleveldb1 libnspr4 libnss3 librados2 librbd1 libreadline5 librtmp0 [localhost][DEBUG ] libsnappy1 libssh2-1 libtcmalloc-minimal4 libunwind7 python-cephfs [localhost][DEBUG ] python-flask python-jinja2 python-markupsafe python-rados python-rbd [localhost][DEBUG ] python-requests python-six python-werkzeug sdparm xfsprogs [localhost][DEBUG ] Use ‘apt-get autoremove‘ to remove them. [localhost][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. [localhost][INFO ] Running command: wget -O release.asc http://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/release.asc [localhost][WARNIN] --2015-10-21 14:42:23-- http://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/release.asc [localhost][WARNIN] Resolving git.ceph.com (git.ceph.com)... 67.205.20.229 [localhost][WARNIN] Connecting to git.ceph.com (git.ceph.com)|67.205.20.229|:80... connected. [localhost][WARNIN] HTTP request sent, awaiting response... 200 OK [localhost][WARNIN] Length: 1645 (1.6K) [text/plain] [localhost][WARNIN] Saving to: `release.asc‘ [localhost][WARNIN] [localhost][WARNIN] 0K . 100% 147M=0s [localhost][WARNIN] [localhost][WARNIN] 2015-10-21 14:43:07 (147 MB/s) - `release.asc‘ saved [1645/1645] [localhost][WARNIN] [localhost][INFO ] Running command: apt-key add release.asc [localhost][DEBUG ] OK [localhost][DEBUG ] add deb repo to /etc/apt/sources.list.d/ [localhost][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update [localhost][DEBUG ] Hit http://mirrors.163.com wheezy Release.gpg [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates Release.gpg [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports Release.gpg [localhost][DEBUG ] Get:1 http://mirrors.163.com wheezy/updates Release.gpg [1,554 B] [localhost][DEBUG ] Hit http://mirrors.163.com wheezy Release [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates Release [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports Release [localhost][DEBUG ] Get:2 http://mirrors.163.com wheezy/updates Release [102 kB] [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/main Sources [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/non-free Sources [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/contrib Sources [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/main amd64 Packages [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/non-free amd64 Packages [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/contrib amd64 Packages [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/contrib Translation-en [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/main Translation-en [localhost][DEBUG ] Hit http://mirrors.163.com wheezy/non-free Translation-en [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main Sources [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free Sources [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib Sources [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main amd64 Packages/DiffIndex [localhost][DEBUG ] Hit http://mirrors.ustc.edu.cn wheezy Release.gpg [localhost][DEBUG ] Hit http://mirrors.ustc.edu.cn wheezy Release [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free amd64 Packages [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib amd64 Packages [localhost][DEBUG ] Hit http://mirrors.ustc.edu.cn wheezy/main amd64 Packages [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib Translation-en [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main Translation-en [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free Translation-en [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main Sources/DiffIndex [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free Sources [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib Sources [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main amd64 Packages/DiffIndex [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free amd64 Packages [localhost][DEBUG ] Ign http://mirrors.ustc.edu.cn wheezy/main Translation-en_HK [localhost][DEBUG ] Ign http://mirrors.ustc.edu.cn wheezy/main Translation-en [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib amd64 Packages [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib Translation-en [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main Translation-en/DiffIndex [localhost][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free Translation-en [localhost][DEBUG ] Get:3 http://mirrors.163.com wheezy/updates/main Sources [195 kB] [localhost][DEBUG ] Get:4 http://mirrors.163.com wheezy/updates/non-free Sources [14 B] [localhost][DEBUG ] Get:5 http://mirrors.163.com wheezy/updates/contrib Sources [14 B] [localhost][DEBUG ] Get:6 http://mirrors.163.com wheezy/updates/main amd64 Packages [320 kB] [localhost][DEBUG ] Get:7 http://mirrors.163.com wheezy/updates/non-free amd64 Packages [14 B] [localhost][DEBUG ] Get:8 http://mirrors.163.com wheezy/updates/contrib amd64 Packages [14 B] [localhost][DEBUG ] Get:9 http://mirrors.163.com wheezy/updates/contrib Translation-en [523 B] [localhost][DEBUG ] Get:10 http://mirrors.163.com wheezy/updates/main Translation-en [183 kB] [localhost][DEBUG ] Get:11 http://mirrors.163.com wheezy/updates/non-free Translation-en [14 B] [localhost][DEBUG ] Hit http://ceph.com wheezy Release.gpg [localhost][DEBUG ] Hit http://ceph.com wheezy Release [localhost][DEBUG ] Hit http://ceph.com wheezy/main amd64 Packages [localhost][DEBUG ] Ign http://ceph.com wheezy/main Translation-en_HK [localhost][DEBUG ] Ign http://ceph.com wheezy/main Translation-en [localhost][DEBUG ] Fetched 802 kB in 27s (29.7 kB/s) [localhost][DEBUG ] Reading package lists... [localhost][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-mds radosgw [localhost][DEBUG ] Reading package lists... [localhost][DEBUG ] Building dependency tree... [localhost][DEBUG ] Reading state information... [localhost][DEBUG ] The following extra packages will be installed: [localhost][DEBUG ] ceph-common [localhost][DEBUG ] Recommended packages: [localhost][DEBUG ] btrfs-tools libradosstriper1 ceph-fs-common ceph-fuse [localhost][DEBUG ] The following NEW packages will be installed: [localhost][DEBUG ] ceph ceph-common ceph-mds radosgw [localhost][DEBUG ] 0 upgraded, 4 newly installed, 0 to remove and 1 not upgraded. [localhost][DEBUG ] Need to get 0 B/45.7 MB of archives. [localhost][DEBUG ] After this operation, 142 MB of additional disk space will be used. [localhost][DEBUG ] Selecting previously unselected package ceph-common. [localhost][DEBUG ] (Reading database ... 28995 files and directories currently installed.) [localhost][DEBUG ] Unpacking ceph-common (from .../ceph-common_0.94.4-1~bpo70+1_amd64.deb) ... [localhost][DEBUG ] Selecting previously unselected package ceph. [localhost][DEBUG ] Unpacking ceph (from .../ceph_0.94.4-1~bpo70+1_amd64.deb) ... [localhost][DEBUG ] Selecting previously unselected package ceph-mds. [localhost][DEBUG ] Unpacking ceph-mds (from .../ceph-mds_0.94.4-1~bpo70+1_amd64.deb) ... [localhost][DEBUG ] Selecting previously unselected package radosgw. [localhost][DEBUG ] Unpacking radosgw (from .../radosgw_0.94.4-1~bpo70+1_amd64.deb) ... [localhost][DEBUG ] Processing triggers for man-db ... [localhost][DEBUG ] Setting up ceph-common (0.94.4-1~bpo70+1) ... [localhost][DEBUG ] Setting up ceph (0.94.4-1~bpo70+1) ... [localhost][DEBUG ] Setting up ceph-mds (0.94.4-1~bpo70+1) ... [localhost][DEBUG ] Setting up radosgw (0.94.4-1~bpo70+1) ... [localhost][INFO ] Running command: ceph --version [localhost][DEBUG ] ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a) [ceph_deploy.install][DEBUG ] Detecting platform for host node1 ... [node1][DEBUG ] connected to host: node1 [node1][DEBUG ] detect platform information from remote host [node1][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: debian 7.9 wheezy [node1][INFO ] installing Ceph on node1 [node1][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates [node1][DEBUG ] Reading package lists... [node1][DEBUG ] Building dependency tree... [node1][DEBUG ] Reading state information... [node1][DEBUG ] ca-certificates is already the newest version. [node1][DEBUG ] The following packages were automatically installed and are no longer required: [node1][DEBUG ] cryptsetup-bin gdisk libaio1 libboost-program-options1.49.0 [node1][DEBUG ] libboost-system1.49.0 libboost-thread1.49.0 libcephfs1 libcryptsetup4 [node1][DEBUG ] libcurl3-gnutls libfcgi0ldbl libgoogle-perftools4 libicu48 libjs-jquery [node1][DEBUG ] libleveldb1 libnspr4 libnss3 librados2 librbd1 libreadline5 librtmp0 [node1][DEBUG ] libsnappy1 libssh2-1 libtcmalloc-minimal4 libunwind7 python-cephfs [node1][DEBUG ] python-flask python-jinja2 python-markupsafe python-rados python-rbd [node1][DEBUG ] python-requests python-six python-werkzeug sdparm xfsprogs [node1][DEBUG ] Use ‘apt-get autoremove‘ to remove them. [node1][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. [node1][INFO ] Running command: wget -O release.asc http://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/release.asc [node1][WARNIN] --2015-10-21 14:43:43-- http://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/release.asc [node1][WARNIN] Resolving git.ceph.com (git.ceph.com)... 67.205.20.229 [node1][WARNIN] Connecting to git.ceph.com (git.ceph.com)|67.205.20.229|:80... connected. [node1][WARNIN] HTTP request sent, awaiting response... 200 OK [node1][WARNIN] Length: 1645 (1.6K) [text/plain] [node1][WARNIN] Saving to: `release.asc‘ [node1][WARNIN] [node1][WARNIN] 0K . 100% 147M=0s [node1][WARNIN] [node1][WARNIN] 2015-10-21 14:44:30 (147 MB/s) - `release.asc‘ saved [1645/1645] [node1][WARNIN] [node1][INFO ] Running command: apt-key add release.asc [node1][DEBUG ] OK [node1][DEBUG ] add deb repo to /etc/apt/sources.list.d/ [node1][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update [node1][DEBUG ] Hit http://mirrors.163.com wheezy Release.gpg [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates Release.gpg [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports Release.gpg [node1][DEBUG ] Get:1 http://mirrors.163.com wheezy/updates Release.gpg [1,554 B] [node1][DEBUG ] Hit http://mirrors.163.com wheezy Release [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates Release [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports Release [node1][DEBUG ] Get:2 http://mirrors.163.com wheezy/updates Release [102 kB] [node1][DEBUG ] Hit http://mirrors.163.com wheezy/main Sources [node1][DEBUG ] Hit http://mirrors.163.com wheezy/non-free Sources [node1][DEBUG ] Hit http://mirrors.163.com wheezy/contrib Sources [node1][DEBUG ] Hit http://mirrors.163.com wheezy/main amd64 Packages [node1][DEBUG ] Hit http://mirrors.163.com wheezy/non-free amd64 Packages [node1][DEBUG ] Hit http://mirrors.163.com wheezy/contrib amd64 Packages [node1][DEBUG ] Hit http://mirrors.163.com wheezy/contrib Translation-en [node1][DEBUG ] Hit http://mirrors.163.com wheezy/main Translation-en [node1][DEBUG ] Hit http://mirrors.163.com wheezy/non-free Translation-en [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main Sources [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free Sources [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib Sources [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main amd64 Packages/DiffIndex [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free amd64 Packages [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib amd64 Packages [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib Translation-en [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main Translation-en [node1][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free Translation-en [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main Sources/DiffIndex [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free Sources [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib Sources [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main amd64 Packages/DiffIndex [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free amd64 Packages [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib amd64 Packages [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib Translation-en [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main Translation-en/DiffIndex [node1][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free Translation-en [node1][DEBUG ] Get:3 http://mirrors.163.com wheezy/updates/main Sources [195 kB] [node1][DEBUG ] Get:4 http://mirrors.163.com wheezy/updates/non-free Sources [14 B] [node1][DEBUG ] Get:5 http://mirrors.163.com wheezy/updates/contrib Sources [14 B] [node1][DEBUG ] Get:6 http://mirrors.163.com wheezy/updates/main amd64 Packages [320 kB] [node1][DEBUG ] Get:7 http://mirrors.163.com wheezy/updates/non-free amd64 Packages [14 B] [node1][DEBUG ] Get:8 http://mirrors.163.com wheezy/updates/contrib amd64 Packages [14 B] [node1][DEBUG ] Get:9 http://mirrors.163.com wheezy/updates/contrib Translation-en [523 B] [node1][DEBUG ] Get:10 http://mirrors.163.com wheezy/updates/main Translation-en [183 kB] [node1][DEBUG ] Get:11 http://mirrors.163.com wheezy/updates/non-free Translation-en [14 B] [node1][DEBUG ] Hit http://ceph.com wheezy Release.gpg [node1][DEBUG ] Hit http://ceph.com wheezy Release [node1][DEBUG ] Hit http://ceph.com wheezy/main amd64 Packages [node1][DEBUG ] Ign http://ceph.com wheezy/main Translation-en_HK [node1][DEBUG ] Ign http://ceph.com wheezy/main Translation-en [node1][DEBUG ] Fetched 802 kB in 13s (59.7 kB/s) [node1][DEBUG ] Reading package lists... [node1][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-mds radosgw [node1][DEBUG ] Reading package lists... [node1][DEBUG ] Building dependency tree... [node1][DEBUG ] Reading state information... [node1][DEBUG ] The following extra packages will be installed: [node1][DEBUG ] ceph-common [node1][DEBUG ] Recommended packages: [node1][DEBUG ] btrfs-tools libradosstriper1 ceph-fs-common ceph-fuse [node1][DEBUG ] The following NEW packages will be installed: [node1][DEBUG ] ceph ceph-common ceph-mds radosgw [node1][DEBUG ] 0 upgraded, 4 newly installed, 0 to remove and 1 not upgraded. [node1][DEBUG ] Need to get 0 B/45.7 MB of archives. [node1][DEBUG ] After this operation, 142 MB of additional disk space will be used. [node1][DEBUG ] Selecting previously unselected package ceph-common. [node1][DEBUG ] (Reading database ... 31882 files and directories currently installed.) [node1][DEBUG ] Unpacking ceph-common (from .../ceph-common_0.94.4-1~bpo70+1_amd64.deb) ... [node1][DEBUG ] Selecting previously unselected package ceph. [node1][DEBUG ] Unpacking ceph (from .../ceph_0.94.4-1~bpo70+1_amd64.deb) ... [node1][DEBUG ] Selecting previously unselected package ceph-mds. [node1][DEBUG ] Unpacking ceph-mds (from .../ceph-mds_0.94.4-1~bpo70+1_amd64.deb) ... [node1][DEBUG ] Selecting previously unselected package radosgw. [node1][DEBUG ] Unpacking radosgw (from .../radosgw_0.94.4-1~bpo70+1_amd64.deb) ... [node1][DEBUG ] Processing triggers for man-db ... [node1][DEBUG ] Setting up ceph-common (0.94.4-1~bpo70+1) ... [node1][DEBUG ] Setting up ceph (0.94.4-1~bpo70+1) ... [node1][DEBUG ] Setting up ceph-mds (0.94.4-1~bpo70+1) ... [node1][DEBUG ] Setting up radosgw (0.94.4-1~bpo70+1) ... [node1][INFO ] Running command: ceph --version [node1][DEBUG ] ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a) [ceph_deploy.install][DEBUG ] Detecting platform for host node2 ... [node2][DEBUG ] connected to host: node2 [node2][DEBUG ] detect platform information from remote host [node2][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: debian 7.9 wheezy [node2][INFO ] installing Ceph on node2 [node2][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates [node2][DEBUG ] Reading package lists... [node2][DEBUG ] Building dependency tree... [node2][DEBUG ] Reading state information... [node2][DEBUG ] ca-certificates is already the newest version. [node2][DEBUG ] The following packages were automatically installed and are no longer required: [node2][DEBUG ] cryptsetup-bin gdisk libaio1 libboost-program-options1.49.0 [node2][DEBUG ] libboost-system1.49.0 libboost-thread1.49.0 libcephfs1 libcryptsetup4 [node2][DEBUG ] libcurl3-gnutls libfcgi0ldbl libgoogle-perftools4 libicu48 libjs-jquery [node2][DEBUG ] libleveldb1 libnspr4 libnss3 librados2 librbd1 libreadline5 librtmp0 [node2][DEBUG ] libsnappy1 libssh2-1 libtcmalloc-minimal4 libunwind7 openssh-blacklist [node2][DEBUG ] openssh-blacklist-extra python-cephfs python-flask python-jinja2 [node2][DEBUG ] python-markupsafe python-rados python-rbd python-requests python-six [node2][DEBUG ] python-werkzeug sdparm xfsprogs [node2][DEBUG ] Use ‘apt-get autoremove‘ to remove them. [node2][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. [node2][INFO ] Running command: wget -O release.asc http://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/release.asc [node2][WARNIN] --2015-10-21 14:44:50-- http://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/release.asc [node2][WARNIN] Resolving git.ceph.com (git.ceph.com)... 67.205.20.229 [node2][WARNIN] Connecting to git.ceph.com (git.ceph.com)|67.205.20.229|:80... connected. [node2][WARNIN] HTTP request sent, awaiting response... 200 OK [node2][WARNIN] Length: 1645 (1.6K) [text/plain] [node2][WARNIN] Saving to: `release.asc‘ [node2][WARNIN] [node2][WARNIN] 0K . 100% 128M=0s [node2][WARNIN] [node2][WARNIN] 2015-10-21 14:45:34 (128 MB/s) - `release.asc‘ saved [1645/1645] [node2][WARNIN] [node2][INFO ] Running command: apt-key add release.asc [node2][DEBUG ] OK [node2][DEBUG ] add deb repo to /etc/apt/sources.list.d/ [node2][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update [node2][DEBUG ] Hit http://mirrors.163.com wheezy Release.gpg [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates Release.gpg [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports Release.gpg [node2][DEBUG ] Get:1 http://mirrors.163.com wheezy/updates Release.gpg [1,554 B] [node2][DEBUG ] Hit http://mirrors.163.com wheezy Release [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates Release [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports Release [node2][DEBUG ] Get:2 http://mirrors.163.com wheezy/updates Release [102 kB] [node2][DEBUG ] Hit http://mirrors.163.com wheezy/main Sources [node2][DEBUG ] Hit http://mirrors.163.com wheezy/non-free Sources [node2][DEBUG ] Hit http://mirrors.163.com wheezy/contrib Sources [node2][DEBUG ] Hit http://mirrors.163.com wheezy/main amd64 Packages [node2][DEBUG ] Hit http://mirrors.163.com wheezy/non-free amd64 Packages [node2][DEBUG ] Hit http://mirrors.163.com wheezy/contrib amd64 Packages [node2][DEBUG ] Hit http://mirrors.163.com wheezy/contrib Translation-en [node2][DEBUG ] Hit http://mirrors.163.com wheezy/main Translation-en [node2][DEBUG ] Hit http://mirrors.163.com wheezy/non-free Translation-en [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main Sources [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free Sources [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib Sources [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main amd64 Packages/DiffIndex [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free amd64 Packages [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib amd64 Packages [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib Translation-en [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main Translation-en [node2][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free Translation-en [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main Sources/DiffIndex [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free Sources [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib Sources [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main amd64 Packages/DiffIndex [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free amd64 Packages [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib amd64 Packages [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib Translation-en [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main Translation-en/DiffIndex [node2][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free Translation-en [node2][DEBUG ] Get:3 http://mirrors.163.com wheezy/updates/main Sources [195 kB] [node2][DEBUG ] Get:4 http://mirrors.163.com wheezy/updates/non-free Sources [14 B] [node2][DEBUG ] Get:5 http://mirrors.163.com wheezy/updates/contrib Sources [14 B] [node2][DEBUG ] Get:6 http://mirrors.163.com wheezy/updates/main amd64 Packages [320 kB] [node2][DEBUG ] Get:7 http://mirrors.163.com wheezy/updates/non-free amd64 Packages [14 B] [node2][DEBUG ] Get:8 http://mirrors.163.com wheezy/updates/contrib amd64 Packages [14 B] [node2][DEBUG ] Get:9 http://mirrors.163.com wheezy/updates/contrib Translation-en [523 B] [node2][DEBUG ] Get:10 http://mirrors.163.com wheezy/updates/main Translation-en [183 kB] [node2][DEBUG ] Hit http://ceph.com wheezy Release.gpg [node2][DEBUG ] Get:11 http://mirrors.163.com wheezy/updates/non-free Translation-en [14 B] [node2][DEBUG ] Hit http://ceph.com wheezy Release [node2][DEBUG ] Hit http://ceph.com wheezy/main amd64 Packages [node2][DEBUG ] Ign http://ceph.com wheezy/main Translation-en_HK [node2][DEBUG ] Ign http://ceph.com wheezy/main Translation-en [node2][DEBUG ] Fetched 802 kB in 15s (52.5 kB/s) [node2][DEBUG ] Reading package lists... [node2][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-mds radosgw [node2][DEBUG ] Reading package lists... [node2][DEBUG ] Building dependency tree... [node2][DEBUG ] Reading state information... [node2][DEBUG ] The following packages were automatically installed and are no longer required: [node2][DEBUG ] openssh-blacklist openssh-blacklist-extra [node2][DEBUG ] Use ‘apt-get autoremove‘ to remove them. [node2][DEBUG ] The following extra packages will be installed: [node2][DEBUG ] ceph-common [node2][DEBUG ] Recommended packages: [node2][DEBUG ] btrfs-tools libradosstriper1 ceph-fs-common ceph-fuse [node2][DEBUG ] The following NEW packages will be installed: [node2][DEBUG ] ceph ceph-common ceph-mds radosgw [node2][DEBUG ] 0 upgraded, 4 newly installed, 0 to remove and 1 not upgraded. [node2][DEBUG ] Need to get 0 B/45.7 MB of archives. [node2][DEBUG ] After this operation, 142 MB of additional disk space will be used. [node2][DEBUG ] Selecting previously unselected package ceph-common. [node2][DEBUG ] (Reading database ... 31897 files and directories currently installed.) [node2][DEBUG ] Unpacking ceph-common (from .../ceph-common_0.94.4-1~bpo70+1_amd64.deb) ... [node2][DEBUG ] Selecting previously unselected package ceph. [node2][DEBUG ] Unpacking ceph (from .../ceph_0.94.4-1~bpo70+1_amd64.deb) ... [node2][DEBUG ] Selecting previously unselected package ceph-mds. [node2][DEBUG ] Unpacking ceph-mds (from .../ceph-mds_0.94.4-1~bpo70+1_amd64.deb) ... [node2][DEBUG ] Selecting previously unselected package radosgw. [node2][DEBUG ] Unpacking radosgw (from .../radosgw_0.94.4-1~bpo70+1_amd64.deb) ... [node2][DEBUG ] Processing triggers for man-db ... [node2][DEBUG ] Setting up ceph-common (0.94.4-1~bpo70+1) ... [node2][DEBUG ] Setting up ceph (0.94.4-1~bpo70+1) ... [node2][DEBUG ] Setting up ceph-mds (0.94.4-1~bpo70+1) ... [node2][DEBUG ] Setting up radosgw (0.94.4-1~bpo70+1) ... [node2][INFO ] Running command: ceph --version [node2][DEBUG ] ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a) [ceph_deploy.install][DEBUG ] Detecting platform for host node3 ... [node3][DEBUG ] connected to host: node3 [node3][DEBUG ] detect platform information from remote host [node3][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: debian 7.9 wheezy [node3][INFO ] installing Ceph on node3 [node3][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates [node3][DEBUG ] Reading package lists... [node3][DEBUG ] Building dependency tree... [node3][DEBUG ] Reading state information... [node3][DEBUG ] ca-certificates is already the newest version. [node3][DEBUG ] The following packages were automatically installed and are no longer required: [node3][DEBUG ] cryptsetup-bin gdisk libaio1 libboost-program-options1.49.0 [node3][DEBUG ] libboost-system1.49.0 libboost-thread1.49.0 libcephfs1 libcryptsetup4 [node3][DEBUG ] libcurl3-gnutls libfcgi0ldbl libgoogle-perftools4 libicu48 libjs-jquery [node3][DEBUG ] libleveldb1 libnspr4 libnss3 librados2 librbd1 libreadline5 librtmp0 [node3][DEBUG ] libsnappy1 libssh2-1 libtcmalloc-minimal4 libunwind7 openssh-blacklist [node3][DEBUG ] openssh-blacklist-extra python-cephfs python-flask python-jinja2 [node3][DEBUG ] python-markupsafe python-rados python-rbd python-requests python-six [node3][DEBUG ] python-werkzeug sdparm xfsprogs [node3][DEBUG ] Use ‘apt-get autoremove‘ to remove them. [node3][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. [node3][INFO ] Running command: wget -O release.asc http://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/release.asc [node3][WARNIN] --2015-10-21 14:45:56-- http://git.ceph.com/?p=ceph.git;a=blob_plain;f=keys/release.asc [node3][WARNIN] Resolving git.ceph.com (git.ceph.com)... 67.205.20.229 [node3][WARNIN] Connecting to git.ceph.com (git.ceph.com)|67.205.20.229|:80... connected. [node3][WARNIN] HTTP request sent, awaiting response... 200 OK [node3][WARNIN] Length: 1645 (1.6K) [text/plain] [node3][WARNIN] Saving to: `release.asc‘ [node3][WARNIN] [node3][WARNIN] 0K . 100% 144M=0s [node3][WARNIN] [node3][WARNIN] 2015-10-21 14:46:39 (144 MB/s) - `release.asc‘ saved [1645/1645] [node3][WARNIN] [node3][INFO ] Running command: apt-key add release.asc [node3][DEBUG ] OK [node3][DEBUG ] add deb repo to /etc/apt/sources.list.d/ [node3][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update [node3][DEBUG ] Hit http://mirrors.163.com wheezy Release.gpg [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates Release.gpg [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports Release.gpg [node3][DEBUG ] Get:1 http://mirrors.163.com wheezy/updates Release.gpg [1,554 B] [node3][DEBUG ] Hit http://mirrors.163.com wheezy Release [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates Release [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports Release [node3][DEBUG ] Get:2 http://mirrors.163.com wheezy/updates Release [102 kB] [node3][DEBUG ] Hit http://mirrors.163.com wheezy/main Sources [node3][DEBUG ] Hit http://mirrors.163.com wheezy/non-free Sources [node3][DEBUG ] Hit http://mirrors.163.com wheezy/contrib Sources [node3][DEBUG ] Hit http://mirrors.163.com wheezy/main amd64 Packages [node3][DEBUG ] Hit http://mirrors.163.com wheezy/non-free amd64 Packages [node3][DEBUG ] Hit http://mirrors.163.com wheezy/contrib amd64 Packages [node3][DEBUG ] Hit http://mirrors.163.com wheezy/contrib Translation-en [node3][DEBUG ] Hit http://mirrors.163.com wheezy/main Translation-en [node3][DEBUG ] Hit http://mirrors.163.com wheezy/non-free Translation-en [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main Sources [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free Sources [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib Sources [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main amd64 Packages/DiffIndex [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free amd64 Packages [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib amd64 Packages [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/contrib Translation-en [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/main Translation-en [node3][DEBUG ] Hit http://mirrors.163.com wheezy-updates/non-free Translation-en [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main Sources/DiffIndex [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free Sources [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib Sources [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main amd64 Packages/DiffIndex [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free amd64 Packages [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib amd64 Packages [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/contrib Translation-en [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/main Translation-en/DiffIndex [node3][DEBUG ] Hit http://mirrors.163.com wheezy-backports/non-free Translation-en [node3][DEBUG ] Get:3 http://mirrors.163.com wheezy/updates/main Sources [195 kB] [node3][DEBUG ] Get:4 http://mirrors.163.com wheezy/updates/non-free Sources [14 B] [node3][DEBUG ] Get:5 http://mirrors.163.com wheezy/updates/contrib Sources [14 B] [node3][DEBUG ] Get:6 http://mirrors.163.com wheezy/updates/main amd64 Packages [320 kB] [node3][DEBUG ] Get:7 http://mirrors.163.com wheezy/updates/non-free amd64 Packages [14 B] [node3][DEBUG ] Get:8 http://mirrors.163.com wheezy/updates/contrib amd64 Packages [14 B] [node3][DEBUG ] Get:9 http://mirrors.163.com wheezy/updates/contrib Translation-en [523 B] [node3][DEBUG ] Get:10 http://mirrors.163.com wheezy/updates/main Translation-en [183 kB] [node3][DEBUG ] Get:11 http://mirrors.163.com wheezy/updates/non-free Translation-en [14 B] [node3][DEBUG ] Hit http://ceph.com wheezy Release.gpg [node3][DEBUG ] Hit http://ceph.com wheezy Release [node3][DEBUG ] Hit http://ceph.com wheezy/main amd64 Packages [node3][DEBUG ] Ign http://ceph.com wheezy/main Translation-en_HK [node3][DEBUG ] Ign http://ceph.com wheezy/main Translation-en [node3][DEBUG ] Fetched 802 kB in 22s (35.4 kB/s) [node3][DEBUG ] Reading package lists... [node3][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-mds radosgw [node3][DEBUG ] Reading package lists... [node3][DEBUG ] Building dependency tree... [node3][DEBUG ] Reading state information... [node3][DEBUG ] The following packages were automatically installed and are no longer required: [node3][DEBUG ] openssh-blacklist openssh-blacklist-extra [node3][DEBUG ] Use ‘apt-get autoremove‘ to remove them. [node3][DEBUG ] The following extra packages will be installed: [node3][DEBUG ] ceph-common [node3][DEBUG ] Recommended packages: [node3][DEBUG ] btrfs-tools libradosstriper1 ceph-fs-common ceph-fuse [node3][DEBUG ] The following NEW packages will be installed: [node3][DEBUG ] ceph ceph-common ceph-mds radosgw [node3][DEBUG ] 0 upgraded, 4 newly installed, 0 to remove and 1 not upgraded. [node3][DEBUG ] Need to get 0 B/45.7 MB of archives. [node3][DEBUG ] After this operation, 142 MB of additional disk space will be used. [node3][DEBUG ] Selecting previously unselected package ceph-common. [node3][DEBUG ] (Reading database ... 31897 files and directories currently installed.) [node3][DEBUG ] Unpacking ceph-common (from .../ceph-common_0.94.4-1~bpo70+1_amd64.deb) ... [node3][DEBUG ] Selecting previously unselected package ceph. [node3][DEBUG ] Unpacking ceph (from .../ceph_0.94.4-1~bpo70+1_amd64.deb) ... [node3][DEBUG ] Selecting previously unselected package ceph-mds. [node3][DEBUG ] Unpacking ceph-mds (from .../ceph-mds_0.94.4-1~bpo70+1_amd64.deb) ... [node3][DEBUG ] Selecting previously unselected package radosgw. [node3][DEBUG ] Unpacking radosgw (from .../radosgw_0.94.4-1~bpo70+1_amd64.deb) ... [node3][DEBUG ] Processing triggers for man-db ... [node3][DEBUG ] Setting up ceph-common (0.94.4-1~bpo70+1) ... [node3][DEBUG ] Setting up ceph (0.94.4-1~bpo70+1) ... [node3][DEBUG ] Setting up ceph-mds (0.94.4-1~bpo70+1) ... [node3][DEBUG ] Setting up radosgw (0.94.4-1~bpo70+1) ... [node3][INFO ] Running command: ceph --version [node3][DEBUG ] ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a) root@admin-node:/home/my-cluster#
在所有节点，看看都安装了什么东西。
root@node1:~# find / -iname "*ceph*" /lib/udev/rules.d/95-ceph-osd.rules /lib/udev/rules.d/60-ceph-partuuid-workaround.rules /lib/modules/3.2.0-4-amd64/kernel/fs/ceph /lib/modules/3.2.0-4-amd64/kernel/fs/ceph/ceph.ko /lib/modules/3.2.0-4-amd64/kernel/net/ceph /lib/modules/3.2.0-4-amd64/kernel/net/ceph/libceph.ko /lib/modules/3.16.0-0.bpo.4-amd64/kernel/fs/ceph /lib/modules/3.16.0-0.bpo.4-amd64/kernel/fs/ceph/ceph.ko /lib/modules/3.16.0-0.bpo.4-amd64/kernel/net/ceph /lib/modules/3.16.0-0.bpo.4-amd64/kernel/net/ceph/libceph.ko /var/cache/apt/archives/ceph-common_0.94.4-1~bpo70+1_amd64.deb /var/cache/apt/archives/ceph-mds_0.94.4-1~bpo70+1_amd64.deb /var/cache/apt/archives/python-cephfs_0.94.4-1~bpo70+1_amd64.deb /var/cache/apt/archives/libcephfs1_0.94.4-1~bpo70+1_amd64.deb /var/cache/apt/archives/ceph_0.94.4-1~bpo70+1_amd64.deb /var/lib/ceph /var/lib/apt/lists/ceph.com_debian-hammer_dists_wheezy_Release.gpg /var/lib/apt/lists/ceph.com_debian-hammer_dists_wheezy_Release /var/lib/apt/lists/ceph.com_debian-hammer_dists_wheezy_main_binary-amd64_Packages /var/lib/dpkg/info/ceph-common.prerm /var/lib/dpkg/info/ceph.conffiles /var/lib/dpkg/info/ceph-common.md5sums /var/lib/dpkg/info/ceph.postinst /var/lib/dpkg/info/libcephfs1.postinst /var/lib/dpkg/info/libcephfs1.md5sums /var/lib/dpkg/info/ceph-mds.conffiles /var/lib/dpkg/info/ceph-mds.postinst /var/lib/dpkg/info/ceph-mds.list /var/lib/dpkg/info/python-cephfs.list /var/lib/dpkg/info/ceph.prerm /var/lib/dpkg/info/ceph-mds.prerm /var/lib/dpkg/info/python-cephfs.prerm /var/lib/dpkg/info/python-cephfs.postinst /var/lib/dpkg/info/ceph-mds.md5sums /var/lib/dpkg/info/ceph-common.list /var/lib/dpkg/info/ceph.shlibs /var/lib/dpkg/info/libcephfs1.list /var/lib/dpkg/info/ceph-common.conffiles /var/lib/dpkg/info/ceph.list /var/lib/dpkg/info/libcephfs1.postrm /var/lib/dpkg/info/ceph-common.postrm /var/lib/dpkg/info/python-cephfs.md5sums /var/lib/dpkg/info/ceph-common.postinst /var/lib/dpkg/info/ceph.md5sums /var/lib/dpkg/info/libcephfs1.shlibs /var/lib/dpkg/info/ceph.postrm /var/log/ceph /etc/ceph /etc/init.d/ceph /etc/logrotate.d/ceph /etc/rc3.d/S16ceph /etc/rc1.d/K02ceph /etc/rc6.d/K02ceph /etc/rc0.d/K02ceph /etc/rc4.d/S16ceph /etc/apt/sources.list.d/ceph.list /etc/apt/preferences.d/ceph.pref /etc/bash_completion.d/ceph /etc/init/ceph-mon-all-starter.conf /etc/init/ceph-mon.conf /etc/init/ceph-mon-all.conf /etc/init/ceph-osd-all.conf /etc/init/ceph-mds.conf /etc/init/ceph-create-keys.conf /etc/init/ceph-mds-all.conf /etc/init/ceph-osd.conf /etc/init/ceph-osd-all-starter.conf /etc/init/ceph-all.conf /etc/init/ceph-mds-all-starter.conf /etc/rc2.d/S16ceph /etc/rc5.d/S16ceph /usr/lib/libcephfs.so.1 /usr/lib/ceph /usr/lib/ceph/ceph_common.sh /usr/lib/libcephfs.so.1.0.0 /usr/lib/python2.6/dist-packages/ceph_argparse.py /usr/lib/python2.6/dist-packages/ceph_rest_api.py /usr/lib/python2.6/dist-packages/cephfs.pyc /usr/lib/python2.6/dist-packages/ceph_argparse.pyc /usr/lib/python2.6/dist-packages/cephfs.py /usr/lib/python2.6/dist-packages/ceph_rest_api.pyc /usr/lib/python2.7/dist-packages/ceph_argparse.py /usr/lib/python2.7/dist-packages/ceph_rest_api.py /usr/lib/python2.7/dist-packages/cephfs.pyc /usr/lib/python2.7/dist-packages/ceph_argparse.pyc /usr/lib/python2.7/dist-packages/cephfs.py /usr/lib/python2.7/dist-packages/ceph_rest_api.pyc /usr/sbin/ceph-disk-prepare /usr/sbin/ceph-create-keys /usr/sbin/ceph-disk /usr/sbin/ceph-disk-activate /usr/bin/ceph-objectstore-tool /usr/bin/ceph-dencoder /usr/bin/ceph /usr/bin/ceph-authtool /usr/bin/ceph-rbdnamer /usr/bin/ceph-run /usr/bin/ceph-osd /usr/bin/ceph-mds /usr/bin/cephfs-journal-tool /usr/bin/ceph-debugpack /usr/bin/ceph-mon /usr/bin/ceph-post-file /usr/bin/ceph-crush-location /usr/bin/ceph-syn /usr/bin/ceph-conf /usr/bin/ceph-brag /usr/bin/ceph-clsinfo /usr/bin/ceph-rest-api /usr/bin/cephfs-table-tool /usr/libexec/ceph /usr/libexec/ceph/ceph-osd-prestart.sh /usr/share/man/man8/ceph-conf.8.gz /usr/share/man/man8/ceph-authtool.8.gz /usr/share/man/man8/ceph-deploy.8.gz /usr/share/man/man8/ceph-rest-api.8.gz /usr/share/man/man8/ceph-post-file.8.gz /usr/share/man/man8/ceph-create-keys.8.gz /usr/share/man/man8/ceph-clsinfo.8.gz /usr/share/man/man8/ceph-dencoder.8.gz /usr/share/man/man8/ceph-debugpack.8.gz /usr/share/man/man8/ceph-syn.8.gz /usr/share/man/man8/ceph-mon.8.gz /usr/share/man/man8/ceph-disk.8.gz /usr/share/man/man8/ceph-run.8.gz /usr/share/man/man8/ceph-mds.8.gz /usr/share/man/man8/ceph.8.gz /usr/share/man/man8/ceph-osd.8.gz /usr/share/man/man8/ceph-rbdnamer.8.gz /usr/share/ceph /usr/share/ceph/known_hosts_drop.ceph.com /usr/share/ceph/id_dsa_drop.ceph.com.pub /usr/share/ceph/id_dsa_drop.ceph.com /usr/share/lintian/overrides/ceph /usr/share/pyshared/ceph_argparse.py /usr/share/pyshared/ceph_rest_api.py /usr/share/pyshared/cephfs.py /usr/share/doc/libcephfs1 /usr/share/doc/ceph /usr/share/doc/ceph/sample.ceph.conf.gz /usr/share/doc/ceph-common /usr/share/doc/ceph-mds /usr/share/doc/python-cephfs root@node1:~#
添加ceph monitor
root@admin-node:/home/my-cluster# ceph-deploy mon create-initial root@admin-node:/home/my-cluster# ll total 72 -rw------- 1 root root 71 Oct 21 14:48 ceph.bootstrap-mds.keyring -rw------- 1 root root 71 Oct 21 14:48 ceph.bootstrap-osd.keyring -rw------- 1 root root 71 Oct 21 14:48 ceph.bootstrap-rgw.keyring -rw------- 1 root root 63 Oct 21 14:48 ceph.client.admin.keyring -rw-r--r-- 1 root root 253 Oct 21 14:41 ceph.conf -rw-r--r-- 1 root root 44601 Oct 21 14:48 ceph.log -rw------- 1 root root 73 Oct 21 14:41 ceph.mon.keyring
*******************************
如果报错了，monitor ？ does not exist in monmap
请清除各节点的ceph程序和配置文件
ceph-deploy purge localhost node1 node2 node3
ceph-deploy purgedata localhost node1 node2 node3
admin节点 reboot
admin节点执行 ceph-deploy --help
admin节点如果执行不了，那么就安装
apt-get remove ceph-deploy
apt-get install ceph-deploy
回到【创建集群】，再来一次就好了
*********************************
Add two OSDs
可以把其他的盘mount到这个文件夹，不mount也可以 ssh node2 sudo mkdir /var/local/osd0 exit ssh node3 sudo mkdir /var/local/osd1 exit
prepare the OSDs.
root@admin-node:/home/my-cluster# ceph-deploy osd prepare node2:/var/local/osd0 node3:/var/local/osd1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.28): /usr/bin/ceph-deploy osd prepare node2:/var/local/osd0 node3:/var/local/osd1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] disk : [(‘node2‘, ‘/var/local/osd0‘, None), (‘node3‘, ‘/var/local/osd1‘, None)]
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : prepare
[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fe1cced41b8>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] fs_type : xfs
[ceph_deploy.cli][INFO ] func : <function osd at 0x7fe1ccec4140>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node2:/var/local/osd0: node3:/var/local/osd1:
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: debian 7.9 wheezy
[ceph_deploy.osd][DEBUG ] Deploying osd to node2
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][WARNIN] osd keyring does not exist yet, creating one
[node2][DEBUG ] create a keyring file
[node2][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host node2 disk /var/local/osd0 journal None activate False
[node2][INFO ] Running command: ceph-disk -v prepare --cluster ceph --fs-type xfs -- /var/local/osd0
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
[node2][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/local/osd0
[node2][INFO ] checking OSD status...
[node2][INFO ] Running command: ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node2 is now ready for osd use.
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[node3][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: debian 7.9 wheezy
[ceph_deploy.osd][DEBUG ] Deploying osd to node3
[node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node3][WARNIN] osd keyring does not exist yet, creating one
[node3][DEBUG ] create a keyring file
[node3][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host node3 disk /var/local/osd1 journal None activate False
[node3][INFO ] Running command: ceph-disk -v prepare --cluster ceph --fs-type xfs -- /var/local/osd1
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
[node3][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
[node3][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/local/osd1
[node3][INFO ] checking OSD status...
[node3][INFO ] Running command: ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node3 is now ready for osd use.Activate the OSDs.
root@admin-node:/home/my-cluster# ceph-deploy osd activate node2:/var/local/osd0 node3:/var/local/osd1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.28): /usr/bin/ceph-deploy osd activate node2:/var/local/osd0 node3:/var/local/osd1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : activate
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f25526731b8>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function osd at 0x7f2552663140>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : [(‘node2‘, ‘/var/local/osd0‘, None), (‘node3‘, ‘/var/local/osd1‘, None)]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node2:/var/local/osd0: node3:/var/local/osd1:
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: debian 7.9 wheezy
[ceph_deploy.osd][DEBUG ] activating host node2 disk /var/local/osd0
[ceph_deploy.osd][DEBUG ] will use init type: sysvinit
[node2][INFO ] Running command: ceph-disk -v activate --mark-init sysvinit --mount /var/local/osd0
[node2][WARNIN] DEBUG:ceph-disk:Cluster uuid is 43f7498a-653b-4b23-92c5-a89f311721df
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[node2][WARNIN] DEBUG:ceph-disk:Cluster name is ceph
[node2][WARNIN] DEBUG:ceph-disk:OSD uuid is 1334758b-76f3-4067-99a5-3d8850db1d42
[node2][WARNIN] DEBUG:ceph-disk:Allocating OSD id...
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 1334758b-76f3-4067-99a5-3d8850db1d42
[node2][WARNIN] DEBUG:ceph-disk:OSD id is 0
[node2][WARNIN] DEBUG:ceph-disk:Initializing OSD...
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/local/osd0/activate.monmap
[node2][WARNIN] got monmap epoch 1
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /var/local/osd0/activate.monmap --osd-data /var/local/osd0 --osd-journal /var/local/osd0/journal --osd-uuid 1334758b-76f3-4067-99a5-3d8850db1d42 --keyring /var/local/osd0/keyring
[node2][WARNIN] 2015-10-21 16:26:32.403850 7f8cee8e0840 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
[node2][WARNIN] 2015-10-21 16:26:32.483448 7f8cee8e0840 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
[node2][WARNIN] 2015-10-21 16:26:32.484407 7f8cee8e0840 -1 filestore(/var/local/osd0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
[node2][WARNIN] 2015-10-21 16:26:32.543063 7f8cee8e0840 -1 created object store /var/local/osd0 journal /var/local/osd0/journal for osd.0 fsid d1b6c3d3-6439-47fa-9c94-3d8055fd07ff
[node2][WARNIN] 2015-10-21 16:26:32.543141 7f8cee8e0840 -1 auth: error reading file: /var/local/osd0/keyring: can‘t open /var/local/osd0/keyring: (2) No such file or directory
[node2][WARNIN] 2015-10-21 16:26:32.543237 7f8cee8e0840 -1 created new key in keyring /var/local/osd0/keyring
[node2][WARNIN] DEBUG:ceph-disk:Marking with init system sysvinit
[node2][WARNIN] DEBUG:ceph-disk:Authorizing OSD key...
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring auth add osd.0 -i /var/local/osd0/keyring osd allow * mon allow profile osd
[node2][WARNIN] added key for osd.0
[node2][WARNIN] DEBUG:ceph-disk:ceph osd.0 data dir is ready at /var/local/osd0
[node2][WARNIN] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/osd/ceph-0 -> /var/local/osd0
[node2][WARNIN] DEBUG:ceph-disk:Starting ceph osd.0...
[node2][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/service ceph --cluster ceph start osd.0
[node2][DEBUG ] === osd.0 ===
[node2][WARNIN] create-or-move updating item name ‘osd.0‘ weight 0.01 at location {host=node2,root=default} to crush map
[node2][DEBUG ] Starting Ceph osd.0 on node2...
[node2][DEBUG ] starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
[node2][INFO ] checking OSD status...
[node2][INFO ] Running command: ceph --cluster=ceph osd stat --format=json
[node2][INFO ] Running command: chkconfig ceph on
[node2][ERROR ] Traceback (most recent call last):
[node2][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/vendor/remoto/process.py", line 118, in run
[node2][ERROR ] reporting(conn, result, timeout)
[node2][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/vendor/remoto/log.py", line 13, in reporting
[node2][ERROR ] received = result.receive(timeout)
[node2][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/vendor/remoto/lib/vendor/execnet/gateway_base.py", line 704, in receive
[node2][ERROR ] raise self._getremoteerror() or EOFError()
[node2][ERROR ] RemoteError: Traceback (most recent call last):
[node2][ERROR ] File "<string>", line 1036, in executetask
[node2][ERROR ] File "<remote exec>", line 12, in _remote_run
[node2][ERROR ] File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
[node2][ERROR ] errread, errwrite)
[node2][ERROR ] File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
[node2][ERROR ] raise child_exception
[node2][ERROR ] OSError: [Errno 2] No such file or directory
[node2][ERROR ]
[node2][ERROR ]
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: chkconfig ceph onadmin节点：
拷贝配置文件和admin key到 所有节点
localhost就是admin节点的主机名
root@admin-node:/home/my-cluster# ceph-deploy admin admin-node node1 node2 node3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.28): /usr/bin/ceph-deploy admin localhost node1 node2 node3
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f689c939e60>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : [‘localhost‘, ‘node1‘, ‘node2‘, ‘node3‘]
[ceph_deploy.cli][INFO ] func : <function admin at 0x7f689cd89050>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to localhost
[localhost][DEBUG ] connected to host: localhost
[localhost][DEBUG ] detect platform information from remote host
[localhost][DEBUG ] detect machine type
[localhost][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node2
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node3
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.confroot@admin-node:/home/my-cluster# root@admin-node:/home/my-cluster# chmod +r /etc/ceph/ceph.client.admin.keyring root@admin-node:/home/my-cluster# ll /etc/ceph/ceph.client.admin.keyring -rw-r--r-- 1 root root 63 Oct 21 15:19 /etc/ceph/ceph.client.admin.keyring
Check your cluster’s health.
root@admin-node:/home/my-cluster# ceph -s
cluster fbb629be-b5af-4211-9170-c6ca6740fafb
health HEALTH_WARN
64 pgs degraded
64 pgs stuck degraded
64 pgs stuck unclean
64 pgs stuck undersized
64 pgs undersized
monmap e1: 1 mons at {node1=172.16.66.133:6789/0}
election epoch 2, quorum 0 node1
osdmap e5: 1 osds: 1 up, 1 in
pgmap v10: 64 pgs, 1 pools, 0 bytes data, 0 objects
6345 MB used, 2419 MB / 9258 MB avail
64 active+undersized+degraded本文出自 “生命不息，折腾不止。” 博客，请务必保留此出处http://990487026.blog.51cto.com/10133282/1704880
02 【ceph】Storage Cluster Quick Start --> Create a Cluster
标签：【ceph】storage cluster quick start --> create a cluster
原文地址：http://990487026.blog.51cto.com/10133282/1704880
