码迷，mamicode.com
首页 > 其他好文 > 详细

基于CentOS 7.3 安装Ceph Jewel 10.2.9

时间：2019-08-05 14:14:10      阅读：110      评论：0      收藏：0      [点我收藏+]

标签：ase   tab   osd   ack   init   com   flags   guid   添加   

https://www.lijiawang.org/posts/intsall-ceph.html

 

配置说明：

采用了4台centos7.3系统的虚拟机，1台Ceph-Master作为安装节点，NTP Server；3台Ceph节点，既作为OSD节点，也作为Monitor节点。每个OSD节点有6个盘：300G的系统盘，3个2TB作为SATA池的OSD，800GB作为SSD池的OSD，240GB SSD盘作为日志盘。

环境准备

这里安装centos7.3的操作系统我就不多说了，下面我说一下环境准备工作。

1.检查操作系统的版本

1
2
# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

 

2.查看系统内核版本

1
2
# uname -r
3.10.0-514.26.2.el7.x86_64

 

3.关闭防火墙和selinux

1
2
3
4
sed -i ‘s/SELINUX=.*/SELINUX=disabled/‘ /etc/selinux/config
setenforce 0
systemctl stop firewalld
systemctl disable firewalld

 

4.查看设备信息（所有的OSD节点）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 300G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 299G 0 part
├─cl-root 253:0 0 50G 0 lvm /
├─cl-swap 253:1 0 2G 0 lvm [SWAP]
└─cl-home 253:2 0 247G 0 lvm /home
sdb 8:16 0 2T 0 disk
sdc 8:32 0 2T 0 disk
sdd 8:48 0 2T 0 disk
sde 8:64 0 800G 0 disk
sdf 8:80 0 240G 0 disk
sr0 11:0 1 1024M 0 rom

 

5.查看网卡配置（所有OSD节点）

1
2
3
4
5
6
7
8
9
10
11
12
13
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:06:c8:a4 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.11/24 brd 172.16.0.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:06:c8:ae brd ff:ff:ff:ff:ff:ff
inet 192.168.0.11/24 brd 192.168.0.255 scope global eth1
valid_lft forever preferred_lft forever

 

6.在所有节点配置hosts解析

1
2
3
4
172.16.0.10 ceph-master
172.16.0.11 ceph-node-1
172.16.0.12 ceph-node-2
172.16.0.13 ceph-node-3

 

7.安装基础的软件包(所有节点)

1
yum install tree nmap sysstat lrzsz dos2unix wegt git net-tools -y

 

8.建立SSH通信（在ceph-master节点上执行）
（1）生成秘钥文件

1
ssh-keygen -t rsa

 

（2）拷贝秘钥文件

1
2
3
4
ssh-copy-id root@ceph-master
ssh-copy-id root@ceph-node-1
ssh-copy-id root@ceph-node-2
ssh-copy-id root@ceph-node-3

 

环境准备基本完成

配置NTP服务

首先我们要在所有的节点上安装NTP服务

1
# yum install -y ntp

 

在CEPH-MASTER节点上配置

1.修改NTP配置文件/etc/ntp.conf

1
2
3
4
5
6
7
8
# vim /etc/ntp.conf
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
restrict 172.16.0.0 mask 255.255.255.0 nomodify notrap
server 127.127.1.0 minpoll 4
fudge 127.127.1.0 stratum 0

 

2.修改配置文件/etc/ntp/step-tickers

1
2
3
# vim /etc/ntp/step-tickers
#0.centos.pool.ntp.org
127.127.1.0

 

3.启动NTP服务，并设置开机启动

1
systemctl enable ntpd ; systemctl start ntpd

 

在所有的OSD节点配置

1.修改NTP配置文件/etc/ntp.conf

1
2
3
4
5
6
# vim /etc/ntp.conf
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 172.16.0.10

 

2.启动NTP服务，并设置开机启动

1
systemctl enable ntpd ; systemctl start ntpd

 

验证NTP

在所有节点执行

1
2
3
4
5
# ntpq -p
 
remote refid st t when poll reach delay offset jitter
==============================================================================
*ceph-master .LOCL. 1 u 16 64 377 0.269 0.032 0.269

 

前面有 * 表示已经同步。

安装ceph

####跟新系统源(所有节点执行)

1
2
3
4
5
6
7
8
rm -rf /etc/yum.repos.d/*.repo
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
sed -i ‘/aliyuncs/d‘ /etc/yum.repos.d/CentOS-Base.repo
sed -i ‘s/$releasever/7/g‘ /etc/yum.repos.d/CentOS-Base.repo
sed -i ‘/aliyuncs/d‘ /etc/yum.repos.d/epel.repo
yum clean all
yum makecache fast

 

安装CEPH-DEPLOY

以下在ceph-master节点上执行

1
# yum install http://mirrors.163.com/ceph/rpm-jewel/el7/noarch/ceph-deploy-1.5.38-0.noarch.rpm

 

查看CEPH-DEPLOY版本

1
2
# ceph-deploy --version
1.5.38

创建CEPH集群

1
ceph-deploy new ceph-node-1 ceph-node-2 ceph-node-3

编辑CEPH配置文件

在global下添加以下配置：

1
2
3
4
5
# vim ceph.conf
[global]
mon_clock_drift_allowed = 5
osd_journal_size = 20480
public_network=172.16.0.0/24

 

安装CEPH

直接指定源地址，不用担心ceph.com官方源无法下载了。

1
ceph-deploy install --release jewel --repo-url http://mirrors.163.com/ceph/rpm-jewel/el7 --gpg-url http://mirrors.163.com/ceph/keys/release.asc ceph-master ceph-node-1 ceph-node-2 ceph-node-3

 

检查CEPH版本

1
2
ceph -v
ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)

初始化MOM节点

1
ceph-deploy mon create-initial

查看集群状态（在OSD节点上查看）

1
2
3
4
5
6
7
8
9
10
11
# ceph -s
cluster 60597e53-ad29-44bd-8dcd-db6aeae6f580
health HEALTH_ERR
no osds
monmap e2: 3 mons at {ceph-node-1=172.16.0.11:6789/0,ceph-node-2=172.16.0.12:6789/0,ceph-node-3=172.16.0.13:6789/0}
election epoch 6, quorum 0,1,2 ceph-node-1,ceph-node-2,ceph-node-3
osdmap e1: 0 osds: 0 up, 0 in
flags sortbitwise,require_jewel_osds
pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
64 creating

配置管理节点ceph-master

为什么每次查看ceph –s的只能在OSD节点上执行，而不能在Master节点上执行？
用ceph-deploy把配置文件和admin密钥拷贝到Master节点

1
# ceph-deploy admin ceph-master

 

确保你对ceph.client.admin.keyring有正确的操作权限

1
# chmod +r /etc/ceph/ceph.client.admin.keyring

 

检查集群的健康状况

1
2
3
4
5
6
7
8
9
10
11
# ceph -s
cluster 60597e53-ad29-44bd-8dcd-db6aeae6f580
health HEALTH_ERR
no osds
monmap e2: 3 mons at {ceph-node-1=172.16.0.11:6789/0,ceph-node-2=172.16.0.12:6789/0,ceph-node-3=172.16.0.13:6789/0}
election epoch 6, quorum 0,1,2 ceph-node-1,ceph-node-2,ceph-node-3
osdmap e1: 0 osds: 0 up, 0 in
flags sortbitwise,require_jewel_osds
pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
64 creating

 

对OSD节点配置

磁盘分区

将240G的SSD盘分出4个20G的分区用作 journal

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# fdisk /dev/sdf
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x9ec0a047.
Command (m for help): g
Building a new GPT disklabel (GUID: 31D328DD-E9A0-4306-9C99-8D42F7BA8008)
Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-503316446, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-503316446, default 503316446): +20G
Created partition 1
Command (m for help): n
Partition number (2-128, default 2):
First sector (41945088-503316446, default 41945088):
Last sector, +sectors or +size{K,M,G,T,P} (41945088-503316446, default 503316446): +20G
Created partition 2
Command (m for help): n
Partition number (3-128, default 3):
First sector (83888128-503316446, default 83888128):
Last sector, +sectors or +size{K,M,G,T,P} (83888128-503316446, default 503316446): +20G
Created partition 3
Command (m for help): n
Partition number (4-128, default 4):
First sector (125831168-503316446, default 125831168):
Last sector, +sectors or +size{K,M,G,T,P} (125831168-503316446, default 503316446): +20G
Created partition 4
Command (m for help): p
Disk /dev/sdf: 257.7 GB, 257698037760 bytes, 503316480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
# Start End Size Type Name
1 2048 41945087 20G Linux filesyste
2 41945088 83888127 20G Linux filesyste
3 83888128 125831167 20G Linux filesyste
4 125831168 167774207 20G Linux filesyste
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

 

查看分区

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 300G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 299G 0 part
├─cl-root 253:0 0 50G 0 lvm /
├─cl-swap 253:1 0 2G 0 lvm [SWAP]
└─cl-home 253:2 0 247G 0 lvm /home
sdb 8:16 0 2T 0 disk
sdc 8:32 0 2T 0 disk
sdd 8:48 0 2T 0 disk
sde 8:64 0 800G 0 disk
sdf 8:80 0 240G 0 disk
├─sdf1 8:81 0 20G 0 part
├─sdf2 8:82 0 20G 0 part
├─sdf3 8:83 0 20G 0 part
└─sdf4 8:84 0 20G 0 part
sr0 11:0 1 1024M 0 rom

修改 JOURNAL 分区权限

1
chown ceph:ceph /dev/sdf[1-4]

添加OSD

在ceph-master执行
先部署SATA

1
ceph-deploy osd prepare ceph-node-1:/dev/sdb:/dev/sdf1 ceph-node-1:/dev/sdc:/dev/sdf2 ceph-node-1:/dev/sdd:/dev/sdf3 ceph-node-2:/dev/sdb:/dev/sdf1 ceph-node-2:/dev/sdc:/dev/sdf2 ceph-node-2:/dev/sdd:/dev/sdf3 ceph-node-3:/dev/sdb:/dev/sdf1 ceph-node-3:/dev/sdc:/dev/sdf2 ceph-node-3:/dev/sdd:/dev/sdf3

 

再部署SSD

1
ceph-deploy osd prepare ceph-node-1:/dev/sde:/dev/sdf4 ceph-node-2:/dev/sde:/dev/sdf4 ceph-node-3:/dev/sde:/dev/sdf4

 

先部署SATA，再部署SSD，这样序号就会连续了…
查看集群状态

1
2
3
4
5
6
7
8
9
10
# ceph -s
cluster f9dba93b-c40f-4b31-8e6d-6ea830cad944
health HEALTH_OK
monmap e1: 3 mons at {ceph-node-1=172.16.0.11:6789/0,ceph-node-2=172.16.0.12:6789/0,ceph-node-3=172.16.0.13:6789/0}
election epoch 6, quorum 0,1,2 ceph-node-1,ceph-node-2,ceph-node-3
osdmap e56: 12 osds: 12 up, 12 in
flags sortbitwise,require_jewel_osds
pgmap v142: 128 pgs, 1 pools, 0 bytes data, 0 objects
413 MB used, 20821 GB / 20821 GB avail
128 active+clean

 

查看OSD的分布

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 20.33363 root default
-2 6.77788 host ceph-node-1
0 1.99899 osd.0 up 1.00000 1.00000
1 1.99899 osd.1 up 1.00000 1.00000
4 1.99899 osd.4 up 1.00000 1.00000
9 0.78090 osd.9 up 1.00000 1.00000
-3 6.77788 host ceph-node-2
2 1.99899 osd.2 up 1.00000 1.00000
3 1.99899 osd.3 up 1.00000 1.00000
5 1.99899 osd.5 up 1.00000 1.00000
10 0.78090 osd.10 up 1.00000 1.00000
-4 6.77788 host ceph-node-3
6 1.99899 osd.6 up 1.00000 1.00000
7 1.99899 osd.7 up 1.00000 1.00000
8 1.99899 osd.8 up 1.00000 1.00000
11 0.78090 osd.11 up 1.00000 1.00000

 

安装完毕

ceph安装完毕

基于CentOS 7.3 安装Ceph Jewel 10.2.9

标签：ase   tab   osd   ack   init   com   flags   guid   添加   

原文地址：https://www.cnblogs.com/wangmo/p/11302293.html
踩
(0)
赞
(0)
   
举报
评论 一句话评论（0）
登录后才能评论！
分享档案
更多>
2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)
周排行
mamicode.com排行更多图片
更多
友情链接
兰亭集智   国之画   百度统计   站长统计   阿里云   chrome插件   新版天听网
关于我们 - 联系我们 - 留言反馈
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码！