pve+ceph超融合（七）-刪除添加壞的硬盤

在使用過程中，當集群硬盤故障什么更換新的硬盤呢，下面我們來演示

查看ceph狀態
root@pve-1:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.43822 root default
-3 0.14607 host pve-1
0 hdd 0.04869 osd.0 up 1.00000 1.00000
3 hdd 0.04869 osd.3 up 1.00000 1.00000
7 hdd 0.04869 osd.7 up 1.00000 1.00000
-5 0.14607 host pve-2
2 hdd 0.04869 osd.2 up 1.00000 1.00000
4 hdd 0.04869 osd.4 up 1.00000 1.00000
6 hdd 0.04869 osd.6 up 1.00000 1.00000
-7 0.14607 host pve-3
1 hdd 0.04869 osd.1 up 1.00000 1.00000
5 hdd 0.04869 osd.5 up 1.00000 1.00000
8 hdd 0.04869 osd.8 up 1.00000 1.00000

都是正常的，等下我把一個硬盤去掉
ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.43822 root default
-3 0.14607 host pve-1
0 hdd 0.04869 osd.0 up 1.00000 1.00000
3 hdd 0.04869 osd.3 up 1.00000 1.00000
7 hdd 0.04869 osd.7 up 1.00000 1.00000
-5 0.14607 host pve-2
2 hdd 0.04869 osd.2 up 1.00000 1.00000
4 hdd 0.04869 osd.4 up 1.00000 1.00000
6 hdd 0.04869 osd.6 up 1.00000 1.00000
-7 0.14607 host pve-3
1 hdd 0.04869 osd.1 up 1.00000 1.00000
5 hdd 0.04869 osd.5 up 1.00000 1.00000
8 hdd 0.04869 osd.8 down 1.00000 1.00000

osd.8變成down了

然后我們模擬刪除硬盤和添加硬盤
ceph osd out osd.8
ceph auth del osd.8
ceph osd rm 8
root@pve-1:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.43822 root default
-3 0.14607 host pve-1
0 hdd 0.04869 osd.0 up 1.00000 1.00000
3 hdd 0.04869 osd.3 up 1.00000 1.00000
7 hdd 0.04869 osd.7 up 1.00000 1.00000
-5 0.14607 host pve-2
2 hdd 0.04869 osd.2 up 1.00000 1.00000
4 hdd 0.04869 osd.4 up 1.00000 1.00000
6 hdd 0.04869 osd.6 up 1.00000 1.00000
-7 0.14607 host pve-3
1 hdd 0.04869 osd.1 up 1.00000 1.00000
5 hdd 0.04869 osd.5 up 1.00000 1.00000
8 hdd 0.04869 osd.8 DNE 0

osd.8 的狀態是DNE

刪除故障節點的ceph磁盤，操作如下：
ceph osd crush rm osd.8
root@pve-1:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.38953 root default
-3 0.14607 host pve-1
0 hdd 0.04869 osd.0 up 1.00000 1.00000
3 hdd 0.04869 osd.3 up 1.00000 1.00000
7 hdd 0.04869 osd.7 up 1.00000 1.00000
-5 0.14607 host pve-2
2 hdd 0.04869 osd.2 up 1.00000 1.00000
4 hdd 0.04869 osd.4 up 1.00000 1.00000
6 hdd 0.04869 osd.6 up 1.00000 1.00000
-7 0.09738 host pve-3
1 hdd 0.04869 osd.1 up 1.00000 1.00000
5 hdd 0.04869 osd.5 up 1.00000 1.00000
osd.8已經找不到了
說明刪除成功

查看我們添加的硬盤信息
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 19.5G 0 part
├─pve-swap 253:0 0 2.4G 0 lvm [SWAP]
├─pve-root 253:1 0 4.8G 0 lvm /
├─pve-data_tmeta 253:2 0 1G 0 lvm
│ └─pve-data 253:4 0 8G 0 lvm
└─pve-data_tdata 253:3 0 8G 0 lvm
└─pve-data 253:4 0 8G 0 lvm
sdb 8:16 0 50G 0 disk
├─sdb1 8:17 0 100M 0 part /var/lib/ceph/osd/ceph-1
└─sdb2 8:18 0 49.9G 0 part
sdc 8:32 0 50G 0 disk
├─sdc1 8:33 0 100M 0 part /var/lib/ceph/osd/ceph-5
└─sdc2 8:34 0 49.9G 0 part
sdd 8:48 0 50G 0 disk
sr0 11:0 1 655.3M 0 rom
pveceph createosd /dev/sdd
root@pve-3:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.43822 root default
-3 0.14607 host pve-1
0 hdd 0.04869 osd.0 up 1.00000 1.00000
3 hdd 0.04869 osd.3 up 1.00000 1.00000
7 hdd 0.04869 osd.7 up 1.00000 1.00000
-5 0.14607 host pve-2
2 hdd 0.04869 osd.2 up 1.00000 1.00000
4 hdd 0.04869 osd.4 up 1.00000 1.00000
6 hdd 0.04869 osd.6 up 1.00000 1.00000
-7 0.14607 host pve-3
1 hdd 0.04869 osd.1 up 1.00000 1.00000
5 hdd 0.04869 osd.5 up 1.00000 1.00000
8 hdd 0.04869 osd.8 up 1.00000 1.00000

從ceph集群中刪除物理節點，操作如下：

ceph osd crush rm pve-3

從集群中刪除故障節點
登錄集群中任意正常節點，執行如下指令進行驅逐操作：
pvecm delnode pve-3
故障機恢復操作

最好全部干掉，重新安裝系統，并用新的ip地址，加入集群。