(93条消息) ceph-创建一个使用该rule-ssd规则的存储池_大飞哥2的博客-CSDN博客

crush class实验

luminous版本的ceph新增了一个功能crush class,这个功能又可以称为磁盘智能分组。因为这个功能就是根据磁盘类型自动的进行属性的关联,然后进行分类。无需手动修改crushmap,极大的减少了人为的操作。以前的操作有多麻烦可以看看:ceph crushmap

ceph中的每个osd设备都可以选择一个class类型与之关联,默认情况下,在创建osd的时候会自动识别设备类型,然后设置该设备为相应的类。通常有三种class类型:hdd,ssd,nvme。

由于当前实验环境下没有ssd和nvme设备,只好修改class标签,假装为有ssd设备,然后进行实验。

一,实验环境

  1. [root@node3 ~]# cat /etc/redhat-release
  2. CentOS Linux release 7.3.1611 (Core)
  3. [root@node3 ~]# ceph -v
  4. ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)

二,修改crush class:

1,查看当前集群布局:

  1. [root@node3 ~]# ceph osd tree
  2. ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
  3. -1 0.05878 root default
  4. -3 0.01959 host node1
  5. 0 hdd 0.00980 osd.0 up 1.00000 1.00000
  6. 3 hdd 0.00980 osd.3 up 1.00000 1.00000
  7. -5 0.01959 host node2
  8. 1 hdd 0.00980 osd.1 up 1.00000 1.00000
  9. 4 hdd 0.00980 osd.4 up 1.00000 1.00000
  10. -7 0.01959 host node3
  11. 2 hdd 0.00980 osd.2 up 1.00000 1.00000
  12. 5 hdd 0.00980 osd.5 up 1.00000 1.00000

可以看到只有第二列为CLASS,只有hdd类型。
通过查看crush class,确实只有hdd类型

  1. [root@node3 ~]# ceph osd crush class ls
  2. [
  3. "hdd"
  4. ]

2,删除osd.0,osd.1,osd.2的class:

  1. [root@node3 ~]# for i in 0 1 2;do ceph osd crush rm-device-class osd.$i;done
  2. done removing class of osd(s): 0
  3. done removing class of osd(s): 1
  4. done removing class of osd(s): 2

再次通过命令ceph osd tree查看osd.0,osd.1,osd.2的class

  1. [root@node3 ~]# ceph osd tree
  2. ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
  3. -1 0.05878 root default
  4. -3 0.01959 host node1
  5. 0 0.00980 osd.0 up 1.00000 1.00000
  6. 3 hdd 0.00980 osd.3 up 1.00000 1.00000
  7. -5 0.01959 host node2
  8. 1 0.00980 osd.1 up 1.00000 1.00000
  9. 4 hdd 0.00980 osd.4 up 1.00000 1.00000
  10. -7 0.01959 host node3
  11. 2 0.00980 osd.2 up 1.00000 1.00000
  12. 5 hdd 0.00980 osd.5 up 1.00000 1.00000

可以发现osd.0,osd.1,osd.2的class为空

3,设置osd.0,osd.1,osd.2的class为ssd:

  1. [root@node3 ~]# for i in 0 1 2;do ceph osd crush set-device-class ssd osd.$i;done
  2. set osd(s) 0 to class 'ssd'
  3. set osd(s) 1 to class 'ssd'
  4. set osd(s) 2 to class 'ssd'

再次通过命令ceph osd tree查看osd.0,osd.1,osd.2的class

  1. [root@node3 ~]# ceph osd tree
  2. ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
  3. -1 0.05878 root default
  4. -3 0.01959 host node1
  5. 3 hdd 0.00980 osd.3 up 1.00000 1.00000
  6. 0 ssd 0.00980 osd.0 up 1.00000 1.00000
  7. -5 0.01959 host node2
  8. 4 hdd 0.00980 osd.4 up 1.00000 1.00000
  9. 1 ssd 0.00980 osd.1 up 1.00000 1.00000
  10. -7 0.01959 host node3
  11. 5 hdd 0.00980 osd.5 up 1.00000 1.00000
  12. 2 ssd 0.00980 osd.2 up 1.00000 1.00000

可以看到osd.0,osd.1,osd.2的class变为ssd
再查看一下crush class:

  1. [root@node3 ~]# ceph osd crush class ls
  2. [
  3. "hdd",
  4. "ssd"
  5. ]

可以看到class中多出了一个名为ssd的class

4,创建一个优先使用ssd设备的crush rule:

创建了一个rule的名字为:rule-ssd,在root名为default下的rule

[root@node3 ~]# ceph osd crush rule create-replicated rule-ssd default  host ssd 

查看集群的rule:

  1. [root@node3 ~]# ceph osd crush rule ls
  2. replicated_rule
  3. rule-ssd

可以看到多出了一个名为rule-ssd的rule
通过下面的命令下载集群crushmap查看有哪些变化:

  1. [root@node3 ~]# ceph osd getcrushmap -o crushmap
  2. 20
  3. [root@node3 ~]# crushtool -d crushmap -o crushmap
  4. [root@node3 ~]# cat crushmap
  5. # begin crush map
  6. tunable choose_local_tries 0
  7. tunable choose_local_fallback_tries 0
  8. tunable choose_total_tries 50
  9. tunable chooseleaf_descend_once 1
  10. tunable chooseleaf_vary_r 1
  11. tunable chooseleaf_stable 1
  12. tunable straw_calc_version 1
  13. tunable allowed_bucket_algs 54
  14. # devices
  15. device 0 osd.0 class ssd
  16. device 1 osd.1 class ssd
  17. device 2 osd.2 class ssd
  18. device 3 osd.3 class hdd
  19. device 4 osd.4 class hdd
  20. device 5 osd.5 class hdd
  21. # types
  22. type 0 osd
  23. type 1 host
  24. type 2 chassis
  25. type 3 rack
  26. type 4 row
  27. type 5 pdu
  28. type 6 pod
  29. type 7 room
  30. type 8 datacenter
  31. type 9 region
  32. type 10 root
  33. # buckets
  34. host node1 {
  35. id -3 # do not change unnecessarily
  36. id -4 class hdd # do not change unnecessarily
  37. id -9 class ssd # do not change unnecessarily
  38. # weight 0.020
  39. alg straw2
  40. hash 0 # rjenkins1
  41. item osd.0 weight 0.010
  42. item osd.3 weight 0.010
  43. }
  44. host node2 {
  45. id -5 # do not change unnecessarily
  46. id -6 class hdd # do not change unnecessarily
  47. id -10 class ssd # do not change unnecessarily
  48. # weight 0.020
  49. alg straw2
  50. hash 0 # rjenkins1
  51. item osd.1 weight 0.010
  52. item osd.4 weight 0.010
  53. }
  54. host node3 {
  55. id -7 # do not change unnecessarily
  56. id -8 class hdd # do not change unnecessarily
  57. id -11 class ssd # do not change unnecessarily
  58. # weight 0.020
  59. alg straw2
  60. hash 0 # rjenkins1
  61. item osd.2 weight 0.010
  62. item osd.5 weight 0.010
  63. }
  64. root default {
  65. id -1 # do not change unnecessarily
  66. id -2 class hdd # do not change unnecessarily
  67. id -12 class ssd # do not change unnecessarily
  68. # weight 0.059
  69. alg straw2
  70. hash 0 # rjenkins1
  71. item node1 weight 0.020
  72. item node2 weight 0.020
  73. item node3 weight 0.020
  74. }
  75. # rules
  76. rule replicated_rule {
  77. id 0
  78. type replicated
  79. min_size 1
  80. max_size 10
  81. step take default
  82. step chooseleaf firstn 0 type host
  83. step emit
  84. }
  85. rule rule-ssd {
  86. id 1
  87. type replicated
  88. min_size 1
  89. max_size 10
  90. step take default class ssd
  91. step chooseleaf firstn 0 type host
  92. step emit
  93. }
  94. # end crush map

可以看到在root default下多了一行: id -12 class ssd。在rules下,多了一个rule rule-ssd其id为1

5,创建一个使用该rule-ssd规则的存储池:

  1. [root@node3 ~]# ceph osd pool create ssdpool 64 64 rule-ssd
  2. pool 'ssdpool' created

查看ssdpool的信息可以看到使用的crush_rule 为1,也就是rule-ssd

  1. [root@node3 ~]# ceph osd pool ls detail
  2. pool 1 'ssdpool' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 39 flags hashpspool stripe_width 0

6,创建对象测试ssdpool:

创建一个对象test并放到ssdpool中:

  1. [root@node3 ~]# rados -p ssdpool ls
  2. [root@node3 ~]# echo "hahah" >test.txt
  3. [root@node3 ~]# rados -p ssdpool put test test.txt
  4. [root@node3 ~]# rados -p ssdpool ls
  5. test

查看该对象的osd组:

  1. [root@node3 ~]# ceph osd map ssdpool test
  2. osdmap e46 pool 'ssdpool' (1) object 'test' -> pg 1.40e8aab5 (1.35) -> up ([1,2,0], p1) acting ([1,2,0], p1)

可以看到该对象的osd组使用的都是ssd磁盘,至此验证成功。可以看出crush class相当于一个辨别磁盘类型的标签。

 

作者:http://www.cnblogs.com/sisimi/p/7804138.html

其他参考:https://blog-fromsomedude.rhcloud.com/2017/05/16/Luminous-series-CRUSH-devices-class/

三,参考文献:

  1. ceph luminous 新功能之磁盘智能分组
  2. CRUSH MAPS
  3. ceph crushmap

发表评论

邮箱地址不会被公开。 必填项已用*标注