linux nfs挂载 测速,CentOS下使用fio测试挂载了CEPH rbd磁盘的读写性能|4k随机/顺序读写...
背景简介
继续上文的ceph性能调整，有兴趣的可以看看：
使用rbd bench/bench-write自带测速工具和fio对ceph image进行性能测试|测速跑分
文章目录[隐藏] 前言 集群配置 开始测试 前言 最近在折腾CEPH集群，测试环境是4台ceph物理节点，每台8~12个SATA/SAS机械硬盘，共43个OSD。性能远低于预期，目前正在调整配置中…… 正好实测了一下性能，这里分享出来大家参考。对CEPH Pool的性能测试可以看前一篇： 集群配置 4 CEPH节点：SH-1001 SH-1003 SH-1004 SH-1005 2Manager：SH-1001 SH-1005 3Monitor：SH-1004 SH-2001 SH-2003 开始测试 内网测速 root@SH-1005:~# iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.1.0.1, port 42784 [ 5] local 10.1.0.5 port 5201 connected to 10.1.0.1 port 42786 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 […]
使用rados自带测速工具对ceph pool进行性能测试|测速跑分
文章目录[隐藏] 前言 集群配置 开始测试 前言 最近在折腾CEPH集群，测试环境是4台ceph物理节点，每台8~12个SATA/SAS机械硬盘，共43个OSD。性能远低于预期，目前正在调整配置中…… 正好实测了一下性能，这里分享出来大家参考。简要信息通过ceph -s即可获取： 集群配置 4 CEPH节点：SH-1001 SH-1003 SH-1004 SH-1005 2Manager：SH-1001 SH-1005 3Monitor：SH-1004 SH-2001 SH-2003 开始测试 内网测速 root@SH-1005:~# iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.1.0.1, port 42784 [ 5] local 10.1.0.5 port 5201 connected to 10.1.0.1 port 42786 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 […]
测试环境的Ceph集群配置同前文，所以就不赘述了。将上述的Ceph pool中新建raw格式的ceph rbd，并挂载在虚拟机的/vz下，qemu cache设置为writeback
[root@hn03 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 126G 0 126G 0% /dev
tmpfs 126G 12K 126G 1% /dev/shm
tmpfs 126G 1008K 126G 1% /run
tmpfs 126G 0 126G 0% /sys/fs/cgroup
/dev/sda4 165G 5.7G 151G 4% /
tmpfs 126G 20K 126G 1% /tmp
/dev/sda2 976M 170M 739M 19% /boot
/dev/mapper/vhs_jelastic--host--3-vz 8.0T 17G 7.6T 1% /vz
/dev/ploop13141p1 9.8G 1.3G 8.0G 14% /vz/pfcache
10.1.3.2:/home/nfs/backup/hn03 33T 24T 7.3T 77% /vz/backups
/dev/ploop48143p1 241G 548M 230G 1% /vz/root/2629
tmpfs 26G 0 26G 0% /run/user/0
开始测试
4K随机写性能
fio -filename=test_4kwrite -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=10000M -numjobs=100 -runtime=60 -group_reporting -name=test4kwrite
[root@hn03 ~]# fio -filename=/vz/test_4kwrite -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=100M -numjobs=10 -runtime=60 -group_reporting -name=test4kwrite
test4kwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.14
Starting 10 threads
test4kwrite: Laying out IO file (1 file / 100MiB)
Jobs: 10 (f=10): [w(10)][100.0%][w=4336KiB/s][w=1084 IOPS][eta 00m:00s]
test4kwrite: (groupid=0, jobs=10): err= 0: pid=292932: Sun Dec 13 16:10:38 2020
write: IOPS=2454, BW=9820KiB/s (10.1MB/s)(577MiB/60196msec)
clat (usec): min=141, max=1677.1k, avg=4061.46, stdev=52237.91
lat (usec): min=142, max=1677.1k, avg=4062.74, stdev=52237.96
clat percentiles (usec):
| 1.00th=[ 157], 5.00th=[ 161], 10.00th=[ 163],
| 20.00th=[ 167], 30.00th=[ 174], 40.00th=[ 180],
| 50.00th=[ 184], 60.00th=[ 190], 70.00th=[ 200],
| 80.00th=[ 215], 90.00th=[ 258], 95.00th=[ 10945],
| 99.00th=[ 53216], 99.50th=[ 68682], 99.90th=[1216349],
| 99.95th=[1350566], 99.99th=[1619002]
bw ( KiB/s): min= 159, max=48638, per=100.00%, avg=14139.18, stdev=1074.69, samples=836
iops : min= 39, max=12158, avg=3534.29, stdev=268.68, samples=836
lat (usec) : 250=89.14%, 500=4.80%, 750=0.06%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.10%, 10=0.72%, 20=1.17%, 50=2.83%
lat (msec) : 100=0.88%, 250=0.09%, 500=0.01%, 1000=0.02%, 2000=0.13%
cpu : usr=0.27%, sys=2.23%, ctx=295619, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,147774,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=9820KiB/s (10.1MB/s), 9820KiB/s-9820KiB/s (10.1MB/s-10.1MB/s), io=577MiB (605MB), run=60196-60196msec
Disk stats (read/write):
dm-0: ios=1/148479, merge=0/0, ticks=74/176011, in_queue=218493, util=96.85%, aggrios=1/147862, aggrmerge=0/636, aggrticks=74/158079, aggrin_queue=159152, aggrutil=96.39%
sdb: ios=1/147862, merge=0/636, ticks=74/158079, in_queue=159152, util=96.39%
4k随机读
[root@hn03 ~]# fio -filename=/vz/test_4kread -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=4k -size=100M -numjobs=10 -runtime=60 -group_reporting -name=test4kread
test4kread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.14
Starting 10 threads
Jobs: 2 (f=2): [_(1),r(1),_(1),r(1),_(6)][100.0%][r=42.4MiB/s][r=10.8k IOPS][eta 00m:00s]
test4kread: (groupid=0, jobs=10): err= 0: pid=303812: Sun Dec 13 16:17:48 2020
read: IOPS=7743, BW=30.2MiB/s (31.7MB/s)(1000MiB/33062msec)
clat (usec): min=6, max=465810, avg=1267.16, stdev=6697.91
lat (usec): min=7, max=465811, avg=1268.53, stdev=6697.91
clat percentiles (usec):
| 1.00th=[ 145], 5.00th=[ 167], 10.00th=[ 186], 20.00th=[ 239],
| 30.00th=[ 478], 40.00th=[ 709], 50.00th=[ 766], 60.00th=[ 824],
| 70.00th=[ 881], 80.00th=[ 955], 90.00th=[ 1090], 95.00th=[ 1270],
| 99.00th=[ 20579], 99.50th=[ 37487], 99.90th=[ 74974], 99.95th=[119014],
| 99.99th=[283116]
bw ( KiB/s): min= 552, max=62816, per=100.00%, avg=31085.23, stdev=2034.02, samples=648
iops : min= 138, max=15704, avg=7770.71, stdev=508.54, samples=648
lat (usec) : 10=0.04%, 20=0.05%, 50=0.05%, 100=0.02%, 250=21.29%
lat (usec) : 500=8.64%, 750=16.49%, 1000=37.87%
lat (msec) : 2=13.23%, 4=0.26%, 10=0.57%, 20=0.47%, 50=0.79%
lat (msec) : 100=0.16%, 250=0.05%, 500=0.01%
cpu : usr=1.08%, sys=6.70%, ctx=256844, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=256000,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=30.2MiB/s (31.7MB/s), 30.2MiB/s-30.2MiB/s (31.7MB/s-31.7MB/s), io=1000MiB (1049MB), run=33062-33062msec
Disk stats (read/write):
dm-0: ios=255350/0, merge=0/0, ticks=305061/0, in_queue=306628, util=99.92%, aggrios=255590/0, aggrmerge=0/0, aggrticks=305374/0, aggrin_queue=305203, aggrutil=99.72%
sdb: ios=255590/0, merge=0/0, ticks=305374/0, in_queue=305203, util=99.72%
顺序写
[root@hn03 ~]# fio -filename=/vz/test_seqwrite -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1G -size=10G -numjobs=10 -runtime=60 -group_reporting -name=testseqwrite
testseqwrite: (g=0): rw=write, bs=(R) 1024MiB-1024MiB, (W) 1024MiB-1024MiB, (T) 1024MiB-1024MiB, ioengine=psync, iodepth=1
...
fio-3.14
Starting 10 threads
testseqwrite: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(9),W(1)][13.6%][eta 12m:23s]
testseqwrite: (groupid=0, jobs=10): err= 0: pid=307042: Sun Dec 13 16:21:50 2020
write: IOPS=0, BW=159MiB/s (167MB/s)(18.0GiB/115971msec)
clat (msec): min=20002, max=64182, avg=50736.31, stdev=14795.11
lat (msec): min=20103, max=64581, avg=50949.70, stdev=14763.00
clat percentiles (msec):
| 1.00th=[17113], 5.00th=[17113], 10.00th=[17113], 20.00th=[17113],
| 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
| 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
| 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
| 99.99th=[17113]
bw ( MiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=8
iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=8
lat (msec) : >=2000=100.00%
cpu : usr=0.42%, sys=0.43%, ctx=380, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,18,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=18.0GiB (19.3GB), run=115971-115971msec
Disk stats (read/write):
dm-0: ios=0/37098, merge=0/0, ticks=0/17552840, in_queue=17710730, util=99.92%, aggrios=0/36944, aggrmerge=0/179, aggrticks=0/17266133, aggrin_queue=17265866, aggrutil=99.22%
sdb: ios=0/36944, merge=0/179, ticks=0/17266133, in_queue=17265866, util=99.22%
顺序读
[root@hn03 ~]# fio -filename=/vz/test_seqread -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1G -size=10G -numjobs=10 -runtime=60 -group_reporting -name=testseqread
testseqread: (g=0): rw=read, bs=(R) 1024MiB-1024MiB, (W) 1024MiB-1024MiB, (T) 1024MiB-1024MiB, ioengine=psync, iodepth=1
...
fio-3.14
Starting 10 threads
testseqread: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(6),R(1),_(3)][60.0%][eta 00m:46s]
testseqread: (groupid=0, jobs=10): err= 0: pid=319596: Sun Dec 13 16:30:05 2020
read: IOPS=1, BW=1043MiB/s (1094MB/s)(70.0GiB/68729msec)
clat (msec): min=967, max=10553, avg=8866.24, stdev=2072.27
lat (msec): min=967, max=10553, avg=8866.24, stdev=2072.27
clat percentiles (msec):
| 1.00th=[ 969], 5.00th=[ 2869], 10.00th=[ 5403], 20.00th=[ 9060],
| 30.00th=[ 9329], 40.00th=[ 9329], 50.00th=[ 9463], 60.00th=[ 9597],
| 70.00th=[ 9731], 80.00th=[ 9866], 90.00th=[10134], 95.00th=[10134],
| 99.00th=[10537], 99.50th=[10537], 99.90th=[10537], 99.95th=[10537],
| 99.99th=[10537]
bw ( MiB/s): min=20447, max=20480, per=100.00%, avg=20472.62, stdev= 2.11, samples=60
iops : min= 15, max= 20, avg=18.69, stdev= 0.30, samples=60
lat (msec) : 1000=1.43%, 2000=1.43%, >=2000=97.14%
cpu : usr=0.00%, sys=3.04%, ctx=4613, majf=0, minf=6158
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=70,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=1043MiB/s (1094MB/s), 1043MiB/s-1043MiB/s (1094MB/s-1094MB/s), io=70.0GiB (75.2GB), run=68729-68729msec
Disk stats (read/write):
dm-0: ios=143360/4, merge=0/0, ticks=9562103/248, in_queue=9565573, util=99.98%, aggrios=143360/4, aggrmerge=0/1, aggrticks=9515022/245, aggrin_queue=9514757, aggrutil=99.95%
sdb: ios=143360/4, merge=0/1, ticks=9515022/245, in_queue=9514757, util=99.95%
This article is under CC BY-NC-SA 4.0 license.
Please quote the original link：https://www.liujason.com/article/1121.html
