DRBD (Distributed Replicated Block Device) 実測ベンチマーク

測定環境

  • 測定日時:2020年 10月 8日 木曜日 18:22:18 JST
  • ハードウェア:Intel Core i7-9700K 3.60GHz
  • ホストOS:Windows10 Home
  • 仮想マシン:VirtualBox 6.1.14 r140239 (Qt5.6.2)
  • クライアント:CentOS Linux release 8.2.2004 (Core)
  • DRBD:kmod-drbd90.x86_64 9.0.23-1.el8_2.elrepo
  • primary , secondary 2台構成
  • 内部ネットワーク速度:3.71 Gbit/s (別途スループット測定)

定義

cat > /etc/drbd.d/r1.res

resource r1 {
    floating ipv4 192.168.32.1:7788 {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/cl/drdb-1;
            meta-disk    internal;
        }
    }
    floating ipv4 192.168.32.2:7788 {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/cl/drdb-1;
            meta-disk    internal;
        }
    }
    net {
        protocol           C;
        max-buffers      8000;
        max-epoch-size   8000;
    }
    disk {
        al-extents       3389;
        disk-barrier      no;
        disk-flushes      no;
    }
}

結果

ランダムライト LOCAL xfs

[root@centos1 /]# fio -filename=/tmp/benchmark -direct=1 -rw=randwrite -bs=4k -size=512M -numjobs=64 -runtime=10 -group_reporting -name=file1
file1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 64 processes

Jobs: 64 (f=64): [w(64)][30.0%][r=0KiB/s,w=155MiB/s][r=0,w=39.7k IOPS][eta 00m:0
Jobs: 64 (f=64): [w(64)][40.0%][r=0KiB/s,w=160MiB/s][r=0,w=40.0k IOPS][eta 00m:0
Jobs: 64 (f=64): [w(64)][50.0%][r=0KiB/s,w=151MiB/s][r=0,w=38.6k IOPS][eta 00m:0
Jobs: 64 (f=64): [w(64)][70.0%][r=0KiB/s,w=95.0MiB/s][r=0,w=24.3k IOPS][eta 00m:
Jobs: 64 (f=64): [w(64)][80.0%][r=0KiB/s,w=155MiB/s][r=0,w=39.6k IOPS][eta 00m:0
Jobs: 64 (f=64): [w(64)][90.0%][r=0KiB/s,w=152MiB/s][r=0,w=38.0k IOPS][eta 00m:0
Jobs: 64 (f=64): [w(64)][100.0%][r=0KiB/s,w=154MiB/s][r=0,w=39.4k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=64): err= 0: pid=5680: Thu Oct  8 18:10:33 2020
  write: IOPS=37.5k, BW=147MiB/s (154MB/s)(1468MiB/10007msec)
    clat (usec): min=40, max=532953, avg=1699.64, stdev=11235.49
     lat (usec): min=41, max=532953, avg=1699.87, stdev=11235.51
    clat percentiles (usec):
     |  1.00th=[    69],  5.00th=[    91], 10.00th=[   139], 20.00th=[   269],
     | 30.00th=[   379], 40.00th=[   482], 50.00th=[   594], 60.00th=[   742],
     | 70.00th=[  1004], 80.00th=[  1500], 90.00th=[  3032], 95.00th=[  6128],
     | 99.00th=[ 16712], 99.50th=[ 19268], 99.90th=[ 26608], 99.95th=[291505],
     | 99.99th=[513803]
   bw (  KiB/s): min=    7, max=10108, per=1.35%, avg=2028.47, stdev=1197.32, samples=1266
   iops        : min=    1, max= 2527, avg=506.90, stdev=299.37, samples=1266
  lat (usec)   : 50=0.02%, 100=6.23%, 250=12.20%, 500=23.36%, 750=18.48%
  lat (usec)   : 1000=9.57%
  lat (msec)   : 2=15.17%, 4=7.27%, 10=4.77%, 20=2.51%, 50=0.36%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%
  cpu          : usr=0.05%, sys=2.03%, ctx=382202, majf=0, minf=545
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,375694,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=1468MiB (1539MB), run=10007-10007msec

Disk stats (read/write):
    dm-0: ios=0/370306, merge=0/0, ticks=0/380993, in_queue=380993, util=79.17%, aggrios=0/375660, aggrmerge=0/34, aggrticks=0/378876, aggrin_queue=281126, aggrutil=79.11%
  sda: ios=0/375660, merge=0/34, ticks=0/378876, in_queue=281126, util=79.11%

シーケンシャルライト LOCAL xfs

[root@centos1 /]# fio -filename=/tmp/benchmark -direct=1 -rw=write -bs=4k -size=512M -numjobs=64 -runtime=10 -group_reporting -name=file1
file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 64 processes
file1: Laying out IO file (1 file / 512MiB)

Jobs: 64 (f=64): [W(64)][30.0%][r=0KiB/s,w=157MiB/s][r=0,w=40.2k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][50.0%][r=0KiB/s,w=145MiB/s][r=0,w=37.0k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][54.5%][r=0KiB/s,w=150MiB/s][r=0,w=38.3k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][63.6%][r=0KiB/s,w=148MiB/s][r=0,w=37.0k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][72.7%][r=0KiB/s,w=144MiB/s][r=0,w=36.9k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][81.8%][r=0KiB/s,w=147MiB/s][r=0,w=37.5k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][90.9%][r=0KiB/s,w=146MiB/s][r=0,w=37.5k IOPS][eta 00m:0
Jobs: 63 (f=63): [W(11),E(1),W(52)][100.0%][r=0KiB/s,w=137MiB/s][r=0,w=35.0k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=64): err= 0: pid=5520: Thu Oct  8 18:09:13 2020
  write: IOPS=37.8k, BW=148MiB/s (155MB/s)(1550MiB/10491msec)
    clat (usec): min=43, max=515606, avg=1634.38, stdev=7420.51
     lat (usec): min=43, max=515606, avg=1634.61, stdev=7420.60
    clat percentiles (usec):
     |  1.00th=[    84],  5.00th=[   117], 10.00th=[   221], 20.00th=[   379],
     | 30.00th=[   498], 40.00th=[   594], 50.00th=[   676], 60.00th=[   766],
     | 70.00th=[   873], 80.00th=[  1074], 90.00th=[  2737], 95.00th=[  8225],
     | 99.00th=[ 15139], 99.50th=[ 20055], 99.90th=[ 31065], 99.95th=[ 39584],
     | 99.99th=[505414]
   bw (  KiB/s): min=   44, max= 5360, per=1.63%, avg=2467.58, stdev=831.20, samples=1278
   iops        : min=   11, max= 1340, avg=616.72, stdev=207.81, samples=1278
  lat (usec)   : 50=0.01%, 100=4.06%, 250=7.47%, 500=18.74%, 750=28.35%
  lat (usec)   : 1000=18.57%
  lat (msec)   : 2=11.05%, 4=3.44%, 10=4.04%, 20=3.75%, 50=0.47%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%
  cpu          : usr=0.02%, sys=2.36%, ctx=396763, majf=0, minf=809
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,396742,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1550MiB (1625MB), run=10491-10491msec

Disk stats (read/write):
    dm-0: ios=0/396742, merge=0/0, ticks=0/358015, in_queue=358015, util=79.06%, aggrios=0/395179, aggrmerge=0/1537, aggrticks=0/361835, aggrin_queue=288639, aggrutil=75.40%
  sda: ios=0/395179, merge=0/1537, ticks=0/361835, in_queue=288639, util=75.40%

シーケンシャルライト DRBD PROTOCOL A ext4

[root@centos1 /]# fio -filename=/share/benchmark2 -direct=1 -rw=write -bs=4k -size=512M -numjobs=64 -runtime=10 -group_reporting -name=file1
file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 64 processes

Jobs: 64 (f=64): [W(64)][27.3%][r=0KiB/s,w=14.7MiB/s][r=0,w=3765 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][36.4%][r=0KiB/s,w=21.1MiB/s][r=0,w=5404 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][45.5%][r=0KiB/s,w=8677KiB/s][r=0,w=2169 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][50.0%][r=0KiB/s,w=8705KiB/s][r=0,w=2176 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][60.0%][r=0KiB/s,w=28.0MiB/s][r=0,w=7416 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][70.0%][r=0KiB/s,w=11.5MiB/s][r=0,w=2947 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][81.8%][r=0KiB/s,w=4296KiB/s][r=0,w=1074 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][90.9%][r=0KiB/s,w=6838KiB/s][r=0,w=1709 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][100.0%][r=0KiB/s,w=7290KiB/s][r=0,w=1822 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=64): err= 0: pid=5312: Thu Oct  8 18:00:35 2020
  write: IOPS=3194, BW=12.5MiB/s (13.1MB/s)(125MiB/10021msec)
    clat (usec): min=54, max=567158, avg=20013.42, stdev=57528.75
     lat (usec): min=54, max=567158, avg=20013.66, stdev=57528.77
    clat percentiles (usec):
     |  1.00th=[    71],  5.00th=[    78], 10.00th=[    83], 20.00th=[    97],
     | 30.00th=[   124], 40.00th=[   153], 50.00th=[ 14222], 60.00th=[ 15533],
     | 70.00th=[ 16319], 80.00th=[ 22676], 90.00th=[ 39060], 95.00th=[ 70779],
     | 99.00th=[337642], 99.50th=[530580], 99.90th=[557843], 99.95th=[557843],
     | 99.99th=[566232]
   bw (  KiB/s): min=    8, max=  568, per=1.61%, avg=205.22, stdev=146.58, samples=1233
   iops        : min=    2, max=  142, avg=51.07, stdev=36.75, samples=1233
  lat (usec)   : 100=21.03%, 250=26.99%, 500=0.52%, 750=0.05%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=0.02%, 10=0.04%, 20=29.45%, 50=14.27%
  lat (msec)   : 100=5.57%, 250=0.82%, 500=0.20%, 750=1.00%
  cpu          : usr=0.01%, sys=0.76%, ctx=64572, majf=0, minf=761
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32014,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=12.5MiB/s (13.1MB/s), 12.5MiB/s-12.5MiB/s (13.1MB/s-13.1MB/s), io=125MiB (131MB), run=10021-10021msec

Disk stats (read/write):
    drbd0: ios=1/31924, merge=0/0, ticks=0/2971, in_queue=2971, util=46.29%, aggrios=1/32026, aggrmerge=0/0, aggrticks=0/2398, aggrin_queue=2398, aggrutil=45.70%
    dm-2: ios=1/32026, merge=0/0, ticks=0/2398, in_queue=2398, util=45.70%, aggrios=1/32021, aggrmerge=0/5, aggrticks=0/2458, aggrin_queue=59, aggrutil=45.70%
  sdb: ios=1/32021, merge=0/5, ticks=0/2458, in_queue=59, util=45.70%

シーケンシャルライト DRBD PROTOCOL C ext4

[root@centos1 /]# fio -filename=/share/benchmark2 -direct=1 -rw=write -bs=4k -size=1G -numjobs=64 -runtime=10 -group_reporting -name=file1
file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 64 processes
file1: Laying out IO file (1 file / 1024MiB)

Jobs: 64 (f=64): [W(64)][30.0%][r=0KiB/s,w=10.7MiB/s][r=0,w=2726 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][40.0%][r=0KiB/s,w=11.0MiB/s][r=0,w=2825 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][54.5%][r=0KiB/s,w=6895KiB/s][r=0,w=1723 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][63.6%][r=0KiB/s,w=6174KiB/s][r=0,w=1543 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][72.7%][r=0KiB/s,w=6140KiB/s][r=0,w=1535 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][81.8%][r=0KiB/s,w=4412KiB/s][r=0,w=1103 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][90.9%][r=0KiB/s,w=5494KiB/s][r=0,w=1373 IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][100.0%][r=0KiB/s,w=5499KiB/s][r=0,w=1374 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=64): err= 0: pid=5133: Thu Oct  8 17:52:02 2020
  write: IOPS=1809, BW=7238KiB/s (7411kB/s)(70.9MiB/10028msec)
    clat (usec): min=238, max=167893, avg=35309.35, stdev=40145.12
     lat (usec): min=238, max=167893, avg=35309.55, stdev=40145.12
    clat percentiles (usec):
     |  1.00th=[   277],  5.00th=[   297], 10.00th=[   310], 20.00th=[   326],
     | 30.00th=[   347], 40.00th=[   400], 50.00th=[ 39060], 60.00th=[ 41157],
     | 70.00th=[ 50594], 80.00th=[ 78119], 90.00th=[ 95945], 95.00th=[108528],
     | 99.00th=[139461], 99.50th=[147850], 99.90th=[164627], 99.95th=[166724],
     | 99.99th=[166724]
   bw (  KiB/s): min=   39, max=  208, per=1.56%, avg=112.94, stdev=45.49, samples=1278
   iops        : min=    9, max=   52, avg=28.15, stdev=11.40, samples=1278
  lat (usec)   : 250=0.04%, 500=45.24%, 750=3.55%, 1000=0.14%
  lat (msec)   : 2=0.15%, 4=0.06%, 10=0.09%, 20=0.05%, 50=20.03%
  lat (msec)   : 100=22.97%, 250=7.68%
  cpu          : usr=0.01%, sys=0.62%, ctx=36442, majf=0, minf=718
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,18145,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=7238KiB/s (7411kB/s), 7238KiB/s-7238KiB/s (7411kB/s-7411kB/s), io=70.9MiB (74.3MB), run=10028-10028msec

Disk stats (read/write):
    drbd0: ios=0/17875, merge=0/0, ticks=0/6082, in_queue=6082, util=71.37%, aggrios=0/18190, aggrmerge=0/0, aggrticks=0/1589, aggrin_queue=1589, aggrutil=70.21%
    dm-2: ios=0/18190, merge=0/0, ticks=0/1589, in_queue=1589, util=70.21%, aggrios=0/18155, aggrmerge=0/35, aggrticks=0/1441, aggrin_queue=54, aggrutil=70.21%
  sdb: ios=0/18155, merge=0/35, ticks=0/1441, in_queue=54, util=70.21%
[root@centos1 /]#

シーケンシャルライト DRBD PROTOCOL C xfs

[root@centos1 share]# fio -filename=/share/benchmark2 -direct=1 -rw=write -bs=4k -size=512M -numjobs=64 -runtime=10 -group_reporting -name=file1
file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 64 processes
file1: Laying out IO file (1 file / 512MiB)

Jobs: 64 (f=64): [W(64)][30.0%][r=0KiB/s,w=63.6MiB/s][r=0,w=16.3k IOPS][eta 00m:
Jobs: 64 (f=64): [W(64)][40.0%][r=0KiB/s,w=64.0MiB/s][r=0,w=16.4k IOPS][eta 00m:
Jobs: 64 (f=64): [W(64)][50.0%][r=0KiB/s,w=65.9MiB/s][r=0,w=16.9k IOPS][eta 00m:
Jobs: 64 (f=64): [W(64)][60.0%][r=0KiB/s,w=63.6MiB/s][r=0,w=16.3k IOPS][eta 00m:
Jobs: 64 (f=64): [W(64)][70.0%][r=0KiB/s,w=63.1MiB/s][r=0,w=16.1k IOPS][eta 00m:
Jobs: 64 (f=64): [W(64)][80.0%][r=0KiB/s,w=61.4MiB/s][r=0,w=15.7k IOPS][eta 00m:
Jobs: 64 (f=64): [W(64)][90.0%][r=0KiB/s,w=65.3MiB/s][r=0,w=16.7k IOPS][eta 00m:
Jobs: 64 (f=64): [W(64)][100.0%][r=0KiB/s,w=63.2MiB/s][r=0,w=16.2k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=64): err= 0: pid=69958: Fri Oct  9 16:10:15 2020
  write: IOPS=16.2k, BW=63.4MiB/s (66.5MB/s)(635MiB/10007msec)
    clat (usec): min=340, max=107156, avg=3936.31, stdev=1842.69
     lat (usec): min=340, max=107156, avg=3936.63, stdev=1842.68
    clat percentiles (usec):
     |  1.00th=[ 1991],  5.00th=[ 2442], 10.00th=[ 2704], 20.00th=[ 3032],
     | 30.00th=[ 3294], 40.00th=[ 3556], 50.00th=[ 3785], 60.00th=[ 4047],
     | 70.00th=[ 4293], 80.00th=[ 4555], 90.00th=[ 4948], 95.00th=[ 5473],
     | 99.00th=[ 8586], 99.50th=[13042], 99.90th=[27132], 99.95th=[31589],
     | 99.99th=[67634]
   bw (  KiB/s): min=  712, max= 1440, per=1.56%, avg=1015.03, stdev=52.44, samples=1271
   iops        : min=  178, max=  360, avg=253.70, stdev=13.13, samples=1271
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.03%
  lat (msec)   : 2=1.01%, 4=57.59%, 10=40.57%, 20=0.58%, 50=0.20%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=0.04%, sys=1.39%, ctx=322624, majf=0, minf=570
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,162525,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=63.4MiB/s (66.5MB/s), 63.4MiB/s-63.4MiB/s (66.5MB/s-66.5MB/s), io=635MiB (666MB), run=10007-10007msec

Disk stats (read/write):
    drbd0: ios=0/160667, merge=0/0, ticks=0/610046, in_queue=610046, util=95.81%, aggrios=0/162529, aggrmerge=0/0, aggrticks=0/24455, aggrin_queue=24455, aggrutil=95.48%
    dm-2: ios=0/162529, merge=0/0, ticks=0/24455, in_queue=24455, util=95.48%, aggrios=0/162093, aggrmerge=0/362, aggrticks=0/21962, aggrin_queue=5331, aggrutil=95.49%
  sdb: ios=0/162093, merge=0/362, ticks=0/21962, in_queue=5331, util=95.49%

シーケンシャルライト DRBD PROTOCOL A xfs

[root@centos1 share]# fio -filename=/share/benchmark2 -direct=1 -rw=write -bs=4k -size=512M -numjobs=64 -runtime=10 -group_reporting -name=file1
file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 64 processes

Jobs: 64 (f=64): [W(64)][30.0%][r=0KiB/s,w=113MiB/s][r=0,w=28.9k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][40.0%][r=0KiB/s,w=113MiB/s][r=0,w=28.9k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][50.0%][r=0KiB/s,w=112MiB/s][r=0,w=28.6k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][60.0%][r=0KiB/s,w=111MiB/s][r=0,w=28.5k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][70.0%][r=0KiB/s,w=107MiB/s][r=0,w=27.4k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][80.0%][r=0KiB/s,w=109MiB/s][r=0,w=27.9k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][90.0%][r=0KiB/s,w=109MiB/s][r=0,w=27.0k IOPS][eta 00m:0
Jobs: 64 (f=64): [W(64)][100.0%][r=0KiB/s,w=109MiB/s][r=0,w=27.9k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=64): err= 0: pid=70115: Fri Oct  9 16:11:42 2020
  write: IOPS=28.2k, BW=110MiB/s (115MB/s)(1101MiB/10008msec)
    clat (usec): min=81, max=65167, avg=2267.34, stdev=1453.24
     lat (usec): min=81, max=65168, avg=2267.59, stdev=1453.24
    clat percentiles (usec):
     |  1.00th=[  857],  5.00th=[ 1237], 10.00th=[ 1385], 20.00th=[ 1549],
     | 30.00th=[ 1663], 40.00th=[ 1778], 50.00th=[ 1893], 60.00th=[ 2057],
     | 70.00th=[ 2311], 80.00th=[ 2671], 90.00th=[ 3326], 95.00th=[ 4293],
     | 99.00th=[ 7570], 99.50th=[11338], 99.90th=[17957], 99.95th=[20317],
     | 99.99th=[22676]
   bw (  KiB/s): min= 1184, max= 2227, per=1.56%, avg=1761.55, stdev=128.67, samples=1270
   iops        : min=  296, max=  556, avg=440.39, stdev=32.16, samples=1270
  lat (usec)   : 100=0.01%, 250=0.09%, 500=0.22%, 750=0.39%, 1000=0.98%
  lat (msec)   : 2=54.94%, 4=37.38%, 10=5.38%, 20=0.56%, 50=0.05%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.06%, sys=2.65%, ctx=434663, majf=0, minf=629
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,281964,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=1101MiB (1155MB), run=10008-10008msec

Disk stats (read/write):
    drbd0: ios=0/278192, merge=0/0, ticks=0/545442, in_queue=545442, util=91.65%, aggrios=0/281968, aggrmerge=0/0, aggrticks=0/42690, aggrin_queue=42690, aggrutil=91.71%
    dm-2: ios=0/281968, merge=0/0, ticks=0/42690, in_queue=42690, util=91.71%, aggrios=0/281538, aggrmerge=0/376, aggrticks=0/41970, aggrin_queue=12313, aggrutil=91.70%
  sdb: ios=0/281538, merge=0/376, ticks=0/41970, in_queue=12313, util=91.70%

結果まとめ

  • シーケンシャルライト 512MB
    • ローカル xfs:155 MB/s
    • DRBD PROTOCOL C ext4:7.2 MB/s
    • DRBD PROTOCOL A ext4:13.1 MB/s
    • DRBD PROTOCOL C xfs:66.5 MB/s
    • DRBD PROTOCOL A xfs:115 MB/s

DRBDで利用するファイルシステムはext4ではなくxfsを利用すること。