AN!Cluster Tutorial 2 - Disk Performance Tuning
Jump to navigation
Jump to search
|
Alteeve Wiki :: How To :: AN!Cluster Tutorial 2 - Disk Performance Tuning |
The goal of this tutorial is to show the process of tuning an Anvil! to maximize storage performance for a given workload.
 |
Warning: This is NOT complete and is largely a dumping ground of raw data. It's probably useless to most people at this time. |
The Workload
Our client specified the following 'fio' test as the simulated workload to optimize:
PATH="/data0/test"
fio --name certification --filename $PATH --ioengine libaio --direct 1 --bs 4M --rw randwrite --size 2g --iodepth 64 --numjobs 4 --runtime 30 --time_based --group_reportingThe '$PATH' will, eventually, be an xfs partition in a server hosted on an Anvil!.
The Setup
Each node is built on two Fujitsu RX2540 M1 nodes. The storage configuration is;
- 24x 1.8 TB 10krpm self-encrypting SAS drives.
- RAID level 6 using 22 drives in the array and 2 hot-spares.
- AVAGO MegaRAID 9361-8i Controller with 1 GiB of FBWC and SafeStore drive encryption management.
- DRBD v8.4.6.
- 4 KiB Native sector size drives using UEFI boot.
This configuration is designed for maximum availability and security. Performance is critical, but secondary to these priorities. As such, this tutorial will likely fall short of the maximum performance possible.
The Optimization Process
Tuning will be done in stages;
- Optimize local storage performance.
- Optimize network performance.
- Optimize DRBD performance (testing on the raw /dev/drbd0 device).
- Optimize clustered LVM performance.
- Optimize raw virtio block device performance.
- Optimize xfs performance inside the server.
Optimize
Local Storage
Local storage optimization requires tuning the RAID controller and virtual disk properties. Write-back caching is used and cached data is protected by flash-backed storage on the controller.
Strip Size Tuning
These tests are performed with no background operating in progress, optimal array and full disk encryption enabled.
Recorded test result is from 5 loops, each running for 60 seconds.
Call: /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
All tests performed with no disk cache, 4kib native sectors (no emulation) and a 'Direct' IO policy.
The kernel is set to default values (using tuned) for /dev/sda;
- scheduler is deadline
- nr_requests is 128
- rq_affinity is 1
- queue_depth is 256
- rotational is 1.
| Write Speed | Strip Size | Note | Raw output |
|---|---|---|---|
| 1136.36 MiB/s | 64 KiB | Base-line | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [64 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 15:58:42]
- Estimated End Time: [2015-12-08, 16:03:42]
Please wait, the first loop has started...
Pass: [1], speed: [1149.50 MiB/sec].
Pass: [2], speed: [1133.90 MiB/sec].
Pass: [3], speed: [1158.80 MiB/sec].
Pass: [4], speed: [1141.30 MiB/sec].
Pass: [5], speed: [1153.60 MiB/sec].
Average write speed: [1147.42 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [64 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 15:58:43]
- Estimated End Time: [2015-12-08, 16:03:43]
Please wait, the first loop has started...
Pass: [1], speed: [1119.20 MiB/sec].
Pass: [2], speed: [1130.20 MiB/sec].
Pass: [3], speed: [1156.70 MiB/sec].
Pass: [4], speed: [1126.40 MiB/sec].
Pass: [5], speed: [1149.30 MiB/sec].
Average write speed: [1136.36 MiB/sec]. |
| 1111.44 MiB/s | 128 KiB | No appreciable difference over 64 KiB | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [128 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 16:54:15]
- Estimated End Time: [2015-12-08, 16:59:15]
Please wait, the first loop has started...
Pass: [1], speed: [1153.50 MiB/sec].
Pass: [2], speed: [1116.40 MiB/sec].
Pass: [3], speed: [1092.20 MiB/sec].
Pass: [4], speed: [1115.70 MiB/sec].
Pass: [5], speed: [1104.20 MiB/sec].
Average write speed: [1116.40 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [128 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 16:54:16]
- Estimated End Time: [2015-12-08, 16:59:16]
Please wait, the first loop has started...
Pass: [1], speed: [1130.60 MiB/sec].
Pass: [2], speed: [1117.10 MiB/sec].
Pass: [3], speed: [1091.20 MiB/sec].
Pass: [4], speed: [1108.90 MiB/sec].
Pass: [5], speed: [1109.40 MiB/sec].
Average write speed: [1111.44 MiB/sec]. |
| 1309.90 MiB/s | 256 KiB | Notable improvement, also default value | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 17:38:22]
- Estimated End Time: [2015-12-08, 17:43:22]
Please wait, the first loop has started...
Pass: [1], speed: [1357.20 MiB/sec].
Pass: [2], speed: [1303.30 MiB/sec].
Pass: [3], speed: [1300.10 MiB/sec].
Pass: [4], speed: [1274.50 MiB/sec].
Pass: [5], speed: [1317.90 MiB/sec].
Average write speed: [1310.60 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 17:38:22]
- Estimated End Time: [2015-12-08, 17:43:22]
Please wait, the first loop has started...
Pass: [1], speed: [1351.10 MiB/sec].
Pass: [2], speed: [1324.90 MiB/sec].
Pass: [3], speed: [1285.50 MiB/sec].
Pass: [4], speed: [1272.60 MiB/sec].
Pass: [5], speed: [1315.40 MiB/sec].
Average write speed: [1309.90 MiB/sec]. |
| 1206.18 MiB/s | 512 KiB | Drop from 256 KiB strip size | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [512 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 18:19:04]
- Estimated End Time: [2015-12-08, 18:24:04]
Please wait, the first loop has started...
Pass: [1], speed: [1227.60 MiB/sec].
Pass: [2], speed: [1199.50 MiB/sec].
Pass: [3], speed: [1216.50 MiB/sec].
Pass: [4], speed: [1197.20 MiB/sec].
Pass: [5], speed: [1190.10 MiB/sec].
Average write speed: [1206.18 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [512 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 18:19:04]
- Estimated End Time: [2015-12-08, 18:24:04]
Please wait, the first loop has started...
Pass: [1], speed: [1212.90 MiB/sec].
Pass: [2], speed: [1227.60 MiB/sec].
Pass: [3], speed: [1212.50 MiB/sec].
Pass: [4], speed: [1206.10 MiB/sec].
Pass: [5], speed: [1202.50 MiB/sec].
Average write speed: [1212.32 MiB/sec]. |
| 1045.58 MiB/s | 1024 KiB | Slowest option | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [1024 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 19:10:51]
- Estimated End Time: [2015-12-08, 19:15:51]
Please wait, the first loop has started...
Pass: [1], speed: [1045.80 MiB/sec].
Pass: [2], speed: [1059.20 MiB/sec].
Pass: [3], speed: [1054.40 MiB/sec].
Pass: [4], speed: [1041.40 MiB/sec].
Pass: [5], speed: [1028.10 MiB/sec].
Average write speed: [1045.78 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [1024 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 19:10:53]
- Estimated End Time: [2015-12-08, 19:15:53]
Please wait, the first loop has started...
Pass: [1], speed: [1047.20 MiB/sec].
Pass: [2], speed: [1044.80 MiB/sec].
Pass: [3], speed: [1037.40 MiB/sec].
Pass: [4], speed: [1060.10 MiB/sec].
Pass: [5], speed: [1038.40 MiB/sec].
Average write speed: [1045.58 MiB/sec]. |
The best strip size is 256 KiB. With that, we will tune other components now.
Other VD Tuning
 |
Note: Disk caching is not tested. It does give a significant performance boost, but it is not safe and can't be considered for use in an Anvil!. |
| Write Speed | Emulation Type | IO Policy | Read-Ahead | Note | Raw output |
|---|---|---|---|---|---|
| 1282.36 MiB/s | Default (4kn) | Direct | Read-Ahead | Repeat of 256 KiB test for new baseline | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 00:37:23]
- Estimated End Time: [2015-12-09, 00:42:23]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1352.90 MiB/sec].
Pass: [2], speed: [1296.60 MiB/sec].
Pass: [3], speed: [1229.20 MiB/sec].
Pass: [4], speed: [1289.60 MiB/sec].
Pass: [5], speed: [1283.80 MiB/sec].
Average write speed: [1290.42 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 00:37:24]
- Estimated End Time: [2015-12-09, 00:42:24]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1344.40 MiB/sec].
Pass: [2], speed: [1279.30 MiB/sec].
Pass: [3], speed: [1248.50 MiB/sec].
Pass: [4], speed: [1259.50 MiB/sec].
Pass: [5], speed: [1280.10 MiB/sec].
Average write speed: [1282.36 MiB/sec]. |
| 1251.12 MiB/s | Force | Direct | Read-Ahead | Forcing 512b emulation hurts performance a bit. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [512E-force]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-08, 23:56:53]
- Estimated End Time: [2015-12-09, 00:01:53]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1266.50 MiB/sec].
Pass: [2], speed: [1262.90 MiB/sec].
Pass: [3], speed: [1263.70 MiB/sec].
Pass: [4], speed: [1280.50 MiB/sec].
Pass: [5], speed: [1279.80 MiB/sec].
Average write speed: [1270.68 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [512E-force]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-08, 23:57:00]
- Estimated End Time: [2015-12-09, 00:02:00]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1258.30 MiB/sec].
Pass: [2], speed: [1251.20 MiB/sec].
Pass: [3], speed: [1254.40 MiB/sec].
Pass: [4], speed: [1264.80 MiB/sec].
Pass: [5], speed: [1226.90 MiB/sec].
Average write speed: [1251.12 MiB/sec]. |
| 1269.04 MiB/s | Default (4kn) | Direct | No Read-Ahead | Disabled read-ahead caching. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [No Read-Ahead]
- Start Time: ....... [2015-12-09, 00:46:25]
- Estimated End Time: [2015-12-09, 00:51:25]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1257.30 MiB/sec].
Pass: [2], speed: [1281.30 MiB/sec].
Pass: [3], speed: [1282.70 MiB/sec].
Pass: [4], speed: [1265.10 MiB/sec].
Pass: [5], speed: [1297.50 MiB/sec].
Average write speed: [1276.78 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [No Read-Ahead]
- Start Time: ....... [2015-12-09, 00:46:24]
- Estimated End Time: [2015-12-09, 00:51:24]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1269.30 MiB/sec].
Pass: [2], speed: [1253.80 MiB/sec].
Pass: [3], speed: [1274.60 MiB/sec].
Pass: [4], speed: [1262.10 MiB/sec].
Pass: [5], speed: [1285.40 MiB/sec].
Average write speed: [1269.04 MiB/sec]. |
| 1268.54 MiB/s | Default (4kn) | Cached | Read-Ahead | No appreciable different. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 00:53:36]
- Estimated End Time: [2015-12-09, 00:58:36]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1285.90 MiB/sec].
Pass: [2], speed: [1262.90 MiB/sec].
Pass: [3], speed: [1281.50 MiB/sec].
Pass: [4], speed: [1285.40 MiB/sec].
Pass: [5], speed: [1261.70 MiB/sec].
Average write speed: [1275.48 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 00:53:35]
- Estimated End Time: [2015-12-09, 00:58:35]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1275.30 MiB/sec].
Pass: [2], speed: [1260.70 MiB/sec].
Pass: [3], speed: [1272.30 MiB/sec].
Pass: [4], speed: [1266.20 MiB/sec].
Pass: [5], speed: [1268.20 MiB/sec].
Average write speed: [1268.54 MiB/sec]. |
The default settings; direct IO policy, read-ahead caching and 4k native sector sizes are best.
Kernel Tuning
| Write Speed | Scheduler | Queue Requests | Queue Depth | Request Affinity | Note | Raw output |
|---|---|---|---|---|---|---|
| 1261.30 MiB/s | deadline | 128 | 256 | 1 | Repeat of the VD tuning for a new baseline | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 01:01:06]
- Estimated End Time: [2015-12-09, 01:06:06]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1278.80 MiB/sec].
Pass: [2], speed: [1279.40 MiB/sec].
Pass: [3], speed: [1278.70 MiB/sec].
Pass: [4], speed: [1286.60 MiB/sec].
Pass: [5], speed: [1282.40 MiB/sec].
Average write speed: [1281.18 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 01:01:06]
- Estimated End Time: [2015-12-09, 01:06:06]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1268.40 MiB/sec].
Pass: [2], speed: [1256.90 MiB/sec].
Pass: [3], speed: [1262.70 MiB/sec].
Pass: [4], speed: [1257.60 MiB/sec].
Pass: [5], speed: [1260.90 MiB/sec].
Average write speed: [1261.30 MiB/sec]. |
| 905.81 MiB/s | cfq | 128 | 256 | 1 | Complete Fair Queuing; Significantly slower, though it might be worth testing later with 'noop' in the guest. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 01:09:48]
- Estimated End Time: [2015-12-09, 01:14:48]
OS Configuration (sda):
- Scheduler: ...... [cfq]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [918.06 MiB/sec].
Pass: [2], speed: [915.54 MiB/sec].
Pass: [3], speed: [897.26 MiB/sec].
Pass: [4], speed: [910.41 MiB/sec].
Pass: [5], speed: [915.18 MiB/sec].
Average write speed: [911.29 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 01:09:48]
- Estimated End Time: [2015-12-09, 01:14:48]
OS Configuration (sda):
- Scheduler: ...... [cfq]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [902.63 MiB/sec].
Pass: [2], speed: [889.31 MiB/sec].
Pass: [3], speed: [918.16 MiB/sec].
Pass: [4], speed: [918.74 MiB/sec].
Pass: [5], speed: [900.21 MiB/sec].
Average write speed: [905.81 MiB/sec]. |
| 1275.40 MiB/s | noop | 128 | 256 | 1 | Iiiinteresting. Leave scheduling to the controller? Test against CPU load later with DRBD. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 01:16:43]
- Estimated End Time: [2015-12-09, 01:21:43]
OS Configuration (sda):
- Scheduler: ...... [noop]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1335.90 MiB/sec].
Pass: [2], speed: [1284.30 MiB/sec].
Pass: [3], speed: [1268.70 MiB/sec].
Pass: [4], speed: [1260.90 MiB/sec].
Pass: [5], speed: [1289.10 MiB/sec].
Average write speed: [1287.78 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 01:16:43]
- Estimated End Time: [2015-12-09, 01:21:43]
OS Configuration (sda):
- Scheduler: ...... [noop]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1285.40 MiB/sec].
Pass: [2], speed: [1267.40 MiB/sec].
Pass: [3], speed: [1272.70 MiB/sec].
Pass: [4], speed: [1275.40 MiB/sec].
Pass: [5], speed: [1276.10 MiB/sec].
Average write speed: [1275.40 MiB/sec]. |
| 1267.64 MiB/s | deadline | 975 | 256 | 1 | No real difference on its own. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:09:25]
- Estimated End Time: [2015-12-09, 17:14:25]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [975]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1268.20 MiB/sec].
Pass: [2], speed: [1266.40 MiB/sec].
Pass: [3], speed: [1285.60 MiB/sec].
Pass: [4], speed: [1284.70 MiB/sec].
Pass: [5], speed: [1284.10 MiB/sec].
Average write speed: [1277.80 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:09:24]
- Estimated End Time: [2015-12-09, 17:14:24]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [975]
- Queue Depth: .... [256]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1270.50 MiB/sec].
Pass: [2], speed: [1273.60 MiB/sec].
Pass: [3], speed: [1258.60 MiB/sec].
Pass: [4], speed: [1270.30 MiB/sec].
Pass: [5], speed: [1265.20 MiB/sec].
Average write speed: [1267.64 MiB/sec]. |
| 1417.52 MiB/s | deadline | 975 | 919 | 1 | Tried to set '975', was changed to '919'. Significant boost in performance! | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:17:10]
- Estimated End Time: [2015-12-09, 17:22:10]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [975]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1427.80 MiB/sec].
Pass: [2], speed: [1463.40 MiB/sec].
Pass: [3], speed: [1417.10 MiB/sec].
Pass: [4], speed: [1408.20 MiB/sec].
Pass: [5], speed: [1467.40 MiB/sec].
Average write speed: [1436.78 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:17:09]
- Estimated End Time: [2015-12-09, 17:22:09]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [975]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1433.20 MiB/sec].
Pass: [2], speed: [1399.10 MiB/sec].
Pass: [3], speed: [1415.90 MiB/sec].
Pass: [4], speed: [1411.60 MiB/sec].
Pass: [5], speed: [1427.80 MiB/sec].
Average write speed: [1417.52 MiB/sec]. |
| 1864.72 MiB/s | deadline | 100000 | 919 | 1 | O_O - What are the risks involved with so much cached IO? | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:24:04]
- Estimated End Time: [2015-12-09, 17:29:04]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [100000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1874.80 MiB/sec].
Pass: [2], speed: [1858.10 MiB/sec].
Pass: [3], speed: [1836.20 MiB/sec].
Pass: [4], speed: [1920.90 MiB/sec].
Pass: [5], speed: [1833.60 MiB/sec].
Average write speed: [1864.72 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:24:03]
- Estimated End Time: [2015-12-09, 17:29:03]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [100000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1889.20 MiB/sec].
Pass: [2], speed: [1904.40 MiB/sec].
Pass: [3], speed: [1828.10 MiB/sec].
Pass: [4], speed: [1851.20 MiB/sec].
Pass: [5], speed: [1892.90 MiB/sec].
Average write speed: [1873.16 MiB/sec]. |
| 1839.74 MiB/s | deadline | 10000 | 919 | 1 | Same as nr_requests = 100,000 | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:35:53]
- Estimated End Time: [2015-12-09, 17:40:53]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [10000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1930.90 MiB/sec].
Pass: [2], speed: [1870.80 MiB/sec].
Pass: [3], speed: [1816.90 MiB/sec].
Pass: [4], speed: [1854.90 MiB/sec].
Pass: [5], speed: [1805.20 MiB/sec].
Average write speed: [1855.74 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:35:53]
- Estimated End Time: [2015-12-09, 17:40:53]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [10000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1854.20 MiB/sec].
Pass: [2], speed: [1840.20 MiB/sec].
Pass: [3], speed: [1823.50 MiB/sec].
Pass: [4], speed: [1861.50 MiB/sec].
Pass: [5], speed: [1819.30 MiB/sec].
Average write speed: [1839.74 MiB/sec]. |
| 1877.18 MiB/s | deadline | 5000 | 919 | 1 | Still the same performance | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:46:40]
- Estimated End Time: [2015-12-09, 17:51:40]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [5000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1890.70 MiB/sec].
Pass: [2], speed: [1863.90 MiB/sec].
Pass: [3], speed: [1887.60 MiB/sec].
Pass: [4], speed: [1881.10 MiB/sec].
Pass: [5], speed: [1916.40 MiB/sec].
Average write speed: [1887.94 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:46:42]
- Estimated End Time: [2015-12-09, 17:51:42]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [5000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1880.30 MiB/sec].
Pass: [2], speed: [1892.10 MiB/sec].
Pass: [3], speed: [1879.70 MiB/sec].
Pass: [4], speed: [1845.10 MiB/sec].
Pass: [5], speed: [1888.70 MiB/sec].
Average write speed: [1877.18 MiB/sec]. |
| 1374.22 MiB/s | deadline | 2000 | 919 | 1 | Significant drop back down in performance. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:55:30]
- Estimated End Time: [2015-12-09, 18:00:30]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [2000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1390.70 MiB/sec].
Pass: [2], speed: [1368.90 MiB/sec].
Pass: [3], speed: [1375.70 MiB/sec].
Pass: [4], speed: [1376.50 MiB/sec].
Pass: [5], speed: [1401.20 MiB/sec].
Average write speed: [1382.60 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 17:55:28]
- Estimated End Time: [2015-12-09, 18:00:28]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [2000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1382.30 MiB/sec].
Pass: [2], speed: [1362.20 MiB/sec].
Pass: [3], speed: [1352.20 MiB/sec].
Pass: [4], speed: [1398.60 MiB/sec].
Pass: [5], speed: [1375.80 MiB/sec].
Average write speed: [1374.22 MiB/sec]. |
| 1585.52 MiB/s | deadline | 3000 | 919 | 1 | Creeping back up... | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:01:46]
- Estimated End Time: [2015-12-09, 18:06:46]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [3000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1588.70 MiB/sec].
Pass: [2], speed: [1597.40 MiB/sec].
Pass: [3], speed: [1656.40 MiB/sec].
Pass: [4], speed: [1574.80 MiB/sec].
Pass: [5], speed: [1563.90 MiB/sec].
Average write speed: [1596.24 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:01:47]
- Estimated End Time: [2015-12-09, 18:06:47]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [3000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1593.90 MiB/sec].
Pass: [2], speed: [1569.40 MiB/sec].
Pass: [3], speed: [1568.80 MiB/sec].
Pass: [4], speed: [1611.20 MiB/sec].
Pass: [5], speed: [1584.30 MiB/sec].
Average write speed: [1585.52 MiB/sec]. |
| 1872.30 MiB/s | deadline | 4000 | 919 | 1 | This is the ideal Queue Request size. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:07:46]
- Estimated End Time: [2015-12-09, 18:12:46]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [4000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1842.40 MiB/sec].
Pass: [2], speed: [1857.20 MiB/sec].
Pass: [3], speed: [1920.10 MiB/sec].
Pass: [4], speed: [1839.50 MiB/sec].
Pass: [5], speed: [1902.30 MiB/sec].
Average write speed: [1872.30 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:07:45]
- Estimated End Time: [2015-12-09, 18:12:45]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [4000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1872.20 MiB/sec].
Pass: [2], speed: [1871.60 MiB/sec].
Pass: [3], speed: [1852.10 MiB/sec].
Pass: [4], speed: [1921.20 MiB/sec].
Pass: [5], speed: [1880.50 MiB/sec].
Average write speed: [1879.52 MiB/sec]. |
| 1740.86 MiB/s | deadline | 3500 | 919 | 1 | Climbing back up | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:14:11]
- Estimated End Time: [2015-12-09, 18:19:11]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [3500]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1738.20 MiB/sec].
Pass: [2], speed: [1733.10 MiB/sec].
Pass: [3], speed: [1722.30 MiB/sec].
Pass: [4], speed: [1759.30 MiB/sec].
Pass: [5], speed: [1751.40 MiB/sec].
Average write speed: [1740.86 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:14:10]
- Estimated End Time: [2015-12-09, 18:19:10]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [3500]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1747.20 MiB/sec].
Pass: [2], speed: [1765.30 MiB/sec].
Pass: [3], speed: [1724.20 MiB/sec].
Pass: [4], speed: [1735.40 MiB/sec].
Pass: [5], speed: [1813.10 MiB/sec].
Average write speed: [1757.04 MiB/sec]. |
| 1403.90 MiB/s | noop | 4000 | 919 | 1 | Stick with deadline. Apparently, 'noop' can suffer latency spikes anyway. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:23:50]
- Estimated End Time: [2015-12-09, 18:28:50]
OS Configuration (sda):
- Scheduler: ...... [noop]
- Queue Requests: . [4000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1423.30 MiB/sec].
Pass: [2], speed: [1425.10 MiB/sec].
Pass: [3], speed: [1404.90 MiB/sec].
Pass: [4], speed: [1399.60 MiB/sec].
Pass: [5], speed: [1420.30 MiB/sec].
Average write speed: [1414.64 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:23:50]
- Estimated End Time: [2015-12-09, 18:28:50]
OS Configuration (sda):
- Scheduler: ...... [noop]
- Queue Requests: . [4000]
- Queue Depth: .... [919]
- Request Affinity: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1404.90 MiB/sec].
Pass: [2], speed: [1407.40 MiB/sec].
Pass: [3], speed: [1388.50 MiB/sec].
Pass: [4], speed: [1404.50 MiB/sec].
Pass: [5], speed: [1414.20 MiB/sec].
Average write speed: [1403.90 MiB/sec]. |
| 1833.34 MiB/s | deadline | 4000 | 919 | 0 | Disabling the request affinity didn't change anything | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:38:43]
- Estimated End Time: [2015-12-09, 18:43:43]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [4000]
- Queue Depth: .... [919]
- Request Affinity: [0]
Please wait, the first loop has started...
Pass: [1], speed: [1835.80 MiB/sec].
Pass: [2], speed: [1850.10 MiB/sec].
Pass: [3], speed: [1894.80 MiB/sec].
Pass: [4], speed: [1747.10 MiB/sec].
Pass: [5], speed: [1838.90 MiB/sec].
Average write speed: [1833.34 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:38:43]
- Estimated End Time: [2015-12-09, 18:43:43]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [4000]
- Queue Depth: .... [919]
- Request Affinity: [0]
Please wait, the first loop has started...
Pass: [1], speed: [1843.50 MiB/sec].
Pass: [2], speed: [1935.10 MiB/sec].
Pass: [3], speed: [1806.90 MiB/sec].
Pass: [4], speed: [1875.50 MiB/sec].
Pass: [5], speed: [1952.10 MiB/sec].
Average write speed: [1882.62 MiB/sec]. |
| 1827.18 MiB/s | deadline | 4000 | 919 | 2 | No real difference. Leave this as '1', which is default. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:52:04]
- Estimated End Time: [2015-12-09, 18:57:04]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [4000]
- Queue Depth: .... [919]
- Request Affinity: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1823.10 MiB/sec].
Pass: [2], speed: [1908.20 MiB/sec].
Pass: [3], speed: [1819.50 MiB/sec].
Pass: [4], speed: [1942.30 MiB/sec].
Pass: [5], speed: [1919.40 MiB/sec].
Average write speed: [1882.50 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [256 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Emulation Type: ... [default]
- Read-Ahead Caching: [Read-Ahead]
- Start Time: ....... [2015-12-09, 18:52:03]
- Estimated End Time: [2015-12-09, 18:57:03]
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [4000]
- Queue Depth: .... [919]
- Request Affinity: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1822.80 MiB/sec].
Pass: [2], speed: [1814.30 MiB/sec].
Pass: [3], speed: [1817.20 MiB/sec].
Pass: [4], speed: [1814.30 MiB/sec].
Pass: [5], speed: [1867.30 MiB/sec].
Average write speed: [1827.18 MiB/sec]. |
Decision is to set:
- /sys/block/sda/device/queue_depth to 919.
- /sys/block/sda/queue/scheduler to deadline (default).
- /sys/block/sda/queue/rq_affinity to 1 (default).
- /sys/block/sda/queue/nr_requests to 4000.
|
Note: These values may not apply when DRBD is layered on top later. Retest! |
Tuning the 'deadline' Scheduler
To see how these change effect things, the tests will be run both with stock queue depth and queue requests and again with the optimized values from above.
| Write Speed | Queue Requests | Queue Depth | FIFO Batch | Front Merges | Read Expire | Write Expire | Writes Starved | Note | Raw output |
|---|---|---|---|---|---|---|---|---|---|
| 1275.86 MiB/s | 128 | 256 | 16 | 1 | 500 ms | 5000 ms | 2 | Baseline with default values. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1289.70 MiB/sec].
Pass: [2], speed: [1271.70 MiB/sec].
Pass: [3], speed: [1291.70 MiB/sec].
Pass: [4], speed: [1280.40 MiB/sec].
Pass: [5], speed: [1285.20 MiB/sec].
Average write speed: [1283.74 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1296.60 MiB/sec].
Pass: [2], speed: [1282.40 MiB/sec].
Pass: [3], speed: [1272.20 MiB/sec].
Pass: [4], speed: [1267.50 MiB/sec].
Pass: [5], speed: [1260.60 MiB/sec].
Average write speed: [1275.86 MiB/sec]. |
| 1261.62 MiB/s | 128 | 256 | 16 | 1 | 500 ms | 5000 ms | 1 | Setting minimum 'write starved' to minimize read preference | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1273.40 MiB/sec].
Pass: [2], speed: [1281.80 MiB/sec].
Pass: [3], speed: [1268.40 MiB/sec].
Pass: [4], speed: [1290.20 MiB/sec].
Pass: [5], speed: [1268.80 MiB/sec].
Average write speed: [1276.52 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [1]
Please wait, the first loop has started...
Pass: [1], speed: [1257.30 MiB/sec].
Pass: [2], speed: [1260.80 MiB/sec].
Pass: [3], speed: [1272.90 MiB/sec].
Pass: [4], speed: [1272.30 MiB/sec].
Pass: [5], speed: [1244.80 MiB/sec].
Average write speed: [1261.62 MiB/sec]. |
| 1267.56 MiB/s | 128 | 256 | 16 | 1 | 500 ms | 5000 ms | 4 | Try doubling the default to give more time to coalesce writes | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [4]
Please wait, the first loop has started...
Pass: [1], speed: [1278.90 MiB/sec].
Pass: [2], speed: [1268.50 MiB/sec].
Pass: [3], speed: [1283.90 MiB/sec].
Pass: [4], speed: [1292.10 MiB/sec].
Pass: [5], speed: [1287.10 MiB/sec].
Average write speed: [1282.10 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [4]
Please wait, the first loop has started...
Pass: [1], speed: [1289.80 MiB/sec].
Pass: [2], speed: [1269.90 MiB/sec].
Pass: [3], speed: [1249.10 MiB/sec].
Pass: [4], speed: [1268.80 MiB/sec].
Pass: [5], speed: [1260.20 MiB/sec].
Average write speed: [1267.56 MiB/sec]. |
| 1260.82 MiB/s | 128 | 256 | 16 | 1 | 500 ms | 5000 ms | 8 | Push the write starved unreasonably high. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [8]
Please wait, the first loop has started...
Pass: [1], speed: [1276.70 MiB/sec].
Pass: [2], speed: [1269.10 MiB/sec].
Pass: [3], speed: [1274.90 MiB/sec].
Pass: [4], speed: [1272.70 MiB/sec].
Pass: [5], speed: [1272.70 MiB/sec].
Average write speed: [1273.22 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [8]
Please wait, the first loop has started...
Pass: [1], speed: [1264.20 MiB/sec].
Pass: [2], speed: [1268.20 MiB/sec].
Pass: [3], speed: [1268.70 MiB/sec].
Pass: [4], speed: [1243.30 MiB/sec].
Pass: [5], speed: [1259.70 MiB/sec].
Average write speed: [1260.82 MiB/sec]. |
| 1269.98 MiB/s | 128 | 256 | 16 | 1 | 500 ms | 500 ms | 2 | Trying a low write expire to flush out data more quickly | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [500]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1285.20 MiB/sec].
Pass: [2], speed: [1278.90 MiB/sec].
Pass: [3], speed: [1289.50 MiB/sec].
Pass: [4], speed: [1274.50 MiB/sec].
Pass: [5], speed: [1280.10 MiB/sec].
Average write speed: [1281.64 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [500]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1286.40 MiB/sec].
Pass: [2], speed: [1277.20 MiB/sec].
Pass: [3], speed: [1251.70 MiB/sec].
Pass: [4], speed: [1261.10 MiB/sec].
Pass: [5], speed: [1273.50 MiB/sec].
Average write speed: [1269.98 MiB/sec]. |
| 1275.88 MiB/s | 128 | 256 | 16 | 1 | 5000 ms | 500 ms | 2 | Flipping the read/write priority | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [5000]
- Write Expire: .. [500]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1272.70 MiB/sec].
Pass: [2], speed: [1278.50 MiB/sec].
Pass: [3], speed: [1279.10 MiB/sec].
Pass: [4], speed: [1275.60 MiB/sec].
Pass: [5], speed: [1281.20 MiB/sec].
Average write speed: [1277.42 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [5000]
- Write Expire: .. [500]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1264.60 MiB/sec].
Pass: [2], speed: [1267.70 MiB/sec].
Pass: [3], speed: [1284.50 MiB/sec].
Pass: [4], speed: [1273.90 MiB/sec].
Pass: [5], speed: [1288.70 MiB/sec].
Average write speed: [1275.88 MiB/sec]. |
| 1278.60 MiB/s | 128 | 256 | 32 | 1 | 500 ms | 5000 ms | 2 | Doubling the fifo buffer size | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [32]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1278.80 MiB/sec].
Pass: [2], speed: [1285.30 MiB/sec].
Pass: [3], speed: [1282.50 MiB/sec].
Pass: [4], speed: [1289.30 MiB/sec].
Pass: [5], speed: [1290.90 MiB/sec].
Average write speed: [1285.36 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [32]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1285.40 MiB/sec].
Pass: [2], speed: [1273.70 MiB/sec].
Pass: [3], speed: [1278.20 MiB/sec].
Pass: [4], speed: [1274.60 MiB/sec].
Pass: [5], speed: [1281.10 MiB/sec].
Average write speed: [1278.60 MiB/sec]. |
| 1274.46 MiB/s | 128 | 256 | 64 | 1 | 500 ms | 5000 ms | 2 | Silly fifo batch size, still no change. | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [64]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1301.90 MiB/sec].
Pass: [2], speed: [1298.80 MiB/sec].
Pass: [3], speed: [1289.60 MiB/sec].
Pass: [4], speed: [1305.70 MiB/sec].
Pass: [5], speed: [1285.80 MiB/sec].
Average write speed: [1296.36 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [64]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1292.20 MiB/sec].
Pass: [2], speed: [1274.90 MiB/sec].
Pass: [3], speed: [1259.10 MiB/sec].
Pass: [4], speed: [1272.30 MiB/sec].
Pass: [5], speed: [1273.80 MiB/sec].
Average write speed: [1274.46 MiB/sec]. |
| 1277.90 MiB/s | 128 | 256 | 16 | 1 | 500 ms | 500 ms | 2 | Matching read and write expire times | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [500]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1272.90 MiB/sec].
Pass: [2], speed: [1295.30 MiB/sec].
Pass: [3], speed: [1287.30 MiB/sec].
Pass: [4], speed: [1277.20 MiB/sec].
Pass: [5], speed: [1270.60 MiB/sec].
Average write speed: [1280.66 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [128]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [500]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [1278.80 MiB/sec].
Pass: [2], speed: [1275.30 MiB/sec].
Pass: [3], speed: [1277.10 MiB/sec].
Pass: [4], speed: [1281.70 MiB/sec].
Pass: [5], speed: [1276.60 MiB/sec].
Average write speed: [1277.90 MiB/sec]. |
| 2060.44 MiB/s | 4000 | 256 | 16 | 1 | 500 ms | 5000 ms | 2 | Increased Queue requests only | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [4000]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [2197.70 MiB/sec].
Pass: [2], speed: [2055.90 MiB/sec].
Pass: [3], speed: [2024.60 MiB/sec].
Pass: [4], speed: [2122.40 MiB/sec].
Pass: [5], speed: [2056.90 MiB/sec].
Average write speed: [2091.50 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
OS Configuration (sda):
- Scheduler: ...... [deadline]
- Queue Requests: . [4000]
- Queue Depth: .... [256]
- Request Affinity: [1]
Deadline Scheduler Configuration (sda):
- FIFO Batch size: [16]
- Read Expire: ... [500]
- Write Expire: .. [5000]
- Front Merges: .. [1]
- Writes Starved: [2]
Please wait, the first loop has started...
Pass: [1], speed: [2017.90 MiB/sec].
Pass: [2], speed: [2036.60 MiB/sec].
Pass: [3], speed: [2098.20 MiB/sec].
Pass: [4], speed: [2058.40 MiB/sec].
Pass: [5], speed: [2091.10 MiB/sec].
Average write speed: [2060.44 MiB/sec]. |
Decision is to leave everything at default except for the queue requests, which we'll set to 4000.
Network Tuning
Local Storage
Local storage optimization requires tuning the RAID controller and virtual disk properties. Write-back caching is used and cached data is protected by flash-backed storage on the controller.
iperf3 Testing
Recorded test result is from 5 loops, each running for 60 seconds.
Client Call: iperf3 --client 10.10.10.2 --port 7788 --time 30 --interval 0 Server Call: iperf3 --port 7788 --server --one-off --bind 10.10.10.2 --interval 0
| Network Speed | Note | Raw output | |
|---|---|---|---|
| Any questions, feedback, advice, complaints or meanderings are welcome. | |||
| Alteeve's Niche! | Alteeve Enterprise Support | Community Support | |
| © 2025 Alteeve. Intelligent Availability® is a registered trademark of Alteeve's Niche! Inc. 1997-2025 | |||
| legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions. | |||