resilver slowdown....

[Michelle Sullivan]

After retuning for priority to the resilver, I see massive (very good)
performance.... then at some random time later (or so it seemed) the
performance would drop off very quickly always at least to half the
performance, sometimes as much as a quarter of the speed...


At first I though it was memory based, then observing system with 19G
(of 32G) free figured it must be based on the stuff on the disk got some
16T of very small files, and another 16T of video files so figured it
was probably going from large files to small files etc... Nope, after a
reboot and resetting the tuning params to the same each time after the
reboot I'd be seeing an immediate and sustained performance... Watching
it more it seems this is directly related:

[***@colossus ~]# sysctl -a | grep \\.arc_
vfs.zfs.arc_meta_limit: 4294967296
vfs.zfs.arc_meta_used: 5529107352
vfs.zfs.arc_min: 8589934592
vfs.zfs.arc_max: 17179869184

Specifically when the vfs.zfs.arc_meta_used reaches and then exceeds
vfs.zfs.arc_meta_limit..

This is the average output of 'gstat -I 5s' when the limit is exceeded:

dT: 5.001s w: 5.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 49 49 2016 192.2 0 0 0.0 82.7| mfid0
0 50 50 1981 206.5 0 0 0.0 82.4| mfid1
0 49 49 1988 194.4 0 0 0.0 80.7| mfid2
0 51 51 1984 218.5 0 0 0.0 83.8| mfid3
0 50 50 1993 215.5 0 0 0.0 82.5| mfid4
0 51 51 2067 204.5 0 0 0.0 83.1| mfid5
0 54 54 1983 181.0 0 0 0.0 77.2| mfid6
0 49 49 1982 207.3 0 0 0.0 83.2| mfid7
0 49 49 1983 204.0 0 0 0.0 76.4| mfid8
0 51 51 2130 213.5 0 0 0.0 84.0| mfid9
0 51 51 2040 213.2 0 0 0.0 84.6| mfid10
74 142 0 0 0.0 142 2362 988.5 100.2| mfid11
0 52 52 1988 145.1 0 0 0.0 79.3| mfid12
0 50 50 1978 166.8 0 0 0.0 90.1| mfid13
0 101 0 0 0.0 101 2104 173.6 91.6| mfid14
0 0 0 0 0.0 0 0 3.2 0.1| ada0
0 0 0 0 0.0 0 0 3.1 0.1| ada1
0 0 0 0 0.0 0 0 0.0 0.0| raid/r0
0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p1
0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p2
0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p3
0 0 0 0 0.0 0 0 0.0 0.0| label/Boot
0 0 0 0 0.0 0 0 0.0 0.0|
gptid/a9e5606c-c587-11e3-9aac-ac220bcb8f6c
0 0 0 0 0.0 0 0 0.0 0.0| label/Swap
0 0 0 0 0.0 0 0 0.0 0.0| label/RootFS

Before the limit is reached... (ie immediately after a reboot) and the
following tuning is done:
[***@colossus ~]# sysctl vfs.zfs.resilver_min_time_ms=5000; sysctl
vfs.zfs.resilver_delay=0 ; sysctl vfs.zfs.vdev.max_pending=10000; sysctl
vfs.zfs.top_maxinflight=3200 ; sysctl vfs.zfs.scan_idle=1
vfs.zfs.resilver_min_time_ms: 3000 -> 5000
vfs.zfs.resilver_delay: 2 -> 0
vfs.zfs.vdev.max_pending: 10 -> 10000
vfs.zfs.top_maxinflight: 32 -> 3200
vfs.zfs.scan_idle: 50 -> 1

The following is the average output of the same gstat:

dT: 5.001s w: 5.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 115 115 6396 304.1 0 0 0.0 100.5| mfid0
0 117 117 6532 298.3 0 0 0.0 100.6| mfid1
0 112 112 6379 235.1 0 0 0.0 104.9| mfid2
0 114 114 6349 302.4 0 0 0.0 97.7| mfid3
0 117 117 6529 300.0 0 0 0.0 98.1| mfid4
25 112 112 6070 266.4 0 0 0.0 101.5| mfid5
39 106 106 5773 245.7 0 0 0.0 100.7| mfid6
25 110 110 6037 266.1 0 0 0.0 103.2| mfid7
35 111 111 6001 250.6 0 0 0.0 104.2| mfid8
35 110 110 5925 255.6 0 0 0.0 99.3| mfid9
11 116 116 6270 259.2 0 0 0.0 105.3| mfid10
207 374 0 0 0.0 374 4694 576.2 103.5| mfid11
8 115 115 6287 255.3 0 0 0.0 103.0| mfid12
1 113 113 6359 202.3 0 0 0.0 107.6| mfid13
95 268 0 0 0.0 268 4972 207.5 105.6| mfid14
0 0 0 0 0.0 0 0 3.4 0.1| ada0
0 0 0 0 0.0 0 0 3.5 0.1| ada1
0 0 0 0 0.0 0 0 0.0 0.0| raid/r0
0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p1
0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p2
0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p3
0 0 0 0 0.0 0 0 0.0 0.0| label/Boot
0 0 0 0 0.0 0 0 0.0 0.0|
gptid/a9e5606c-c587-11e3-9aac-ac220bcb8f6c
0 0 0 0 0.0 0 0 0.0 0.0| label/Swap
0 0 0 0 0.0 0 0 0.0 0.0| label/RootFS


Is this a bug?

9.3-p33 on LSI 9260-16i-512mb set in 15 3Tb SATA3 RAID 0 arrays, no
read-ahead, write-back, cached.

Regards,

[Michelle Sullivan]

Post by Michelle Sullivan
After retuning for priority to the resilver, I see massive (very good)
performance.... then at some random time later (or so it seemed) the
performance would drop off very quickly always at least to half the
performance, sometimes as much as a quarter of the speed...
At first I though it was memory based, then observing system with 19G
(of 32G) free figured it must be based on the stuff on the disk got
some 16T of very small files, and another 16T of video files so
figured it was probably going from large files to small files etc...
Nope, after a reboot and resetting the tuning params to the same each
time after the reboot I'd be seeing an immediate and sustained
vfs.zfs.arc_meta_limit: 4294967296
vfs.zfs.arc_meta_used: 5529107352
vfs.zfs.arc_min: 8589934592
vfs.zfs.arc_max: 17179869184

Yup think it is related to the meta_limit and meta_used...

Resilvering at 203M/s (according to zpool status)
[***@colossus ~]# sysctl -a | grep \\.arc_
vfs.zfs.arc_meta_limit: 4294967296
vfs.zfs.arc_meta_used: 2187549528
vfs.zfs.arc_min: 8589934592
vfs.zfs.arc_max: 17179869184

Noticed it had slowed again.. around 64M/s ...
[***@colossus ~]# sysctl -a | grep \\.arc_
vfs.zfs.arc_meta_limit: 4294967296
vfs.zfs.arc_meta_used: 4295111952
vfs.zfs.arc_min: 8589934592
vfs.zfs.arc_max: 17179869184

Rebooted and back to >120M/s (can't get a reading atm - too soon after
the reboot)