ZFS, SSD and encryption

Discussion:

(too old to reply)

Nikos Kastanas

2016-07-22 12:48:50 UTC

I have a Lenovo X220 laptop running FreeBSD 10.3-RELEASE with ZFS and
encryption on a plain HDD. I am considering buying a Samsung Pro 850 SSD to
boost performance but I am not sure if TRIM and ZFS+Encryption work well
together. After some research online, I found *this page*
<https://www.freebsd.org/doc/faq/all-about-zfs.html>which states the
following:

*Note: *
ZFS TRIM may not work with all configurations, such as a ZFS filesystem on
a GELI-backed device.

From what I can understand from the above note, I should not use the
encryption option when installing FreeBSD with ZFS on an SSD. TRIM will not
work correctly and therefore the SSD performace will be impacted.

Do I understand it correctly? Or is there something that I miss?

After asking in the community forums
<https://forums.freebsd.org/threads/57011/> a member suggested that i
should instead ask here.

Sorry if this is a stupid question, but I really want to be sure, since
SSDs are expensive and I don't want to make a mistake that will affect its
performance.

Thank you in advance.

Karl Denninger

2016-07-22 14:27:37 UTC

Permalink

Post by Nikos Kastanas
I have a Lenovo X220 laptop running FreeBSD 10.3-RELEASE with ZFS and
encryption on a plain HDD. I am considering buying a Samsung Pro 850 SSD to
boost performance but I am not sure if TRIM and ZFS+Encryption work well
together. After some research online, I found *this page*
<https://www.freebsd.org/doc/faq/all-about-zfs.html>which states the
*Note: *
ZFS TRIM may not work with all configurations, such as a ZFS filesystem on
a GELI-backed device.
From what I can understand from the above note, I should not use the
encryption option when installing FreeBSD with ZFS on an SSD. TRIM will not
work correctly and therefore the SSD performace will be impacted.

Meh. Simply not true. The reason for the "supported feature" flag here
is that this machine was recently rolled forward to 11.0-BETA1, but I
have not upgraded the pools yet from the feature set of 10.2.

[***@NewFS ~]$ zpool status zsr
pool: zsr
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not
support
the features. See zpool-features(7) for details.
scan: scrub repaired 0 in 0h6m with 0 errors on Sun Jul 17 03:12:01 2016
config:

NAME STATE READ WRITE CKSUM
zsr ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da8p4.eli ONLINE 0 0 0
da9p4.eli ONLINE 0 0 0

errors: No known data errors

[***@NewFS ~]$ gpart show da8
=> 34 468862061 da8 GPT (224G)
34 2014 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 20971520 2 freebsd-zfs [bootme] (10G)
20975616 134217728 3 freebsd-swap (64G)
155193344 313667584 4 freebsd-zfs (150G)
468860928 1167 - free - (584K)

da8: <ATA INTEL SSDSC2BP24 0420> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number BTJR41210025240AGN
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 228936MB (468862128 512 byte sectors)

***@NewFS:/var/log # sysctl -a|grep trim
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
vfs.zfs.vdev.trim_max_pending: 10000
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.trim_on_init: 1
kstat.zfs.misc.zio_trim.failed: 0
kstat.zfs.misc.zio_trim.unsupported: 25748
kstat.zfs.misc.zio_trim.success: 6120223
kstat.zfs.misc.zio_trim.bytes: 295371051008

And as you can see, TRIM is definitely working (on the devices that can
handle it); there are also spinning rust disks in this machine, thus the
"unsupported" reports as well.

HOWEVER, I do suggest (strongly!) that you NOT use the particular SSD
you are intending to buy as it has no power-loss protection. Instead,
buy an Intel 730-series drive (that's what's in this machine); it has
that protection and it is *EXTREMELY IMPORTANT* as otherwise any power
event has the potential of silent corruption which is catastrophic --
especially on an encrypted volume!

That same machine has two other 730s running a Postgresql database (also
Geli-encrypted) and they're just fine in terms of their wear leveling
and such; the media "wearout" indicator shows that 95% of the device's
life remains and they currently have 10,000 power-on-hours.

They'll wear out in something like another 20 years at present use
rates.... :)

The 480MB version of that drive is currently available for roughly
$250. It is not the fastest SSD out there but the differences between
it and others are small and I have *verified* that the power-loss data
protection works on these units. IMHO they're the only "consumer" style
priced devices that I find acceptable for this reason; the S3500/S3700s
are good too, but a hell of a lot more money and unless you need the
write endurance IMHO not worth it.

The 730 series hits the sweet spot in that it has power-loss protection
that *works* and yet they're reasonably priced. I own a bunch of them;
they're in my production servers under FreeBSD and also on my Win10
desktop machine.

--
Karl Denninger
***@denninger.net <mailto:***@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

Nikos Kastanas

2016-07-22 18:44:18 UTC

Permalink

Post by Karl Denninger

Post by Nikos Kastanas
I have a Lenovo X220 laptop running FreeBSD 10.3-RELEASE with ZFS and
encryption on a plain HDD. I am considering buying a Samsung Pro 850 SSD

Post by Karl Denninger

Post by Nikos Kastanas
boost performance but I am not sure if TRIM and ZFS+Encryption work well
together. After some research online, I found *this page*
<https://www.freebsd.org/doc/faq/all-about-zfs.html>which states the
*Note: *
ZFS TRIM may not work with all configurations, such as a ZFS filesystem

Post by Karl Denninger

Post by Nikos Kastanas
a GELI-backed device.
From what I can understand from the above note, I should not use the
encryption option when installing FreeBSD with ZFS on an SSD. TRIM will

not

Post by Karl Denninger

Post by Nikos Kastanas
work correctly and therefore the SSD performace will be impacted.

Meh. Simply not true. The reason for the "supported feature" flag here
is that this machine was recently rolled forward to 11.0-BETA1, but I
have not upgraded the pools yet from the feature set of 10.2.
pool: zsr
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not
support
the features. See zpool-features(7) for details.
scan: scrub repaired 0 in 0h6m with 0 errors on Sun Jul 17 03:12:01 2016
NAME STATE READ WRITE CKSUM
zsr ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da8p4.eli ONLINE 0 0 0
da9p4.eli ONLINE 0 0 0
errors: No known data errors
=> 34 468862061 da8 GPT (224G)
34 2014 - free - (1.0M)
2048 1024 1 freebsd-boot (512K)
3072 1024 - free - (512K)
4096 20971520 2 freebsd-zfs [bootme] (10G)
20975616 134217728 3 freebsd-swap (64G)
155193344 313667584 4 freebsd-zfs (150G)
468860928 1167 - free - (584K)
da8: <ATA INTEL SSDSC2BP24 0420> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number BTJR41210025240AGN
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 228936MB (468862128 512 byte sectors)
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
vfs.zfs.vdev.trim_max_pending: 10000
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.trim_on_init: 1
kstat.zfs.misc.zio_trim.failed: 0
kstat.zfs.misc.zio_trim.unsupported: 25748
kstat.zfs.misc.zio_trim.success: 6120223
kstat.zfs.misc.zio_trim.bytes: 295371051008
And as you can see, TRIM is definitely working (on the devices that can
handle it); there are also spinning rust disks in this machine, thus the
"unsupported" reports as well.
HOWEVER, I do suggest (strongly!) that you NOT use the particular SSD
you are intending to buy as it has no power-loss protection. Instead,
buy an Intel 730-series drive (that's what's in this machine); it has
that protection and it is *EXTREMELY IMPORTANT* as otherwise any power
event has the potential of silent corruption which is catastrophic --
especially on an encrypted volume!
That same machine has two other 730s running a Postgresql database (also
Geli-encrypted) and they're just fine in terms of their wear leveling
and such; the media "wearout" indicator shows that 95% of the device's
life remains and they currently have 10,000 power-on-hours.
They'll wear out in something like another 20 years at present use
rates.... :)
The 480MB version of that drive is currently available for roughly
$250. It is not the fastest SSD out there but the differences between
it and others are small and I have *verified* that the power-loss data
protection works on these units. IMHO they're the only "consumer" style
priced devices that I find acceptable for this reason; the S3500/S3700s
are good too, but a hell of a lot more money and unless you need the
write endurance IMHO not worth it.
The 730 series hits the sweet spot in that it has power-loss protection
that *works* and yet they're reasonably priced. I own a bunch of them;
they're in my production servers under FreeBSD and also on my Win10
desktop machine.
--
Karl Denninger
/The Market Ticker/
/[S/MIME encrypted email preferred]/

Thank you for your answer. So I guess the warning in the FAQ is probably

outdated.

I will seriously consider your suggestion considering the Intel SSD.

Thank you for your help

Eric A. Borisch

2016-07-22 19:02:37 UTC

Permalink

Post by Karl Denninger

Post by Nikos Kastanas
I have a Lenovo X220 laptop running FreeBSD 10.3-RELEASE with ZFS and
encryption on a plain HDD. I am considering buying a Samsung Pro 850 SSD

Post by Nikos Kastanas
a GELI-backed device.
From what I can understand from the above note, I should not use the
encryption option when installing FreeBSD with ZFS on an SSD. TRIM will

not

Post by Nikos Kastanas
work correctly and therefore the SSD performace will be impacted.

Meh. Simply not true.

It will not work on 10.3, but will work (as Karl demonstrates) on
11.x. Here's the commit to head enabling it:

https://svnweb.freebsd.org/base?view=revision&revision=286444

And here's what is in 10.3 (BIO_DELETE case returns EOPNOTSUPP):

https://svnweb.freebsd.org/base/releng/10.3/sys/geom/eli/g_eli.c?revision=296373&view=markup#l319

- Eric

Nikos Kastanas

2016-07-22 19:14:49 UTC

Permalink

Post by Karl Denninger

Post by Nikos Kastanas
I have a Lenovo X220 laptop running FreeBSD 10.3-RELEASE with ZFS
and encryption on a plain HDD. I am considering buying a Samsung Pro
850 SSD

Post by Nikos Kastanas
boost performance but I am not sure if TRIM and ZFS+Encryption work
well together. After some research online, I found *this page*
<https://www.freebsd.org/doc/faq/all-about-zfs.html>which states the
*Note: *
ZFS TRIM may not work with all configurations, such as a ZFS
filesystem

Post by Nikos Kastanas
a GELI-backed device.
From what I can understand from the above note, I should not use the
encryption option when installing FreeBSD with ZFS on an SSD. TRIM
will

not

Post by Nikos Kastanas
work correctly and therefore the SSD performace will be impacted.

Meh. Simply not true.

It will not work on 10.3, but will work (as Karl demonstrates) on 11.x.
https://svnweb.freebsd.org/base?view=revision&revision=286444

https://svnweb.freebsd.org/base/releng/10.3/sys/geom/eli/g_eli.c?revision=296373&view=markup#l319

- Eric

Thank you for your answer and the clarification.

Karl Denninger

2016-07-22 19:27:11 UTC

Permalink

Post by Nikos Kastanas

Post by Nikos Kastanas
I have a Lenovo X220 laptop running FreeBSD 10.3-RELEASE with

ZFS and

Post by Nikos Kastanas
encryption on a plain HDD. I am considering buying a Samsung Pro

850 SSD to

Post by Nikos Kastanas
boost performance but I am not sure if TRIM and ZFS+Encryption

work well

Post by Nikos Kastanas
together. After some research online, I found *this page*
<https://www.freebsd.org/doc/faq/all-about-zfs.html>which states the
*Note: *
ZFS TRIM may not work with all configurations, such as a ZFS

filesystem on

Post by Nikos Kastanas
a GELI-backed device.
From what I can understand from the above note, I should not use the
encryption option when installing FreeBSD with ZFS on an SSD.

TRIM will not

Post by Nikos Kastanas
work correctly and therefore the SSD performace will be impacted.

Meh. Simply not true.
It will not work on 10.3, but will work (as Karl demonstrates) on
https://svnweb.freebsd.org/base?view=revision&revision=286444
https://svnweb.freebsd.org/base/releng/10.3/sys/geom/eli/g_eli.c?revision=296373&view=markup#l319
- Eric

Note that the system in question (from which the stats were pulled) was
on 10.2 for an extended period of time, with SSDs, and with
Geli-encrypted disks. It was fine with no performance issues; whether
there is a problem with earlier releases has much to do with the disks
in question.

In the case of the Intel 730s it works perfectly well even though TRIM
is not passed through in that case.

--
Karl Denninger
***@denninger.net <mailto:***@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

Eric A. Borisch

2016-07-22 20:10:00 UTC

Permalink

Post by Karl Denninger

Post by Nikos Kastanas

Post by Nikos Kastanas
I have a Lenovo X220 laptop running FreeBSD 10.3-RELEASE with

ZFS and

Post by Nikos Kastanas
encryption on a plain HDD. I am considering buying a Samsung Pro

850 SSD to

Post by Nikos Kastanas
boost performance but I am not sure if TRIM and ZFS+Encryption

work well

filesystem on

Post by Nikos Kastanas
a GELI-backed device.
From what I can understand from the above note, I should not use the
encryption option when installing FreeBSD with ZFS on an SSD.

TRIM will not

Post by Nikos Kastanas
work correctly and therefore the SSD performace will be impacted.

Note that the system in question (from which the stats were pulled) was
on 10.2 for an extended period of time, with SSDs, and with
Geli-encrypted disks. It was fine with no performance issues; whether
there is a problem with earlier releases has much to do with the disks
in question.
In the case of the Intel 730s it works perfectly well even though TRIM
is not passed through in that case.

Fair, but the original question was if "TRIM will not work correctly
and therefore the SSD performace will be impacted" -- and the answer
is that TRIM+GELI does not "work correctly" for 10.3, but it does for
11.x. This is only a performance (and not "is my data safe")
statement.

As you allude to, how much this impacts performance depends on the
drive, partitioning / provisioning, and workload.

- Eric

Karl Denninger

2016-07-22 20:51:19 UTC

Permalink

Post by Eric A. Borisch

Post by Karl Denninger

Post by Nikos Kastanas

Post by Nikos Kastanas
I have a Lenovo X220 laptop running FreeBSD 10.3-RELEASE with

ZFS and

Post by Nikos Kastanas
encryption on a plain HDD. I am considering buying a Samsung Pro

850 SSD to

Post by Nikos Kastanas
boost performance but I am not sure if TRIM and ZFS+Encryption

work well

filesystem on

Post by Nikos Kastanas
a GELI-backed device.
From what I can understand from the above note, I should not use the
encryption option when installing FreeBSD with ZFS on an SSD.

TRIM will not

Post by Nikos Kastanas
work correctly and therefore the SSD performace will be impacted.

Note that the system in question (from which the stats were pulled) was
on 10.2 for an extended period of time, with SSDs, and with
Geli-encrypted disks. It was fine with no performance issues; whether
there is a problem with earlier releases has much to do with the disks
in question.
In the case of the Intel 730s it works perfectly well even though TRIM
is not passed through in that case.

Fair, but the original question was if "TRIM will not work correctly
and therefore the SSD performace will be impacted" -- and the answer
is that TRIM+GELI does not "work correctly" for 10.3, but it does for
11.x. This is only a performance (and not "is my data safe")
statement.
As you allude to, how much this impacts performance depends on the
drive, partitioning / provisioning, and workload.
- Eric

Note the two-part original statement-of-claim.

1. TRIM will not work on GELI encrypted disks before 11.x (true)
2. Performance will be severely impacted due to #1 (not necessarily and,
with proper SSD selection, generally false)

This is far more dependent on the SSD in question than the workload.
The machine from which those stats were pulled, now on 11.0-BETA1, was
on 10.2-STABLE for an extended period of time and has production loads
on it, including quite-heavy Postgresql DBMS use with a lot of table
write activity.

I never saw any evidence of performance degradation at all over that
time, and it has run with GELI-encrypted providers since I stuffed
AESNI-equipped processors in that particular machine (making FDE
reasonable in terms of CPU overhead) a few years back.

It just happens to be that one of the good choices in SSD selection from
a standpoint of not having performance go to hell correlates with your
data being safe as well.

I'm sure there are corner cases where #2 is true even with a
carefully-considered SSD selection but in the time I've been using the
Intel 730s I've not run into them.

The big benefit of TRIM is that by immediately notifying the drive of a
"freed" block it is able to choose how and when to consolidate and
pre-erase said blocks for later use. Since NVRAM has to be erased in
fairly large blocks (frequently 4MB in size or even more) to be
re-written and the erase cycle is, relatively speaking, somewhat slow
being able to do this "in advance" of the demand to write a block can
bring quite-material performance improvements. In addition, due to the
relatively large unit of erasure vis-a-vis the typical allocation size
on a file system there are a lot of other storage management
considerations that come into play; being able to do some or all of that
before the user demands a new block be written can be of great benefit
to performance. But exactly how much this matters is dependent on many
things with one of the most-important being how well the microcode in
the drive is written in the first place......

I suspect the odds of running into performance trouble in the OP's
environment, that being a laptop-style computer for what are probably
"personal" (as opposed to "server") sort of workloads, if the OP uses
one of the Intel 730s as the SSD in question, approaches zero -- and of
course somewhere down the road the OP will probably roll forward to 11.x.

--
Karl Denninger
***@denninger.net <mailto:***@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/