Adding a New ZFS Encrypted Disk on Ubuntu

I recently got a second-hand SSD for my work computer and I wanted to make use of it by moving my home dataset to the new SSD while keeping it encrypted and using Ubuntu's boot time unlock feature. After a day of fiddling with it I got it to work but the solution wasn't particularly obvious so I thought I'd share it!

Hardware: x86_64
OS: Ubuntu 21.10
Kernel: 5.13.0-21-generic

Background

Recent Ubuntu releases have support for installing ZFS on the root filesystem using built-in ZFS encryption, but it works in a surprising way. A volume is created to hold the the key file and is available in /dev/zvol. That volume is formatted with LUKS. You can the format like this:

$ sudo cryptsetup luksDump /dev/zvol/rpool/keystore 
LUKS header information
Version:       	2
Epoch:         	3
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
...

During boot the typical process of looking for LUKS volumes takes place and this volume is unlocked and mounted on /run/keystore/rpool. Inside the mount you'll find the key used to unlock the ZFS datasets:

$ ls -l  /run/keystore/rpool
total 20
drwx------ 2 root root 16384 Nov 22 18:38 lost+found
-rw------- 1 root root    32 Nov 22 18:38 system.key

Looking at rpool you can see that it's using the key in that location:

$ zfs get keylocation rpool 
NAME   PROPERTY     VALUE                                  SOURCE
rpool  keylocation  file:///run/keystore/rpool/system.key  local

Creating an Encrypted Pool and Dataset

This is the easy part. Since we know were to find the system key we can just create a new pool using the key. These options are taken from this blog post about using ZFS encryption in Ubuntu 20.04

$ sudo zpool create -f \
                -o ashift=12 \
                -O compression=lz4 \
                -O acltype=posixacl \
                -O xattr=sa \
                -O relatime=on \
                -O normalization=formD \
                -O canmount=off \
                -O dnodesize=auto \
                -O sync=disabled \
                -O recordsize=1M \
                -O encryption=aes-256-gcm \
                -O keylocation=file:///run/keystore/rpool/system.key \
                -O keyformat=raw \
                newpool /path/to/device

If that worked you should be able to get the keystatus of the new pool and see that it's available:

$ zfs get keystatus newpool 
NAME     PROPERTY   VALUE        SOURCE
newpool  keystatus  available    -

Now you can create an encrypted dataset on that pool:

$ sudo zfs create newpool/newdata

Now you have an encrypted dataset! Unfortunately, it won't be remounted after a reboot. Figuring out how to do that was the harder part.

Making it Permanent

Mounting encrypted volumes is done using systemd. During early boot a script called zfs-mount-generator creates the Systemd units that are necessary to make sure that the key is available to encrypted volumes and that zfs load-key is called for each volume. This is the manual for zfs-mount-generator. zfs-mount-generator will not see your new pool unless you update its cache located in /etc/zfs/zfs-list.cache which is not done automatically. It seems you have to trick it into working. Here's how. Start by creating an empty file with your pool name:

$ sudo touch /etc/zfs/zfs-list.cache/newpool

Now trigger an event that will cause a zedlet to update the cached information in that file:

$ sudo zfs set canmount=on newpool

Verify that the newpool file has contents:

$ cat /etc/zfs/zfs-list.cache/newpool

All done! Your new pool will be automatically decrypted using the same key as your system and be ready after the next reboot.

Conclusion

I think it's a bug in Ubuntu that creation of files in /etc/zfs/zfs-list.cache isn't done automatically. For non-encrypted mounts they're not necessary so perhaps it's an oversight. I hope this helps you!