Ubuntu on ZFS grub/boot failure

[MAFoElffen]

I'm back. Busy week at work. Worked until late last nigh. The had to work on my truck, to get is ready for work for this week.

I was thinking about htis in the back of my head while I was working...

First do snapshots. recursively using the "-r" flag. The datasets yu want to make sanpshots of are bpool/boot, rpool/ROOT, and rpool/USERDATA. Then send/receive, replicating the whole datasets using the "-R" flag... To an external disk. That way you can snapshot and restore without carrying any of set pool options that are causing this problem.

I mount my backup disk as a pool called "backups", mounted at a directory called /backups...

sudo su --
zfs snapshot -r bpool/BOOT@migrate
zfs snapshot -r rpool/ROOT@migrate
zfs snapshot -r rpool/USERDATA@migrate
zfs send -R bpool/BOOT@migrate > /backups/bpool-boot.migrate
zfs send -R rpool/ROOT@migrate > /backups/rpool-root.migrate
zfs send -R rpool/USERDATA@migrate > backups/rpool-userdata.migrate

This will be the easiest way to migrate to a newer larger disk, and not carry over the problem...

First do this to find the name of the disk:

ls -e7 -o name,size,model

For example sake, lets just say it turned up as "sdb". We then use that value to find the unique disk ID

ls -l /dev/disk/by-id | grep -e 'sdb'

Lets say it came back like this

lrwxrwxrwx 1 root root  9 Feb 18 10:26 ata-Samsung_SSD_870_QVO_2TB_S6R4NJ0R624350W -> ../../sdb

We then use that value, to create a reusable variable, so we don't have to retype it each time we need to use it

DISK=/dev/disk/by-id/ata-Samsung_SSD_870_QVO_2TB_S6R4NJ0R624350W

Then we create new partitions on that disk. This is what the canned Ubuntu install does by rule of thumb:

NAME    LABEL                     SIZE     FSTYPE    MOUNTPOINT
sda                               <whatever size>         
 +-sda1                           512M     vfat      /boot/efi
 +-sda2                             2G     swap      [SWAP]
 +-sda3 bpool                       2G     zfs_mem 
 +-sda4 rpool                    <balance> zfs_mem

I use different sizes for things to correct size issues for what I do:

I set EFI at between 750M to 1G, depending if I am going to add any EFI apps.
I set swap at 1.5 to 2 times the RAM
I set bpool at 3G to allow room for snaphots
I give what balance to rpool, or whatever I decide as appropriate (I usually reserve 10%-20% as reserve for emergencies).

That would translate to someting like this, adjusting for your planned needs:

sudo su --
## Add partitions
# For ESP EFI
sgdisk     -n1:1M:1G      -c1:EFI    -t1:EF00 $DISK

# For Linux Swap
sgdisk     -n2:0:+32G   -c2:swap   -t2:BF02 $DISK

# For bpool boot
sgdisk     -n3:0:+3G     -c3:bpool  -t3:BE00 $DISK

# For rpool root
sgdisk     -n4:0:0   -c4:rpool  -t4:BF00 $DISK

Then we are going to do a few things...

copy over the EFI for old to new:

sudo dd if=/dev/sda1 of=/dev/sdb11

The easiest way to migrate rpool, is to attach ${DISK}-part4 to rpool, (will create it as a mirror of the old). After it finishes resilvering, then "detach" the old partion then remove the old partition, then set "autexpand=on"...

There is actually 3 wyas to do that, with either add/remove, replace or attach/detach. I find with people, that attach/detach has a better success rate, and you can verify it before the old is removed...

The commands are going to look something like this

zfs attach ${DISK}-part4 rpool
zpool status -v rpool

Until it is done resilvering, then it will look similar to this

  pool: rpool
 state: ONLINE
config:

    NAME                                           STATE     READ WRITE CKSUM
    rpool                                          ONLINE       0     0     0
      mirror-0                                     ONLINE       0     0     0
      5d133589-303b-1940-80fc-568124015599         ONLINE       0     0     0
      ata-Samsung_SSD_870_QVO_2TB_S6R4NJ0R624350W  ONLINE       0     0     0 

errors: No known data errors

So then you are going to detach the old pool

zfs detach 5d133589-303b-1940-80fc-568124015599 rpool

For bpool, it going to be a bit different... First, we capture the UUID variable thta was used during its origianl install...

UUID=$(zfs list | awk -F "_" '/bpool\/BOOT\/ubuntu_/ {print $2}' | awk '{print $1}')

Then rename the old bpool, so it doesnt conflist wiht the new one, so we still have access to both

umount /boot/efi
zpool rename bpool bpool-bak
zpool set mouuntpoint=/mnt bpool-bak

Then create the new bpool, with the new options:

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -o cachefile=/etc/zfs/zpool.cache \
    -o feature@async_destroy=enabled \
    -o feature@empty_bpobj=active \
    -o feature@lz4_compress=active \
    -o feature@multi_vdev_crash_dump=disabled \
    -o feature@spacemap_histogram=active \
    -o feature@enabled_txg=active \
    -o feature@hole_birth=active \
    -o feature@extensible_dataset=disabled \
    -o feature@embedded_data=active \
    -o feature@bookmarks=disabled \
    -o feature@filesystem_limits=disabled \
    -o feature@large_blocks=disabled \
    -o feature@large_dnode=disabled \
    -o feature@sha512=disabled \
    -o feature@skein=disabled \
    -o feature@edonr=disabled \
    -o feature@userobj_accounting=disabled \
    -o feature@encryption=disabled \
    -o feature@project_quota=disabled \
    -o feature@device_removal=disabled \
    -o feature@obsolete_counts=disabled \
    -o feature@zpool_checkpoint=disabled \
    -o feature@spacemap_v2=disabled \
    -o feature@allocation_classes=disabled \
    -o feature@resilver_defer=disabled \
    -o feature@bookmark_v2=disabled \
    -o feature@redaction_bookmarks=disabled \
    -o feature@redacted_datasets=disabled \
    -o feature@bookmark_written=disabled \
    -o feature@log_spacemap=disabled \
    -o feature@livelist=disabled \
    -o feature@device_rebuild=disabled \
    -o feature@zstd_compress=disabled \
    -o feature@draid=disabled \
    -o feature@zilsaxattr=disabled \
    -o feature@head_errlog=disabled \
    -o feature@blake3=disabled \
    -o feature@block_cloning=disabled \
    -o feature@vdev_zaps_v2=disabled \
    -o compatibility=grub2,ubuntu-22.04 \
    -O devices=off \
    -O acltype=posixacl \
    -O xattr=sa \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off -O mountpoint=/boot -R /mnt \
    bpool ${DISK}-part3
    
zfs create -o canmount=off -o mountpoint=none bpool/BOOT
zfs create -o mountpoint=/boot bpool/BOOT/ubuntu_$UUID

ls /mnt  # To verify it is there...

Then copy (rsync) over the filesystem from old-to-new, or restore via send/recieve from the dataset snapshot backup from your recovery pool.

rsync would look like this

rsync -r /boot/ /mnt/boot

send/receive would look like this... Note that the dataset cannot exist before it is restored from full...)

zfs destroy bpool/BOOT/ubuntu_$UUID
zfs destroy bpool/BOOT
zfs receive -F -d bpool/BOOT < /backups/bpool-boot.migrate

[MAFoElffen]

PART 2 -

That way, the old bpool and rpool still exist... The new ones are populated...

Then you would intialize/format the swap partition, then update the /mnt/etc/fstab file to point it to the new EFI partition on that drive to mount to /boot/efi.

Export the old bpool-old pool. Export the new bpool, then import it, which will then move the mountpoint from /mnt to /boot.


Reinstall grub2, update grub, then re-gen the initramfs images...

Exit the chroot, umount and export all. Shutdown and remove the old drive...

Test.

I think I covered everything. Maybe 1fallen can look it over to see if I skipped over anything. Or add any clarifications.

[dragonomnus]

Thanks for the update.
Reading through this reminded me I forgot to mention 1 other issue I encountered when following your prior steps, being the exit process was not completing correctly.
The part at the end where you instructed to exit chroot, then "mount | grep" etc, then "zpool export -a"... If failed with "device busy".
I shall need to test it again, but IIRC, the exit succeeds but both the umount & zpool export would fail with "device busy".
I tried repeating that process but did the umount part before the exit and I think it returned no error, but the zpool exit always returned "device busy".
I am not certain if that is part of the issue, or an error on my part (95% certain I followed your instructions correctly), or some unrelated issue, but thought it worth mentioning as it could be a point of failure in this process.

Regarding the backup, I think I shall try to create the backup including any possible errors present.
Reasoning:
If ZFS send/receive omits copying problematic zpool options (though there should be none) because it sends only the dataset to a new pool instead of copying the pool, there would be no harm copying potential problems as part of the backup because I could use ZFS send to restore if needed. Basically, LIVE>BACKUP>RESTORE, it is irrelevant to RESTORE whether potential problems are skipped during LIVE>BACKUP or BACKUP>RESTORE, so copying them as part of LIVE>BACKUP could allow us to work more with the issues if needed and would cause no problem if later needed to restore from.
Regarding methods of migration/backup and considering an exact clone of the drive, my understanding is dd would copy the partitions exactly & entirely including all blank space, correct?
But gparted is able to skip copying blank space of partitions IF it can read the partition, which it cannot read ZFS (yet) but can read vfat of the EFI partition, so it should generally be better to use for cloning drives/partitions, correct?
Then I can use ZFS attach/detach or send/receive where appropriate for ZFS partitions.
I am not certain if I am misremembering what I read of those differences between dd/gparted or if the information was incorrect or what, so clarification there would be great.
While it would not apply as much to my present situation, there is a big difference in copy time (whether SSD or HDD) and SSD wear when you copy an entire partition including the 60% or 90% free space vs skipping the free space.

Regarding variables in Linux, if creating them is done as you indicated by merely typing "VNAME=string" then how do you delete them? Windows uses the SET command to manage varaibles and I can use a modifier to create a temporary variable which survives until I close the terminal session, is that how the variable created using the method you instructed functions? I need to learn more of Linux variables soon.

Regarding the partition sizes, they were created as you said Ubuntu defaults are, but what to adjust them to is another question.
I would see no need for increasing the size of the EFI partition, but I am not familiar with EFI apps. What extra software could you or would you want to install to EFI? Guess I shall need to research this also.
I seriously doubt I would want a 128G swap partition, and saw no problems with the default 2G even with 75% of my 64GB RAM filled at times (largely due to ZFS caching, but I shall add cache drives later). Unless you know some other benefit of a larger swap partition?
bpool becomes the biggest question. zfs-auto-snapshot filling bpool with old snapshots was a problem, but they were between 200M & 300M each so I would need to expand it to something more like 4G or 5G to store all it creates.
Alternatively, I would prefer to learn how to alter zfs-auto-snapshot in multiple ways, including altering its snapshot name scheme and its retention policy. If I can limit its monthly snapshot retention to 2 or 3 then 2G for bpool should be more than sufficient, the question is would that be better than expanding bpool to 4G or 5G to store an entire year of snapshots?
Regarding its naming scheme, it is needlessly lengthy & redundant. It prefixes snapshots with @zfs-auto-snap_ which is rather self-evident. If you are looking at the ZFS snapshots then you already know you are working with ZFS so starting with @zfs is useless while adding needless length, and same goes for the -snap part. The -auto part is self-evident when you look at the snapshots with names of _hourly/_daily/etc, so none of that prefixing should be there but I am yet to determine how to alter it, but that is also not relevant to the primary issues of this thread.

Regarding how to migrate, add/remove would not function for this because that zfs command adds the new drive as an additional vdev to the pool to expand the pool's capacity or add optional devices (SLOG, spare, etc), attach/detach would be required to clone the pool. Also, my understanding is zfs-remove can safely remove only optional vdevs, not vdevs comprising any part of the main storage, which some people call "a limitation of ZFS" but that same thing is true of all striped RAID arrays as far as I am aware with the exception of splitting a mirror which ZFS can easily do.

Regarding the part where you instruct to detach then rename the old bpool, that is 1 part of this I can offer a suggestion! Use "zfs split" instead because zfs-split splits mirrored drives leaving all intact and directly renames the split pools in 1 command. That said, I am yet to test how it functions with mirrors of more than 2 drives as it says it splits ALL mirrors and retains merely 1, but it could be poorly worded or possibly require naming each split. I shall need to re-read & play with it at some point using more than 2 drives, or use files as pools (purely to test & play, not for actual use).

Regarding creating the new bpool, is it necessary to specify all those options? I know some must be specified at pool creation as they cannot be changed later (ashift for example), but I sadly cannot copy the command on the PC in question because Firefox is not displaying correctly when booting from the LiveUSB on it. Firefox displayed correctly when I originally used the LiveUSB to install but perhaps the GPU I swapped in later is not compatible with some driver (or ?). Hrm, I wonder if this could also be an indication of a problem... It should not be an issue because Ubuntu functioned well with the new GPU, rebooted correctly, Firefox displayed correctly, etc. At least until the reboot which never rebooted... So many variables in this issue.
If those options must all be specified, that also creates extra issues. For example, I know some of those affect file permissions/timestamps/etc between Linux/Windows, and the primary purpose of this is to learn using Linux as a server for a typical Windows environment so I need to learn better which options there must be set to what to allow that to function correctly, but are you thinking that could also be part of the problem with why it is not booting? At least for bpool/rpool, I did a default Ubuntu on ZFS install so they should have Ubuntu's default options, whatever those are. I shall compare them later.
To clarify, I technically have 2 extra zpools for hosting files to the network for clients so I think the options for bpool & rpool can be set to what Linux needs or prefers with no issues. I think, feel free to correct as I am not certain here.

Regarding your zfs-send/receive, your instruction is different than what I read before. I read it needed to be "zfs send source | zfs receive destination", not accounting for options. I shall try it your method, but also wonder if there are differences/benefits to either method.

Finally, I never modified fstab before (FileSystem TABle, correct?), so guess I shall learn now.
I shall read through this again and try to learn it all before I try it, and shall post if I have further questions before I implement. Or to inform you of the results.

[1fallen]

PART 2 -
I think I covered everything. Maybe 1fallen can look it over to see if I skipped over anything. Or add any clarifications.

You did and very nicely in Post #11
That should get the OP back and running with fresh install and migrate back all needed pools.

I'm not sure what's up with his booting issue though, quite puzzling.

[dragonomnus]

Worked on this when I could over the last few days.
Seemingly achieved progress but the PC now fails differently.

I shall post later when I have more time to write out the changes to your instructions which were needed.
For example, "zpool rename" is impossible because there is no such option for the zpool command.

However, aside from again being unable to unmount all zpools at the end because some device is always busy (as I explained before), the entire process seemed to succeed this time.
Until I rebooted to attempt to boot to the new SSD install.
The prior boot issue was it began booting to the Ubuntu recovery menu, was unable to boot to the Ubuntu install, but could put me in grub.
The present issue is it boots directly to grub.
I used:

     set root=(hdX,gptX)
linux /BOOT/ubuntu_UUID/@/vmlinuz root=ZFS=rpool/ROOT/ubuntu_UUID boot=zfs
initrd /BOOT/ubuntu_UUID/@/initrd.img
boot

which seemingly began to boot the Ubuntu install, I saw a few fails but they passed too quickly, I shall try to get them later.
However, it dropped me directly to the command line of "root@mypc" instead of loading the GUI as normal.
The info given there is insufficient to know what I should do, but I tried multiple options ("exit", "systemctl reboot", pressing Control-D).
They either did effectively nothing or resulted:
"Starting default target
Failed to start default target: Transaction for graphical.target/start is destructive (emergency.target has 'start' job queued, but 'stop' is included in transaction)."

To summarize the present other info:
The original SSD should be intact as it was but is disconnected from the motherboard (in the case, though).
I copied the EFI & swap partition to an extra HDD, then mirrored bpool & rpool to it, so it should also be a perfect backup.
The new NVMe also has backups of bpool & rpool so I can re-mirror or re-send them to the active bpool & rpool if needed with no need to reconnect the old SSD or backup HDD.

I shall explain further after I sleep & have time, but that is the present summary.

[MAFoElffen]

Sorry command is "zfs rename". Sorry. Did triple shifts. Slept in my car between shifts at the new locations...

Got home, and got my truck ready for this week. which such be slower.

[dragonomnus]

Hey, sorry, was dealing with bigger issues last week.
Was your week slower as you expected?

Any thoughts on how to deal with that new boot issue I mentioned in my prior post?
I am curious what caused that new problem as much as how to solve it.
Is that a problem any person experiencing this new Ubuntu on ZFS boot failure would also experience when trying to fix the issue?
I can possibly try snapshot rollbacks to correct the problems but I think (need to verify) I have no snapshots on bpool prior to the kernel update.
But.... I am 98% certain Ubuntu wanted a reboot *after* the kernel update and it rebooted successfully, it was other odd failures a few days later which convinced me to reboot again and *that* reboot never rebooted.
That leaves me more curious, but it could be some other event I am forgetting or unaware of.
I definitely have older snapshots on rpool though so could possibly do a rollback pre-update on it, but I am not completely certain I am performing all these steps to reinstall grub through chroot correctly because they are not fixing the issue for me as it did you and because of not being able to completely cleanly unmount all datasets & pools after doing the various steps.

I should have time to work on it again later this week.
Either way, thanks for all your help & time thus far.

[MAFoElffen]

Actually... This week is slower. In fact tomorrow I'm on standby. I'll find ot about 7am if I have to rush somewhere.

Dang. Feast or famine. Just when I thought things were starting to get consistent.

[dragonomnus]

Been waiting nearly 2 months for instructions or advice regarding that last issue, but seems none shall be provided.
Thankfully I was mostly busy, but that also meant I had little time to work on it myself so it is yet to be fixed.
I began working on it again recently and was able to photograph some of the errors from the failed boot, uploaded to my Google Drive so I shall link that later.

The 1st error seems to be regarding the unknown command line parameters from the boot command, but those should be correct according to multiple other posts & such I read, and that /BOOT/ section seems to not function if I put bpool before it as I read is needed with the root=ZFS=rpool/ part.
No knowledge if that "Optimal transfer size" part is normal or a relevant issue, but the kernel taint errors I would guess are but no idea how to solve them.
I can see it is failing multiple occurrences of the /usr/bin/unshare part, but uncertain if it is trying to mount that folder on each of those devices or whether they are important errors.
I next see it reads from /dev/disk/by-uuid/9507-4A55 and starts & finishes an fsck on it, it being the gpt1 partition which holds /boot/efi.
Next possible issue I see is "systemd-udev-settle.service is deprecated", but I know not whether that is a problem or how to fix it (yet) if it is.
Next errors I see are "filesystem 'bpool/BOOT/ubuntu_j8657t' cannot be mounted" and "boot.mount: Failed with result 'exit-code'.", but why is this if it can read the efi partition & the boot command reads /BOOT/ from gpt3?
That seems to lead to the subsequent errors of failing to mount /boot and the following tasks which need it, correct?
After that come the sysinit.target errors, but I know nothing of it so shall try to research that some, but not certain if it is presently important or shall be fixed if we can solve the prior issue of being unable to mount /boot so advice here would be helpful.
After that are the results of blkid so you can see which partitions are on which devices & such.

https://drive.google.com/drive/folde...1V?usp=sharing

This is as much as I can presently do with what I know & was able to find, so further instruction or advice is needed if I am to solve this problem.
At this point, I could alternatively consider installing fresh but would need to also learn how to migrate installed software & OS configuration, or manually reinstall & modify everything which would not be easy because 3/4 of it required multiple attempts to convince to function because I know very little of Linux.

[MAFoElffen]

That is a lot of errors and warnings. Some that you think are errors are just warnings.

Remind me, which version of Ubuntu with which version of Grub2?