ZFS rollback snapshot general question

[Bobb Shires]

Recently upgraded my 14.4 installation to 15.1 and something went very wrong, so I decided to rollback the snapshot the installer script made for me (THANK YOU).

When I hit enter on the rollback command the screen went completely blank, there was some brief disk activity, and the keyboard was unresponsive (like no caps lock response etc).

I waited a few minutes and hit the power button, then powered back on and... it worked! Back to 14.4

My question is, I'm thinking that's not the expected behavior and I did something wrong? How and when should the ifs rollback command be invoked, and what should I expect to see?
I was logged in as me (regular user) and did su then zfs rollback. Thanks!

[Kevin P. Neal]

You did a zfs rollback of the current root filesystem on a running system?
I'm not surprised it went badly. I'm not sure the kernel expects filesystems
that programs are currently running out of to blow away programs with no
way to page any required pages back in, for example.

Normally when a running program, or indeed any open file, is deleted the
inode remains valid until the program ends/the file is closed. (That's why
the system call is "unlink()" and not "delete()".) A ZFS rollback would
blow away the inode. If that case isn't explicitly supported then it will
probably blow up on you.

After a rollback of the root filesystem (and /usr), what should the kernel
do about all the running programs including /sbin/init (which has a PID
of 1)? A rollback would result in probably all running programs being
killed, but that leaves no program left to do anything including ask what
shell to use for a single user mode command line.

Really, the rollback should have been blocked and not allowed.
--
Kevin P. Neal http://www.pobox.com/~kpn/
"14. Re-reading No. 13, I realize that it's quite possible I'm losing my
mind. I'm glad that for the most part I'm not aware it's happening."
-- from "20 things I'm thankful for": Fortune, Nov 29, 2004, page 230

[Edward Sanford Sutton, III]

On 6/25/26 18:11, Kevin P. Neal wrote:
> On Mon, Jun 22, 2026 at 12:57:23PM -0400, Bobb Shires wrote:
>> Recently upgraded my 14.4 installation to 15.1 and something went
very wrong, so I decided to rollback the snapshot the installer script
made for me (THANK YOU).
>>
>> When I hit enter on the rollback command the screen went completely
blank, there was some brief disk activity, and the keyboard was
unresponsive (like no caps lock response etc).
>>
>> I waited a few minutes and hit the power button, then powered back
on and... it worked! Back to 14.4

ZFS snapshots are nearly instant. ZFS rollbacks and snapshot deletions
are not; time it takes depends how much difference needs to be properly
accounted for.

>> My question is, I'm thinking that's not the expected behavior and I
did something wrong? How and when should the ifs rollback command be
invoked, and what should I expect to see?

Rolling back to a snapshot removes everything that came after it
(probably what you wanted). If you have ZFS boot environments and the
boot menu enabled then you can select which boot environment you want to
use and it switches you to the chosen state but does not blow away all
states after it to do so.

You should invoke a ZFS snapshot rollback when you need to revert a
dataset to a previous state and are okay with all newer changes to it
being lost. Once complete, What you should see is any snapshots and data
changes that were created after the snapshot you roll back should be
gone. Like with backups it is wise to make sure files are not in use
when changing files that are normally in use. For all datasets including
what you booted from when the system is installed on ZFS it is wise to
reboot into single user mode with `shutdown now` and then doing the
rollback to minimize anything running or launching while you work. Once
complete you would then want to reboot. It is also safe to do such
operation when booted from separate media to avoid anything being in use.

Boot environments are created by taking a snapshot of one (=shallow boot
environment) or several (=deep boot environment) ZFS datasets and then
promoting those snapshots to clones which makes them both able to be
edited. If you have a working boot environment, you can modify other
boot environments at will without breaking the current one.

Assuming you have boot environments, you likely wanted to pick a
different one at the boot menu and then destroy the broken one that
would have normally booted when not changing it at the menu. There was a
thread here recently about the issues with unintuitive naming but it
comes down to you need to know which one is which to properly pick what
to boot to and pick what to delete.

As as reminder, any data outside the boot environments datasets is not
managed by them; a software rollback without rolling back other data
files is not always compatible, particularly with many databases.

>> I was logged in as me (regular user) and did su then zfs rollback.
Thanks!
>
> You did a zfs rollback of the current root filesystem on a running
system?
> I'm not surprised it went badly. I'm not sure the kernel expects
filesystems
> that programs are currently running out of to blow away programs with no
> way to page any required pages back in, for example.
>
> Normally when a running program, or indeed any open file, is deleted the
> inode remains valid until the program ends/the file is closed.
(That's why
> the system call is "unlink()" and not "delete()".) A ZFS rollback would
> blow away the inode. If that case isn't explicitly supported then it will
> probably blow up on you.
>
> After a rollback of the root filesystem (and /usr), what should the
kernel
> do about all the running programs including /sbin/init (which has a PID
> of 1)? A rollback would result in probably all running programs being
> killed, but that leaves no program left to do anything including ask what
> shell to use for a single user mode command line.
>
> Really, the rollback should have been blocked and not allowed.

You do have good points about issues to consider but with applying that
same logic we should have to boot from separate media to upgrade too.

[Bobb Shires]

Hi Kevin... Yes, I knew I had done something incorrectly.
Thanks for the reminder.

[Kevin P. Neal]

On Fri, Jun 26, 2026 at 10:14:17AM -0400, Bobb Shires wrote:
> Hi Kevin... Yes, I knew I had done something incorrectly.
> Thanks for the reminder.

I meant it as an explanation of why it went south on you, that's all.

ZFS has too many footguns.

> On Thu, Jun 25, 2026, at 21:11, Kevin P. Neal wrote:
> > Really, the rollback should have been blocked and not allowed.

Seriously, this particular case really should have been stopped by the
kernel. And why did it give you a script/command without a warning that
it shouldn't be used on a running system? That's not very friendly.

The cases that always get me are the ones where I accidentally mount a
filesystem over top of another one, typically the root filesystem. That's
a bad day when I have to drive out to the datacenter because my server
can't reboot with / overlaid by some other dataset like, for example, a
root filesystem for a newer version of FreeBSD. Wow do I hate that.

And I hate that I keep finding more ways to make the same mistake. For
example, a "zfs send | zfs receive" of /usr/local to a new boot environment
will mount the new /usr/local over top of the already mounted /usr/local
and ruin my ongoing poudriere builds. Well, unless I remember to use the
"-u" option on the receiving end. Which I didn't remember on my upgrade
to 15.1. *sigh*

I have used zfs rollback before, but never in a case where the kernel
should have stopped me. You got unlucky. Myself, I find other ways to get
unlucky instead.

I like ZFS, but man is it easy to get into trouble with it.
--
Kevin P. Neal from "Crito" by Plato, 360 BCE:
"But why, my dear Crito, should we care about the opinion of the many?
Good men, and they are the persons who are worth considering, will
think of these things truly as they happened." - Socrates

[Edward Sanford Sutton, III]

On 6/26/26 20:30, Kevin P. Neal wrote:
> On Fri, Jun 26, 2026 at 10:14:17AM -0400, Bobb Shires wrote:
>> Hi Kevin... Yes, I knew I had done something incorrectly.
>> Thanks for the reminder.
>
> I meant it as an explanation of why it went south on you, that's all.
>
> ZFS has too many footguns.
>
>> On Thu, Jun 25, 2026, at 21:11, Kevin P. Neal wrote:
>>> Really, the rollback should have been blocked and not allowed.
>
> Seriously, this particular case really should have been stopped by the
> kernel. And why did it give you a script/command without a warning that
> it shouldn't be used on a running system? That's not very friendly.
>
> The cases that always get me are the ones where I accidentally mount a
> filesystem over top of another one, typically the root filesystem. That's
> a bad day when I have to drive out to the datacenter because my server
> can't reboot with / overlaid by some other dataset like, for example, a
> root filesystem for a newer version of FreeBSD. Wow do I hate that.
>
> And I hate that I keep finding more ways to make the same mistake. For
> example, a "zfs send | zfs receive" of /usr/local to a new boot environment
> will mount the new /usr/local over top of the already mounted /usr/local
> and ruin my ongoing poudriere builds. Well, unless I remember to use the
> "-u" option on the receiving end. Which I didn't remember on my upgrade
> to 15.1. *sigh*

I avoid such misuse something like:

zfs send -LeRI older_snapshot pool_name@newer_snapshot|zfs recv -Fx
mountpoint -x sharesmb -x sharenfs -x checksum -x compression -x atime
-x refreservation -x readonly backup_pool/backups/pool_name

and restore with:

zfs send -bLeRv backup_pool/backups/pool_name@newer_snapshot|zfs recv
pool_name

this creates an override of each of the -x properties so that the
properties of the backup are what are actually used on the backup copy
and -b will use the non-overridden property. This makes it possible to
improve quality of life with things like backups getting a different
higher compression while also performing necessary adjustments like
overriding mountpoints to avoid the backup automounting on top of the
running system (a truly horrible thing to have ZFS do to its
unsuspecting users and needs big warnings added for in the docs; makes
ZFS users appreciate the days and ways of fstab).

Though a different issue, refreservation if set based on a pool's size
in just another example of how transferring between two different pools
could go anywhere from an annoyance to drastically wrong just because of
an overlooked ZFS setting. There may be more that are needs and wants to
override but this likely matched a lot of cases and made a good example.

I know my memory is not good enough to remember these commands properly
to guarantee I can avoid breaking things, the less important parts I'd
still prefer to get right without lots of slow reading, and I'd prefer
that even typos are not something I have to worry about. Such commands I
save in a file and I "carefully" copy+edit to avoid all these issues;
I'd use a script but I never took the time to write one that accounts
for the always-changing snapshot names without being error prone.

If I wasn't working from such notes and scripts, I would never include
`recv` in a backup and only use it on a restore do to the problems that
ZFS defaults during replication cause.