"zfs list -t snapshot" missing entries

[Aneurin Price]

Hello all,

I've been testing Darik Horn's Linux version of zfs-auto-snapshot
today. This is the first time I've tried using snapshots with zfs-fuse
(I'm using 0.7.0), and I'm confused about why 'zfs list' doesn't seem
to show all the snapshots made, unless they're listed explicitly.

For example:

$zfs list 'srv/video@zfs-auto-snap_frequent-2011-12-22-1400'
NAME USED AVAIL REFER
MOUNTPOINT
srv/video@zfs-auto-snap_frequent-2011-12-22-1400 0 - 3.83T -

So there's a snapshot named
'srv/video@zfs-auto-snap_frequent-2011-12-22-1400', but:

$zfs list -t snapshot|grep video
srv/video@zfs-auto-snap_hourly-2011-12-22-1117 0
- 3.79T -
srv/video@zfs-auto-snap_frequent-2011-12-22-1300 58K
- 3.82T -
srv/video@zfs-auto-snap_frequent-2011-12-22-1345 26K
- 3.82T -
srv/video@zfs-auto-snap_frequent-2011-12-22-1415 0
- 3.83T -

That snapshot isn't listed. What am I missing here?
The 14:00 snapshot should already have been automatically removed by
now, but zfs-auto-snapshot doesn't even know it's there since 'zfs
list' isn't listing it; I'm worried that I'm going to end up with
hundreds of snapshots hanging around that I can't remove without
guessing that they're there so I can name them explicitly.

Can anyone tell me what's happening?

Thanks,
Nye

[Darik Horn]

On Thu, Dec 22, 2011 at 09:21, Aneurin Price <aneuri...@gmail.com> wrote:
> I've been testing Darik Horn's Linux version of zfs-auto-snapshot
> today. This is the first time I've tried using snapshots with zfs-fuse
> (I'm using 0.7.0), and I'm confused about why 'zfs list' doesn't seem
> to show all the snapshots made, unless they're listed explicitly.

Use `zfs list -t all` or `zfs list -t snapshot` and you should get the
desired result.

By default, snapshots are no longer displayed in the `zfs list` output.

--
Darik Horn <daj...@vanadac.com>

[Aneurin Price]

On 22 December 2011 15:45, Darik Horn <daj...@vanadac.com> wrote:
> On Thu, Dec 22, 2011 at 09:21, Aneurin Price <aneuri...@gmail.com> wrote:
>> I've been testing Darik Horn's Linux version of zfs-auto-snapshot
>> today. This is the first time I've tried using snapshots with zfs-fuse
>> (I'm using 0.7.0), and I'm confused about why 'zfs list' doesn't seem
>> to show all the snapshots made, unless they're listed explicitly.
>
> Use `zfs list -t all` or `zfs list -t snapshot` and you should get the
> desired result.

Um, I already am - look at the examples I pasted in :P
(And apologies for the wrapping BTW; damn Gmail)

Thanks,
Nye

[Darik Horn]

On Thu, Dec 22, 2011 at 09:51, Aneurin Price <aneuri...@gmail.com> wrote:
> Um, I already am - look at the examples I pasted in :P
> (And apologies for the wrapping BTW; damn Gmail)

The zfs-auto-snapshot program for Linux never invokes a bare `zfs
list` command, so you need to explain why the default behavior of `zfs
list` is causing a problem. Both instances of it are qualified with
the `-t` parameter.

Snapshots are not deleted by zfs-auto-snapshot unless the `--keep`
parameter is given. Look at the bundled cron parts for an example of
how to invoke zfs-auto-snapshot properly.

--
Darik Horn <daj...@vanadac.com>

[Darik Horn]

> Um, I already am - look at the examples I pasted in :P
> (And apologies for the wrapping BTW; damn Gmail)

BTW, if the stuff that you're pasting here is being stripped or
mangled, then please post it as an attachment, or put it into a github
issue here:

https://github.com/dajhorn/zfs-auto-snapshot/issues/new

--
Darik Horn <daj...@vanadac.com>

[Aneurin Price]

On 22 December 2011 16:04, Darik Horn <daj...@vanadac.com> wrote:
> On Thu, Dec 22, 2011 at 09:51, Aneurin Price <aneuri...@gmail.com> wrote:
>> Um, I already am - look at the examples I pasted in :P
>> (And apologies for the wrapping BTW; damn Gmail)
>
> The zfs-auto-snapshot program for Linux never invokes a bare `zfs
> list` command, so you need to explain why the default behavior of `zfs
> list` is causing a problem.  Both instances of it are qualified with
> the `-t` parameter.
>

I think you've misunderstood me; this isn't a problem with
zfs-auto-snapshot, so far as I can tell - I just mentioned it for
completeness in case it turns out to be relevant. I'll restate the
most relevant parts which might help to clarify:

There exists a snapshot named
'srv/video@zfs-auto-snap_frequent-2011-12-22-1400', which can be
demonstrated by giving it explicitly as an argument to 'zfs list'.
(This is just an example; there are other snapshots with the same
issue.)

Calling 'zfs list -t snapshot' (or indeed 'zfs list -H -t snapshot -S
creation -o name' as zfs-auto-snapshot does) shows a list of several
snapshots, *but not all of them*. In particular,
'srv/video@zfs-auto-snap_frequent-2011-12-22-1400' is not listed even
though it's verified to exist.

Does that make it clearer?

[Darik Horn]

On Thu, Dec 22, 2011 at 10:18, Aneurin Price <aneuri...@gmail.com> wrote:
>
> I think you've misunderstood me; this isn't a problem with
> zfs-auto-snapshot, so far as I can tell - I just mentioned it for
> completeness in case it turns out to be relevant. I'll restate the
> most relevant parts which might help to clarify:
>
> There exists a snapshot named
> 'srv/video@zfs-auto-snap_frequent-2011-12-22-1400', which can be
> demonstrated by giving it explicitly as an argument to 'zfs list'.
> (This is just an example; there are other snapshots with the same
> issue.)
>
> Calling 'zfs list -t snapshot' (or indeed 'zfs list -H -t snapshot -S
> creation -o name' as zfs-auto-snapshot does) shows a list of several
> snapshots, *but not all of them*. In particular,
> 'srv/video@zfs-auto-snap_frequent-2011-12-22-1400' is not listed even
> though it's verified to exist.
>
> Does that make it clearer?

Yes, my bad.

Please create a recursive snapshot on the pool like this:

# zfs snapshot -r srv@test-recursive

And a regular snapshot like this:

# zfs snapshot srv@test-nonrecursive

Do they both appear in the snapshot list?

# zfs list -t snapshot | grep @test

--
Darik Horn <daj...@vanadac.com>

[Aneurin Price]

On 22 December 2011 16:38, Darik Horn <daj...@vanadac.com> wrote:
> On Thu, Dec 22, 2011 at 10:18, Aneurin Price <aneuri...@gmail.com> wrote:
>>
>> I think you've misunderstood me; this isn't a problem with
>> zfs-auto-snapshot, so far as I can tell - I just mentioned it for
>> completeness in case it turns out to be relevant. I'll restate the
>> most relevant parts which might help to clarify:
>>
>> There exists a snapshot named
>> 'srv/video@zfs-auto-snap_frequent-2011-12-22-1400', which can be
>> demonstrated by giving it explicitly as an argument to 'zfs list'.
>> (This is just an example; there are other snapshots with the same
>> issue.)
>>
>> Calling 'zfs list -t snapshot' (or indeed 'zfs list -H -t snapshot -S
>> creation -o name' as zfs-auto-snapshot does) shows a list of several
>> snapshots, *but not all of them*. In particular,
>> 'srv/video@zfs-auto-snap_frequent-2011-12-22-1400' is not listed even
>> though it's verified to exist.
>>
>> Does that make it clearer?
>
> Yes, my bad.
>
> Please create a recursive snapshot on the pool like this:
>
>  # zfs snapshot -r srv@test-recursive
>

Hmm, now I have a sinking feeling:

cannot create snapshot 'srv/broken@test-recursive': dataset is busy
no snapshots were created

That dataset (srv/broken) has been sitting around for so long I'd
forgotten about it. It can't be destroyed without crashing zfs :(.

Everything scrubs just fine, but now I'm wondering if the problem is
simply that my pool is borked.

> And a regular snapshot like this:
>
>  # zfs snapshot srv@test-nonrecursive
>
> Do they both appear in the snapshot list?
>
>  # zfs list -t snapshot | grep @test
>

The non-recursive version works, and shows up.
I guess it may not be relevant in light of that fact that the pool
seems to be damaged, but I did try the following:
# zfs snapshot -r srv/video@test-recursive
# zfs snapshot srv/video@test-nonrecursive

# zfs list -t snapshot|grep @test

srv/video@test-recursive 0
- 3.83T -

All the snapshots that aren't showing up were made with '-r' though,
so I don't think there's any correlation between it being a recursive
snapshot or not. So far it seems frustratingly random.

I take it you've not seen anything like this before - any ideas short
of destroying and recreating the pool?

Thanks for your time,
Nye

[Emmanuel Anne]

Well eventually the estimation of how many snapshots were created with their names could help to see if the problem is because there were too many snapshots (?) or because of something else.
(and subvolumes also maybe).

The ideal would be to be able to reproduce the creation of all these snapshots on a disk image to see what happens (with a script ?).

2011/12/22 Aneurin Price <aneuri...@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

--
my zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

[Aneurin Price]

On 22 December 2011 17:23, Emmanuel Anne <emmanu...@gmail.com> wrote:
> Well eventually the estimation of how many snapshots were created with their
> names could help to see if the problem is because there were too many
> snapshots (?) or because of something else.
> (and subvolumes also maybe).
>

There are only a few dozen, so that shouldn't be a problem. Since it's
not causing me any immediate trouble, I think I'll leave it until
after Christmas to investigate further.

(Actually, I might see what zfsonlinux does, since that should be
fairly quick and painless. I'll report back if it does anything
different.)

Nye

[Emmanuel Anne]

Just be careful to test zfs on linux first on a separate pool, just to be on the safe side.

2011/12/22 Aneurin Price <aneuri...@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Seth Heeren]

On Dec 22, 4:21 pm, Aneurin Price <aneurin.pr...@gmail.com> wrote:

> Can anyone tell me what's happening?
>
> Thanks,
> Nye

No this behaviour isn't known to me; I have used snapshots very
intensively (with my own 5-minute auto-snapshot service on linux) for
a long time, though admittedly not since upgrading to 0.7.0. Never had
any trouble with that (though, unrelated: the reason I stopped doing
the many many snapshots was that sending/receiving incremental
datasets was getting extremely slow on the receiving side; creating
many - empty - snapshots on the receiving end takes several seconds
_per dataset_ _per snapshot_ regardless of whether it contains any
data. Annoying when you are replaying them. Luckily, the performance
is as pathetic on Solaris).

You can see what really happens in terms of snapshots and possibly
temp snapshot holds (just a suspicion there) by doing

zpool history tank

to see the holds being acquired/released:

zpool history -i tank

This should also clearly tell you whether you are piling up those
snapshots that should have been expired. Can you analyse the history
to see whether you can get any more information from that?

I must admit, that it does remind me of the infamous put_nvlist bug we
had in 0.6.0 (which failed to grow the client buffer large enough when
receiving large server replies from zfs-fuse). Something like that
could be happening here too. I'll have a look if you can't find
anything helpful from the zpool history.

Regards
Seth

[Aneurin Price]

On 22 December 2011 18:49, Aneurin Price <aneuri...@gmail.com> wrote:
> (Actually, I might see what zfsonlinux does, since that should be
> fairly quick and painless. I'll report back if it does anything
> different.)

And it works! All the snapshots are listed by ZoL. I wonder if it's
just because it uses a newer version of ZFS (I haven't upgraded the
pool since I want to be able to go back to zfs-fuse).

And with this puzzle piece now I really am going to forget about it
until after Christmas.

[sgheeren]

Thanks for the other angle on this issue. I might have a a look at that
some of these days :)

[hornowitz]

Hello guys,

first of all thanks to Darik for his auto-snapshot work! It works quite nice on my home zol servers.

But I also have some old boxes around, that still run zfs-fuse (never touch a running... etc)...

So I thougt I will try and use dariks zfs-auto-snapshot script there as well and noticed the same problem as decribed above.

After some debugging I've found out, that it works perfectly if you remove the com.sun:auto-snapshot-desc='-' at the snapshot taking:

Original:

zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'hornvmzpool@zfs-auto-snap_daily-2012-10-29-0947'

Modified:

zfs snapshot -r 'hornvmzpool@zfs-auto-snap_daily-2012-10-29-0947'

Sounds like a bug to me, that zfs-fuse doesn't know how to handle the com.sun:auto-snapshot-desc='-' when the snapshot is taken.

I modified darik's script now and it looks like it is working. I am still testing.

Thanks all for your great work. I love having zfs on linux! (fuse or native...)

Bye,

Dominic