Ran into an zfs list 'hangs' issue today

[jasonbelec]

So I can list pools, no problem, pool status no problem, but trying to
list zfs filesystems just goes to infinity. Data dropped from mount
points on the system, but still shows to be on disk under pool info.

Tried just listing snapshots, should be thousands, just hangs.

Tried exporting the pool, says its busy even after a fresh restart of
the system. Even when forced.

I've mounted another pool and was thinking I'd send all the data from
A to B, kinda hard if you can't see what you have.

Never had this before, did a quick Google, nothing seems relevant and
I searched here to see if I had missed someone else's education.

Any ideas, suggestions? Come on, before I figure it out, its for a
thousand Jeopardy points!!!

[Alex Blewitt]

On 4 Apr 2011, at 21:41, jasonbelec wrote:

> So I can list pools, no problem, pool status no problem, but trying to
> list zfs filesystems just goes to infinity. Data dropped from mount
> points on the system, but still shows to be on disk under pool info.
>
> Tried just listing snapshots, should be thousands, just hangs.

Yeah, that's what I see when I have a significant number of filesystems. Regular destruction of temporary ones seems to be the order of the day to fix that problem.

Are you creating snapshots whilst this operation is happening? That could trigger a restart. What does the last few hundred lines of 'zpool history PoolName' show you?

Alex

[jasonbelec]

well 'zpool history PoolName' just showed me lots of
snapshots. :) ) )~

I turned said creation off. List does work but takes sometime. So I
will sort and nuke snapshots until its down to something sane. I guess
I now know yet another thing to be wary of! ROFL

First time this caught my attention, but it was the first time I tried
this much detail.

I'll work up a script to filter so I can destroy, yes that's the dark
side pulling me.......

Thanks for the the help Alex.

[Alex Blewitt]

I keep 24 hourly snapshots, 35 daily snapshots, 12 monthly snapshots and then annual ones.

I posted it to my blog a while ago:

http://alblue.bandlem.com/2008/11/crontab-generated-zfs-snapshots.html

Note that I had to fix the hourly generation to skip if a scrub was in progress, since if you (create|destroy) snapshots whilst a scrub is in progress on MacZFS74 it restarts the scrub. So I skip if zpool status | grep "progress" gives me a zero return code or some such. I should really go back and update that post ...

Alex

Sent from my iPhone 4

[Jason Belec]

Excellent, probably what I messed up, will go through it tonight!

Thanks

--

Jason Belec

Sent from my iPhone

[jasonbelec]

Wow, so it seems I found a few other people that have run into this on
FreeBSD, they thought 15-20 minutes was a long wait! Ha! Try 3
hrs!!!!!!

So I killed the script running under Launchd, nuked all the snapshots
after copying, that took a 28hrs.... ROFL

System is back to normal, luckily no errors occurred.

Looked at logs, time when things happen, and seems I created this
issue, as as Alex mentioned some things should not run at the same
time, ever under ZFS, apparently even posts on this list and my brain
do not desire to play together. So today I'll tweak the snapshot/
backup scripts to allow for a little more flexibility considering I'm
trying to run a snapshot every minute.

I do think I'll change things to only snapshot during work hours,
scrub after and then sync snaps to the other system. Hopefully that
solves the massive amount of snapshots killing the system issue....
well the zfs requests of the system issue anyway.

Hopefully anyone else will glean useful information from this.

Jason

On Apr 6, 6:33 pm, Jason Belec <jasonbe...@rogers.com> wrote:
> Excellent, probably what I messed up, will go through it tonight!
>
> Thanks
>
> --
>
> Jason Belec
>
> Sent from my iPhone
>

> On 2011-04-06, at 6:09 PM, Alex Blewitt <alex.blew...@gmail.com> wrote:
>
>
>
> > I keep 24 hourly snapshots, 35 daily snapshots, 12 monthly snapshots and then annual ones.
>
> > I posted it to my blog a while ago:
>
> >http://alblue.bandlem.com/2008/11/crontab-generated-zfs-snapshots.html
>
> > Note that I had to fix the hourly generation to skip if a scrub was in progress, since if you (create|destroy) snapshots whilst a scrub is in progress on MacZFS74 it restarts the scrub. So I skip if zpool status | grep "progress" gives me a zero return code or some such. I should really go back and update that post ...
>
> > Alex
>
> > Sent from my iPhone 4
>