race condition with umem

[sgheeren]

Gents,

I wonder if anyone could have a look at the new info uncovered at issue #25.

There is a racecondition that crashes debug=2 builds of zfs-fuse
directly at launch, on certain setups. It has been established (see
comments) that this race can be suppressed by inserting 'magic sleeps'.
However, I'm currently at a loss as to what subsystems are interacting
in the first place.

Now I now that some of the guys (Emmanuel?) have been looking at
replacing the onnv-gate included version of libumem with 'stock' libumem
in the past? Perhaps they can look at this with their prior insight in
the working of libumem / it's integration with zfs-fuse.

--- some detail (head to the bug report for more:
http://zfs-fuse.net/issues/25)

Of note seems that the daemon SIGABRTs on the VM_SLEEP | VMC_IDENTIFIER
allocation for taskq_id_arena in taskq_init /reliably/. This appears to
be some kind of a special one, for some reason.

Further more my comments about static (shared/duplicated) data might
trigger someone with additional info. This is certainly not a threading
issue (well, not libpthread anyway).

Any help greatly appreciated.

Seth

[sgheeren]

relevant web link

http://zfs-fuse.sourcearchive.com/documentation/0.6.0~betaplus-p433snapshot/

(who put this there?)

[Emmanuel Anne]

Well I am almost sure I can't reproduce this with my repository because I build debug builds all the time and of course they never crash on me like that. And the guy uses a system which is very similar to mine (not even a multi processor which could have explained the thing).
So it's very very probably another difference between 0.6.x and my repository.

I'll try to check his backtrace later (to see if the line numbers match something usefull in my source) - but I won't cross check with the 0.6 source, too much trouble...

For the comment about disabling assertions in non debug builds : the idea is not from me, it's done like that in the public libumem release. Actually most of these checks are really "sanity" checks, and can safety be disabled once you have made sure that they work correctly in the debug builds.

And about the crash itself, libumem is very dangerous at startup, you are not allowed to call malloc (its malloc) while it is initializing but the problem is that it's initializing in the background. So if ever something calls malloc while it's initializing, you are dead.
I find it's a crazy architecture, but anyway we have to use it...

2010/2/11 sgheeren <sghe...@hotmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

--
zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

[sgheeren]

Emmanuel Anne wrote:
> I'll try to check his backtrace later (to see if the line numbers
> match something usefull in my source) - but I won't cross check with
> the 0.6 source, too much trouble...

Far too much trouble indeed, mostly since it never was reported on the
0.6.0 branch to begin with?!

>
> For the comment about disabling assertions in non debug builds : the
> idea is not from me, it's done like that in the public libumem
> release. Actually most of these checks are really "sanity" checks, and
> can safety be disabled once you have made sure that they work
> correctly in the debug builds.

+1

>
> And about the crash itself, libumem is very dangerous at startup, you
> are not allowed to call malloc (its malloc) while it is initializing
> but the problem is that it's initializing in the background. So if
> ever something calls malloc while it's initializing, you are dead.

Okay, so that's probably what's happening here anyway. Any doc on how to
prevent that race from happening?

> I find it's a crazy architecture, but anyway we have to use it...

Thanks for sharing these thoughts. It helps my understanding a bit

[Emmanuel Anne]

I might have spoken too fast here, even with Rudd-O latest source from his repository I was able to reproduce the problem only once !
After this, everytime I tried to launch the daemon, it worked...

(and with my sources never).

So...
1st it would be better to find a way to reproduce this more than once in a million times or so.
After that there are 2 ways to fix it :
1) move the libumem init outside this thread mess to be sure it will be initialized before anything else happens (probably the best way).
2) Add a sleep, or some kind of test to be sure libumem init is over.

I don't think changing the umem version we use would change anything to this... but it would be better if it was easier to reproduce this !

I'll try with the other computer then...

2010/2/12 sgheeren <sghe...@hotmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[sgheeren]

Emmanuel Anne wrote:
> I might have spoken too fast here, even with Rudd-O latest source from
> his repository I was able to reproduce the problem only once !
> After this, everytime I tried to launch the daemon, it worked...
>
> (and with my sources never).
>
> So...
> 1st it would be better to find a way to reproduce this more than once
> in a million times or so.

Did you read my notes in the issue? I have this setup documented.

> After that there are 2 ways to fix it :
> 1) move the libumem init outside this thread mess to be sure it will
> be initialized before anything else happens (probably the best way).

Erm.. Did you read my notes in the issue? There is no thread mess at the
point of SIGABRT. This is what stupified me.

> 2) Add a sleep, or some kind of test to be sure libumem init is over.

This is what I demonstrated to prove that timing is involved.

>
> I don't think changing the umem version we use would change anything
> to this... but it would be better if it was easier to reproduce this !

Nobody said so (I think). I, for one, am firmly _against_ dropping in a
different version of umem than what is used upstream. I asked for your
help (and other people too), because I think I recall you have looked
into libumem before...

>
> I'll try with the other computer then...

Ok, good luck!

[Emmanuel Anne]

Ok, I was able to reproduce it at will by using a remote host (and it was very very slow actually but it crashed everytime I launched zfs-fuse this way !).

So the easiest fix is simply to replace VM_SLEEP by VM_NOSLEEP in taskq_init. Anyway this just means that the memory allocation can go to sleep if the memory is not available when you pass VM_SLEEP, it might be usefull in kernel mode while some memory is freed, but here when you just start zfs-fuse it's probably totally uselss.

With just this single VM_SLEEP replaced, it works all the time.
Notice that I had to do this already in taskq_create_common (taskq.c too), there is a comment for zfs-fuse explaining why.

The only real issue here is that it's very hard to reproduce if you don't have the right machine for it !

2010/2/13 sgheeren <sghe...@hotmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[sgheeren]

Emmanuel Anne wrote:

Ok, I was able to reproduce it at will by using a remote host (and it was very very slow actually but it crashed everytime I launched zfs-fuse this way !).

So the easiest fix is simply to replace VM_SLEEP by VM_NOSLEEP in taskq_init. Anyway this just means that the memory allocation can go to sleep if the memory is not available when you pass VM_SLEEP, it might be usefull in kernel mode while some memory is freed, but here when you just start zfs-fuse it's probably totally uselss.

With just this single VM_SLEEP replaced, it works all the time.

Good thinking

Notice that I had to do this already in taskq_create_common (taskq.c too), there is a comment for zfs-fuse explaining why.

Even better thinking. Now why'd I not think of that myself :)

The only real issue here is that it's very hard to reproduce if you don't have the right machine for it !

It's always like that with timing issues.

Thanks again for thinking along. I think you can mark that issue as closed/up for retest? I will do it if you prefer

[Emmanuel Anne]

I don't think I have admin rights on this, so do it, I'll push my commit...

2010/2/13 sgheeren <sghe...@hotmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[sgheeren]

Bug closed (http://zfs-fuse.net/issues/25):

So I went and retested my own setup ($0.085 down the drain for less than
10 minutes :))

It works fine now.

I pushed (cherry-pick) the relevant commit to zfs-fuse.net/official
master branches. So pulling the official master should get you the fix.

Closing

Emmanuel Anne wrote:
> I don't think I have admin rights on this, so do it, I'll push my
> commit...
>

> 2010/2/13 sgheeren <sghe...@hotmail.com <mailto:sghe...@hotmail.com>>

>
> Emmanuel Anne wrote:
>> Ok, I was able to reproduce it at will by using a remote host
>> (and it was very very slow actually but it crashed everytime I
>> launched zfs-fuse this way !).
>>
>> So the easiest fix is simply to replace VM_SLEEP by VM_NOSLEEP in
>> taskq_init. Anyway this just means that the memory allocation can
>> go to sleep if the memory is not available when you pass
>> VM_SLEEP, it might be usefull in kernel mode while some memory is
>> freed, but here when you just start zfs-fuse it's probably
>> totally uselss.
>>
>> With just this single VM_SLEEP replaced, it works all the time.
> Good thinking
>
>> Notice that I had to do this already in taskq_create_common
>> (taskq.c too), there is a comment for zfs-fuse explaining why.
> Even better thinking. Now why'd I not think of that myself :)
>>
>> The only real issue here is that it's very hard to reproduce if
>> you don't have the right machine for it !
> It's always like that with timing issues.
>
> Thanks again for thinking along. I think you can mark that issue
> as closed/up for retest? I will do it if you prefer
> --
> To post to this group, send email to zfs-...@googlegroups.com

> <mailto:zfs-...@googlegroups.com>