[RFC][Re-Send] Fix SERVER-2114 (Don't use select() timeouts for fast coarse timing)

[Calvin Owens]

I'd really like to get this merged, but so far it's been ignored. I'm very open to discussing

this and doing further work on it if that is desired.

I also have a changeset that only effects Linux (where this is the biggest win), if that's more

acceptable, although IMHO eliminating the timeouts everywhere is better.

Pull request: https://github.com/mongodb/mongo/pull/416

Patch: https://http.snarkywidgets.com/mongo-timer-fix-v3.patch

If the community is totally opposed to this, can I at least hear that from somebody?

Thanks,

Calvin

[Alberto Lerner]

Calvin,

I don't think there is any opposition. But it's a tricky patch on at least two grounds.

First, making this kind of decision should be based on further evidence besides a single benchmark clobbered together in a laptop. Actual hypervisors are actually very good at detecting the guest os doesn't really need attention. Also, if you're spinning a mongod, how often is it going to stay without getting any request for longer than the time out? So getting more data about this things should make the effectiveness (or need) of this patch clearer.

The second point is that it is quite common not to block in a select and use the timeouts to, for instance, trigger timed callbacks (taks that are in the background that can be woken up when the server load is not hight). We don't exactly do this right now but when the timeout was put in place, it was also with this goal in mind.

There is also the point that that file is on the center of a couple of improvements we're planning for this cycle, related to connection handling (actually loss thereof). And there's only so much change that can be done before we start wondering if we're introducing one too many changes. 

Do those points seem reasonable?

Alberto.

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-dev?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Calvin Owens]

On Wednesday 05/08 at 15:25 -0400, Alberto Lerner wrote:
> Calvin, I don't think there is any opposition.
> But it's a tricky patch on at least two grounds.
>
> First, making this kind of decision should be
> based on further evidence besides a single
> benchmark clobbered together in a laptop.

Of course, it was more of a proof of concept sort
of thing.

> Actual hypervisors are actually very good at
> detecting the guest os doesn't really need
> attention.

Hypervisors can do some clever things, but there's
no way that select() wakeup is getting optimized
away - and that wakeup would be more expensive on
a virtualized platform, not less.

> Also, if you're spinning a mongod, how often is
> it going to stay without getting any request for
> longer than the time out? So getting more data
> about this things should make the effectiveness
> (or need) of this patch clearer.

I think you're forgetting that the timeout has
overhead even if it never happens. Consider the
following:

https://http.snarkywidgets.com/select_example.c

Below is a quick run of that code on an idle AWS
instance: (I ran this about 10 times, the numbers
were consistent).

$ gcc -std=gnu99 -O3 select_example.c
$ time ./a.out

real 0m0.964s
user 0m0.216s
sys 0m0.748s
$ gcc -std=gnu99 -O3 -DDO_TIMEOUT select_example.c
$ time ./a.out

real 0m1.454s
user 0m0.264s
sys 0m1.188s

As you can see, even when the timeout never
triggers, it has non-negligible overhead. That's a
50% performance hit. Even though it's no great
amount of time, it's worth recognizing that the
timeouts aren't by any means free.

> The second point is that it is quite common not
> to block in a select and use the timeouts to,
> for instance, trigger timed callbacks (taks that
> are in the background that can be woken up when
> the server load is not hight). We don't exactly
> do this right now but when the timeout was put
> in place, it was also with this goal in mind.

That makes sense, but the timeout would definitely
be longer than 10ms in that case, wouldn't it?

> There is also the point that that file is on the
> center of a couple of improvements we're
> planning for this cycle, related to connection
> handling (actually loss thereof). And there's
> only so much change that can be done before we
> start wondering if we're introducing one too
> many changes.

That's fair, of course. I'm happy to wait, just
wanted to know if anybody was listening. :)

Thanks very much,
Calvin

[Alberto Lerner]

Calvin,

Sure thing. And of course we're listening. 

Just one comment:

> As you can see, even when the timeout never
> triggers, it has non-negligible overhead. That's a
> 50% performance hit. Even though it's no great
> amount of time, it's worth recognizing that the
> timeouts aren't by any means free.

I'm not arguing it is free. But this experiment gives no better data to evaluate how costly it is if you consider user performance perspective. You see, the example code is not performing the tasks that an actual server does. All I can safely read from this experiment is that, once in a while, it takes 500ns more to issue one system call whose cost gets amortized in many different ways. I suspect that, in terms of user visible performance, abolishing the time out might not make even in the tenths of a percent difference in user visible performance. (But that is more of a hunch than anything else.) Put it in a different way, for all I know, I can shave 500ns or better by, say, doing some perf analysis to find a hot loop and trying to reuse cache lines better there without giving up any possibility (e.g, issuing timed callbacks). 

Making these decisions require more realistic and more varied benchmarks -- ideally ones that can be reusable in the future to prevent performance regressions. It remains a bit tricky to use a percentage number like that and think that "it's x times faster" would actually be so to the end user.

Alberto.

Calvin

[Calvin Owens]

On Wednesday 05/08 at 21:10 -0400, Alberto Lerner wrote:
> Calvin,

> Sure thing. And of course we're listening.
> Just one comment:
>> As you can see, even when the timeout never
>> triggers, it has non-negligible overhead. That's a
>> 50% performance hit. Even though it's no great
>> amount of time, it's worth recognizing that the
>> timeouts aren't by any means free.
>
> I'm not arguing it is free. But this experiment gives no better data to
> evaluate how costly it is if you consider user performance perspective.
> You see, the example code is not performing the tasks that an actual
> server does. All I can safely read from this experiment is that, once in a
> while, it takes 500ns more to issue one system call whose cost gets
> amortized in many different ways. I suspect that, in terms of user visible
> performance, abolishing the time out might not make even in the tenths of
> a percent difference in user visible performance. (But that is more of a
> hunch than anything else.)

Sorry, I wasn't meaning to imply that eliminating the timeout was some
gigantic win in terms of this syscall latency - you're right of course
that it's likely not.

That said, while that example I sent is *very* pessimistic, it is true
that select() is invoked between every connection in the Listener (which
is it's own thread - I think), which is what I was trying to model there.
Those calls could be quite frequent on a very busy server, although
obviously not approaching 1M/sec. It's not a good model, but I think
it's not necessarily a bad one in this case.

Thanks very much,
Calvin

> You received this message because you are subscribed to a topic in the
> Google Groups "mongodb-dev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/mongodb-dev/a0Peho3ikts/unsubscribe?hl=en-US.
> To unsubscribe from this group and all its topics, send an email to