> On 10/03/2012 10:55 PM, Avi Kivity wrote:
>> On 10/03/2012 04:29 PM, Raghavendra K T wrote:
>>> * Avi Kivity <a...@redhat.com> [2012-09-27 14:03:59]:
>>>> On 09/27/2012 01:23 PM, Raghavendra K T wrote:
>>> [...]
>>>>> 2) looking at the result (comparing A & C) , I do feel we have
>>>>> significant in iterating over vcpus (when compared to even vmexit)
>>>>> so We still would need undercommit fix sugested by PeterZ
>>>>> (improving by
>>>>> 140%). ?
>>>> Looking only at the current runqueue? My worry is that it misses a lot
>>>> of cases. Maybe try the current runqueue first and then others.
>>> ( I thought lot about && or || . Both seem to have their own cons ).
>>> But that should be only when we have short term imbalance, as PeterZ
>>> told.
>> I'm missing the context. What is p_rq?
> p_rq is the run queue of target vcpu.
> What I was trying below was to address Rik concern. Suppose
> rq of source vcpu has one task, but target probably has two task,
> with a eligible vcpu waiting to be scheduled.
>> What I mean was:
>> if can_yield_to_process_in_current_rq
>> do that
>> else if can_yield_to_process_in_other_rq
>> do that
>> else
>> return -ESRCH
> I think you are saying we have to check the run queue of the
> source vcpu, if we have a vcpu belonging to same VM and try yield to
> that? ignoring whatever the target vcpu we received for yield_to.
> Or is it that kvm_vcpu_yield_to should now check the vcpus of same vm
> belonging to same run queue first. If we don't succeed, go again for
> a vcpu in different runqueue.
Right. Prioritize vcpus that are cheap to yield to. But may return bad
results if all vcpus on the current runqueue are spinners, so probably
not a good idea.
> Does it add more overhead especially in <= 1x scenario?
The current runqueue should have just our vcpu in that case, so low
overhead. But it's a bad idea due to the above scenario.
-- error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
> Again the numbers are ridiculously high for arch_local_irq_restore.
> Maybe there's a bad perf/kvm interaction when we're injecting an
> interrupt, I can't believe we're spending 84% of the time running the
> popf instruction.
Smells like a software fallback that doesn't do NMI, hrtimer based
sampling typically hits popf where we re-enable interrupts.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
> On 10/04/2012 12:49 PM, Raghavendra K T wrote:
> > On 10/03/2012 10:35 PM, Avi Kivity wrote:
> >> On 10/03/2012 02:22 PM, Raghavendra K T wrote:
> >>>> So I think it's worth trying again with ple_window of 20000-40000.
> >>> Hi Avi,
> >>> I ran different benchmarks increasing ple_window, and results does not
> >>> seem to be encouraging for increasing ple_window.
> >> So, 44% degradation even with no overcommit? That's surprising.
> > Yes. Kernbench was run with #threads = #vcpu * 2 as usual. Is it
> > spending 8 times the original ple_window cycles for 16 vcpus
> > significant?
> A PLE exit when not overcommitted cannot do any good, it is better to
> spin in the guest rather that look for candidates on the host. In fact
> when we benchmark we often disable PLE completely.
Agreed. However, I really do not understand why the kernbench regressed
with bigger ple_window. It should stay the same or improve. Raghu, do
you have perf data for the kernbench runs?
> >> Both the 4k and the 32k results are crazy. Why is
> >> arch_local_irq_restore() so prominent? Do you have a very high
> >> interrupt rate in the guest?
> > How to measure if I have high interrupt rate in guest?
> > From /proc/interrupt numbers I am not able to judge :(
> 'vmstat 1'
> > I went back and got the results on a 32 core machine with 32 vcpu guest.
> > Strangely, I got result supporting the claim that increasing ple_window
> > helps for non-overcommitted scenario.
> So maybe something was wrong with the first measurement.
OK, this is more in line with what I expected for kernbench. FWIW, in
order to show an improvement for a larger ple_window, we really need a
workload which we know has a longer lock holding time (without factoring
in LHP). We have noticed this on IO based locks mostly. We saw it with
a massive disk IO test (qla2xxx lock), and also with a large web serving
test (some vfs related lock, but I forget what exactly it was).
> Again the numbers are ridiculously high for arch_local_irq_restore.
> Maybe there's a bad perf/kvm interaction when we're injecting an
> interrupt, I can't believe we're spending 84% of the time running the
> popf instruction.
I do have a feeling that ebizzy just has too many variables and LHP is
just one of many problems. However, am I curious what perf kvm from
host shows as Avi suggested below.
> On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
>> Again the numbers are ridiculously high for arch_local_irq_restore.
>> Maybe there's a bad perf/kvm interaction when we're injecting an
>> interrupt, I can't believe we're spending 84% of the time running the
>> popf instruction.
> Smells like a software fallback that doesn't do NMI, hrtimer based
> sampling typically hits popf where we re-enable interrupts.
Good nose, that's probably it. Raghavendra, can you ensure that the PMU
is properly exposed? 'dmesg' in the guest will tell. If it isn't, -cpu
host will expose it (and a good idea anyway to get best performance).
-- error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
> On Wed, Oct 03, 2012 at 04:56:57PM +0200, Avi Kivity wrote:
>> On 10/03/2012 04:17 PM, Raghavendra K T wrote:
>>> * Avi Kivity <a...@redhat.com> [2012-09-30 13:13:09]:
>>>> On 09/30/2012 01:07 PM, Gleb Natapov wrote:
>>>>> On Sun, Sep 30, 2012 at 10:18:17AM +0200, Avi Kivity wrote:
>>>>>> On 09/28/2012 08:16 AM, Raghavendra K T wrote:
>>>>>>>> +struct pv_sched_info {
>>>>>>>> + unsigned long sched_bitmap;
>>>>>>> Thinking, whether we need something similar to cpumask here?
>>>>>>> Only thing is we are representing guest (v)cpumask.
>>>> Use the index into the vcpu table as the bitmap index then. In fact
>>>> it's better because then the lookup to get the vcpu pointer is trivial.
>>> Did you mean, while setting the bitmap,
>>> we should do
>>> for (i = 1..n)
>>> if (kvm->vcpus[i] == vcpu) set ith position in bitmap?
>> You can store i in the vcpu itself:
>> set_bit(vcpu->index, &kvm->preempted);
> This will make the fact that vcpus are stored in an array into API
> instead of implementation detail :( There were patches for vcpu
> destruction that changed the array to rculist. Well, it will be still
> possible to make the array rcu protected and copy it every time vcpu is
> deleted/added I guess.
If IUC, summary is, we are going with
- Let vcpu array be rcu protected.
- we use index inside vcpu and should be updated when a vcpu is
added/deleted.
> On 10/04/2012 12:49 PM, Raghavendra K T wrote:
>> On 10/03/2012 10:35 PM, Avi Kivity wrote:
>>> On 10/03/2012 02:22 PM, Raghavendra K T wrote:
>>>>> So I think it's worth trying again with ple_window of 20000-40000.
>>>> Hi Avi,
>>>> I ran different benchmarks increasing ple_window, and results does not
>>>> seem to be encouraging for increasing ple_window.
>>> So, 44% degradation even with no overcommit? That's surprising.
>> Yes. Kernbench was run with #threads = #vcpu * 2 as usual. Is it
>> spending 8 times the original ple_window cycles for 16 vcpus
>> significant?
> A PLE exit when not overcommitted cannot do any good, it is better to
> spin in the guest rather that look for candidates on the host. In fact
> when we benchmark we often disable PLE completely.
>>>> I also got perf top output to analyse the difference. Difference comes
>>>> because of flushtlb (and also spinlock).
>>> Both the 4k and the 32k results are crazy. Why is
>>> arch_local_irq_restore() so prominent? Do you have a very high
>>> interrupt rate in the guest?
>> How to measure if I have high interrupt rate in guest?
>> From /proc/interrupt numbers I am not able to judge :(
> 'vmstat 1'
Thanks you. 'll save this. Apart from in,cs I think r: The number of processes waiting for run time, would be useful for me in vmstat.
>> I went back and got the results on a 32 core machine with 32 vcpu guest.
>> Strangely, I got result supporting the claim that increasing ple_window
>> helps for non-overcommitted scenario.
[ 0.017997] Performance Events: 16-deep LBR, SandyBridge events, Intel PMU driver.
[ 0.018868] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
So I 'll try it on x240 again.
( Some how mx3850 -cpu host resulted in
[ 0.026995] Performance Events: unsupported p6 CPU model 26 no PMU driver, software events only.
I think qemu needs some fix as pointed in
http://www.mail-archive.com/...@vger.kernel.org/msg55836.html
> Again the numbers are ridiculously high for arch_local_irq_restore.
> Maybe there's a bad perf/kvm interaction when we're injecting an
> interrupt, I can't believe we're spending 84% of the time running the
> popf instruction.
> On 10/04/2012 12:56 PM, Raghavendra K T wrote:
>> On 10/03/2012 10:55 PM, Avi Kivity wrote:
>>> On 10/03/2012 04:29 PM, Raghavendra K T wrote:
>>>> * Avi Kivity <a...@redhat.com> [2012-09-27 14:03:59]:
>>>>> On 09/27/2012 01:23 PM, Raghavendra K T wrote:
>>>> [...]
>>>>>> 2) looking at the result (comparing A & C) , I do feel we have
>>>>>> significant in iterating over vcpus (when compared to even vmexit)
>>>>>> so We still would need undercommit fix sugested by PeterZ
>>>>>> (improving by
>>>>>> 140%). ?
>>>>> Looking only at the current runqueue? My worry is that it misses a lot
>>>>> of cases. Maybe try the current runqueue first and then others.
>>>> ( I thought lot about && or || . Both seem to have their own cons ).
>>>> But that should be only when we have short term imbalance, as PeterZ
>>>> told.
>>> I'm missing the context. What is p_rq?
>> p_rq is the run queue of target vcpu.
>> What I was trying below was to address Rik concern. Suppose
>> rq of source vcpu has one task, but target probably has two task,
>> with a eligible vcpu waiting to be scheduled.
>>> What I mean was:
>>> if can_yield_to_process_in_current_rq
>>> do that
>>> else if can_yield_to_process_in_other_rq
>>> do that
>>> else
>>> return -ESRCH
>> I think you are saying we have to check the run queue of the
>> source vcpu, if we have a vcpu belonging to same VM and try yield to
>> that? ignoring whatever the target vcpu we received for yield_to.
>> Or is it that kvm_vcpu_yield_to should now check the vcpus of same vm
>> belonging to same run queue first. If we don't succeed, go again for
>> a vcpu in different runqueue.
> Right. Prioritize vcpus that are cheap to yield to. But may return bad
> results if all vcpus on the current runqueue are spinners, so probably
> not a good idea.
Okay. 'll drop vcpu from same rq idea now.
>> Does it add more overhead especially in <= 1x scenario?
> The current runqueue should have just our vcpu in that case, so low
> overhead. But it's a bad idea due to the above scenario.
> On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
>> On 10/04/2012 12:49 PM, Raghavendra K T wrote:
>>> On 10/03/2012 10:35 PM, Avi Kivity wrote:
>>>> On 10/03/2012 02:22 PM, Raghavendra K T wrote:
>>>>>> So I think it's worth trying again with ple_window of 20000-40000.
>>>>> Hi Avi,
>>>>> I ran different benchmarks increasing ple_window, and results does not
>>>>> seem to be encouraging for increasing ple_window.
>>>> So, 44% degradation even with no overcommit? That's surprising.
>>> Yes. Kernbench was run with #threads = #vcpu * 2 as usual. Is it
>>> spending 8 times the original ple_window cycles for 16 vcpus
>>> significant?
>> A PLE exit when not overcommitted cannot do any good, it is better to
>> spin in the guest rather that look for candidates on the host. In fact
>> when we benchmark we often disable PLE completely.
> Agreed. However, I really do not understand why the kernbench regressed
> with bigger ple_window. It should stay the same or improve. Raghu, do
> you have perf data for the kernbench runs?
>> This will make the fact that vcpus are stored in an array into API
>> instead of implementation detail :( There were patches for vcpu
>> destruction that changed the array to rculist. Well, it will be still
>> possible to make the array rcu protected and copy it every time vcpu is
>> deleted/added I guess.
> If IUC, summary is, we are going with
> - Let vcpu array be rcu protected.
That's for the future. For now ->vcpus[] is statically allocated.
> - we use index inside vcpu and should be updated when a vcpu is
> added/deleted.
Yes.
-- error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
* Avi Kivity <a...@redhat.com> [2012-10-04 17:00:28]:
> On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
> > On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
> >> Again the numbers are ridiculously high for arch_local_irq_restore.
> >> Maybe there's a bad perf/kvm interaction when we're injecting an
> >> interrupt, I can't believe we're spending 84% of the time running the
> >> popf instruction.
> > Smells like a software fallback that doesn't do NMI, hrtimer based
> > sampling typically hits popf where we re-enable interrupts.
> Good nose, that's probably it. Raghavendra, can you ensure that the PMU
> is properly exposed? 'dmesg' in the guest will tell. If it isn't, -cpu
> host will expose it (and a good idea anyway to get best performance).
Hi Avi, you are right. SandyBridge machine result was not proper.
I cleaned up the services, enabled PMU, re-ran all the test again.
Here is the summary:
We do get good benefit by increasing ple window. Though we don't
see good benefit for kernbench and sysbench, for ebizzy, we get huge
improvement for 1x scenario. (almost 2/3rd of ple disabled case).
Let me know if you think we can increase the default ple_window
itself to 16k.
I am experimenting with V2 version of undercommit improvement(this) patch
series, But I think if you wish to go for increase of
default ple_window, then we would have to measure the benefit of patches
when ple_window = 16k.
I can respin the whole series including this default ple_window change.
I also have the perf kvm top result for both ebizzy and kernbench.
I think they are in expected lines now.
Improvements
================
16 core PLE machine with 16 vcpu guest
base = 3.6.0-rc5 + ple handler optimization patches
base_pleopt_16k = base + ple_window = 16k
base_pleopt_32k = base + ple_window = 32k
base_pleopt_nople = base + ple_gap = 0
kernbench, hackbench, sysbench (time in sec lower is better)
ebizzy (rec/sec higher is better)
On Wed, 2012-10-10 at 00:21 +0530, Raghavendra K T wrote:
> * Avi Kivity <a...@redhat.com> [2012-10-04 17:00:28]:
> > On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
> > > On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
> > >> Again the numbers are ridiculously high for arch_local_irq_restore.
> > >> Maybe there's a bad perf/kvm interaction when we're injecting an
> > >> interrupt, I can't believe we're spending 84% of the time running the
> > >> popf instruction.
> > > Smells like a software fallback that doesn't do NMI, hrtimer based
> > > sampling typically hits popf where we re-enable interrupts.
> > Good nose, that's probably it. Raghavendra, can you ensure that the PMU
> > is properly exposed? 'dmesg' in the guest will tell. If it isn't, -cpu
> > host will expose it (and a good idea anyway to get best performance).
> Hi Avi, you are right. SandyBridge machine result was not proper.
> I cleaned up the services, enabled PMU, re-ran all the test again.
> Here is the summary:
> We do get good benefit by increasing ple window. Though we don't
> see good benefit for kernbench and sysbench, for ebizzy, we get huge
> improvement for 1x scenario. (almost 2/3rd of ple disabled case).
> Let me know if you think we can increase the default ple_window
> itself to 16k.
> I am experimenting with V2 version of undercommit improvement(this) patch
> series, But I think if you wish to go for increase of
> default ple_window, then we would have to measure the benefit of patches
> when ple_window = 16k.
> I can respin the whole series including this default ple_window change.
> I also have the perf kvm top result for both ebizzy and kernbench.
> I think they are in expected lines now.
> Improvements
> ================
> 16 core PLE machine with 16 vcpu guest
> base = 3.6.0-rc5 + ple handler optimization patches
> base_pleopt_16k = base + ple_window = 16k
> base_pleopt_32k = base + ple_window = 32k
> base_pleopt_nople = base + ple_gap = 0
> kernbench, hackbench, sysbench (time in sec lower is better)
> ebizzy (rec/sec higher is better)
Does 'perf kvm top' not give host samples at the same time? Would be
nice to see the host overhead as a function of varying ple window. I
would expect that to be the major difference between 4/16/32k window
sizes.
A big concern I have (if this is 1x overcommit) for ebizzy is that it
has just terrible scalability to begin with. I do not think we should
try to optimize such a bad workload.
I ran 'perf sched map' on the dbench workload for medium and large VMs,
and I thought I would share some of the results. I think it helps to
visualize what's going on regarding the yielding.
These files are png bitmaps, generated from processing output from 'perf
sched map' (and perf data generated from 'perf sched record'). The Y
axis is the host cpus, each row being 10 pixels high. For these tests,
there are 80 host cpus, so the total height is 800 pixels. The X axis
is time (in microseconds), with each pixel representing 1 microsecond.
Each bitmap plots 30,000 microseconds. The bitmaps are quite wide
obviously, and zooming in/out while viewing is recommended.
Each row (each host cpu) is assigned a color based on what thread is
running. vCPUs of the same VM are assigned a common color (like red,
blue, magenta, etc), and each vCPU has a unique brightness for that
color. There are a maximum of 12 assignable colors, so in any VMs >12
revert to vCPU color of gray. I would use more colors, but it becomes
harder to distinguish one color from another. The white color
represents missing data from perf, and black color represents any thread
which is not a vCPU.
For the following tests, VMs were pinned to host NUMA nodes and to
specific cpus to help with consistency and operate within the
constraints of the last test (gang scheduler).
Here is a good example of PLE. These are 10-way VMs, 16 of them (as
described above only 12 of the VMs have a color, rest are gray).
If you zoom out and look at the whole bitmap, you may notice the 4ms
intervals of the scheduler. They are pretty well aligned across all
cpus. Normally, for cpu bound workloads, we would expect to see each
thread to run for 4 ms, then something else getting to run, and so on.
That is mostly true in this test. We have 2x over-commit and we
generally see the switching of threads at 4ms. One thing to note is
that not all vCPU threads for the same VM run at exactly the same time,
and that is expected and the whole reason for lock-holder preemption.
Now, if you zoom in on the bitmap, you should notice within the 4ms
intervals there is some task switching going on. This is most likely
because of the yield_to initiated by the PLE handler. In this case
there is not that much yielding to do. It's quite clean, and the
performance is quite good.
Below is an example of PLE, but this time with 20-way VMs, 8 of them.
CPU over-commit is still 2x.
This one looks quite different. In short, it's a mess. The switching
between tasks can be lower than 10 microseconds. It basically never
recovers. There is constant yielding all the time.
Below is again 8 x 20-way VMs, but this time I tried out Nikunj's gang
scheduling patches. While I am not recommending gang scheduling, I
think it's a good data point. The performance is 3.88x the PLE result.
Note that the task switching intervals of 4ms are quite obvious again,
and this time all vCPUs from same VM run at the same time. It
represents the best possible outcome.
Anyway, I thought the bitmaps might help better visualize what's going
on.
> I ran 'perf sched map' on the dbench workload for medium and large VMs,
> and I thought I would share some of the results. I think it helps to
> visualize what's going on regarding the yielding.
> These files are png bitmaps, generated from processing output from 'perf
> sched map' (and perf data generated from 'perf sched record'). The Y
> axis is the host cpus, each row being 10 pixels high. For these tests,
> there are 80 host cpus, so the total height is 800 pixels. The X axis
> is time (in microseconds), with each pixel representing 1 microsecond.
> Each bitmap plots 30,000 microseconds. The bitmaps are quite wide
> obviously, and zooming in/out while viewing is recommended.
> Each row (each host cpu) is assigned a color based on what thread is
> running. vCPUs of the same VM are assigned a common color (like red,
> blue, magenta, etc), and each vCPU has a unique brightness for that
> color. There are a maximum of 12 assignable colors, so in any VMs >12
> revert to vCPU color of gray. I would use more colors, but it becomes
> harder to distinguish one color from another. The white color
> represents missing data from perf, and black color represents any thread
> which is not a vCPU.
> For the following tests, VMs were pinned to host NUMA nodes and to
> specific cpus to help with consistency and operate within the
> constraints of the last test (gang scheduler).
> Here is a good example of PLE. These are 10-way VMs, 16 of them (as
> described above only 12 of the VMs have a color, rest are gray).
> If you zoom out and look at the whole bitmap, you may notice the 4ms
> intervals of the scheduler. They are pretty well aligned across all
> cpus. Normally, for cpu bound workloads, we would expect to see each
> thread to run for 4 ms, then something else getting to run, and so on.
> That is mostly true in this test. We have 2x over-commit and we
> generally see the switching of threads at 4ms. One thing to note is
> that not all vCPU threads for the same VM run at exactly the same time,
> and that is expected and the whole reason for lock-holder preemption.
> Now, if you zoom in on the bitmap, you should notice within the 4ms
> intervals there is some task switching going on. This is most likely
> because of the yield_to initiated by the PLE handler. In this case
> there is not that much yielding to do. It's quite clean, and the
> performance is quite good.
> Below is an example of PLE, but this time with 20-way VMs, 8 of them.
> CPU over-commit is still 2x.
> This one looks quite different. In short, it's a mess. The switching
> between tasks can be lower than 10 microseconds. It basically never
> recovers. There is constant yielding all the time.
> Below is again 8 x 20-way VMs, but this time I tried out Nikunj's gang
> scheduling patches. While I am not recommending gang scheduling, I
> think it's a good data point. The performance is 3.88x the PLE result.
> Note that the task switching intervals of 4ms are quite obvious again,
> and this time all vCPUs from same VM run at the same time. It
> represents the best possible outcome.
> Anyway, I thought the bitmaps might help better visualize what's going
> on.
> On Wed, 2012-10-10 at 00:21 +0530, Raghavendra K T wrote:
>> * Avi Kivity <a...@redhat.com> [2012-10-04 17:00:28]:
>>> On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
>>>> On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
>>>>> Again the numbers are ridiculously high for arch_local_irq_restore.
>>>>> Maybe there's a bad perf/kvm interaction when we're injecting an
>>>>> interrupt, I can't believe we're spending 84% of the time running the
>>>>> popf instruction.
>>>> Smells like a software fallback that doesn't do NMI, hrtimer based
>>>> sampling typically hits popf where we re-enable interrupts.
>>> Good nose, that's probably it. Raghavendra, can you ensure that the PMU
>>> is properly exposed? 'dmesg' in the guest will tell. If it isn't, -cpu
>>> host will expose it (and a good idea anyway to get best performance).
>> Hi Avi, you are right. SandyBridge machine result was not proper.
>> I cleaned up the services, enabled PMU, re-ran all the test again.
>> Here is the summary:
>> We do get good benefit by increasing ple window. Though we don't
>> see good benefit for kernbench and sysbench, for ebizzy, we get huge
>> improvement for 1x scenario. (almost 2/3rd of ple disabled case).
>> Let me know if you think we can increase the default ple_window
>> itself to 16k.
>> I am experimenting with V2 version of undercommit improvement(this) patch
>> series, But I think if you wish to go for increase of
>> default ple_window, then we would have to measure the benefit of patches
>> when ple_window = 16k.
>> I can respin the whole series including this default ple_window change.
>> I also have the perf kvm top result for both ebizzy and kernbench.
>> I think they are in expected lines now.
>> Improvements
>> ================
>> 16 core PLE machine with 16 vcpu guest
>> base = 3.6.0-rc5 + ple handler optimization patches
>> base_pleopt_16k = base + ple_window = 16k
>> base_pleopt_32k = base + ple_window = 32k
>> base_pleopt_nople = base + ple_gap = 0
>> kernbench, hackbench, sysbench (time in sec lower is better)
>> ebizzy (rec/sec higher is better)
>> perf kvm top observation for kernbench and ebizzy (nople, 4k, 32k window)
>> ========================================================================
> Does 'perf kvm top' not give host samples at the same time? Would be
> nice to see the host overhead as a function of varying ple window. I
> would expect that to be the major difference between 4/16/32k window
> sizes.
No, I did something like this
perf kvm --guestvmlinux ./vmlinux.guest top -g -U -d 3. Yes that is a
good idea.
(I am getting some segfaults with perf top, I think it is already fixed
but yet to see the patch that fixes)
> A big concern I have (if this is 1x overcommit) for ebizzy is that it
> has just terrible scalability to begin with. I do not think we should
> try to optimize such a bad workload.
I think my way of running dbench has some flaw, so I went to ebizzy.
Could you let me know how you generally run dbench?
> On 10/10/12 11:54 AM, Raghavendra K T wrote:
>> No, I did something like this
>> perf kvm --guestvmlinux ./vmlinux.guest top -g -U -d 3. Yes that is a
>> good idea.
>> (I am getting some segfaults with perf top, I think it is already fixed
>> but yet to see the patch that fixes)
> What version of perf: perf --version
perf version 2.6.32-279.el6.x86_64.debug
(I searched that it is fixed in 288. could not dig-out actual patch
though)
On Wed, 2012-10-10 at 23:13 +0530, Raghavendra K T wrote:
> On 10/10/2012 07:54 PM, Andrew Theurer wrote:
> > I ran 'perf sched map' on the dbench workload for medium and large VMs,
> > and I thought I would share some of the results. I think it helps to
> > visualize what's going on regarding the yielding.
> > These files are png bitmaps, generated from processing output from 'perf
> > sched map' (and perf data generated from 'perf sched record'). The Y
> > axis is the host cpus, each row being 10 pixels high. For these tests,
> > there are 80 host cpus, so the total height is 800 pixels. The X axis
> > is time (in microseconds), with each pixel representing 1 microsecond.
> > Each bitmap plots 30,000 microseconds. The bitmaps are quite wide
> > obviously, and zooming in/out while viewing is recommended.
> > Each row (each host cpu) is assigned a color based on what thread is
> > running. vCPUs of the same VM are assigned a common color (like red,
> > blue, magenta, etc), and each vCPU has a unique brightness for that
> > color. There are a maximum of 12 assignable colors, so in any VMs >12
> > revert to vCPU color of gray. I would use more colors, but it becomes
> > harder to distinguish one color from another. The white color
> > represents missing data from perf, and black color represents any thread
> > which is not a vCPU.
> > For the following tests, VMs were pinned to host NUMA nodes and to
> > specific cpus to help with consistency and operate within the
> > constraints of the last test (gang scheduler).
> > Here is a good example of PLE. These are 10-way VMs, 16 of them (as
> > described above only 12 of the VMs have a color, rest are gray).
> This looks very nice to visualize what is happening. Beginning of the > graph looks little messy but later it is clear.
> > If you zoom out and look at the whole bitmap, you may notice the 4ms
> > intervals of the scheduler. They are pretty well aligned across all
> > cpus. Normally, for cpu bound workloads, we would expect to see each
> > thread to run for 4 ms, then something else getting to run, and so on.
> > That is mostly true in this test. We have 2x over-commit and we
> > generally see the switching of threads at 4ms. One thing to note is
> > that not all vCPU threads for the same VM run at exactly the same time,
> > and that is expected and the whole reason for lock-holder preemption.
> > Now, if you zoom in on the bitmap, you should notice within the 4ms
> > intervals there is some task switching going on. This is most likely
> > because of the yield_to initiated by the PLE handler. In this case
> > there is not that much yielding to do. It's quite clean, and the
> > performance is quite good.
> > Below is an example of PLE, but this time with 20-way VMs, 8 of them.
> > CPU over-commit is still 2x.
> > This one looks quite different. In short, it's a mess. The switching
> > between tasks can be lower than 10 microseconds. It basically never
> > recovers. There is constant yielding all the time.
> > Below is again 8 x 20-way VMs, but this time I tried out Nikunj's gang
> > scheduling patches. While I am not recommending gang scheduling, I
> > think it's a good data point. The performance is 3.88x the PLE result.
> > Note that the task switching intervals of 4ms are quite obvious again,
> > and this time all vCPUs from same VM run at the same time. It
> > represents the best possible outcome.
> > Anyway, I thought the bitmaps might help better visualize what's going
> > on.
On Wed, 2012-10-10 at 23:24 +0530, Raghavendra K T wrote:
> On 10/10/2012 08:29 AM, Andrew Theurer wrote:
> > On Wed, 2012-10-10 at 00:21 +0530, Raghavendra K T wrote:
> >> * Avi Kivity <a...@redhat.com> [2012-10-04 17:00:28]:
> >>> On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
> >>>> On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
> >>>>> Again the numbers are ridiculously high for arch_local_irq_restore.
> >>>>> Maybe there's a bad perf/kvm interaction when we're injecting an
> >>>>> interrupt, I can't believe we're spending 84% of the time running the
> >>>>> popf instruction.
> >>>> Smells like a software fallback that doesn't do NMI, hrtimer based
> >>>> sampling typically hits popf where we re-enable interrupts.
> >>> Good nose, that's probably it. Raghavendra, can you ensure that the PMU
> >>> is properly exposed? 'dmesg' in the guest will tell. If it isn't, -cpu
> >>> host will expose it (and a good idea anyway to get best performance).
> >> Hi Avi, you are right. SandyBridge machine result was not proper.
> >> I cleaned up the services, enabled PMU, re-ran all the test again.
> >> Here is the summary:
> >> We do get good benefit by increasing ple window. Though we don't
> >> see good benefit for kernbench and sysbench, for ebizzy, we get huge
> >> improvement for 1x scenario. (almost 2/3rd of ple disabled case).
> >> Let me know if you think we can increase the default ple_window
> >> itself to 16k.
> >> I am experimenting with V2 version of undercommit improvement(this) patch
> >> series, But I think if you wish to go for increase of
> >> default ple_window, then we would have to measure the benefit of patches
> >> when ple_window = 16k.
> >> I can respin the whole series including this default ple_window change.
> >> I also have the perf kvm top result for both ebizzy and kernbench.
> >> I think they are in expected lines now.
> >> Improvements
> >> ================
> >> 16 core PLE machine with 16 vcpu guest
> >> base = 3.6.0-rc5 + ple handler optimization patches
> >> base_pleopt_16k = base + ple_window = 16k
> >> base_pleopt_32k = base + ple_window = 32k
> >> base_pleopt_nople = base + ple_gap = 0
> >> kernbench, hackbench, sysbench (time in sec lower is better)
> >> ebizzy (rec/sec higher is better)
> > Does 'perf kvm top' not give host samples at the same time? Would be
> > nice to see the host overhead as a function of varying ple window. I
> > would expect that to be the major difference between 4/16/32k window
> > sizes.
> No, I did something like this
> perf kvm --guestvmlinux ./vmlinux.guest top -g -U -d 3. Yes that is a
> good idea.
> (I am getting some segfaults with perf top, I think it is already fixed
> but yet to see the patch that fixes)
> > A big concern I have (if this is 1x overcommit) for ebizzy is that it
> > has just terrible scalability to begin with. I do not think we should
> > try to optimize such a bad workload.
> I think my way of running dbench has some flaw, so I went to ebizzy.
> Could you let me know how you generally run dbench?
I mount a tmpfs and then specify that mount for dbench to run on. This
eliminates all IO. I use a 300 second run time and number of threads is
equal to number of vcpus. All of the VMs of course need to have a
synchronized start.
I would also make sure you are using a recent kernel for dbench, where
the dcache scalability is much improved. Without any lock-holder
preemption, the time in spin_lock should be very low:
On Wed, 10 Oct 2012 09:24:55 -0500, Andrew Theurer <haban...@linux.vnet.ibm.com> wrote:
> Below is again 8 x 20-way VMs, but this time I tried out Nikunj's gang
> scheduling patches. While I am not recommending gang scheduling, I
> think it's a good data point. The performance is 3.88x the PLE result.
That looks pretty good and serves the purpose. And the result says it all.
> Note that the task switching intervals of 4ms are quite obvious again,
> and this time all vCPUs from same VM run at the same time. It
> represents the best possible outcome.
> Anyway, I thought the bitmaps might help better visualize what's going
> on.
> On Wed, 2012-10-10 at 23:13 +0530, Raghavendra K T wrote:
>> On 10/10/2012 07:54 PM, Andrew Theurer wrote:
>>> I ran 'perf sched map' on the dbench workload for medium and large VMs,
>>> and I thought I would share some of the results. I think it helps to
>>> visualize what's going on regarding the yielding.
>>> These files are png bitmaps, generated from processing output from 'perf
>>> sched map' (and perf data generated from 'perf sched record'). The Y
>>> axis is the host cpus, each row being 10 pixels high. For these tests,
>>> there are 80 host cpus, so the total height is 800 pixels. The X axis
>>> is time (in microseconds), with each pixel representing 1 microsecond.
>>> Each bitmap plots 30,000 microseconds. The bitmaps are quite wide
>>> obviously, and zooming in/out while viewing is recommended.
>>> Each row (each host cpu) is assigned a color based on what thread is
>>> running. vCPUs of the same VM are assigned a common color (like red,
>>> blue, magenta, etc), and each vCPU has a unique brightness for that
>>> color. There are a maximum of 12 assignable colors, so in any VMs >12
>>> revert to vCPU color of gray. I would use more colors, but it becomes
>>> harder to distinguish one color from another. The white color
>>> represents missing data from perf, and black color represents any thread
>>> which is not a vCPU.
>>> For the following tests, VMs were pinned to host NUMA nodes and to
>>> specific cpus to help with consistency and operate within the
>>> constraints of the last test (gang scheduler).
>>> Here is a good example of PLE. These are 10-way VMs, 16 of them (as
>>> described above only 12 of the VMs have a color, rest are gray).
>> This looks very nice to visualize what is happening. Beginning of the
>> graph looks little messy but later it is clear.
>>> If you zoom out and look at the whole bitmap, you may notice the 4ms
>>> intervals of the scheduler. They are pretty well aligned across all
>>> cpus. Normally, for cpu bound workloads, we would expect to see each
>>> thread to run for 4 ms, then something else getting to run, and so on.
>>> That is mostly true in this test. We have 2x over-commit and we
>>> generally see the switching of threads at 4ms. One thing to note is
>>> that not all vCPU threads for the same VM run at exactly the same time,
>>> and that is expected and the whole reason for lock-holder preemption.
>>> Now, if you zoom in on the bitmap, you should notice within the 4ms
>>> intervals there is some task switching going on. This is most likely
>>> because of the yield_to initiated by the PLE handler. In this case
>>> there is not that much yielding to do. It's quite clean, and the
>>> performance is quite good.
>>> Below is an example of PLE, but this time with 20-way VMs, 8 of them.
>>> CPU over-commit is still 2x.
>>> This one looks quite different. In short, it's a mess. The switching
>>> between tasks can be lower than 10 microseconds. It basically never
>>> recovers. There is constant yielding all the time.
>>> Below is again 8 x 20-way VMs, but this time I tried out Nikunj's gang
>>> scheduling patches. While I am not recommending gang scheduling, I
>>> think it's a good data point. The performance is 3.88x the PLE result.
> On Wed, 2012-10-10 at 23:24 +0530, Raghavendra K T wrote:
>> On 10/10/2012 08:29 AM, Andrew Theurer wrote:
>>> On Wed, 2012-10-10 at 00:21 +0530, Raghavendra K T wrote:
>>>> * Avi Kivity <a...@redhat.com> [2012-10-04 17:00:28]:
>>>>> On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
>>>>>> On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
[...]
>>> A big concern I have (if this is 1x overcommit) for ebizzy is that it
>>> has just terrible scalability to begin with. I do not think we should
>>> try to optimize such a bad workload.
>> I think my way of running dbench has some flaw, so I went to ebizzy.
>> Could you let me know how you generally run dbench?
> I mount a tmpfs and then specify that mount for dbench to run on. This
> eliminates all IO. I use a 300 second run time and number of threads is
> equal to number of vcpus. All of the VMs of course need to have a
> synchronized start.
> I would also make sure you are using a recent kernel for dbench, where
> the dcache scalability is much improved. Without any lock-holder
> preemption, the time in spin_lock should be very low:
Hi Andrew,
I ran the test with dbench with tmpfs. I do not see any improvements in
dbench for 16k ple window.
So it seems apart from ebizzy no workload benefited by that. and I
agree that, it may not be good to optimize for ebizzy.
I shall drop changing to 16k default window and continue with other
original patch series. Need to experiment with latest kernel.
(PS: Thanks for pointing towards, perf in latest kernel. It works fine.)
Results:
dbench run for 120 sec 30 sec warmup 8 iterations using tmpfs
base = 3.6.0-rc5 with ple handler optimization patch.
x => base + ple_window = 4k
+ => base + ple_window = 16k
* => base + ple_gap = 0
dbench 1x overcommit case
=========================
N Min Max Median Avg Stddev
x 8 5322.5 5519.05 5482.71 5461.0962 63.522276
+ 8 5255.45 5530.55 5496.94 5455.2137 93.070363
* 8 5350.85 5477.81 5408.065 5418.4338 44.762697
dbench 2x overcommit case
==========================
N Min Max Median Avg Stddev
x 8 3054.32 3194.47 3137.33 3132.625 54.491615
+ 8 3040.8 3148.87 3088.615 3088.1887 32.862336
* 8 3031.51 3171.99 3083.6 3097.4612 50.526977
On Mon, 2012-10-15 at 17:40 +0530, Raghavendra K T wrote:
> On 10/11/2012 01:06 AM, Andrew Theurer wrote:
> > On Wed, 2012-10-10 at 23:24 +0530, Raghavendra K T wrote:
> >> On 10/10/2012 08:29 AM, Andrew Theurer wrote:
> >>> On Wed, 2012-10-10 at 00:21 +0530, Raghavendra K T wrote:
> >>>> * Avi Kivity <a...@redhat.com> [2012-10-04 17:00:28]:
> >>>>> On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
> >>>>>> On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
> [...]
> >>> A big concern I have (if this is 1x overcommit) for ebizzy is that it
> >>> has just terrible scalability to begin with. I do not think we should
> >>> try to optimize such a bad workload.
> >> I think my way of running dbench has some flaw, so I went to ebizzy.
> >> Could you let me know how you generally run dbench?
> > I mount a tmpfs and then specify that mount for dbench to run on. This
> > eliminates all IO. I use a 300 second run time and number of threads is
> > equal to number of vcpus. All of the VMs of course need to have a
> > synchronized start.
> > I would also make sure you are using a recent kernel for dbench, where
> > the dcache scalability is much improved. Without any lock-holder
> > preemption, the time in spin_lock should be very low:
> Hi Andrew,
> I ran the test with dbench with tmpfs. I do not see any improvements in
> dbench for 16k ple window.
> So it seems apart from ebizzy no workload benefited by that. and I
> agree that, it may not be good to optimize for ebizzy.
> I shall drop changing to 16k default window and continue with other
> original patch series. Need to experiment with latest kernel.
Thanks for running this again. I do believe there are some workloads,
when run at 1x overcommit, would benefit from a larger ple_window [with
he current ple handling code], but I do not also want to potentially
degrade >1x with a larger window. I do, however, think there may be a
another option. I have not fully worked this out, but I think I am on
to something.
I decided to revert back to just a yield() instead of a yield_to(). My
motivation was that yield_to() [for large VMs] is like a dog chasing its
tail, round and round we go.... Just yield(), in particular a yield()
which results in yielding to something -other- than the current VM's
vcpus, helps synchronize the execution of sibling vcpus by deferring
them until the lock holder vcpu is running again. The more we can do to
get all vcpus running at the same time, the far less we deal with the
preemption problem. The other benefit is that yield() is far, far lower
overhead than yield_to()
This does assume that vcpus from same VM do not share same runqueues.
Yielding to a sibling vcpu with yield() is not productive for larger VMs
in the same way that yield_to() is not. My recent results include
restricting vcpu placement so that sibling vcpus do not get to run on
the same runqueue. I do believe we could implement a initial placement
and load balance policy to strive for this restriction (making it purely
optional, but I bet could also help user apps which use spin locks).
For 1x VMs which still vm_exit due to PLE, I believe we could probably
just leave the ple_window alone, as long as we mostly use yield()
instead of yield_to(). The problem with the unneeded exits in this case
has been the overhead in routines leading up to yield_to() and the
yield_to() itself. If we use yield() most of the time, this overhead
will go away.
Here is a comparison of yield_to() and yield():
dbench with 20-way VMs, 8 of them on 80-way host:
no PLE 426 +/- 11.03%
no PLE w/ gangsched 32001 +/- .37%
PLE with yield() 29207 +/- .28%
PLE with yield_to() 8175 +/- 1.37%
Yield() is far and way better than yield_to() here and almost approaches
gang sched result. Here is a link for the perf sched map bitmap:
The thrashing is way down and sibling vcpus tend to run together,
approximating the behavior of the gang scheduling without needing to
actually implement gang scheduling.
I did test a smaller VM:
dbench with 10-way VMs, 16 of them on 80-way host:
no PLE 6248 +/- 7.69% no PLE w/ gangsched 28379 +/- .07%
PLE with yield() 29196 +/- 1.62%
PLE with yield_to() 32217 +/- 1.76%
There is some degrade from yield() to yield_to() here, but nearly as
large as the uplift we see on the larger VMs. Regardless, I have an
idea to fix that: Instead of using yield() all the time, we could use
yield_to(), but limit the rate per vcpu to something like 1 per jiffie.
All other exits use yield(). That rate of yield_to() should be more
than enough for the smaller VMs, and the result should be hopefully just
the same as the current code. I have not coded this up yet, but it's my
next step.
I am also hopeful the limitation of yield_to() will also make the 1x
issue just go away as well (even with 4096 ple_window). The vast
majority of exits will result in yield() which should be harmless.
Keep in mind this did require ensuring sibling vcpus do not share host
runqueues -I do think that can be possible given some optional scheduler
tweaks.
> Here is the summary:
> We do get good benefit by increasing ple window. Though we don't
> see good benefit for kernbench and sysbench, for ebizzy, we get huge
> improvement for 1x scenario. (almost 2/3rd of ple disabled case).
> Let me know if you think we can increase the default ple_window
> itself to 16k.
I think so, there is no point running with untuned defaults.
> I can respin the whole series including this default ple_window change.
So it seems we want dynamic PLE windows. As soon as we enter overcommit
we need to decrease the window.
-- error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
> On 10/09/2012 08:51 PM, Raghavendra K T wrote:
>> Here is the summary:
>> We do get good benefit by increasing ple window. Though we don't
>> see good benefit for kernbench and sysbench, for ebizzy, we get huge
>> improvement for 1x scenario. (almost 2/3rd of ple disabled case).
>> Let me know if you think we can increase the default ple_window
>> itself to 16k.
> I think so, there is no point running with untuned defaults.
Oaky.
>> I can respin the whole series including this default ple_window change.
> So it seems we want dynamic PLE windows. As soon as we enter overcommit
> we need to decrease the window.
Okay.
I have some rough idea on the implementation. I 'll try that after this
V2 experiments are over.
So in brief, I have this in my queue priority wise
1) V2 version of this patch series( in progress)
2) default PLE window
3) preemption notifiers
4) Pv spinlock
> On Mon, 2012-10-15 at 17:40 +0530, Raghavendra K T wrote:
>> On 10/11/2012 01:06 AM, Andrew Theurer wrote:
>>> On Wed, 2012-10-10 at 23:24 +0530, Raghavendra K T wrote:
>>>> On 10/10/2012 08:29 AM, Andrew Theurer wrote:
>>>>> On Wed, 2012-10-10 at 00:21 +0530, Raghavendra K T wrote:
>>>>>> * Avi Kivity <a...@redhat.com> [2012-10-04 17:00:28]:
>>>>>>> On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
>>>>>>>> On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
>> [...]
>>>>> A big concern I have (if this is 1x overcommit) for ebizzy is that it
>>>>> has just terrible scalability to begin with. I do not think we should
>>>>> try to optimize such a bad workload.
>>>> I think my way of running dbench has some flaw, so I went to ebizzy.
>>>> Could you let me know how you generally run dbench?
>>> I mount a tmpfs and then specify that mount for dbench to run on. This
>>> eliminates all IO. I use a 300 second run time and number of threads is
>>> equal to number of vcpus. All of the VMs of course need to have a
>>> synchronized start.
>>> I would also make sure you are using a recent kernel for dbench, where
>>> the dcache scalability is much improved. Without any lock-holder
>>> preemption, the time in spin_lock should be very low:
>> Hi Andrew,
>> I ran the test with dbench with tmpfs. I do not see any improvements in
>> dbench for 16k ple window.
>> So it seems apart from ebizzy no workload benefited by that. and I
>> agree that, it may not be good to optimize for ebizzy.
>> I shall drop changing to 16k default window and continue with other
>> original patch series. Need to experiment with latest kernel.
> Thanks for running this again. I do believe there are some workloads,
> when run at 1x overcommit, would benefit from a larger ple_window [with
> he current ple handling code], but I do not also want to potentially
> degrade >1x with a larger window. I do, however, think there may be a
> another option. I have not fully worked this out, but I think I am on
> to something.
> I decided to revert back to just a yield() instead of a yield_to(). My
> motivation was that yield_to() [for large VMs] is like a dog chasing its
> tail, round and round we go.... Just yield(), in particular a yield()
> which results in yielding to something -other- than the current VM's
> vcpus, helps synchronize the execution of sibling vcpus by deferring
> them until the lock holder vcpu is running again. The more we can do to
> get all vcpus running at the same time, the far less we deal with the
> preemption problem. The other benefit is that yield() is far, far lower
> overhead than yield_to()
> This does assume that vcpus from same VM do not share same runqueues.
> Yielding to a sibling vcpu with yield() is not productive for larger VMs
> in the same way that yield_to() is not. My recent results include
> restricting vcpu placement so that sibling vcpus do not get to run on
> the same runqueue. I do believe we could implement a initial placement
> and load balance policy to strive for this restriction (making it purely
> optional, but I bet could also help user apps which use spin locks).
> For 1x VMs which still vm_exit due to PLE, I believe we could probably
> just leave the ple_window alone, as long as we mostly use yield()
> instead of yield_to(). The problem with the unneeded exits in this case
> has been the overhead in routines leading up to yield_to() and the
> yield_to() itself. If we use yield() most of the time, this overhead
> will go away.
> Here is a comparison of yield_to() and yield():
> dbench with 20-way VMs, 8 of them on 80-way host:
> no PLE 426 +/- 11.03%
> no PLE w/ gangsched 32001 +/- .37%
> PLE with yield() 29207 +/- .28%
> PLE with yield_to() 8175 +/- 1.37%
> Yield() is far and way better than yield_to() here and almost approaches
> gang sched result. Here is a link for the perf sched map bitmap:
> The thrashing is way down and sibling vcpus tend to run together,
> approximating the behavior of the gang scheduling without needing to
> actually implement gang scheduling.
> I did test a smaller VM:
> dbench with 10-way VMs, 16 of them on 80-way host:
> no PLE 6248 +/- 7.69% > no PLE w/ gangsched 28379 +/- .07%
> PLE with yield() 29196 +/- 1.62%
> PLE with yield_to() 32217 +/- 1.76%
Hi Andrew, Results are encouraging.
> There is some degrade from yield() to yield_to() here, but nearly as
> large as the uplift we see on the larger VMs. Regardless, I have an
> idea to fix that: Instead of using yield() all the time, we could use
> yield_to(), but limit the rate per vcpu to something like 1 per jiffie.
> All other exits use yield(). That rate of yield_to() should be more
> than enough for the smaller VMs, and the result should be hopefully just
> the same as the current code. I have not coded this up yet, but it's my
> next step.
I personally feel rate limiting yield_to may be a good idea.
> I am also hopeful the limitation of yield_to() will also make the 1x
> issue just go away as well (even with 4096 ple_window). The vast
> majority of exits will result in yield() which should be harmless.
> Keep in mind this did require ensuring sibling vcpus do not share host
> runqueues -I do think that can be possible given some optional scheduler
> tweaks.
I think this is a concern (placing). Having rate limit alone may
suffice.May be tuning that taking into overcommitted/non-overcommitted
scenario also into account would be better.
Okay below is my V2 implementation I am experimenting
1) check source -and- target runq to decide on exiting the ple handler
2)
vcpu_on_spin()
{
.....
if yield_to_same_vm did not succeed and we are overcommitted
yield()
}
I think combining your thoughts and (2) complicates scenario a bit.
anyways let me see how my experiment goes. I will also check how yield
performs without any pinning.
