dev.ix.0.queueX.interrupt_rate

[Slawa Olhovchenkov]

I have '82599ES 10-Gigabit SFI/SFP+ Network Connection' and see
strange:

# sysctl dev.ix.0 | grep interrupt_rate
dev.ix.0.queue7.interrupt_rate: 500000
dev.ix.0.queue6.interrupt_rate: 500000
dev.ix.0.queue5.interrupt_rate: 31250
dev.ix.0.queue4.interrupt_rate: 31250
dev.ix.0.queue3.interrupt_rate: 31250
dev.ix.0.queue2.interrupt_rate: 500000
dev.ix.0.queue1.interrupt_rate: 500000
dev.ix.0.queue0.interrupt_rate: 500000

Last -stable, no tuning. Is this normal?

_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

[Eric Joyner]

What's strange about it? The interrupt rate only changes if traffic goes
out on the queue, and so if whatever applications you use don't utilize the
core the queue is bound to, then the interrupt rate won't change.

Or are you confused about the units? I know ixl(4) uses usecs instead of
Hz, so ix(4) should probably change to use that at some point.

[Slawa Olhovchenkov]

On Mon, Aug 24, 2015 at 10:13:05PM +0000, Eric Joyner wrote:

> What's strange about it? The interrupt rate only changes if traffic goes
> out on the queue, and so if whatever applications you use don't utilize the
> core the queue is bound to, then the interrupt rate won't change.
>
> Or are you confused about the units? I know ixl(4) uses usecs instead of
> Hz, so ix(4) should probably change to use that at some point.

By default `hw.ix.max_interrupt_rate: 31250` and I confused about some
queue exceed this.

And I can set dev.ix.0.queue4.interrupt_rate to 31250 (by sysctl) and
this changes accepted.

I am to be out of it.

[Babak Farrokhi]

You may want to disable interrupt moderation and manually configure
interrupt rate per queue if you have high pps. I have had a lot of
headaches with ix driver and fixed it by manually setting interrupt
rates.

sysctl hw.ix.enable_aim=0


—
Babak

[Gerrit Kühn]

On Mon, 24 Aug 2015 22:29:26 +0300 Slawa Olhovchenkov <s...@zxy.spb.ru>
wrote about dev.ix.0.queueX.interrupt_rate:

SO> Last -stable, no tuning. Is this normal?

From 10.2-rel (and still having severe performance issues with NFS as
reported before):

dev.ix.0.queue7.interrupt_rate: 31250
dev.ix.0.queue6.interrupt_rate: 100000
dev.ix.0.queue5.interrupt_rate: 83333
dev.ix.0.queue4.interrupt_rate: 100000
dev.ix.0.queue3.interrupt_rate: 500000
dev.ix.0.queue2.interrupt_rate: 31250
dev.ix.0.queue1.interrupt_rate: 31250
dev.ix.0.queue0.interrupt_rate: 500000



cu
Gerrit

[Rick Macklem]

Gerritt Kuhn wrote:
> On Mon, 24 Aug 2015 22:29:26 +0300 Slawa Olhovchenkov <s...@zxy.spb.ru>
> wrote about dev.ix.0.queueX.interrupt_rate:
>
> SO> Last -stable, no tuning. Is this normal?
>
> From 10.2-rel (and still having severe performance issues with NFS as
> reported before):
>
> dev.ix.0.queue7.interrupt_rate: 31250
> dev.ix.0.queue6.interrupt_rate: 100000
> dev.ix.0.queue5.interrupt_rate: 83333
> dev.ix.0.queue4.interrupt_rate: 100000
> dev.ix.0.queue3.interrupt_rate: 500000
> dev.ix.0.queue2.interrupt_rate: 31250
> dev.ix.0.queue1.interrupt_rate: 31250
> dev.ix.0.queue0.interrupt_rate: 500000
>

If you have tso enabled, you could try this patch:
https://reviews.freebsd.org/D3477

If TSO is disabled, then we don't have an explanation for poor NFS performance
yet. If you haven't seen it, you might want to keep an eye on this thread:
http://docs.FreeBSD.org/cgi/mid.cgi?55DC1B5A.8010109

rick

[Gerrit Kühn]

On Tue, 25 Aug 2015 07:55:49 -0400 (EDT) Rick Macklem
<rmac...@uoguelph.ca> wrote about Re: dev.ix.0.queueX.interrupt_rate:


RM> If you have tso enabled, you could try this patch:
RM> https://reviews.freebsd.org/D3477
RM>
RM> If TSO is disabled, then we don't have an explanation for poor NFS
RM> performance yet.

I tried both with or without TSO, and it does not appear to make any
difference for me. I get about 50MB/s net write speed and about 200MB/s
read. Even my 1GBE interface perform better in terms of writing.

RM> If you haven't seen it, you might want to keep an eye
RM> on this thread: http://docs.FreeBSD.org/cgi/mid.cgi?55DC1B5A.8010109

Yes, I am watching this. Thanks for the pointer.

[Slawa Olhovchenkov]

On Tue, Aug 25, 2015 at 04:04:41PM +0200, Gerrit K"uhn wrote:

> On Tue, 25 Aug 2015 07:55:49 -0400 (EDT) Rick Macklem
> <rmac...@uoguelph.ca> wrote about Re: dev.ix.0.queueX.interrupt_rate:
>
>
> RM> If you have tso enabled, you could try this patch:
> RM> https://reviews.freebsd.org/D3477
> RM>
> RM> If TSO is disabled, then we don't have an explanation for poor NFS
> RM> performance yet.
>
> I tried both with or without TSO, and it does not appear to make any
> difference for me. I get about 50MB/s net write speed and about 200MB/s
> read. Even my 1GBE interface perform better in terms of writing.
>
> RM> If you haven't seen it, you might want to keep an eye
> RM> on this thread: http://docs.FreeBSD.org/cgi/mid.cgi?55DC1B5A.8010109
>
> Yes, I am watching this. Thanks for the pointer.

For discover poor network performance you need:

1. check cables
2. capture interface statisctics from all ports on paths on network
switchs
3. capture NIC statisctics on hosts (netstat -nbeI)
4. ifconfig -a
5. netstat -rn
6. /var/log/dmesg.boot

[Gerrit Kühn]

On Tue, 25 Aug 2015 19:45:08 +0300 Slawa Olhovchenkov <s...@zxy.spb.ru>
wrote about Re: dev.ix.0.queueX.interrupt_rate:

SO> For discover poor network performance you need:

[...]

I am already through this, see
<https://lists.freebsd.org/pipermail/freebsd-net/2015-June/042536.html>

I just wanted to let /you/ know that my interrupt rates look similar (you
asked for that).
I think there are a couple of people on the mailing lists reporting
performance issues with ix interfaces and (especially?) NFS lately. Would
be great if we could find and fix the root cause(s)...


cu
Gerrit

[Slawa Olhovchenkov]

On Wed, Aug 26, 2015 at 08:31:14AM +0200, Gerrit K"uhn wrote:

> On Tue, 25 Aug 2015 19:45:08 +0300 Slawa Olhovchenkov <s...@zxy.spb.ru>
> wrote about Re: dev.ix.0.queueX.interrupt_rate:
>
> SO> For discover poor network performance you need:
>
> [...]
>
> I am already through this, see
> <https://lists.freebsd.org/pipermail/freebsd-net/2015-June/042536.html>

I am don't subscribe to freebsd-net.
I am quick look into thread and don't see this information.
For you case collet also `top -PHS` and may be you need capture nfs
seession by tcpdump and analize this by wireshark (may be solaris
servered synchronus write).

> I just wanted to let /you/ know that my interrupt rates look similar (you
> asked for that).

My case is very special -- I am realy have 3Mpps syn-flood and try to
mitigate this by netmap filtering.

> I think there are a couple of people on the mailing lists reporting
> performance issues with ix interfaces and (especially?) NFS lately. Would
> be great if we could find and fix the root cause(s)...

Nobody do instrumental analysis. I see only guesting.

My experience talk: before solve software problem make sure no
hardware problem. For this need to collect `show interface`, netstat
and etc.

My experience talk: before tuning make sure what tuning need and so.
For this need capturing nfs session and analise by wireshark.
May be just solaris on every write do sync.