NUMA support; tweaking TCP for GPRS

[Sean Hamilton]

Greetings -hackers,

I have two unrelated questions.

First, what is the status of NUMA support in FreeBSD? Is
there a performance penalty on Nehalem-class systems,
compared with Linux, which advertises NUMA awareness? Google
seems to turn up very little on this subject.

Second, I am using a FreeBSD server to talk to equipment
which has a GPRS internet connection. This is fairly high
latency (approximately one second RTT) and is prone to
bursts of packet loss, or bursts of extremely high latency
-- perhaps up to a minute. These intervals cause many
retransmissions, which I presume is a good strategy over the
internet, but not so good for GPRS.

For my application, latency is mostly irrelevant. However,
data over GPRS is very expensive, so I would like to reduce
as much as possible the number of TCP retransmissions made
on the FreeBSD side, possibly at the expense of latency.

So, I am looking for suggestions on how to achieve this, via
sysctl, setsockopt, etc. There seems to be a lot of
literature regarding TCP tuning, but usually the focus is on
improving performance, not reducing network traffic. The
"rexmit_min" and "rexmit_sop" sysctls mentioned in tcp(4)
seem interesting, but it's not clear to me exactly how they
might be adjusted for this purpose.

Thanks in advance,

--
Sean Hamilton <s...@bel.bc.ca>
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

[Gary Jennejohn]

On Thu, 12 Nov 2009 18:08:35 -0800
Sean Hamilton <s...@bel.bc.ca> wrote:

> Second, I am using a FreeBSD server to talk to equipment
> which has a GPRS internet connection. This is fairly high
> latency (approximately one second RTT) and is prone to
> bursts of packet loss, or bursts of extremely high latency
> -- perhaps up to a minute. These intervals cause many
> retransmissions, which I presume is a good strategy over the
> internet, but not so good for GPRS.
>
> For my application, latency is mostly irrelevant. However,
> data over GPRS is very expensive, so I would like to reduce
> as much as possible the number of TCP retransmissions made
> on the FreeBSD side, possibly at the expense of latency.
>
> So, I am looking for suggestions on how to achieve this, via
> sysctl, setsockopt, etc. There seems to be a lot of
> literature regarding TCP tuning, but usually the focus is on
> improving performance, not reducing network traffic. The
> "rexmit_min" and "rexmit_sop" sysctls mentioned in tcp(4)
> seem interesting, but it's not clear to me exactly how they
> might be adjusted for this purpose.
>
> Thanks in advance,
>

Highjacking this thread.

This won't help you, but it would be interesting to have the new
Delay Tolerant Networking protocol developed by Vint Cerf for NASA
in FreeBSD. Supposedly it's already in Android.

---
Gary Jennejohn

[John Baldwin]

On Thursday 12 November 2009 9:08:35 pm Sean Hamilton wrote:
> Greetings -hackers,
>
> I have two unrelated questions.
>
> First, what is the status of NUMA support in FreeBSD? Is
> there a performance penalty on Nehalem-class systems,
> compared with Linux, which advertises NUMA awareness? Google
> seems to turn up very little on this subject.

Yes, there is a penalty. I have some very simplistic NUMA support in a p4
branch (//depot/user/jhb/numa/...) that just makes threads allocate memory
close to the current CPU when requesting a free page. It's not very general
purpose, but it can be useful if you pin all your tasks to specific packages.
I haven't tested it with non-pinned workloads to see what effect it has.

--
John Baldwin

[Sean Hamilton]

On Fri, Dec 18, 2009 at 12:47:11PM +1100,
Lawrence Stewart wrote:
> Sean Hamilton wrote:
> > [...]

> > Second, I am using a FreeBSD server to talk to equipment
> > which has a GPRS internet connection. This is fairly high
> > latency (approximately one second RTT) and is prone to
> > bursts of packet loss, or bursts of extremely high latency
> > -- perhaps up to a minute. These intervals cause many
> > retransmissions, which I presume is a good strategy over the
> > internet, but not so good for GPRS.
> >
> > For my application, latency is mostly irrelevant. However,
> > data over GPRS is very expensive, so I would like to reduce
> > as much as possible the number of TCP retransmissions made
> > on the FreeBSD side, possibly at the expense of latency.
> >
> > So, I am looking for suggestions on how to achieve this, via
> > sysctl, setsockopt, etc. There seems to be a lot of
> > literature regarding TCP tuning, but usually the focus is on
> > improving performance, not reducing network traffic. The
> > "rexmit_min" and "rexmit_sop" sysctls mentioned in tcp(4)
> > seem interesting, but it's not clear to me exactly how they
> > might be adjusted for this purpose.
>

> Just back from 6 months of travel and a bit busy right now catching up
> on things, but ping me in a week or two about the TCP issue and I can
> provide some insights.

At this point, the issue is mostly resolved -- we were
seeing a huge number of TCP retransmissions, but this was
due to braindead and undocumented behaviour on the GPRS
device side, not FreeBSD's fault at all.

Specifically, the TCP stack on the device (eCos-based, but
possibly using the TCP stack in the GSM chipset -- not sure)
was setting some of its parameters based on the requested
GPRS service class. For instance, if the GPRS service class
indicated a desired average RTT of 0.5s, then it would just
retransmit every 0.5s, or something like that -- the exact
behaviour is undocumented, but clearly laughably broken.

Anyway, I am still really interested in what you might have
to say about this. I have already increased the minimum
retransmission interval to a few seconds, with some success.