Whats with this -> sendto: No buffer space available

[Kal Torak]

This keeps happening...

I have adsl running on my gateway using pppoe with userppp
and its internal nat enabled, also a fairly simple ipfw config
blocking a few ports...

Now every so often all data stops and checking on the gateway
trying to send something I get the "sendto: No buffer space available"
error and have to reboot and everything is fine again...

It seems to happen at random, sometimes its fine for a week,
sometimes only a few days and it happens... There is nothing
of interest logged in ppp.log or security or messages...

What causes this buffer to fill up and not empty? Here in australia
telstras adsl is really crap and you often cant get more than 2
hops past the peer... Im wondering if its something to do with
apps like bitchx trying to constantly reconnect when the network
is down filling up the buffer???

But shouldnt the buffer eventualy empty as things time out or something?
There is really not that much that should be trying to connect besides
a few irc sessions...

Anyway any ideas? Should I tweek some settings? Increase the max users
in the kernel? This is a really annoying little problem...

Thanks!!

To Unsubscribe: send mail to majo...@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message

[Andrew Cowan]

I use Helstra ADSL in Melbourne on freeBSD 4.3 and have never had any
problems.
I will dig up my kernel config, ppp.conf, whatever etc, if no one knows the
actual cause of the problem and see if we can find anything.

Andy

[ja...@sage-american.com]

...do you have a limit set on the firewall logging compiled in the
kernel...??? packets can overload the system as I undertand it....

At 02:20 PM 11.28.2001 +1100, Andrew Cowan wrote:
>I use Helstra ADSL in Melbourne on freeBSD 4.3 and have never had any
>problems.
>I will dig up my kernel config, ppp.conf, whatever etc, if no one knows the
>actual cause of the problem and see if we can find anything.
>
>Andy
>
>-----Original Message-----
>From: owner-free...@FreeBSD.ORG
>[mailto:owner-free...@FreeBSD.ORG]On Behalf Of Kal Torak
>Sent: Wednesday, 28 November 2001 14:07 PM
>To: FreeBSD Stable; FreeBSD ISP

Best regards,
Jack L. Stone,
Server Admin

Sage-American
http://www.sage-american.com
ja...@sage-american.com

[Brandon S. Allbery]

On Tue, 2001-11-27 at 22:06, Kal Torak wrote:
> Now every so often all data stops and checking on the gateway
> trying to send something I get the "sendto: No buffer space available"
> error and have to reboot and everything is fine again...

Beats the heck out of me, but it happens constantly on my laptop when
I'm using the VPN (mpd; dynamic addresses mean I can't use IPSEC).
Also, it constantly loses loopback packets. Usually it recovers by
itself after a minute or so. I've tried fiddling with everything I can
find and am considering instrumenting the kernel to syslog a stack trace
any time something wants to return ENOBUFS. It's really annoying.

I also see similar problems on the machine which is my PPP router / NAT
gateway.

--
brandon s. allbery [os/2][linux][solaris][japh]
all...@kf8nh.apk.net
system administrator [WAY too many hats]
all...@ece.cmu.edu
electrical and computer engineering
KF8NH
carnegie mellon university ["better check the oblivious first"
-ke6sls]

[Colin Campbell]

Hi,

Methinks you is out of "mbufs". Try a "netstat -m". You can maybe increase
mbufs by increasing "maxusers" or definitely by adding the fllowing to
your kernel config:

options NMBCLUSTERS=8192 #network mbuf clusters

Note the 8192 might be too big for your system.

Of course you'll need to rebuild your kernel.

On Wed, 28 Nov 2001, Kal Torak wrote:

> This keeps happening...
>
> I have adsl running on my gateway using pppoe with userppp
> and its internal nat enabled, also a fairly simple ipfw config
> blocking a few ports...
>
> Now every so often all data stops and checking on the gateway
> trying to send something I get the "sendto: No buffer space available"
> error and have to reboot and everything is fine again...

Colin

[Richard Chew]

Have you actually tried using netstat to actually work out your Send and
Receive Queues and also whether you have enough mbufs?

Also, try tweaking net.inet.tcp.sendspace and net.inet.tcp.recvspace and
the equivalent for udp.

Hope this helps.

Cheers,

Richard
-----------
Richard Chew
Tel: 02 6208 1913 (International: +61 2 6208 1913)
Telstra Internet Network Development

[Rowan Crowe]

bOn Wed, 28 Nov 2001, Kal Torak wrote:

> This keeps happening...
>
> I have adsl running on my gateway using pppoe with userppp
> and its internal nat enabled, also a fairly simple ipfw config
> blocking a few ports...
>
> Now every so often all data stops and checking on the gateway
> trying to send something I get the "sendto: No buffer space available"
> error and have to reboot and everything is fine again...
>
> It seems to happen at random, sometimes its fine for a week,
> sometimes only a few days and it happens... There is nothing
> of interest logged in ppp.log or security or messages...
>
> What causes this buffer to fill up and not empty? Here in australia
> telstras adsl is really crap and you often cant get more than 2
> hops past the peer... Im wondering if its something to do with
> apps like bitchx trying to constantly reconnect when the network
> is down filling up the buffer???

I see this error occasionally on a 56k modem connection. Sometimes the
modem half dies (CD light on but not passing data), and usually drops out
within a few minutes. If there's a lot of outbound data in this period,
the error pops up.

Presumably the error refers to an output buffer being full, in my case
most likely because the modem has signalled it cannot take any more data,
and ultimately packets cannot be shed from the packet queue. I haven't
used PPPoE but I would guess it means that your client cannot reach the
other end of the tunnel, which is effectively the same thing - output
queue fills and overflows.

Cheers.

--
Rowan Crowe
camrecord.com / camdiscover.com / Sensation Internet Services
Melbourne, Australia

[Rob Secombe]

Hi,

I agree. I believe it is as a result of the output queue filling as a
result of the link dying. In my case our customer was using a Alcatel ADSL
modem and it was the ethernet link between the FreeBSD box and the modem
that was falling over. I suspect that the NIC was trying to negotiate
100baseTX <full-duplex> or the like and the modem is only capable of
10baseT/UTP. The link led on the modem goes out and the media status on the
NIC was inactive in the failed state. The problem was resolved thus:

ifconfig_rl1="media 10baseT/UTP -mediaopt full-duplex up"

I have a bunch of these "out there" running on different providers,
including Tel$tra, without any problems.

Cheers

Rob.

At 15:51 28/11/01 +1100, Rowan Crowe wrote:
>bOn Wed, 28 Nov 2001, Kal Torak wrote:
>

>> This keeps happening...
>>
>> I have adsl running on my gateway using pppoe with userppp
>> and its internal nat enabled, also a fairly simple ipfw config
>> blocking a few ports...
>>
>> Now every so often all data stops and checking on the gateway
>> trying to send something I get the "sendto: No buffer space available"
>> error and have to reboot and everything is fine again...
>>
>> It seems to happen at random, sometimes its fine for a week,
>> sometimes only a few days and it happens... There is nothing
>> of interest logged in ppp.log or security or messages...
>>
>> What causes this buffer to fill up and not empty? Here in australia
>> telstras adsl is really crap and you often cant get more than 2
>> hops past the peer... Im wondering if its something to do with
>> apps like bitchx trying to constantly reconnect when the network
>> is down filling up the buffer???
>

>I see this error occasionally on a 56k modem connection. Sometimes the
>modem half dies (CD light on but not passing data), and usually drops out
>within a few minutes. If there's a lot of outbound data in this period,
>the error pops up.
>
>Presumably the error refers to an output buffer being full, in my case
>most likely because the modem has signalled it cannot take any more data,
>and ultimately packets cannot be shed from the packet queue. I haven't
>used PPPoE but I would guess it means that your client cannot reach the
>other end of the tunnel, which is effectively the same thing - output
>queue fills and overflows.
>
>Cheers.
>
>
>--
>Rowan Crowe
>camrecord.com / camdiscover.com / Sensation Internet Services
>Melbourne, Australia
>
>
>
>
>

>To Unsubscribe: send mail to majo...@FreeBSD.org

>with "unsubscribe freebsd-isp" in the body of the message

[Mark Sergeant]

I have the same problem and I have this option in my rc.conf. I also have 256
set for max users n 8096 for mbufs, suffice it to say the only time I have ever
noticed a problem is when Telstra has issues. after Telstra comes back a
ifconfig down n up combined with a stop n start of ppp fixes things fine.

Cheers,

Mark

--
Mark Sergeant | url: http://www.snsonline.net/
Unix Systems Administrator | email: mser...@snsonline.net
disclaimer: http://www.snsonline.net/disclaimer/ | mobile: +61 4 1271 42631

[Greg Black]

Rob Secombe wrote:

| I agree. I believe it is as a result of the output queue filling as a
| result of the link dying. In my case our customer was using a Alcatel ADSL
| modem and it was the ethernet link between the FreeBSD box and the modem
| that was falling over. I suspect that the NIC was trying to negotiate
| 100baseTX <full-duplex> or the like and the modem is only capable of
| 10baseT/UTP. The link led on the modem goes out and the media status on the
| NIC was inactive in the failed state. The problem was resolved thus:
|
| ifconfig_rl1="media 10baseT/UTP -mediaopt full-duplex up"

What kind of modem? My Alcatel ADSL modem states in its manual
that it provides a 10BaseT half-duplex interface and that's the
way I run it. I've never seen these problems, but I had a long
wait for Telstra to put the ADSL on the correct line and I've
only had my ADSL service running for a few days so far.

[Rob Secombe]

Alcatel Speed Touch Home.

Rob.

At 17:06 28/11/01 +1000, Greg Black wrote:

>Rob Secombe wrote:
>
>| I agree. I believe it is as a result of the output queue filling as a
>| result of the link dying. In my case our customer was using a Alcatel ADSL
>| modem and it was the ethernet link between the FreeBSD box and the modem
>| that was falling over. I suspect that the NIC was trying to negotiate
>| 100baseTX <full-duplex> or the like and the modem is only capable of
>| 10baseT/UTP. The link led on the modem goes out and the media status on the
>| NIC was inactive in the failed state. The problem was resolved thus:
>|
>| ifconfig_rl1="media 10baseT/UTP -mediaopt full-duplex up"
>

>What kind of modem? My Alcatel ADSL modem states in its manual
>that it provides a 10BaseT half-duplex interface and that's the
>way I run it. I've never seen these problems, but I had a long
>wait for Telstra to put the ADSL on the correct line and I've
>only had my ADSL service running for a few days so far.
>

[Greg Black]

I wrote:

| Rob Secombe wrote:
|
| | I agree. I believe it is as a result of the output queue filling as a
| | result of the link dying. In my case our customer was using a Alcatel ADSL
| | modem and it was the ethernet link between the FreeBSD box and the modem
| | that was falling over. I suspect that the NIC was trying to negotiate
| | 100baseTX <full-duplex> or the like and the modem is only capable of
| | 10baseT/UTP. The link led on the modem goes out and the media status on the
| | NIC was inactive in the failed state. The problem was resolved thus:
| |
| | ifconfig_rl1="media 10baseT/UTP -mediaopt full-duplex up"
|
| What kind of modem? My Alcatel ADSL modem states in its manual
| that it provides a 10BaseT half-duplex interface and that's the
| way I run it. I've never seen these problems, but I had a long
| wait for Telstra to put the ADSL on the correct line and I've
| only had my ADSL service running for a few days so far.

Sigh. I left out the type of modem I was using. It's an
Alcatel Speed Touch Home ADSL modem.

[Andrew Khanoff]

On Tue, 27 Nov 2001 ja...@sage-american.com wrote:

> Date: Tue, 27 Nov 2001 21:27:44 -0600

> ...do you have a limit set on the firewall logging compiled in the
> kernel...??? packets can overload the system as I undertand it....
>

hi, all
I've got pretty much the same errors on both of my 4.4-STABLE
routers(3.5-STABLE based work seems like ok). More to say, the second one
is router with very little network load and with no ipfw rules except
allow ip from any to any, and 100Mbps NIC onboard(connected btw in
10baseTx hub). netstat -m gives only 9 per cent use of mbuffers. What's
up? Something strange is going on after lattest cvsups. Services running:
squid, snmpd(running this one and polling it with mrtg encreases
probability of "network lockup" in several times), pop3, sendmail, sshd.
TIA for any help in this problem.

-------------------------
wsi# netstat -m
68/384/4096 mbufs in use (current/peak/max):
68 mbufs allocated to data
64/100/1024 mbuf clusters in use (current/peak/max)
296 Kbytes allocated to network (9% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
--------------------------
wsi# uname -a

FreeBSD abc.kiev.ua 4.4-STABLE FreeBSD 4.4-STABLE #0: Fri Nov 9 10:43:08
EET 2001 en...@abc.kiev.ua:/usr/src/sys/compile/WSI i386

WBR
Andrew Khanoff

"UCT" Internet service provider
System administrator
--
AK11361-RIPE | ANK1-UANIC phone: +380442208170
ICQ#:9157815
AS20785 MNT

[Ian Pulsford]

I had the same problem last night as I was using several newsreaders to
access nntpcache on my FreeBSD 4.4-STABLE dialup router. It was fixed
by hanging up the dialup connection and reconnecting. Also running
squid, apache, icecast, postgresql. However I don't think it ran out of
mbufs:

137/640/16384 mbufs in use (current/peak/max):
132 mbufs allocated to data
5 mbufs allocated to packet headers
131/390/4096 mbuf clusters in use (current/peak/max)
940 Kbytes allocated to network (7% of mb_map in use)

0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

Also have large send and recieve buffers set in /etc/sysctl.conf:

net.inet.tcp.sendspace=65535
net.inet.tcp.recvspace=65535

It also happened about a month ago but I decided to let it slide unless
it repeated.

IanP

[Jonathan Hanna]

Anyone here of more progress on this problem?

I had one case where a restart of natd fixed it. The
ipfw rules were also flushed and reset as part of that.
I am on semi-static DHCP with the appropriate natd arguments.

Jonathan Hanna <jha...@shaw.ca>

[Doug Silver]

It happens all the time on one of my nameservers, but only when amanda is
trying to back it up. Then it says this:

> Dec 4 23:00:43 HOST syslogd: sendto: No buffer space available
> Dec 5 00:00:36 HOST syslogd: sendto: No buffer space available

It does not affect the machine other than this message.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Doug Silver
Network Manager
Quantified Systems, Inc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Kal Torak]

Doug Silver wrote:

> It happens all the time on one of my nameservers, but only when amanda is
> trying to back it up. Then it says this:
>
>
>>Dec 4 23:00:43 HOST syslogd: sendto: No buffer space available
>>Dec 5 00:00:36 HOST syslogd: sendto: No buffer space available
>>
>
> It does not affect the machine other than this message.

Hmmm, no affect???

When ever I get this message the machine can no longer send any data
to the network on that interface... The interface has to be reset...

Perhaps the cause of your message is due to lack of mbufs for the load
its getting, you should probably increase them...

The problem the rest of us have is different... I am almost 100% sure
this is being caused by a problem upstream, the data cant get out so
the sendto buffer fills and seems to lock up the interface... I have a
feeling this may be specific to PPP connections, anyone finding this
locks there interface on a non PPP connection??? (pppoe is still ppp
could this even be specific to pppoe??)

Anyway to my way of thinking this is an error in the handeling of this
kind of situation and needs to be looked at... Is there anything that
I can log or dump next time this happens that would be helpfull in fixing
this???

[Jonathan Hanna]

No PPP involved with me, and I think with many others. I agree that the
"no affect" above does look like ordinary buffer exhaustion, though I also have
a working network except for one interface (or maybe divert socket?).

A message that seems not to have made it to the list:

From: Freddie Cash <fc...@ocis.net>
To: Jonathan Hanna <jha...@shaw.ca>
Subject: Re: Whats with this -> sendto: No buffer space available
Cc: freeb...@freebsd.org

...

On December 5, 2001 12:46 am, you wrote:
> Anyone here of more progress on this problem?

> I had one case where a restart of natd fixed it. The
> ipfw rules were also flushed and reset as part of that.
> I am on semi-static DHCP with the appropriate natd arguments.

We've run into this problem quite often. We have a dozen or so FreeBSD
4.x boxes set up as NAT/ipfw firewall/gateways with 3C509B cards.
Every so often, the links will just die and the only way to bring it
back is to down the card for a couple seconds to clear the buffers.

The cause, for us, appears to be these wonderful viral worms
propagating through Outlook. A flurry of outgoing connections are
made, NAT'd on the internal card, and nothing comes back, filling the
buffer and blocking all outbound traffic. Fortunately, the external
card continues to work and we can log in remotely to restart the card,
scan the logs, and track down the infected computer.

It's still a royal pain in the arse, though. :) Damn, I really wish
people would smarten up and not open every single piece of mail they
get.

PS: This might not make it to the list as the reverse DNS on our main
NAT box is hooped. Could you forward this to the isp@ list if it
doesn't show up there shortly? Thanks.

--
Cheers,
Freddie _ _ _ _ _ _ _ _ _ _ PhoenixTek Consulting
fc...@bigfoot.com _ _ _ _ _ Unix, Network, Security Administration
_ _ _ _ _ _ _ _ _ _ _ _ _ _ (250) 376-5885

Jonathan Hanna <jha...@shaw.ca>

[Brandon S. Allbery]

On Wed, 2001-12-05 at 23:28, Kal Torak wrote:
> The problem the rest of us have is different... I am almost 100% sure
> this is being caused by a problem upstream, the data cant get out so
> the sendto buffer fills and seems to lock up the interface... I have a
> feeling this may be specific to PPP connections, anyone finding this
> locks there interface on a non PPP connection??? (pppoe is still ppp
> could this even be specific to pppoe??)

Add PPTP to the list; that's how I lose. And in my case the "problem"
is that when I see it I'm using wavelan to either a machine with a 28.8
dialup, or a congested access point. Also, ppp on the former locks up
in this fashion.

However, in the case of the PPTP connection there are some differences:

- it tends to drop loopback packets as an early indication that it's
about to report ENOBUFS;

- it recovers by itself after a few minutes.

Note that "netstat -m" shows this machine nowhere near any mbuf limits,
and increasing #mbufs or #mbclusters has no effect.

--
brandon s. allbery [os/2][linux][solaris][japh]
all...@kf8nh.apk.net
system administrator [WAY too many hats]
all...@ece.cmu.edu
electrical and computer engineering
KF8NH
carnegie mellon university ["better check the oblivious first"
-ke6sls]

[Kal Torak]

Jonathan Hanna wrote:

>
> No PPP involved with me, and I think with many others. I agree that the
> "no affect" above does look like ordinary buffer exhaustion, though I also have
> a working network except for one interface (or maybe divert socket?).

Hmmm, perhaps this is related to NAT then???

Only the interface running NAT locks up for me, everything else works
normaly... Is everyone with this problem running NAT or a divert socket
of some kind on the interface that dies???

[Bri]

I have an interface that dies and its running

IPFIREWALL
IPDIVERT

options in the kernel and the external interface dies the machine becomes
unstable and reboots well sometimes other times you have to follow the
emergency shutdown procedure it does when that system has become unstable.

its a 3com card running on xl0

and of course with all this I'm running divert socket to.

-----Original Message-----
From: owner-free...@FreeBSD.ORG
[mailto:owner-free...@FreeBSD.ORG]On Behalf Of Kal Torak

Sent: 07 December 2001 05:38
Cc: FreeBSD Stable; FreeBSD ISP
Subject: Re: Whats with this -> sendto: No buffer space available

[Jeff Lynch]

On Fri, 7 Dec 2001, Kal Torak wrote:

> Jonathan Hanna wrote:
>
> >
> > No PPP involved with me, and I think with many others. I agree that the
> > "no affect" above does look like ordinary buffer exhaustion, though I also have
> > a working network except for one interface (or maybe divert socket?).
>
> Hmmm, perhaps this is related to NAT then???

We see this without using NAT between a 3.x and 4.x box sitting
on fastether feeds at two separate colo spaces. Latency is pretty
consistent between 30-40ms. We've turned off encryption/compression/stat
without help, but switching to tcp seems to help. But we are moving
traffic off the vtun as quickly as possible, so that could also be
a factor to the noticable increase in stability.

============================================================================
Jeffrey A. Lynch | JORSM Internet, Regional Internet Services
email: je...@jorsm.com | 7 Area Codes in Chicagoland and NW Indiana
Voice: (219)322-2180 | 100Mbps+ Connectivity, 56K-DS3, V.90, ISDN
Autoresponse: in...@jorsm.com | Quality Service, Affordable Prices
http://www.jorsm.com | Serving Gov, Biz, Residential Since 1995

[Jonathan Hanna]

Latest data point here: this time it was fixed mysteriously.
After noticing it was down, "netstat -m" showed no serious mbuf
use or peak use. "ep0" had:
ep0: flags=cc43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,LINK2,MULTICAST> mtu 1500
and the IP address was set up correctly

"tcpdump -n -i ep0" worked, and after that the behavior was back to normal.
No restart of natd was done, no firewall rules flushed or reset.
This is on 4.3-STABLE FreeBSD 4.3-STABLE #3: Sun Jul 15 00:27:41 PDT 2001

On 07-Dec-01 Bri wrote:
> I have an interface that dies and its running
>
> IPFIREWALL
> IPDIVERT
>
> options in the kernel and the external interface dies the machine becomes
> unstable and reboots well sometimes other times you have to follow the
> emergency shutdown procedure it does when that system has become unstable.
>
> its a 3com card running on xl0
>
> and of course with all this I'm running divert socket to.
> -----Original Message-----
> From: owner-free...@FreeBSD.ORG
> [mailto:owner-free...@FreeBSD.ORG]On Behalf Of Kal Torak
> Sent: 07 December 2001 05:38
> Cc: FreeBSD Stable; FreeBSD ISP
> Subject: Re: Whats with this -> sendto: No buffer space available
>
>
> Jonathan Hanna wrote:
>
>>
>> No PPP involved with me, and I think with many others. I agree that the
>> "no affect" above does look like ordinary buffer exhaustion, though I also
> have
>> a working network except for one interface (or maybe divert socket?).
>
> Hmmm, perhaps this is related to NAT then???
>
> Only the interface running NAT locks up for me, everything else works
> normaly... Is everyone with this problem running NAT or a divert socket
> of some kind on the interface that dies???

Jonathan Hanna <jha...@shaw.ca>

[Haikal Saadh]

>
>
> On Fri, 7 Dec 2001, Kal Torak wrote:
>
> > Jonathan Hanna wrote:
> >
> > >
> > > No PPP involved with me, and I think with many others. I
> agree that
> > > the "no affect" above does look like ordinary buffer exhaustion,
> > > though I also have a working network except for one interface (or
> > > maybe divert socket?).
> >
> > Hmmm, perhaps this is related to NAT then???
>
> We see this without using NAT between a 3.x and 4.x box
> sitting on fastether feeds at two separate colo spaces.
> Latency is pretty consistent between 30-40ms. We've turned
> off encryption/compression/stat without help, but switching
> to tcp seems to help. But we are moving traffic off the vtun
> as quickly as possible, so that could also be a factor to the
> noticable increase in stability.
>

Sorry if I'm a bit late to this party, but I've had this come up once,
and it was on a dial up link to a box (ppp link, connected to another
ordinary modem at 33.6k), and I was concurrently running nmap and saint,
and needless to say it happened about halfway through the scans.

IPFirewall was running, but I had it set to allow any from any to any.
NAT was not running, and had never been running on that box either.

And no I'm not a script kiddie, I was asked to see if that box had any
major holes open :).

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

[Jonathan Hanna]

On 07-Dec-01 Jonathan Hanna wrote:
>
> Latest data point here: this time it was fixed mysteriously.
> After noticing it was down, "netstat -m" showed no serious mbuf
> use or peak use. "ep0" had:
> ep0: flags=cc43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,LINK2,MULTICAST> mtu 1500
> and the IP address was set up correctly
>
> "tcpdump -n -i ep0" worked, and after that the behavior was back to normal.
> No restart of natd was done, no firewall rules flushed or reset.
> This is on 4.3-STABLE FreeBSD 4.3-STABLE #3: Sun Jul 15 00:27:41 PDT 2001

I think I may have found the problem; it may be a bug in the ep 3c509 driver.
The OACTIVE flag being set was a clue and I think tcpdump setting promiscuous
mode may have reset the adapter.

I have sent mail to the maintainer, but I am having some mail problems and am
not sure it was received. Can anyone confirm that my number (3) question below
does identify the problem?

I am having ENBUFS problems with a 486 running 4.X stable on an 10BaseT ep interface.
The ENOBUFS comes from ip_output seeing that the send queue is full. When this
happens the OACTIVE flag is set and the interface is stuck untill ifconfig down/up is done.

Looking at the ep driver, a few questions arise:

(This is $FreeBSD: src/sys/dev/ep/if_ep.c,v 1.95.2.2 2000/07/17 21:24:26 archie Exp $)

1) in ep_if_start:

if (inw(BASE + EP_W1_FREE_TX) < len + pad + 4) {
/* no room in FIFO */
outw(BASE + EP_COMMAND, SET_TX_AVAIL_THRESH | (len + pad + 4));
/* make sure */
if (inw(BASE + EP_W1_FREE_TX) < len + pad + 4) {
ifp->if_flags |= IFF_OACTIVE;
return;
}
} else {
outw(BASE + EP_COMMAND, SET_TX_AVAIL_THRESH | EP_THRESH_DISABLE);
}

If the second test of available TX space decides there is enough, should
there be a disabling of the threshold as there is in the else clause?

2) in ep_intr:

void
ep_intr(arg)
void *arg;
{
struct ep_softc *sc;
register int status;
struct ifnet *ifp;
int x;

x = splbio();

sc = (struct ep_softc *)arg;

if (sc->gone)
return;

Should there not be an splx(x) if sc->gone is true?

3) in ep_intr:

rescan:

while ((status = inw(BASE + EP_STATUS)) & S_5_INTS) {

/* first acknowledge all interrupt sources */
outw(BASE + EP_COMMAND, ACK_INTR | (status & S_MASK));

if (status & (S_RX_COMPLETE | S_RX_EARLY)) {
old_status = status;
epread(sc);
continue;
}

It seems that if an RX is done then any TX conditions are lost as an ack of all
sources has been done. I have verified that on the next look after the epread
TX conditions have disappeared without being handled. If OACTIVE is set, then
no if_start will not kikl the driver out of this condition. I suspect this is
the problem I am seeing.

4) if_timer is set when a packet is sent out, but disabled as soon as any TX space
is available. Any reason not to wait for completion?

Thanks.

[Nuno Subtil]

On Tue, Feb 05, 2002 at 07:50:14PM -0800, Jonathan Hanna wrote:
>
> On 07-Dec-01 Jonathan Hanna wrote:
> >
> > Latest data point here: this time it was fixed mysteriously.
> > After noticing it was down, "netstat -m" showed no serious mbuf
> > use or peak use. "ep0" had:
> > ep0: flags=cc43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,LINK2,MULTICAST> mtu 1500
> > and the IP address was set up correctly
> >
> > "tcpdump -n -i ep0" worked, and after that the behavior was back to normal.
> > No restart of natd was done, no firewall rules flushed or reset.
> > This is on 4.3-STABLE FreeBSD 4.3-STABLE #3: Sun Jul 15 00:27:41 PDT 2001

I have also seen these symptoms, but on an ISDN interface. The machine does
NAT, and sometimes it just stops working. I can ssh into it through the
"internal" NIC (an ISA NE2000 clone), but no traffic ever goes out through the
ISDN interface --- any program that generates traffic through that interface
returns the same error: "No buffer space available". netstat -m also shows
nothing apparently wrong with mbuf usage. This happens about once or twice
every two months or so, and a reboot fixes it. I can't recall seeing any
special flags on the interface when this happens, nor do I remember trying to
run tcpdump on the interface, but I can try that if it happens again.

The ISDN card is a mysterious "combo" card which apparently does video-capture
as well. The ISDN part works with the Teles S0/16.3 driver, modified to accept
the card's signature. NAT is done through ipnat, and the ppp connection is
managed by the userland ppp daemon. It runs 4.3-STABLE from around the
beginning of May 2001.

Nuno Subtil
nsu...@vega.dhis.co.it.pt

[Nuno Subtil]

Karl Dietz asked for this to be forwarded to the lists, so here it is.

----- Forwarded message from Karl Dietz <Karl....@TRIPLAN.com> -----

From: "Karl Dietz" <Karl....@TRIPLAN.com>
Date: Wed, 6 Feb 2002 19:02:51 +0100
To: "Jonathan Hanna" <jha...@shaw.ca>, <nsu...@vega.dhis.co.it.pt>
Subject: Re: Whats with this -> sendto: No buffer space available

X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4807.1700
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300

Meanwhile I have browsed through the code of ep and vx.

> 3) in ep_intr:
>
> rescan:
>
> while ((status = inw(BASE + EP_STATUS)) & S_5_INTS) {
>
> /* first acknowledge all interrupt sources */
> outw(BASE + EP_COMMAND, ACK_INTR | (status & S_MASK));
>
> if (status & (S_RX_COMPLETE | S_RX_EARLY)) {
> old_status = status;
> epread(sc);
> continue;
> }
>
> It seems that if an RX is done then any TX conditions are lost as an ack
of all
> sources has been done. I have verified that on the next look after the
epread
> TX conditions have disappeared without being handled. If OACTIVE is set,

the vx driver (which is to be folded back into ep) has a more verbose
comment on the code in question:

/*
* Acknowledge any interrupts. It's important that we do this
* first, since there would otherwise be a race condition.
* Due to the i386 interrupt queueing, we may get spurious
* interrupts occasionally.
*/

But the handler is written in a way that tx interupts get lost only if
a card failure occurs.

Anyway the modified driver has transfered more than 4 million packets
already without a problem (two interfaces, some flood pings, and the
whole internet traffic)

Regards,
Karl

PS: Can one of you forward this to the list?

----- End forwarded message -----