uv_write fails with ENOBUFS

[Ashish]

On Windows, if I call uv_write continuously from a loop, it intermittently returns -1 with error ENOBUFS

How can I avoid this? (OR just retrying uv_write after some delay?)

Please advice.

Tnx,

Ashish

[Saúl Ibarra Corretgé]

On 10/4/13 2:40 PM, Ashish wrote:
>
> On Windows, if I call uv_write continuously from a loop, it
> intermittently returns -1 with error ENOBUFS
> How can I avoid this? (OR just retrying uv_write after some delay?)
>

Assuming you allocated a non-empty buffer in the alloc_cb, that could be
the translation of WSAENOBUFS, which means the system buffers are full
AFAIS, you may need to back off and retry writing a bit later.

If you port is > 5000 this might apply to you:
http://support.microsoft.com/kb/196271


Cheers,

PS: I'm no Windows expert.

--
Sa�l Ibarra Corretg�
http://bettercallsaghul.com

[Ashish]

Sa�l Ibarra Corretg�
http://bettercallsaghul.com

I tried these tips by editing registry but it didn't help me.

When I run simulator for 10K connections, along with ENOBUFS (on server side) 

I am also getting ECONNREFUSED (on client side) for some connections. 

I believe both these errors because of not having enough internal storage.

How can we increase allocations for tcp internal buffers through libuv ? 

(I read somewhere that using setsockopt‎ we can increase internal tcp allocations. 

However I am not sure in libuv paradigm I don't know how to do that)

Tnx,

Ashish

[Ashish]

Hi,

Is there any way to proactively avoid  ENOBUFS error before calling uv_write (rather than dealing with it after uv_write returns -1) ?

I can find out how much data is still pending to send ("data size written by uv_write" minus "data size sent we get in after_write")

But how to find out available window size?

Sorry if am sounding annoying by asking question on this again and again but I am really stuck on this for long time and badly in need of solution :-(

Tnx,

Ashish

[Sam Roberts]

Do you have an example?

What do you expect to happen when you call uv_write() in a loop? If
you continuously write faster than tcp can deliver to receipient, data
needs to be buffered, and eventually those buffers will fill. Have you
tried doing your next write in the uv_write_cb, so you write as fast
as o/s will accept without blocking?
Sam

[Ashish]

On Wednesday, October 30, 2013 10:40:03 PM UTC+5:30, Sam Roberts wrote:

Do you have an example?

Yes. In my server code I am processing all incoming requests in multiple threads for e.g. say eight threads.

Now as each of these eight threads finishes request processing, event loop callback after_thread gets called

which invokes uv_write to write responses to socket. (Then this callback again start another thread to process next request)

 

Thus, uv_write gets called repetitively (as good as if we call it in a loop) and for massive number of requests it results in  ENOBUFS

I had that mechanism to write one response and in callback to write next.

But it becomes way too slow and we miss performance benefits of asynchronous writes.

In production scenario we do not have control over speed of recipients (

clients). (They might go slow coz of external reasons like network congestion etc.) 

and so we need to know how much maximum data these internal buffers can hold

So before calling uv_write we can decide, rather than calling and getting failure afterwards

(Also IMHO once we get ENOBUFS, TCP stream is potentially screwed up anyways so its 

not recommended to hold back and continue after wait. The best bet is to avoid getting into 

the situation in the first place by knowing how much data is outstanding on the link and 

managing it actively before the OS has to step in by running out of resources)

 

[Sam Roberts]

On Wed, Oct 30, 2013 at 11:03 AM, Ashish <aashee...@gmail.com> wrote:
> On Wednesday, October 30, 2013 10:40:03 PM UTC+5:30, Sam Roberts wrote:
> I had that mechanism to write one response and in callback to write next.
> But it becomes way too slow and we miss performance benefits of asynchronous
> writes.

Are you sure? As in, you benchmarked on RECEIVE side? Because while
you might be memcopy data into the kernel really quickly, so it looks
"fast", if the system is running out of bufers, then it could mean you
are making a big pile of unsent data in the kernel really "fast", not
that the receipient is geting data as fast as they can.

> (Also IMHO once we get ENOBUFS, TCP stream is potentially screwed up anyways
> so its
> not recommended to hold back and continue after wait.

Not true on Linux, doubt if its true on Windows, should be temporary.
I've seen this on dgram connections more often, not on tcp. We dealt
with by busy looping on write... waiting till ENOBUFS stopped, then
continuing. That worked well because of specifics of what we doing,
load generation, but might not work for you.

Its not clear whether you tried Saul's suggestion of pausing briefly
and trying again. Or tried to up your net stacks buffer capacity.

> The best bet is to
> avoid getting into
> the situation in the first place by knowing how much data is outstanding on
> the link and
> managing it actively before the OS has to step in by running out of
> resources)

The buf pool referred to is global to net stack, I think, not specific
to a single link. It could be caused by sheer number of connections,
not data on specfic link.

Anyhow, the logic is close to identical either way:
- you want to check before, and then pause before checking again (I
assume), and then write eventually when bufs avail
- but this is almost same as just writing, and then when ENOBUFS
occurs, pause, then write same data again, until success

Since the first would require doing something outside of uv that is
windows specific and be prone to race conditions anyhow, and latter is
possible, why not try latter?

Sam

[Bert Belder]

On Wednesday, October 30, 2013 3:48:55 PM UTC+1, Ashish wrote:
> Is there any way to proactively avoid  ENOBUFS error before calling uv_write (rather than dealing with it after uv_write returns -1) ?

Not really.

 
> I can find out how much data is still pending to send ("data size written by uv_write" minus "data size sent we get in after_write")
> But how to find out available window size?

Libuv doesn't have an API to query the window size. However this also doesn't help. ENOBUFS happens when your application exceeds it's non-paged pool allocation quota. The maximum size of the non-paged is extremely variable between versions of windows; most likely you are on 32-bit XP or something; it doesn't happen very quickly on more modern versions of windows, and I've never been able to replicate this behaviour on an x64 version.

> Sorry if am sounding annoying by asking question on this again and again but I am really stuck on this for long time and badly in need of solution :-(

This should probably all be handled in libuv itself. We could easily just queue up write requests in userspace and make sure there are never more than a handful of WSASend operations pending for any socket. However nobody currently has time to do this so you'll have to work around it or do the libuv patch yourself.

- Bert

[Ashish]

On Thursday, October 31, 2013 12:04:17 AM UTC+5:30, Sam Roberts wrote:

while you might be memcopy data into the kernel really quickly, so it looks
"fast", if the system is running out of bufers, then it could mean you
are making a big pile of unsent data in the kernel really "fast", not
that the receipient is geting data as fast as they can.

This was so true!! I just revisited the design I had assessed and rejected in the past.

But seems that was only correct approach. I just misinterpreted results. I was only 

overburdening kernel buffers and thought performance was great:-( 

Many thanks for pinpointing the issue. Made my day :-)

cheers!