Re: [nodejs] Why does htop show mutiple threads for node.js ?

[Ben Noordhuis]

On Mon, Nov 26, 2012 at 9:17 PM, Stefan Happer <stefan...@gmail.com> wrote:
> Sorry for maybe dumb question but why does "htop" show multiple threads for
> node.js ? Isn't node.js single-threaded?
>
> This is on a 4-core server running Ubuntu 12.04 LTS

node.js is single-threaded insofar that JS code is executed from one
thread only. Some operations however, notably file operations, are
handed off to a thread pool.

As to why, it's because of inherent limitations in today's popular
operating systems: none support true asynchronous file operations.

[Ben Noordhuis]

On Thu, Nov 29, 2012 at 2:43 AM, Tolga Tekin <ctt...@yahoo.com> wrote:
> Actually that is not completely true - Both Windows and MacOSX's native file
> APIs support async file operations.
> The limitation might be coming from linux posix api.

I'm aware of them but neither are really usable.

Windows AIO only works with files opened in direct I/O mode, meaning
you bypass the disk cache. When a file system doesn't support AIO, it
either raises an error (in which case you have to fall back to
user-space threads) or silently (!) switches to synchronous I/O.

OS X AIO is implemented with kernel threads but the default settings
are way too low to do anything useful, you have to tweak a bunch of
sysctls first. FreeBSD has similar issues (the implementations are
very similar) and you need to manually load a kernel module first,
otherwise everything fails with ENOSYS.

Native Linux AIO has the same issues as Windows. glibc's POSIX AIO
implementation doesn't even bother with it, it always uses user-space
threads.

The only operating system I know of that has a remotely usable
implementation is Solaris and who uses that?

[Jérémy Lal]

On 29/11/2012 03:41, Ben Noordhuis wrote:
> The only operating system I know of that has a remotely usable
> implementation is Solaris and who uses that?

Starts with a J and ends with a t...
:)

[mscdex]

Javascript? ;-)

[Stefan Happer]

Ben, thank you for your answer. That explains why some of my node apps use 5 threads and others use just one (and later two, when express.js starts to server HTTP requests)

[Bert Belder]

The problem in windows is mostly that it silently switches to synchronous AIO in many cases, some of which are documented but others are not. This can be bypassed by using direct mode I/O but it has many other disadvantages.

As for solaris, I am not convinced that is has usable kernel AIO. As far as I know solaris kaio will (transparently) fall back to using threads when reading from or writing to ordinary files.

[Alex Kocharin]

Hi Ben,


Would it make sense to use AIO if filesystem supports it? How faster it would be? Did somebody run any benchmarks?


I don't know what Windows is (sorry, just trolling), but Linux have a lot of filesystems and it's open. If just one filesystem supports AIO and work with node.js much better, other will follow. If something silently fall back to sync IO, file a bugreport against it, it's as simple as this.


Somebody could even write a npm package that does AIO for a specified filesystem. So, anything come up to one question: is it worth it? Any benchmarks?


--
// alex

29.11.2012, 06:41, "Ben Noordhuis" <in...@bnoordhuis.nl>:

> On Thu, Nov 29, 2012 at 2:43 AM, Tolga Tekin <ctt...@yahoo.com> wrote:
>

>> О©╫Actually that is not completely true - Both Windows and MacOSX's native file
>> О©╫APIs support async file operations.
>> О©╫The limitation might be coming from linux posix api.

>
> I'm aware of them but neither are really usable.
>
> Windows AIO only works with files opened in direct I/O mode, meaning

> you bypass the disk cache. О©╫When a file system doesn't support AIO, it

> either raises an error (in which case you have to fall back to
> user-space threads) or silently (!) switches to synchronous I/O.
>
> OS X AIO is implemented with kernel threads but the default settings
> are way too low to do anything useful, you have to tweak a bunch of

> sysctls first. О©╫FreeBSD has similar issues (the implementations are

> very similar) and you need to manually load a kernel module first,
> otherwise everything fails with ENOSYS.
>

> Native Linux AIO has the same issues as Windows. О©╫glibc's POSIX AIO

> implementation doesn't even bother with it, it always uses user-space
> threads.
>
> The only operating system I know of that has a remotely usable
> implementation is Solaris and who uses that?
>

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Ben Noordhuis]

On Wed, Dec 5, 2012 at 9:35 AM, Alex Kocharin <al...@kocharin.ru> wrote:
> Would it make sense to use AIO if filesystem supports it? How faster it would be? Did somebody run any benchmarks?

The answer lies somewhere between 'not at all' and 'lots'.

From anecdotal evidence, AIO on FreeBSD and Solaris can give a nice
boost. The Samba people reported a 50% increase in random I/O
throughput when they switched to AIO on FreeBSD.

On the other hand, Linux AIO only works with files opened in O_DIRECT
mode. You no longer benefit from the disk cache so in general it will
be slower, possibly a lot slower. O_DIRECT only makes sense for very
specialized workloads.

OS X is not worth considering even if it wasn't broken. No one runs
his production systems on OS X.

> I don't know what Windows is (sorry, just trolling), but Linux have a lot of filesystems and it's open. If just one filesystem supports AIO and work with node.js much better, other will follow. If something silently fall back to sync IO, file a bugreport against it, it's as simple as this.

Yes, well... I've been following the discussions on the LKML since
2006 or 2007. I've little hope of seeing it fixed in our lifetimes.
:-(

[Ryan Schmidt]

On Dec 5, 2012, at 04:11, Ben Noordhuis <in...@bnoordhuis.nl> wrote:

> OS X is not worth considering even if it wasn't broken. No one runs
> his production systems on OS X.

Certainly many users use node on OS X on their development systems, and Apple does sell capable servers and a server OS, so why shouldn't I use node on OS X in production? Calling 66 million users' choice of OS "broken" doesn't seem constructive, and given the lengths that this community has recently gone to to be inclusive of and add support for Windows, I'm surprised at this dismissive attitude with regard to such a popular UNIX OS.

[Alex Kocharin]

Hi Ryan,

1) The talk is about AIO, not about OS (and yes, it is broken anywhere except Solaris according to prev. discussion, didn't check though)
2) The talk is about performance (which means things will still work, but a bit slower)
3) The talk is about production servers where bits of performance do matter (do 66 million people even exist in this industry?)

--
Regards,
Alex

05.12.2012, 14:57, "Ryan Schmidt" <googl...@ryandesign.com>:

> On Dec 5, 2012, at 04:11, Ben Noordhuis <in...@bnoordhuis.nl> wrote:
>

>> О©╫OS X is not worth considering even if it wasn't broken. О©╫No one runs
>> О©╫his production systems on OS X.

>
> Certainly many users use node on OS X on their development systems, and Apple does sell capable servers and a server OS, so why shouldn't I use node on OS X in production? Calling 66 million users' choice of OS "broken" doesn't seem constructive, and given the lengths that this community has recently gone to to be inclusive of and add support for Windows, I'm surprised at this dismissive attitude with regard to such a popular UNIX OS.
>

[Ben Noordhuis]

On Wed, Dec 5, 2012 at 11:11 AM, Ben Noordhuis <in...@bnoordhuis.nl> wrote:
> On the other hand, Linux AIO only works with files opened in O_DIRECT
> mode. You no longer benefit from the disk cache so in general it will
> be slower, possibly a lot slower. O_DIRECT only makes sense for very
> specialized workloads.

I should probably have mentioned that the situation has much improved
with recent kernels (though still not perfect) and that O_DIRECT is no
longer a hard requirement for reliable AIO. If you're writing bespoke
software and control the kernel it's getting deployed on, AIO may be a
feasible option.

For node.js it's more complicated. It runs on a broad range of
kernels and there are still lots of 2.6.18 machines out there, or even
older.

I don't think sniffing the kernel version is going to cut it: distro
vendors tend to back-port and forward-port kernel patches a lot. For
example, the delta between a mainline 2.6.32 kernel and RHEL 2.6.32 is
_massive_.

[Ben Noordhuis]

That particular comment was about the AIO implementation, which is so
limited in the stock configuration that 'broken' is an apt
description.

There is a lot wrong with OS X from a system programmer's perspective
but I hardly ever complain about it in public. I would if I thought
it'd shame Apple's kernel engineers into fixing their shit but that
seems highly unlikely.