Dropping support for unix datagram sockets

[Bert Belder]

Hey all,

The node core team wants to drop node's support for unix datagram
sockets in v0.6. We think nobody is using it anyway.
But we might be wrong. If you *are* using this functionality, speak up
now or forever hold your peace.

- Bert

[Ben Noordhuis]

Another thing is that multicast support for dgram sockets is - at
least for now - disabled. I'd like to know how many people actually
use that and for what (the what is optional, just genuinely curious).

[Ted Young]

Pretty sure they're still using it at etsy: https://github.com/etsy/statsd

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Elijah Insua]

> The node core team wants to drop node's support for unix datagram
> sockets in v0.6.

Is this a permanent removal or will it be reintroduced after 0.6?

-- Elijah

[Matt Stevens]

On Aug 23, 7:34 pm, Ben Noordhuis <i...@bnoordhuis.nl> wrote:
> Another thing is that multicast support for dgram sockets is - at
> least for now - disabled. I'd like to know how many people actually
> use that and for what (the what is optional, just genuinely curious).

I tinkered with a UPnP/DLNA media server, the ability to join a UDP
multicast group is needed there to advertise the services. Based on
this thread it sounds like others are interested in the same thing:

https://groups.google.com/forum/#!topic/nodejs/OG74K2jHPc0

[Bert Belder]

> Pretty sure they're still using it at etsy: https://github.com/etsy/statsd

It looks like etsy uses only udp datagram sockets.

> Is this a permanent removal or will it be reintroduced after 0.6?

Permanent removal.

[Joshua Holbrook]

Might I suggest moving the code for datagram sockets to userspace?
Surely, if a few people are using them, they would be interested in
maintaining the code as a separate project?

Just my $0.02,

--Josh

[hij1nx.dev]

what is the motivation to remove it?

-- 
hij1nx
http://twitter.com/hij1nx

http://github.com/hij1nx

[Ben Combee]

webOS developers are using datagrams for UDP both to and from
services. We're not dead, and I'd hate to see that removed from the
platform.

[Ben Noordhuis]

On Wed, Aug 24, 2011 at 05:05, Ben Combee <ben.c...@gmail.com> wrote:
> webOS developers are using datagrams for UDP both to and from
> services.  We're not dead, and I'd hate to see that removed from the
> platform.

*UNIX* dgrams, not UDP dgrams. I just spent the last week slaving over
UDP support in libuv.

[Dean Landolt]

On Tue, Aug 23, 2011 at 11:05 PM, Ben Combee <ben.c...@gmail.com> wrote:

webOS developers are using datagrams for UDP both to and from
services.  We're not dead, and I'd hate to see that removed from the
platform.

I'm pretty sure Bert's only referring to the datagram flavor of unix domain sockets, not removing UDP support altogether.

[Thomas Shinnick]

This is indeed an opportunity to establish future directions on how to preserve stable provision of function sets.

I'm assuming that in the push to achieve compatibility with Windows the node team have discovered that that requires a step backwards, removing 'Unix' data sockets from core.

How do you accomplish something like this without total "deprecation"?  That is, how to provide a path for functionality to enter the core, when appropriate, _and_ for functionality to *leave* the core when appropriate?  Without removing that functionality and breaking people's use of node?

I've seen instances where it was felt code just wasn't of high enough utility to justify being in the core.  And so it was said it should stay in and be provided at the user-land level. 

Isn't this also the answer for functions which have become "less needed" or "difficult to provide" in the core?  There should be the ability to shift core code into user-land modules.  There should be the precedent that this has been done. 

Agreeing with Joshua., but asking for a bit more, I'd like to see the core team agree that functions formerly provided in core, but that need to now be removed, would naturally be moved into user space modules.  By default, not by exception...

[Joshua Holbrook]

> I'd like to see the core team agree that functions formerly provided in core, but that need to
> now be removed, would naturally be moved into user space modules. By default, not by
> exception.

I would also like to see this. I think it's an approach that jibes
well with Node's philosophies regarding "core vs. userspace" and
whatnot.

--Josh

[Aikar]

This is a really bad idea. Regardless if anyone's using it it not, a
major networking piece should be done in core once and done well.

[Aria Stewart]

On Tuesday, August 23, 2011 at 9:34 PM, Ben Noordhuis wrote:

> On Wed, Aug 24, 2011 at 03:23, Bert Belder <bertb...@gmail.com (mailto:bertb...@gmail.com)> wrote:
> > The node core team wants to drop node's support for unix datagram
> > sockets in v0.6. We think nobody is using it anyway.
> > But we might be wrong. If you *are* using this functionality, speak up
> > now or forever hold your peace.
>
> Another thing is that multicast support for dgram sockets is - at
> least for now - disabled. I'd like to know how many people actually
> use that and for what (the what is optional, just genuinely curious).

I was planning on writing an implementation of Resilient Streaming Protocol, but it's UDP-multicast, not UNIX-domain.

----
Aria Stewart

[Ryan Grove]

Cluster uses unix datagram sockets: https://github.com/LearnBoost/cluster/

See https://github.com/LearnBoost/cluster/blob/master/lib/master.js#L129

- Ryan

[Ben Combee]

Ah, thanks for the clarification. Unix domain datagrams aren't being
used. Nevermind.

[shimonchayim]

At this point our arch is based on cluster. For another project we are
using udp datagrams and if I understand correctly that is not going
away right?

Also, on a different note, why do we want to deprecate functionality
if you want to support Windows. Just throw and exception or something
when unix sockets are used with Windows. This way folks who care about
Windows won't use the code and folks who don't care carry forward what
they have done.


On Aug 23, 9:15 pm, Ryan Grove <r...@wonko.com> wrote:

[马涛]

I am using the unix datagram in my app for receiving local log and trans to the log center
 server

2011/8/24 Bert Belder <bertb...@gmail.com>

[Isaac Schlueter]

Cluster is fairly popular. Breaking it should at least have really
good justification, or better yet, a clear workaround.

[Jeroen Janssen]

Hi,

just to clarify.. does this mean that from the
http://nodejs.org/docs/v0.4.11/api/dgram.html#dgram.createSocket only
'unix_dgram' will be removed?
(i.e. the rest: udp4 and udp6 will keep on existing?)

Best regards,

Jeroen

[Lee Treveil]

this is going away: socket(AF_UNIX, SOCK_DGRAM, 0)

correct?

[Lee Treveil]

Looking at the source sheds some light:

https://github.com/joyent/node/blob/master/src/node_net.cc#L227

[Jeroen Janssen]

Yes,

Multicast UDP support is needed for UPnP and there are also some other
discovery/announcement protocols that depend on it.

http://en.wikipedia.org/wiki/Simple_Service_Discovery_Protocol
http://en.wikipedia.org/wiki/Service_Location_Protocol
http://en.wikipedia.org/wiki/UPNP

I am not sure if it uses UDP, but Bonjour/ZeroConf also uses a
multicast mechanism in order to work.

[Renato Elias]

Yep !

I´m use multicast for auto-discover, :´(

Renato Elias
Tel: +55 (011) 7694-3872

[Bradley Meck]

I use it to keep my secure RPC off the network entirely and give permissions to access the RPC. If this can be duplicated (the permissions, I can firewally the box) it sounds fine. If this can be moved to userland this is fine, but I suspect some people will end up using Windows Pipes for similar purposes as Unix Domain Sockets.

[Ben Noordhuis]

On Wed, Aug 24, 2011 at 11:22, Jeroen Janssen <jeroen....@gmail.com> wrote:
> just to clarify.. does this mean that from the
> http://nodejs.org/docs/v0.4.11/api/dgram.html#dgram.createSocket only
> 'unix_dgram' will be removed?
> (i.e. the rest: udp4 and udp6 will keep on existing?)

Yes, that's correct. unix_dgram is out, udp4 and udp6 are in.
Broadcast and multicast support is missing at the moment but we'll add
those.

[Ben Noordhuis]

On Wed, Aug 24, 2011 at 09:28, Isaac Schlueter <i...@izs.me> wrote:
> Cluster is fairly popular.  Breaking it should at least have really
> good justification, or better yet, a clear workaround.

Use UDP sockets or streaming UNIX sockets.

I've taken a peek at cluster's source. The sockets are used for a
JSON-RPC like protocol. It doesn't seem like they need unix_dgram as
such (please correct me if I'm wrong).

[Matt]

Can you answer hij1nx's question? What's the motivation for removing it?

--

[Ben Noordhuis]

On Wed, Aug 24, 2011 at 16:18, Matt <hel...@gmail.com> wrote:
> Can you answer hij1nx's question? What's the motivation for removing it?

Sure: hard to get right cross-platform and almost no one uses it. We
have limited developer resources so spending a man week on something
that has only a handful of users didn't seem worthwhile, there are
bigger fish to fry.

[Matt]

That seems fair enough... But then is there a reason the release can't get pushed back a man week to get it working? Seems like it might not be the easiest thing to add in as a userland option, since you'd have to hook into the event loop directly, rather than the eio_custom() stuff...

Or alternatively just make it non-Windows only in 6.0 and if someone wants it for windows ask for patches.

--

[Nicholas Campbell]

Pretty certain a dependency like Cluster means that quite a few more than "almost no one" use it. In addition to what Issac said, there are 6 packages that depend on cluster (within NPM alone -> http://search.npmjs.org/#/cluster), Haraka and Calipso being somewhat popular as well.

IMO, If it isn't broken out to user land then push the release a week and fix it.

- Nick Campbell

http://digitaltumbleweed.com

--

[Subbu Allamaraju]

Could you elaborate on your data points about "almost no one uses it"?

As Issac pointed out, Cluster uses it, and it is fairly popular.

Subbu

[Ryan Dahl]

Everyone should relax a little and understand what is being asked.

- Unix domain sockets are a very important feature and we will
continue to support it forever

- UDP is a very important feature and we will continue to support it forever

What we're removing support for is socket(AF_UNIX, SOCK_DGRAM, 0). It
is almost certain that no one is using this. We want to remove it
because the equivalent does not exist on windows and it's a strange
functionality.

[Ryan Dahl]

Oops - Cluster is actually using this. TJ will need to frame his
protocol for transport over unix stream sockets. This is fairly minor.

Unrelated to the AF_UNIX SOCK_DGRAM discussion, we're likely to be
changing our API for sharing file descriptors across processes which
will probably require cluster (and similar modules) to be rewritten
totally. A discussion about that can be moved to the node-dev list.

[Marco Rogers]

+1 to removing it if it helps move to 0.6 more quickly.

My feeling is that this is a major horizon in node development. Once true Windows compatibility is achieved there will be a huge influx of new users as well as new and varied use cases. Lots of things will probably get a second look in terms of what "core" is. The sooner we reach this milestone, the sooner we can start sorting things out.

As for Cluster, those guys are pretty smart and I'm sure they don't *need* unix dgrams to achieve what they need to. They'll probably just use some other socket type and no one will be the wiser.

With that said, I think what people would like to hear is that Ryan and Joyent have a strategy for evolving node in a sane fashion across Windows and *nix systems. In general I think people would like to hear that node will still take advantage of nice features that are well supported on unix, even if they can't be properly reproduced on Windows.

:Marco

[David Goehrig]

I've been using them for mDNS discovery. If you drop support I guess I'll have to fork.

Dave

-=-=- da...@nexttolast.com -=-=-

On Aug 23, 2011, at 9:23 PM, Bert Belder <bertb...@gmail.com> wrote:

> Hey all,
>
> The node core team wants to drop node's support for unix datagram
> sockets in v0.6. We think nobody is using it anyway.
> But we might be wrong. If you *are* using this functionality, speak up
> now or forever hold your peace.
>
> - Bert
>

[Diogo Resende]

Why not just loopback/socket file? Usually they don't even go down the
network stack.

---
Diogo R.

[David Goehrig]

On Wed, Aug 24, 2011 at 8:47 AM, Ben Noordhuis <in...@bnoordhuis.nl> wrote:

I've taken a peek at cluster's source. The sockets are used for a
JSON-RPC like protocol. It doesn't seem like they need unix_dgram as
such (please correct me if I'm wrong).

Using unix streams would significantly complicate the code in cluster's master.js as it would need to maintain an individual socket connection / session per child node process.  Using dgrams you don't suffer from all of that connection maintenance overhead if you don't need it. I have an project that does the same thing for communicating between node & a server written in Erlang for message passing on the same box.  

--
-=-=-=-=-=-=-=-=-=-=- http://blog.dloh.org/

[Ryan Dahl]

On Wed, Aug 24, 2011 at 4:12 AM, David Goehrig <da...@nexttolast.com> wrote:
> I've been using them for mDNS discovery. If you drop support I guess I'll have to fork.

You're using unix datagram sockets for mDNS discovery?

[Mikeal Rogers]

node.js succeeds because it limits it's scope.

We don't specify the language, we don't build the VM, and we don't go too far up the stack building a huge stdlib.

0.6 is adding another dimension, node.js doesn't do Unix (or Windows) specific APIs. Those go in to userland modules now.

It makes sense to me. If we want Windows support to be sustainable node should not make an effort to support features that don't appear in both Unix and Windows.

Of course, userland modules are free to provide APIs for all kinds of OS specific features.

-Mikeal

[Matt]

I understand what's being asked. My argument though is that I don't see a way currently to implement this in userland unless you provide a way to say "Give me all events on this FD" in your C++ API. I could be missing that though - the C++ side of things is an area that needs documented (not a criticism per-se, just something to go on the list).

If it's implementable in userland with the current API (i.e. if I'm wrong about the above) then I say scrap it. People who want unix-only features can write those as external libraries.

Matt.

[Shimon Doodkin]

its bad idea node.js as i understood was planed to be used create
network apps.

[Joshua Holbrook]

Shimon said:
> its bad idea node.js as i understood was planed to be used create
network apps.

This is true. On the other hand, we ain't Twisted either (which, for
example, has irc-specific code in its core). It's okay for
functionality to live entirely in userspace, really! ;)

Matt said:
> My argument though is that I don't see a way currently to implement this in userland unless you provide a way to say "Give me all events on this FD" in your C++ API.

This is a good point. How hard would it be to implement unix datagram
sockets in userspace? Is it doable?

Matt said:
> If it's implementable in userland with the current API (i.e. if I'm wrong about the above) then I say scrap it. People who want unix-only features can write those as external libraries.

I think "scrap it" is dangerous here, since I think the node core devs
should be open to the idea of moving such code into external libraries
instead of just killing it.

Speaking of: Would anybody here that currently uses unix datagram
sockets be interested in maintaining such a project? I mean, if nobody
uses them then the answer is obviously "no," but it sounds like some
people might be.

--Josh

[Tim Caswell]

My only requirement/request is to be able to implement mDNS
advertisement and discovery without having to resort to a C++ addon. If
I understand correctly, the dgram unix stuff is unrelated, but we are
still missing the broadcast and multicast parts needed.

[Matt]

On Wed, Aug 24, 2011 at 3:37 PM, Tim Caswell <t...@creationix.com> wrote:

On 08/24/2011 05:41 AM, Ben Noordhuis wrote:

On Wed, Aug 24, 2011 at 11:22, Jeroen Janssen<jeroen.janssen@gmail.com>  wrote:

just to clarify.. does this mean that from the
http://nodejs.org/docs/v0.4.11/api/dgram.html#dgram.createSocket only
'unix_dgram' will be removed?
(i.e. the rest: udp4 and udp6 will keep on existing?)

Yes, that's correct. unix_dgram is out, udp4 and udp6 are in.
Broadcast and multicast support is missing at the moment but we'll add
those.

My only requirement/request is to be able to implement mDNS advertisement and discovery without having to resort to a C++ addon.  If I understand correctly, the dgram unix stuff is unrelated, but we are still missing the broadcast and multicast parts needed.

Yes it's unrelated, but I think everything you need for mDNS should be there. You'll have to construct your own DNS packets though.

See tests/simple/test-dgram-multicast.js

Matt.

[Bert Belder]

On Aug 24, 9:37 pm, Tim Caswell <t...@creationix.com> wrote:
> My only requirement/request is to be able to implement mDNS
> advertisement and discovery without having to resort to a C++ addon.  If
> I understand correctly, the dgram unix stuff is unrelated, but we are
> still missing the broadcast and multicast parts needed.

Yes, the dgram unix stuff is unrelated.
Yes, broadcast/multicast is still missing, but this will be re-added
before 0.6.

[Matt]

On Wed, Aug 24, 2011 at 3:35 PM, Joshua Holbrook <josh.h...@gmail.com> wrote:

Matt said:

> My argument though is that I don't see a way currently to implement this in userland unless you provide a way to say "Give me all events on this FD" in your C++ API.

This is a good point. How hard would it be to implement unix datagram
sockets in userspace? Is it doable?

Probably, but the "how" is likely to change with UV. Right now you could probably (I presume) create your own ev_io in C++ and ev_init() that.

 Matt said:

> If it's implementable in userland with the current API (i.e. if I'm wrong about the above) then I say scrap it. People who want unix-only features can write those as external libraries.

I think "scrap it" is dangerous here, since I think the node core devs
should be open to the idea of moving such code into external libraries
instead of just killing it.

Well that's what I mean by scrap it. Very much like my node-fs-ext library.

Matt.

[tjholowaychuk]

Cluster doesn't need to use local sockets, I can change that, I'm fine
with the removal

On Aug 24, 4:42 am, Renato Elias <renato.el...@gmail.com> wrote:
> Yep !
>
> I´m use multicast for auto-discover, :´(
>
> Renato Elias
> Tel: +55 (011) 7694-3872
>

[Isaac Schlueter]

On Wed, Aug 24, 2011 at 21:15, tjholowaychuk <tjholo...@gmail.com> wrote:
> Cluster doesn't need to use local sockets, I can change that, I'm fine
> with the removal

Well that settles that, then. :)

[David Goehrig]

I have a small daemon (in Erlang) that is rebroadcasting on a unix socket locally, multiple processes in node listen to it and use the unix dgram for communication.  Could I have them listen for UDP?  Sure, but that would suck resources on the network stack side, and unix dgram sockets are significantly more reliable in my tests.

It's really handy for also having node communicate to erlang via ports this way, much much easier than having to deal with all the stupid session overhead.

Dave

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
-=-=-=-=-=-=-=-=-=-=- http://blog.dloh.org/

[Ben Noordhuis]

On Fri, Aug 26, 2011 at 03:48, David Goehrig <da...@nexttolast.com> wrote:
> I have a small daemon (in Erlang) that is rebroadcasting on a unix socket
> locally, multiple processes in node listen to it and use the unix dgram for
> communication.  Could I have them listen for UDP?  Sure, but that would suck
> resources on the network stack side, and unix dgram sockets are
> significantly more reliable in my tests.
> It's really handy for also having node communicate to erlang via ports this
> way, much much easier than having to deal with all the stupid session
> overhead.

If there's demand, I'll post a module that provides API-compatible
unix_dgram support.

I find it hard to believe that loopback UDP performs significantly
worse or less reliable than UNIX sockets - I know those parts of the
linux and freebsd source well. Show me numbers.