I want to bind a zeromq socket to an available TCP port assigned by the OS. However the standard way of doing this, binding to port 0, fails with EINVAL. Indeed, there is a check in the code saying that 0 is not a valid port. Is there any reason for this? Is there another way of doing this?
Also it would be useful to have a way to fetch the port number that was assigned, although personally I can live without it.
Regards,
-- Pierre Ynard "Une âme dans un corps, c'est comme un dessin sur une feuille de papier." _______________________________________________ zeromq-dev mailing list zeromq-...@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
On Wed, Jan 25, 2012 at 8:58 AM, Pierre Ynard <linkfa...@yahoo.fr> wrote: > I want to bind a zeromq socket to an available TCP port assigned by the > OS. However the standard way of doing this, binding to port 0, fails > with EINVAL. Indeed, there is a check in the code saying that 0 is not a > valid port. Is there any reason for this? Is there another way of doing > this?
If you're using the high level C binding this works. It's not a core libzmq functionality. Some other bindings may also provide it.
Thanks, but I don't really want to write an ugly for loop to (badly) reinvent a feature of the socket API. Also this code ignores the OS settings for local port assignation.
Why is it disabled in libzmq?
-- Pierre Ynard "Une âme dans un corps, c'est comme un dessin sur une feuille de papier." _______________________________________________ zeromq-dev mailing list zeromq-...@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
On Wed, Jan 25, 2012 at 11:22 AM, Pierre Ynard <linkfa...@yahoo.fr> wrote: > Thanks, but I don't really want to write an ugly for loop to (badly) > reinvent a feature of the socket API. Also this code ignores the OS > settings for local port assignation.
> Why is it disabled in libzmq?
No good reason I can see. If you want it, make the change and send a pull request to libzmq.
Pierre Ynard <linkfa...@yahoo.fr> wrote: > > If you're using the high level C binding this works. It's not a core > > libzmq functionality. Some other bindings may also provide it.
> Thanks, but I don't really want to write an ugly for loop to (badly) > reinvent a feature of the socket API. Also this code ignores the OS > settings for local port assignation.
+1, I have proposed this functionality before, but we could not figure out a way to make the API work. Its obvious that many people need this, and have implemented it with the same hack as czmq.
> Why is it disabled in libzmq?
It is disabled because in ZeroMQ, by design, there is no way to work with individual underlying connections of a socket. Hence, there is no API equivalent to e.g. getsockname() which you could use to get the port back from the OS.
If you can think of a way to make this work, without exposing the details of the individual connections to the application, and without a backward-incompatible API change, please propose it (or even better, make a patch).
On Wed, Jan 25, 2012 at 6:19 PM, Martin Lucina <mar...@lucina.net> wrote: > It is disabled because in ZeroMQ, by design, there is no way to work > with individual underlying connections of a socket. Hence, there is no > API equivalent to e.g. getsockname() which you could use to get the port > back from the OS.
The functionality regards binding not connections, this isn't the concern.
> If you can think of a way to make this work, without exposing the > details of the individual connections to the application, and > without a backward-incompatible API change, please propose it (or > even better, make a patch).
Current working code would not break by returning the port number via the zmq_bind API.
The other solution, if this is too ugly (which is debatable) is via a zmq_getsockopt call. That was your suggestion previously, iirc. Both are worth exploring.
I don't recall any valid technical concerns with this change proposal, it came down to opinion.
Pieter Hintjens <p...@imatix.com> wrote: > On Wed, Jan 25, 2012 at 6:19 PM, Martin Lucina <mar...@lucina.net> wrote:
> > It is disabled because in ZeroMQ, by design, there is no way to work > > with individual underlying connections of a socket. Hence, there is no > > API equivalent to e.g. getsockname() which you could use to get the port > > back from the OS.
> The functionality regards binding not connections, this isn't the concern.
s/connections/endpoints/.
> > If you can think of a way to make this work, without exposing the > > details of the individual connections to the application, and > > without a backward-incompatible API change, please propose it (or > > even better, make a patch).
> Current working code would not break by returning the port number via > the zmq_bind API.
How? zmq_bind() does not have an out parameter. You mean abusing the (int) return value of zmq_bind() to return a port number? That's horrible and not all transports have such a thing as a port number, so it's not generic either.
> The other solution, if this is too ugly (which is debatable) is via a > zmq_getsockopt call. That was your suggestion previously, iirc. Both > are worth exploring.
> I don't recall any valid technical concerns with this change proposal, > it came down to opinion.
The major technical problem which everyone (including me) forgets every time this comes up is that you can bind a single ZeroMQ socket to multiple endpoints. Cf. what I said above, some of the bound endpoints may not even have such a thing as a "port number".
To create an API for this you basically end up with some way to enumerate the bound endpoints and get them to the application. And at that point we're coming dangerously close to letting people enumerate connected endpoints, etc.
On Wed, Jan 25, 2012 at 6:58 PM, Martin Lucina <mar...@lucina.net> wrote: > How? zmq_bind() does not have an out parameter. You mean abusing the > (int) return value of zmq_bind() to return a port number? That's > horrible and not all transports have such a thing as a port number, > so it's not generic either.
Yes, it's horrid, I didn't claim it was neat :-) But it is technically compatible since it will not break current working code.
> The major technical problem which everyone (including me) forgets every > time this comes up is that you can bind a single ZeroMQ socket to > multiple endpoints. Cf. what I said above, some of the bound endpoints > may not even have such a thing as a "port number".
Speak for yourself, I did not forget about this :) Binds, as I said deliberately, are synchronous. That means you can bind and then immediately ask for the port number.
Further, 95% of applications bind exactly one time. And further to that, it is an extreme edge case (one we can ignore IMO) to bind twice to system-provided ports.
> To create an API for this you basically end up with some way to > enumerate the bound endpoints and get them to the application. And at > that point we're coming dangerously close to letting people enumerate > connected endpoints, etc.
Please read Martin's excellent posting about solving the right problems only and not the full range of theoretical problems. It is amazing how even experienced engineers can focus so intensely on solving, perfectly, the wrong problem.
No-one is asking for an API that lets you bind 10 times to port zero and get all ten results. 100% of use cases here are for binding once to port zero and getting exactly one result back.
It is bad design process to exaggerate the user's requirements into absurd extremity and then state there is no simple solution. It's far wiser to understand the actual problem people are trying to solve, and solve that simply and minimally.
The elegant and minimal solution is of course to return the system assigned port via a zmq_getsockopt and I'd make that patch except it's evening here in Dallas and there's a bar with barely-clad waitresses that has a cold beer waiting for me.
On 26/01/2012, at 12:28 PM, Pieter Hintjens wrote:
> The elegant and minimal solution is of course to return the system > assigned port via a zmq_getsockopt and I'd make that patch except it's > evening here in Dallas and there's a bar with barely-clad waitresses > that has a cold beer waiting for me.
Later in the nigh there'll be a beer with bearly clad waitresses surfing in it
Pieter Hintjens <p...@imatix.com> wrote: > No-one is asking for an API that lets you bind 10 times to port zero > and get all ten results. 100% of use cases here are for binding once > to port zero and getting exactly one result back.
Indeed. In fact, the OP was specifically asking for a use case where he could bind an unspecified number of times to port zero, and get zero results back. :-)
> It is bad design process to exaggerate the user's requirements into > absurd extremity and then state there is no simple solution. It's far > wiser to understand the actual problem people are trying to solve, and > solve that simply and minimally.
Given that an API will stick around for a long time, it is good API design process to extrapolate current and possible future use cases and arrive at the simplest possible API covering all these use cases.
On Wed, Jan 25, 2012 at 10:05 PM, Martin Lucina <mar...@lucina.net> wrote: > Indeed. In fact, the OP was specifically asking for a use case where he > could bind an unspecified number of times to port zero, and get zero > results back. :-)
Pierre may clarify whether "unspecified" is 1, or N > 1.
> Given that an API will stick around for a long time, it is good API > design process to extrapolate current and possible future use cases > and arrive at the simplest possible API covering all these use cases.
That road leads to AMQP/1.0. Solve the core majority problem minimally, period. It is terrible API design (indeed, design in general) to comingle abstract philosophically potential problems with real ones and end up with incredibly complex solutions of which people only use 10%.
Even if (some) users claim to require N, which is unproven here and IMO inaccurate, it would be necessary to measure the cost of solving 1 vs. the cost of solving N, and measure the value to all users of that solution. Often solving N is simpler than solving 1. In this case, it's not. Solving N would satisfy 0.01% of users and make a solution horrible to 99.9% of users. Quod Erat Demonstrandum.
If anyone on this list needs to bind one socket multiple times to port 0, please do raise a hand.
Not to mention the "binds are synchronous" statement that you seem to be ignoring in the pursuit of complexity.
On Thu, Jan 26, 2012 at 05:19, Pieter Hintjens <p...@imatix.com> wrote: > If anyone on this list needs to bind one socket multiple times to port > 0, please do raise a hand.
I don't, but I have an idea. Not sure if this was already proposed, but what about allowing to "upgrade" TCP socket to ZeroMQ socket? I.e. allow someone to bind a socket in a way he want and then "import" it into ZeroMQ socket as one of the connections _______________________________________________ zeromq-dev mailing list zeromq-...@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
On Thu, Jan 26, 2012 at 4:05 AM, Martin Lucina <mar...@lucina.net> wrote: > > It is bad design process to exaggerate the user's requirements into > > absurd extremity and then state there is no simple solution. It's far > > wiser to understand the actual problem people are trying to solve, and > > solve that simply and minimally.
> Given that an API will stick around for a long time, it is good API > design process to extrapolate current and possible future use cases > and arrive at the simplest possible API covering all these use cases.
Is it possible that this API should simply be on a different calls, a zmq_bind_assign or similar, to avoid this clash? This could then work with all transports:
tcp/pgm: bind to random port ipc: bind to random free name in system tmp inproc: bind to random free name
That can either have a return parameter which gives you the bound address, or it can be passed back via a second argument or similar.
> > Indeed. In fact, the OP was specifically asking for a use case where > > he could bind an unspecified number of times to port zero, and get > > zero results back. :-)
> Pierre may clarify whether "unspecified" is 1, or N > 1.
In my case I would only need to bind the socket to one port.
Regards,
-- Pierre Ynard "Une âme dans un corps, c'est comme un dessin sur une feuille de papier." _______________________________________________ zeromq-dev mailing list zeromq-...@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
On Thu, Jan 26, 2012 at 2:17 AM, Ian Barber <ian.bar...@gmail.com> wrote: > Is it possible that this API should simply be on a different calls, a > zmq_bind_assign or similar, to avoid this clash? This could then work with > all transports:
> tcp/pgm: bind to random port > ipc: bind to random free name in system tmp > inproc: bind to random free name
That would work, but so would a less surprising zmq_bind plus a getsockopt with return value depending on transport type.
On Thu, Jan 26, 2012 at 07:53:06AM -0600, Pieter Hintjens wrote: > On Thu, Jan 26, 2012 at 2:17 AM, Ian Barber <ian.bar...@gmail.com> wrote:
> > Is it possible that this API should simply be on a different calls, a > > zmq_bind_assign or similar, to avoid this clash? This could then work with > > all transports:
> > tcp/pgm: bind to random port > > ipc: bind to random free name in system tmp > > inproc: bind to random free name
> That would work, but so would a less surprising zmq_bind plus a > getsockopt with return value depending on transport type.
Forgive me if I'm being dense, but couldn't the N case work if you made sure to do the getsockopt() call after each zmq_bind()?
---------------------------------------------------------------------- The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt. _______________________________________________ zeromq-dev mailing list zeromq-...@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> On Thu, Jan 26, 2012 at 07:53:06AM -0600, Pieter Hintjens wrote: > > On Thu, Jan 26, 2012 at 2:17 AM, Ian Barber <ian.bar...@gmail.com> wrote:
> > > Is it possible that this API should simply be on a different calls, a > > > zmq_bind_assign or similar, to avoid this clash? This could then work with > > > all transports:
> > > tcp/pgm: bind to random port > > > ipc: bind to random free name in system tmp > > > inproc: bind to random free name
> > That would work, but so would a less surprising zmq_bind plus a > > getsockopt with return value depending on transport type.
> Forgive me if I'm being dense, but couldn't the N case work if you made > sure to do the getsockopt() call after each zmq_bind()?
Yup, that would work. Sorry for missing the point about zmq_bind() being synchronous.
Ian also made a great point that this can be extended to "bind to unnamed endpoint" (for want of a better name) for all transports. So, what I think we want is that the getsockopt API should return a string (so as to be usable for multiple transports, plus returning a different-data-type-per-transport is a PITA).
It'd also be nice to define a consistent way to specify this "unnamed endpoint" for all transports that might want to provide such functionality. ":0" happens to work for the tcp:// case, but does not really make sense for inproc:// or ipc://.
This leaves us with something like this proposal:
zmq_bind(foo, "tcp://XXXX:*"); // "tcp://*:*" if you want INADDR_ANY char endpoint [ZMQ_ENDPOINT_MAX]; zmq_getsockopt(foo, ZMQ_GET_ENDPOINT, endpoint, sizeof endpoint);
inproc:// semantics would be identical to ipc:// (w/o the /tmp/ prefix obviously).
The use of "*" seems fairly uncontroversial -- note that this means an ipc:// endpoint cannot therefore contain "*" which is an (albeit niche) backward-incompatible change.
Thoughts? I'll ping Martin Sustrik tomorrow to see if he thinks there's any reason why this wouldn't work; I believe he's ignoring this thread as TL;DR :-)
On Thu, Jan 26, 2012 at 03:57:37PM +0100, Martin Lucina wrote: > aj.le...@quantum.com said: > > On Thu, Jan 26, 2012 at 07:53:06AM -0600, Pieter Hintjens wrote: > > > On Thu, Jan 26, 2012 at 2:17 AM, Ian Barber <ian.bar...@gmail.com> > > > wrote: > > Forgive me if I'm being dense, but couldn't the N case work if you > > made sure to do the getsockopt() call after each zmq_bind()?
> Yup, that would work. Sorry for missing the point about zmq_bind() > being synchronous.
> Ian also made a great point that this can be extended to "bind to > unnamed endpoint" (for want of a better name) for all transports. So, > what I think we want is that the getsockopt API should return a string > (so as to be usable for multiple transports, plus returning a > different-data-type-per-transport is a PITA).
> It'd also be nice to define a consistent way to specify this "unnamed > endpoint" for all transports that might want to provide such > functionality. ":0" happens to work for the tcp:// case, but does not > really make sense for inproc:// or ipc://.
> This leaves us with something like this proposal:
> zmq_bind(foo, "tcp://XXXX:*"); // "tcp://*:*" if you want > INADDR_ANY char endpoint [ZMQ_ENDPOINT_MAX]; zmq_getsockopt(foo, > ZMQ_GET_ENDPOINT, endpoint, sizeof endpoint);
> => endpoint is filled as "ipc:///tmp/Xyz358hfA7".
> inproc:// semantics would be identical to ipc:// (w/o the /tmp/ prefix > obviously).
> The use of "*" seems fairly uncontroversial -- note that this means an > ipc:// endpoint cannot therefore contain "*" which is an (albeit > niche) backward-incompatible change.
I think the "*" is fine, and it makes sense to me to extend it to all endpoint types. I'm a bit concerned about having the full ${transport}://${path/port} string passed back. That means that the app needs to parse that string out to get what it needs, when what it really wants is just the ${path/port} depending on the transport. Is there a reason to include more than the part of the transport string that was wildcarded?
> Thoughts? I'll ping Martin Sustrik tomorrow to see if he thinks > there's any reason why this wouldn't work; I believe he's ignoring > this thread as TL;DR :-)
Thanks, -- AJ Lewis Software Engineer Quantum Corporation
Work: 651 688-4346
---------------------------------------------------------------------- The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt. _______________________________________________ zeromq-dev mailing list zeromq-...@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> => endpoint is filled as "ipc:///tmp/Xyz358hfA7".
>> inproc:// semantics would be identical to ipc:// (w/o the /tmp/ prefix >> obviously).
>> The use of "*" seems fairly uncontroversial -- note that this means an >> ipc:// endpoint cannot therefore contain "*" which is an (albeit >> niche) backward-incompatible change.
> I think the "*" is fine, and it makes sense to me to extend it to all > endpoint types. I'm a bit concerned about having the full > ${transport}://${path/port} string passed back. That means that the app > needs to parse that string out to get what it needs, when what it really > wants is just the ${path/port} depending on the transport. Is there a > reason to include more than the part of the transport string that was > wildcarded?
The immediate use case I can think of is to make an ad hoc channel for communicating with a particular client and communicating the new endpoint to the client via an already established channel. While that my code (and possibly the client) already "knows" the transport and the hostname and just needs to know the port, what it really wants to communicate is the endpoint that the client should connect to. If I just get the port number, then I still need to reconstruct the whole endpoint string in order to connect to the new bound socket. I think this is universal. Every time you bind a socket, you want something connect to it, and that something needs the full endpoint address in order to connect.
-- Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thu, Jan 26, 2012 at 8:34 AM, AJ Lewis <aj.le...@quantum.com> wrote: > Forgive me if I'm being dense, but couldn't the N case work if you made > sure to do the getsockopt() call after each zmq_bind()?
You're not being dense and this is exactly what I meant by "bind is synchronous", unlike a connect. The code example makes it easier to understand. Thanks!
On Thu, Jan 26, 2012 at 8:57 AM, Martin Lucina <mar...@lucina.net> wrote: > zmq_bind(foo, "tcp://XXXX:*"); // "tcp://*:*" if you want INADDR_ANY > zmq_bind(foo, "ipc://*"); > The use of "*" seems fairly uncontroversial -- note that this means an > ipc:// endpoint cannot therefore contain "*" which is an (albeit niche) > backward-incompatible change.
This is also what we did for CZMQ. Consistent and unsurprising.
On Thu, Jan 26, 2012 at 03:51:17PM +0000, Robert Kern wrote: > On 1/26/12 3:07 PM, AJ Lewis wrote: > > On Thu, Jan 26, 2012 at 03:57:37PM +0100, Martin Lucina wrote:
> >> This leaves us with something like this proposal:
> >> => endpoint is filled as "ipc:///tmp/Xyz358hfA7".
> >> inproc:// semantics would be identical to ipc:// (w/o the /tmp/ > >> prefix obviously).
> >> The use of "*" seems fairly uncontroversial -- note that this means > >> an ipc:// endpoint cannot therefore contain "*" which is an (albeit > >> niche) backward-incompatible change.
> > I think the "*" is fine, and it makes sense to me to extend it to > > all endpoint types. I'm a bit concerned about having the full > > ${transport}://${path/port} string passed back. That means that the > > app needs to parse that string out to get what it needs, when what > > it really wants is just the ${path/port} depending on the transport. > > Is there a reason to include more than the part of the transport > > string that was wildcarded?
> The immediate use case I can think of is to make an ad hoc channel for > communicating with a particular client and communicating the new > endpoint to the client via an already established channel. While that > my code (and possibly the client) already "knows" the transport and > the hostname and just needs to know the port, what it really wants to > communicate is the endpoint that the client should connect to. If I > just get the port number, then I still need to reconstruct the whole > endpoint string in order to connect to the new bound socket. I think > this is universal. Every time you bind a socket, you want something > connect to it, and that something needs the full endpoint address in > order to connect.
Hrm...what I was thinking of was the initial establishment of a connection. For TCP for example, this could be via a portmapper or zeroconf, in which case all you care about is the port number. I could see the full string being useful in the pre-established connection case, or for inproc, where the binding thread creates the worker threads and simply passes the inproc string on.
I guess having that be the standard and either having a helper function in zmq proper, or in the language bindings, to do the parsing out of the wildcard part probably makes the most sense.
Regards, -- AJ Lewis Software Engineer Quantum Corporation
Work: 651 688-4346
---------------------------------------------------------------------- The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt. _______________________________________________ zeromq-dev mailing list zeromq-...@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
On Thu, Jan 26, 2012 at 8:57 AM, Martin Lucina <mar...@lucina.net> wrote: > This leaves us with something like this proposal: > => endpoint is filled as "tcp://XXXX:12345".
Missed this. It won't work. Bind interfaces are not the same as connect addresses. The caller needs to know the port number, it can then reconstruct a full address and broadcast that to peers, e.g. via a central broker or out-of-band UDP (we do this in several applications using CZMQ). Returning the bound interface + port just creates extra parsing work for applications.
Further, you need to support "*" in the bind address, just treat ":*" at the end as special. Otherwise you break the very common and necessary use case of "bind to all interfaces".
> > > I think the "*" is fine, and it makes sense to me to extend it to > > > all endpoint types. I'm a bit concerned about having the full > > > ${transport}://${path/port} string passed back. That means that the > > > app needs to parse that string out to get what it needs, when what > > > it really wants is just the ${path/port} depending on the transport. > > > Is there a reason to include more than the part of the transport > > > string that was wildcarded?
> > The immediate use case I can think of is to make an ad hoc channel for > > communicating with a particular client and communicating the new > > endpoint to the client via an already established channel. While that > > my code (and possibly the client) already "knows" the transport and > > the hostname and just needs to know the port, what it really wants to > > communicate is the endpoint that the client should connect to. If I > > just get the port number, then I still need to reconstruct the whole > > endpoint string in order to connect to the new bound socket. I think > > this is universal. Every time you bind a socket, you want something > > connect to it, and that something needs the full endpoint address in > > order to connect.
> Hrm...what I was thinking of was the initial establishment of a connection. > For TCP for example, this could be via a portmapper or zeroconf, in which > case all you care about is the port number. I could see the full string being > useful in the pre-established connection case, or for inproc, where the > binding thread creates the worker threads and simply passes the inproc > string on.
> I guess having that be the standard and either having a helper function in > zmq proper, or in the language bindings, to do the parsing out of the wildcard > part probably makes the most sense.
Perhaps you can have a getsockopt that returns just the "port" bound, and another that returns the full endpoint.
