[zeromq-dev] Seldom publisher is not listened to for long ?
Martin Raspaud
Hash: SHA1
Hi all,
I have a very simple pub/sub scheme (1 pub, 1 sub) talking over an IPSec
tunnel.
This seems to work great when publishing messages regularly. However, a
typical use case for us would be to publish one message every two or
three hours. In those cases we have noticed that the subscriber after a
while is just not receiving any messages anymore (no obvious error
message or so).
We fixed the problem by implementing a heartbeat, but I'm a bit
sceptical. ZeroMQ should know better, right ? Or maybe is this not a zmq
problem, maybe it's related to an underlying network layer ? Does anyone
have an idea on this ?
I've been searching around the list and the guide, but I did not seem to
find anything related to my problem here... Please shout if I missed it :)
Best regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJN0MMRAAoJEBdvyODiyJI4bXcH/3/lbsFyYY1qJ5BQgAp0F4iK
AgQsazItDnGGuOA6MZsBr5eK6B48TVlpt5tpYH8DEYEVxoAnpC9S2Ro7e8KpMxzA
CrDegSa/6S/Jyw3AhY/Mv/LG0iV4H7093ImwZcKvOcU8QJB1DCy6Gc+2aQVeH6wi
sqXH0QzwbsZVlWhlWAUbVcIXhmX0aDQSxRfAdcRndGI1vtE+7bvFubzV95EmM7q1
Og17mquNpukm8HV7SnEapAnCrNMOVDewktdFSLRUCGFpSHXCbaJ9dFAyNpRjx9vF
0aCsb4gN9M/qoKW7+VjIzhTvVnm4c82lq8dl4Gq6GSmDV0aovfXYX0a7zmN5MTs=
=imxG
-----END PGP SIGNATURE-----
Martin Sustrik
> This seems to work great when publishing messages regularly. However, a
> typical use case for us would be to publish one message every two or
> three hours. In those cases we have noticed that the subscriber after a
> while is just not receiving any messages anymore (no obvious error
> message or so).
>
> We fixed the problem by implementing a heartbeat, but I'm a bit
> sceptical. ZeroMQ should know better, right ? Or maybe is this not a zmq
> problem, maybe it's related to an underlying network layer ? Does anyone
> have an idea on this ?
Hm. Looks like a bug. Maybe some misinteraction between 0mq and TCP
keepalives?
So the use case is to have a single pub/sub pair connected over TCP,
publish message every 3 hours, right?
Martin
_______________________________________________
zeromq-dev mailing list
zerom...@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Martin Raspaud
Hash: SHA1
On 16/05/11 08:38, Martin Sustrik wrote:
> Hm. Looks like a bug. Maybe some misinteraction between 0mq and TCP
> keepalives?
>
> So the use case is to have a single pub/sub pair connected over TCP,
> publish message every 3 hours, right?
Yes, that sounds about right.
regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJN0MtQAAoJEBdvyODiyJI4AIkIANG9G72WIpFMBMQZs53tB9M/
udgQKITIa7mwn0gN0PXDF8157780cLhquKH0jZQU68hWFsNPbYCEk9dkXGL1ZRlM
9m7mA2PvcP95WofvdABkCqAh9w1e1KsE5nMHXOFN290ojz8Xj2UqapYunkXbPaw4
Sf//2+1Ol+QpJksfcOno6AKMJWCbJ2kfxWjPZR095hb5c47LcyDjVkjHwNQ/0xfa
TOXcD0HW4JJyLRZQXlmuwHyvAmkOLoviHtb8mb/HrwJruyW0c/ZQfZb/U2oU/BQj
f8+6CZXoNnhudnqhneLNhf4ANkB8gFCSFTugrVA7dRFbZqRjYv0xXojrS/pbtn0=
=nVoq
-----END PGP SIGNATURE-----
Pieter Hintjens
> I have a very simple pub/sub scheme (1 pub, 1 sub) talking over an IPSec
> tunnel.
>
> This seems to work great when publishing messages regularly. However, a
> typical use case for us would be to publish one message every two or
> three hours. In those cases we have noticed that the subscriber after a
> while is just not receiving any messages anymore (no obvious error
> message or so).
>
> We fixed the problem by implementing a heartbeat, but I'm a bit
> sceptical. ZeroMQ should know better, right ? Or maybe is this not a zmq
> problem, maybe it's related to an underlying network layer ? Does anyone
> have an idea on this ?
It's possible the tunnel itself has a timeout for inactivity, and you
can presumably raise this. Sending a keep-alive heartbeat is an OK
workaround but obviously not as good as understanding what's actually
going on.
-Pieter
Michael Holm
> I have a very simple pub/sub scheme (1 pub, 1 sub) talking over an IPSec
> tunnel.
>
> This seems to work great when publishing messages regularly. However, a
> typical use case for us would be to publish one message every two or
> three hours. In those cases we have noticed that the subscriber after a
> while is just not receiving any messages anymore (no obvious error
> message or so).
>
> We fixed the problem by implementing a heartbeat, but I'm a bit
> sceptical. ZeroMQ should know better, right ? Or maybe is this not a zmq
> problem, maybe it's related to an underlying network layer ? Does anyone
> have an idea on this ?
It's possible the tunnel itself has a timeout for inactivity, and you
can presumably raise this. Sending a keep-alive heartbeat is an OK
workaround but obviously not as good as understanding what's actually
going on.
-Pieter
IPSec tunnels typically disconnect after a time period and then will automatically reconnect as soon as someone tries to access a vlan routed over that connection. What is weird is that if you have an established socket connection I would think that is enough to keep the tunnel alive. There might be an option set on the tunnel that disconnects after n minutes if no traffic has been sent even if there is an existing connection. You need to chat with the team responsible for setting up the tunnel.
Michael
Tibra Trading Europe Limited is Authorised and Regulated by the FSA (Company No. 06061713)
Tibra Global Services Pty Limited London Branch (Company No. BR014370)
Tibra Equities Europe Limited (Company No. 07283912)
Registered in England & Wales - Level 11, 5 Aldermanbury Square London EC2V 7HR
The contents of this email including any attachments are confidential. If you have received this email in error, please advise the sender by return email and delete this email. Any unauthorised use of the contents of the email is prohibited and you must not disseminate, copy or distribute the message or use the information contained in the email or its attachments in any way.
The views or opinions expressed are the author's own and may not reflect the views or opinions of Tibra. Tibra does not guarantee the integrity of any emails or attached files. E-mails may be interfered with, may contain computer viruses or other defects. Under no circumstances do we accept liability for any loss or damage which may result from your receipt of this message or any attachments.
Martin Raspaud
Hash: SHA1
On 16/05/11 10:57, Michael Holm wrote:
> 2011/5/16 Martin Raspaud <martin....@smhi.se>:
>
>> I have a very simple pub/sub scheme (1 pub, 1 sub) talking over an
>> IPSec tunnel.
>
> It's possible the tunnel itself has a timeout for inactivity, and
> you can presumably raise this. Sending a keep-alive heartbeat is an
> OK workaround but obviously not as good as understanding what's
> actually going on.
>
> -Pieter
>
> IPSec tunnels typically disconnect after a time period and then will
> automatically reconnect as soon as someone tries to access a vlan
> routed over that connection. What is weird is that if you have an
> established socket connection I would think that is enough to keep
> the tunnel alive. There might be an option set on the tunnel that
> disconnects after n minutes if no traffic has been sent even if there
> is an existing connection. You need to chat with the team responsible
> for setting up the tunnel.
Hi Michael,
I'll do that, thanks for the input !
Regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJN0OokAAoJEBdvyODiyJI4A1gH/1pqPLCzatBGXkkcORggLSS2
4rDyNMzJk1dWpwJP+DShM7h1ieYOGro511aZ9/ayEjKUrgXPnxiwtGK000Gj9N9/
Q5SYcp4CFcx4YAQkwGBd0lcjR9CZgTBquJ7+e4+zaEXT9XDZ74HxJw1F5r7sN6Ny
xd8YKqnEY4djatEFrpIIamzVel/ZdMZzhbnPWnx371jvoXrnUqYObEgJRYTcc5pL
00u4f7meD91RNwvToayB4KXiDIRzYlPiMUcXdUlcBuTxafq06nbL72fLehCzgCpS
356I461u/F9M99A08p1LI9M/I1njjiV89BOp8cFVzk7x6RXetfUZ6cGzNBYrDag=
=o1T/
-----END PGP SIGNATURE-----
Martin Raspaud
Hash: SHA1
On 16/05/11 08:38, Martin Sustrik wrote:
>
> Hm. Looks like a bug. Maybe some misinteraction between 0mq and TCP
> keepalives?
Hi again,
I investigated a bit more and talked to the IT guys keeping up our IPSec
tunnel here.
It seems the idle timeout for the tunnel is 30 minutes. What about the
TCP keepalives ? I read somewhere in this list they are > 2 hours, that
could be problematic, right ?
Best regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJN0Q49AAoJEBdvyODiyJI4ZSAH/2WxPVd2IkVnfKxh7DYxwETe
AQEv0k3bjGQkTt6gAcU6spvNwip8evWNbtc/rU1TNhB7/dmzETOGuvYkYnUTgfTT
B21q8Sg17QvVl82xL2uL/VG8KUG05aSsR7//aS6e6AF0JLukHzHhVFfpY5wTtWt8
8LrvPxwmO2IUT3CCdf7sQYduNoZZxMVENElOZj9pg58+5E2JsMeCYbCEDZ/ujN/Z
v3u+qVNI9f3LW78yy3BwRANQi6oAHtluBFArxb4+aAW/AC1vUXUWHFAgYFmbjYSP
vY/gKI56D5FwHR+4Rz6FRsucLxMWVY+kg0FYYeMDNXF25NdmjQEM90uG/o3/sj0=
=UjjX
-----END PGP SIGNATURE-----
Martin Sustrik
> It seems the idle timeout for the tunnel is 30 minutes. What about the
> TCP keepalives ? I read somewhere in this list they are> 2 hours, that
> could be problematic, right ?
TCP spec states that keepalives should be send at least after 2 hours.
The actual time would depend on the TCP stack you are using.
Martin Raspaud
Hash: SHA1
On 16/05/11 13:50, Martin Sustrik wrote:
> On 05/16/2011 01:45 PM, Martin Raspaud wrote:
>
>> It seems the idle timeout for the tunnel is 30 minutes. What about the
>> TCP keepalives ? I read somewhere in this list they are> 2 hours, that
>> could be problematic, right ?
>
> TCP spec states that keepalives should be send at least after 2 hours.
> The actual time would depend on the TCP stack you are using.
Yes indeed. On my system it's two hours also. I see there low-level
socket options to change this. Is there a way to do this in zeromq
(access the low-level socket and add an option) ? Otherwise I could
maybe change this system-wide...
Best regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJN0R9DAAoJEBdvyODiyJI4ms0H/1Axtl1vzNZI7cwEmBlhCk0l
+O19Na0DHuwBN2x5you6uThybsNKKAtzIWw2Aht2cdlbe/aDUjI/3mqeQ4HVahrj
G1CA303ed8A4JAiKfZvZPC4/4y9R1imJIg+Q2m0hwaCX3ZCeQZ2We3Dd5iL6qM+w
NUyq24lOeIvqw55w1bOZ7GwGNQeRO4I//WL8ZKUctMgBRXj5KX9kWec7OrXGlvuP
paKmKg7uJwUrJmC0vfwHD639EGUi29li+041kmCtIqAhAkcfw3W4Xqe/Xhyovok+
6dLVqKCYCb6SSyrPVzFOOeYZA6+Y4VaJ3ShbVJHncW16JDnrnhQN4maF8SY30Nc=
=oPjE
-----END PGP SIGNATURE-----
Martin Sustrik
> Yes indeed. On my system it's two hours also. I see there low-level
> socket options to change this. Is there a way to do this in zeromq
> (access the low-level socket and add an option) ? Otherwise I could
> maybe change this system-wide...
It would be good to do so for diagnostics: Disable TCP keepalives and
check whether the problem persists. If it does the problem isn't caused
by keepalives.