Server stops responding

[Gili]

Hi,

I am running Version Citrix-3.2.2.4 'Marshal West' under Windows 7, 64-bit, under Cygwin. Chrome 32.0.1700.107 m on the client end.

I've noticed that after a few hours/days the server is up, clients are no longer able to get back ICE candidates from the server. The service itself seems to be up, but the log file is empty (nothing is getting logged, no indication that anyone is even trying to connect) using the default logging configuration. I was not able to figure out whether this is a client-side or server-side problem because I don't know how to investigate it on the client-side... but I *did* notice that if I restart the service, everything works just fine again.

This has happened to me multiple times already. Every time I restart the service, everything begins working just fine. As a result, I suspect this is a server-side bug or misconfiguration.

How do I go about tracking this down? Should I increase the logging level? If so, what should I look for?

 

Thank you,

Gili

[Oleg Moskalenko]

Hi Gili

thanks for notifying us about the problem. We are actually have been aware of the Cygwin problem. After a period of (heavy) usage, it stops responding. We do know how to reproduce the problem in Cygwin. Our investigation has shown that this is something Cygwin-related... but we do not know the exact reason for the  problem. Cygwin has a unique socket functionality that is not exactly a UNIX functionality, but it is not a Windows behavior, that's a unique flavor that is hard to adjust application to.

Cygwin platform is not a reliable platform for a network server. We would recommend using it only for R&D, not for a real production environment. For production, any "real" UNIX platform is recommended - especially, as of now, the CentOS 6.5, ArchLinux or FreeBSD 9.x.

If you absolutely must use a desktop platform, can you switch to Mac OS X ? That's pretty much bulletproof, the TURN server runs really well in Mac OS X. That's one of the project development platforms so it is very well tested. A Linux desktop Fedora 20 would be a good choice, too.

We will continue researching why the Cygwin version stops working after a while.

Regards,
Oleg

[cowwoc]

Hi Oleg,

I'm glad to know I'm not the only one seeing this :)

I've already got a theory for what's causing this problem. I think my Windows service was misconfigured. I neglected to define the "Tcpip" and "Afd" as dependencies, which means that the service could have gotten launched before these were available. I made a similar mistake in the past when configuring Tomcat as a service and it also failed on a seemingly random basis (depending on whether it happens to launch before the dependencies or not). So one theory is that the server didn't actually stop running after hours/days but rather it never worked in the first place after a particular reboot but I didn't notice until much later. I've posted http://stackoverflow.com/q/21715865/14731 in the hopes of helping others avoid this mistake.

If you don't hear back from me in the next month, it means this fixed the problem for me. Otherwise, I think we discuss how to debug this problem further. What, if anything, can we do to debug the server once it gets into a "bad state"? Is there any way for us to investigate what Chrome sees from its side? Currently, Chrome doesn't throw an error if it decides to skip a TURN server so we don't know what, if anything, it sends and receives from the server.

Final point: I've attached two log files for your review. Here is what I noticed...

I have a log file from February 7th (attached). Then the next 4 days I have one log file per day which is completely empty. Then I rebooted the computer and the log file from February 11th was generated. The 4 days of empty logs is suspicious because I suspect I tried using TURN in those days (I did LAN testing so I wouldn't have noticed if TURN didn't work, but Chrome should have been hitting the server in any case).

When I compared these files, one thing I noticed is that when the server was in a bad state, the turn server instances were being initialized in a serial manner:

0: turn server id=0 created
0: turn server id=1 created
0: turn server id=2 created
...

whereas when the server was in a good state we see initialization happening out of order:

0: turn server id=1 created
0: turn server id=0 created
0: turn server id=2 created
...

I don't know whether this is meaningful or not but I thought I'd bring it to your attention.

Thanks,
Gili

--
You received this message because you are subscribed to a topic in the Google Groups "TURN Server project rfc5766-turn-server" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/turn-server-project-rfc5766-turn-server/NaJVSWnD7CI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to turn-server-project-rfc57...@googlegroups.com.
To post to this group, send email to turn-server-project...@googlegroups.com.
Visit this group at http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

[Oleg Moskalenko]

Hi Gili

looking at your traces, I suspect that you are starting the turnserver without -L option - so it uses all IPs that is able to find. That may be a problem for you. Either define -L <192....> address as the command-line option, or upgrade to version 3.2.2.6 that uses a smarter listener/relay policy.

I am not sure whether the service configuration is related to the problem.

In my case, I suspect, it was exhausting the open sockets limit and hanging after that - but I could not drill down.

I checked the logs, both logs look fine.

I'll let you know if I'll find anything.

Regards,
Oleg

To unsubscribe from this group and all its topics, send an email to turn-server-project-rfc5766-turn-server+unsubscribe@googlegroups.com.
To post to this group, send email to turn-server-project-rfc5766-turn-...@googlegroups.com.

[cowwoc]

Hi Oleg,

It should be fairly easy to investigate whether the problem is caused by the open sockets limit.

Simply do something similar to http://blogs.technet.com/b/yongrhee/archive/2011/12/19/how-to-troubleshoot-a-handle-leak.aspx

I would do the following:

• Create a tool that opens sockets in a loop (equivalent of TestLimit64) to find out the maximum number of sockets.
  
• Have your tool eat up (MAX - 20) sockets.
• Launch your server and fire automated tests against it (make sure to use less than 20 concurrent requests).

If something is really wrong then someone (server or cygwin) will leak sockets and eventually hit the max. If not, you should be able to run this test for many hours without a problem.

Another test is to force the server to run out of sockets (start with MAX-20, then eat up the remaining sockets while the server is running) in order to ensure that the server handles such failures properly.

Give it a go and let me know if it helps.

Gili

To unsubscribe from this group and all its topics, send an email to turn-server-project-rfc57...@googlegroups.com.
To post to this group, send email to turn-server-project...@googlegroups.com.

[Oleg Moskalenko]

Gili, yes, I am going to do something like that. In addition to system-wide and per-user socket limits, there may be also per-process socket limit (then the technique described will not work).

We tested that condition multiple times on UNIX systems, this is a part of our standard test suite, so how the TURN server handles this situation we know pretty well. I am rather concerned how the Cygwin handles the situation.

I'll let you know the outcome of my investigation.

Regards,
Oleg

--
You received this message because you are subscribed to the Google Groups "TURN Server project rfc5766-turn-server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to turn-server-project-rfc57...@googlegroups.com.

[Oleg Moskalenko]

OK, I found why it is stopping in my tests.

I have too many Cygwin processes running when it happens - I guess hundreds of them, some are TURN-related, some are not. All of them are getting stuck and jammed by something. All are unresponsive and I have to kill them from the task manager forcefully. That looks more like a general Cygwin environment problem.

I suppose that you are experiencing something else.

When that happens again, please open the TaskManager and check what you see there. Also, try to connect to the turnserver process thru telnet (or putty), address 127.0.0.1:5766. Is it responding to the telnet ? If you can connect, then run ps and pc commands and post the output here.

Regards,
Oleg

To post to this group, send email to turn-server-project-rfc5766-turn-s...@googlegroups.com.

Visit this group at http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "TURN Server project rfc5766-turn-server" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/turn-server-project-rfc5766-turn-server/NaJVSWnD7CI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to turn-server-project-rfc5766-turn-server+unsubscribe@googlegroups.com.
To post to this group, send email to turn-server-project-rfc5766-turn-...@googlegroups.com.
Visit this group at http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

--

You received this message because you are subscribed to the Google Groups "TURN Server project rfc5766-turn-server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to turn-server-project-rfc5766-turn-server+unsubscribe@googlegroups.com.

[cowwoc]

Hi Oleg,

1. Adding the service dependencies did not help me. I still ran into the problem.
2. I ran into it immediately after rebooting my computer, which means that this issue cannot be related to the maximum file or socket handles (since usage is low after reboot).
   
3. Unfortunately, I only read your email now so I didn't run the test in time. I restarted turnserver and it started working as usual.

I am familiar with "ps" but what is the "pc command" you referred to?

I'm going to run a new test, where I either restart the service multiple times and retest or reboot the entire computer multiple times. Hopefully I'll figure out how to reproduce it.

PS: I had to reboot my machine due to Windows updates. Hopefully the problem is not related to that (booting after installing updates) because if so this will be virtually impossible to reproduce :)

Gili

[Oleg Moskalenko]

Gili, ps and pc commands are commands in the telnet monitor to the turnserver. Those are not shell commands. The "ps" shows sessions, the "pc" shows the general configuration state.

Let me know the results.

Regards,
Oleg

[Oleg Moskalenko]

Also, when you recompile the sources after upgrade to a new version, do not forget to run "make clean" before that.

Oleg

[cowwoc]

Okay, here is what I see: The problem occurs right after a reboot. Restarting the service past that point always fixes the problem so I'm guessing there is a synchronization/dependency problem if you launch the service too early.

While the server is in a "bad state":

It responds to telnet.
"ps" returns:

Total sessions: 0

"pc" returns:

  verbose: ON
  daemon process: OFF
  stale-nonce: OFF (*)
  stun-only: OFF (*)
  no-stun: OFF (*)
  secure-stun: OFF (*)
  do-not-use-config-file: OFF
  RFC5780 support: ON
  net engine version: 2
  net engine: UDP thread per network endpoint
  enforce fingerprints: OFF
  mobility: OFF (*)
  udp-self-balance: OFF
  pidfile: /var/run/turnserver.pid
  process user ID: 18
  process group ID: 544
  process dir: /cygdrive/c/Windows/system32

  cipher-list: DEFAULT
  ec-curve-name: prime256v1
  DH-key-length: 1066
  Certificate Authority file: empty
  Certificate file: turn_server_cert.pem
  Private Key file: turn_server_pkey.pem
  allowed SHA types: SHA1 and SHA256

  Listener addr: 192.168.232.1
  Listener addr: 192.168.253.1
  Listener addr: ::1
  Listener addr: 127.0.0.1
  no-udp: OFF
  no-tcp: OFF
  no-dtls: ON
  no-tls: ON
  SSLv2: OFF
  SSLv3: OFF
  TLSv1.0: OFF
  TLSv1.1: OFF
  TLSv1.2: OFF
  listener-port: 3478
  tls-listener-port: 5349
  alt-listener-port: 0
  alt-tls-listener-port: 0


  Relay addr: 192.168.232.1
  Relay addr: 192.168.253.1
  Relay addr: ::1
  server-relay: OFF
  no-udp-relay: OFF (*)
  no-tcp-relay: OFF (*)
  min-port: 49152
  max-port: 65535
  no-multicast-peers: OFF (*)
  no-loopback-peers: OFF (*)

  DB type: file
  DB: turnuserdb.conf

  Long-term authorization mechanism: ON
  Short-term authorization mechanism: OFF
  Anonymous credentials: OFF
  REST API: OFF
  Realm: vtlr

  total-quota: 0 (**)
  user-quota: 0 (**)
  total-current-allocs: 0
  max-bps: 0

  cli-max-output-sessions: 256 (**)

  (Note 1: parameters with (*) are toggleable)

  (Note 2: parameters with (**) are changeable)

"ipconfig" returns:

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::b0c5:3fbf:7ea2:bffe%10
   IPv4 Address. . . . . . . . . . . : 192.168.1.100
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1

Ethernet adapter VMware Network Adapter VMnet1:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::2d6d:64d5:52a:4f45%17
   IPv4 Address. . . . . . . . . . . : 192.168.232.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

Ethernet adapter VMware Network Adapter VMnet8:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::cdd7:c5b2:d74c:78c4%18
   IPv4 Address. . . . . . . . . . . : 192.168.253.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

Tunnel adapter isatap.{C24FB618-5B59-4665-AD57-0E9AB00D2835}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Tunnel adapter isatap.{3832925B-7883-4FE5-ACC9-826DA7E50A59}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Tunnel adapter isatap.{4853B252-C7FE-4E59-AB35-C222198725AC}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Does this give you any ideas? What else can I test to see what is going on?

I'm going to send you my verbose logs privately (since they contain sensitive information).

Thanks,
Gili

To unsubscribe from this group and all its topics, send an email to turn-server-project-rfc57...@googlegroups.com.
To post to this group, send email to turn-server-project...@googlegroups.com.

Visit this group at http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "TURN Server project rfc5766-turn-server" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/turn-server-project-rfc5766-turn-server/NaJVSWnD7CI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to turn-server-project-rfc57...@googlegroups.com.
To post to this group, send email to turn-server-project...@googlegroups.com.

Visit this group at http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "TURN Server project rfc5766-turn-server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to turn-server-project-rfc57...@googlegroups.com.
To post to this group, send email to turn-server-project...@googlegroups.com.

[cowwoc]

I think I found the problem.

I re-ran "pc" when the server was in a good state and noticed one important difference:

[...]
  Listener addr: 192.168.1.100
[...]
  Relay addr: 192.168.1.100

This is my computer's local IP. Notice that it is missing when the server is in a bad state. This means that incoming requests pass through my firewall, to 192.168.1.100 but the server isn't listening on that IP. Why would "turnserver" not see this IP on bootup?

Thanks,
Gili

[cowwoc]

I'm wondering how other services (like MySQL) do this. Its Windows Service doesn't declare any dependency, it launches on startup, and yet it is able to listen on all devices. I'm thinking there is a way to internally wait for the LAN to come up before scanning for available interfaces.

Gili

[Oleg Moskalenko]

Gili, that's great that we are moving with the investigation.

I was already saying that you have to set the listener and the relay addresses EXPLICITLY, and do not rely on the automatic discovery. Use -L and -E command-line options.

I do not think that that's application responsibility to wait till the TCP/IP stack is configured. I believe that's the configuration manager responsibility. I am not very familiar with MS Windows; but in Unix, that would be rather problematic to start an application service BEFORE the network system is configured; if you start a service in a script in the usual multi-user level 3,  then you can pretty much be sure that the network service is already ready. I suppose that MS Windows must have some tools to ensure that the services are getting started in the proper order, too. Honestly, that's a system-level functionality, and I do not want to put it into the TURN server if it can be done by the system utilities, in a better and cleaner way.

Please try to find how Windows handles the services ordering... and let us know.

Thanks
Oleg

[cowwoc]

Hi Oleg,

First, a correction: the MySQL service was a bad example because it turns out I only have it configure to use named pipes (it's not accessible over TCP/IP).

The problem, as I understand it, isn't so much that the networking stack is not available (it is) but rather that the computer hasn't connected acquired an IP address from the network yet. This is equivalent to booting up the computer, waiting for all services to launch, pulling the LAN cable, then restart turnserver. It won't find 192.168.1.100 because at the moment it checks it is not available.

So, that being said I added:

listening-ip=192.168.1.100
relay-ip=192.168.1.100

to the configuration. Rebooted. And the problem stills occurs. I ran a quick test and discovered that according to http://www.yougetsignal.com/tools/open-ports/ the port is closed when the problem occurs, and open when I restart the server.

In other words, turnserver is attempting to bind to 192.168.1.100 before the computer acquires that IP address and as a result the binding fails.

I suspect, but cannot prove, that you should be getting back an error when binding to an unavailable ip/port but that error is going unnoticed. Where should I be looking in the source-code?

Ideally the server should go into a retry loop for some configurable time and eventually the address will become available.

Gili

[cowwoc]

Ha! I was right.

Looking at the log file, I now see:

0: Trying to bind fd 44 to <192.168.1.100:3478>: errno=125
0: Cannot bind UDP/DTLS listener socket to addr 192.168.1.100:3478
0: Trying to bind fd 45 to <192.168.1.100:3478>: errno=125
0: Cannot bind TCP/TLS listener socket to addr 192.168.1.100:3478

So:

1. The server shouldn't keep on running after such a failure. It should treat this as a fatal error and shut down.
2. The server should go into a retry loop as I mentioned.

I am surprised you are getting errno 125 though because that corresponds to "port in use" when in fact I believe the port is unavailable for other reasons.

Gili

[Oleg Moskalenko]

125 is reported by the system, so that's what I have.

The problem is that the TURN server is supposed to be used on a system with static IPs. Running it on a DHCP-provisioned system is rather a hack than a normal situation...

I'll think what can be done.

Oleg

[cowwoc]

Oleg,

I suggest making the following changes:

1. Issue a warning on startup if the user doesn't bind to a specific IP address. You can give the example where the server launches before the IP address becomes available, hence it never gets bound to.
2. Use a retry-loop to bind to the specified address/ports and if any of them fail, it should treat it as a fatal error.

Implementing this will not break anything for static IP users, but it will help those of us using DHCP.

I also think I have a solution for #1 (that will guarantee the IP is available). More on that in a bit.

Gili

[Oleg Moskalenko]

OK, collect all your proposals and let me know. I ll combine them with an anycast fix and i ll provide a comprehensive solution.

Sent from my iPhone

[cowwoc]

Okay. I've run out of options (none of them worked).

My proposal remains:

1. Issue a warning on startup if the user doesn't bind to a specific IP address. You can give the example where the server launches before the IP address becomes available, hence it never gets bound to.

1. Use a retry-loop to bind to the specified addresses/ports. If *any* of them fail, the server should shut down with an error. This will prevent such silent failures in the future.
   

Gili

[Silvia Pfeiffer]

Gili,

Have you tried starting the turnserver after DHCP? Here's an
explanation on how to change your registry for it:
http://support.microsoft.com/kb/193888

HTH,
Silvia.

[cowwoc]

Hi Silvia,

Yes I did, but it did not help. I even tried "LanmanWorkstation" which
is one of the last services to launch but ended up with similar results.
The only workaround that seems to help is using a start type of
"Automatic (Delayed Start)". Once Oleg implements the retry loop, this
problems will go away and I should be able to revert to a normal service
type.

Gili

[Oleg Moskalenko]

I'd say that this is a very good idea proposed by Silvia. I am not familiar with MS Windows but I was pretty much sure that something like that must be available, in the configuration - and after reading the link I see that this is indeed possible.

Regards,
Oleg

[Oleg Moskalenko]

Gili, why it did not help ? Because the DHCP system signals "ready" but it really has not completed the IP negotiations yet ? That's weird.

turn-server-project-rfc5766-turn-server+unsubscribe@googlegroups.com.

To post to this group, send email to

turn-server-project-rfc5766-turn-...@googlegroups.com.

Visit this group at
http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "TURN Server project rfc5766-turn-server" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/turn-server-project-rfc5766-turn-server/NaJVSWnD7CI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to

turn-server-project-rfc5766-turn-server+unsubscribe@googlegroups.com.

To post to this group, send email to

turn-server-project-rfc5766-turn-...@googlegroups.com.

Visit this group at
http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google
Groups "TURN Server project rfc5766-turn-server" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to

turn-server-project-rfc5766-turn-server+unsubscribe@googlegroups.com.

To post to this group, send email to

turn-server-project-rfc5766-turn-...@googlegroups.com.

Visit this group at
http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"TURN Server project rfc5766-turn-server" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to

turn-server-project-rfc5766-turn-server+unsubscribe@googlegroups.com.

To post to this group, send email to

turn-server-project-rfc5766-turn-...@googlegroups.com.

[cowwoc]

Correct.

The DHCP client service is up, but it does not mean it acquired an IP address from a server. Again, the scenario I compare this to is the computer is fully booted but the network cable is disconnected. In this case, the DHCP client is done loading, but it does not mean that the local interface is up.

Gili

turn-server-project-rfc57...@googlegroups.com.

To post to this group, send email to

turn-server-project...@googlegroups.com.

Visit this group at
http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "TURN Server project rfc5766-turn-server" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/turn-server-project-rfc5766-turn-server/NaJVSWnD7CI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to

turn-server-project-rfc57...@googlegroups.com.

To post to this group, send email to

turn-server-project...@googlegroups.com.

Visit this group at
http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google
Groups "TURN Server project rfc5766-turn-server" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to

turn-server-project-rfc57...@googlegroups.com.

To post to this group, send email to

turn-server-project...@googlegroups.com.

Visit this group at
http://groups.google.com/group/turn-server-project-rfc5766-turn-server.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"TURN Server project rfc5766-turn-server" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to

turn-server-project-rfc57...@googlegroups.com.

To post to this group, send email to

turn-server-project...@googlegroups.com.

[Oleg Moskalenko]

I filed Issue 101 about the DHCP problem. I fixed it in SVN, and I 'll include the fix in the maintenance release 3.2.2.7, in a bout a week from now.

Regards,
Oleg

[cowwoc]

Hi Oleg,

I looked over the changes. They should work in theory ;) I'll test 3.2.2.7 when it comes out and let you know.

Thanks,
Gili