UDP Send Errors

[David Lewanda]

Hi,

I've just started evaluating this STUN/TURN server as a replacement for an existing STUN/TURN server in a project I'm working on. The extensive options for security and scalability are quite impressive and tempting. I have pulled down the latest code and tried it out on my own CentOS 6 EC2 instance. It builds and installs fine, and works great when making STUN connections between a client elsewhere on the Internet and one on the same EC2 instance. When I need TURN, however, the client appears to connect for a few seconds, but invariably ends up dropping the connection and spinning for a while. Eventually it may make a connection after a number of tries, but often I end up killing it because it's taking way to long before it can establish the connection. In looking at the verbose (-v) logging for the turnserver application, I am seeing a repeated error of ("udp send: Connection refused") and/or ("udp send: Invalid argument) without a timestamp or other logging decoration. I believe this is coming from the perror call in ns_ioalib_engine_impl.c on line 2244, from the udp_send call on line 2239 of the same file. I am currently running without authentication and have set the -X option to my EC2 elastic IP and the -L option to my eth0's EC2 adapter. I've tried adding some logging to the code myself, but can't seem to trace the path that would be causing this problem. Any advice would be greatly welcomed!

Thanks,

Dave

[Oleg Moskalenko]

Hi Dave

it sounds like something is wrong in the connectivity between the EC2 instance and your client. Try to run wireshark on EC2, take the trace and see what is going on. Double-check your -X and -L settings. Check your turnserver.conf file - see if there is conflict with your -X and -L settings. If nothing helps, put the debug print in the line 2238 (print dest_addr) and send it to me. If you will not be able to figure out the wireshark traces, send them to me.

Regards,
Oleg

[Oleg Moskalenko]

David, also get the AMI that I set in Amazon, instantiate it and see how I set the TURN server in EC2, as an example.

Also, remember that you cannot use DNS names in the parameters - only absolute addresses.

On Tuesday, July 16, 2013 1:42:39 PM UTC-7, David Lewanda wrote:

[David Lewanda]

Oleg,

Thank you for your prompt reply. I will grab your AMI later tonight or tomorrow and give it a spin. I have checked the -L and -X options and they look right to me. I am using absolute addresses and not DNS names as the parameters as you pointed out. I will also try to get Wireshark/tcpdump traces from the server and send them to you if I can't figure it out myself. I will also debug that one line.

Dave

[Oleg Moskalenko]

Dave, also check the EC2 instance network filters (security groups). Your filters may be, are not allowing UDP packets on some ports. As far as I remember, the default filter would not work. You have to allow UDP traffic to your listening ports (default are 3478, 3479, 5349, 5350), may be you will need 443 and 444) and to your relay ports (default are 49152-65535).

[Oleg Moskalenko]

By the way, "udp send: Connection refused" may be not a real problem - this message is usually emitted when the client died and the server received ICMP error packet. That must not be a problem, this is a normal situation, and the server supposed to recover from that.

 The message "udp send: Invalid argument" may be a more interesting thing. Most likely this is a sign that the server is trying to connect to a badly formatted (or formed) IP address. try to check the IP addresses in the STUN/TURN messages in the wireshark.

[David Lewanda]

I thought of that and threw the security group wide open, with everything allowed in. I will investigate further shortly.

[David Lewanda]

One interesting fact we've discovered over here is that our TURN client works fine when connecting from a different machine to the TURN server, but only seems to have the trouble when the TURN client is running on the same machine as the server. Does that help at all?

[Oleg Moskalenko]

On the same machine ?!

That's strange. This is the easiest variant and I cannot imagine what would be the problem. I was thinking about NATs, filewalls, ICMP, lots of things, but ON THE SAME MACHINE... this is bizarre.

The only problem that can be is if you used -d (or --listening-device) option, or if you used a similar option in the client program (you are binding the socket to the network device with that option). That would essentially destroy the local loopback functionality and the local clients would not be able to connect properly to the TURN server. Are you using something like that ?

Regards,
Oleg
 

[David Lewanda]

We tried with the -d option and with out with similar results. The client application is binding to the same eth0 device as the -d option is specifying when it is used, as well as binding to the same IP address as the -L option. So you're saying that's not allowed by the client?

[Oleg Moskalenko]

In general case, when you are using the "bind to device" socket option, it somehow screws the local loopback communications in Linux. I do not have precise explanation for that, but that is a fact.

The "device" option is used for optimization and for "firewalling" purposes (when TURN server is a sort of "firewall" between two networks). This is Linux-only option. Usually people do not need it.

If without the "device" option... I've never seen a problem with "collocated" local loopback clients. In general case, a TURN server is not supposed to work with the local clients - but nothing in this TURN server design and implementation prevents it from working with the local clients. All the "examples" scripts are working in the local environment.

Another thing that I can think of is the Linux network filtering options in the kernel. Are you sure that there is nothing there ?

Regards,
Oleg

[Oleg Moskalenko]

David, check this please:

https://groups.google.com/forum/?fromgroups=#!topic/discuss-webrtc/-DPY-BDj_TM

Oleg

[David Lewanda]

Oleg,

Thanks for the tip. That's not exactly our situation. In our case, one end of the connection is on a PC here, and the other end is on the same machine as our EC2 instance. Both machines have access to the Internet. After further investigation, it seems that it just takes a long time and many attempts to connect, which is much longer than if both ends are separate from the TURN server. I am attaching a sample tcpdump (wireshark) capture from our server when the situation occurs. Please let me know if you have any suggestions as to what might be going on and/or how to resolve it.

Thanks,

Dave

[Oleg Moskalenko]

David, indeed, I see some very strange network patterns.

For example, see the STUN packets 149,150,151,152, and the replies. The network configuration does not seems very normal to me.

The networks 10.x and 54.x are on the same EC2 system, right ? I see the packets are spinning inside back and force.

I do not see any TURN packets - it uses just STUN.

Can you always use separate systems ?

Regards,
Oleg

[David Lewanda]

Oleg,

Thanks so much for the quick response. The 10.x IP address is the one assigned to eth0 by EC2. The 54.x address is the public elastic IP assigned back to the EC2 instance. I will double check our setup and make sure there are TURN packets included and try to capture it again. We are looking to make a single simple appliance image where all the server components, including our server (which is the co-located TURN client) on the same machine, so if we can support both separate and combined scenarios, that would be ideal. Please let us know if you have any other suggestions.

Dave

[Oleg Moskalenko]

David, with loopback interface there may be hard-to-catch tricks. For example, a "peer A" in TURN communications is listening to address 0.0.0.0:12345 (all packets to port 12345 on all system addresses). Then, say, another peer B establishes a connection to the TURN server and through the TURN server sends a packet to the peer A on address 127.0.0.1:12345. Peer A accepts that packet and sends back the response through the socket bound to 0.0.0.0 interface. But the system may choose another local address, like, 10.1.2.3 as the source of the response packet from peer A to the TURN server. And the TURN server, because it has permissions established on 127.0.0.1 for peer A, will reject that packet, silently, as it has to do according to the specs.

I personally observed such a case. So I am just warning you that strange things are possible, locally. You can double-check that packets are getting sent/received on the right addresses.

[Oleg Moskalenko]

By the way, David, I totally forgot about one thing. If you are using our TURN Server AMI image, then, for the security purpose, in the /etc/turnserver.conf file the loopback peers are explicitly forbidden. So, it makes the loopback connections impossible. Find and comment out that option, and then restart the server. That must fix the problem.

Regards,
Oleg

[David Lewanda]

Hi Oleg,

Thanks for the heads up. Unfortunately I was tied up and not able to continue working on this project for a bit, so I haven't made any progress of late, but I'm getting back to it this week. I never did get a chance to try your AMI on EC2, and always was feeding the parameters through the command line without even setting up a configuration file. Any chance that loopback peers is off by default? Do I need to explicitly enable that feature?

Thanks,

Dave

[Oleg Moskalenko]

Hi Dave

no, the loopback is ON by default. If you check all the "example" scripts they use the loopback interface and they work.

The best solution would be to run the wireshark locally and go packet-by-packet to figure out why its not working. That must be something very simple.

Regards,
Oleg

[David Lewanda]

Oleg,

Thanks again. I will likely do that some point this week and report back if I figure it out or have any other questions. I really appreciate your effort and prompt responses!

Dave

[David Lewanda]

Oleg,

I wanted to let you know that I had some time last week to resume working on this, and I was able to get everything working. It turned out to be an issue with setting the -L option to the EC2 instance's eth0 address. If I took that out, but used -d eth0, it started working just as well as I would have expected. I haven't had a chance to investigate why that option would be a limiting factor, or if it would work with other NAT'ed setups outside EC2, but it works for us now, so that's good enough to go forward. Thanks again for your assistance.

Dave

[Oleg Moskalenko]

David, in EC2 environment -L must not be used. For that kind of environment, -X has to be used.

Regards,
Oleg

[David Lewanda]

Oleg,

Thanks for confirming that. It had been tripping us up earlier. Just as a heads up, we're looking into https://github.com/jordansissel/fpm as a way to package up the server as a CentOS 6 RPM (and maybe other distros and package managers in the future). We'll be sure to let you know how it goes.

Dave