Confusion with udp-receive!

40 views
Skip to first unread message

David Storrs

unread,
Sep 25, 2019, 1:08:16 AM9/25/19
to Racket Users
udp-receive! is giving me unexpected results when my local machine ->
router -> server shows the UDP port of the process running on the
local machine instead of the one from the router. I'm not sure how to
get the router's port instead.


The AWS server does this:
(define-values (len senders-host senders-port) (udp-receive! socket buffer))

What I'm actually getting is:

senders-host: <public IP address of my router>
senders-port: 25890 ; this is the UDP port bound by the process on the
local machine

What I'm expecting is:

senders-host: <public IP address of my router>
senders-port: <port number that my router chose when it relayed the
message from my machine to the AWS server>

I've been digging through the RFCs for UDP and Traditional NAT
(https://www.ietf.org/rfc/rfc768.txt and
https://www.ietf.org/rfc/rfc3022.txt) to make sure that I haven't
randomly gotten confused about how routers work but it seems to be
what I recall: The local machine sends to the router using the port
number 25890, the router rewrites it to an arbitrary port number
chosen on the fly, the AWS server sees the router's assigned port and
not 25890.

What am I missing here? I'm sure it's something embarrassingly obvious.



Simplified form of code for reference:

-------------------
shared code
-------------------
(struct Message (message-id attributes) #:prefab)
(struct Binding-Request Message () #:prefab)
(struct Binding-Success-Response Message () #:prefab)
(struct transport-address (ip port) #:prefab)

(define (write-to-bytes v)
(define out (open-output-bytes))
(write v out)
(get-output-bytes out))

-------------------
local machine:
-------------------
(define socket (udp-open-socket #f #f))
(udp-bind! socket #f 25890 #t)
(thread
(thunk
(define-values (len senders-host senders-port) (udp-receive! socket buffer))
(log-msg-debug "host: ~a, port ~a, buffer ~a " senders-host
senders-port buffer)))

(udp-send-to socket default-host default-port
(wrap (Binding-Request 17 (hash))))

-----------------
AWS server:
-----------------
The server has its own UDP socket (bound to 54545, fwiw) and a receive
loop that identifies the Binding-Request and routes it to the
following code:

(define-values (len senders-host senders-port) (udp-receive!
socket buffer)
(define mapped-address (transport-address senders-host senders-port))
(define msg-out
(Binding-Success-Response mid
(hasheq 'mapped-address
mapped-address)))
(udp-send-to socket senders-host senders-port (wrap msg-out))

Alex Harsanyi

unread,
Sep 25, 2019, 3:16:31 AM9/25/19
to Racket Users
Do you know what port the router is using for NAT?  Are you sure that the router is not simply choosing the same port, so 25890 is both your local port and the port used by the router?

Alex.

David Storrs

unread,
Sep 25, 2019, 9:29:07 AM9/25/19
to Alex Harsanyi, Racket Users


On Wed, Sep 25, 2019, 3:16 AM Alex Harsanyi <alexha...@gmail.com> wrote:
Do you know what port the router is using for NAT?  Are you sure that the router is not simply choosing the same port, so 25890 is both your local port and the port used by the router?

I haven't yet 100% ruled it out, but it doesn't look like it. I tried sending traffic to <public IP>:25890 and it was not received.  It's possible that the port went stale and was released before my sending went out, but that seems unlikely, as it should persist for seconds or tens of seconds.My next step is to try again with a flood ping, just to be sure.

Regards, it really shouldn't be doing that. If so, it's leaking information about the inner network to the outside, and that's not what I'd expect from the latest version of the FOSS OpenWRT.


--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/0fc8af6a-c50c-4a90-ba8e-64718161379e%40googlegroups.com.

David Storrs

unread,
Sep 25, 2019, 12:49:51 PM9/25/19
to Alex Harsanyi, Racket Users
We (my business partner and I) ran tcpdump on the router and
determined that no, it is not using the local port. At first it bound
to 65395 and then after we stopped/started the process it bound to a
different port (49428) as expected.

After a bit of digging in the racket source code I note that the
various UDP functions are an FFI into librktio. This leaves me with
two questions:

1) Is it possible that there is a bug in the underlying C code?

2) Why does Racket use a hand-rolled io library instead of a more
standard net stack element? Is it for portability or...?

Matthew Flatt

unread,
Sep 25, 2019, 2:57:19 PM9/25/19
to David Storrs, Alex Harsanyi, Racket Users
At Wed, 25 Sep 2019 12:49:36 -0400, David Storrs wrote:
> 1) Is it possible that there is a bug in the underlying C code?

It's always possible. But if I understand the original problem, it
seems unlikely that a bug manages to exactly reconstruct a port number
that has been replaced in a UDP packet by a NAT.


Just to make sure we're talking about the same thing, I'm enclosing
complete code based on your fragments. If I run "server.rkt" first and
then "client.rkt", I see

me: 0.0.0.0 25891
reply from: ZZZZ 54545
reply: #s(transport-address "XXXX" YYYY)

If I run as-is, using "localhost" as the server shown by ZZZZ, then
XXXX and YYYY are `127.0.0.1` and `25891`, as expected. I change
"client.rkt" to connect to a remote host where my client is behind a
NAT, then XXXX is my NAT's address and YYYY is something other than
25891 --- and that YYYY persists over multiple runs of "client.rkt" and
"server.rkt" in some complicated way that does indeed seem to be at
least tens of seconds. (My NAT is an AirPort.)

If I understand, you're seeing YYYY as always 25891, and you have good
reason to think that the NAT isn't just using 25891 as a convenient
substitute for 25891. I have no explanation.


> 2) Why does Racket use a hand-rolled io library instead of a more
> standard net stack element? Is it for portability or...?

As you say, rktio provides a portable interface non-blocking I/O and
some other OS services.

I forget which libraries I tried back around 1995-96. There wasn't an
obvious choice like libuv at the time. I remember there was library for
Mac OS Classic that provided a Unix-like layer, but I wasn't happy with
it. Some frameworks were starting to build up abstractions, I think,
but only by buying into a lot more of a framework. So, after building
up abstractions that provide exactly what Racket needs over a decade or
two, it would be a lot of work to rebuild on a different library, and
I'm not seeing a pressing need.
common.rkt
server.rkt
client.rkt

George Neuner

unread,
Sep 25, 2019, 3:33:40 PM9/25/19
to David Storrs, racket users
Hi David,

On 9/25/2019 1:08 AM, David Storrs wrote:
> udp-receive! is giving me unexpected results when my local machine ->
> router -> server shows the UDP port of the process running on the
> local machine instead of the one from the router. I'm not sure how to
> get the router's port instead.

NAT routing is transparent:  you are - or at least, should be - seeing
the port number of the remote machine.

But remember that - unlike TCP - UDP can send on the same port that it
listens to.  If the "client" is sending on the same port as the server
is listening, you won't be able to distinguish them by port alone.


> The AWS server does this:
> (define-values (len senders-host senders-port) (udp-receive! socket buffer))
>
> What I'm actually getting is:
>
> senders-host: <public IP address of my router>
> senders-port: 25890 ; this is the UDP port bound by the process on the
> local machine
>
> What I'm expecting is:
>
> senders-host: <public IP address of my router>
> senders-port: <port number that my router chose when it relayed the
> message from my machine to the AWS server>

You won't get the address of the router - you'll get the address of the
client.  (see below)


> I've been digging through the RFCs for UDP and Traditional NAT
> (https://www.ietf.org/rfc/rfc768.txt and
> https://www.ietf.org/rfc/rfc3022.txt) to make sure that I haven't
> randomly gotten confused about how routers work but it seems to be
> what I recall: The local machine sends to the router using the port
> number 25890, the router rewrites it to an arbitrary port number
> chosen on the fly, the AWS server sees the router's assigned port and
> not 25890.
>
> What am I missing here? I'm sure it's something embarrassingly obvious.

What you are overlooking is that UDP is a level 4 [end-to-end] protocol,
but you are trying to get level 3 [routing] information.

Neither UDP nor TCP normally records the path a packet takes - only the
routers themselves know what they do to forward the packet.  The path
between 2 machines is never fixed, and it may change abruptly due to
congestion or outage along the way.    "traceroute" uses a special
(level 3) IP packet which explicitly commands each router along the way
... and the end destination as well if the packet gets there ... to send
back a response to the traceroute initiator.  The path is discovered
dynamically from the responses.


Hope this helps,
George

George Neuner

unread,
Sep 25, 2019, 3:41:54 PM9/25/19
to Matthew Flatt, racket users


On 9/25/2019 2:57 PM, Matthew Flatt wrote:
> At Wed, 25 Sep 2019 12:49:36 -0400, David Storrs wrote:
> > 1) Is it possible that there is a bug in the underlying C code?
>
> It's always possible. But if I understand the original problem, it
> seems unlikely that a bug manages to exactly reconstruct a port number
> that has been replaced in a UDP packet by a NAT.

The NAT  [or last router]  port will be in the IP header, not the UDP
header - the UDP header contains the address:port of the remote client.
AFAIK, the IP (level 3) information is not passed up to the application
by the network stack when the application is using a level 4 protocol:
UDP or TCP.

George

David Storrs

unread,
Sep 25, 2019, 3:44:10 PM9/25/19
to Matthew Flatt, Alex Harsanyi, Racket Users
On Wed, Sep 25, 2019 at 2:57 PM Matthew Flatt <mfl...@cs.utah.edu> wrote:
>
> At Wed, 25 Sep 2019 12:49:36 -0400, David Storrs wrote:
> > 1) Is it possible that there is a bug in the underlying C code?
>
> It's always possible. But if I understand the original problem, it
> seems unlikely that a bug manages to exactly reconstruct a port number
> that has been replaced in a UDP packet by a NAT.
>
>
> Just to make sure we're talking about the same thing, I'm enclosing
> complete code based on your fragments. If I run "server.rkt" first and
> then "client.rkt", I see
>
> me: 0.0.0.0 25891
> reply from: ZZZZ 54545
> reply: #s(transport-address "XXXX" YYYY)
>
> If I run as-is, using "localhost" as the server shown by ZZZZ, then
> XXXX and YYYY are `127.0.0.1` and `25891`, as expected. I change
> "client.rkt" to connect to a remote host where my client is behind a
> NAT, then XXXX is my NAT's address and YYYY is something other than
> 25891 --- and that YYYY persists over multiple runs of "client.rkt" and
> "server.rkt" in some complicated way that does indeed seem to be at
> least tens of seconds. (My NAT is an AirPort.)
>
> If I understand, you're seeing YYYY as always 25891, and you have good
> reason to think that the NAT isn't just using 25891 as a convenient
> substitute for 25891. I have no explanation.

Interesting. When I run server.rkt and client.rkt on my local machine
then I, like you, get 127.0.0.1 and 25891. When I put the code onto
my AWS instance so that there is NAT between us, I *still* get 25891
on both the server and the client. If it's working for you then there
must be something wrong with my network.

Would it be possible for you to give me the IP of the remote machine
you used and leave server.rkt running there for an hour or so? That
way I could try it against a different remote machine without needing
to spin up and provision a different VM.

David Storrs

unread,
Sep 25, 2019, 4:36:43 PM9/25/19
to George Neuner, racket users
Hi Greg,
I don't think this is correct. NAPT is defined here:
https://www.ietf.org/rfc/rfc3022.txt The relevant section is 4.1 (NB:
"TU port" means "TCP or UDP port")

-------- start quote
4.1. IP, TCP, UDP and ICMP Header Manipulations

In Basic NAT model, the IP header of every packet must be modified.
This modification includes IP address (source IP address for outbound
packets and destination IP address for inbound packets) and the IP
checksum.

For TCP ([TCP]) and UDP ([UDP]) sessions, modifications must include
update of checksum in the TCP/UDP headers. This is because TCP/UDP
checksum also covers a pseudo header which contains the source and
destination IP addresses. As an exception, UDP headers with 0
checksum should not be modified. As for ICMP Query packets ([ICMP]),
no further changes in ICMP header are required as the checksum in
ICMP header does not cover IP addresses.

In NAPT model, modifications to IP header are similar to that of
Basic NAT. For TCP/UDP sessions, modifications must be extended to
include translation of TU port (source TU port for outbound packets
and destination TU port for inbound packets) in the TCP/UDP header.
ICMP header in ICMP Query packets must also be modified to replace
the query ID and ICMP header checksum. Private host query ID must be
translated into assigned ID on the outbound and the exact reverse on
the inbound. ICMP header checksum must be corrected to account for
Query ID translation.

-------- end quote

The firewall that I'm behind, like most modern NAT instances, is a
NAPT instance -- a single external address that maps multiple internal
addresses based on a semi-randomly-chosen port assigned when the
initial connection is made. As stated above, the differences between
the two are:

Basic NAT rewrites
- IP header
-- IP address for source (on outbound) or destination (on inbound)
-- IP checksum
UDP header: UDP checksum

NAPT rewrites:
- IP header
-- IP address for source (on outbound) or destination (on inbound)
-- IP checksum
- UDP header:
-- UDP checksum
-- TU port for source (on outbound) or destination (on inbound)

In short, the connection should go like this:

local machine (192.168.1.221, port 25891) sends message to
<AWS_SERVER_IP>:54545 by way of router (192.168.1.1). At each step
the source/destination are:

local machine -> router
(source: 192.168.1.221:25891
dest: <AWS_SERVER_IP>:54545)

router -> AWS-server
(s: <ROUTER_IP>:XXXXX <--- arbitrary port number for this session
d: <AWS_SERVER_IP>:54545)

AWS-server -> router
(s: <AWS_SERVER_IP>:54545
d: <ROUTER_IP>:XXXXX)

router -> local machine
(s: <AWS_SERVER_IP>:54545
d: 192.168.1.221:25891


Did I misunderstand something? If not, then I think that the
udp-receive! on the AWS server should return IP address <ROUTER_IP>
and port <XXXXX>. This matches when Matt Flatt is seeing, but not
what I'm seeing. As yet, I have no explanation, but it's obviously
something to do with my specific setup.



>
>
> Hope this helps,
> George

Alex Harsanyi

unread,
Sep 25, 2019, 7:45:21 PM9/25/19
to Racket Users
Is there any tunneling involved for connecting to your AWS instance?

There is only one copy of the source port in an IP+UDP datagram, and this needs to be whatever the router is using for NAT, otherwise it would not be able to route replies back to your machine on the local network.  If you have some kind of tunneling set up to access the AWS servers, the router will wrap the original UDP packet into another UDP packet with its own source port, leaving the original source port unchanged.

You could also run tcpdump on your AWS server and dump the entire contents of the packet to see what the server actually receives.

Alex.


On Thursday, September 26, 2019 at 12:49:51 AM UTC+8, David Storrs wrote:
We (my business partner and I) ran tcpdump on the router and
determined that no, it is not using the local port.  At first it bound
to 65395 and then after we stopped/started the process it bound to a
different port (49428) as expected.

After a bit of digging in the racket source code I note that the
various UDP functions are an FFI into librktio.  This leaves me with
two questions:

1) Is it possible that there is a bug in the underlying C code?

2) Why does Racket use a hand-rolled io library instead of a more
standard net stack element?  Is it for portability or...?

>> To unsubscribe from this group and stop receiving emails from it, send an email to racket...@googlegroups.com.

David Storrs

unread,
Sep 25, 2019, 8:17:07 PM9/25/19
to Alex Harsanyi, Racket Users
On Wed, Sep 25, 2019 at 7:45 PM Alex Harsanyi <alexha...@gmail.com> wrote:
>
> Is there any tunneling involved for connecting to your AWS instance?
>
> There is only one copy of the source port in an IP+UDP datagram, and this needs to be whatever the router is using for NAT, otherwise it would not be able to route replies back to your machine on the local network. If you have some kind of tunneling set up to access the AWS servers, the router will wrap the original UDP packet into another UDP packet with its own source port, leaving the original source port unchanged.

I'll have to check.

>
> You could also run tcpdump on your AWS server and dump the entire contents of the packet to see what the server actually receives.

Did that, and it's getting the 25890 port (the one for my local
machine instead of the router).
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/a64515ef-640f-4762-ba07-283987d07ef6%40googlegroups.com.

David Storrs

unread,
Sep 26, 2019, 4:55:24 PM9/26/19
to Alex Harsanyi, Racket Users
On Wed, Sep 25, 2019 at 8:16 PM David Storrs <david....@gmail.com> wrote:
>
> On Wed, Sep 25, 2019 at 7:45 PM Alex Harsanyi <alexha...@gmail.com> wrote:
> >
> > Is there any tunneling involved for connecting to your AWS instance?
> >
> > There is only one copy of the source port in an IP+UDP datagram, and this needs to be whatever the router is using for NAT, otherwise it would not be able to route replies back to your machine on the local network. If you have some kind of tunneling set up to access the AWS servers, the router will wrap the original UDP packet into another UDP packet with its own source port, leaving the original source port unchanged.
>
> I'll have to check.

Okay, we checked and there is definitely no tunneling going on.

Does anyone have any further thoughts on this?
Reply all
Reply to author
Forward
0 new messages