UDP forwarder

[Kōshin]

I am trying to set up a UDPForwarder so that I can proxy all UDP
packets received by my stack out on a separate (non-gvisor)
connection. At the moment, my code sometimes only ever processes one
UDP packet, and then hangs:



udpForwarder := udp.NewForwarder(s, func(r *udp.ForwarderRequest) {
var wq waiter.Queue
ep, err := r.CreateEndpoint(&wq)
if err != nil {
log.Printf("error accepting connection: %v", err)
return
}

conn := gonet.NewUDPConn(&wq, ep)
defer conn.Close()

buf := make([]byte, mtu)
for {
n, _, err := conn.ReadFrom(buf)
if err == net.ErrClosed {
log.Printf("UDP connection closed, exiting the read loop", n)
break
}
if err != nil {
log.Printf("error reading udp packet with conn.ReadFrom:
%v, ignoring", err)
continue
}

log.Printf("read a UDP packet with %d bytes", n)
go proxyPacket(conn, buf[:n]) // how would I make sure the
connection is not closed while this runs?
}
})

myStack.SetTransportProtocolHandler(udp.ProtocolNumber,
udpForwarder.HandlePacket)

I'm not too sure about the semantics of UDPForwarder. Do I need to
relinquish ownership of anything or otherwise clean up anything that
is passed to UDPForwarder handler? Can I create multiple endpoints
from the same udp.ForwarderRequest? What I really want to do is
receive UDP packets and send them out to the public internet (using an
ordinary net.Dial etc), then when I receive packets back from the
public internet, send them back through the netstack I created. Would
it perhaps make more sense to not create the endpoint when the UDP
packet is received, but instead when I have a packet to send? Would it
make sense or to create a new endpoint for every UDP packet I wish to
send? If I create an endpoint when I have a packet to send, how can I
send packets with a spoofed "from" address (I already have
myStack.SetSpoofing(myNIC, true), I'm just wondering how I would
specify an arbitrary "from" IPv4 address when sending a UDP packet.)

Thank you very much for any help you can offer here.

[Kevin Krakauer]

Where is it hanging? You can either panic or printf-debug your way to figuring out where threads are blocking. My guess is that the `conn.ReadFrom` loop is blocking in a forever. Returning n == 0 should be enough to indicate that there are, for now, no more bytes. UDP connections aren't explicitly closed like TCP, so this might just never break out of the loop. It looks like we have a test that uses udp.NewForwarder here.

You don't need to clean up anything, but you do need to return control back to netstack. So starting a goroutine, as you're doing, is a good option.

The rest of your questions are increasingly up to you and require separate design. You can "fan out" and "fan in" packets to/from the internet as you've described, but you'll need to decide where those packets go. You could implement this yourself in Go (we have some NAT support in gVisor, but it's not designed to be used outside of where we have it now), or you could use iptables, or you could use something simpler: you can proxy packets. You can do this as soon as they're received, or wait until later. You can "spoof" the source address directly by opening a host UDP socket and sending the payload to the same destination, and spawn goroutines to copy bytes in both directions. This last option isn't that much code, it just requires you to know where you're sending bytes.

Kevin

--
You received this message because you are subscribed to the Google Groups "gVisor Users [Public]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gvisor-users/CAPhZ8gdMf-N8Ee9oP6p041MaxLz5TaVgd3d2ZDawPOUSCSgxUw%40mail.gmail.com.

[Kōshin]

> Where is it hanging? You can either panic or printf-debug your way to figuring out where threads are blocking. My guess is that the `conn.ReadFrom` loop is blocking in a forever.

Yes, this turned out to be exactly what was happening! I was blocking
in the read loop, hence not returning control back to netstack. After
putting the main for loop from the code above into a goroutine, it
works as expected.

> Returning n == 0 should be enough to indicate that there are, for now, no more bytes.

Very helpful, thank you.

> UDP connections aren't explicitly closed like TCP, so this might just never break out of the loop. It looks like we have a test that uses udp.NewForwarder here.

Also very helpful - thanks again.

>
> You don't need to clean up anything, but you do need to return control back to netstack. So starting a goroutine, as you're doing, is a good option.
>
> The rest of your questions are increasingly up to you and require separate design. You can "fan out" and "fan in" packets to/from the internet as you've described, but you'll need to decide where those packets go. You could implement this yourself in Go (we have some NAT support in gVisor, but it's not designed to be used outside of where we have it now), or you could use iptables, or you could use something simpler: you can proxy packets. You can do this as soon as they're received, or wait until later. You can "spoof" the source address directly by opening a host UDP socket and sending the payload to the same destination, and spawn goroutines to copy bytes in both directions. This last option isn't that much code, it just requires you to know where you're sending bytes.

Yes, this last option is what I want. I have it working now. I worry
though that I create a new endpoint each time I get a
udp.ForwarderRequest, and it's not exactly clear where I should close
the endpoint, since there isn't a clear notion of "closed" for the
other side of the proxied UDP connection (i.e the side that sends UDP
packets out to the internet via net.Dial). Won't this cause the number
of endpoints in the stack to grow unbounded over time?

[Kevin Krakauer]

> Won't this cause the number of endpoints in the stack to grow unbounded over time?

Largely true. Since all the packets are entering netstack with a single source IP and single destination IP:port, there is an upper bound of 65K connections. After that, any packet from the application will reuse an existing connection. But that's pretty inefficient.

Handling this is going to be application-specific: you can limit the number of endpoints explicitly or have them time out, or if you know more about the application sending packets you may have some notion of when connections are "finished". That could be based on timing (e.g. the app only ever sends small bursts of packets over the course of a few seconds), or you could inspect the data itself.