Using gVisor's tcpip/transport to test a more lightweight implementation

[Pat Whitti]

Hello peeps! 

I'm currently developing a TCP stack for use on the RP2040 and wanted to be able to test the implementation on a PC. I came across the gvisor project and it looks like EXACTLY what I'd want to use. Only issue is: how?! A quick scan seems like I could create operating system pipes via pipefs.NewConnectedPipeFDs which has been so graciously provided by gvisor. However, there's not much documentation on how to use it.

A little on my NIC abstraction interface below:

type NIC interface {

RecvEthHandle(handler func(pkt []byte) error)

SendEth(pkt []byte) error

MAC() net.HardwareAddr

}

And that's really all there is to it. Send/receive raw ethernet packets over wifi. No ethernet CRC calculation required.
I need to have what I send to my NIC from my TCP implementation be routed to gVisor and have gVisor's output go through the handler callback I set.
I've been looking at replicating this publicly available gVisor example, but with my TCP lib.

Thanks in advance for any help/suggestions you send this way!

Patricio Whittingslow, maintainer of the TinyGo RP2040 SDK

[Kevin Krakauer]

Hi Pat,

Happy to help. First: what's the idea behind using gVisor? I may be misunderstanding, but if you want to test your TCP stack on a PC, you could use the following setup:

[ your TCP stack ] <---> [ AF_PACKET socket on veth device ] <---> [ test program with callback that opens a TCP socket on the other end of the veth ]

AF_PACKET lets you write a packet with ethernet headers. But if you need gVisor:

You're probably not looking for `pipefs.NewConnectedPipeFDs`. That's something that's called inside gVisor when a program makes a `pipe()` syscall, not something that you're likely to call directly.

The example is a good starting point for using gVisor's netstack, which can be used separately from the rest of gVisor. I'm guessing you just want the netstack.

But I'm not sure I totally understand your setup. Is this right? Lemme know if I'm off here:

[ your TCP stack ] <---> [ NIC interface ] <---> [ gVisor netstack ] <---> [ callback ]

This is probably doable but a bit awkward -- gVisor exposes "endpoints", but your stack wants to speak directly to a NIC. I'd suggest structuring the test more like this:

[ your TCP stack ] <---> [ AF_PACKET socket on veth device ] <---> [ gVisor netstack on the other end of the veth ] <---> [ callback ]

With gVisor netstack, you can create an endpoint to read from and write to for the callback. But also: 

Hope that's helpful, but let us know if you have more questions and/or I got the test setup completely wrong :)

Kevin

--
You received this message because you are subscribed to the Google Groups "gVisor Users [Public]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/a0326c6c-5a51-410a-a3e3-753cc3077d97n%40googlegroups.com.

[Pat Whitti]

> but if you want to test your TCP stack on a PC, you could use the following setup:

Yes! I want to test the stack on my PC and not have to find bugs once running on a constrained device such as the RP2040.

>  [your TCP stack ] <---> [ AF_PACKET socket on veth device ] <---> [ test program with callback that opens a TCP socket on the other end of the veth ]

This sounds like it could work, but I'm unfamiliar with C. I understand I'd have to interface with linux syscalls, probably need to be root user. If I can avoid that and just use pure Go I'd rather do that (if it's not too hard)

> I'm guessing you just want the netstack.

From what I can tell, only the tcpip package is enough + the stack.Stack to manage it.

> [ your TCP stack ] <---> [ NIC interface ] <---> [ gVisor netstack ] <---> [ callback ]

Yes, this is what I was trying my hand at building but I'm running into difficulties understanding the subtleties in how LinkEndpoint works. I really wish there was some basic interface builder for small dumb things like this.

> [ your TCP stack ] <---> [ AF_PACKET socket on veth device ] <---> [ gVisor netstack on the other end of the veth ] <---> [ callback ]

I'm open to the idea, but I've already tried TUN/TAPing in the past and I did not get far- ran into issues with the packets I was receiving being incomplete. I really don't know where to start looking on how I'd set this up. 

So overall: Yes, I feel like you understand what I want to do. I've got my cute, simple little netstack which is quite manageable and I'm just overwhelmed by the complexity of setting up gVisor and/or AF_PACKET.

Is there a gVisor example that sets up most of what I want to do?

Pat

[Kevin Krakauer]

If you want to avoid syscalls entirely, you can try creating a packet socket on a loopback device within netstack and having your TCP stack write to it. And then you'd have a regular TCP endpoint on the other end with your callback. AFAIK nobody has tried doing this, so I'm not sure whether you'd run into problems. People typically use netstack to parse packets, whereas you actually want to plug a different stack into netstack.

But I really think your best bet is the veth:
[your TCP stack ] <---> [ AF_PACKET socket on veth device ] <---> [ test program with callback that opens a socket on the other end of the veth ]

- There's no example of using netstack like I described at the top, and it's very possible you'll run into issues as we've never tried to do that before.

- The veth setup is just way simpler: you don't even need netstack. Your callback program can just listen on a regular TCP socket. If you need to inspect the entire packet, it can listen on an AF_PACKET socket.

- C isn't required, it can all be done in Go. You can use the unix package to create AF_PACKET sockets as we do in several cases within gVisor. Then you can read/write directly to the socket.

- Creating a veth device can be done from command line, and the internet has a bunch of examples.

You would have to run with sudo/root unless you're feeling ambitious and want to deal with network namespaces. And sorry we don't have examples of the gVisor-only setup: it's an uncommon use case.

Kevin

To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/06c64bc1-05e5-4eb6-a13c-a11332f6df0fn%40googlegroups.com.