No response from gcp vm

[Robert Raszuk]

Hi,

Apologies if this has been already answered, but I am not able to find some hints. 

When I run packetdrill from my home machine to GCP the server is not responding. This is the same for any test script. 

Is this a known issue ? Any workaround ? 

client: 

robert@nuc2-kom:~/TCP/git/packetdrill/gtests/net/tcp/ts_recent$ sudo ../../packetdrill/packetdrill -v --wire_client --wire_client_dev=eno1 --wire_server_ip=34.118.112.244   --local_ip=172.16.0.1 --gateway_ip=172.16.0.2 --netmask_ip=255.255.255.0 --remote_ip=172.16.1.1/24 -v reset_tsval.pkt

socket syscall: 1629931171.591686
setsockopt syscall: 1629931171.591706
bind syscall: 1629931171.591714
listen syscall: 1629931171.591737

server: 

root@rr-vm1-waw:~/TCP/packetdrill/gtests/net/packetdrill# ./packetdrill -v --wire_server --wire_server_dev=ens4

inbound injected packet:  0.000341 S 0:0(0) win 20000 <mss 1000,sackOK,TS val 100 ecr 0>

Many thx in advance,

R.

[Robert Raszuk]

Hi, 

After a bit of more investigation I think the issue is that GCP VMs are behind NAT with filtering on only src IP addresses allocated to the native GCP subnets. 

So if packetdrill is using some overlay address of 172.16... it may never connect back to the real src in the Internet. 

Not sure what is the best way to get around this. 

Btw a side two questions ... 

1. Is there any way to use packedrill direct p2p without any overlays ? 

2. What are the default  TCP ports test suites are using ? /* Question for GCP FW to allow them in */

Thank you,

R.

[Neal Cardwell]

Hi,

Thanks for the question.

Unfortunately, packetdrill's "Remote Mode" only supports pairs of machines that are on the same layer 2 broadcast domain (e.g., same physical or virtual Ethernet switch or hub); it will not work for paths like the one you are trying, where the packets would have to travel through a router.

Scenarios where I have used packetdrill's Remote Mode successfully:

  (1) two physical machines on the same physical Ethernet switch

  (2) two VMWare virtual machines on the same physical machine (IIRC "Bridged networking" and "Host-only networking" both work)

This is touched on briefly in the packetdrill USENIX login article in the "Remote Mode" section,  but I'm sorry this is not more clear. 

best regards,

neal

--
You received this message because you are subscribed to the Google Groups "packetdrill" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packetdrill...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/CA%2Bb%2BERnN44NZyfDr-ZGnAzzJgY%3DLHWhbcQFMAEzOW5fOER2%2Bzw%40mail.gmail.com.

[Robert Raszuk]

Hi Neal,

I see so - if I am not mistaken - the way it works and the reason for requiring the same L2 link is due to the way client and server operate in respect to setting up independent TCP sessions to each other. 

Clearly it is not so much about routing (we could feed each end with peer's real IP address to use), but traversing NATs in this case will not work. 

Many thx - will try to just use p2p wireguard encap between my endpoints. Not sure how it will affect TCP timings - should not that much I hope. 

R.

[Robert Raszuk]

Hi, 

Just an update ... the wireguard tunnel does not seem to help. 

The communication seems to be working fine but packtdrill seems to not get any responses and does not print anything after listen syscall... 

Main reason for the effort is to measure correctness of TCP behaviour across chain of TCP port forwarders. 

Client: 

robert@nuc2-kom:~/TCP/git/packetdrill/gtests/net/tcp/blocking$ sudo ../../packetdrill/packetdrill -v --wire_client --wire_client_dev=wg0 --wire_server_ip=10.10.10.80   --local_ip=172.16.0.1 --gateway_ip=172.16.0.2 --netmask_ip=255.255.255.0 --remote_ip=172.16.1.1/24 blocking-accept.pkt
socket syscall: 1629991381.191720
setsockopt syscall: 1629991381.191738
bind syscall: 1629991381.191765
listen syscall: 1629991381.191770
^C

Server: 

raszuk@rr-vm1-waw:~/TCP/packetdrill/gtests/net/packetdrill$ sudo ./packetdrill -v --wire_server --wire_server_dev=wg0
inbound injected packet:  0.101050 S 0:0(0) win 32792 <mss 1000,nop,wscale 7>

Client tcpdump: 

root@nuc2-kom:~# tcpdump -i wg0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
17:26:51.614888 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [S], seq 1028864393, win 65535, options [mss 1380,sackOK,TS val 1532714946 ecr 0,nop,wscale 2], length 0
17:26:51.635920 IP 10.10.10.80.tproxy > nuc2-kom.35352: Flags [S.], seq 2383691374, ack 1028864394, win 65072, options [mss 1340,sackOK,TS val 1472353682 ecr 1532714946,nop,wscale 7], length 0
17:26:51.635959 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [.], ack 1, win 16384, options [nop,nop,TS val 1532714967 ecr 1472353682], length 0
17:26:51.636023 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [P.], seq 1:9, ack 1, win 16384, options [nop,nop,TS val 1532714967 ecr 1472353682], length 8
17:26:51.636039 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [P.], seq 9:190, ack 1, win 16384, options [nop,nop,TS val 1532714967 ecr 1472353682], length 181
17:26:51.636162 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [P.], seq 190:711, ack 1, win 16384, options [nop,nop,TS val 1532714967 ecr 1472353682], length 521
17:26:51.657784 IP 10.10.10.80.tproxy > nuc2-kom.35352: Flags [.], ack 9, win 509, options [nop,nop,TS val 1472353704 ecr 1532714967], length 0
17:26:51.657810 IP 10.10.10.80.tproxy > nuc2-kom.35352: Flags [.], ack 190, win 508, options [nop,nop,TS val 1472353704 ecr 1532714967], length 0
17:26:51.823437 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [P.], seq 711:719, ack 9, win 16384, options [nop,nop,TS val 1532715155 ecr 1472353739], length 8
17:26:51.824384 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [P.], seq 719:727, ack 9, win 16384, options [nop,nop,TS val 1532715156 ecr 1472353739], length 8
17:26:51.824566 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [P.], seq 727:731, ack 9, win 16384, options [nop,nop,TS val 1532715156 ecr 1472353739], length 4
17:26:51.843246 IP 10.10.10.80.tproxy > nuc2-kom.35352: Flags [.], ack 719, win 504, options [nop,nop,TS val 1472353891 ecr 1532715155], length 0
17:26:51.843266 IP 10.10.10.80.tproxy > nuc2-kom.35352: Flags [.], ack 727, win 504, options [nop,nop,TS val 1472353891 ecr 1532715156], length 0
17:26:51.843266 IP 10.10.10.80.tproxy > nuc2-kom.35352: Flags [.], ack 731, win 504, options [nop,nop,TS val 1472353892 ecr 1532715156], length 0
17:26:53.569535 IP nuc2-kom.35352 > 10.10.10.80.tproxy: Flags [F.], seq 731, ack 9, win 16384, options [nop,nop,TS val 1532716901 ecr 1472353892], length 0
17:26:53.629356 IP 10.10.10.80.tproxy > nuc2-kom.35352: Flags [.], ack 732, win 504, options [nop,nop,TS val 1472355678 ecr 1532716901], length 0

Server tcpdump: 

root@rr-vm1-waw:/etc/wireguard# tcpdump -i wg0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
15:26:51.626357 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [S], seq 1028864393, win 65535, options [mss 1380,sackOK,TS val 1532714946 ecr 0,nop,wscale 2], length 0
15:26:51.626424 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [S.], seq 2383691374, ack 1028864394, win 65072, options [mss 1340,sackOK,TS val 1472353682 ecr 1532714946,nop,wscale 7], length 0
15:26:51.647763 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [.], ack 1, win 16384, options [nop,nop,TS val 1532714967 ecr 1472353682], length 0
15:26:51.648194 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [P.], seq 1:9, ack 1, win 16384, options [nop,nop,TS val 1532714967 ecr 1472353682], length 8
15:26:51.648222 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [.], ack 9, win 509, options [nop,nop,TS val 1472353704 ecr 1532714967], length 0
15:26:51.648234 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [P.], seq 9:190, ack 1, win 16384, options [nop,nop,TS val 1532714967 ecr 1472353682], length 181
15:26:51.648244 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [.], ack 190, win 508, options [nop,nop,TS val 1472353704 ecr 1532714967], length 0
15:26:51.649725 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [P.], seq 190:711, ack 1, win 16384, options [nop,nop,TS val 1532714967 ecr 1472353682], length 521
15:26:51.649750 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [.], ack 711, win 504, options [nop,nop,TS val 1472353706 ecr 1532714967], length 0
15:26:51.683310 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [P.], seq 1:9, ack 711, win 504, options [nop,nop,TS val 1472353739 ecr 1532714967], length 8
15:26:51.703289 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [.], ack 9, win 16384, options [nop,nop,TS val 1532715023 ecr 1472353739], length 0
15:26:51.834488 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [P.], seq 711:719, ack 9, win 16384, options [nop,nop,TS val 1532715155 ecr 1472353739], length 8
15:26:51.834532 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [.], ack 719, win 504, options [nop,nop,TS val 1472353891 ecr 1532715155], length 0
15:26:51.835218 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [P.], seq 719:727, ack 9, win 16384, options [nop,nop,TS val 1532715156 ecr 1472353739], length 8
15:26:51.835240 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [.], ack 727, win 504, options [nop,nop,TS val 1472353891 ecr 1532715156], length 0
15:26:51.835512 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [P.], seq 727:731, ack 9, win 16384, options [nop,nop,TS val 1532715156 ecr 1472353739], length 4
15:26:51.835527 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [.], ack 731, win 504, options [nop,nop,TS val 1472353892 ecr 1532715156], length 0
15:26:51.935635 unknown ip 0
15:26:53.580342 IP 10.10.10.70.35352 > rr-vm1-waw.tproxy: Flags [F.], seq 731, ack 9, win 16384, options [nop,nop,TS val 1532716901 ecr 1472353892], length 0
15:26:53.621735 IP rr-vm1-waw.tproxy > 10.10.10.70.35352: Flags [.], ack 732, win 504, options [nop,nop,TS val 1472355678 ecr 1532716901], length 0

Rgs,

R.

[Neal Cardwell]

Yes, the packetrill code is not structured to support that kind of scenario.

If you want to extend it to support something like that, you could start by:

(1) looking at wire_server_netdev_send(), here:

https://github.com/google/packetdrill/blob/827ea9d9e2db79dbbd79a51dd5a43a1097dd901e/gtests/net/packetdrill/wire_server_netdev.c#L159

Right now it is structured to send a raw ethernet frame to the device under test. AFAICT you would need to at least generalize that code to be able to try sending a raw IP packet (trusting the server's kernel to forward the packet to the default gateway).

(2) looking at route_traffic_to_wire_server(), here:

https://github.com/google/packetdrill/blob/827ea9d9e2db79dbbd79a51dd5a43a1097dd901e/gtests/net/packetdrill/wire_client_netdev.c#L64

Right now it is structured to set the route for the IP under test to have a gateway address that is the IP of the packetdrill server. Presumably that needs to be generalized to skip that command for your scenario.

If you get this support to work, I am happy to review a git pull request with the patch(es).

best regards,

neal