Packet dropped by VM with IP FW on host

[Christian Gagneraud]

Hi there,

I'm pretty new to vagrant, and i like it very much.
But I have a strange networking problem, I think the VM is dropping packets when it should not.

My host is a HW machine connected on internet in a data center via eth0 (let's say IP 1.2.3.4).
I'm trying to create a local virtual network on which I attach a couple of VMs.
I have tried several way to achieve this but always end up with the same problem.

Right now, i'm creating a couple of peered veth interfaces, and a vbridge.
the first veth is is named veth_host.1 and veth_host.2.
veth_host.2 has the address 192.168.50.1 on the host machine, veth_host.1 is plugged into the vbridge.
I then create a second veth pair, named veth_guest.1 and veth_guest.2, veth_guest.1 is plugged
into the vbridge and veth_guest.2 is used by the Vagrantfile using:
vm.network :public_network, ip: "192.168.50.10", bridge: "veth_master.2"
Similarly, I plugged another VM into this bridge. (i've tried private_network, and VBox "internal", didn't solve the problem)

Now I configure the host to act as a gateway for the local 192.168.50.0/24 network, and I forward port 80/443
from the host's internet iface to the IP address 192.168.50.10.

With this setup, i can connect to 1.2.3.4:80/443 from another machine on the internet, I can as well
connect to 1.2.3.4:80/443 from the second VM, but I cannot connect to 1.2.3.4:80/44 from either
1.2.3.4 (the host itself) or 192.168.50.10 (the guest itself).

This has driven me nuts for the last few days.

To troubleshoot this, i'm using
- on host and guest: "tcpdump -n -v -e -i any" (no name resolution, any interface, verbose with ethernet headers)
- "nc -v -k -l 80" in the guest
- "nc -w 2 1.2.3.4" on the client side
- when the client runs on the host, i use "-s" to choose eth0 or veth_host.2 as the source address.

When the client is another machine accessing 1.2.3.4 or the second guest, everything is fine, i see these packets:
- first, client->1.2.3.4, followed by 192.168.50.1->192.168.50.10
- then 192.168.50.10->192.168.50.1 followed by 1.2.3.4->client.
And the 2 "nc" communicate both ways.

For the case of the client being the host or the guest itself, I can clearly see on the host that
veth_guest.2 receives a 192.168.50.1->192.168.50.10 SYN packet, but inside the guest, i cannot see this packet, I only see the original 192.168.50.10->1.2.3.4 SYN packet.
Which lead me to the conclusion that , for some reasons, the incoming SYN packet is dropped by the VM.

The guest machine has it's firewall disabled (iptables flush, inc nat, ACCEPT everywhere), the host starts with firewall disbaled too, just to make sure it doesn't mess up my experiments. See script at end of email to see how i'm setting up the virtual network and the gateway.

Does anyone knows what would make VBox decide to drop packets?
Any point out, advice or general comment would be greatly appreciated.

Thanks,
Chris

Vagrantfile:
------------------------------------------------------------------
WORKER_COUNT=2
Vagrant.require_version ">= 2.0.2"  # Required for private network to work properly
Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/xenial64"
  config.vm.provision :shell, path: "src/provision-vm.sh"
  config.vm.provision "shell", run: "always",
                      inline: "ip route del default; ip route add default via 192.168.50.1"
  config.vm.provision "shell", run: "always",
                      inline: "echo 'nameserver 8.8.8.8' > /etc/resolv.conf"
  config.vm.define "master" do |master|
    master.vm.network :public_network, ip: "192.168.50.10", bridge: "veth_master.2"
    master.vm.hostname = "master.local"
  end
  config.vm.define "master" do |leader|
    leader.vm.network :public_network, ip: "192.168.50.11", bridge: "veth_leader.2"
    leader.vm.hostname = "leader.local"
  end
  (1..WORKER_COUNT).each do |i|
    config.vm.define "worker#{i}" do |worker|
      worker.vm.network :public_network, ip: "192.168.50.#{20+i}", bridge: "veth_worker#{i}.2"
      worker.vm.hostname = "worker#{i}.local"
    end
  end
end
-------------------------------------------------------------------

setup_vm_networking.sh (run on host):
-------------------------------------------------------------------
WORKER_COUNT=2

function add_patch_cable()
{
    name=$1
    ip link add dev veth_$name.1 type veth peer name veth_$name.2
    ip link set veth_$name.1 up
    ip link set veth_$name.2 up
    ip link set veth_$name.1 master vbridge0
    # switch off checksum off-loading                                                                                                                                      
    ethtool -K veth_$name.1 tx off
    ethtool -K veth_$name.2 tx off
}

# Create a virtual bridge                                                                                                                                                  
ip link add name vbridge0 type bridge
ip link set vbridge0 up
ethtool -K vbridge0 tx off
# Add host to the bridge, set IP address and add route to the network                                                                                                      
add_patch_cable host
ip addr add 192.168.50.1 dev veth_host.2
ip route add 192.168.50.0/24 dev veth_host.2
# Add all VMs to the bridge                                                                                                                                                
add_patch_cable master
add_patch_cable leader
for i in $(seq 1 $WORKER_COUNT); do
    add_patch_cable worker$i
done

# Allow routing                                                                                                                                                            
sysctl net.ipv4.ip_forward=1
# Forward public ports to master VM                                                                                                                                        
PUBLIC_IP="1.2.3.4"
DEST_IP="192.168.50.10"
PORTS="80 443 7999"
for port in $PORTS; do
    iptables -t nat -A PREROUTING -p tcp -d $PUBLIC_IP --dport $port -j DNAT --to-destination $DEST_IP:$port
done

# For vm initiated outbound traffic                                                                                                                                        
iptables -t nat -A POSTROUTING -j MASQUERADE

-------------------------------------------------------------------

[Christian Gagneraud]

On 4 March 2018 at 12:39, Christian Gagneraud <chg...@gmail.com> wrote:
> Hi there,
>
> I'm pretty new to vagrant, and i like it very much.
> But I have a strange networking problem, I think the VM is dropping packets
> when it should not.

I forgot some important details:
- All host and guests are Ubuntu-16.04.
- Host runs vagrant 2.0.2, VBox 5.2.8r121009 (official release, from their ppa)
- guests use vbox addition 5.0 (vbox doesn't provide guest addition in
their ppa!?!)

[Alvaro Miranda Aguilera]

hello

whats wrong on using a virtualbox box switch for this?

since in the way you are doing it today i am not sure you will find more people using it or having the same issues

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/mitchellh/vagrant/issues
IRC: #vagrant on Freenode
---
You received this message because you are subscribed to the Google Groups "Vagrant" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vagrant-up+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vagrant-up/CABxGUTiQUp8N_KjJDbfPgc-F%3Dv72bQeggGxJ941EogWkiCpNsQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Alvaro

[Christian Gagneraud]

On 5 March 2018 at 08:29, Alvaro Miranda Aguilera <kik...@gmail.com> wrote:
> hello
>
> whats wrong on using a virtualbox box switch for this?
>
> since in the way you are doing it today i am not sure you will find more
> people using it or having the same issues

I couldn't get it working with "plain" VBox network, i tried vagrant's
"public_network", "private_network" or using the "--intnet" trick.
I could have use the forwarded_port, unfortunately to listen on
1.2.3.4:80 and 1.2.3.4:443 I would have to run the VM as root.
So I had no choice but to use port forwarding.

What i've recently discovered is that my problem has actually a name:
"Hairpin NAT"
And there's a possible alternative (what i'm using right now):
Split-horizon DNS"

You can find details at https://serverfault.com/a/557776

I'm still interested to know if I can use Vagrant to have this
scenario working, or if the problem is purely on VBox side (which I
believe is)

Thanks,
Chris

[Alvaro Miranda Aguilera]

hello

vagrant will do what you tell vagrant to do and will be limited to what your user can do.

Vagrant networking is very basic and cover the most general user cases, not all the features that are new into virtualbox are present in vagrant (read as features that have been created after the vagrant virtualbox provider was done)

so if you think some of the features are useful, you can try to get them working on Vagrant and sumbit a Pull Request

If you use private_networking, and use the same network on all the boxes, all of them will be connected to the same virtualbox network (like a switch) and you should be able to use . vm_ip:80 or vm_ip:443

Alvaro.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/mitchellh/vagrant/issues
IRC: #vagrant on Freenode
---
You received this message because you are subscribed to the Google Groups "Vagrant" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vagrant-up+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vagrant-up/CABxGUTjTCz8L2mdN9SvN51Mg1SVYb9f9DDRJ8e7sOwX%3Do7rKXQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Alvaro