DNS Not working with CoreOS on libvirt

[Morten Minke]

Hello,

Is there anyone who uses CoreOS on the Linux/KVM/Qemu/Libvirt environment.

I have a very weird problem that the networking is not correctly working and I cannot get my hands on the cause. I have setup libvirt according to the CoreOS libvirt documentation. However, I do not use bridged networking mode, but NAT mode for testing purposes (I do not yet have dedicated hardware).

My problem started when I tried to run etcd on the CoreOS node. It didn't boot. Investigating things, it showed that it could not find discovery.etc.io.

Then I started trying other things, such as ping(ing) www.google.com while using 8.8.8.8 as my DNS setting. Whatever I do, it constantly returns with a 'host not found' error.

On my host which runs the virtual machine, I used Wireshark to see what happens and it shows that the DNS request is actually made to the outside world and that a response is sent back.

For some reason though, the message doesn't get through to CoreOS.

The strange thing is that when I execute 

ping 8.8.8.8

on my coreos instance I do get a proper response.

I also installed a clear VM on the same libvirt network with and Ubuntu image. This image works fine, I can access the network/internet, ping hosts etc.

I am really getting frustrated for not finding the cause and not understanding the problem fully I guess.

Hopefully someone can help me out.

My configs are:

#cloud-config


hostname: coreos-node1


ssh_authorized_keys:
 - ssh-rsa AAAA........


coreos:
  units:
    - name: 10-static-eth0.network
      content: |
        [Match]
        MACAddress=52:54:00:fe:b3:c1


        [Network]
        Address=172.16.1.11/24
        Gateway=172.16.1.1
        DNS=8.8.8.8
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start
  etcd:
    name: coreos-node1
    discovery: https://discovery.etcd.io/95bc2bf8ae4690b17ea8d995a2234745


users:
  - name: morten
    passwd: $6$rounds=4096$L8FHm22KK2B$bZrnYqibeXp3X4.jWuO/Z2Ku6vm5rEV5TwDsiM8IkOkrpviA1FOiTiBJtNy.zqiATWnQv0afx89SU7xCNbdau.
    groups:
      - sudo
      - docker

My libvirt VM definition is:

<domain type='kvm'>
  <name>coreos-node1</name>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/coreos-node1/coreos_production_qemu_image.img'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <controller type='usb' index='0'>
    </controller>
    <filesystem type='mount' accessmode='squash'>
      <source dir='/var/lib/libvirt/images/coreos-node1/configdrive/'/>
      <target dir='config-2'/>
      <readonly/>
    </filesystem>
    <interface type='network'>
      <mac address='52:54:00:fe:b3:c1'/>
      <source network='coreos'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
    </sound>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
    <memballoon model='virtio'>
    </memballoon>
  </devices>
</domain>

On the coreos instance, my /etc/resolve.conf file looks like:

# This file is managed by systemd-resolved(8). Do not edit.
#
# Third party programs must not access this file directly, but
# only through the symlink at /etc/resolv.conf. To manage
# resolv.conf(5) in a different way, replace the symlink by a
# static file or a different symlink.


nameserver 8.8.8.8

I probably am doing something really stupid, but I cant see what :=(

[Jonathan Boulle]

Is it only DNS traffic showing this issue? What if you try something like hitting Google directly, e.g. curl 76.14.93.178

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Morten Minke]

Direct traffic works fine. I even tried https, but offcourse that fails because the google.com certificate is not valid for the ip address, but it shows, the certificate exchange also works for 443.

I have to try to install the docker toolbox image, but offcourse I need DNS for that as well.

Adding entries in the host file works too.

[Morten Minke]

I managed to get toolbox and within that tcpdump installed (took me some edits in the hosts files). The result of a ping request to one of our newssites is:

20:45:22.157325 IP coreos-node1.56188 > 8.8.8.8.domain: 40116+ A? www.nu.nl. (27)

20:45:22.173656 IP 8.8.8.8.domain > coreos-node1.58978: 40116 5/0/0 CNAME www-nu-nl.gslb.sanomaservices.nl., A 62.69.174.75, A 62.69.166.15, A 62.69.166.18, A 62.69.174.78 (135)

20:45:22.173679 IP coreos-node1 > 8.8.8.8: ICMP coreos-node1 udp port 58978 unreachable, length 171

So, as far as I can see DNS entries come back correctly, but somehow udp traffic after that is a problem. But I do not completely understand what the last line should do.

Hopefully this helps understanding the problem!

[Morten Minke]

It seems that configuring systemd-networkd with a fixed ip address is not working correctly. I have the following observation:

The following WORKS using the cloud-config file:

[Match]

MACAddress=52:54:00:fe:b3:c1

[Network]

DHCP=both

The following DOES NOT WORK using the cloud-config file (leaving everything else exactly the same):

[Match]

MACAddress=52:54:00:fe:b3:c1

[Network]

DNS=8.8.8.8

[Address]

Address=172.16.1.11/24

[Route]

Gateway=172.16.1.1

Hopefully this helps determining whether I do something wrong or whether this is a bug while the network is configured!

Any help would be much appreciated.

Morten

[Alex Crawford]

What does `ip addr` return when you statically configure the interface?

-Alex

---

From: Morten Minke
Sent: ‎10/‎11/‎2014 8:06
To: coreo...@googlegroups.com
Subject: Re: DNS Not working with CoreOS on libvirt

--

[Morten Minke]

Hello Alex,

See below different outputs from ip. The strange thing is that I can ssh into the VM. Certain traffic works. using curl on an IP address works, using it on a DNS does not work. I get the feeling that tcp traffic works and udp traffic has problems, but I am not a network guru so I am not sure and do not know how to verify the differences between the DHCP and the static configuration.

core@coreos-node1 ~ $ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:fe:b3:c1 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.11/24 brd 172.16.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fefe:b3c1/64 scope link
       valid_lft forever preferred_lft forever

core@coreos-node1 ~ $ ip route
default via 172.16.1.1 dev eth0
172.16.1.0/24 dev eth0  proto kernel  scope link  src 172.16.1.11

core@coreos-node1 ~ $ ip route show table local
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1
broadcast 172.16.1.0 dev eth0  proto kernel  scope link  src 172.16.1.11
local 172.16.1.11 dev eth0  proto kernel  scope host  src 172.16.1.11
broadcast 172.16.1.255 dev eth0  proto kernel  scope link  src 172.16.1.11

[Seán C. McCord]

Would you be up to providing a packet capture from your host machine?

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Seán C. McCord

ule...@gmail.com

CyCore Systems

[Morten Minke]

The following is the tcpdump output from the HOST when a fixed IP address is used. This is the output from the moment the VM is booted:

$ sudo tcpdump -i virbr1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on virbr1, link-type EN10MB (Ethernet), capture size 65535 bytes
21:18:34.171427 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
21:18:34.547626 ARP, Request who-has 172.16.1.1 tell 172.16.1.11, length 28
21:18:34.547649 ARP, Reply 172.16.1.1 is-at 52:54:00:99:3c:f5 (oui Unknown), length 28
21:18:34.547681 IP 172.16.1.11.37424 > google-public-dns-a.google.com.domain: 2856+ A? coreos-node1. (30)
21:18:34.547719 IP 172.16.1.11.37424 > google-public-dns-a.google.com.domain: 2137+ AAAA? coreos-node1. (30)
21:18:34.596865 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 2137 NXDomain 0/1/0 (105)
21:18:34.597125 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 2856 NXDomain 0/1/0 (105)
21:18:34.683830 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [S], seq 822672812, win 29200, options [mss 1460,sackOK,TS val 4294670655 ecr 0,nop,wscale 7], length 0
21:18:34.846637 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [S.], seq 784171707, ack 822672813, win 14480, options [mss 1460,sackOK,TS val 306613741 ecr 4294670655,nop,wscale 8], length 0
21:18:34.846796 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 1, win 229, options [nop,nop,TS val 4294670818 ecr 306613741], length 0
21:18:34.847247 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 1:144, ack 1, win 229, options [nop,nop,TS val 4294670818 ecr 306613741], length 143
21:18:34.906430 IP6 :: > ff02::1:fffe:b3c1: ICMP6, neighbor solicitation, who has fe80::5054:ff:fefe:b3c1, length 24
21:18:34.940451 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
21:18:35.012434 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [.], ack 144, win 61, options [nop,nop,TS val 306613782 ecr 4294670818], length 0
21:18:35.013375 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [.], seq 1:1403, ack 144, win 61, options [nop,nop,TS val 306613782 ecr 4294670818], length 1402
21:18:35.013519 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 1403, win 251, options [nop,nop,TS val 4294670985 ecr 306613782], length 0
21:18:35.013744 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [.], seq 1403:2805, ack 144, win 61, options [nop,nop,TS val 306613782 ecr 4294670818], length 1402
21:18:35.013861 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 2805, win 274, options [nop,nop,TS val 4294670985 ecr 306613782], length 0
21:18:35.014242 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [P.], seq 2805:3900, ack 144, win 61, options [nop,nop,TS val 306613782 ecr 4294670818], length 1095
21:18:35.014330 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 3900, win 296, options [nop,nop,TS val 4294670985 ecr 306613782], length 0
21:18:35.015161 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 144:411, ack 3900, win 296, options [nop,nop,TS val 4294670986 ecr 306613782], length 267
21:18:35.015247 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 411:417, ack 3900, win 296, options [nop,nop,TS val 4294670986 ecr 306613782], length 6
21:18:35.015272 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 417:458, ack 3900, win 296, options [nop,nop,TS val 4294670986 ecr 306613782], length 41
21:18:35.183576 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [.], ack 458, win 65, options [nop,nop,TS val 306613825 ecr 4294670986], length 0
21:18:35.185473 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [P.], seq 3900:3947, ack 458, win 65, options [nop,nop,TS val 306613826 ecr 4294670986], length 47
21:18:35.185815 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 458:756, ack 3947, win 296, options [nop,nop,TS val 4294671157 ecr 306613826], length 298
21:18:35.387841 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [P.], seq 3947:4413, ack 756, win 70, options [nop,nop,TS val 306613875 ecr 4294671157], length 466
21:18:35.389504 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 756:1031, ack 4413, win 318, options [nop,nop,TS val 4294671361 ecr 306613875], length 275
21:18:35.545851 IP 172.16.1.11.35387 > google-public-dns-a.google.com.domain: 40664+ A? coreos-node1. (30)
21:18:35.545964 IP 172.16.1.11.35387 > google-public-dns-a.google.com.domain: 19652+ AAAA? coreos-node1. (30)
21:18:35.589481 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [P.], seq 4413:4755, ack 1031, win 74, options [nop,nop,TS val 306613927 ecr 4294671361], length 342
21:18:35.590167 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 1031:1232, ack 4755, win 340, options [nop,nop,TS val 4294671561 ecr 306613927], length 201
21:18:35.596108 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 40664 NXDomain 0/1/0 (105)
21:18:35.596262 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 141
21:18:35.740200 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 19652 NXDomain 0/1/0 (105)
21:18:35.740369 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 141
21:18:35.754325 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.11.40761: Flags [P.], seq 4755:5352, ack 1232, win 78, options [nop,nop,TS val 306613968 ecr 4294671561], length 597
21:18:35.794535 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 5352, win 361, options [nop,nop,TS val 4294671766 ecr 306613968], length 0
21:18:35.908450 IP6 fe80::5054:ff:fefe:b3c1 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
21:18:35.908471 IP6 fe80::5054:ff:fefe:b3c1 > ip6-allrouters: ICMP6, router solicitation, length 16
21:18:36.109720 IP 172.16.1.11.60614 > google-public-dns-a.google.com.domain: 34480+ A? 1.pool.ntp.org. (32)
21:18:36.109770 IP 172.16.1.11.60614 > google-public-dns-a.google.com.domain: 58372+ AAAA? 1.pool.ntp.org. (32)
21:18:36.147098 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 34480 4/0/0 A 82.197.221.30, A 213.154.236.182, A 131.211.8.244, A 37.139.4.57 (96)
21:18:36.147183 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:18:36.157993 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 58372 0/1/0 (96)
21:18:36.158209 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:18:36.872522 IP6 fe80::5054:ff:fefe:b3c1 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
21:18:38.092057 IP 172.16.1.11.ntp > ntp4.bit.nl.ntp: NTPv4, Client, length 48
21:18:38.104335 IP ntp4.bit.nl.ntp > 172.16.1.11.ntp: NTPv4, Server, length 48
21:18:39.608647 ARP, Request who-has 172.16.1.11 tell 172.16.1.1, length 28
21:18:39.608798 ARP, Reply 172.16.1.11 is-at 52:54:00:fe:b3:c1 (oui Unknown), length 28
21:18:39.916440 IP6 fe80::5054:ff:fefe:b3c1 > ip6-allrouters: ICMP6, router solicitation, length 16
21:18:40.551020 IP 172.16.1.11.35387 > google-public-dns-a.google.com.domain: 40664+ A? coreos-node1. (30)
21:18:40.551077 IP 172.16.1.11.35387 > google-public-dns-a.google.com.domain: 19652+ AAAA? coreos-node1. (30)
21:18:40.568796 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 40664 NXDomain 0/1/0 (105)
21:18:40.568943 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 141
21:18:40.602971 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 19652 NXDomain 0/1/0 (105)
21:18:40.603094 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 141
21:18:41.114920 IP 172.16.1.11.60614 > google-public-dns-a.google.com.domain: 34480+ A? 1.pool.ntp.org. (32)
21:18:41.115015 IP 172.16.1.11.60614 > google-public-dns-a.google.com.domain: 58372+ AAAA? 1.pool.ntp.org. (32)
21:18:41.153622 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 58372 0/1/0 (96)
21:18:41.153859 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:18:41.157244 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 34480 4/0/0 A 131.211.8.244, A 213.154.236.182, A 87.195.109.106, A 213.136.0.252 (96)
21:18:41.157461 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:18:43.924624 IP6 fe80::5054:ff:fefe:b3c1 > ip6-allrouters: ICMP6, router solicitation, length 16
21:18:46.120061 IP 172.16.1.11.40736 > google-public-dns-a.google.com.domain: 14032+ A? 1.pool.ntp.org. (32)
21:18:46.120139 IP 172.16.1.11.40736 > google-public-dns-a.google.com.domain: 62628+ AAAA? 1.pool.ntp.org. (32)
21:18:46.161836 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 62628 0/1/0 (96)
21:18:46.161935 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:18:46.162378 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 14032 4/0/0 A 82.197.221.30, A 94.228.220.14, A 131.211.8.244, A 213.154.236.182 (96)
21:18:46.162454 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:18:51.125149 IP 172.16.1.11.40736 > google-public-dns-a.google.com.domain: 14032+ A? 1.pool.ntp.org. (32)
21:18:51.125187 IP 172.16.1.11.40736 > google-public-dns-a.google.com.domain: 62628+ AAAA? 1.pool.ntp.org. (32)
21:18:51.142840 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 14032 4/0/0 A 82.197.221.30, A 213.154.236.182, A 131.211.8.244, A 37.139.4.57 (96)
21:18:51.142922 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:18:51.143895 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 62628 0/1/0 (96)
21:18:51.143938 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:18:56.130685 IP 172.16.1.11.55874 > google-public-dns-a.google.com.domain: 7690+ A? 2.pool.ntp.org. (32)
21:18:56.130799 IP 172.16.1.11.55874 > google-public-dns-a.google.com.domain: 1164+ AAAA? 2.pool.ntp.org. (32)
21:18:56.174923 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 1164 4/0/0 AAAA 2001:7b8:3:32:213:136:0:252, AAAA 2001:7b8:633:1:213:154:236:182, AAAA 2001:888:2000:d::a6, AAAA 2001:418:3ff::1:53 (144)
21:18:56.175122 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 180
21:18:56.204841 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 7690 4/0/0 A 81.4.121.228, A 213.109.127.195, A 83.247.2.80, A 213.239.154.12 (96)
21:18:56.204993 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:19:01.135763 IP 172.16.1.11.55874 > google-public-dns-a.google.com.domain: 7690+ A? 2.pool.ntp.org. (32)
21:19:01.135801 IP 172.16.1.11.55874 > google-public-dns-a.google.com.domain: 1164+ AAAA? 2.pool.ntp.org. (32)
21:19:01.191144 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 7690 4/0/0 A 80.69.83.153, A 81.4.121.228, A 93.94.224.67, A 213.109.127.195 (96)
21:19:01.191195 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:19:01.267887 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 1164 4/0/0 AAAA 2a01:7c8:aaaa:4ac::1, AAAA 2a01:7c8:c020:3::191, AAAA 2a00:d880:6:108::6bd5, AAAA 2a03:b0c0:2:d0::16:7001 (144)
21:19:01.268034 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 180
21:19:06.140965 IP 172.16.1.11.38477 > google-public-dns-a.google.com.domain: 45651+ A? 2.pool.ntp.org. (32)
21:19:06.141034 IP 172.16.1.11.38477 > google-public-dns-a.google.com.domain: 59427+ AAAA? 2.pool.ntp.org. (32)
21:19:06.186833 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 45651 4/0/0 A 93.94.224.67, A 5.200.6.34, A 83.247.2.80, A 87.253.152.191 (96)
21:19:06.186984 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:19:06.189558 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 59427 4/0/0 AAAA 2a01:788:a000:0:91:148:192:49, AAAA 2a00:f80:6::66, AAAA 2a00:f10:103:201:ba27:ebff:fefd:984, AAAA 2001:7b8:633:1:213:154:236:182 (144)
21:19:06.189703 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 180
21:19:11.146225 IP 172.16.1.11.38477 > google-public-dns-a.google.com.domain: 45651+ A? 2.pool.ntp.org. (32)
21:19:11.146327 IP 172.16.1.11.38477 > google-public-dns-a.google.com.domain: 59427+ AAAA? 2.pool.ntp.org. (32)
21:19:11.162838 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 59427 4/0/0 AAAA 2001:7b8:3:32:213:136:0:252, AAAA 2001:7b8:633:1:213:154:236:182, AAAA 2001:888:2000:d::a6, AAAA 2001:418:3ff::1:53 (144)
21:19:11.163042 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 180
21:19:11.163110 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 45651 4/0/0 A 80.69.83.153, A 81.4.121.228, A 93.94.224.67, A 213.109.127.195 (96)
21:19:11.163298 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:19:16.151412 IP 172.16.1.11.53943 > google-public-dns-a.google.com.domain: 28292+ A? 3.pool.ntp.org. (32)
21:19:16.151477 IP 172.16.1.11.53943 > google-public-dns-a.google.com.domain: 7917+ AAAA? 3.pool.ntp.org. (32)
21:19:16.156593 ARP, Request who-has 172.16.1.1 tell 172.16.1.11, length 28
21:19:16.156615 ARP, Reply 172.16.1.1 is-at 52:54:00:99:3c:f5 (oui Unknown), length 28
21:19:16.224505 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 7917 0/1/0 (96)
21:19:16.224616 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
21:19:16.270924 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 28292 4/0/0 A 87.195.109.207, A 87.195.109.220, A 83.98.201.134, A 5.39.184.5 (96)
21:19:16.271073 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132
^C
109 packets captured
109 packets received by filter
0 packets dropped by kernel

Below is the output of the same interface with DHCP configured. (Strange note: After changing the configuration, I have to start the VM twice for the changes to fully take place!)

$ sudo tcpdump -i virbr1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on virbr1, link-type EN10MB (Ethernet), capture size 65535 bytes
21:25:37.940827 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:fe:b3:c1 (oui Unknown), length 281
21:25:37.940960 IP 172.16.1.1.bootps > 172.16.1.124.bootpc: BOOTP/DHCP, Reply, length 300
21:25:37.941381 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
21:25:37.941425 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:fe:b3:c1 (oui Unknown), length 293
21:25:37.945643 IP 172.16.1.1.bootps > 172.16.1.124.bootpc: BOOTP/DHCP, Reply, length 306
21:25:38.112375 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
21:25:38.225192 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:fe:b3:c1 (oui Unknown), length 281
21:25:38.225391 IP 172.16.1.1.bootps > 172.16.1.124.bootpc: BOOTP/DHCP, Reply, length 300
21:25:38.225484 ARP, Request who-has 172.16.1.1 tell 172.16.1.124, length 28
21:25:38.225507 ARP, Reply 172.16.1.1 is-at 52:54:00:99:3c:f5 (oui Unknown), length 28
21:25:38.225558 IP 172.16.1.124 > 172.16.1.1: ICMP 172.16.1.124 udp port bootpc unreachable, length 336
21:25:38.236452 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:fe:b3:c1 (oui Unknown), length 293
21:25:38.240124 IP 172.16.1.1.bootps > 172.16.1.124.bootpc: BOOTP/DHCP, Reply, length 306
21:25:38.240192 IP 172.16.1.124 > 172.16.1.1: ICMP 172.16.1.124 udp port bootpc unreachable, length 342
21:25:38.268821 IP 172.16.1.124.40262 > 172.16.1.1.domain: 38554+ A? coreos-node1. (30)
21:25:38.268872 IP 172.16.1.124.40262 > 172.16.1.1.domain: 3387+ AAAA? coreos-node1. (30)
21:25:38.268907 IP 172.16.1.1.domain > 172.16.1.124.40262: 38554* 1/0/0 A 172.16.1.124 (46)
21:25:38.270460 IP 172.16.1.1.domain > 172.16.1.124.40262: 3387* 0/0/0 (30)
21:25:38.338394 IP6 :: > ff02::1:fffe:b3c1: ICMP6, neighbor solicitation, who has fe80::5054:ff:fefe:b3c1, length 24
21:25:38.352445 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [S], seq 3706930176, win 29200, options [mss 1460,sackOK,TS val 4294670279 ecr 0,nop,wscale 7], length 0
21:25:38.545962 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [S.], seq 1557898911, ack 3706930177, win 14480, options [mss 1460,sackOK,TS val 306719658 ecr 4294670279,nop,wscale 8], length 0
21:25:38.546146 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 1, win 229, options [nop,nop,TS val 4294670472 ecr 306719658], length 0
21:25:38.546858 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 1:144, ack 1, win 229, options [nop,nop,TS val 4294670473 ecr 306719658], length 143
21:25:38.730603 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [.], ack 144, win 61, options [nop,nop,TS val 306719706 ecr 4294670473], length 0
21:25:38.731128 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [.], seq 1:1403, ack 144, win 61, options [nop,nop,TS val 306719706 ecr 4294670473], length 1402
21:25:38.731294 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 1403, win 251, options [nop,nop,TS val 4294670657 ecr 306719706], length 0
21:25:38.731558 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [.], seq 1403:2805, ack 144, win 61, options [nop,nop,TS val 306719706 ecr 4294670473], length 1402
21:25:38.731673 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 2805, win 274, options [nop,nop,TS val 4294670658 ecr 306719706], length 0
21:25:38.732324 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [P.], seq 2805:3900, ack 144, win 61, options [nop,nop,TS val 306719706 ecr 4294670473], length 1095
21:25:38.732445 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 3900, win 296, options [nop,nop,TS val 4294670659 ecr 306719706], length 0
21:25:38.736095 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 144:411, ack 3900, win 296, options [nop,nop,TS val 4294670662 ecr 306719706], length 267
21:25:38.736216 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 411:417, ack 3900, win 296, options [nop,nop,TS val 4294670662 ecr 306719706], length 6
21:25:38.736244 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 417:458, ack 3900, win 296, options [nop,nop,TS val 4294670662 ecr 306719706], length 41
21:25:38.901471 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [.], ack 458, win 65, options [nop,nop,TS val 306719754 ecr 4294670662], length 0
21:25:38.901731 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [P.], seq 3900:3947, ack 458, win 65, options [nop,nop,TS val 306719754 ecr 4294670662], length 47
21:25:38.902054 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 458:756, ack 3947, win 296, options [nop,nop,TS val 4294670828 ecr 306719754], length 298
21:25:39.107577 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [P.], seq 3947:4413, ack 756, win 70, options [nop,nop,TS val 306719806 ecr 4294670828], length 466
21:25:39.108487 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 756:1031, ack 4413, win 318, options [nop,nop,TS val 4294671035 ecr 306719806], length 275
21:25:39.269901 IP 172.16.1.124.52884 > 172.16.1.1.domain: 62854+ A? coreos-node1. (30)
21:25:39.269935 IP 172.16.1.124.52884 > 172.16.1.1.domain: 8211+ AAAA? coreos-node1. (30)
21:25:39.269990 IP 172.16.1.1.domain > 172.16.1.124.52884: 62854* 1/0/0 A 172.16.1.124 (46)
21:25:39.271790 IP 172.16.1.1.domain > 172.16.1.124.52884: 8211* 0/0/0 (30)
21:25:39.310673 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [P.], seq 4413:4755, ack 1031, win 74, options [nop,nop,TS val 306719857 ecr 4294671035], length 342
21:25:39.311662 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 1031:1232, ack 4755, win 340, options [nop,nop,TS val 4294671238 ecr 306719857], length 201
21:25:39.341636 IP6 fe80::5054:ff:fefe:b3c1 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
21:25:39.341680 IP6 fe80::5054:ff:fefe:b3c1 > ip6-allrouters: ICMP6, router solicitation, length 16
21:25:39.475291 IP ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https > 172.16.1.124.50773: Flags [P.], seq 4755:5352, ack 1232, win 78, options [nop,nop,TS val 306719898 ecr 4294671238], length 597
21:25:39.483467 IP6 fe80::5054:ff:fefe:b3c1 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
21:25:39.515591 IP 172.16.1.124.50773 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [.], ack 5352, win 361, options [nop,nop,TS val 4294671442 ecr 306719898], length 0
21:25:39.923865 IP 172.16.1.124.51797 > 172.16.1.1.domain: 17873+ A? 1.pool.ntp.org. (32)
21:25:39.923949 IP 172.16.1.124.51797 > 172.16.1.1.domain: 14393+ AAAA? 1.pool.ntp.org. (32)
21:25:39.924063 IP 172.16.1.1.domain > 172.16.1.124.51797: 17873 4/0/0 A 193.227.197.2, A 91.82.85.61, A 178.32.44.208, A 199.7.177.206 (96)
21:25:39.924118 IP 172.16.1.1.domain > 172.16.1.124.51797: 14393 0/0/0 (32)
21:25:39.925167 IP 172.16.1.124.57528 > 172.16.1.1.domain: 63467+ A? 2.pool.ntp.org. (32)
21:25:39.925221 IP 172.16.1.124.57528 > 172.16.1.1.domain: 42676+ AAAA? 2.pool.ntp.org. (32)
21:25:39.925312 IP 172.16.1.1.domain > 172.16.1.124.57528: 63467 4/0/0 A 87.195.109.207, A 87.195.109.220, A 80.69.83.153, A 81.171.44.131 (96)
21:25:39.925365 IP 172.16.1.1.domain > 172.16.1.124.57528: 42676 4/0/0 AAAA 2a00:f80:6::66, AAAA 2a00:d880:3:1::f69c:7f41, AAAA 2001:418:3ff::1:53, AAAA 2a00:1ca8:e:4::b3b6:1d74 (144)
21:25:39.926449 IP 172.16.1.124.45226 > 172.16.1.1.domain: 40226+ A? 3.pool.ntp.org. (32)
21:25:39.926593 IP 172.16.1.1.domain > 172.16.1.124.45226: 40226 4/0/0 A 195.191.112.251, A 178.21.23.127, A 213.109.127.82, A 129.250.35.250 (96)
21:25:39.927024 IP 172.16.1.124.45226 > 172.16.1.1.domain: 43171+ AAAA? 3.pool.ntp.org. (32)
21:25:39.927150 IP 172.16.1.1.domain > 172.16.1.124.45226: 43171 0/0/0 (32)
21:25:41.910790 IP 172.16.1.124.ntp > ntp.monshouwer.eu.ntp: NTPv4, Client, length 48
21:25:41.910905 IP 172.16.1.124.ntp > ntp.raqxs.nl.ntp: NTPv4, Client, length 48
21:25:41.910934 IP 172.16.1.124.ntp > ntp4.bit.nl.ntp: NTPv4, Client, length 48
21:25:41.935511 IP ntp.raqxs.nl.ntp > 172.16.1.124.ntp: NTPv4, Server, length 48
21:25:41.935803 IP ntp4.bit.nl.ntp > 172.16.1.124.ntp: NTPv4, Server, length 48
21:25:41.936099 IP ntp.monshouwer.eu.ntp > 172.16.1.124.ntp: NTPv4, Server, length 48
21:25:42.475446 IP6 fe80::5054:ff:fefe:b3c1 > ip6-allrouters: ICMP6, router solicitation, length 16
21:25:43.353498 IP6 fe80::5054:ff:fefe:b3c1 > ip6-allrouters: ICMP6, router solicitation, length 16
21:25:43.910812 IP 172.16.1.124.ntp > yikes.bl2.tolna.net.ntp: NTPv4, Client, length 48
21:25:43.963807 IP yikes.bl2.tolna.net.ntp > 172.16.1.124.ntp: NTPv4, Server, length 48
21:25:46.725670 IP6 fe80::5054:ff:fefe:b3c1 > ip6-allrouters: ICMP6, router solicitation, length 16
21:25:46.936599 ARP, Request who-has 172.16.1.124 tell 172.16.1.1, length 28
21:25:46.936670 ARP, Reply 172.16.1.124 is-at 52:54:00:fe:b3:c1 (oui Unknown), length 28
21:25:47.361527 IP6 fe80::5054:ff:fefe:b3c1 > ip6-allrouters: ICMP6, router solicitation, length 16
21:25:50.975912 IP6 fe80::5054:ff:fefe:b3c1.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
21:25:52.100088 IP6 fe80::5054:ff:fefe:b3c1.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
21:25:54.433734 IP6 fe80::5054:ff:fefe:b3c1.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
^C
78 packets captured
79 packets received by filter
0 packets dropped by kernel

Hope this helps.

Morten

[Seán C. McCord]

> 21:18:46.120061 IP 172.16.1.11.40736 > google-public-dns-a.google.com.domain: 14032+ A? 1.pool.ntp.org. (32)

> 21:18:46.120139 IP 172.16.1.11.40736 > google-public-dns-a.google.com.domain: 62628+ AAAA? 1.pool.ntp.org. (32)

> 21:18:46.161836 IP google-public-dns-a.google.com.domain > 172.16.1.11.37424: 62628 0/1/0 (96)

> 21:18:46.161935 IP 172.16.1.11 > google-public-dns-a.google.com: ICMP 172.16.1.11 udp port 37424 unreachable, length 132

So that's your problem:  the NAT is getting screwed up.  The responses are being directed to a different port than the requests.  The CoreOS box receives the replies, but ignores them, because that are coming back on the wrong port.

I've always hated the default networking setup of qemu and libvirt; hence, I've never used it and can't comment on it.  There's also the issue that Amazon does NATing, as well.  

<rant>Hey Amazon, are you ever going to get IPv6?  DigitalOcean finally has it. </rant>

Anyway, would you care to post your 'iptables -v -t nat -L' output for both cases?

--

You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Morten Minke]

Hi Seán,

Thank you so much for your help up until now. It is really appreciated. Below you can find the output of the iptables -v -t nat -L on the host. Please remember that I have to 'networks' configured for libvirt because I thought it would be best to separate the network stuff while experimenting. That is why there is 172.16 and 192.168 entries in the table. I also have docker installed which created the 172.17 network segment and apparently also a NAT entry.

I have no idea (yet) what the  base-address.mcast.net/24 means, but it seems that that rule captures all the trafic.

Boot with fixed IP address

$ sudo iptables -v -t nat -L

Chain PREROUTING (policy ACCEPT 16481 packets, 1243K bytes)

 pkts bytes target     prot opt in     out     source               destination         

  121  8283 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 4049 packets, 563K bytes)

 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 15524 packets, 1084K bytes)

 pkts bytes target     prot opt in     out     source               destination         

    0     0 DOCKER     all  --  any    any     anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 15497 packets, 1082K bytes)

 pkts bytes target     prot opt in     out     source               destination         

   28  2261 MASQUERADE  all  --  any    any     172.17.0.0/16       !172.17.0.0/16       

   28  2260 RETURN     all  --  any    any     172.16.1.0/24        base-address.mcast.net/24 

    0     0 RETURN     all  --  any    any     172.16.1.0/24        255.255.255.255     

    1    60 MASQUERADE  tcp  --  any    any     172.16.1.0/24       !172.16.1.0/24        masq ports: 1024-65535

   15   965 MASQUERADE  udp  --  any    any     172.16.1.0/24       !172.16.1.0/24        masq ports: 1024-65535

    0     0 MASQUERADE  all  --  any    any     172.16.1.0/24       !172.16.1.0/24       

   33  2479 RETURN     all  --  any    any     192.168.122.0/24     base-address.mcast.net/24 

    0     0 RETURN     all  --  any    any     192.168.122.0/24     255.255.255.255     

    0     0 MASQUERADE  tcp  --  any    any     192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535

    0     0 MASQUERADE  udp  --  any    any     192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535

    0     0 MASQUERADE  all  --  any    any     192.168.122.0/24    !192.168.122.0/24    

Chain DOCKER (2 references)

 pkts bytes target     prot opt in     out     source               destination              

Based on the differences in the configuration, I have put the DNS for coreos to my libvirt host 172.16.1.1 (as was also given by the DHCP configuration) . With that configuration, DNS works, but I see other traffic failing in logfiles. For some reason libvirt is messing up the NAT as you mentioned. However, what a good configuration should look like is a big question to me.

Morten

21:18:35.015161 IP 172.16.1.11.40761 > ec2-54-183-31-94.us-west-1.compute.amazonaws.com.https: Flags [P.], seq 144:411, ack 3900, win 296, options [<span

...