TTL expired in transit to qemu virtual machine.

[Mimiko]

Hello.

I've setup qemu/kvm and installed several virtual machines. Access and ping to some virtuals are ok, but one have a stable problem not receiving
correctly packets. First, this is the environment:

>uname -a
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.84-1 x86_64 GNU/Linux

>libvirtd --version
libvirtd (libvirt) 0.9.12.3

>cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eth0 eth1
bond-mode balance-alb
bond-miimon 100
bond-downdelay 200
bond-updelay 200

auto br0
iface br0 inet static
address 10.10.10.10
netmask 255.255.0.0
vlan-raw-device bond0
bridge_ports bond0
bridge_stp off
bridge_fd 0
bridge_ageing 0
bridge_maxwait 0

Virtual machines connects to LAN using bridge:
>virt-install .... --network=bridge:br0,model=virtio ....

All virtuals has networking configuret like that. Also in iptables is an entry to allow access to virtuals:

>iptables -L FORWARD -nv
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
X X ACCEPT all -- br0 br0 0.0.0.0/0 0.0.0.0/0

Most virtuals does not have networking problems, but some times they can't be reached. For now only one virtual machines have this problem:
From windows machine ping virtual machine:

>ping 10.10.10.3

Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.
Reply from 10.10.10.10: TTL expired in transit.

>tracert -d 10.10.10.3

Tracing route to 10.10.10.3 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.10.10.10
2 <1 ms <1 ms <1 ms 10.10.10.10
3 <1 ms <1 ms * 10.10.10.10
4 1 ms <1 ms <1 ms 10.10.10.10
5 <1 ms <1 ms * 10.10.10.10

So packet goes round on interfaces of server hosting virtuals.

Virtuals are linux different flavour and one windows. This problem may occur on any of this virtuals.

I've observed that for this particular virtual, which have problem, the arp table of host assigned self mac to the virtual's IP, not the mac
configured for virtual machine.

What could be the problem?

--
Mimiko desu.

[Dominik George]

Hi,

> >iptables -L FORWARD -nv
>Chain FORWARD (policy DROP 0 packets, 0 bytes)
>pkts bytes target prot opt in out source
>destination
>X X ACCEPT all -- br0 br0 0.0.0.0/0
>0.0.0.0/0

What is that supposed to do?

Forwarding on the IP layer, by definition, happens between different interfaces.

Although this rule is most likely a no-op, you should remove it to reduce confusion.

-nik

[Igor Cicimov]

Hi Mimiko,

On Fri, Mar 17, 2017 at 9:58 PM, Mimiko <vbv...@gmail.com> wrote:

Hello.

I've setup qemu/kvm and installed several virtual machines. Access and ping to some virtuals are ok, but one have a stable problem not receiving correctly packets. First, this is the environment:

>uname -a
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.84-1 x86_64 GNU/Linux

That's an really old kernel, I don't start anything virtual these days without at least 3.13.x kernel.

Hmmm, this doesn't make much sense to me, more specifically this part:

        vlan-raw-device bond0
        bridge_ports bond0

Whats the purpose exactly of the vlan? Usually, and that is how I do it, you would split the VLAN's coming from the switch trunk port over the bond and attach them to separate bridges lets say:

# VLAN51

auto br-vlan51

iface br-vlan51 inet manual

    vlan-raw-device bond0 

    bridge_ports bond0.51

    bridge_stp off

    bridge_maxwait 0

    bridge_fd 0

# VLAN52

auto br-vlan52

iface br-vlan52 inet manual

    vlan-raw-device bond0 

    bridge_ports bond0.52

    bridge_stp off

    bridge_maxwait 0

    bridge_fd 0

If the intention was to pass through the tagged traffic to the VM's then vlan-raw-device part is not needed at all.

Yep typical routing loop.

  

Virtuals are linux different flavour and one windows. This problem may occur on any of this virtuals.

I've observed that for this particular virtual, which have problem, the arp table of host assigned self mac to the virtual's IP, not the mac configured for virtual machine.

That's strange indeed, except if br0 is used by something else like libvrit network that sets up the interface for proxy-arp. What's the output of:

# brctl showmacs br0

# ip route show

# arp -n

on the host, and:

# ip link show

# ip route show

# arp -n

on the problematic vm and on one of the good vm's?

To find the loop I would start by doing ping between good and bad vm (both directions in turns) and check the traffic on the host interface:

# tcpdump -ennqt -i br0 \( arp or icmp \)

and corresponding network devices on both vm's too.

 

What could be the problem?

Any sysctl settings you might have changed on the host?

 

--
Mimiko desu.

[Mimiko]

This rule allows traffic to virtuals. Without this rule virtuals does not have access to LAN. Yes, I know that bridge setup does not need iptables,
but reality is this.


--
Mimiko desu.

[Mimiko]

On 18.03.2017 07:22, Igor Cicimov wrote:
> >uname -a
> Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.84-1 x86_64 GNU/Linux
>
> That's an really old kernel, I don't start anything virtual these days without at least 3.13.x kernel.

I regularly do apt-get upgrade, but not to next Debian version. So, how this kernel be old for Debian 7?

> Hmmm, this doesn't make much sense to me, more specifically this part:
>
> vlan-raw-device bond0
> bridge_ports bond0
>
> Whats the purpose exactly of the vlan? Usually, and that is how I do it, you would split the VLAN's coming from the switch trunk port over the bond
> and attach them to separate bridges lets say:

Don't mind that line. I putted it as a reminder to continue extend networking in using vlans. For now there is no vlan used.

> That's strange indeed, except if br0 is used by something else like libvrit network that sets up the interface for proxy-arp. What's the output of:
>
> # brctl showmacs br0

port no mac addr is local? ageing timer
1 bridge mac yes 0.00
8 fe:54:00:13:76:5b yes 0.00
5 fe:54:00:44:ac:39 yes 0.00
6 fe:54:00:a8:35:fb yes 0.00
2 fe:54:00:c2:64:2d yes 0.00
4 fe:54:00:d5:17:fb yes 0.00
3 fe:54:00:de:83:2d yes 0.00


> # ip route show

10.0.0.0/8 via 10.10.10.254 dev br0
10.10.0.0/16 dev br0 proto kernel scope link src 10.10.10.10


> # arp -n

gives a long list for local and virtual ip/mac

>
> on the host, and:
>
> # ip link show

I can't connect using ssh, only vnc.
This shows lo and eth0 with corresponding mac and is UP

> # ip route show
10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.10.3
169.254.0.0/16 dev eth0 scope link metric 1002
default via 10.10.10.254 dev eth0

> # arp -n

gives a long list for local and virtual ip/mac

>
> on the problematic vm and on one of the good vm's?

Thouse 3 commands shows the same (except ip and mac) on both vm's.

>
> To find the loop I would start by doing ping between good and bad vm (both directions in turns) and check the traffic on the host interface:
>
> # tcpdump -ennqt -i br0 \( arp or icmp \)
>
> and corresponding network devices on both vm's too.

This gives me lot of output regarding arp and icmp on LAN.

> Any sysctl settings you might have changed on the host?

net.ipv4.ip_forward=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2


--
Mimiko desu.

[davi...@freevolt.org]

On Mon, 20 Mar 2017, Mimiko wrote:
> On 18.03.2017 07:22, Igor Cicimov wrote:
>> >uname -a
>> Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.84-1 x86_64 GNU/Linux
>>
>> That's an really old kernel, I don't start anything virtual these days
>> without at least 3.13.x kernel.
>
> I regularly do apt-get upgrade, but not to next Debian version. So, how this
> kernel be old for Debian 7?

TLDR:

1. I have no clues to offer regarding OP's virtualisation/networking
troubles, but I wish them the best of fortune.

2. OP's kernel version does not look quite current to me, even for
wheezy (debian 7) and it might be a matter of concern (though I claim
no particular expertise).

Longer version:

It is not clear to me whether your question

> I regularly do apt-get upgrade, but not to next Debian version. So, how this
> kernel be old for Debian 7?

is a request for information, or merely rhetorical (ie, an assertion
that your kernel is in fact current relative to debian 7).

In case you were asking, I'm still using debian 7 too,

$ cat /etc/debian_version
7.11

and, for what it's worth, in case it interests you

$ uname -a
Linux leviathan 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686 GNU/Linux

I notice the version you posted here (3.2.84-1) is slightly older than
mine. If like myself you plan on running the 3.2.0-4-* release for a
while, It might[1] be worth upgrading at least to version 3.2.86-1:

https://security-tracker.debian.org/tracker/CVE-2016-5195

There is also a newer kernel release available, for those who choose
to install it explicitly[3], from the wheezy-backports repository[2],
(see the 3.16.0-0.bpo releases near the end):

$ apt-cache search '^linux-image-*'
linux-image-3.2.0-4-486 - Linux 3.2 for older PCs
linux-image-3.2.0-4-686-pae - Linux 3.2 for modern PCs
linux-image-3.2.0-4-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-686-pae
linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs
linux-image-3.2.0-4-rt-686-pae - Linux 3.2 for modern PCs, PREEMPT_RT
linux-image-3.2.0-4-rt-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-rt-686-pae
linux-image-2.6-486 - Linux for older PCs (dummy package)
linux-image-2.6-686 - Linux for modern PCs (dummy package)
linux-image-2.6-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-2.6-686-pae - Linux for modern PCs (dummy package)
linux-image-2.6-amd64 - Linux for 64-bit PCs (dummy package)
linux-image-486 - Linux for older PCs (meta-package)
linux-image-686 - Linux for modern PCs (dummy package)
linux-image-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-686-pae - Linux for modern PCs (meta-package)
linux-image-amd64 - Linux for 64-bit PCs (meta-package)
linux-image-rt-686-pae - Linux for modern PCs (meta-package), PREEMPT_RT
linux-image-3.16.0-0.bpo.4-586 - Linux 3.16 for older PCs
linux-image-3.16.0-0.bpo.4-686-pae - Linux 3.16 for modern PCs
linux-image-3.16.0-0.bpo.4-686-pae-dbg - Debugging symbols for Linux 3.16.0-0.bpo.4-686-pae
linux-image-3.16.0-0.bpo.4-amd64 - Linux 3.16 for 64-bit PCs
linux-image-586 - Linux for older PCs (meta-package)
linux-image-686-pae-dbg - Debugging symbols for Linux 686-pae configuration (meta-package)

But regarding backports, do see https://backports.debian.org/FAQ/

| Q: Is there security support for packages from backports.debian.org?
|
| A: Unfortunately not. This is done on a best effort basis by the
| people who track the package, usually the ones who originally did
| upload the package into backports. When security related bugs are
| fixed in Debian unstable the backporter is permitted to upload the
| package from directly there instead of having to wait until the fix
| hits testing. You can see the open issues for wheezy-backports in
| the security tracker (though there may be false positives too, the
| version compare isn't perfect yet)

[1] The security-tracker page for CVE-2016-5195 lists version 78-1 as
'vulnerable', and 86-1 as 'fixed'. It is unclear to me what this
is supposed to say about OP's kernel, which is apparently at 84-1.

[2] https://backports.debian.org/Instructions/#index2h2
[3] https://backports.debian.org/Instructions/#index3h2

[Mimiko]

On 21.03.2017 02:25, davi...@freevolt.org wrote:
> It is not clear to me whether your question
>
>> I regularly do apt-get upgrade, but not to next Debian version. So, how this kernel be old for Debian 7?
>
> is a request for information, or merely rhetorical (ie, an assertion
> that your kernel is in fact current relative to debian 7).
>
> In case you were asking, I'm still using debian 7 too,
>
> $ cat /etc/debian_version
> 7.11
>
> and, for what it's worth, in case it interests you
>
> $ uname -a
> Linux leviathan 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686 GNU/Linux
>
> I notice the version you posted here (3.2.84-1) is slightly older than
> mine. If like myself you plan on running the 3.2.0-4-* release for a
> while, It might[1] be worth upgrading at least to version 3.2.86-1:

Well, doing regulat apt-get update && apt-get upgrade, I fought that kernel is also upgraded. I've seen this several times. How comes it wasn't
updated to 3.2.86-1.

As for apt-cache search '^linux-image-*' :

linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs

linux-image-3.2.0-4-amd64-dbg - Debugging symbols for Linux 3.2.0-4-amd64
linux-image-3.2.0-4-rt-amd64 - Linux 3.2 for 64-bit PCs, PREEMPT_RT
linux-image-3.2.0-4-rt-amd64-dbg - Debugging symbols for Linux 3.2.0-4-rt-amd64

linux-image-2.6-amd64 - Linux for 64-bit PCs (dummy package)

linux-image-amd64 - Linux for 64-bit PCs (meta-package)

linux-image-rt-amd64 - Linux for 64-bit PCs (meta-package), PREEMPT_RT

Then doing apt-get install linux-image-amd64 says it already latest version.

Anyway, this slightly different (minor) version difference could be the problem?

--
Mimiko desu.

[Greg Wooledge]

On Thu, Mar 23, 2017 at 08:36:38AM +0200, Mimiko wrote:
> Well, doing regulat apt-get update && apt-get upgrade, I fought that kernel
> is also upgraded. I've seen this several times. How comes it wasn't updated
> to 3.2.86-1.

Run "uname -a" (and also "uptime") and post the results. After a kernel
package is upgraded, you do have to reboot to run the new kernel, and
uptime will tell us how long ago you did that.

> As for apt-cache search '^linux-image-*' :
> linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs

[...]

> Then doing apt-get install linux-image-amd64 says it already latest version.

It's important to note that the kernel package name is *not* the same
as the exact kernel version. That's why you want to run "uname -a".
It will show both of them.

The kernel package name 3.2.0-4-amd64 refers to the ABI (module
compatibility) of the kernel. A module built for any kernel with this
package name will work with any other version, so you don't have to
rebuild all of your third party kernel modules every time you upgrade
the kernel.

[Mimiko]

On 23.03.2017 14:14, Greg Wooledge wrote:
> Run "uname -a" (and also "uptime") and post the results.

You a right. I didn't do a restart after last kernel update.

After a restart, TTL problem didn't occur anymore.

Also I've noted that the particular vm had INPUT default policy to DROP and no icmp accepted. Strange that some time I did get an answer on ping.
Accepting icmp requests gives me stable replies.

Thank you all for answering.

--
Mimiko desu.