Troubleshooting an AppVM that suddenly can't connect to the network

[Andrew Todd]

I'm having a problem that I'm very confused about.

I have several AppVMs based on the same fedora-30 TemplateVM. Some of them are completely cut off from the Internet, some have firewall restrictions, some connect through sys-firewall but are not restricted.

Today, one specific AppVM has started refusing to connect to anything. Even if I try to connect by IP address, it claims it can't route. Example:

$ ssh us...@192.168.34.22
ssh: connect to host 192.168.34.22 port 22: No route to host
$ ip r
default via 10.137.0.6 dev eth0
10.137.0.6 dev eth0 scope link

I've gone through the few settings and error logs that I can think of, but nothing seems to be unusual about this particular AppVM. I've checked qvm-firewall and the rules should be allowing appropriate traffic to pass. I have not changed any settings. I have tried rebooting the system once or twice as well, with no effect.

It seems like I can get to sys-firewall across the link-local connection, but that's all. After that, nothing seems to work. Every other AppVM I've tried is working fine. Where should I be looking next? Thank you.

[unman]

Look at what's happening at sys-net - First at the vif interface leading
to sys-firewall, then at eth0.
If you're not familiar with traffic sniffing, then you can use counters in
iptables or nftables.
iptables -L -nvZ will zero counters, try to connect, then same command
will show counters of traffic. You shuld see increase on the FORWARD
chain.

You can insert specific rules targeting ssh traffic if you will.
What template are you using for sys-net and sys-firewall?

[Andrew Todd]

Thanks, you are right that I don't have any experience with packet sniffing, however, after making sure that nothing else was using the network and running iptables -L -nvZ on both sys-net and sys-firewall... there's no packets at all getting there from this AppVM. Which doesn't make any sense to me, the default route is definitely there:

$ ip r
default via 10.137.0.6 dev eth0
10.137.0.6 dev eth0 scope link

and in qvm-firewall both DNS and ICMP are allowed, yet neither is working. Is there more debugging I can do on the AppVM itself?

Both sys-net and sys-firewall are based on the fedora-30 template. There is only one AppVM affected, all others based on the same template are working fine. In fact, I copied the complete contents of the bad AppVM's home directory to a new AppVM, and it's also having no problems connecting to the network. I would like to find the root cause, but that's my workaround for now. Thanks for any more advice you can offer.