Update Killed All Non-Tor Internet Access!

[Axon]

After updating dom0 and all my templatevms yesterday and rebooting, I
can now no longer access the internet except through VMs which have my
torvm as their netvm. (Of course, my netvm itself can connect to the
internet, which I verified with the terminal.)

I'm pretty sure it was the update, since everything was working fine
right before rebooting to apply the update. I didn't change anything else.

I tried creating a new, plain firewallvm to connect through. No joy.

I'm sure I'm not the only one experiencing this, as my setup is pretty
standard. Any idea what could be causing this?

[Axon]

My templatevm's yum.log shows that the following were updated (note that
this is just a vanilla Qubes template; I've never installed anything
there myself):

Aug 15 03:37:22 Updated: 1:NetworkManager-glib-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:23 Updated: selinux-policy-3.11.1-100.fc18.noarch
Aug 15 03:37:23 Updated: gstreamer1-1.0.9-1.fc18.x86_64
Aug 15 03:37:24 Updated: gstreamer1-plugins-base-1.0.9-1.fc18.x86_64
Aug 15 03:37:24 Updated: libnm-gtk-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:25 Updated: 2:libwbclient-4.0.8-1.fc18.x86_64
Aug 15 03:37:26 Updated: 2:samba-libs-4.0.8-1.fc18.x86_64
Aug 15 03:37:27 Updated: 2:samba-common-4.0.8-1.fc18.x86_64
Aug 15 03:37:27 Updated: 2:libsmbclient-4.0.8-1.fc18.x86_64
Aug 15 03:37:28 Updated: nm-connection-editor-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:29 Updated: 1:NetworkManager-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:30 Updated: hplip-common-3.13.7-1.fc18.x86_64
Aug 15 03:37:31 Updated: hplip-libs-3.13.7-1.fc18.x86_64
Aug 15 03:37:32 Updated: 1:hpijs-3.13.7-1.fc18.x86_64
Aug 15 03:37:35 Updated: xulrunner-23.0-2.fc18.x86_64
Aug 15 03:37:36 Updated: paps-libs-0.6.8-26.fc18.x86_64
Aug 15 03:37:37 Updated: paps-0.6.8-26.fc18.x86_64
Aug 15 03:37:40 Updated: firefox-23.0-1.fc18.x86_64
Aug 15 03:37:41 Updated: hplip-3.13.7-1.fc18.x86_64
Aug 15 03:37:41 Updated: libsane-hpaio-3.13.7-1.fc18.x86_64
Aug 15 03:37:42 Updated: network-manager-applet-0.9.8.2-1.fc18.x86_64
Aug 15 03:37:42 Updated: 2:samba-client-4.0.8-1.fc18.x86_64
Aug 15 03:37:43 Updated: gstreamer1-plugins-good-1.0.9-1.fc18.x86_64
Aug 15 03:37:43 Updated: gstreamer1-plugins-bad-free-1.0.9-1.fc18.x86_64
Aug 15 03:37:44 Updated: selinux-policy-targeted-3.11.1-100.fc18.noarch
Aug 15 03:37:45 Updated: selinux-policy-devel-3.11.1-100.fc18.noarch
Aug 15 03:37:46 Updated: selinux-policy-doc-3.11.1-100.fc18.noarch
Aug 15 03:37:47 Updated: binutils-2.23.51.0.1-10.fc18.x86_64
Aug 15 03:37:48 Updated: gnupg-1.4.14-1.fc18.x86_64
Aug 15 03:37:48 Updated: boost-date-time-1.50.0-7.fc18.x86_64
Aug 15 03:37:53 Updated: thunderbird-17.0.8-1.fc18.x86_64


And dom0's yum.log shows the following were updated yesterday:

2:libwbclient-4.0.8-1.fc18.x86_64
2:samba-libs-4.0.8-1.fc18.x86_64
2:samba-common-4.0.8-1.fc18.x86_64
system-config-keyboard-base-1.3.1-14.fc18.x86_64
system-config-keyboard-1.3.1-14.fc18.x86_64
2:libsmbclient-4.0.8-1.fc18.x86_64
boost-program-options-1.50.0-7.fc18.x86_64
perf-3.10.6-100.fc18.x86_64
binutils-2.23.51.0.1-10.fc18.x86_64
gstreamer1-1.0.9-1.fc18.x86_64


Could it be the NetworkManager updates in the templatevm? I'm going to
try to figure out if/how I can try rolling those back.

[Joanna Rutkowska]

Did you try to reboot the whole system? Perhaps it was some restarted
VMs not reconnecting to the net/firewallvms (issue #722)?

j.

[Zrubecz Laszlo]

On 16 August 2013 12:11, Axon <ax...@openmailbox.org> wrote:

> Could it be the NetworkManager updates in the templatevm? I'm going to try
> to figure out if/how I can try rolling those back.

If you only use one default template, then surely no.

If your can acces any network from your netVM, then your template (and
dom0) is fine.

1. Try to reassign the the netvm for your AppVM first...
2. check the firewall settings (if it's any)
3. dump your traffic on the netVM to see if AppVM traffic is going to
the right direction...

--
Zrubi


--
Zrubi

[Axon]

Yeah, I rebooted several times earlier when I thought it might be a DNS
problem. Changed the DNS server settings in NetworkManager, but that
didn't affect it.

[Axon]

On 08/16/13 04:05, Zrubecz Laszlo wrote:
> On 16 August 2013 12:11, Axon <ax...@openmailbox.org> wrote:
>
>> Could it be the NetworkManager updates in the templatevm? I'm going to try
>> to figure out if/how I can try rolling those back.
>
> If you only use one default template, then surely no.
>

The torvm has its own template, but everything else (including netvm and
firewallvm) uses the default template.

> If your can acces any network from your netVM, then your template (and
> dom0) is fine.
>

Actually, I just discovered that my templatevms can connect to the
update servers over my plain connection. (Will expand on this in another
message.)

> 1. Try to reassign the the netvm for your AppVM first...

Ok, I tried changing some AppVM's NetVM to netvm (instead of the usual
firewallvm). No change.

Also, as said above, I tried creating a new firewallvm (connected to
netvm), and assigning that as the NetVM for some AppVMs, with no change.

> 2. check the firewall settings (if it's any)

Right, no firewall settings. (Anyway, I checked all the AppVMs, some of
which have firewall settings, and some of which don't.)

> 3. dump your traffic on the netVM to see if AppVM traffic is going to
> the right direction...

Running sudo tcpdump in the netvm, it looks like traffic is going in the
right direction. But I'm not sure if I'm reading it correctly, and there
are some other odd things in there that I don't understand. I attached a
partial output. Would you mind taking a look? (For the output, I just
started tcpdump, then tried to connect to qubes-os.org with a dispvm.)

One thing I noticed in the tcpdump output is a lot of lines like this:

06:17:44.271406 IP netvm.38028 > 10.137.2.1.domain: 7512+ A?
qubes-os.org. (30)
06:17:44.271418 IP netvm.38028 > 10.137.2.1.domain: 49126+ AAAA?
qubes-os.org. (30)
06:17:49.273462 IP netvm.53980 > 10.137.2.254.domain: 7512+ A?
qubes-os.org. (30)
06:17:49.273509 IP netvm.53980 > 10.137.2.254.domain: 49126+ AAAA?
qubes-os.org. (30)

Doesn't the "AAAA" stuff have to do with IPv6? I'm pretty sure my
equipment only handles IPv4, but I'm not sure if this is normal.

>
> --
> Zrubi
>
>

[Axon]

I'll start by saying sorry for the long message. I've tried to <snip>
where possible to exclude extraneous things. :(
As I mentioned in a previous message, I tried to see if I can rollback
the updates to NetworkManager:


[user@fedora-18-x64-clone-1 ~]$ sudo yum downgrade NetworkManager*
Loaded plugins: langpacks, post-transaction-actions, presto,
refresh-packagekit, yum-qubes-hooks
No Match for available package:
1:NetworkManager-devel-0.9.7.0-12.git20121004.fc18.i686
No Match for available package:
1:NetworkManager-devel-0.9.7.0-12.git20121004.fc18.x86_64
No Match for available package:
1:NetworkManager-glib-devel-0.9.7.0-12.git20121004.fc18.i686
No Match for available package:
1:NetworkManager-glib-devel-0.9.7.0-12.git20121004.fc18.x86_64
No Match for available package: NetworkManager-l2tp-0.9.6-2.fc18.x86_64
No Match for available package:
NetworkManager-openswan-0.9.3.995-3.git20120302.fc18.x86_64
No Match for available package:
NetworkManager-ssh-0.0.3-0.8.20130419git3d5321b.fc18.x86_64
No Match for available package:
NetworkManager-ssh-gnome-0.0.3-0.8.20130419git3d5321b.fc18.x86_64
No Match for available package:
1:NetworkManager-wimax-0.9.7.0-12.git20121004.fc18.x86_64
Resolving Dependencies
--> Running transaction check
---> Package NetworkManager.x86_64 1:0.9.7.0-12.git20121004.fc18 will be
a downgrade
---> Package NetworkManager.x86_64 1:0.9.8.2-1.fc18 will be erased
---> Package NetworkManager-glib.x86_64 1:0.9.7.0-12.git20121004.fc18
will be a downgrade
---> Package NetworkManager-glib.x86_64 1:0.9.8.2-1.fc18 will be erased
--> Finished Dependency Resolution
Error: Package: nm-connection-editor-0.9.8.2-1.fc18.x86_64 (@updates)
Requires: NetworkManager-glib >= 1:0.9.8.0
Removing: 1:NetworkManager-glib-0.9.8.2-1.fc18.x86_64 (@updates)
NetworkManager-glib = 1:0.9.8.2-1.fc18
Downgraded By:
1:NetworkManager-glib-0.9.7.0-12.git20121004.fc18.x86_64 (fedora)
NetworkManager-glib = 1:0.9.7.0-12.git20121004.fc18
Error: Package: network-manager-applet-0.9.8.2-1.fc18.x86_64 (@updates)
Requires: NetworkManager-glib >= 1:0.9.8.0
Removing: 1:NetworkManager-glib-0.9.8.2-1.fc18.x86_64 (@updates)
NetworkManager-glib = 1:0.9.8.2-1.fc18
Downgraded By:
1:NetworkManager-glib-0.9.7.0-12.git20121004.fc18.x86_64 (fedora)
NetworkManager-glib = 1:0.9.7.0-12.git20121004.fc18
Error: Package: network-manager-applet-0.9.8.2-1.fc18.x86_64 (@updates)
Requires: NetworkManager >= 1:0.9.8.0
Removing: 1:NetworkManager-0.9.8.2-1.fc18.x86_64 (@updates)
NetworkManager = 1:0.9.8.2-1.fc18
Downgraded By:
1:NetworkManager-0.9.7.0-12.git20121004.fc18.x86_64 (fedora)
NetworkManager = 1:0.9.7.0-12.git20121004.fc18
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest


Just for the sake of completeness, I tried both of the suggestions
above, which didn't work.

I then did some more searching and learned that there is a handy way to
undo yum changes in Fedora using yum history.[1][2] First, I ran "sudo
yum history" to find the ID of the update. For me it was 16. Then I ran
sudo yum history undo 16:


[user@fedora-18-x64-clone-1 ~]$ sudo yum history undo 16
Loaded plugins: langpacks, post-transaction-actions, presto,
refresh-packagekit, yum-qubes-hooks
Undoing transaction 16, from Thu Aug 15 03:37:21 2013
Updated NetworkManager-1:0.9.8.1-3.git20130514.fc18.x86_64
@updates
Update 1:0.9.8.2-1.fc18.x86_64
@updates
Updated NetworkManager-glib-1:0.9.8.1-3.git20130514.fc18.x86_64
@updates
Update 1:0.9.8.2-1.fc18.x86_64
@updates

<snip>

Updated network-manager-applet-0.9.8.1-4.git20130514.fc18.x86_64
@updates
Update 0.9.8.2-1.fc18.x86_64
@updates
Updated nm-connection-editor-0.9.8.1-4.git20130514.fc18.x86_64
@updates
Update 0.9.8.2-1.fc18.x86_64
@updates

<snip>

Failed to downgrade: 1:NetworkManager-0.9.8.1-3.git20130514.fc18.x86_64
Failed to downgrade: 1:NetworkManager-glib-0.9.8.1-3.git20130514.fc18.x86_64

<snip>

Failed to downgrade:
network-manager-applet-0.9.8.1-4.git20130514.fc18.x86_64
Failed to downgrade: nm-connection-editor-0.9.8.1-4.git20130514.fc18.x86_64

<snip>

Dependencies Resolved

=======================================================================
Package Arch
Version
Repository Size
=======================================================================
Downgrading:

<snip>

Not available:

<snip>

NetworkManager x86_64
1:0.9.8.1-3.git20130514.fc18
- 0.0

<snip>

nm-connection-editor x86_64
0.9.8.1-4.git20130514.fc18
- 0.0
NetworkManager-glib x86_64
1:0.9.8.1-3.git20130514.fc18
- 0.0

<snip>

Transaction Summary
=========================================================================
Downgrade 3 Packages
Not available 28 Packages

Total download size: 1.4 M
Is this ok [y/N]:


Ok, but the Fedora docs recommend enabling the updates-testing repo[3]
if you're going to try to use yum history undo. So I did that ("sudo
yum-config-manager --enable updates-testing").

And ran "sudo yum downgrade NetworkManager*" again with the same results.

Then I ran "sudo yum history undo 16" again with the same results.

Apparently this guy[4] had a similar problem a couple of years ago and
solved it by downloading the RPMs on another machine and manually
placing them in the yum cache path. So, I figured I would try this too,
but when I search for the packages which yum tells me are missing, I
can't seem to find them. I figure I'm just not looking for them
correctly (or not looking in the right place). Will continue the search.


---

[1]
https://docs.fedoraproject.org/en-US/Fedora/14/html/Software_Management_Guide/ch05s16.html

[2]
https://docs.fedoraproject.org/en-US/Fedora/16/html/System_Administrators_Guide/sec-Yum-Transaction_History.html

[3] https://fedoraproject.org/wiki/QA:Updates_Testing

[Axon]

Ok, so my system is set up as follows:

NetVM: netvm
ProxyVM 1: firewallvm
ProxyVM 2: torvm
TemplateVM 1: fedora-18-x64 (vanilla default)
TemplateVM 1: fedora-18-x64-net (standard Qubes Tor installation)
AppVM 1: untrusted (NetVM: firewallvm)
AppVM 2: untrusted-tor (NetVM: torvm)
[A bunch of other AppVMs which all use firewallvm as their NetVM.]

And I've established the following with respect to connectivity:

netvm: can connect
torvm: can connect (my tests from inside the VM didn't work, but it
obviously must be able to, since AppVMs which use it can connect
successfully)
firewallvm: ??? (when I issue "host google.com", it just times out, but
when I issue "ping 173.194.66.102", it works?!)
fedora-18-x64: can communicate with the update servers
fedora-18-x64-net: can communicate with the update servers
untrusted: cannot connect
untrusted-tor: can connect (http, pop3, smtp all work)

So, what I'm having trouble figuring out is whether the problem is with
the firewallvm, or something else.

It seems like a DNS issue, but I already changed my DNS several times
(and of course rebooted the whole system after each change). I tried my
own ISPs DNS server, OpenDNs, and Google Public DNS. I had it set to
OpenDNS before (when everything was working correctly) because my ISP's
DNS was causing problems (an unrelated Windows machine on my local
network wasn't able to connect at all a couple days ago using my ISP's
DNS until I changed it to OpenDNS). I wonder if somehow the update to
NetworkManager broke my ability to set the DNS server or something.

[Axon]

Sorry for sending so many messages, but I just wanted to add one more clue:

I have a VM that I use to access my router's administration page. It
uses firewallvm as its NetVM, and it's fine (i.e. I can connect to the
router and everything as normal).

I should also add that I have other computers here which have been
connected normally the whole time (some Windows, some Linux, including
Fedora).

Also, I know this is rather obvious from my previous messages, but if I
*switch* an AppVMs NetVM from firewallvm to torvm, then I can connect
successfully (through tor, of course!).

Lastly, I mentioned some DNS-related things in a previous message, but I
want to add that a computer I have here running plain Fedora 19 has not
had its DNS settings changed at all. It shows that it's still using my
default ISP DNS, and its connection is fine.

So, given all of the information above and in the previous messages, I
think I can confidently say:

It's not a hardware problem.
It's not an ISP problem.
It's not a NetVM problem.
It's not an AppVM problem.
It doesn't *seem* like a ProxyVM problem. (But what else could it be?!)

This is a really strange phenomenon, right?? I mean, how is this even
possible?

Or maybe it only seems really strange to me because there are things I
am ignorant about or do not understand. :)

[ix4...@gmail.com]

On 16 August 2013 19:06, Axon <ax...@openmailbox.org> wrote:

On 08/16/13 07:41, Axon wrote:

On 08/16/13 07:15, Axon wrote:

On 08/16/13 06:33, Axon wrote:

On 08/16/13 04:05, Zrubecz Laszlo wrote:

On 16 August 2013 12:11, Axon <ax...@openmailbox.org> wrote:

<snip>

So, given all of the information above and in the previous messages, I think I can confidently say:

It's not a hardware problem.
It's not an ISP problem.
It's not a NetVM problem.
It's not an AppVM problem.
It doesn't *seem* like a ProxyVM problem. (But what else could it be?!)

I'm afraid I can't help you, but I had the exact same behaviour yesterday. After what was a massive update for some reason (even though I always update the template when I see the notification), I rebooted the machine and then realised I had no network.

Tried a couple of things with tshark/tcpdump on netvm/firewallvm vif and eth0 interfaces, gave myself a headache trying to remember iptables rules etc, had fun reading about "network administratively prohibited" and "host administratively prohibited" ICMP Destination Unreachable messages (which was what my sniffer were showing me) and then I noticed that it was a name resolution issue. Hardcoding the IP of my local (physical) router (which runs a DNS forwarder) in /etc/resolv.conf would make the AppVM resolve names.

So I thought fine, fired up the templateVM and hardcoded the IP of my router there, and then rebooted. The machine appeared to work fine after the reboot and I thought I was very clever, until I noticed that /etc/resolv.conf of my active VMs had been returned to their pre-intervention values (probably by NetworkManager) of
nameserver 10.137.2.1

nameserver 10.137.2.254

So, no idea wtf happened, but it all works now. :-(

Alex

[inf...@gmail.com]

On 08/16/13 22:11, ix4...@gmail.com wrote:

On 16 August 2013 19:06, Axon <ax...@openmailbox.org> wrote:

On 08/16/13 07:41, Axon wrote:

On 08/16/13 07:15, Axon wrote:

On 08/16/13 06:33, Axon wrote:

On 08/16/13 04:05, Zrubecz Laszlo wrote:

On 16 August 2013 12:11, Axon <ax...@openmailbox.org> wrote:

<snip>

So, given all of the information above and in the previous messages, I think I can confidently say:

It's not a hardware problem.
It's not an ISP problem.
It's not a NetVM problem.
It's not an AppVM problem.
It doesn't *seem* like a ProxyVM problem. (But what else could it be?!)

Just had what I assume is same problem (no network in AppVMs after a main template update), except that for me even net access via TorVM didn't work

I had to cure by restoring template from backup

before i did, I notice that in a Terminal in the NetVM I could ping websites by name (i.e. DNS working), however
...in the FirewallVM I could ping by IP address but not by name (DNS not working)

hope helps - I still have the bad template if want me run a any debugs

CB

[Marek Marczykowski-Górecki]

The DNS traffic is DNATed at each netvm/firewallvm. So each VM's DNS points at
its "uplink" VM magic IP, which is redirected to the next VM (or outside
network in final netvm).
So start with checking if it is done correctly in your netvm:
sudo iptables -t nat -nvL PR-QBS

It should point to your real DNS. If not, try:
sudo /usr/lib/qubes/qubes-setup-dnat-to-ns
(it should copy settings from /etc/resolv.conf).

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Axon]

Thank you so much for replying! I was beginning to think that I was
going insane. :)

>
> The DNS traffic is DNATed at each netvm/firewallvm. So each VM's DNS points at
> its "uplink" VM magic IP, which is redirected to the next VM (or outside
> network in final netvm).
> So start with checking if it is done correctly in your netvm:
> sudo iptables -t nat -nvL PR-QBS
>
> It should point to your real DNS. If not, try:
> sudo /usr/lib/qubes/qubes-setup-dnat-to-ns
> (it should copy settings from /etc/resolv.conf).
>

It worked! Marek saves the day again! :)

Just a couple of minor points to help others with the same issue:

The file is actually qubes_setup_dnat_to_ns (at least on my system).
Also, I think you accidentally left out "sh." So, full command: "sudo sh
qubes_setup_dnat_to_ns".
Lastly, running this command changed the destination IP address range to
the range where my firewallvm lives (not the external DNS server IP
addresses, which are in /etc/resolv.conf).

Thank you, Marek! :)

[inf...@gmail.com]

hear hear :-)

Just a couple of minor points to help others with the same issue:

The file is actually qubes_setup_dnat_to_ns (at least on my system).
Also, I think you accidentally left out "sh." So, full command: "sudo sh qubes_setup_dnat_to_ns".
Lastly, running this command changed the destination IP address range to the range where my firewallvm lives (not the external DNS server IP addresses, which are in /etc/resolv.conf).

Although I got my network back through restoring the main template from backup, I am still wondering how to prevent recurrence when I next update

(BTW I hadn't noticed but discovered the reason my TorVM wasn't working was here but have now fixed)

do we know which package caused the problem in the update (or why?)

CB

[Axon]

So, after I reboot, I had to apply this again. Maybe I need to do it in
the template upon which netvm is based to get it to stick across reboots?

Anyway, I think I was mistaken about what it did and did not change,
since it looks like it does indeed cause the "udp dpt" portion of the
iptables output to point to my real external proxy. So, sorry about that. :)

A similar (perhaps related?) issue: I've noticed that when I reboot the
whole machine, I have to start up torvm, then shut it down, then start
it again, before connections to the Tor network can actually be made.
Not sure why, and obviously this is not at all a difficult workaround,
but I thought I would share the data point.

[Krypton Radius]

Hello. I am experiencing the same behavior after a recent dom0 and template update. I had been experimenting with some settings, and so blamed this on myself and wiped my drive. I did a fresh install with no problems. Then, after one dom0 update and a template update done side-by-side, I'm stuck with no working network connection at all. So I can verify at least that this is a problem with a fresh install followed by a dom0 update and a template update. I will try again to see if I can isolate the problem package.

> --
> You received this message because you are subscribed to the Google Groups "qubes-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users...@googlegroups.com.
> To post to this group, send email to qubes...@googlegroups.com.
> Visit this group at http://groups.google.com/group/qubes-users.
> For more options, visit https://groups.google.com/groups/opt_out.

[Marek Marczykowski-Górecki]

> <https://groups.google.com/forum/#%21msg/qubes-users/fyBVmxIpbSs/R5mxUcIEZAQJ>

> but have now fixed)
>
> do we know which package caused the problem in the update (or why?)

I haven't observed such behavior on my system (last template update two days
ago or so)...
Perhaps NetworkManager no longer call this script after connecting to the
network (it is called from /etc/NetworkManager/dispatcher.d/qubes-nmhook).
Check the logs in netvm - perhaps there is some hint...

[inf...@gmail.com]

established trial & error the prob in the template update is in
NetworkManager

CB

[inf...@gmail.com]

 I copied logs from:
-  BAD (updated NetworkManager - network not working in AppVMs) and
- GOOD (template not updated about a month - network working fine)

...versions of NetVM into attached LibreOffice spreadsheet (separate sheets)

but could not see what the problem might be ?

CB

[Axon]

I can report that the problem recurs on my system whenever the netvm is
shut down and started back up. (In other words, not just when dom0 is
restarted, but also after, e.g., running qvm-backup.) I have to apply
Marek's fix, then restart all the ProxyVMs in order to get a connection
in any AppVMs. (Oddly, it now appears that this is true of Tor-connected
VMs, as well. As I relayed in my previous messages, this was not the
case before.)

[Marek Marczykowski-Górecki]

It looks to be the case of:
https://bugzilla.redhat.com/show_bug.cgi?id=974811

For now the workaround (until Fedora releases the fix) is to do in template:
sudo systemctl enable NetworkManager-dispatcher.service

[Gorka Alonso]

Just some feedback, that workaround works great :-)

[Franz]

I tried this but am getting this error:

Failed to issue method call: no such file or directory

Wired Network suddenly stopped working but I did NOT recently update the system. Tried with another ethernet cable (working with other computer). Also tried running another live linux distribution to check if it is an hardware problem, but the other linux works.

[user@netvm ~]$ ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif2.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.137.1.1  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::fcff:ffff:feff:ffff  prefixlen 64  scopeid 0x20<link>
        ether fe:ff:ff:ff:ff:ff  txqueuelen 32  (Ethernet)
        RX packets 58  bytes 5333 (5.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 53  bytes 5753 (5.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[user@netvm ~]$ ping google.com
ping: unknown host google.com
[user@netvm ~]$

any idea

best

Franz

[Marek Marczykowski-Górecki]

ifconfig -a ?
I can't see eth0 above, so this can be a network driver issue.

[Christopher Lee]

Just wanted to summarize my own experience with working around this bug, in case it's helpful to others:

Symptoms:
* appVMs (e.g. work) would just time out on any network access, failing to resolve hostname via DNS
* running dig in appVM would fail the same way (e.g. dig www.ucla.edu gets no nameserver response)
* running same dig command in netvm works!
* netvm is able to ping successfully (e.g. ping www.ucla.edu)
* other appVMs are able to ping successfully if given the IP address (e.g. ping 128.97.27.37)
* netvm and other appVMs show quite different nameserver IP in /etc/resolv.conf, e.g. netvm:
[user@netvm ~]$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.1.1

[user@work ~]$ cat /etc/resolv.conf
nameserver 10.137.2.1
nameserver 10.137.2.254

* other appVM can dig successfully if given explicit DNS server IP address (e.g. dig @192.168.1.1 www.ucla.edu)

* running Marek's test query in netvm shows no mapping of "bad" nameserver address to "good" nameserver address
(sudo iptables -t nat -nvL PR-QBS)

* running Marek's work-around command in netvm indeed adds nameserver mapping
[user@netvm ~]$ sudo sh /usr/lib/qubes/qubes_setup_dnat_to_ns
[user@netvm ~]$ sudo iptables -t nat -nvL PR-QBS
Chain PR-QBS (1 references)
 pkts bytes target     prot opt in     out     source               destination        
  201 13265 DNAT       udp  --  *      *       0.0.0.0/0            10.137.1.1           udp dpt:53 to:192.168.1.1

* other appVMs can now resolve hostnames successfully (e.g. dig www.ucla.edu). 

Success!  Presumably you have to re-run the workaround if you reboot netvm.

I'm curious how fedora can be rolling out such buggy updates to production f18 users?!?  Is the problem that this bug would affect almost no one except Qubes users with this complicated netvm - appVM separation?

[Franz]

ifconfig -a gives exactly the same. No mention of eth0.

Do I have to reinstall Qubes?

Best

Franz

[Marek Marczykowski-Górecki]

Check if you still have network device assigned to your netvm (in Qubes
manager or qvm-prefs/qvm-pci). If you have, check kernel messages (dmesg) in
netvm.

[Marek Marczykowski-Górecki]

Yes, Qubes networking relies on NetworkManager dispatch scripts, which are
currently broken in Fedora. Related bugreport:
https://bugzilla.redhat.com/show_bug.cgi?id=974811

You need to run 'systemctl enable NetworkManager-dispatcher.service' in
template (then reboot).

[Axon]

Not sure if you caught Marek's earlier message in this thread in which
he stated this, but if you run

sudo systemctl enable NetworkManager-dispatcher.service

in the TemplateVM upon which your netvm is based, then you should not
have to re-run the workaround fix above after each netvm reboot.

>
> I'm curious how fedora can be rolling out such buggy updates to production
> f18 users?!? Is the problem that this bug would affect almost no one
> except Qubes users with this complicated netvm - appVM separation?
>

Yes, I wonder about that too. I suspect it must be something like that.

[Franz]

This gives an error:

failed to issue method call: No such file or directory

Best Franz

[Franz]

No I do not have any network device assigned to my netvm.

Best

Franz

[Marek Marczykowski-Górecki]

So... assign some (especially that you want to use) :)

[Franz]

IT WORKS MAREK IT WORKS!!

I just assigned ethernet controller and network controller, rebooted the machine and it works.

Many thanks. Well, I'll need to understand how this bitcoin thing work to send some bitcoin to you.

Best

Franz

[Abel Luck]

Marek Marczykowski-Górecki:

> Yes, Qubes networking relies on NetworkManager dispatch scripts, which are
> currently broken in Fedora. Related bugreport:
> https://bugzilla.redhat.com/show_bug.cgi?id=974811
>
> You need to run 'systemctl enable NetworkManager-dispatcher.service' in
> template (then reboot).
>

Just ran into this bug. Running that systemctl command fixed it for me.
Thanks.

~abel

[Christopher Lee]

I'm now experiencing an annoying variant of this bug: now even the netvm fails to name resolve, off a standard WiFi connection. it can ping by IP address just fine, but no DNS. WiFi connections have always worked automatically in the past, so this seems like I'm being bitten by the same Network Manager bug in a new way. Anyone have any advice?

neither of Marek's workarounds solves this.

$%&#ing Fedora "updates"!

[Tokyo Jeff]

I'm not really sure if this is the same issue, but I was having DNS
issues also. I'm not exactly sure if this is supposed to be like this:

[root@netvm ~]# iptables -t nat -nvL PR-QBS

Chain PR-QBS (1 references)
pkts bytes target prot opt in out source destination

0 0 DNAT udp -- * * 0.0.0.0/0
10.137.1.1 udp dpt:53 to:10.137.2.1
0 0 DNAT udp -- * * 0.0.0.0/0
10.137.1.254 udp dpt:53 to:10.137.2.254

The DNAT to 10.137.2.x is my firewallvm. I'm not sure if there is a bug
here, but it doesn't make sense to me (yet) why netvm would use the
firewallvm for DNS resolution.

My fix was to add to run the following in netvm which resolves the DNS
issue:

# Use Google DNS
sudo iptables -t nat -I PR-QBS -p udp -d $DNS1 --dport 53 -j DNAT --to
8.8.8.8
sudo iptables -t nat -I PR-QBS -p udp -d $DNS2 --dport 53 -j DNAT --to
8.8.4.4

Also, of note, I added above commands to rc.local in my netvm, but it
appears that rc.local is run prior to the firewall being configured in
the netvm, so these rules get overwritten. Currently, I am running the
script manually after a reboot. Should the qubes_firewall_user_script
be executed in non proxy VMs? It doesn't appear to be run in netvm. I
think this would be a worthwhile addition to allow iptables rule
customizations that persist across reboots.

TJ.

[inf...@gmail.com]

On 08/30/13 20:28, Christopher Lee wrote:
> $%&#ing Fedora "updates"!

I am v.surprised one of the major distros has not fixed a regression bug
after many weeks which breaks networking on a normal update

What's the protocol to light a fire under this?

?

CB

[Marek Marczykowski-Górecki]

On 31.08.2013 11:20, Tokyo Jeff wrote:
> On 08/30/13 21:28, Christopher Lee wrote:
>> I'm now experiencing an annoying variant of this bug: now even the netvm
>> fails to name resolve, off a standard WiFi connection. it can ping by IP
>> address just fine, but no DNS. WiFi connections have always worked
>> automatically in the past, so this seems like I'm being bitten by the same
>> Network Manager bug in a new way. Anyone have any advice?
>>
>> neither of Marek's workarounds solves this.
>>
>>
>> $%&#ing Fedora "updates"!
>>
> I'm not really sure if this is the same issue, but I was having DNS issues
> also. I'm not exactly sure if this is supposed to be like this:
>
> [root@netvm ~]# iptables -t nat -nvL PR-QBS
> Chain PR-QBS (1 references)
> pkts bytes target prot opt in out source destination
> 0 0 DNAT udp -- * * 0.0.0.0/0 10.137.1.1
> udp dpt:53 to:10.137.2.1
> 0 0 DNAT udp -- * * 0.0.0.0/0 10.137.1.254
> udp dpt:53 to:10.137.2.254
>
> The DNAT to 10.137.2.x is my firewallvm. I'm not sure if there is a bug here,
> but it doesn't make sense to me (yet) why netvm would use the firewallvm for
> DNS resolution.

I think this is the same bug - qubes script isn't called by NetworkManager, so
PR-QBS rules aren't regenerated when entries in /etc/resolv.conf is set.
Have you tried "sudo systemctl enable NetworkManager-dispatcher.service" in
template?

Above strange entries are from the fact that at netvm startup /etc/resolv.conf
is in the state as in templatevm (which sends DNS to firewallvm), so initial
firewall rules points at firewallvm.

> My fix was to add to run the following in netvm which resolves the DNS issue:
>
> # Use Google DNS
> sudo iptables -t nat -I PR-QBS -p udp -d $DNS1 --dport 53 -j DNAT --to 8.8.8.8
> sudo iptables -t nat -I PR-QBS -p udp -d $DNS2 --dport 53 -j DNAT --to 8.8.4.4
>
> Also, of note, I added above commands to rc.local in my netvm, but it appears
> that rc.local is run prior to the firewall being configured in the netvm, so
> these rules get overwritten. Currently, I am running the script manually
> after a reboot. Should the qubes_firewall_user_script be executed in non
> proxy VMs? It doesn't appear to be run in netvm. I think this would be a
> worthwhile addition to allow iptables rule customizations that persist across
> reboots.

You can also regenerate DNAT rules based on /etc/resolv.conf by "sudo sh
/usr/lib/qubes/qubes_setup_dnat_to_ns".

[Tokyo Jeff]

Yes, this does in fact resolve the DNS issues. Qubes prerouting now
looks like this:

[root@netvm config]# iptables -t nat -nvL PR-QBS

Chain PR-QBS (1 references)
pkts bytes target prot opt in out source destination

10 678 DNAT udp -- * * 0.0.0.0/0
10.137.1.1 udp dpt:53 to:10.0.1.1

However, I continue to have the issue that there is no good place to add
custom firewall rules in a netvm, such as forwarding services from the
outside world to a particular appvm. Any iptables rules set in rc.local
are overwritten by qubes' scripts when a connection to the network is
established. Would the addition of calling the
qubes_firewall_user_script in non-proxy VMs be considered? This would
be a logical location for user customization of the firewall. Even in an
appvm this would be beneficial (e.g. to accept the incoming services
forwarded there).

[Marek Marczykowski-Górecki]

Some time ago there was thread about external services on qubes-devel.

For now, you can add own scripts to /etc/NetworkManager/dispatcher.d (called
after establishing network connection). NetworkManager is active only in netvm.
If you do not want to place them in template, you can store them in /rw/config
and copy/symlink to /etc in rc.local.

[Robert William Fuller]

I know this is an old thread, but it's still relevant! I tried the
approach you suggested. I wrote an /rw/config/rc.local that looks like
this:

#!/bin/sh
cp rc.local2 /etc/NetworkManager/dispatcher.d

The script works if I run it manually. The problem seems to be that the
network VM no longer invokes /rw/config/rc.local.

Rob

[Robert William Fuller]

Although, I bet a fully qualified path for rc.local2 would fix that....
Whoops.

Rob

[pfl...@gmail.com]

Hello,
I've experienced the same issue but notices there's no entry for PR-QBS in my netvm's iptables config. Should I just add it manually and execute the script? It should have been added during the default installation right? Thanks in advance for your help I really need to get this to work.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Oct 04, 2015 at 05:10:28AM -0700, pfl...@gmail.com wrote:
> Hello,
> I've experienced the same issue but notices there's no entry for PR-QBS in my netvm's iptables config. Should I just add it manually and execute the script? It should have been added during the default installation right? Thanks in advance for your help I really need to get this to work.

What template do you use for netvm (fedora 20 or 21)?
Check status of iptables service:
sudo systemctl status iptables

If there are any errors, please send exact message.

Do you have /etc/sysconfig/iptables file? Or a symlink to iptables.qubes
there?
Check /etc/sysconfig/iptables-config - there should be
"IPTABLES_DATA=/etc/sysconfig/iptables.qubes" at the end.

- --

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWEb1jAAoJENuP0xzK19cs6fwH/0bSfmR9KxWzvklGvOjdUCBB
aD8muyUfsrTTMWPGujj89u0xM4xM2hYdI4vHnxMCKq6OTyS7jXhevWuCcU1R5RRX
6NBzp9LS4TbN0Ls9GGe6LctOUMhP0xuXPoHSyoZLKyR0obz6/srTwWoiSVEbeBtv
arBjGXpxosZyIgzJpdEVHdIoiALjazdpjHwFtlod7GNDNwXV6A6Ea1VqCwddNb7G
vgv+t4D5FlgibaEdMFw1BOvn6JFbOWI/1j3HJs3Am1Ms24rpn0pybIkggCfOejif
Krv5ptq2R1X+Z0a5EzdPirQKx/lGhKGRARY8IY/StBaXey/3Sth3axYAGLh91+Y=
=csDP
-----END PGP SIGNATURE-----

[pfl...@gmail.com]

Hello Marek,
thanks for your reply. Here are the details:

> What template do you use for netvm (fedora 20 or 21)?

fedora-20-x64

> Check status of iptables service:
> sudo systemctl status iptables

[user@netvm ~]$ sudo systemctl status iptables
iptables.service - IPv4 firewall with iptables
Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled)
Active: inactive (dead)
start condition failed at Sun 2015-10-04 19:04:23 EDT; 1h 15min ago
ConditionPathExists=/etc/sysconfig/iptables was not met

Oct 04 19:04:23 netvm systemd[1]: Started IPv4 firewall with iptables.

> Do you have /etc/sysconfig/iptables file? Or a symlink to iptables.qubes
> there?
> Check /etc/sysconfig/iptables-config - there should be
> "IPTABLES_DATA=/etc/sysconfig/iptables.qubes" at the end.

Yes I do have the file. And the entry is there:
### Automatically added by Qubes:
# Override default rules location on Qubes
IPTABLES_DATA=/etc/sysconfig/iptables.qubes

I have also followed some steps that have been suggested regarding PR-QBS and executing the script: /usr/lib/qubes/qubes-setup-dnat-to-ns

I've noticed the tiniproxy service is not running and logs show it's constantly trying to restart unsuccessfully. This all happened after the regular templateVM update after the initial installation. Now, as you may imagine, I cant neither update templateVM nor dom0.
NetVM can ping, browse and do everything network wise.
Any other VM is unable to do so.
I'm currently downloading release 3 but would definitely like to fix this.
Thanks a lot for your kind response,
Pedro.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Oct 04, 2015 at 05:29:40PM -0700, pfl...@gmail.com wrote:
> Hello Marek,
> thanks for your reply. Here are the details:
>
> > What template do you use for netvm (fedora 20 or 21)?
> fedora-20-x64
>
> > Check status of iptables service:
> > sudo systemctl status iptables
> [user@netvm ~]$ sudo systemctl status iptables
> iptables.service - IPv4 firewall with iptables
> Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled)
> Active: inactive (dead)
> start condition failed at Sun 2015-10-04 19:04:23 EDT; 1h 15min ago
> ConditionPathExists=/etc/sysconfig/iptables was not met

Ok, missing /etc/sysconfig/iptables is the problem. Can you check what
was the previous qubes-core-vm package version? Simply execute "grep
qubes-core-vm /var/log/yum.log" and you should get all the package
history.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWEcdqAAoJENuP0xzK19cs2FYIAJf89Sp2EHejzgXFgMy4Qs7M
+ck6BADEzwtoiuqSqPMaRlQjJjoMnRauC8K4JStmJkuSVj768cZP2+rqIdoPpGmt
1wdnZROyriJ0N7c3T4r51aul/VWE6XoOU1ejhjng4zj25dkQe2TkC4OcD1zqF+mV
/sKOoIo1ijpaNlv6N0508aP+mFUiBublO5cO9wdYo3KuGXlEPcsRzhbXBkl2LaNY
k2egUgbRnmh17Lhx72JjOpwLndeUKa7p2ABV/fXRIUolJLfgt4aKRPbWkQ+FROTp
ffCA5zr1lNsYevY4e1k0SFYGXXdbNXQ8s0oEiAphg9MNPPF9bZZKcSaV/cKzykU=
=iGb0
-----END PGP SIGNATURE-----

[pfl...@gmail.com]

Thanks for the quick reply!
Here it is:

> Ok, missing /etc/sysconfig/iptables is the problem. Can you check what
> was the previous qubes-core-vm package version? Simply execute "grep
> qubes-core-vm /var/log/yum.log" and you should get all the package
> history.

[user@netvm ~]$ sudo grep -i qubes-core-vm /var/log/yum.log
Mar 25 16:18:27 Installed: qubes-core-vm-kernel-placeholder-1.0-2.fc20.x86_64
Sep 22 14:16:14 Installed: qubes-core-vm-2.1.41-1.fc20.x86_64
Sep 22 14:16:16 Installed: qubes-core-vm-systemd-2.1.41-1.fc20.x86_64
Oct 01 16:07:04 Updated: qubes-core-vm-2.1.66-1.fc20.x86_64
Oct 01 16:07:05 Updated: qubes-core-vm-systemd-2.1.66-1.fc20.x86_64

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

One more thing: what is result of:
ls -l /etc/sysconfig/ip*tables*

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWEfIoAAoJENuP0xzK19csywAH/j3n5jU7ikiHpeHItpX2Quhq
oQ/jVZSxMKR0d5U7HPfu2HgAE4auFIb4CCdt/4qAzQpD0VEahuAbu4R8KmkUEeJr
jGFnf2uDLtry5Y1pSqwP4+tFkFsp7uwm51KtgKE1AFet+urIKqUhL4q5n20mjp+W
jJ6As8XGrG3Gk6Ia9x4PgpM6+C5aCXcd3OUEspyZ29HRtsSmEHSRcNfymOpPjcbJ
qSafdflq7CqaV5k2IxxU4RhQ+UG63XPmqTukw++fz8LT6QYwUvrbzZfwiju7BwM7
lyUYeAlZqHDu71D1oQO3aAN+BHdnKGy7yOEo82rNMMyRsVLnZMQiJrnHFjMGNe0=
=OS/9
-----END PGP SIGNATURE-----

[pfl...@gmail.com]

> One more thing: what is result of:
> ls -l /etc/sysconfig/ip*tables*
>

Here it is:

[user@netvm ~]$ ls -l /etc/sysconfig/ip*tables*
-rw------- 1 root root 1877 Oct 1 16:07 /etc/sysconfig/ip6tables-config
-r-------- 1 root root 206 Jul 17 10:16 /etc/sysconfig/ip6tables.qubes
-rw------- 1 root root 1862 Oct 1 16:07 /etc/sysconfig/iptables-config
-r-------- 1 root root 942 Jul 17 10:16 /etc/sysconfig/iptables.qubes

Thanks!
Pedro.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ok. The problem should be fixed in package currently in testing
repository. But to install it, you need network access first...
To fix that just to install an update, open netvm terminal and:
sudo touch /etc/sysconfig/iptables
sudo systemctl restart iptables
sudo systemctl restart qubes-network
sudo systemctl restart qubes-updates-proxy

And in firewallvm:
sudo touch /etc/sysconfig/iptables
sudo systemctl restart iptables
sudo systemctl restart qubes-firewall

Then start your template and update the package:
sudo yum update --enablerepo=qubes-*-testing

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWE51bAAoJENuP0xzK19csF4EH/R5JP4QYPnJ6WGT1CE4bILwe
mO3CgxxxQOd8wNs+CdsxAYX2vn4NbgfpmnEwfoYqdbY1lT3DmgyzRdFZKSBR7hwb
p0N/qr8pZTmx26+/tvfczklVz91SNB4333SEZLukxXq53c5hhlOr0AtxLNOhvX2F
StAVz4j7OFMYc8Z7cI7S/rT5DW+QzWVrhjCUtfC890JTgB5NGtM1tYpE/ipoWEod
dsv8rFQh3lJEDqwUaiD2cmmJqpBQqL4Mb1Pa8exxAATKvUkKiGySh/3H5F/HMmkK
7NEbXBmfFczDisYZvlhkDEeJDe7+v0ArkOwLYPGp+iZekDR+XR118JYWzLrBrQ4=
=qvp1
-----END PGP SIGNATURE-----

[pfl...@gmail.com]

Hello Marek, thanks for your answer.
Update: Release 3 solved the issue. Will test your suggested steps in a Release 2 installation we're trying to set up. Thanks again for your kind help!
Pedro.

[fernandoval...@gmail.com]

Yes! This also fixed my connectivity issues!!!
Thank you very much!

Now I have a couple questions:

I would like to know if there is any way to change the install menu to avoid the vms to startup automatically so I'll be able to finish the installation without errors as explained in https://groups.google.com/forum/#!topic/qubes-users/EcfV9pDhhLA and aply the script from https://groups.google.com/forum/#!msg/qubes-users/o8eahbAg3q0/oUPBOpIRSXkJ and then manually bring the vms up after.

If this is not possible and I have to do clean r2 installation and then update, I should sudo yum update --enablerepo=qubes-*-testin instead of general yum update?