Qubes 4: Windows 7 HVM loses internet connection after installing Qubes Windows Tools
Otto Kratik
Created a new Windows 7 SP1 HVM by using an .iso as I've done many times successfully in the past on Qubes 3.2. Everything works fine, HVM is created and runs correctly, internet connection is intact and functioning as expected.
Downloaded Qubes Windows Tools 4.0 and installed them into the HVM as per documentation. Installation of QWT 4.0 completely *breaks* HVM and places it into a totally irrecoverable state, as detailed here near the top:
https://github.com/QubesOS/qubes-issues/issues/3585
"at the following reboot, windows fails to boot and tries to repair files, which as usual doesn't fix anything (the VM might boot at some point, without the tools installed)"
Deleted the HVM, and started over from scratch by creating a new one. This time, I installed Qubes Windows Tools version 3.2.2.3 instead of 4.0. That worked perfectly fine, and the usual QWT-enhanced Windows HVM appeared with full screen resolution, mouse pointer integration etc. Except, the internet connection suddenly completely went offline and stopped functioning.
As per Issue 1 again from the same source, I tried the following instructions:
https://github.com/QubesOS/qubes-issues/issues/3585
--------
Issue 1 - Networking isn't set up properly
The PV adapter's status is stuck at "Identifying"; pinging an ip works but pinging a host fails, indicating a problem with DNS resolution. A tcpdump on sys-firewall indeed shows that DNS requests are sent to the gateway's ip and are rejected. The reason is that in R4 VMs are now using the exposed "/qubes-{primary,secondary}-dns" values, while R3.2's Windows Tools still use /qubes-gateway (whose IP in R4 is different from /qubes-primary-dns).
Workaround: disable the "Qubes Network Setup" service (with gui/msconfig, or sc config "QubesNetworkSetup" start= disabled in a command prompt) and configure the network manually.
Settings:
DNS{1,2}: 10.139.1.1, 10.139.1.2
Subnet: 255.255.255.0
IP: in dom0, qvm-prefs vmname ip will output the VM's ip. Caveat: a cloned/restored/... VM will likely have its IP changed so you'll have to remember to update your network settings.
----
Implementing this attempted fix did *NOT* solve the problem, and the lack of internet connectivity persists despite doing everything suggested. All other VMs on the system have their internet connections working perfectly fine.
What are the next suggested steps to try? Should that fix have worked regardless of using QWT 3.2.2.3 rather than 4.0, as long as the base system is Qubes 4 instead of 3.2? If not, what options should I be using for my specific situation? What do I do from here to get internet connectivity back?
Ivan Mitev
there's probably an issue with your ip settings in the windows HVM. Make
sure that the IP you've manually configured in windows matches the one
assigned to the VM (in dom0, run `qvm-prefs vmname visible_ip`).
Also check that the ip/gateways/dns ips you've configured are properly
applied in the vm (`ipconfig /all` in a command prompt), it could be
that the "Qubes Network Setup" service is still active for some reason.
Then, if everything looks OK, try to ping the VM's IP from the VM (this
should work), then the gateway, then the DNS IPs.
Otto Kratik
> Hi,
> Since you mention that the network is functional without QWT installed
> there's probably an issue with your ip settings in the windows HVM. Make
> sure that the IP you've manually configured in windows matches the one
> assigned to the VM (in dom0, run `qvm-prefs vmname visible_ip`).
> Also check that the ip/gateways/dns ips you've configured are properly
> applied in the vm (`ipconfig /all` in a command prompt), it could be
> that the "Qubes Network Setup" service is still active for some reason.
>
> Then, if everything looks OK, try to ping the VM's IP from the VM (this
> should work), then the gateway, then the DNS IPs.
I'll give this a try, thanks. So far in the Windows HVM I have not put any value under "gateway" because that is not mentioned in the instructions from Issue3585 above. Only IP address, Subnet mask, and DNS server fields are filled out. Default gateway is left empty.
What value, if anything, should go under Gateway in the VM? The ip address shown by Qubes as belonging to the network-providing VM itself, ie Sys-Net or Sys-Firewall, namely 10.137.0.6 ? Or something else?
Also, I am presuming the values listed for DNS servers are universal constants at the moment in Qubes 4, meaning 10.139.1.1 and 10.139.1.2 are absolute values for all installations and not dynamically dependent on specific configuration?
Ivan Mitev
On 11/12/18 9:47 PM, Otto Kratik wrote:
> On Monday, November 12, 2018 at 1:52:44 PM UTC-5, Ivan Mitev wrote:
>> Hi,
>> Since you mention that the network is functional without QWT installed
>> there's probably an issue with your ip settings in the windows HVM. Make
>> sure that the IP you've manually configured in windows matches the one
>> assigned to the VM (in dom0, run `qvm-prefs vmname visible_ip`).
>> Also check that the ip/gateways/dns ips you've configured are properly
>> applied in the vm (`ipconfig /all` in a command prompt), it could be
>> that the "Qubes Network Setup" service is still active for some reason.
>>
>> Then, if everything looks OK, try to ping the VM's IP from the VM (this
>> should work), then the gateway, then the DNS IPs.
>
>
> I'll give this a try, thanks. So far in the Windows HVM I have not put any value under "gateway" because that is not mentioned in the instructions from Issue3585 above. Only IP address, Subnet mask, and DNS server fields are filled out. Default gateway is left empty.
Actually the issue was originally meant to document the problems with
QWT on R4 but it slowly evolved into a step-by-step guide.
I've also added a note about QWT 4 breaking *new* HVMs (I thought the
breakage was only when updating from QWT3 to QWT4). It seems it's a
hit-or-miss process, IIRC some users managed to have QWT4 running.
> What value, if anything, should go under Gateway in the VM? The ip address shown by Qubes as belonging to the network-providing VM itself, ie Sys-Net or Sys-Firewall, namely 10.137.0.6 ? Or something else?
a fancy vpn/firewall setup, it's likely 10.137.0.6.
> Also, I am presuming the values listed for DNS servers are universal constants at the moment in Qubes 4, meaning 10.139.1.1 and 10.139.1.2 are absolute values for all installations and not dynamically dependent on specific configuration?
python scripts in dom0.
Achim Patzner
You do not need to quote a full message as a block; just coppy what you
really refer to.
> Since you mention that the network is functional without QWT
> installed there's probably an issue with your ip settings in the
> windows HVM.
and updated at what point.
I've just finished setting up a new Windows 7 HVM, too. The up to now
best route for me was installing an original Windows 7 SP 1 medium and
then spend about two days of updating it (including 28 reboots...)
before even trying to install the tools package.
Achim
Achim Patzner
I've also added a note about QWT 4 breaking *new* HVMs (I thought thebreakage was only when updating from QWT3 to QWT4). It seems it's ahit-or-miss process, IIRC some users managed to have QWT4 running.
What value, if anything, should go under Gateway in the VM? The ip address shown by Qubes as belonging to the network-providing VM itself, ie Sys-Net or Sys-Firewall, namely 10.137.0.6 ? Or something else?The ip output by `qvm-prefs vmname visible_gateway` ; if you don't havea fancy vpn/firewall setup, it's likely 10.137.0.6.
Ivan Mitev
On 11/13/18 1:44 PM, Achim Patzner wrote:
> On 20181113 at 06:44 +0200 Ivan Mitev wrote:
>> I've also added a note about QWT 4 breaking *new* HVMs (I thought the
>> breakage was only when updating from QWT3 to QWT4). It seems it's a
>> hit-or-miss process, IIRC some users managed to have QWT4 running.
>
> The real problem with these tools is not being able to install and
> deinstall them in steps. Somewhere along the way I lost libvirt and
> there is no easy way to just put it where it belongs. Using the
> installer to "repair" the system breaks it because it is messing with
> the drivers. If you uninstall completely you break the system with the
> reinstallation. All in all it worked better NOT to use the Qubes tools
> but the XEN installers and add the Qubes video driver later.
components but I agree that it'd be more flexible to have the tools
split in several packages.
>>> What value, if anything, should go under Gateway in the VM? The ip address shown by Qubes as belonging to the network-providing VM itself, ie Sys-Net or Sys-Firewall, namely 10.137.0.6 ? Or something else?
>>
>> The ip output by `qvm-prefs vmname visible_gateway` ; if you don't have
>> a fancy vpn/firewall setup, it's likely 10.137.0.6.
>
> This is another joke I'm not understanding. Ok, no DHCP for the
> unwashed masses. But if I have qubes-rpc working, why not inject the
> necessary settings using this mechanism?
There is a dhcp server in XEN's stub domain, that's why networking works
out of the box on plain windows VMs. The problem is that the PV network
driver (installed by QWT or manually from XEN windows PV drivers)
bypasses the stub-domain networking [1].
Workarounds:
- use QWT 4 (if it works for you).
- or, use QWT 3 and configure the network manually
- or, don't use the network PV driver, which should be perfectly fine
for VMs used for casual browsing.
Re- rpc settings: IIRC the qubes network service provided by QWT reads
the ip/dns/gw/... values from the exposed keys [2] and sets the network
accordingly.
[1] https://groups.google.com/forum/#!topic/qubes-users/EXAcxrD7ZQU
[2] https://www.qubes-os.org/doc/vm-interface/
>
>
> Achim
>
Otto Kratik
> Oh, I see. I've updated the issue to add a mention about the gateway.
> Actually the issue was originally meant to document the problems with
> QWT on R4 but it slowly evolved into a step-by-step guide.
> a fancy vpn/firewall setup, it's likely 10.137.0.6.
Thanks - I added the sys-firewall gateway value and that seemed to do the trick in restoring connectivity (which is of course, entirely obvious in hindsight). A couple of oddities I noticed though:
With everything manually configured and working, I can successfully ping the VM's own ip address and the gateway from within the VM, however I can *NOT* ping the DNS servers at all.
Attempting to ping 10.139.1.1 or 10.139.1.2 results in:
Response from 10.128.100.62: Destination net unreachable
I have no idea what that IP address above is. Obviously DNS resolution is working since I can lookup websites correctly as expected, but the ping attempt either fails with that reply or times out completely, every single time.
Also, if I delete the DNS entries from adapter IPv4 config completely and then do "ipconfig /all" from command line, they seem to get magically filled in by themselves, with one slight change:
10.138.1.1 <-- (note the 138 instead of 139)
10.139.1.2
..And everything continues to work fine in terms of connectivity. The Qubes Network Setup service is definitely disabled and stopped, so I am not quite sure how that auto-fill is occurring.
I can also use other externally operated DNS like:
8.8.8.8
4.4.4.4
1.1.1.1
And it gets saved correctly in ipconfig and also produces full connectivity. I am going to try garbage values and see what happens, but it almost seems like the HVM is somehow routing its DNS queries automatically regardless of entered values, but maybe not.
> I've also added a note about QWT 4 breaking *new* HVMs (I thought the
> breakage was only when updating from QWT3 to QWT4). It seems it's a
> hit-or-miss process, IIRC some users managed to have QWT4 running.
Hit or miss, yes... possibly partially related to the state of updates in Windows 7 at the time QWT4 is installed. Those reporting success (in this thread and issue 3585) seem to have installed updates into Win7 first before installing the guest tools. In my case I tried installing QWT4 into a fresh Win7 SP1 with no updates applied yet, and it broke completely. So that might be the crux, though it's just a hypothesis.
At some point if I have the 2-3 days needed to fully update Windows 7, I may try removing QWT3 and installing QWT4 to see what happens. Of course I will try this in a clone, since I have no idea how easy or difficult it actually is to uninstall QWT3223 cleanly, and it's far more likely I'll break something in the attempt. Is it just a question of selecting "Remove" from the internal Win7 "Add/Remove Programs", and then installing QWT4 anew? Or is there a more elaborate procedure required?
Ivan Mitev
On 11/13/18 9:15 PM, Otto Kratik wrote:
> On Monday, November 12, 2018 at 11:44:08 PM UTC-5, Ivan Mitev wrote:
>> Oh, I see. I've updated the issue to add a mention about the gateway.
>> Actually the issue was originally meant to document the problems with
>> QWT on R4 but it slowly evolved into a step-by-step guide.
>> The ip output by `qvm-prefs vmname visible_gateway` ; if you don't have
>> a fancy vpn/firewall setup, it's likely 10.137.0.6.
>
> Thanks - I added the sys-firewall gateway value and that seemed to do the trick in restoring connectivity (which is of course, entirely obvious in hindsight). A couple of oddities I noticed though:
>
> With everything manually configured and working, I can successfully ping the VM's own ip address and the gateway from within the VM, however I can *NOT* ping the DNS servers at all.
>
> Attempting to ping 10.139.1.1 or 10.139.1.2 results in:
>
> Response from 10.128.100.62: Destination net unreachable
forgot about that when I advised to ping them - sorry.
> I have no idea what that IP address above is. Obviously DNS resolution is working since I can lookup websites correctly as expected, but the ping attempt either fails with that reply or times out completely, every single time.
>
> Also, if I delete the DNS entries from adapter IPv4 config completely and then do "ipconfig /all" from command line, they seem to get magically filled in by themselves, with one slight change:
>
> 10.138.1.1 <-- (note the 138 instead of 139)
> 10.139.1.2
>
> ..And everything continues to work fine in terms of connectivity. The Qubes Network Setup service is definitely disabled and stopped, so I am not quite sure how that auto-fill is occurring.
With the qubes network service disabled the only way I'd think of would
be to get the ips from the dhcp server running in the xen stub domain
but this shouldn't work with the PV driver (which is the reason
automatic settings work without QWT - see my other email in reply to
Achim Patzner) - and if it did the vm's ip/mask/gw should have been set
too automatically.
Also AFAIK there's no 10.138.1.1 ip used in R4, so it must be coming
from QWT.
> I can also use other externally operated DNS like:
>
> 8.8.8.8
> 4.4.4.4
> 1.1.1.1
>
>
> And it gets saved correctly in ipconfig and also produces full connectivity. I am going to try garbage values and see what happens, but it almost seems like the HVM is somehow routing its DNS queries automatically regardless of entered values, but maybe not.
rules in sys-firewall and sys-net are only valid for 10.139.1.{1,2} (at
least on my setup).
>> I've also added a note about QWT 4 breaking *new* HVMs (I thought the
>> breakage was only when updating from QWT3 to QWT4). It seems it's a
>> hit-or-miss process, IIRC some users managed to have QWT4 running.
>
> Hit or miss, yes... possibly partially related to the state of updates in Windows 7 at the time QWT4 is installed. Those reporting success (in this thread and issue 3585) seem to have installed updates into Win7 first before installing the guest tools. In my case I tried installing QWT4 into a fresh Win7 SP1 with no updates applied yet, and it broke completely. So that might be the crux, though it's just a hypothesis.
*must* be installed in order to be able to use the PV storage driver,
so it might a requirement for other stuff. Windows being windows, it's
really a hit-or-miss process.
> At some point if I have the 2-3 days needed to fully update Windows 7, I may try removing QWT3 and installing QWT4 to see what happens. Of course I will try this in a clone, since I have no idea how easy or difficult it actually is to uninstall QWT3223 cleanly, and it's far more likely I'll break something in the attempt. Is it just a question of selecting "Remove" from the internal Win7 "Add/Remove Programs", and then installing QWT4 anew? Or is there a more elaborate procedure required?
using larger update packs, like described in the following guide:
https://plugable.com/2016/06/08/windows-7-wont-update-what-to-do/
advice: clone the VM before each important step and always keep at least
the past 2 clones; I remember discarding a clone after the VM
successfuly booted, only to find out that it wouldn't boot next time.
FWIW I never managed to update from QWT3 to QWT4, whatever I tried -
updating "over" QWT3, removing QWT3 first (through add/remove
programes), ...
advice 2: fully update the VM first and then mess with QWT. Again, clone
first before testing anything.
Good luck - you'll need some :)