I am running a Qubes R4.0 install on Thinkpad X250, fully `*-testing` updated, and after restoring most of my R3.2 qubes and getting almost ready to switch fully to the new version, something happened with `qubesd` or `qubes-db-dom0`.

Most `qvm-*` command fail with:

```
Traceback (most recent call last):
  File "/usr/bin/qvm-start", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.5/site-packages/qubesadmin/tools/qvm_start.py", line 179, in main
    domain.start()
  File "/usr/lib/python3.5/site-packages/qubesadmin/vm/__init__.py", line 100, in start
    self.qubesd_call(self._method_dest, 'admin.vm.Start')
  File "/usr/lib/python3.5/site-packages/qubesadmin/base.py", line 68, in qubesd_call
    payload_stream)
  File "/usr/lib/python3.5/site-packages/qubesadmin/app.py", line 483, in qubesd_call
    return self._parse_qubesd_response(return_data)
  File "/usr/lib/python3.5/site-packages/qubesadmin/base.py", line 102, in _parse_qubesd_response
    raise exc_class(format_string, *args)
  File "/usr/lib/python3.5/site-packages/qubesadmin/exc.py", line 29, in __init__
    message_format % tuple(int(d) if d.isdigit() else d for d in args),
TypeError: not enough arguments for format string
```

The first occurrence was when attempting to restore a backup:
```
qubesadmin.backup: Error restoring VM p-android, skipping: not enough arguments for format string
```

When the issue started, all qubes that were active were still operating. And `qvm-run <qube> <command>` worked fine on them. Other commands like `qvm-ls` work ok.

After a reboot, all ServiceVMs are not starting, so the complete system is broken.

I did some analysis on the traceback and the error seems to be simply that an Exception is raised without any data. But I do not know how to enable more debugging or provide more specific data.

Tryed to restart `dom0` daemons, look into /var/log/qubes/* and /var/log/libvirt/* without getting any clues about what is failing.

Only change from stock install is that I had to `lvextend` the default `pool00` volume to have enough disk space, and the extension segment is on a second, permanent SSD.

Should I raise an issue?
Any debugging tips?

Cheers,
///Pablo

You have almost exactly described my scenario.

You can work around the TypeError exception by editing /usr/lib/python3.5/site-packages/qubesadmin/exc.py, as I did.  Please note, however, that the lines _must_ be indented with spaces (not tabs); as Python3 is very particular about indentation style.

If, as I suspect, the root cause of your problem is a lack of metadata space on pool00; you can confirm this by typing "sudo lvs" into a console.  You will then need to figure out a way to enlarge that metadata volume.

> If, as I suspect, the root cause of your problem is a lack of metadata space on pool00; you can confirm this by typing "sudo lvs" into a console.  You will then need to figure out a way to enlarge that metadata volume.

Yes, you are right, the `pool00` volume metadata was >96% when this happened. The thing is that the volume metadata was set to a quite small size after install (96mb on a 46gb pool) and after install was on ~20% usage. I started to use the system, testing stuff with DispVMs, restoring my debian templates and some work VMs. After a couple of days of usage the metadata climbed very little, to 27-28%.

I tried to have a second pool to hold my machines, precisely to avoid issues with thin provisioning on the pool holding `root` and `swap` and services vms. But the lack of support for cloning/moving between pools made that effort moot.

So I `lvextend`ed `pool00` and forgot to properly enlarge it's  `pool00_tmeta` counterpart.

When doing some more customization, including restoring more larger sized qubes and cloning/renaming qubes it seems the metadata usage climbed really fast and hit this bug.

Unfortunately, could not recover from that.

It looks like qubes lvm actions while metadata was full may have corrupted the metadata somehow, since I could enlarge and repair the thin metadata from a live cd, but many of the volumes that where in use where never available again. The -private and -snap for the qubes that were running (not sure how to discard them) and also all the volumes of the qubes being restored and services vms are lost ("NOT available" as lvm status)

I remember there was some Saltstack magic to recreate the services vms, but could not find anything for R4.0... So I had to revert to R3.2 for the time being.

I will keep the failing install for debugging, or may be able to recover if someone can provide any tips about:

- How to recreate sys-net, sys-firewall and sys-usb on a R4.0 system
- how to recover a qube whose -snap volumes are no longer available (I have no problem losing these short-term data)

Thanks for pointing to the right direction!

On Thu, Apr 12, 2018 at 08:19:10AM -0700, Pablo Di Noto wrote:
> > techg...@gmail.com
> > If, as I suspect, the root cause of your problem is a lack of metadata space on pool00; you can confirm this by typing "sudo lvs" into a console.  You will then need to figure out a way to enlarge that metadata volume.
>
> Yes, you are right, the `pool00` volume metadata was >96% when this happened. The thing is that the volume metadata was set to a quite small size after install (96mb on a 46gb pool) and after install was on ~20% usage. I started to use the system, testing stuff with DispVMs, restoring my debian templates and some work VMs. After a couple of days of usage the metadata climbed very little, to 27-28%.
>
> I tried to have a second pool to hold my machines, precisely to avoid issues with thin provisioning on the pool holding `root` and `swap` and services vms. But the lack of support for cloning/moving between pools made that effort moot.
>
> So I `lvextend`ed `pool00` and forgot to properly enlarge it's  `pool00_tmeta` counterpart.

> When doing some more customization, including restoring more larger sized qubes and cloning/renaming qubes it seems the metadata usage climbed really fast and hit this bug.
>
> Unfortunately, could not recover from that.
>
> It looks like qubes lvm actions while metadata was full may have corrupted the metadata somehow, since I could enlarge and repair the thin metadata from a live cd, but many of the volumes that where in use where never available again. The -private and -snap for the qubes that were running (not sure how to discard them) and also all the volumes of the qubes being restored and services vms are lost ("NOT available" as lvm status)

> I remember there was some Saltstack magic to recreate the services vms, but could not find anything for R4.0... So I had to revert to R3.2 for the time being.

> I will keep the failing install for debugging, or may be able to recover if someone can provide any tips about:
>
> - How to recreate sys-net, sys-firewall and sys-usb on a R4.0 system
> - how to recover a qube whose -snap volumes are no longer available (I have no problem losing these short-term data)
>
> Thanks for pointing to the right direction!
>

> > > > You could also try to revert to earlier revision using "qvm-volume
> > revert sys-net:private" for example.
>
> Will try that tonight.
>

Unfortunately, that was not possible for the service VMs.
For the only important AppVM that had -back and -back missing, the command `qvm-block revert p-2018d:private` was enough.

> > > I remember there was some Saltstack magic to recreate the services vms, but could not find anything for R4.0... So I had to revert to R3.2 for the time being.
> >
> > https://www.qubes-os.org/doc/salt/
> > Especially links at the bottom:
> > https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/README.rst
>
> Thanks. Will also try that tonight.

The way I recreated all sys-* service VMs was:

# You need to unset sys-firewall as netvm for AppVMs
# and unset sys-net as netvm for sys-firewall:
dom0$ qvm-prefs --set <qubes*> netvm ''
dom0$ qvm-prefs --set sys-firewall netvm ''

# The delete all service VMs qubes:
dom0$ qvm-remove sys-usb
dom0$ qvm-remove sys-firewall
dom0$ qvm-remove sys-net

# Make sure the right salt top files are enabled:
dom0$ sudo qubesctl top.enable qvm.sys-net
dom0$ sudo qubesctl top.enable qvm.sys-firewall
dom0$ sudo qubesctl top.enable qvm.sys-usb
dom0$ sudo qubesctl state.highstate

and that should be all. You have to restore 'sys-firewall' as netvm for all the qubes you want networking and thing should be pretty much as they were after install. Note that all new service VMs will have different IP address than their deceased counterparts, but firewall would be able to cope with that.

Ah! The joy! Now back to R4.0 without massive reinstall.

Thanks Marek, awokd and techg...!

On Fri, April 13, 2018 12:33 am, Pablo Di Noto wrote:

> Ah! The joy! Now back to R4.0 without massive reinstall.
>
>
> Thanks Marek, awokd and techg...!