salt management issues since last qubes (dom0?) upgrade

[balin]

I'm managing my fleet of templates and VMs using the salt stack. Since the most recent qubes upgrade this no longer works.

I'm not at the machine in question right now, but the error that shows up is one of "unclean data returned" and according to the logs related to an ssh wrapper component.

Does anyone have insight into this? What info does it take to figure this out?

[balin]

Ok. More detail:

When calling (as root and in dom0):

    qubesctl --target=fedora-32 state.highstate,

I get

    fedora-32: ERROR (exit code 20, details in /var/log/qubes/mgmt-fedora-32.log)

The relevant part in the referenced log reads:

    2020-11-13 19:27:51,237 calling 'state.highstate'...
    2020-11-13 19:28:13,123 output: fedora-32:
    2020-11-13 19:28:13,125 output:     ----------
    2020-11-13 19:28:13,125 output:     _error:
    2020-11-13 19:28:13,125 output:         Failed to return clean data
    2020-11-13 19:28:13,125 output:     retcode:
    2020-11-13 19:28:13,125 output:         1
    2020-11-13 19:28:13,125 output:     stderr:
    2020-11-13 19:28:13,125 output:         Traceback (most recent call last):
    2020-11-13 19:28:13,125 output:           File "/usr/lib/qubes-vm-connector/ssh-wrapper/ssh", line 101, in <module>
    2020-11-13 19:28:13,125 output:             sys.exit(main())
    2020-11-13 19:28:13,125 output:           File "/usr/lib/qubes-vm-connector/ssh-wrapper/ssh", line 94, in main
    2020-11-13 19:28:13,125 output:             return ssh(args)
    2020-11-13 19:28:13,125 output:           File "/usr/lib/qubes-vm-connector/ssh-wrapper/ssh", line 29, in ssh
    2020-11-13 19:28:13,126 output:             assert args[1] == '/bin/sh'
    2020-11-13 19:28:13,126 output:         AssertionError
    2020-11-13 19:28:13,126 output:     stdout:
    2020-11-13 19:28:13,126 exit code: 20

What is going on here? why did this stop working?

[Viktor Ransmayr]

Just an additional confirmation from my side: I receive the same error message, when  I  try to do the same via the (GUI) Qubes Updater.

This problem started yesterday morning on my system.

[Ludovic Bellier]

Le 13/11/2020 à 19:33, balin a écrit :
> Ok. More detail:
>
> When calling (as root and in dom0):
>     qubesctl --target=fedora-32 state.highstate,
> I get
>     fedora-32: ERROR (exit code 20, details in
> /var/log/qubes/mgmt-fedora-32.log)
>
> The relevant part in the referenced log reads:

>     ...

>     2020-11-13 19:28:13,125 output:         Traceback (most recent
> call last):
>     2020-11-13 19:28:13,125 output:           File
> "/usr/lib/qubes-vm-connector/ssh-wrapper/ssh", line 101, in <module>

>     ...

>
>
> What is going on here? why did this stop working?

Hi Balin,

    it's a known temporary issue, read this:

https://qubes-os.discourse.group/t/updating-vms-throwing-an-ssh-error-any-ideas/1397

[balin]

Thank you for pointing this out. the one liner solution in https://github.com/QubesOS/qubes-issues/issues/6188#issuecomment-726373728 (as applied to the default management template) fixed things for me.