Qube Manager dbus event handling

[donoban]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

I saw Qube Manager should handle dbus signals from 'DomainManager1'
for a fast and cheap refreshing of VMs status.

Is anybody currently working on this? If not I think I could do it. I
started playing with pydbus and I think the current code doesn't need
a very big rewrite for handling all relevant events.

Regards.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEznLCgPSfWTT+LPrmFBMQ2OPtCKUFAlrrF/oACgkQFBMQ2OPt
CKU8eRAAt+lOVzcDAjY7lVk88QCnG/SUM/z30tfY9erOAc+Ctzra7Xt5WjZhleEx
dpTMxFAUxbH4Ax7OCAySAWSRBfbt12+WDBoy74rHtjH5NyviWv8L6mcgdtZRMp9d
dhqXsE3lCMWzfEm5jeQiBgv7p3I9p/9mw1xWoxuNwSn9khttYAZ9YLYYjSn54Zay
i/yn3TVO9dmhdOqmF28HBNTHsxd2tMPnjS3Ip/TZREi5Z89gW433YHxhbLVAknXr
SyJUDQ6TJbugvunegGnDdTSs4inCb6RxUPLDgbj+7MbWGo+0KtmYVGtmHpdq6QDO
Hn3PkxI0FLztzKlhh94AJzA5iCyMKi18E2NrVWgP1Mv5930bRW7Wf/ZjgM32vT7m
lJ9v6Y+l1CuyLd4JF6rWWSGnsolEdCmiQo7CUzoj0uj2lQwoyItcqDOBM5zodPof
DESbzn0zobnqLujQaZLD/tq4kYqjUjdS7I+iMAACqMnKHc2S3HjeZbAy6VDeo78V
jNTPAywMx0WgArLPELO9VE3mB3XNLTT+RDIZYQgOWcgbIhDornQpZxhvmnT2+hWU
5b71Z1TYcn7Vc6vLLzNC8SqejDAFytiH9XV3MFedecSsC8MktkY3dSxIBGDCAFgv
A5yyTEcBJPvu4BSOM7MAthNYg8P+E/IDjGpeVQDd77H+ICGiyZw=
=uFZ6
-----END PGP SIGNATURE-----

[donoban]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/03/18 16:08, donoban wrote:
> Hi,
>
> I saw Qube Manager should handle dbus signals from
> 'DomainManager1' for a fast and cheap refreshing of VMs status.
>
> Is anybody currently working on this? If not I think I could do it.
> I started playing with pydbus and I think the current code doesn't
> need a very big rewrite for handling all relevant events.

I am near achieve this. However I see a little problem with current API.

Dbus objects relative to Qubes domains use 'qid' as identifier, like:

/org/qubes/DomainManager1/domains/13

but VMCollection uses VM names as key so I can't retrieve directly an
vm object using a 'qid'.

This only affects to domain creation (for the other cases the vm was
already loaded into a dict) but maybe would be nice to have a direct
way to get a vm by 'qid'.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEznLCgPSfWTT+LPrmFBMQ2OPtCKUFAlrrZ/8ACgkQFBMQ2OPt
CKU3jA//VspAX17VmL6xvcFp8ztZRevJSrztYL/ELUzBBRW9NfsqoNeEfbV4XImo
B2/c230pLRYXUu3dvCjqLJrxF0f50GLl2IhoYEPb+8OrUFvWPyG83sQ2QbQXfRdx
iMnIbmVDHOvBMFHDE+KGMuv4/+b3ke1rmDC8o6ZR4HMeADyb0pPE5M4xcLUe4N70
lPso9ujI2nwY3TOWS9vkUsTyKHz8fl93hwrB9MMNHuOeML+qpy35PQN4/J1Dn3UX
XZfwjyDUINttXE2e+pFMJuzLyiRvcvzt9yL7Fe+IVSTx2oMLEmaIzoD7tP/n9SLx
/cvkyPE3jQuoU8EW9+isp+Pcl2qnJUYy6HZdomoKoeZRIf2HGfUxjPLHZYWjHWj1
iLqSQ2mjnaVSwBw+YsTirr0IVNdZ1h0YA4tGfFGPzztihQqtHK7wOLyEvgDbcrzl
14OuIuAlXd4qmft1x3CQCjsILudU9FKs7dqsQU44181k6boxC9EFJjS3Sse65KKU
E/WiLmorAvXjsHTSvcqwmaNdumCZsDmDHImIexQs2AAwGq+l7l0wfIJS2URheynd
JJvkJTw3lEjGLjp7pYwhBuKpLSNpJ6A00+45NG7lD60AJvnbBG3D1ioADWldKZRu
ZJpnqBlXcFYx6qB5Y8Z+I3S9aAkLeTgPX5wToZlu0H01v5Nr67I=
=Dat+
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, May 03, 2018 at 09:50:26PM +0200, donoban wrote:
> On 05/03/18 16:08, donoban wrote:
> > Hi,
> >
> > I saw Qube Manager should handle dbus signals from
> > 'DomainManager1' for a fast and cheap refreshing of VMs status.
> >
> > Is anybody currently working on this? If not I think I could do it.
> > I started playing with pydbus and I think the current code doesn't
> > need a very big rewrite for handling all relevant events.
>
> I am near achieve this. However I see a little problem with current API.
>
> Dbus objects relative to Qubes domains use 'qid' as identifier, like:
>
> /org/qubes/DomainManager1/domains/13
>
> but VMCollection uses VM names as key so I can't retrieve directly an
> vm object using a 'qid'.
>
> This only affects to domain creation (for the other cases the vm was
> already loaded into a dict) but maybe would be nice to have a direct
> way to get a vm by 'qid'.

There is ongoing work on domains widget to use qubesadmin module
directly (skipping dbus layer). Copying Marta, who is working on it.

Long story short - dbus services was introduced before we've decided to
implement Admin API in 4.0 (initial plan was to have it in 4.1 or
later). But since we've done it in 4.0, there is very little sense to
use dbus, which serve similar purpose (provide info and events about
domains), but adds additional complexity.

As for Qube Manager, there is a little problem - events handling in
qubesadmin is based on asyncio python module. At the same time, both GTK
and Qt use its own event loop. We have it worked out for GTK, using
gbulb module[1][2], it's simple. But we haven't tried it with Qt. In
theory there is quamash module for it, so hopefully that's simple too.
One catch is that, similar to gbulb, there is no package for it in
Fedora... For gbulb we've created one[6], probably the same needed for
quamash.

By using qubesadmin directly, you avoid this qid problem and
additionally things are simpler and there is less places where state can
get out of sync.

Events handling in qubesadmin: docs[4], example usage[5]:

events_dispatcher = qubesadmin.events.EventsDispatcher(args.app)
events_dispatcher.add_handler('backup-progress',
functools.partial(print_progress, profile_name))
events_task = asyncio.ensure_future(
events_dispatcher.listen_for_events())

List of domain related events can be found here:
https://dev.qubes-os.org/projects/core-admin/en/latest/qubes-vm/qubesvm.html

[1] https://github.com/nathan-hoad/gbulb
[2] https://github.com/QubesOS/qubes-dbus/blob/master/qubesdbus/service.py#L37
[3] https://github.com/harvimt/quamash
[4] https://dev.qubes-os.org/projects/core-admin-client/en/latest/qubesadmin.events.html#module-qubesadmin.events
[5] https://github.com/QubesOS/qubes-core-admin-client/blob/master/qubesadmin/tools/qvm_backup.py#L195-L199
[6] https://github.com/QubesOS/qubes-linux-gbulb

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlrro5UACgkQ24/THMrX
1yySSwf/TZJhCqBBa07Pyev6tW5p9ikH/I0AvVYrFuIoewcJziyQjfyJQDWjKYjx
EHwzfYIdMr8oaa5reaJhcvEG6x/G0WkRd0SFWhZi03evAPtahrnvqwGfZmzMMruF
YaoMeRqvqHXTqLYAu+T4BAxsTXrKwnAh1lxEeYqJWw4aQcQu5b+r130ZkpdTDVqy
2d8iNR/OM6fY4M+F6Hy0Z6cj3EtbdTAsI7na0y4UZPKbt3ZmC6ixK097BYfIbs+b
2xuGdpfdfoEiH9j2CFv9TVAzLotpcs/0oJ8Y2bjC22cDUkCDLkbZ+JzsoZsv8ezQ
7XDTJp4u8Y8XMZWcv+lvJeMbGaSL1Q==
=0rMb
-----END PGP SIGNATURE-----

[donoban]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/04/18 02:04, Marek Marczykowski-Górecki wrote:> There is ongoing

work on domains widget to use qubesadmin module
> directly (skipping dbus layer). Copying Marta, who is working on
> it.
>
> Long story short - dbus services was introduced before we've
> decided to implement Admin API in 4.0 (initial plan was to have it
> in 4.1 or later). But since we've done it in 4.0, there is very
> little sense to use dbus, which serve similar purpose (provide info
> and events about domains), but adds additional complexity.
>
> As for Qube Manager, there is a little problem - events handling
> in qubesadmin is based on asyncio python module. At the same time,
> both GTK and Qt use its own event loop. We have it worked out for
> GTK, using gbulb module[1][2], it's simple. But we haven't tried it
> with Qt. In theory there is quamash module for it, so hopefully
> that's simple too. One catch is that, similar to gbulb, there is no
> package for it in Fedora... For gbulb we've created one[6],
> probably the same needed for quamash.
>
> By using qubesadmin directly, you avoid this qid problem and
> additionally things are simpler and there is less places where
> state can get out of sync.

Thanks for your attention Marek. Bad moment for start doing this hehe
but was funny :)

At least I guess the structure of Qube Manager should be very similar
using dbus or another events system so I can try to finish it which is
nearly finished. Probably will be a better place to start a migration
to qubesadmin events than current version.

There are some things that can't be handled with dbus
(network/template changes) and other which could be but I don't know
if you want (CPU usage, memory...) also I think that there is an
strange bug with qubesadmin[1]

So if nobody is working on Qube Manager migration at this moment I
would like to finish the dbus-version and then consider the other api :)

Regards.


[1] https://github.com/QubesOS/qubes-issues/issues/3875
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEznLCgPSfWTT+LPrmFBMQ2OPtCKUFAlrrqNIACgkQFBMQ2OPt
CKX+yxAAj25/RFtbDhf7FFGeIEAbAx35HhX5Rr87HxgL50n+Qp2URiZ3Q0LWCCHk
56mf3p3m+O62BHqJ/oX7MI3t6FRR28/8SZK+Y6crjFlzuW6zqhCEklACmgrs4sjj
ftKu/nC6pZKyk1nCKe5Vqde51TD+4xw3iS/1qYcul8vgQ653h8FWg+zuG8w3GlcW
iMCLZ0va2IsO6uF68jIFKk55pHI5ZJhD3cpx5Ljq34YKbNB5mscwZpQoIlG8csPa
1lBpuh8PrWJVpdAAO41ifYSFXa7h+RTuoxn+30Pz71ykmgTWJEgtu1sJIFjNt/M0
t7V5SNqbDcjAAxJIZHBkk0AZpmZIjglgL7sy4fx9FveGeOw2f9yGnEFqwSAjCPyR
KVep08VF3oX0aiEUZsQ97uQTtAM/60HngiqRKDmamyQ+euMPIUU7KM1huabvAF+u
9npMOx4f8PoaMeu2h0wd7QicW4/X69iZcgXyVSHy3XuP2ogx2MQNHXEn0oiZyMrA
q9MZZkU0BdBlN1pvo+t7lJWpSVlfOCM/wA4Dbh6raYW5iXKfAo9TGDzvDHrHHLSN
wB+Nabsw7yac5G53hPKLM5F7+hO/lPPhEbM1KLGNE4uYhevSqDFIPHT1CBQvc+/k
QUY0LqW/US0oP8i7bLVtF/MsXKiESgkR3fPLsH/GpQ7ogbY8fM4=
=ARU3
-----END PGP SIGNATURE-----

[morf...@gmail.com]

Is the issue described in this thread responsible for the Qube Manager no longer showing live status updates of the Qubes?

[awokd]

On Thu, July 19, 2018 1:11 am, morf...@gmail.com wrote:
> Is the issue described in this thread responsible for the Qube Manager no
> longer showing live status updates of the Qubes?

Yes, but the good news is the dbus version should show live updates again.
If you upgrade dom0 to current-testing, you can try it out right now.

[morf...@gmail.com]

Sweet status updates batman, it works!

Thanks very much, that is halfway home. Now I am off in search of restoring the CPU/RAM stats for each Qube to the QubeManager, which sounds like its going to be a lot tougher!

[Chris Laprise]

On 07/30/2018 08:19 PM, morf...@gmail.com wrote:
> Sweet status updates batman, it works!
>
> Thanks very much, that is halfway home. Now I am off in search of restoring the CPU/RAM stats for each Qube to the QubeManager, which sounds like its going to be a lot tougher!
>

FWIW, here is how I collect CPU + RAM using xentop output:

https://github.com/tasket/Qubes-scripts/blob/master/system-stats-xen

The qubesadmin API doesn't seem to cover these resources, so you could
try the above or go through libvirt-python.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[morf...@gmail.com]

This is fantastic, thank you for sharing this Chris.