[GSoC] Qubes-MIME-Handlers Weekly Progress Report #8

32 views
Skip to first unread message

Andrew Morgan

unread,
Aug 8, 2017, 7:30:19 AM8/8/17
to qubes...@googlegroups.com
Another report with some long-awaited news, enjoy!

Blog post with screenshots:
https://blog.amorgan.xyz/gsoc-weekly-progress-report-8.html

Otherwise text-only is below (with screenshots attached):

---

Hey there, welcome to another report! Since the last one, you may
remember that I had finished setting up the build environment for
Nautilus inside of GNOME Builder and was just getting started on
properly doing the patch for supporting file trust attributes. Well, I'm
happy to report that the patch is now finished! Screenshots and details
below.

## Working Demo

As screen recording in dom0 still isn't really doable yet, you'll have
to settle for some screenshots of it in action :)

![Double-clicking a trusted file opens
normally...](images/open-trusted.png)
*Double-clicking a trusted file opens normally...*

![Everyone knows gedit is the ultimate text
editor.](images/opened-gedit.png)
*Everyone knows gedit is the ultimate text editor.*

![Whereas an untrusted file summons a DispVM!](images/open-untrusted.png)
*Whereas an untrusted file summons a DispVM!*

![Virus free, yes-sir-ee!](images/opened-dispvm.png)
*Virus free, yes-sir-ee!*

This short demo makes use of a majority of all the different components
that have been worked on throughout the project. The now-patched
Nautilus makes a call to the patched NautilusPython which asks our
`qvm_trust` python extension to open a file, which it gives a response
based on what our cli tool, `qvm-file-trust`, says about it, and if it
is determined an untrusted file, `open-file-trust-based` will open it
securely in a DisposableVM. The contents of the file were never parsed
on the local machine!

The low-level part of this process was already working, but it took the
patch to Nautilus allowing us to set off the chain from the GUI to
really bring it all together. Let's talk about how that was done and how
it works.

## Nautilus and NautilusPython

By default, Nautilus does not support extensions written in Python, but
rather only those written in C. To get around this, a _Nautilus C
extension called NautilusPython_ was created. This extension has the
ability to provide a bridging interface from Nautilus' extension
infastructure to Python, handling any calls Python programs make to
Nautilus and calling them from C. Arguments and return variables are all
passed along seamlessly.

This is a really nice feature, as writing extensions in Python are about
90% less work than writing them in C, due to the large amount of
boiler-plate associated with setting up and tearing down an extension.
If you recall, the point of patching Nautilus was to allow extensions to
be able to be notified of when a file is being opened, and block that
request if necessary. I didn't want to subject myself or anyone else to
only being able to write an extension that made use of this new
functionality in C, thus patching NautilusPython as well as Nautilus
itself to add our new `file_open` function was necessary. So how was
each one done then?

Patching NautilusPython to add a new method was relatively simple, you
only need look at the existing methods that an extension can subclass,
find one that is functionally similar your own (I used
`update_file_info` as my guide), then insert your own method
(`file_open` in this case) at all the relevant points as well as create
the appropriate function bodies and set argument/return variables
appropriately.

So overall NautilusPython wasn't too bad really. The real time sink was
Nautilus itself.

## Patching Nautilus

First of all, I almost immediately ditched GNOME Builder after my last
blog post. It's not that it was a bad IDE, far from it actually, I quite
liked the interface and the various tools it contained. The ability to
download a project from git, package and run it inside a Flatpak, and
then send patches upstream all from within the editor was quite
appealing! The first major showstopper however, was that the version of
Nautilus that Qubes makes use of (v3.22.3), doesn't support Flatpak.
While building and running from GNOME Builder without Flatpak seemed
possible, no matter what I tried I could not get extensions to function
properly with the GNOME Builder version.

After a lot of fiddling about, I settled on what I probably should have
been using since the very beginning:
[qubes-builder](https://github.com/qubesos/qubes-builder)!

Not only does it build the correct version (and supports extensions),
but I knew that whatever I did with this build system would be exactly
what users would be using when they built Qubes' templates or got them
from the build servers. So, unfortunately no Flatpak'd builds, but I do
have pre-packaged RPMs with debug-symbols! That's probably even better,
right?

I'll post them in a separate blog post with instructions on how to set
everything up soon.

qubes-builder worked very well and was quite easy to drill into and mess
about with, as it is mostly just a collection of various scripts. I
eventually built a script that ran in dom0 that would build Nautilus in
my development VM, transfer it to my testing VM, delete the old version
and install the new version.

The only downside to this process was that building Nautilus took quite
a while each time (2-3m) which made quick changes quite an agonizing
process. I believe this was due to both having to rebuild _every_ file
each time (as they were packaged in a .tar.xz before being built,
meaning all files were considered to be modified) as well as a call to
`dnf` to fetch any updates was made, _every time_. I found out the
latter when working without internet and realizing my could would not
build! Each of these things probably could have been worked out or
changed if I had felt the need to, but I just stuck with the loading
times and usually had internet, so it was not too terrible.

Was _was_ terrible and time-consuming were a couple of very strange
issues I had during the creation of the patch.

The first was the dreaded _provider issue_. You see, every time Nautilus
calls a method that is meant to be subclassed by a Nautilus extension,
you must _provide a provider_. This can be an `INFO` provider, `MENU`
provider, and so on. Each method only accepts one of each type,
typically corresponding to the category of method you were calling. For
the `file_open` method I decided to stick it under the `INFO` provider
category, as that had to do with information about files, which I felt
'when they were opened' best fell into.

Well, these providers don't just pop out of nowhere. You have to create
them! You can do that by calling the intuitively-named
`nautilus_module_get_extensions_for_type` method, providing it the
`NAUTILUS_TYPE_INFO_PROVIDER` enum. It is then _supposed_ to return a
provider with a type that matches the one you gave it, but no matter
what I did it would always just return a `NULL` pointer! I spent _days_
on this problem, trying loads of different things. Every other call of
`nautilus_module_get_extensions_for_type` had no special setup or
initialization, leading me confused why my own calling of it brought
such different results.

After picking through many lines of code, talking with GNOME people over
IRC who were currently GUADEC'ing, and trying to understand the whole
GNOME development ecosystem, I eventually traced it all back to
NautilusPython not finding an extension that made use of a subclassed
method from the `INFO` method type. But wait, we _do_ have an extension
that does that with `qvm_trust`, so why doesn't it work??

Well I hit my head after this one folks. The version of NautilusPython
that Nautilus was loading in my test VM was the original, from-repo
version, not our modified version. **D'oh!**

"Well that's just a simple fix, I just have to build it and import it
into Nautilus in the testing VM!", I assumed, before realizing that
while my modified NautilusPython built, Nautilus _would not accept it_.
Nautilus would just _hang_ with little to no output.

After a few emails, Marek pointed me in the correct direction with the
`nautilus-3.0` branch of NautilusPython, which, while 6 years since the
last commit, worked with the latest version. Alright no problem. Rebase
my commits on that branch, rebuild, transfer the newly-built
shared-library over and bam, Nautilus runs again! And we have a provider
that's not NULL, hooray!

And then, ladies and gentlemen, problem #2 reared _its_ ugly head.

Here's the code to get a list (`GList*`) of providers to use with a
Nautilus extension:

providers = nautilus_module_get_extensions_for_type
(NAUTILUS_TYPE_INFO_PROVIDER);

While we were getting a _non-null_ provider value, the value we received
was _different_ from the one returned by the method we called. "Wat?"
you say, and I too said the same. For some reason we were returning a
64-bit `GList` pointer from `nautilus_module_get_extensions_for_type`,
but storing a 32-bit `GList` pointer in `providers`.

I tried all sorts of weird tricks to get this fixed. Nobody I talked to
seemed to have a solution, and after wrestling with it for a few days I
begrudingly settled on this very elegant workaround:

// TODO: Providers ends up as 0x55f65ac0
// when method is returning 0x555555f65ac0, very odd...
providers = (GList*) (nautilus_module_get_extensions_for_type
(NAUTILUS_TYPE_INFO_PROVIDER) + 0x555500000000);

Yes I know this is terrible. It did work though, and I needed to stop
hitting this strange bug and get on with patching, so that's where it
currently sits. I'm still actively looking for a proper solution. If
anyone knows why a 64bit pointer might get truncated to a 32bit pointer
after a function call in C please do let me know! You can find the full
code
[here](https://github.com/anoadragon453/nautilus/blob/master/src/nautilus-mime-actions.c#L2496)
and the `type` method
[here](https://github.com/anoadragon453/nautilus/blob/master/src/nautilus-module.c#L269).

After this was out of the way, and a handful of more methods were
implemented, we finally had a working solution between the `qvm_trust`
extension and our patched Nautilus! It was beautiful, really.

## Conclusion and going forward

My [email](https://amorgan.xyz/upstream-request.html) to the [Nautilus
mailing list](https://mail.gnome.org/archives/nautilus-list/) is still
awaiting moderation. GUADEC has now concluded, and with no update on the
situation it seems I need to do some more active poking of people to get
moderation approval. We'll see where that lands, the code still needs to
be cleaned up a bit before sending it upstream anyways, so I'm not too
worried.

At this point in the project, we essentially have the following tasks left:

* Patch KDE Dolphin to provide the same functionality as Nautilus now does
* Finish up some small tasks with the daemon like logging and some more
interfacing with the cli tool
* Code cleanup, linting, tests
* Integrate into main QubesOS repos

I'm going to prioritize the latter 3 bullet points at this point to
ensure we get a really solid core implementation down of the project. I
don't want to spend another couple weeks patching Dolphin only to run
out of time for a general cleanup and other polish. I will try to pour
through the Dolphin code and note down everything that one would need to
do with Dolphin to make it work with file trust attributes, so that in
the future either myself or someone else can give it a shot.

It may end up that Dolphin already has most of the functionality and the
patch will be very small, we'll see!

Anyways, for now I'm going to start on finishing the remaining small
tasks, and will post any questions regarding packaging and/or cleanup to
the qubes-devel mailing list.

As always, you can find the code
[here](https://github.com/anoadragon453/qubes-mime-types).

And as of now, you can also find the patched code for
[Nautilus](https://github.com/anoadragon453/nautilus) and
[NautilusPython](https://github.com/anoadragon453/nautilus-python/tree/nautilus-3.0)
on Github as well. Make sure that when you're inspecting the
NautilusPython code, you're looking at the `nautilus-3.0` branch. That's
where all the important commits are!

That's all for now, thanks for reading!

open-trusted.png
opened-gedit.png
open-untrusted.png
opened-dispvm.png
signature.asc

Andrew Morgan

unread,
Aug 9, 2017, 10:28:35 AM8/9/17
to qubes...@googlegroups.com
Marek,

The git commits for nautilus/python seem like a bit of a mess at the
moment. Are you currently reviewing them individually or can I squash
them into just a few and force push to the repo?

Andrew Morgan

signature.asc

Marek Marczykowski-Górecki

unread,
Aug 9, 2017, 11:04:00 AM8/9/17
to Andrew Morgan, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Feel free to squash them.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZiyRaAAoJENuP0xzK19csug0H+wRi9RLWQWHmMBznsWUFquvL
qF4UOKrUpTd463fUGdzOIIWqlox9gciu7X3mOIsXBmoCTyt9ouq2dsHV80WSpku/
9Wn/smKf01jQNYgRq9ZJ80jOU3TjAZcTwMgG1KynHPTKOJHf4EBQmCsKIjZtyAVk
DbnkHIFxb0npUIFqUqr7qJF/f9Sx0WgTR3LlHrbmIIkXHPr01K9YN/SGqIJrw8Wh
JWV6m0mJHjQXSnJccG58FbLBdA0J/Mf0T20mV+5vQKj1QwH0bOci6+V7PMC9HKN2
xEGPR8dF14gQvwEq1EHWN8kdkjD00ttM1eTdzz4CAUzPyLGsAppP9lGwTsd2B84=
=92f+
-----END PGP SIGNATURE-----

Andrew Morgan

unread,
Aug 9, 2017, 3:03:07 PM8/9/17
to qubes...@googlegroups.com
Hey Marek,

Nautilus has been squashed into one, easily-reviewable commit:
https://github.com/anoadragon453/nautilus/commit/1325725451220447f53e5ab3016423e6d3372682

Currently doing the same for nautilus-python.

Thanks,
Andrew Morgan

signature.asc

Andrew Morgan

unread,
Aug 9, 2017, 4:14:17 PM8/9/17
to qubes...@googlegroups.com
NautilusPython commit has been cleaned up to the best of my abilities
now too:
https://github.com/anoadragon453/nautilus-python/commit/6a9c0a1c4137eb23202e3dab2b8ceaf9bc0eb8c0

Andrew Morgan

signature.asc

Andrew Morgan

unread,
Aug 20, 2017, 12:55:15 AM8/20/17
to qubes...@googlegroups.com
Jean-Philippe,

The top first following commits on these lists are good for code review...

https://github.com/anoadragon453/nautilus-python/commits/file_open
https://github.com/anoadragon453/nautilus/commits/master

NOTE: There's still that crappy work-around in the second link there. It
causes Nautilus to crash unless it's run from gdb (as gdb has 0x5555
pre-pending all/most pointer addresses. Still not sure why the 32-bit
truncation happens).

These files could use some review:

https://github.com/anoadragon453/qubes-mime-types/blob/master/qvm_trust.py
https://github.com/anoadragon453/qubes-mime-types/blob/master/qubesfiletrust/qvm_file_trust.py

Still known issues:

* GDB workaround (Big blocker!)
* update_file_info is somehow called alongside file_open by our
extension. Likely since file_open was implemented alongside
update_file_info in libnautilus-extension, however we never explicitly
call it, so not sure why it's complaining about it. No known reduced
functionality however.
* Nautilus complains something in our extension is not actually a
MenuItem. No known reduced functionality.

Those last couple issues aren't too relevant to the code review, but
just wanted to note them down somewhere. :P

Thanks!
Andrew Morgan

signature.asc

Jean-Philippe Ouellet

unread,
Aug 20, 2017, 3:39:40 PM8/20/17
to Andrew Morgan, qubes-devel
ACK.

Will begin reviewing offline.

Regards,
Jean-Philippe
Reply all
Reply to author
Forward
0 new messages