The future of "remote XUL"

[Boris Zbarsky]

Background: We currently have various provisions for "remote XUL",
wherein a hostname is whitelisted to:

1) Allow parsing XUL coming from that hostname (not normally alllowed
for the web).

2) Allow access to XPConnect-wrapped objects, assuming someone hands
out such an object.

3) Run XBL JS in the same global as the webpage.

4) Allow access to a "cut down" Components object, which has
Components.interfaces but not Components.classes, for example.

This machinery is also used for the "dom.allow_XUL_XBL_for_file" preference.

The question is what we want to do with this going forward. From my
point of view, I would like to eliminate item 4 above, to reduce
complexity. I would also like to eliminate item 2 above, because that
would get us closer to having the invariant that XPConnect is only used
from system principals. These two changes are likely to break some
remote XUL consumers.

The question is whether we should just go ahead and disable remote XUL
altogether, modulo its usage in our test suites and maybe
"dom.allow_XUL_XBL_for_file" (for local testing). It seems like that
might be clearer than a dribble of breakage as we remove bits like items
2/4 above, slowly remove various bindings, slowly remove support for
some XUL tags, etc...

Thoughts? My gut feeling is that we should just turn off remote XUL
outside the IsInAutomation() and maybe the "dom.allow_XUL_XBL_for_file"
case.

-Boris
_______________________________________________
firefox-dev mailing list
firef...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev

[Dave Townsend]

My understanding is that it has been effectively unsupported for some time anyway so I think we should just go ahead and disable it altogether at this point. If we need bits for automated tests then we should work to switch tests away from those if we can.

[Eric Shepherd (Sheppy)]

I would agree that going all-out and disabling remote XUL entirely makes the most sense, except in the cases you mention.

The one potential exception: would it make sense to allow it to be enabled (with it disabled by default) on copies of Firefox set up with Policy Engine, to allow those users the option?

--

Eric Shepherd
Senior Technical Writer
Mozilla
Blog: http://www.bitstampede.com/
Twitter: http://twitter.com/sheppy
Check my Availability

[Nicholas Alexander]

Hi Boris!

I am not expert in this area, but this sounds like a vestigial feature of the Mozilla XUL application layer that should be removed immediately.  Can you elaborate on:

- some of the details of "likely to break remote XUL consumers"?  Which consumers are these -- internal?  External?

- do we have an estimate of how much remote XUL is used in our own test suite?  Is this days/weeks/months of labour to replace?

- do we have any idea of the popularity of `dom.allow_XUL_XBL_for_file`?  Do we expect this usage is all internal?  (I really hope so!)

Sorry to ask for work (before you do the real work),

Nick

[Alex Gaynor]

What was the original intended use case for remote XUL, powerful origins controlled by Mozilla, or enabling developers to build their own powerful origins?

Alex

[Boris Zbarsky]

On 3/27/18 11:49 AM, Alex Gaynor wrote:
> What was the original intended use case for remote XUL, powerful origins
> controlled by Mozilla, or enabling developers to build their own
> powerful origins?

The latter. In particular, it was used for intranet apps.

Note that the remote XUL origins are not terribly powerful. They do
_not_ allow arbitrary XPCOM access, for instance... They do have more
attack surface than normal web pages, of course.

[Boris Zbarsky]

On 3/27/18 11:50 AM, Eric Shepherd (Sheppy) wrote:
> The one potential exception: would it make sense to allow it to be
> enabled (with it disabled by default) on copies of Firefox set up with
> Policy Engine, to allow those users the option?

That would require maintaining the remote XUL infrastructure we have
now, so doesn't seem like a win from the point of view of code or
invariant simplification...

[Alex Gaynor]

Do we have any telemetry on whether people actually use it?

Deleting it entirely seems like the happiest course of action, and if there's extremely low usage that just makes it easier :-) Are there any reasons to want to keep it around, besides backwards compatibility?

Alex

[Boris Zbarsky]

On 3/27/18 12:39 PM, Alex Gaynor wrote:
> Do we have any telemetry on whether people actually use it?

I don't believe we do.

> Are there any reasons to want to keep it around, besides backwards compatibility?

Ease of writing tests and debugging XUL stuff (hence the thought to
maybe keep it for the "allow XUL for file://" case).

Apart from that, I can't think of anything.

[Brian Grinstead]

Wouldn't keeping it for file:// require us to maintain the infrastructure, or is there something that makes it worse in the whitelisted domain case? Also, do we load pages in the content process in both cases?

I’m inclined to say we should remove it for the file:// as well, given comments like https://bugzilla.mozilla.org/show_bug.cgi?id=1442006#c7 and the fact that it will continue to get more broken over time (for example, as we remove XBL bindings). It’s not nearly as convenient, but test cases can still be created by registering the xul file as a chrome uri and loading it in a tab.

Brian

[Axel Hecht]

Also, many of these users might be on institutional ESR deployments with telemetry off. So even if we had probes, they might not send data to us.

We could reach out to the enterprise list, though.

Axel

Am 27.03.18 um 18:39 schrieb Alex Gaynor:

[Boris Zbarsky]

On 3/27/18 11:50 AM, Nicholas Alexander wrote:
> - some of the details of "likely to break remote XUL consumers"?  Which
> consumers are these -- internal?  External?

This is referring to my proposed XPConnect/Components changes, right?

External. We have no internal consumers of remote XUL outside our test
suite, and obviously I would fix the tests as needed (there's not much
needed).

> - do we have an estimate of how much remote XUL is used in our own test
> suite?

https://searchfox.org/mozilla-central/search?q=.xul&case=false&regexp=false&path=mochitest.ini
shows 8 mochitests using it.

https://searchfox.org/mozilla-central/search?q=.xul&path=reftest.list
shows 280-ish reftests using it, mostly testing actual XUL layout
functionality.

https://searchfox.org/mozilla-central/search?q=.xul&case=false&regexp=false&path=crashtests.list
shows ~170 crashtests that are XUL files.

https://searchfox.org/mozilla-central/search?q=%3Cbindings&path=test
shows ~220 XBL files used in tests, though some of those overlap with
one of the earlier lists (e.g. XBL being pulled in by a .xul test).
Presumably we need to address these anyway for XBL removal.

There might be some other bits I'm missing (e.g. browser tests loading
XUL files in tabs or something), but the above seems like a bare minimum
estimate for the usage.

> Is this days/weeks/months of labour to replace?

Definitely not days. Probably doable in "weeks", though a lot of that
depends on what we replace with. Presumably chrome tests loaded with
system principal to replace the mochitests (including the ones using
XBL); that one is not hard to do, but will need auditing the tests to
make sure they still make sense in the aftermath. For the reftests and
crashtests, we need to figure out the right replacement.

> - do we have any idea of the popularity of
> `dom.allow_XUL_XBL_for_file`?  Do we expect this usage is all internal?
> (I really hope so!)

I don't, and I really hope so too....

> Sorry to ask for work (before you do the real work),

Nah, all your questions are totally reasonable. ;)

[Eric Shepherd]

 
 

On Tue, Mar 27, 2018 at 12:33 PM, Boris Zbarsky <bzba...@mit.edu> wrote:

On 3/27/18 11:50 AM, Eric Shepherd (Sheppy) wrote:

The one potential exception: would it make sense to allow it to be enabled (with it disabled by default) on copies of Firefox set up with Policy Engine, to allow those users the option?

That would require maintaining the remote XUL infrastructure we have now, so doesn't seem like a win from the point of view of code or invariant simplification...

​Yes, that's true. My thinking is that a first step could be to change it so it had to be enabled using Policy Engine, and have an ESR release with that, with a notice to users of ESR that that will be the last ESR release to support remote XUL. Then use the following cycles to remove the functionality entirely.​

Sheppy

[Boris Zbarsky]

On 3/27/18 12:56 PM, Brian Grinstead wrote:
> Wouldn't keeping it for file:// require us to maintain the infrastructure, or is there something that makes it worse in the whitelisted domain case?

It's a question of what guarantees we offer for XUL working.

Keeping it for file:// requires keeping some of the infrastructure for
parsing XUL and maybe loading XBL if the pref is flipped.

It does not require us to provide access to Components.interfaces or
allow exposure of XPCOM things in those file:// URLs, as long as we're
OK with "not all XUL" working in the file:// context. Which we are.

Supporting remote XUL in general, on the other hand, involves not
breaking compat with already-deployed stuff which may be relying on the
XPCOM object exposure.

So we could have meaningful code removal and invariant strengthening
even while keeping the file:// case. Certainly until we sort out our
test suites there's no _extra_ win from removing the file:// case in
terms of code removal (though again invariants can maybe be somewhat
stronger if we do remove it).

> Also, do we load pages in the content process in both cases?

I believe we do, but I have not verified.

> I’m inclined to say we should remove it for the file:// as well

OK. Just to be clear, if we expect layout hackers to debug XUL issues,
the file:// case is really useful for that, because it allows using the
layout debugger. I we promise to not make them do that, we can probably
remove the file:// case.

> It’s not nearly as convenient, but test cases can still be created by registering the xul file as a chrome uri and loading it in a tab.

Can't do that in the layout debugger, yes?

[Boris Zbarsky]

On 3/27/18 1:02 PM, Eric Shepherd wrote:
> ​Yes, that's true. My thinking is that a first step could be to change
> it so it had to be enabled using Policy Engine, and have an ESR release
> with that

So presumably targeting 60? The timing is a bit tight...

Note that as things stand the only way to enable remote XUL for a
hostname is by adding permission manager entries. This needs to either
be done by a non-webextension extension, or by running code in a
privileged context (e.g. browser console) or by directly munging the
sqlite database involved. This last is what we do in our automation.

> with a notice to users of ESR that that will be the last ESR
> release to support remote XUL.

We can do that anyway, for 60, without changing how it's enabled, right?

[Brian Grinstead]

> On Mar 27, 2018, at 9:57 AM, Boris Zbarsky <bzba...@mit.edu> wrote:
>
> On 3/27/18 11:50 AM, Nicholas Alexander wrote:
>
>> - do we have any idea of the popularity of `dom.allow_XUL_XBL_for_file`? Do we expect this usage is all internal? (I really hope so!)
>
> I don't, and I really hope so too....

This isn't totally representative, but based on github code search:

“dom.allow_XUL_XBL_for_file” - 330 code results - https://github.com/search?utf8=%E2%9C%93&q=dom.allow_XUL_XBL_for_file&type=Code

And lot of those are duplicates that look like they come from files in mozilla-central:

"dom.allow_XUL_XBL_for_file filename:nsContentUtils.cpp” - 194 code results -https://github.com/search?utf8=%E2%9C%93&q=dom.allow_XUL_XBL_for_file+filename%3AnsContentUtils.cpp&type=Code
“dom.allow_XUL_XBL_for_file filename:reftest-preferences.js” - 89 code results - https://github.com/search?utf8=%E2%9C%93&q=dom.allow_XUL_XBL_for_file+filename%3Areftest-preferences.js&type=Code
“dom.allow_XUL_XBL_for_file filename:ContentPrefs.cpp” - 23 code results - https://github.com/search?utf8=%E2%9C%93&q=dom.allow_XUL_XBL_for_file+filename%3AContentPrefs.cpp&type=Code

If you ignore those that would leave only 24 remaining. I can’t figure out how to group by filename or filter on ‘not in these filenames’ in github search so it’s hard to see exactly what’s left.

[Eric Shepherd (Sheppy)]

On Tue, Mar 27, 2018 at 1:09 PM, Boris Zbarsky <bzba...@mit.edu> wrote:

On 3/27/18 1:02 PM, Eric Shepherd wrote:

​Yes, that's true. My thinking is that a first step could be to change it so it had to be enabled using Policy Engine, and have an ESR release with that

So presumably targeting 60?  The timing is a bit tight...

​Yeah, I suppose. Otherwise you have to wait quite a while. But I think if you were going to make it deprecated for ESR 60, it should be doable, if the decision were made quickly enough.​

 

Note that as things stand the only way to enable remote XUL for a hostname is by adding permission manager entries.  This needs to either be done by a non-webextension extension, or by running code in a privileged context (e.g. browser console) or by directly munging the sqlite database involved.  This last is what we do in our automation.

with a notice to users of ESR that that will be the last ESR release to support remote XUL.

We can do that anyway, for 60, without changing how it's enabled, right?

​In that case, yeah, I would think that if you have to add a permission for it, you're probably okay -- unless there's a desire to have it together with the other Policy Engine stuff just for consistency purposes when customizing Firefox for enterprise use.​

[Brian Grinstead]

OK given all that, I agree with your initial post that we should think about domains and local files/automation separately, since turning it off for domains would be more beneficial and also less work. I’d still like to track what needs to be done to disable it entirely - so far it sounds like migrating existing tests and supporting local development use cases like the layout debugger.

Brian

[Mike Kaply]

The largest and most complex remote XUL app I know of is called Qualoupe and as far as I know, things have already started breaking them before 60.

I've asked them to comment in this conversation.

Mike

[Kris Maglione]

On Tue, Mar 27, 2018 at 03:48:24PM +0000, Dave Townsend wrote:
>My understanding is that it has been effectively unsupported for some time
>anyway so I think we should just go ahead and disable it altogether at this
>point. If we need bits for automated tests then we should work to switch
>tests away from those if we can.

This has been my understanding too. When we removed support for
remote XUL by default, and then eventually moved it to a site
whitelist, those were both stepping stones to removing support
entirely. But it's been officially unsupported since the first
step, and there hasn't been any way to enable it since we
removed legacy extension support.

It's also been progressively breaking over time, and I've been
WONTFIXing bugs about remote XUL with prejudice.

If whatever support is left is causing development friction, we
should just remove it.

[Botond Ballo]

On Tue, Mar 27, 2018 at 11:48 AM, Dave Townsend <dtow...@mozilla.com> wrote:
> My understanding is that it has been effectively unsupported for some time
> anyway so I think we should just go ahead and disable it altogether at this
> point. If we need bits for automated tests then we should work to switch
> tests away from those if we can.

Some rendering codepaths are only exercised by XUL.

For example, in bug 1205630 I fixed a tab-bar rendering issue, and I
had to write a XUL reftest as my regression test, because that's the
only thing I could get to exercise the affected code. I think it's
important that, as long as our browser UI uses the relevant XUL
features, the corresponding test coverage remains in place.

In turn, being able to load a XUL page locally (which I've been doing
with "dom.allow_XUL_XBL_for_file", not sure if there's another way),
is useful for writing such a XUL test.

Cheers,
Botond

[Mark Crutch]

Mike Kaply wrote:

> The largest and most complex remote XUL app I know of is called Qualoupe
> and as far as I know, things have already started breaking them before 60.
>
> I've asked them to comment in this conversation.

I'm the lead developer for Qualoupe, which is a "Laboratory Information Management System" (LIMS) - basically a suite of applications acting as a database front-end for laboratories. We began development back in 2008, when Mozilla were still pitching XUL to third parties as a cross-platform development solution, and HTML/JS/CSS was rather more lacking in capabilities than it is now. We chose to use Remote XUL to give our customers the advantages of a centrally managed system, but with the richness of XUL widgets at the front-end. XUL overlays are also invaluable in allowing us to customise the software for each of our clients, whilst still keeping them on a common base.

In terms of size, the codebase (excluding customisations) runs to over 30k lines of JS, and over 23k lines of XUL. On top of that we have over 16,000 lines of mixed XUL/JS in XBL bindings, which constitute most of the widgets used in the 23k lines of XUL. Adding in the custom code roughly doubles the JS and XUL counts.

We've been persevering with Remote XUL, hoping that a good replacement would become available before it was finally removed. We started work on a migration to XBL2 when that got ratified, only to see it fall by the wayside. We're currently working on a migration to Web Components, but that's been something of a moving target which has wasted a lot of our development time. We had hoped that native Web Components might become available in Firefox before Remote XUL became unusable, but that hasn't been the case, so we've been forced into migrating our code to use the polyfill (which works, but performance isn't great once things get complex).

In practical terms Remote XUL was broken with the move to multi-process Firefox. Popups - which includes menus - don't work in Remote XUL with multi-process enabled. Currently our customers are using 52ESR, but have to make a trip to about:config to disable multi-process, which is obviously less than satisfactory.

I have tried running Qualoupe against Firefox 60 (developer edition), and it's even more fundamentally broken. Our login page - the simplest application in the suite - fails to draw anything to the screen. It looks like it might be an issue with namespaces and our XBL code, but I haven't been able to find a solution. I'm loathe to spend too long on it, as I could easily find myself exploring a rabbit warren of issues, when my time would be better spent working on the migration to Web Components.

When even something as simple as our login screen (two XBL fields, one menupopup) fails to work, I think it's reasonable to say that Remote XUL is no longer functional. Perhaps an application using no XBL might still work, but from our perspective it no longer makes any difference whether or not Remote XUL is officially removed from Firefox, as it doesn't work for Qualoupe and I doubt that anything but the simplest Remote XUL application will work either.

Mark Crutch

Two Fold Software Ltd.

[Boris Zbarsky]

Mark,

Thank you for the summary of where things stand with Qualoupe. I'm
sorry your experience has been so painful...

-Boris

> _______________________________________________
> firefox-dev mailing list
> firefox-dev-4eJtQ...@public.gmane.org
> https://mail.mozilla.org/listinfo/firefox-dev

[Eric Shepherd (Sheppy)]

Do we have a good handle on what things remote XUL allowed that still can't readily be accomplished using open APIs, HTML, and CSS? We should be sure we know what remains to be done in order to make a complete transition away from XUL possible for anyone still relying on it. I don't know what the prioritization is for actually implementing those features, but knowing what they are would be a necessary first step.

firefox-dev-4eJtQOnFJqFAfugRpC6u...@public.gmane.org
https://mail.mozilla.org/listinfo/firefox-dev

_______________________________________________
firefox-dev mailing list
firef...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev

[Mark Crutch]

On Wed, Mar 28, 2018 at 5:26 PM, Eric Shepherd (Sheppy) wrote:

Do we have a good handle on what things remote XUL allowed that still can't
readily be accomplished using open APIs, HTML, and CSS? We should be sure
we know what remains to be done in order to make a complete transition away
from XUL possible for anyone still relying on it. I don't know what the
prioritization is for actually implementing those features, but knowing
what they are would be a necessary first step.

I can give some insight from our perspective, though other Remote XUL consumers may have used things that we didn't, so this list is hardly comprehensive.

1) XBL.
This was the one thing that kept us on Remote XUL for so long. We created complex XBL widgets, and wanted a robust client-side component technology to migrate to. Web Components is the obvious answer, but still not native in Firefox. Using a polyfill works, but performance is bad for nested Web Components, and the lack of true CSS and event encapsulation has resulted in more complex and ugly code than we would otherwise need.

2) XUL overlays.
We've reimplemented this with some JS of our own to achieve a similar effect, but it was a right pain to do so and performance isn't as good as a native solution. I doubt most other Remote XUL consumers are using these, though.

3) XUL Trees

We really only used these as flat tables, rather than genuine trees, but even with that restricted use case, HTML tables don't really cut the mustard. XUL trees have column headers that remain fixed in place when you scroll vertically, sortable columns, and reorderable/resizeable columns. Reimplementing these features in HTML is possible, but not at all easy (I believe the pieces are in place, spec-wise, for fixed headers and sortable columns - but no browser seems to support them). I ended up creating a complex Web Component with two tables (one for the column headers, one for the body), synchronised with scroll event handlers (not good...) and a lot of additional code to try to handle column moving and resizing.

I would guess that most other XUL consumers used these as real trees, which adds a whole other level of complexity, especially as there's a good chance they're using templates and RDF (yet another level of madness and frustration!).

4) XUL Tabpanels

Achievable in HTML, but it would be nice to have a simpler component rather than having to roll your own with suitably styled divs and buttons/links and JS to link it all together. These are a common source of z-index problems as the active "tab" needs to be brought to the front, but not appear above any popups or lightbox-style dialogs (which may not even emanate from within the tabbox).

5) XUL Menubars

Again, nothing that can't be done in HTML with enough effort, but there's a world of difference between creating a menubar (with nested submenus) using a little declarative markup, and trying to roll your own in a Web Component.

6) XUL Popups

Creating a popup or lightbox that can be relied on to appear above everything else in your HTML page is a pain. We all know that just giving something a huge z-index only works on simple pages without positioned elements. In XUL you can just have a <popup>, then tell it to open with a variety of options (too many?) as to where it appears relative to the page, the pointer, or another element - knowing it will always appear on top of everything else, regardless of the structure of your page.

<rant>
The common thing about 3-6 is that they are widgets that are frequently used in desktop applications, but have never been implemented as native HTML elements. This has always seemed ludicrous to me, as HTML takes on ever more "application" roles. I hate to think how many developer hours have been lost to proprietary implementations of tab boxes alone, few of which will have all the keyboard handling and other benefits of a native implementation. I understand the desire not to keep adding more and more elements to HTML, but these are UI stalwarts that have been in use for decades - I think they've proven their worth!
</rant>

Most other things I think are reasonably straightforward in HTML, CSS and JS now, especially with flexbox and grid support in CSS.

The fundamental issue isn't that things *can't* now be done using HTML/CSS/JS, but that doing so is hard, and leads to less than satisfactory results. We've struggled with things like keyboard handling in Web Components, for example, which really hinders accessibility. It can be done, but it takes a huge amount of developer time: consider the implications of writing good keyboard handling code for menus, compared with just being able to add a <menu> tag to your markup.

The obvious answer would be for Mozilla to produce some "official" Web Components to replace the more complex XUL widgets. Every component library I've seen focuses mainly on the simplistic cases, such as buttons and fields. If they deal with menus, tables and trees at all the resultant components tend to end up strongly tied to the rest of the library, in style or usage, without the near-native appearance and behaviour of XUL widgets, and without the flexibility to just drop them into a web page which *isn't* using React (or whichever other JS lib the component library is tied to).

Parity with Chrome on native Web Components would be a huge win, as it might encourage development of more complex component libraries - or at least some standardisation on best practices - but that's probably not what the original question was getting at. Excuse my frustration: I know native Web Components are on their way... but they've been on the way for many years now, and the end of Remote XUL has made them all the more necessary.