Followup: modularity, WebExtensions, and going faster

[Benjamin Smedberg]

I spent a week writing a thing about modularity, webextensions, and going faster. I think it's important for us to decide the module structure of our code especially as we start shipping independent modules/going faster. And I believe that having better module structure, boundaries, and documentation is critical to our teams being more agile and also attracting contributors to the project.

http://benjamin.smedbergs.us/blog/2016-09-03/modularity-and-webextensions/

I personally think that we should double down on WebExtensions as a model and start using that for large parts of Firefox. But Andy McKay and Rob Helper had some good counter-thoughts and I've asked them to post here to elaborate.

In the post I asked everyone to send followups to firefox-dev, so I wanted to start a thread here to collect responses. Over the next months I'd like this to turn into a firm decision about how we're going to build system addons; but I'd like to start by seeing what feedback people have and even whether I've framed the problem correctly.

--BDS

[Gijs Kruitbosch]

My understanding is that the webextensions API surface is fully async, in part to make it possible to eventually move them out-of-process. Some of the web, unfortunately, is still sync (hello, alert/onbeforeunload/inter-window direct JS calls, ...). Some bits of frontend are also expected to react in a synchronous manner. For example, it shouldn't be possible to end up with a situation where the user switches tabs and the location bar updates in a different tick of the event loop than the visualization of the selected tab (ie you'll see your google tab selected but the URL bar says mozilla.org). This fact makes me very skeptical of this paragraph:

At the end of the day, I believe we should end up in a world where most of the Firefox frontend is made available via system addons and glued together using well-defined API surfaces. What would it be like to have our tabbed browser, location bar, search interface, bookmarks system, session restore, etc be independently hackable? It would be awesome! It would mean a higher quality product, with more opportunities for volunteer contributors to improve specific areas. It would likely mean reduced edit/build/run/test cycles. It might even give addon authors the ability to completely replace certain components of the browser if they so choose.

It feels like at least some of the components you're referring to could not realistically be (a / separate) webextension(s) as a result of their needing to be "in sync".

~ Gijs

_______________________________________________
firefox-dev mailing list
firef...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev

[Gregory Szorc]

I agree with much of what you say in your post, especially the bits around lack of clear boundaries around browser modules. Lots of code has evolved into an unholy mess over the past 10+ years. I like the idea of using WebExtensions as an API upon which we build core browser features. If it's good enough for us, it's good enough for add-on authors and that means add-ons have an opportunity to innovate without having to delve too deeply into low-level internals.

What isn't clear to me is where exactly do we draw the line for API support? It's cool to say that awesomebar, bookmarks, session store, etc are loosely coupled and independently hackable - this sounds great from an ivory tower. But the cobweb of inter-dependencies between all these components exists for a reason: it was/is necessary to implement features. There are features like Firefox Sync that put their hands in a lot of cookie jars (not the RFC 2109 kind) and muck around in low-level details that - quite frankly - shouldn't be exposed in any formal API layer. In a WebExtensions world, we'd likely have to expose these things (possibly just to the system add-ons layer) or commit to more stable APIs on the WebExtensions layer. I can see a lot of scenarios where low-level APIs exposed to WebExtensions hamstring future change because of commitment to backwards-compatible APIs. Alternatively, we could see a proliferation of new APIs (e.g. changeFooWidget4()) - and the code behind them - because the existing APIs aren't "just right" and we can't change behavior without breaking backwards compatibility. In short, there is a conflict between system add-ons needing low-level and niche APIs and our desire to maintain stable APIs for 3rd party add-ons. I would appreciate clarification on how exactly these API surfaces will be managed. You allude to this in your post by saying "I think we will have to come up with APIs that are not exposed to all addons, but only to particular system addons where we understand the risks and experimental nature." So I guess my question is... how? Is there an "ACLs for JS functions" feature that was invented since I stopped doing Firefox frontend programming?!

[Stuart Philp]

Some assumptions: It's harder to release larger changes than smaller changes. It's harder to release non-isolated changes than isolated changes.

Potential challenges I can think of:

1. A much higher deployment throughput (multiple small releases vs one large one) requires very mature operational methods. We're making strides here, and perhaps it would all line over next couple of years, but it's something to call out.

2. Maintaining consistency between APIs. We will learn better ways to do things or discover bugs which impact multiple APIs, which means multiple APIs need to be updated. A similar bug in modules A and B could be fixed, but module C goes unnoticed and pops up again in the future. Some modules will lag behind.

3. Handling API inter-dependencies. If your feature relies on multiple API surfaces and one of the contracts changes but another doesn't, you can end up blocking the feature until all contracts are updated, or force the feature to pin to older API versions that can communicate with each other which adds some maintenance overhead.

4. Recreation of environments/builds. Multiple modules revving independently means that for testing we need to specify and test specific combinations of modules for release. I'm not sure how many modules we'd be talking here, but the potential is 100s I'm sure. This also affects reproducing bugs to fix issues. A tool to "define" a specific configuration (combination of modules) could help (and we partially have something like this already I believe?). You also need to test both the individual API contracts (relatively easy) and the integration of those APIs (harder). You still need to test the end user experience as well.

5. Where I've seen this in the past, managing transactions (and synchronous events) typically requires passing IDs around and gets kind of nasty. I'm not sure of the need for this within Firefox but I'm sure some modules would have to be done this way. Can we do both synchronous and asynchronous APIs (and mark them as such)? I would then ask about private vs public APIs (public being addon-hackable).

All fixable problems I think, I like the idea overall

[Richard Newman]

What isn't clear to me is where exactly do we draw the line for API support? It's cool to say that awesomebar, bookmarks, session store, etc are loosely coupled and independently hackable - this sounds great from an ivory tower. But the cobweb of inter-dependencies between all these components exists for a reason: it was/is necessary to implement features.

To echo this: in the past couple of years my opinion is starting to settle that Firefox's vertical component architecture is obsolete in many respects. We very rarely (in my experience, at least) can limit our work to "behind the curtain", wherever that curtain happens to be.

Speaking very superficially, most of the interesting problems occur between components, much of the rest involves poking at components as if they were bad POJOs, and much of what we can do (and think to do) is limited by the artificial boundaries put in place.

If we split apart Firefox along its existing lines, we will get more of what we've got, perhaps faster.

Choosing some different lines will give different results. For example, right now we wouldn't really think to, e.g., up-weight awesomebar results for which we have a saved password, because those two data sources are about as siloed as you can get, and it would be a huge pain in the ass to integrate the two from calling code. At an even greater extreme, implementing decent atomic sync of all browser data sources in the current model is essentially an impossibility, because there's no part of the browser that even pretends to consider them holistically.

My very opinionated position, then, is that putting effort into making Firefox's existing modularity structures "even more so" is not just a waste of time, but actively counterproductive.

[Robert Helmer]

On Thu, Oct 6, 2016 at 8:08 AM, Benjamin Smedberg <benj...@smedbergs.us> wrote:
> I spent a week writing a thing about modularity, webextensions, and going
> faster. I think it's important for us to decide the module structure of our
> code especially as we start shipping independent modules/going faster. And I
> believe that having better module structure, boundaries, and documentation
> is critical to our teams being more agile and also attracting contributors
> to the project.
>
> http://benjamin.smedbergs.us/blog/2016-09-03/modularity-and-webextensions/
>
> I personally think that we should double down on WebExtensions as a model
> and start using that for large parts of Firefox. But Andy McKay and Rob
> Helper had some good counter-thoughts and I've asked them to post here to
> elaborate.

I quite like WebExtensions as a replacement for Firefox extensions.
They are safer for users, easier to understand and control, enable
cross-browser extension development, and they reduce and better define
the extension API surface area we need to support.

We have explored using them for system addons (most likely as a
"hybrid" bootstrap.js + webextension), and there are some significant
drawbacks to using WebExtensions for browser features, which are
mostly due to intentional tradeoffs in the design of Chrome
extensions.

Gijs already pointed out that WebExtension APIs are all async (which
is on purpose), some other issues that come to mind:

1) WebExtensions currently don't (and Chrome likely never will) allow
modification of the UI beyond a few very limited things like adding
toolbar buttons, maybe things like menu entries in the not-too-distant future.

2) We would need to implement a *ton* of internal-only APIs that would
almost certainly never be implemented by any other browsers. Many
would likely only ever have a single caller.

I don't think there's any guarantee that our internal-only
WebExtension-style APIs would be any more stable, well-documented,
etc. than the code we currently consider internal-only. I suspect we'd
be tempted into implementing very general mechanisms that would remove
a lot of the safety arguments in favor of WebExtensions.

3) We're moving towards implementing more of the browser chrome in
HTML/CSS/JS, with some non-standard (and not appropriate to
standardize) APIs - a better approach might be to figure out
how to let Firefox update parts of the browser chrome in a similar way
to how regular web pages work.

For UI-specific features, it doesn't seem easier to use an extension
to modify the DOM or change the CSS, versus just modifying the HTML
and CSS directly.

For JS and binary components that UI features depend on, we could have
a simpler way to deliver just these and still not require restart.

If what we really want is to be able to ship updates to Firefox more
quickly and without requiring restart, and to be able to ship only the
new bits/bits that changed, it feels like there are more direct ways
we could do that safely using existing update mechanisms.

System add-ons are built on top of the extension mechanism because it
already exists and is relatively well understood, but I think we
should step back and re-think before we try to go down the road of
using this as the abstraction to build Firefox generally.

I'm not sure that more modularity is what Firefox needs to move
faster, especially given the lessons we can take from XPCOM from the
last decade or two.

> In the post I asked everyone to send followups to firefox-dev, so I wanted
> to start a thread here to collect responses. Over the next months I'd like
> this to turn into a firm decision about how we're going to build system
> addons; but I'd like to start by seeing what feedback people have and even
> whether I've framed the problem correctly.
>
> --BDS

[Alexandre poirot]

I'm all in for using more WebExtension. That's been a while I've been suggesting to do that:
  http://techno-barje.fr/post/2016/03/14/session-restore-web-extension/
  http://techno-barje.fr/post/2016/03/16/shipping-firefox-features-as-addon/

Me and Vivien prototyped and demoed a browser to a few people at MozLondon.
We ended up releasing that:
  http://techno-barje.fr/post/2016/06/28/html-experiments/
Which is not directly related to web extensions but is part of a bigger plan we have.

We have another version, still to be released. A browser 100% made of WebExtensions.
The current implementation splits the browser in 4 addons:
 * top level document addon, defining the various placeholders used for the other addons. It defines the overall layout of the browser.
   * tabs, just display the tab strips and just that. i.e. document icon, title and the close button,
   * urlbar, just implement the url bar input,
   * "deck", implements the web views via a set of  that display the web content.
These addons are all web extensions and are using chrome.* APIs to implement the browser.
For example urlbar uses chrome.tabs to listen for location change and load another URL. Actually all these addons are heavily based on tabs API.

This prototype is scary as it is yet another html browser written from scratch (tofino, browser.html, ...).
But it exposes the opportunity to rewrite existing pieces of Firefox product and see them working in a brand, new, fresh product.
The session restore addon I released on my blog works as-is in this html browser.

Using WebExtension is a way to move forward and really untie us, not only from XUL, but also from the tall beast Firefox frontend code became. All test pilot addons are meant to be bound to gecko and Firefox frontend. There is no easy way to get them to work easily on Tofino, browser.html or any new thing we would like to come up with. Having such ecosystem with core firefox features implemented as addons would allow having test pilot experiments drastically changing the browser experience and not stick to just hacking (hardly) in browser.xul.

We tried to get some traction at MozLondon but got none once we got back home.
So all this work is just some side projects so far.

[Martin Thomson]

On Fri, Oct 7, 2016 at 2:40 AM, Gijs Kruitbosch
<gijskru...@gmail.com> wrote:
> It feels like at least some of the components you're referring to could not
> realistically be (a / separate) webextension(s) as a result of their needing
> to be "in sync".

Isn't it still true that a synchronous component can depend on an
asynchronous one provided that they don't run on the same thread? As
in, if window.alert() needs to trigger activity on the browser
process, only the content process (thread) needs to block.

[Bill McCloskey]

I think this is a great idea, Benjamin!

I don't think the async nature of the APIs is necessarily a problem. All communication with the content process is already async due to e10s. Somehow we made that work. For UI updates like the location bar/tab strip thing that Gijs mentioned, I think we could add some simple transactionality to the APIs.

I'm also not too concerned about the issue of private versus public interfaces. WebExtensions have a permissions model that would work well here. Also, dealing with these issues ourselves would probably help to clarify some of the policy questions around WebExtension Experiments that have come up recently.

It's true that we probably don't want to expose all our internals through a WebExtensions API. But we don't have to write 100% of the browser using WebExtensions. I think Alexandre's experiment with session restore shows that major parts of the browser could be written this way. We should start with the easiest stuff and see where we get. Perhaps we'll reach a point of diminishing returns, but even then we would have made a lot of progress from where we are now.

-Bill

On Thu, Oct 6, 2016 at 8:08 AM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

[David Teller]

I believe that Alexandre's effort is extremely promising, both for
developing Firefox and for dogfooding the WebExtension API.

Cheers,
David

On 06/10/16 18:25, Alexandre poirot wrote:
> I'm all in for using more WebExtension. That's been a while I've been
> suggesting to do that:
> http://techno-barje.fr/post/2016/03/14/session-restore-web-extension/

> <http://techno-barje.fr/post/2016/03/14/session-restore-web-extension/>
>
> http://techno-barje.fr/post/2016/03/16/shipping-firefox-features-as-addon/
> <http://techno-barje.fr/post/2016/03/16/shipping-firefox-features-as-addon/>

[Mike de Boer]

The intended audience for this post is ‘all’, not just Benjamin. Benjamin, thanks for creating a place where I can share this.

I think Benjamin’s first point that swiftly attempts to validate the assumption that the core limiting factor is in fact the modularity and module boundaries in our codebase is comparing apples to oranges by offsetting process against code architecture.

I think we’re looking for a problem that presumably causes our apparent inability to go faster in the wrong area.

In that vein I was very pleased with Richard Newman’s post to this thread; you should read it!

Before I start, I’d like to point out that our current WebExtensions model is mostly comprised of interfacing with our lower level JS modules, adding a layer of indirection that we most often don’t need in frontend code that builds the browser.

Our current development model skews big code refactors and favours small, incremental iterations that extend the existing code a bit where it fits best, adds a unit test and call it a day.

I hear more and more noises, coming from various directions (product, UX, devtools, addon-sdk, WebExtensions, Servo, B2G, etc.) that are all variations of “Hey, what if we could rewrite Firefox to get rid of all this old, complicated crap?”

This should not be taken seriously. At least, not in the sense that we should actually _try_ to rewrite Firefox in its entirety. It’s in fact a big fat billboard poster on the wall in front of you that says: “This culture sucks, I don’t feel empowered enough to accomplish anything of sufficient value here!”, “Something needs to change! I don’t know what exactly, so rewriting the entire thing is what I can come up with right now!” and “Halp!”

If you read between the lines and talk to all developers - not just the ones who are loud - you’ll have to notice that the current cultural setting is not enabling us to go faster at all.

Rewriting the browser: not faster. Making small changes with le unit test attached: not faster. Adding a layer on top of existing modules: not faster.

What’s missing is that we are not able to _sell_ code refactoring, thus we’re almost never doing it, unless it’s absolutely necessary. Like when Electrolysis happened. (And moving to multiple content processes is going to yield even more refactoring and modularisation of code into a more modern style.) And if we’re not doing it enough, we don’t get a chance to become really good at it, either.

We’re scared to make changes other than small ones, because of their high(er) potential to introduce regressions. Regressions have a bad name, because they cause more work, keep developers busy whilst the calendar says they’re supposed to be working on something else. Managers hate them, because one cannot plan ahead whilst all these issues take priority over all other (future) work. Release Engineering will bounce your uplift requests and push for more strictness. I’d like to call them side-effects, instead, because they’re very valuable:

 * It shows our community from their best and shiniest angle: reporting bugs! Why focus on trying to move this colossus of a body to something that they don’t want to do (write code), but instead capitalise on their oft-emerging strength (file bugs) to feel a part of the community and help the Fox forward?

 * They uncover parts of the codebase that are, yes, not covered by unit tests.

 * When fixed, they are _always_ verified by our community members who filed the bug, showing yet another shiny angle from them.

But even though side-effects have these fine advantages, we wary them, because they’re not part of the process. If you don’t have a good method of dealing with side-effects, good luck pulling a code refactor through without a bruise or two.

But fortunately for us, process has a tendency to evolve when the need arises (again, look at e10s), so the only thing stopping us from moving forward and faster is our inability to sell code refactors.

Why would we need to remove #ifdef’s in our CSS and JS code _right now_? Why do we need to split browser.js or nsSearchService.js up into modules _right now_? It’s never a priority, until the collective frustration - not of the developers _working on_ it, but of all the developers having to _read_ it - reaches the boiling point. We seem to getting closer nowadays, is the feeling I get when I occasionally put out my temperature gauge.

Of course, there are strict rules attached to doing a code refactor of reasonable size and that’s where senior mentorship (note that I’m not using the prefix _leader_ at all) comes in to guide these projects and teams in the right direction with tips and best practices or even contributions. It’s also their responsibility to coordinate efforts and make sure there’s a certain gradual approach to refactoring the frontend, like not running two tracks at the same time.

Another, closely related defect that’s highly likely to have an effect on our inability to move faster is our culture of module ownership. I’m advocating for a complete removal of that concept. The word ‘owner’ has too many connotations that confuse the heck out of people: “Do you own that code like you own a house? What does that mean? Do I need to ask your permission if I want to step on your lawn?“

Instead, the current module ‘owners’ list should be reset to being ‘candidate reviewers’ only; people who have enough understanding of that area of functionality that they’re able to make qualitative judgments on contributions, are available for any question a contributor might have and willingly partake in triage sessions. (The fact that developers do triage in the first place is an interesting subject to me, but out of scope here.)

There is always someone who must _own_ the project. But in that context, the word simply fits. This person must assure that there are always enough candidate reviewers available for the different functional areas.

When you think and talk about modules and architecture, I think that simple faith in the fact that a smart, top-notch developer, like we all are of course, has a natural tendency to take the modern approach to writing new code. I mean, when you look at the most recently written slabs of code, I see the latest ECMAScript features, ReactJS, async-await style control flows, less LOC per module and smaller function bodies. Technically, we ought to be able to refactor very well; it shows.

I think that I’m at a word count currently that’s more than enough for now. I did not permit myself the luxury of a proof-read, so please forgive the broken English if it’s there. Please feel free to reach out if you feel I should clarify/ say more!

Cheers,

Mike.

[Bill McCloskey]

On Fri, Oct 7, 2016 at 9:11 AM, Mike de Boer <mde...@mozilla.com> wrote:

I think Benjamin’s first point that swiftly attempts to validate the assumption that the core limiting factor is in fact the modularity and module boundaries in our codebase is comparing apples to oranges by offsetting process against code architecture.

I think we’re looking for a problem that presumably causes our apparent inability to go faster in the wrong area.

In that vein I was very pleased with Richard Newman’s post to this thread; you should read it!

You make a lot of good points about the need for a culture that encourages refactoring, Mike. I think it would make sense to bring this up in a separate email thread. I'd like to agree with you more, except that this thread is about WebExtensions, and I disagree with you on that point :-).

 

Before I start, I’d like to point out that our current WebExtensions model is mostly comprised of interfacing with our lower level JS modules, adding a layer of indirection that we most often don’t need in frontend code that builds the browser.

Yes, the WebExtensions API won't magically make our code great. But it's a tool, and I do think we need better tools. We wouldn't want to write the browser in assembly, after all. Fundamentally, WebExtensions gives us a typesafe API mechanism. If you haven't seen them yet, you might be interested in the schema files that define WebExtension interfaces. Here's an example:
http://searchfox.org/mozilla-central/source/browser/components/extensions/schemas/tabs.json

From files like this, we automatically check the types of every method parameter. We can also generate documentation automatically (and you can see those docs in MDN). There's also a simple, uniform way of reporting and handling errors.

WebExtensions also enforces separation more strongly than any other mechanism we have now. In theory each extension can run in a separate process, so all communication between them has to go through well-defined APIs.

None of this is necessary to write good code, but it certainly helps.

That said, I agree with you that we need to be better about refactoring our code. We definitely shouldn't wait for a Grand Browser Rewrite to fix all the bad stuff we have. In modules where someone has been ambitious about improving the code (and I think Tim Taubert's work on session store is a great example) we've managed to turn gross code into great code through gradual refactoring. We should encourage more of that!

 

-Bill
 

[Brendan Barnwell]

On 2016-10-06 16:41, Robert Helmer wrote:
> 1) WebExtensions currently don't (and Chrome likely never will) allow
> modification of the UI beyond a few very limited things like adding
> toolbar buttons, maybe things like menu entries in the not-too-distant future.

This is critical, and for me it touches on one of the most fundamental
issues: the functionality of the product is more important than the ease
of development, and WAY more important than the speed of the release
cycle. The starting point needs to be functionality, and then dev
process and speed need to accommodate to that. (Another comment on this
thread re the necessity of consistent ordering between things like the
URL and the selected tab points in a similar direction.)

As far as functionality, I am apprehensive. To my mind, the move to
WebExtensions has already cost Firefox a massive amount of credibility
because it is essentially breaking all existing extensions. If
"doubling down" on this means more blocking off the ability to provide
useful functionality (and breaking existing functionality) in order to
make it easier to provide a severely limited range of functionality, I
don't see that as a benefit. It's like running a sandwich shop and
deciding you're going to make sandwiches faster by just giving everyone
two slices of bread instead of a sandwich.

As far as speed, I'm also apprehensive, because I think the drive to
"go faster" is fundamentally misguided. I would rather have a browser
that releases one coherent, well-designed version a year than a browser
that releases a gazillion tiny upgrades, some of which may break the
workflow I've developed with my existing extensions.

So, basically, I just think the starting point needs to be "what do we
want the browser (and extensions) to be able to do". If everything that
needs to be done can be done with WebExtensions, great. If not, it's
better to back off on WebExtensions than to deliver a crippled product.

--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown

[Andrew McKay]

The focus right now is tooling and development for add-on developers.
Another example is that we are soon going to be taking the schema from
Firefox and running it through tools like web-ext and AMO so
developers can have the same offline linting and validating tools that
Firefox does. But WebExtensions are currently focused on add-on
developers. What works there may not work for other developers, unless
you take the approach that all developers are add-on developers.

I count myself a relatively numpty in Firefox development, so I am
curious... we've tried all this before right? There are already
multiple layers of modules, schemas and all other kinds of things
throughout Firefox I believe. Why did those things work or not work
that WebExtensions might solve?

[Gregory Szorc]

On Thu, Oct 6, 2016 at 5:08 PM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

Having slept on this post, I have a few more comments.

1) It sounds like you are advocating a massive rewrite of the Firefox frontend code. It isn't a complete bottoms-up rewrite. But on a scale of 1 to 10, I think it is solidly a 7. Maybe an 8. In terms of effort required, I think the project is at least on the same scope as e10s. In terms of people resources and the value associated with people time, this project will effectively cost tens of millions of dollars and likely take 2+ years. Where are we going to get the staffing for that? What other features have to be put on hold while we rewrite core features?

2) The more years of experience I accrue, the more skeptical I am that significant rewrites or something resembling significant rewrites are beneficial. The rest of the industry seems to agree that incremental, targeted refactorings yield better results. I am highly dubious that refactoring/rewriting significant existing Firefox frontend features to be WebExtensions will be worthwhile. If we want to implement new features as WebExtensions or convert existing, self-contained features as WebExtensions, go for it. But to advocate for proactively changing the architecture of the existing features sounds scary to me (in fairness, I'm not sure if you are advocating this or whether you are merely saying we should strive for a WebExtensions first architecture).

3) If we're talking about a significant change of architecture to the Firefox frontend and the high costs associated with that change, I'll throw out an alternative idea: (re)write the frontend in a typed language like Rust. I argue that a typed language - and Rust especially - would deliver "well-defined, documented, and enforced module boundaries or interfaces" better than JavaScript ever will.

4) I have significant concerns about the concept of system add-ons. Currently, the composition of Firefox (in terms of code) is essentially deterministic from the commit in mozilla-central it was built from. We know exactly what Firefox code is running (OK, /should/ be running) on users' machines. System add-ons fundamentally change that from ~1 combination of software components to N, where N could be absurdly large (10^3 or larger). It is hard enough to track down intermittent failures in test automation with a well-defined software configuration. System add-ons introduce whole new classes of bugs related to how different versions of add-ons interact with each other. They require more and more complicated testing infrastructure to test various combinations of add-ons. This makes assuring quality harder. System add-ons seem like they add cost to the release process and undermine stability (even if they may make writing code easier - but software development is about a whole lifecycle, not just writing the code).

5) I challenge the necessity that system add-ons or WebExtensions are necessary to "go faster." There are a host of changes we could make to the release process to "go faster." System add-ons feel like they are on the more radical end of the spectrum and thus represent an over-correction from the status quo. Other alternatives include: a) eliminating a train b) shortening release cycles (possibly to days or hours) c) shipping frontend (omni.ja and friends) updates separately from platform/libxul updates. There's also the concern that system add-ons only solve the narrower problem of shipping frontend components. Wouldn't time be better invested in shipping all of Firefox faster? (I understand the post says that modularity is the reason why you believe we can't go faster. But I get the vibe from elsewhere that some feel packaging and shipping are significant reasons.)

6) The emphasis on modularity and module boundaries in this post resembles the latest trend in cloud software: microservices. I'll let readers complete the analogy (there are both pros and cons).

Look, I understand the desire to "go faster" and deliver a better Firefox to users faster. You know from what I've done with my time at Mozilla that I care about developer ergonomics/productivity and contributor engagement. Having worked most of my professional career in server-side software, I understand the viewpoint that shipping monolithic software to clients seems ancient and antiquated in comparison. That explains the desire to adopt more of a server software mindset to delivering Firefox.

We should absolutely "go faster." We can "go faster" by adopting some modern approaches to shipping. We can "go faster" by focusing on software architectures that will make our paid staff and volunteer contributors more productive. But I'll be honest, shipping code updates to core browser features as discrete system add-ons that must conform to specific [WebExtensions] APIs feels like it will cause more harm than good. I'd rather focus on efficiently delivering a known, atomic unit of Firefox (that doesn't rule out tweaking the behavior of code once it has shipped, e.g. turning on a new feature N% at a time). Independently, we can improve developer ergonomics for working on browser "internals." And I can think of several ways we can do that don't involve significant architecture changes or maintaining complicated and backwards-compatible API surfaces. Using a typed programming language and/or static analysis to enforce cross-module "good" behavior, improving support for IDEs and API discoverability, writing JS as "modules first" with an emphasis on reuse and isolated testing, and improving the overall build-edit-test-debug-review-land loop come to mind as more important and far less effort from the perspective of the Firefox front-end engineering team.

[Gregory Szorc]

On Sat, Oct 8, 2016 at 12:12 AM, Robert Helmer <rhe...@mozilla.com> wrote:

On Fri, Oct 7, 2016 at 2:48 PM, Gregory Szorc <g...@mozilla.com> wrote:
> 3) If we're talking about a significant change of architecture to the
> Firefox frontend and the high costs associated with that change, I'll throw
> out an alternative idea: (re)write the frontend in a typed language like
> Rust. I argue that a typed language - and Rust especially - would deliver
> "well-defined, documented, and enforced module boundaries or interfaces"
> better than JavaScript ever will.

+1. I think we should probably stick with HTML/JS/CSS for basic UI
stuff like layout, setting up button events, DOM manipulation etc, but
a lot of the code we have in JSMs right now would be much safer to
implement in Rust, and most likely faster.

> 4) I have significant concerns about the concept of system add-ons.
> Currently, the composition of Firefox (in terms of code) is essentially
> deterministic from the commit in mozilla-central it was built from. We know
> exactly what Firefox code is running (OK, /should/ be running) on users'
> machines. System add-ons fundamentally change that from ~1 combination of
> software components to N, where N could be absurdly large (10^3 or larger).
> It is hard enough to track down intermittent failures in test automation
> with a well-defined software configuration. System add-ons introduce whole
> new classes of bugs related to how different versions of add-ons interact
> with each other. They require more and more complicated testing
> infrastructure to test various combinations of add-ons. This makes assuring
> quality harder. System add-ons seem like they add cost to the release
> process and undermine stability (even if they may make writing code easier -
> but software development is about a whole lifecycle, not just writing the
> code).

I think we're largely in agreement overall (see my earlier post on
this thread), but just to help assuage any concerns about our current
use of system add-ons - we require all extensions that ship with
Firefox to be in-tree, and any out-of-cycle updates should be landed
on the appropriate branches. So, they live alongside other code in
Firefox, are tested as part of the usual automation and QA, etc.

We do have the ability to push arbitrary updates, but we only use
those for hotfix-style activities, similar to the current hotfix.

Any out-of-cycle updates pushed "expire" when the next regular
application update happens.

Thanks, Robert. This does significantly assuage my concerns about the system add-ons model, especially around things becoming out of sync. In my defense, I was thrown off by the line in the blog post about potentially allowing add-ons to replace browser components. Assuming there is no slippery slope argument to be made here, then the conversation is essentially about how we should architect software within mozilla-central. In that case, I refer to rnewman's post on the topic, which was terrific.

FWIW, part of my unease about shipping core features as add-ons - possibly independently from major updates - is reaffirmed by an experience I had several weeks ago. One of our Test Pilot extensions broke enter/return in the awesomebar and I was literally unable to open web pages. My Nightly install was essentially bricked. I can tolerate Nightly crashing excessively every now and then - it is a daily build after all. But in my 5+ years of using Nightly, I don't recall having a Mozilla-maintained frontend feature fundamentally breaking the browser like that. I suppose I could chalk it up to chance and the intrinsic relative instability of Nightly. Yet at the same time, I know that we don't put Test Pilot extensions through the same [absurdly comprehensive] test suite that CI runs for commits to mozilla-central. I'm confident that if we did (or if we developed Test Pilot features in mozilla-central like we do for all other browser features), this breakage would have been detected and we never would have shipped. So when we start talking about developing and shipping features as add-ons outside the existing monorepo and CI infrastructure built around it or allowing core components to be swapped, I get really anxious that product quality and stability will suffer.

 

I agree. I think our problem is that hacking on Firefox is
exceptionally hard, and testing+shipping it is too. I don't think
either of things are hard because there aren't good enough boundaries
between different components.

Pushing updates quickly to just omni.jar and allowing restartless
updates is basically equivalent to moving features to extensions and
updating them that way. It just so happens that today it's easier for
a new team to get going by hacking on an add-on, it's faster and safer
for us to ship these standalone rather than update the whole of
Firefox, and extensions already provide a way to do restartless
install+update.

I don't think it follows that therefore everyone must develop on
extensions and all updates should be done the way extensions are. We
can and should make Firefox easier to hack on and should make shipping
faster + safer.

[Brunoais]

Brendan Barnwell's answer touches a very important point, IMO. Here's a question for discussion:
Why choose firefox over chrome?

Firefox is slightly slower than chrome; specially in a computer with really high amount of RAM available.
If a tab crashes in Firefox, the whole Firefox crashes (in current stable). If a tab crashes in chrome, only that tab suffers (until you hit the 20's and beyond) so you, usually, can just continue working afterwards.
Chrome's v8 is faster in more tests than Firefox's SpiderMonkey than the other way around.
Chrome's interface seems to be the one inspiring Firefox's. At least, for the most part.
Chrome has a more useful developer tools toolset for least used features (E.g. inline svg)
Chrome supports more standards than Firefox (Firefox is in s struggle to implement them).
Both are free of charge.

...But then... With those and more... Why sticking to Firefox?

Firefox allows extensions to have superb power! With extensions, my firefox instance becomes "my browser". Heck just look at my profile! I have (jump to the end of the list when you get the idea):

• Colored tabs
• Tabs opening relative to each other
• A quick access to stored passwords (with a boost in use) with a master password
  
• A share of the master password for all my SessionStorage, LocalStorage, cookies and history. (does not include indexedDB)
• I have a variable proxy selector so I can access some websites I have on my server hidden from most of the world in an effortless way.
• I have an inline translator that is translator independent (I choose which one to use)
• I have tab groups (ex-panorama, now in extension)
• I have a GUI to access to options only available in about:config
• I have an F6 key override
• I have a gif stopper (stops a gif in the current frame at the press of a button)
• I have a mass file downloader (for many files, asks only 1ce where to save them)
  
• I have a profile backup which backs up my profile every X time (super customizable and FREE!!!!!)
• I have a CSS refresher which reloads and reapplies CSS to a page from a changed file (not as useful as it was but I still use it a lot during web development)
• I have a form recovery addon which backs up encrypted (the key is stored in the Firefox's password manager) data on the status of the form. You may never believe how much this has helped me so far.
• Being able to insert tabs in a textarea!!!
• I got the addon bar back! It was getting cluttered up there without this addon.
• ... And that's only 16 of them!

Is Firefox kinda slow? YES! Heck, it is even using ~1.2GB of ram most of the time and some page loads could be ~0.5s faster. It's worth it!!!

Could these extensions be made in WebExtensions? Heck no! Chrome never lets an extension touch Storage stuff. Forget them being protected with a password!
Colored tabs in WebExtensions? Pff... Who wants that, right? Tell that to the +200K users of a featured addon!
Metadata in stored passwords? Boom! Featured addon! (https://addons.mozilla.org/en-GB/firefox/addon/password-categories/)

WebExtensions is not ready for any of this on Chrome. For the time it has been like this, I don't think it will ever be.

In the late years, for many feature requests I have been directly or indirectly bombarded with "If you want feature X, do it yourself or get an addon for it". So far that can be true because the "legacy" extensions can do just about anything.
Now imagine this happened instead:

-  I had an addon on my browser that colors tabs based on the website name and a heuristic of the colors mix in the first page it finds. The browser now only supports WebExtensions. How do I update it so it works on my browser?
- You can't. We now do not support extensions that change the browser UI in any way except for adding a button.

Think as a user now. How do you react? You had an addon you could use and worked, now you cannot have and you can't even expect it to ever work on your browser. EVER!

In conclusion

I think that WebExtensions is a very good tool for the majority of the extensions. It has a very direct API for use and simplifies the code complexity when you just want a small purposed extension that does changes to a web page, potentially with some extras.
We do still need that low level API, though. The one that allows changing the shape of the tabs, place the tabs on the left side of the browser window, change the color of the tabs, add toobars, access about:config configs...

I believe, with both API made in a correct fashion, most extensions will be WebExtensions. Only slightly less safe than a web page (but requires explicit permissions) and cannot do harm without warning. Reviewing those extensions should be a much easier task than the other extensions. The API (already in nightly) to communicate between WebExtensions and "legacy extensions" can be a huge thing here!

Extra thought

While having this e-mail sitting here, I thought:

AFAIK, WebExtensions are limitless when acting inside a web page (I'm relating to content scripts). They have the same power as the legacy extensions.
It is on the background pages that WebExtensions  are limited which is exactly when they need to act on the browser itself and do the awesome (such as coloring the tabs or opening a popup that clearly belongs to the browser and not just a web page).

How about this:

Make 3 code groups:

1. Page scripts (current WebExtensions API)
2. Background pages (current WebExtensions API)
3. Low level API (can only run on the same process as the background pages)
   

1 can only communicate with 2, 2 can communicate with either 1 or 3, 3 can only communicate with 2, 2 and 3 can only share data using the message API (for consistency).
3 is not protected behind a tight permissions system like WebExtensions but it must specify the components it accesses in the manifest file (security). The components requested (as a key:value) are included in a global object since the start of the script.
The scripts for 3 may only be used for features that do not exist in WebExtensions (guaranteed by the manual review and, possibly, to certain extent by the automatic reviewer).

This should ease extension reviewing, keeping compatibility with other browsers (all that works in chrome works here) and stick the tongue out to chrome when you have colored tabs and chrome can't.

Thoughts?

[David Teller]

I have a technical question that may inform a few things.

We currently rely a lot upon lazy loading of modules/features/... to
speed up startup and limit memory usage of features that are not used.
Is this compatible with the WebExtensions model?

Cheers,
David

[Axel Hecht]

I have a couple of thoughts on this, on three lines:

XPCOM. We did have extensive contracts and interfaces, strictly typed and all. My take on the downsides is that we struggled to create future-proof APIs for one. Also, the implementations of those had undocumented ranges, bugs, and side effects that callers depended upon. And some of our APIs never carried fruit.

I also think that tight locking between components isn't as much about the API between them, but how much of the API surface is relied upon.

I wonder how other folks see our storyline to deCOM, and how we'd modularize without the same problem.

WebExtension compat. If we start exposing internals of Firefox needed for Firefox development, we'll fork the API space of WebExtensions. If more browser vendors do so, we'll get into a game of webcompat for WebExtensions. Not sure if that's helpful for the extension ecosystem.

devtools. You quoted devtools as a success story for developing quickly, and I'd actually say the contrary. We had several years of experience with js debuggers and debugging, and only then were able to create an API that was somewhat stable. And even then, IIRC the entry of FxOS and Android and Chrome debugging still opened a bunch of challenges. In that sense, I think devtools is a great example that you can't create good APIs quickly. You need a lot of experience in the field you're tackling, and then you can start defining good APIs, and even those still evolve.

Axel

On 06/10/16 17:08, Benjamin Smedberg wrote:

[Benjamin Smedberg]

Thank you all for the beginnings of a great discussion!

I'd like to follow up on a few points which I think were misunderstood.

• The focus of my post is about requiring good module boundaries for code that wants to move quickly, and the notion that WebExtensions are the right way to do that. Although I like indulging in talk about making the entire browser modular, that is clearly not something we can accomplish soon. I am talking about the pipeline of development that we're building which starts out with test pilot, through SHIELD studies, to system addons. We are shipping these *today*, and shipping is a tedious and fairly high-risk process because we don't have modular isolation or testing.
  
• I also don't think we should limit ourself to WebExtensions as they are today. WebExtensions are purely asynchronous today because of design decisions that Chrome made and we also accept that addons shouldn't be allowed to block the browser UI. That doesn't mean that we can't make API surfaces which are synchronous/blocking, if we feel that's the right tradeoff.
• We can make choices about the "stability" of API surface: both APIs we're building internally, and even APIs we expose to addons. I'm going to follow up separately about what I think we should have learned from XPCOM, but if there's anything we should learn it's that frozen API surfaces are often the wrong choice.
  

We have a stated desire to let certain teams move much more quickly. This includes Hello in the past, it includes a bunch of test pilot and shield studies now, and it will include activity stream work in the future. What I was trying to point out, with a long history of academic research, is that there is a "physics" of software engineering that we cannot avoid: projects and teams move faster when there are fewer modular entanglements, smaller modules in general. We can train new employees and attract new contributors better with smaller boundaries around the code that they have to understand to be productive, and better documentation about where the boundaries are.

Richard is right that if we try to "remodularize" around our existing infrastructure, some things are going to be really hard. It's basically impossible to provide atomic sync with things designed the way they are. But let's say that atomic sync is something that we believe is important: I believe we *have to* spend the time to reorganize the system so that sync as a module that stores passwords, bookmarks, etc is lower/more basic in our module structure, and then we build our bookmarks UI, location bar, etc, on top of that core structure. We are in charge of designing these things.

We always *have* a structure. I believe that currently we have a structure where Firefox is a large monolith which is very difficult to change effectively. To change that, we don't need to and shouldn't rewrite it all. But we should start carving off pieces so that they are smaller and have real boundaries. I believe we should start with the things which are trying to move faster right now. Maybe we should also start at the other end with core infrastructure like sync. If we carefully carve off pieces, keeping a longer-term vision in mind, we will end up in a place where there either is no monolithic center, or we carved it down enough that we can document/understand/hack on it more easily.

--BDS

[Benjamin Smedberg]

On Fri, Oct 7, 2016 at 6:12 PM, Robert Helmer <rhe...@mozilla.com> wrote:

I agree. I think our problem is that hacking on Firefox is
exceptionally hard, and testing+shipping it is too. I don't think
either of things are hard because there aren't good enough boundaries
between different components.

This is where I think we (Robert and I) fundamentally disagree. I believe that hacking on Firefox is exceptionally hard *because* there aren't good boundaries. I think there's a lot to discuss about the nature of those boundaries: JSMs versus WebExtensions versus future Rust versus HTML iframes. But we just don't have effective boundaries right now, and that is one of the reasons why hacking the Firefox frontend is so difficult and exhausting.

--BDS

 

[Brunoais]

I agree with Benjamin here.

[David Teller]

My experience entirely confirms Benjamin's observation.

Simple bugs cannot be fixed easily because of gazillions of unexpected
dependencies. Adding features is a chore because it means that yet more
code will need to be supported forever due to yet further unexpected
dependencies. This is exactly the opposite of what we need if we wish to
recover our long-lost agility.

Moving extensions themselves to WebExtensions will help. Having clearer
boundaries inside our own code is critical, too. I do not know that
basing our boundaries on WebExtensions is the best way to enforce these
boundaries (and it's still not clear to me whether there would be
performance costs involved) but I haven't seen any better counter-proposal.

Orthogonally, static analysis (and other forms of automated code defect
detection) can also be very helpful there.

Cheers,
David

[Robert Helmer]

On Fri, Oct 7, 2016 at 2:48 PM, Gregory Szorc <g...@mozilla.com> wrote:

> 3) If we're talking about a significant change of architecture to the
> Firefox frontend and the high costs associated with that change, I'll throw
> out an alternative idea: (re)write the frontend in a typed language like
> Rust. I argue that a typed language - and Rust especially - would deliver
> "well-defined, documented, and enforced module boundaries or interfaces"
> better than JavaScript ever will.

+1. I think we should probably stick with HTML/JS/CSS for basic UI
stuff like layout, setting up button events, DOM manipulation etc, but
a lot of the code we have in JSMs right now would be much safer to
implement in Rust, and most likely faster.

> 4) I have significant concerns about the concept of system add-ons.
> Currently, the composition of Firefox (in terms of code) is essentially
> deterministic from the commit in mozilla-central it was built from. We know
> exactly what Firefox code is running (OK, /should/ be running) on users'
> machines. System add-ons fundamentally change that from ~1 combination of
> software components to N, where N could be absurdly large (10^3 or larger).
> It is hard enough to track down intermittent failures in test automation
> with a well-defined software configuration. System add-ons introduce whole
> new classes of bugs related to how different versions of add-ons interact
> with each other. They require more and more complicated testing
> infrastructure to test various combinations of add-ons. This makes assuring
> quality harder. System add-ons seem like they add cost to the release
> process and undermine stability (even if they may make writing code easier -
> but software development is about a whole lifecycle, not just writing the
> code).

I think we're largely in agreement overall (see my earlier post on
this thread), but just to help assuage any concerns about our current
use of system add-ons - we require all extensions that ship with
Firefox to be in-tree, and any out-of-cycle updates should be landed
on the appropriate branches. So, they live alongside other code in
Firefox, are tested as part of the usual automation and QA, etc.

We do have the ability to push arbitrary updates, but we only use
those for hotfix-style activities, similar to the current hotfix.

Any out-of-cycle updates pushed "expire" when the next regular
application update happens.

I agree. I think our problem is that hacking on Firefox is
exceptionally hard, and testing+shipping it is too. I don't think
either of things are hard because there aren't good enough boundaries
between different components.

Pushing updates quickly to just omni.jar and allowing restartless
updates is basically equivalent to moving features to extensions and
updating them that way. It just so happens that today it's easier for
a new team to get going by hacking on an add-on, it's faster and safer
for us to ship these standalone rather than update the whole of
Firefox, and extensions already provide a way to do restartless
install+update.

I don't think it follows that therefore everyone must develop on
extensions and all updates should be done the way extensions are. We
can and should make Firefox easier to hack on and should make shipping
faster + safer.

[Robert Helmer]

I do agree that we need to have boundaries, and that we want to do it
right. I'd be happy to be wrong that this will significantly move the
needle on making Firefox hacking more productive so I'll table that :)

I'm having a hard time conceptualizing how WebExtensions would fill
this role exactly.

Would Firefox then be a very basic "core" browser, implemented in
HTML/CSS/JS, with extensions for adding features on top of this
("awesomebar", bookmarks, devtools, etc?) If so then the "core"
browser needs to implement WebExt APIs for each feature to use,
possibly with a "mozilla-only" permission if it's something we
wouldn't want third-party extensions to be able to do?

This feels like it would be an appropriate use of built-in extensions,
by adding features to the existing UI. If an extension fails for any
reason, then the feature is degraded but it continues functioning.
This also encourages us to make more good APIs available to the
third-party addon developer community, since we're dogfooding the APIs
ourselves (for the ones that are not "mozilla-only" which I imagine
there would be quite a few).

What about JS modules that multiple other features depend on, such as
the implementation of the WebExtension APIs themselves? We've so far
largely managed to avoid dealing with dependencies between extensions
(we made an exception for the new "WebExtensions Experiments" type of
addons, since the whole point there is to prototype new WebExt APIs
from an old-style extension).

I have more concerns about this - I think we want a simpler delivery
mechanism for core parts of the browser, and an extension that
implements some bit of functionality that is depended upon by other
features failing could be a lot harder to deal with.

[David Teller]

By the way, Alexandre, did you get a chance to benchmark these
WebExtension-based implementations of browser features? I know it's
pretty hard in the context, but if you by chance have numbers, they
could be useful.

On 06/10/16 18:25, Alexandre poirot wrote:

> I'm all in for using more WebExtension. That's been a while I've been
> suggesting to do that:
> http://techno-barje.fr/post/2016/03/14/session-restore-web-extension/

> <http://techno-barje.fr/post/2016/03/14/session-restore-web-extension/>

[Richard Newman]

Simple bugs cannot be fixed easily because of gazillions of unexpected

dependencies. … Having clearer boundaries inside our own code is critical, too.

To address just this little part (again): lots of interactions and close behavioral coupling between two modules is — in the absence of lazy design, which I don't think is our problem — an indicator that after years of evolution those module boundaries are drawn in the wrong place.

(That's totally normal, but I think our codebase's emphasis on modularity has if anything made the problem worse, not better.)

I think Sync and Places (and Satchel, and pwmgr, and…) are a great example of this: with the benefit of hindsight, having Sync and Places be totally separate modules — indeed, Sync started off as a separate add-on: very Go Faster! — with Sync trying to track very low-level Places changes via observer notifications, is/was entirely the wrong thing to do. Syncing is a cross-cutting concern.

We flipped this around on iOS, with storage exposing very narrow APIs to the front-end and taking on the burden of tracking changes for Sync. Tightly coupled, deliberately very constrained… but small, correct, and simple. The module boundaries are drawn between front-end and storage, not between bookmarks and Sync tracking.

I betcha there are other cross-cutting concerns, too, and I'm curious which other ones people see. Perhaps with the advent of rAC and fancy form fill we wish we didn't have a division between form fill and password management? Maybe we wish the favicon service were more general to support page summaries and tiles?

So yes, clearer boundaries — but they should probably not be the boundaries we have now.

[David Teller]

Of course.

On 10/10/16 22:44, Richard Newman wrote:
> So yes, clearer boundaries — but they should probably not be the
> boundaries we have now.

[Richard Newman]

Would Firefox then be a very basic "core" browser, implemented in
HTML/CSS/JS, with extensions for adding features on top of this
("awesomebar", bookmarks, devtools, etc?)  If so then the "core"
browser needs to implement WebExt APIs for each feature to use,
possibly with a "mozilla-only" permission if it's something we
wouldn't want third-party extensions to be able to do?

My big fear about this is that the choices we would make with those APIs essentially define the pillars of the application for years to come.

Good luck building anything novel on top of chrome.bookmarks, which hard-codes the existence of "Bookmarks Bar" and "Other Bookmarks", is strictly hierarchical, can't be transactionally changed alongside a different data source, etc.

(Heck, good luck building the awesomebar on top of Chrome's bookmark and history APIs.)

And think we have trouble moving fast now? Try hard-coding and publishing an interface that you can't evolve without breaking all UIs built on top!

IMO it's a mistake to treat all interfaces as equal and public, which seems to be the point of the WE side of this discussion.

[Axel Hecht]

I've watched the presentation of github and facebook doing GraphQL at https://www.youtube.com/watch?v=wPPFhcqGcvk with interest.

In a similar vein to what Richard is saying, they're moving the api boundary from what the service provider can offer to what the client needs.

Also, this isn't new to mozilla, per se. http://www-archive.mozilla.org/rdf/doc/aurora.html, just to scare folks.

Axel

On 10/10/16 22:44, Richard Newman wrote:

[Emma Humphries]

On Mon, Oct 10, 2016 at 1:52 PM, Richard Newman <rne...@mozilla.com> wrote:

IMO it's a mistake to treat all interfaces as equal and public, which seems to be the point of the WE side of this discussion.

​My experience has been with building front-end applications, and I can't say that it maps to WebExtensions.

But this conversation reminded me of SoundCloud's talk at the 2013 HTML5Conf on the pain of building your application on your public API, and it captures some of the problems we ran into when we tried rewriting WhiteHat's reporting tools on top of our public rest API.

https://www.youtube.com/watch?v=W8B9xT1QRb0​

​

[Thom Chiovoloni]

A few very ill-advised comments:

On Mon, Oct 10, 2016 at 12:50 AM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

We are shipping these *today*, and shipping is a tedious and fairly high-risk process because we don't have modular isolation or testing.

I don't think that modularity is the problem here. After all, its very possible to have bugs in systems where every component works as specified. (I'd even argue that -- depending on how you define "component" -- this is where most bugs occur.)

On Mon, Oct 10, 2016 at 12:50 AM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

But let's say that atomic sync is something that we believe is important: I believe we *have to* spend the time to reorganize the system so that sync as a module that stores passwords, bookmarks, etc is lower/more basic in our module structure, and then we build our bookmarks UI, location bar, etc, on top of that core structure. We are in charge of designing these things.

While this is true, I feel like it misses the larger point. Perhaps I'm projecting my own feelings onto Richard's comment, but the way I interpreted it, atomic sync was just an example of a feature that necessarily crosses module boundaries. I don't think there's any possible way we could possibly enumerate and plan for all of these -- trying to do so would undoubtedly fail.

And so our choice would be almost exactly the same as it is today: Either we rewrite large parts of Firefox, or we have a low quality implementation of the feature... Except, if anything, the rewrite is even harder, since the boundaries between the modules are even more rigid, and some of them are even on separate shipping schedules... (And while I agree that it's obvious that if we care about the feature, the rewrite is the correct choice, I'm not sure I think that means much.)

I'm not going to be one to say modularity is bad *per se*, but I don't think it's that clear of a win either. As I see it, you're making a trade-off: you give away flexibility (the ability for anything to poke at anything), and you get 1. some code simplicity and 2. the ability to update modules independently in return.

The code simplicity is valuable, certainly (and I'll get to this), but the ability to separate modules to the point where they're effectively different products with different shipping schedules is (to steal a point made by rnewman in IRC -- sorry!) possibly the most Conway's Law thing imaginable.

As for the code simplicity, I'm somewhat skeptical that it would be a real benefit. It seems to me like all that would really happen is that the complexity would be moved to the module boundaries, which is, IMO not where we want it (as I mentioned above, you can still have bugs in a system where every component works as advertised, these bugs are just harder to track down and prevent through testing).

Anyway, it's very possible that I'm being, err, naively negative here (I'm sure there's a better word for this, I simply can't think of it...). I usually try to keep my mouth shut for these discussions, since I'm nigh-undoubtedly the least experienced person commenting here (what with my grand total of ~7 months of hacking on Firefox, and ~7 years programming), but I felt like I needed to say something, despite the fact that I might be missing the bigger picture.

[Jared Hirsch]

Hi all,

I'd like to comment on the contributor-specific arguments made in Benjamin's article, which seem to be:

- contributors struggle because of the module structure and poor APIs;
- contributors ask for better docs, but the docs are unavoidably bad because the APIs are bad.

TL;DR: I think the bigger problem is poor docs, not poor APIs.

I'm a webdev, like many of our contributors. I work on the Test Pilot team, where I've gotten a crash course in Gecko / addon development over the past 1.5 years, starting with building the Universal Search addon.

I'm used to working with a variety of less-than-perfect, evolving, overlapping APIs; that's the Web Platform that webdevs know and love. Inelegance is no barrier to contributing.

However, I'm also accustomed to having good learning resources: blog posts, examples, API docs, and, if all else fails, a carefully-worded spec. And this is where we, at Mozilla, fall short.

I've personally found that the XPCOM interfaces make sense, but I generally have to read a *lot* of source code to learn how to use them.

Insofar as the WebExtension API proposal is intended to make life better for contributors, I'd suggest investing in the docs instead.

Cheers,

Jared

[Andrew Swan]

As a relative newcomer to the Firefox world, I don't think that a lack of boundaries is the biggest hurdle one needs to overcome to hack on Firefox.  By and large, I've found that there are reasonable abstractions and usually at least some useful documentation.  Building blocks like IDL and JSMs provide a reasonable way to encapsulate some chunk of code, tools like dxr or searchfox make finding relevant interfaces/modules easy, and then between comments and MDN we have docs of variable quality.  Taken all together, there is room for improvement but its not bad.  Where I struggle is figuring out how these pieces fit together, especially in conjunction with larger things like e10s, compartments, etc. that transcend individual modules.  To give a concrete example, I was recently working on producing a notification for an event that begins with a script on a content page taking some action.  Finding PopupNotifications.jsm was easy enough and the docs were pretty clear that creating a notification required a <browser> element.  But figuring out how to get from some code running with a window reference in the content process to a place where I could create a notification took a lot of head scratching and more than a few false starts.  Once I figured it all out, I think that all the pieces of code involved are reasonably encapsulated and even have decent reference documentation.  But again, the big barrier was piecing together the big picture, not a lack of modularity in the actual components involved.  This is of course not an argument against modularity, just some observations about "why hacking Firefox is hard" from a newcomer's perspective.

As for WebExtensions, it is true that high-level APIs is an important goal for the project and to the extent we can use them widely, that promotes Benjamin's goal of modularity.  But the WebExtensions project also has other goals that I think conflict with the idea of using them extensively for browser internals.  The most obvious one is safety -- we've relied heavily on manual reviews for pre-webextensions extensions to ensure that when a user installs an extension that says it does one thing, they be confident that it isn't actually doing something else that they may not expect or want.  This process is more automated in webextensions in a few ways.  One is simply that that webextensions run with limited permissions by default and must explicitly declare further permissions that they need.  But another way is that in many places we've drawn a line around built-in parts of the browser and either forbidden or limited extensions from touching them.  As you might imagine, this has upset a lot of extension developers and there's been some lively discussion about this recently on dev-addons that I encourage you to read if you're interested.  But as it pertains to the idea of using WebExtensions internally, billm suggested earlier in this thread that the WebExtensions permissions system could be used here.  I think that's true in principle but extending the permissions system (and related to that, our processes for reviewing and signing extensions) in this way is not currently a goal for WebExtensions development, and it would take some non-zero amount of work to do.

My 2c etc...

-Andrew

[David Teller]

I would agree that modularity is not the problem – or at least not the
best name for the problem.

In my mind, the problem is that we generally make absolutely no
difference between a private-to-feature API, a private-to-test API, a
private-to-feature-set API, a private-to-firefox API and a public API.

Moreover, our codebase abounds with hidden APIs – for instance, APIs
that are actually accessed through magic observer notifications, through
DOM events, or through magic preferences, or through the category
manager, or by giving magic names to XPCOM components or to some extent
through cross-process messages. Again, these APIs are generally designed
to be private-to-feature or private-to-test but are in practice public.

Oh, and we also have a number of subsystems that are actually
(unspecified, hidden) protocols, rather than simple APIs: things must be
done in a given magic order, otherwise breakage.

Oh, and by the way, most of our APIs don't control their inputs, because
they are initially designed for being private-to-feature or private-to-test.

All this needs to be fixed if we want to be able to compete on feature
implementation speed (and quality) again. Not necessarily today and
certainly not in one stroke, but we definitely need to fix it.


So, here is my list of requirements for any proposal for modularity,
assuming a Go Faster world:

1. we need to be able to implement the APIs in at least JS and Rust;

2. we need to be able to access the APIs in at least JS;

3. we need to plan for the fact that many APIs will eventually be
implemented out-of-thread or out-of-process;

4. clear (and enforced) difference between private-to-feature,
private-to-test, private-to-feature-set, private-to-firefox,
private-to-system-addons and public-to-all-addons;

5. typed APIs with rich types (e.g. not just "Object"/"nsISupports"), à
la WebIDL;

6. no hidden APIs;

7. a clear place for the users of these APIs (including add-on authors,
if they can access these APIs) to request new APIs, more documentation, etc.



There are several ways to fulfill these requirements, which can involve
combinations of platform design, static analysis, instrumentation,
coding conventions and reviews. If WebExtensions fit the bill (or can be
made to fit the bill without breakage), since we are supporting them
regardless, I believe we should go for them.

Cheers,
David

On 10/10/16 22:58, Thom Chiovoloni wrote:
> A few very ill-advised comments:
>
> On Mon, Oct 10, 2016 at 12:50 AM, Benjamin Smedberg
> <benj...@smedbergs.us <mailto:benj...@smedbergs.us>> wrote:
>
> We are shipping these *today*, and shipping is a tedious and fairly
> high-risk process because we don't have modular isolation or testing.
>
>
> I don't think that modularity is the problem here. After all, its very
> possible to have bugs in systems where every component works as
> specified. (I'd even argue that -- depending on how you define
> "component" -- this is where most bugs occur.)

[David Teller]

Of course, I should add

8. modularization should not kill startup performance, runtime
performance, binary size or memory usage.

On 11/10/16 10:29, David Teller wrote:
[...]

> So, here is my list of requirements for any proposal for modularity,
> assuming a Go Faster world:
>
> 1. we need to be able to implement the APIs in at least JS and Rust;
>
> 2. we need to be able to access the APIs in at least JS;
>
> 3. we need to plan for the fact that many APIs will eventually be
> implemented out-of-thread or out-of-process;
>
> 4. clear (and enforced) difference between private-to-feature,
> private-to-test, private-to-feature-set, private-to-firefox,
> private-to-system-addons and public-to-all-addons;
>
> 5. typed APIs with rich types (e.g. not just "Object"/"nsISupports"), à
> la WebIDL;
>
> 6. no hidden APIs;
>
> 7. a clear place for the users of these APIs (including add-on authors,
> if they can access these APIs) to request new APIs, more documentation, etc.
>
>

[Axel Hecht]

On 10/10/16 06:50, Benjamin Smedberg wrote:

Thank you all for the beginnings of a great discussion!

I'd like to follow up on a few points which I think were misunderstood.

• The focus of my post is about requiring good module boundaries for code that wants to move quickly, and the notion that WebExtensions are the right way to do that. Although I like indulging in talk about making the entire browser modular, that is clearly not something we can accomplish soon. I am talking about the pipeline of development that we're building which starts out with test pilot, through SHIELD studies, to system addons. We are shipping these *today*, and shipping is a tedious and fairly high-risk process because we don't have modular isolation or testing.
  

Looking at the research others have done, it seems that WebExtensions as they exist today don't enable the experiments we want to ship today.

To take a different spin on this, we would need code that provides the APIs for our experiments, and our experiments, likely to ship in lock step?

This might benefit the shipping process, though I'm curious on the details on how in practice.

I'd expect that to be constrained to changes to the WebExtension part though, and that in scenarios where the API level needs to iterate, we'd be in an environment that's not different to stock firefox code?

• I also don't think we should limit ourself to WebExtensions as they are today. WebExtensions are purely asynchronous today because of design decisions that Chrome made and we also accept that addons shouldn't be allowed to block the browser UI. That doesn't mean that we can't make API surfaces which are synchronous/blocking, if we feel that's the right tradeoff.
• We can make choices about the "stability" of API surface: both APIs we're building internally, and even APIs we expose to addons. I'm going to follow up separately about what I think we should have learned from XPCOM, but if there's anything we should learn it's that frozen API surfaces are often the wrong choice.
  

We have a stated desire to let certain teams move much more quickly. This includes Hello in the past, it includes a bunch of test pilot and shield studies now, and it will include activity stream work in the future. What I was trying to point out, with a long history of academic research, is that there is a "physics" of software engineering that we cannot avoid: projects and teams move faster when there are fewer modular entanglements, smaller modules in general. We can train new employees and attract new contributors better with smaller boundaries around the code that they have to understand to be productive, and better documentation aout where the boundaries are.

I agree that for problem statements that like constrained modularity, building that modularity out and building upon it is going to help the software development process.

Building upon your choice of "physics", I think that this model does have a limited problem space to which it can be applied to. Similar to Newtonian Mechanics, say.

We're probably not good enough yet on taking advantage of the places where the model works.

That said, I don't think that we can categorically assert that the model applies to a problems statement at hand.

That'd be more a math problem than a physics model.

Axel

Richard is right that if we try to "remodularize" around our existing infrastructure, some things are going to be really hard. It's basically impossible to provide atomic sync with things designed the way they are. But let's say that atomic sync is something that we believe is important: I believe we *have to* spend the time to reorganize the system so that sync as a module that stores passwords, bookmarks, etc is lower/more basic in our module structure, and then we build our bookmarks UI, location bar, etc, on top of that core structure. We are in charge of designing these things.

We always *have* a structure. I believe that currently we have a structure where Firefox is a large monolith which is very difficult to change effectively. To change that, we don't need to and shouldn't rewrite it all. But we should start carving off pieces so that they are smaller and have real boundaries. I believe we should start with the things which are trying to move faster right now. Maybe we should also start at the other end with core infrastructure like sync. If we carefully carve off pieces, keeping a longer-term vision in mind, we will end up in a place where there either is no monolithic center, or we carved it down enough that we can document/understand/hack on it more easily.

--BDS

On Thu, Oct 6, 2016 at 11:08 AM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

I spent a week writing a thing about modularity, webextensions, and going faster. I think it's important for us to decide the module structure of our code especially as we start shipping independent modules/going faster. And I believe that having better module structure, boundaries, and documentation is critical to our teams being more agile and also attracting contributors to the project.

http://benjamin.smedbergs.us/blog/2016-09-03/modularity-and-webextensions/

I personally think that we should double down on WebExtensions as a model and start using that for large parts of Firefox. But Andy McKay and Rob Helper had some good counter-thoughts and I've asked them to post here to elaborate.

In the post I asked everyone to send followups to firefox-dev, so I wanted to start a thread here to collect responses. Over the next months I'd like this to turn into a firm decision about how we're going to build system addons; but I'd like to start by seeing what feedback people have and even whether I've framed the problem correctly.

--BDS

[David Bruant]

Le 07/10/2016 à 23:48, Gregory Szorc a écrit :

3) If we're talking about a significant change of architecture to the Firefox frontend and the high costs associated with that change, I'll throw out an alternative idea: (re)write the frontend in a typed language like Rust. I argue that a typed language - and Rust especially - would deliver "well-defined, documented, and enforced module boundaries or interfaces" better than JavaScript ever will.

Le 08/10/2016 à 00:12, Robert Helmer a écrit :

+1. I think we should probably stick with HTML/JS/CSS for basic UI
stuff like layout, setting up button events, DOM manipulation etc, but
a lot of the code we have in JSMs right now would be much safer to
implement in Rust, and most likely faster.

Microsoft faced the challenge of writing large JavaScript applications. They ended up inventing TypeScript for this very purpose. Facebook followed a couple of years later with Flow. Both have differences, but in practice the differences are barely noticeable as far as I'm concerned.
Note that the Firefox devtools team is already using Flow https://github.com/devtools-html/debugger.html

Because of JavaScript limitations and "culture", the type system of either with never be as expressive as the Rust type system can be. However, I've never heard of it as an strong argument to move away from TypeScript or Flow.

Given the advances of JS perf engines the last few years, and things like asm.js and coming primitives low-level like shared memory (which Firefox is the only one to ship under flag at the time?), there are enough options on the table so that language-level performance should not be an argument to switch language.

4) I have significant concerns about the concept of system add-ons. Currently, the composition of Firefox (in terms of code) is essentially deterministic from the commit in mozilla-central it was built from. We know exactly what Firefox code is running (OK, /should/ be running) on users' machines. System add-ons fundamentally change that from ~1 combination of software components to N, where N could be absurdly large (10^3 or larger). It is hard enough to track down intermittent failures in test automation with a well-defined software configuration. System add-ons introduce whole new classes of bugs related to how different versions of add-ons interact with each other.

I lack the proper experience to answer this point with appropriate authority, but I'll try to say something relevant anyway.
I think the idea is that the point of system addons is to find parts of the system that do not interact with one another but only with the Web Extensions API. In that context, the combinations should not matter, only the WE boundary and how the addons interact with it.
Whether this idea is wishful thinking or plausible is the heart of the debate.

Another question is whether the Web Extensions API is the proper boundary. It's misdesigned in many ways (some I would consider major) and allows too many interactions between addons by default. Maybe a tighter API would be a better start to reduce the potential of bad interactions.

David

[David Teller]

On 12/10/16 13:33, David Bruant wrote:
> Microsoft faced the challenge of writing large JavaScript applications.
> They ended up inventing TypeScript for this very purpose. Facebook
> followed a couple of years later with Flow. Both have differences, but
> in practice the differences are barely noticeable as far as I'm concerned.
> Note that the Firefox devtools team is already using Flow
> https://github.com/devtools-html/debugger.html
>
> Because of JavaScript limitations and "culture", the type system of
> either with never be as expressive as the Rust type system can be.
> However, I've never heard of it as an strong argument to move away from
> TypeScript or Flow.

Fwiw, we are currently trying to come up with a strategy to use either
Flow, TypeScript or Google Closure Compiler (which also performs
type-checking on JS) on our codebase.

Of course, the tools only makes sense if we have clean, well-designed
APIs, so the discussion on WebExtensions (or alternatives) is still on.

So far, from this discussion, I have parsed the following proposals:
- [customized] WebExtensions (which intends to answer the question "how
do we present the APIs?");
- Rust (which intends to answer the questions "how do we implement the
APIs"/"how do we encourage APIs to be well-typed");
- don't change anything (which, in my books, doesn't answer any question);
- "we don't understand the question yet".

Am I missing proposals?

Cheers,
David

[Gijs Kruitbosch]

On 12/10/2016 13:29, David Teller wrote:
> On 12/10/16 13:33, David Bruant wrote:
>> Microsoft faced the challenge of writing large JavaScript applications.
>> They ended up inventing TypeScript for this very purpose. Facebook
>> followed a couple of years later with Flow. Both have differences, but
>> in practice the differences are barely noticeable as far as I'm concerned.
>> Note that the Firefox devtools team is already using Flow
>> https://github.com/devtools-html/debugger.html
>>
>> Because of JavaScript limitations and "culture", the type system of
>> either with never be as expressive as the Rust type system can be.
>> However, I've never heard of it as an strong argument to move away from
>> TypeScript or Flow.
> Fwiw, we are currently trying to come up with a strategy to use either
> Flow, TypeScript or Google Closure Compiler (which also performs
> type-checking on JS) on our codebase.
>
> Of course, the tools only makes sense if we have clean, well-designed
> APIs, so the discussion on WebExtensions (or alternatives) is still on.
>
> So far, from this discussion, I have parsed the following proposals:
> - [customized] WebExtensions (which intends to answer the question "how
> do we present the APIs?");
> - Rust (which intends to answer the questions "how do we implement the
> APIs"/"how do we encourage APIs to be well-typed");
> - don't change anything (which, in my books, doesn't answer any question);

Being able to do type checking is orthogonal to using WebExtensions for
browser features.

If you could convince people that it's worth annotating all the existing
JS in browser/ and toolkit/ with these types, you could do that today.
Heck, you could probably use IDL and WebIDL data to infer some of the
types automatically already. Either way, none of that requires the much
larger effort of switching to either Rust or WebExtensions.

I'll note that you're assuming that we all agree type checking is going
to add significant value, and in particular that that value outweighs
the costs of switching technologies to Rust or WebExtensions, or the
cost of annotating all our existing code. However, you have provided no
evidence for that view.

Finally I'll note that the existing WebExtensions APIs are very very
very far from enough to write a browser with. Benjamin already
acknowledged this. We would need to write new ("proprietary" in the
not-specced/in-general-web-use sense) APIs. The implementations of
WebExtensions APIs are not currently type-annotated (that I've been able
to see when reviewing them, anyway), and so AFAICT "switching to
WebExtensions" would just be shifting the annotation problem to the
layer that implements such new APIs.

All of this is of course on top of the fact that significant sets of our
"APIs" (using the term very loosely) are implemented in XBL and XUL, and
none of TypeScript/Flow/... know how to deal with that.


In summary, I'm sympathetic in principle to wanting to do more static
checking on our codebase, but I think "rewrite all the things in other
technology X" is the wrong starting point for achieving that goal. There
might be other reasons why (progressively) rewriting stuff is desirable,
but I don't think requiring that rewrite for the static checking is an
expedient way of getting the static checking, nor does wanting static
checking per se justify a rewrite.

~ Gijs

[David Teller]

On 12/10/16 14:41, Gijs Kruitbosch wrote:
> In summary, I'm sympathetic in principle to wanting to do more static
> checking on our codebase, but I think "rewrite all the things in other
> technology X" is the wrong starting point for achieving that goal. There
> might be other reasons why (progressively) rewriting stuff is desirable,
> but I don't think requiring that rewrite for the static checking is an
> expedient way of getting the static checking, nor does wanting static
> checking per se justify a rewrite.

Good, because that's definitely not what I have in mind. I'm writing that:

1/ JS static checking is currently being planned – I'm not trying to
argue for or against it in this thread, or detail the plan, but if you
wish, we can do that in another thread;

2/ if we have poor modules, static checking is mostly going to confirm
that we have poor modules, but if we have better modules, static
checking is going to be more useful.

I agree that static type-checking is mostly orthogonal with the choice
of a way of modularizing the code (regardless of whether this "way" is a
specific technology, a set of review guidelines or anything inbetween).
On the other hand, I strongly believe that supporting strong typing
(note that I didn't write "static typing") at API boundaries is important.

> Finally I'll note that the existing WebExtensions APIs are very very >
very far from enough to write a browser with. Benjamin already
> acknowledged this. We would need to write new ("proprietary" in the
> not-specced/in-general-web-use sense) APIs. The implementations of
> WebExtensions APIs are not currently type-annotated (that I've been
> able to see when reviewing them, anyway), and so AFAICT "switching to
> WebExtensions" would just be shifting the annotation problem to the
> layer that implements such new APIs.

As a side-note, you may be aware that Alexandre and Vivien already have
a prototype of a browser based on Gecko and WebExtensions. If you wish
to discuss feasibility of writing such a browser, or what is missing
from WebExtensions to do so, you should discuss it with them.

Cheers,
David

[Gijs Kruitbosch]

On 12/10/2016 14:01, David Teller wrote:
> 2/ if we have poor modules, static checking is mostly going to confirm
> that we have poor modules, but if we have better modules, static
> checking is going to be more useful.

This doesn't make a lot of sense to me. The annotations are, I assume,
for globals as well as function/method signatures. Why would it matter
how we organize modules?
(Which isn't to say that we shouldn't pay attention to modular code, but
it doesn't seem like it'll impact static type checking.)

> I agree that static type-checking is mostly orthogonal with the choice
> of a way of modularizing the code (regardless of whether this "way" is a
> specific technology, a set of review guidelines or anything inbetween).
> On the other hand, I strongly believe that supporting strong typing
> (note that I didn't write "static typing") at API boundaries is important.

Right now there are no API boundaries at all in the strict sense of the
word, or alternatively, any JS global or function/method is an API boundary.

I'm also not sure why, even for "properly structured modules" of
whatever kind, we wouldn't care about types used internally, and only
about the API boundaries...

~ Gijs

[David Teller]

On 12/10/16 15:33, Gijs Kruitbosch wrote:
> On 12/10/2016 14:01, David Teller wrote:
>> 2/ if we have poor modules, static checking is mostly going to confirm
>> that we have poor modules, but if we have better modules, static
>> checking is going to be more useful.
> This doesn't make a lot of sense to me. The annotations are, I assume,
> for globals as well as function/method signatures. Why would it matter
> how we organize modules?
> (Which isn't to say that we shouldn't pay attention to modular code, but
> it doesn't seem like it'll impact static type checking.)

Let's call it a hunch (based on my experience of static checking,
including static typing). I'm, of course, willing to be favorably surprised.

>> I agree that static type-checking is mostly orthogonal with the choice
>> of a way of modularizing the code (regardless of whether this "way" is a
>> specific technology, a set of review guidelines or anything inbetween).
>> On the other hand, I strongly believe that supporting strong typing
>> (note that I didn't write "static typing") at API boundaries is
>> important.
> Right now there are no API boundaries at all in the strict sense of the
> word, or alternatively, any JS global or function/method is an API
> boundary.

Indeed. As I understand this thread, this is exactly the essence of the
problem we're discussing.

> I'm also not sure why, even for "properly structured modules" of
> whatever kind, we wouldn't care about types used internally, and only
> about the API boundaries...

We certainly do. If I wrote something that lead you to believe that we
don't, I apologize for the lack of clarity.

Cheers,
David

[David Bruant]

Annotating all the existing JS is not a requirement to use TypeScript or
flow (IIRC Google Closure Compiler is a different story).
Interesting blog post from 2013 of someone switching a 100kLOC lib:
http://news.turbulenz.com/post/47457658846/mostly-painlessly-migrating-a-3d-game-engine-to
"The first step I took was to simply rename each of our .js files to .ts
and create a trivial build step to convert the each of these new .ts
files to JavaScript [with the typescript compiler].
The advantage of this apparently trivial step was that regular
development could continue while we gradually expanded the type
definitions in the same files. This approach is only really viable
because TypeScript extends rather than replaces JavaScript, and it’s
actually very important."
Now, TypeScript has a --allowJs option which makes that you don't even
need to rename the files anymore. I'm sure it's the same for flow.

Anyway, given TypeScript does type inference, "switching" to TypeScript
is merely using the compiler to report error and do nothing else
(--noEmit option) without having to annotate anything.
As a side note, it still surprises me that most JS projects (front-end
or Node) don't use the Typescript (or flow) compiler. It just takes
running it on normal JS code to have errors reported for free with no
consequence; errors that are not reported by any other tool.

Now, to fully benefit from the static type checking, annotations are of
course the way to go, but they can be added gradually over time. From
experience, it's not a bad exercice and can be seen more as a welcome
form of documentation. "Documentation" that is run against the code and
that the compiler invites you to fix when the code changes.

David

[Mike Conley]

I'm trying to synthesize what I've read in this thread. I'm focusing more on the modularity subject, and less on things like tools for ensuring type-correctness or anything like that.

Here's what I'm reading:

1. It might be worth considering re-implementing some browser components using WebExtensions. This would likely involve adding new, probably privileged APIs.
   
2. The current set of modules, as they're defined, may not be optimal these days. rnewman brought up a few examples of where our current module definitions / boundaries hurt us, confuse things, and / or slow us down (Satchel and Password Manager was one such example).

Here are my thoughts. Hopefully I'm not repeating other people here!

1. XPCOM / XPIDL, for better or worse, offered a boundary and interface system. I think those interfaces were originally intended to be pretty stable. Down the line, I seem to recall us deciding that having frozen interfaces there was slowing us down, and so now those interfaces are changed at will. I understand that we were more constrained because more applications were relying on the underlying interfaces, but this all starts to feel kinda familiar. Are we at risk of creating a new set of interfaces and silos that get holes poked through them over time because they slow us down instead of speeding us up? Are we repeating ourselves? If so, what did we learn last time that will make this time better?
   
2. Surely we're not the first large application that's had to grapple with this kind of design decision. Are there any success or failure stories from outside of Mozilla that we can draw inspiration or lessons from?
3. Is there a browser component that we can experiment with, and attempt re-implementing with the WebExtension API to see how it feels? Or, to take rnewman's angle here, is there an opportunity here to draw some new boundary lines and define a new (but small) WebExtension Thing that combines capabilities from several of our current modules for great good?
   

-Mike

[Richard Newman]

As a side note, it still surprises me that most JS projects (front-end or Node) don't use the Typescript (or flow) compiler. It just takes running it on normal JS code to have errors reported for free with no consequence; errors that are not reported by any other tool.

For a few reasons:

• JS build systems are already hairy. Are you going to Babel then TypeScript, then webpack, then…?
• Optional types mean an easy on-ramp, but the cost-benefit exchange starts off negative.

We tried Flow for Tofino. At the time it didn't support Windows, so it was an optional build step… and so it mostly turned into a "make the compiler happy" game of chasing some error that someone else introduced a few days before. We caught one or two small bugs, in exchange for a bunch of irritation. At the time, particularly for Tofino's front-end React code (which already did prop type checking etc.), we decided the juice wasn't worth the squeeze. I hope to revisit that decision at some point

nalexander and I have been using the Google Closure Compiler by way of ClojureScript on Datomish, and it works fine for its purpose, but it isn't much use for type checking (particularly with CLJS on top).

Much of the architectural benefit of types comes from using a sophisticated type system well, so that broad classes of errors or modeling problems are simply impossible to express. It takes a big investment in Flow or TypeScript to start to see the benefits you see from day one with a language like Swift.

(Apologies for dragging out a sidenote!)

[David Teller]

On 12/10/16 16:45, Mike Conley wrote:
> I'm trying to synthesize what I've read in this thread. I'm focusing
> more on the modularity subject, and less on things like tools for
> ensuring type-correctness or anything like that.
>
> Here's what I'm reading:
>

> 1. It might be worth considering re-implementing some browser

> components using WebExtensions. This would likely involve adding
> new, probably privileged APIs.

> 2. The current set of modules, as they're defined, may not be optimal

> these days. rnewman brought up a few examples of where our current
> module definitions / boundaries hurt us, confuse things, and / or
> slow us down (Satchel and Password Manager was one such example).

There is also a discussion on whether we should base our "new" modules
on WebExtensions on something else.

So far, I have seen some suggestions of using Rust as a "something
else", but I believe that this is something of an implementation detail
at that stage.

> Here are my thoughts. Hopefully I'm not repeating other people here!
>

> 1. XPCOM / XPIDL, for better or worse, offered a boundary and interface

> system. I think those interfaces were originally intended to be
> pretty stable. Down the line, I seem to recall us deciding that
> having frozen interfaces there was slowing us down, and so now those
> interfaces are changed at will. I understand that we were more
> constrained because more applications were relying on the underlying
> interfaces, but this all starts to feel kinda familiar. Are we at
> risk of creating a new set of interfaces and silos that get holes
> poked through them over time because they slow us down instead of
> speeding us up? Are we repeating ourselves? If so, what did we learn
> last time that will make this time better?

Good question. In my experience, the main issues with XPCOM/XPIDL are as
follows:

- in practice, we exposed everything as public, something we want to avoid;
- given the granularity, frozen interfaces was a momentum killer but
unfrozen interfaces was an extensions killer;
- also, the resulting types (and error messages) are very un-JavaScript-ish.


> 2. Surely we're not the first large application that's had to grapple

> with this kind of design decision. Are there any success or failure
> stories from outside of Mozilla that we can draw inspiration or
> lessons from?

Good question. I don't know of any.

> 3. Is there a browser component that we can experiment with, and

> attempt re-implementing with the WebExtension API to see how it
> feels? Or, to take rnewman's angle here, is there an opportunity
> here to draw some new boundary lines and define a new (but small)
> WebExtension Thing that combines capabilities from several of our
> current modules for great good?

Have you looked at ochameau's experiment with WebExtension?

See
http://techno-barje.fr/post/2016/03/16/shipping-firefox-features-as-addon/
and followups.

Cheers,

[Richard Newman]

Excellent summary, mconley. (And excellent thoughts besides.)

I think most of the pushback on this list is to the ideas of:

1. Splitting up current pieces (emphasis on the plural) of Firefox, somehow making it easier to work on them. That presupposes that our modules are currently close to correct.

2. Shipping these pieces separately, somehow making it easier to move faster. That presupposes that the workaday troubles of contributing to Firefox are more important than conceptual load.

3. WebExtensions being a fashionable panacea. To me that sounds like B2G WebFoo.

Without these, one reduces the argument to "better modularity would be nice, and it might help to ship some things at different rates when suitable" — and I think we can agree on both of those!

Or, to take rnewman's angle here, is there an opportunity here to draw some new boundary lines and define a new (but small) WebExtension Thing that combines capabilities from several of our current modules for great good?

I'm hoping to end up with Datomish being one such cross-cutting feature. It has a very small defined API surface, is fully asynchronous, and is designed for cross-module use. At some point I'm planning to see if I can implement the stock Places (and Satchel, and…?) APIs on top. At the very least it'll be an interesting performance test!

[Gijs Kruitbosch]

Right, but this is independent of the technology you use. If we expose
the Rust/WebExt stuff to external people (so e.g. add-ons can implement
a replacement location bar) then we're still going to have that problem.

> - given the granularity, frozen interfaces was a momentum killer but
> unfrozen interfaces was an extensions killer;

Yuuup. Hard decisions are hard.

To me, it felt like webextensions (and before that, the SDK) were an
attempt to have a secondary API layer that you used for add-ons, and
your own inter-modular-code could be as unfrozen as you liked. This by
definition means that you can't/shouldn't use said secondary API layer
for your own stuff because then you're back to square 1 where our own
code and add-ons share the same API layer and now everybody is sad.

> - also, the resulting types (and error messages) are very un-JavaScript-ish.

Our JS bindings for XPCOM could be (have been) better. WebIDL is already
better in that respect, but isn't really geared to JS-based
*implementations* (rather than consumption) of static/singleton things,
which is part of why it's (IMO) underused today in frontend-land.

~ Gijs

[Chris Hutten-Czapski]

>>  2. Surely we're not the first large application that's had to grapple
>>     with this kind of design decision. Are there any success or failure
>>     stories from outside of Mozilla that we can draw inspiration or
>>     lessons from?
>
>Good question. I don't know of any.

The BlackBerry 10 browser architected itself primarily as modules and extensions in webtech in 2011-2012 (released January 2013). By increasing the friction necessary to get at privileged APIs (because we didn't autogenerate them particularly well, especially for complex types) we ensured that most operations were completed through extending the support of web standards and renderer performance.

That being said, it was developed rapidly by a small team to support one platform and about six screen sizes, so I'm not sure how much of my experience with it might be of use.

Feel free to pick my brains on the subject at your leisure, but to begin, here's a couple lessons learned:

1) HTML+CSS in 2011 was a poor widget language. Encapsulating components wasn't possible (Hello Custom Elements and Shadow DOM), but restyling existing ones was spectacularly fruitful (webkit's stylability of form components far exceeds gecko's to this day). Specifically, the DOM is a general-purpose layout mechanism so a lot of performance shortcuts that can be made in widget languages cannot be assumed by the renderer leading to terrible slowdowns. If Firefox goes this route, it might benefit from focusing on using CSS to hint at usage (overflow: hidden can be an explicit intent. What I wouldn't have given to have will-change in 2011)

2) APIs are important. Components are important. JS wasn't particularly happy with either. Component loading was a pain at startup. Strings are heavier than you think. i18n has a deep cost. Simply scanning the JS off of disk eats millis out of every user which adds up to hours at userbase scale.

3) A central architecture and design vision was paramount to accomplishing what we did. Before the inter-component framework was put in place, work circled as if in a sink drain. Before the lines had been drawn (albeit in pencil) the frontend had no shape, only colour.

:chutten

[Gijs Kruitbosch]

On 12/10/2016 16:09, Alexandre poirot wrote:

1. XPCOM / XPIDL, for better or worse, offered a boundary and interface system. I think those interfaces were originally intended to be pretty stable. Down the line, I seem to recall us deciding that having frozen interfaces there was slowing us down, and so now those interfaces are changed at will. I understand that we were more constrained because more applications were relying on the underlying interfaces, but this all starts to feel kinda familiar. Are we at risk of creating a new set of interfaces and silos that get holes poked through them over time because they slow us down instead of speeding us up? Are we repeating ourselves? If so, what did we learn last time that will make this time better?
   

We do have some boundaries with XPCOM, some others with JSM, yet some others with message managers.

But that's just too many different kind of boundaries/interfaces. If we only had one border, everything would have been much better.

I agree with having one type of border between internal and external code instead of many. I would also be happy to have only 1 border between different bits of internal code, though I think that's difficult in practice, mostly because you end up with several abstraction levels (if we keep using HTML-like things for frontend, how do you keep CSS for different "modules" separate and isn't that really a different boundary from the boundaries between your JS, etc.)

However, I feel very strongly that we should be using a different border between add-ons and the browser vs. between different internal components of the browser. It seems convenient at first to use the same abstraction, but as David pointed out, this bit (and still bites) us a lot with XPCOM. You freeze things and then you can't change anything. If you don't freeze things, add-ons break all the time. As a simple example: containers/userContextId requires changes to a lot of the tab/browser "interfaces", as well as some XPCOM things such as the cookie service. It's hard to insulate add-ons from that without having 2 separate API layers. Other people have brought this up before: using your public API layer to implement your "own" UI/service leads to pain either/both for you and the consumers of the public API, also on the wider web.

If you look, again, at session restore you will have to go through all of them and it is far from being obvious and will feel very hard to approach to anyone except mozilla employees who have to learn all these things. Web extension abstracts all that. The difference between a content script and a background page is very small.

1. Is there a browser component that we can experiment with, and attempt re-implementing with the WebExtension API to see how it feels? Or, to take rnewman's angle here, is there an opportunity here to draw some new boundary lines and define a new (but small) WebExtension Thing that combines capabilities from several of our current modules for great good?
   

Session restore?
http://blog.techno-barje.fr/post/2016/03/14/session-restore-web-extension/

The big advantage to this particular feature is that it doesn't involve much UI, which is where WebExtension is the most limited.

... but then, is avoiding the UI problem really going to be helpful in the future? How do we transition the experience from a hypothetical successful session restore add-on to other frontend features if we haven't solved that problem? And, most negatively, how do we avoid that becoming boundary type number N + 1 (after XPCOM, JSM, XBL, message managers, system add-ons, ...) that we then don't get rid of and have to teach every contributor (employee or volunteer)?

~ Gijs

[Gervase Markham]

On 12/10/16 16:13, Gijs Kruitbosch wrote:
> To me, it felt like webextensions (and before that, the SDK) were an
> attempt to have a secondary API layer that you used for add-ons, and
> your own inter-modular-code could be as unfrozen as you liked. This by
> definition means that you can't/shouldn't use said secondary API layer
> for your own stuff because then you're back to square 1 where our own
> code and add-ons share the same API layer and now everybody is sad.

Yes, surely this?

Gerv

[Alexandre poirot]

2016-10-12 16:45 GMT+02:00 Mike Conley <mco...@mozilla.com>:

1. It might be worth considering re-implementing some browser components using WebExtensions. This would likely involve adding new, probably privileged APIs.
   

Yes. It would involve introducing new APIs, at least new features in the existing APIs.
For example for session restore, we would need to support a new parameter in addTab to allow loading tabs offloaded in the background (i.e. not load the url, but just display the tab and load it when you switch to it). I don't see any reason to not expose that to all addons.

But we would also need something to save and restore the various storage web api (localStorage, sessionStorage). This time it may be something we want to keep for internal addons?

1. XPCOM / XPIDL, for better or worse, offered a boundary and interface system. I think those interfaces were originally intended to be pretty stable. Down the line, I seem to recall us deciding that having frozen interfaces there was slowing us down, and so now those interfaces are changed at will. I understand that we were more constrained because more applications were relying on the underlying interfaces, but this all starts to feel kinda familiar. Are we at risk of creating a new set of interfaces and silos that get holes poked through them over time because they slow us down instead of speeding us up? Are we repeating ourselves? If so, what did we learn last time that will make this time better?
   

We do have some boundaries with XPCOM, some others with JSM, yet some others with message managers.

But that's just too many different kind of boundaries/interfaces. If we only had one border, everything would have been much better.

If you look, again, at session restore you will have to go through all of them and it is far from being obvious and will feel very hard to approach to anyone except mozilla employees who have to learn all these things. Web extension abstracts all that. The difference between a content script and a background page is very small.

2. Is there a browser component that we can experiment with, and attempt re-implementing with the WebExtension API to see how it feels? Or, to take rnewman's angle here, is there an opportunity here to draw some new boundary lines and define a new (but small) WebExtension Thing that combines capabilities from several of our current modules for great good?
   

[Dan Mosedale]

2016-10-12 8:26 GMT-07:00 Gijs Kruitbosch <gijskru...@gmail.com>:

If you look, again, at session restore you will have to go through all of them and it is far from being obvious and will feel very hard to approach to anyone except mozilla employees who have to learn all these things. Web extension abstracts all that. The difference between a content script and a background page is very small.

1. Is there a browser component that we can experiment with, and attempt re-implementing with the WebExtension API to see how it feels? Or, to take rnewman's angle here, is there an opportunity here to draw some new boundary lines and define a new (but small) WebExtension Thing that combines capabilities from several of our current modules for great good?
   

Session restore?
http://blog.techno-barje.fr/post/2016/03/14/session-restore-web-extension/

The big advantage to this particular feature is that it doesn't involve much UI, which is where WebExtension is the most limited.

... but then, is avoiding the UI problem really going to be helpful in the future? How do we transition the experience from a hypothetical successful session restore add-on to other frontend features if we haven't solved that problem? And, most negatively, how do we avoid that becoming boundary type number N + 1 (after XPCOM, JSM, XBL, message managers, system add-ons, ...) that we then don't get rid of and have to teach every contributor (employee or volunteer)?

​The thing that doesn't seem to have yet been called out in this discussion is the difference between technologies that are local-only, versus ones that have broad documentation and support on the web at large (WebExtensions is arguably such a one).

It seems like some of the real wins going forward involve opportunistically aligning ourselves with existing larger stable development technologies/communities (presumably mostly from the web).  (E.g. incrementally migrate from JSMs to CommonJS). Part of the trick here is to, over the longer term,

slowly let go of old proprietary technologies that are only known here, in favor of technologies known to large swaths of the web-development community, and broadly supported in common tooling.

An example of this is that most (many? all?) of the various Firefox sub-projects that use React have used mocha/chai/sinon for unit testing.  Execution is at least an order of magnitude faster than any of the test frameworks commonly used in-tree, it's well-supported, constantly updated, has lots of interesting build tooling integrations in most IDEs and even in eslint, runs headless, etc. 

Dan

[Brunoais]

I Agree with gijs (quoted e-mail).

I think that the interface for addons should be different from the
interface for internal use. The internal code should not become frozen
due to addons using them. Instead, market research should be done to
know what addons are trying to do and then create API that lasts about
as long as the functionality it is abstracting.
If the functionality disappears, kill the API. If the functionality
fundamentally changes, deprecate stuff and make adjusted API for the
changes.

WebExtensions is a good base for a stable API for extensions but it is a
really bad API for the browser itself. On the other hand, XPCOM / XPID,
with some cleanup and less general objects seem like a good way to
continue for internal use (not for extensions).

Then there's what addons/extensions require and WebExtensions do not
provide. I think that also should be addressed.

> .

[Gijs Kruitbosch]

Aligning ourselves with the web is a good goal, IMO, but seems problematic with WebExtensions. WebExtensions, its async-ness, and its lacking support for actual UI creation, and interfacing with related UI on a single main thread with multiple async components, is (to the best of my knowledge) not a solved problem**. Then there's all the stuff about exposed vs. internally used API surfaces. Using React for the internal implementation of the frontend components would be better-understood (even if I'm personally not a massive React fan...).

The problem then would be how to integrate it piecemeal into the main Firefox UI and replace UI components, how to prioritize that work (which will wholesale break lots of add-ons) and of course whether having the entire thing in react (with flux or redux or whatever the latest thing is called) is the right choice (from the outside, it seems like the reactjs world uses a different toolset every 6 months, so picking something and sticking with it may not be the easiest thing) and how to make sure perf is acceptable (where a lot of our perf measurements right now rely on the infra we have, and so we'd have to rewrite them into something that works both before + after the conversion in order to compare things, etc.). Those are all solvable problems, but there would need to be a commitment that people want to invest in this, and broader agreement on tools and direction.

As for testing, it's not clear to me that those frameworks would be usable for privileged toplevel browser window stuff, which is the primary way we use mochitests these days***. Happy to be proven wrong. Note that mochitests were also originally adapted from web things - the mocha/chai/sinon of yesteryear, if you will - the important part being "adapted". We can do the same thing here, but the idiosyncracies are going to be "a little weird" today, worse in 2-5 years, and we'll like throw "well-supported" and perhaps some of the tooling integrations out of the window. In other words, not sure if that problem won't be reoccurring in due time irrespective of what unittest framework we decide on today.

~ Gijs

** I know someone upthread mentioned transactionality. I don't buy that that solves the problem I flagged up. Happy to discuss that more, but not sure pigeonholing the discussion into that direction is helpful, so not going to do that here.
*** mochitest-plain used to be mostly used by platform, but AIUI these days they use web platform tests, so I don't think we're actually creating many new mochitest-plain tests.

[David Teller]

I have summarized in a previous post [1, 2] my thoughts about our
current XPCOM/XPIDL + jsm boundaries and what I expect from any
replacement boundaries. I believe that this addresses the key parts of
your response.

And yes, definitely, WebIDL APIs are more JS-friendly than XPIDL. I'm
pretty sure that they can be implemented in JS, fwiw. There is just the
problem that they are centralized, but for a clear and well-defined API,
maybe that's not a problem at all.

[1] https://mail.mozilla.org/pipermail/firefox-dev/2016-October/004684.html
[2] https://mail.mozilla.org/pipermail/firefox-dev/2016-October/004685.html

Cheers,
David

[Gijs Kruitbosch]

Yes, you can implement webidl things in JS. However, you can't have a
JS-implemented static WebIDL component:
https://bugzilla.mozilla.org/show_bug.cgi?id=863952 . This means that
things that we do with JSMs or js-implemented (singleton) XPCOM
components right now are usually not easily portable to WebIDL.

~ Gijs

[Ian Bicking]

On Sun, Oct 9, 2016 at 11:50 PM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

• The focus of my post is about requiring good module boundaries for code that wants to move quickly, and the notion that WebExtensions are the right way to do that. Although I like indulging in talk about making the entire browser modular, that is clearly not something we can accomplish soon. I am talking about the pipeline of development that we're building which starts out with test pilot, through SHIELD studies, to system addons. We are shipping these *today*, and shipping is a tedious and fairly high-risk process because we don't have modular isolation or testing.
  

Shipping through Test Pilot is much easier than shipping through Go Faster/System Add-on, simply because the impact of mistakes is so much smaller.  Mistakes hit fewer people, and if you really mess things up someone can uninstall the add-on and we've only lost a feature user, not a Firefox user.  In both cases the release process could be improved, but it's unrelated to WebExtensions.

I also think exactly the kind of pacing of these ideas makes WebExtensions a hard fit, though Hybrid WebExtensions would be a good fit.  Page Shot could be rewritten fairly easily to be 95% WebExtension.  The last 5% could be fixed with probably-acceptable product changes, but we'd be left wondering... is there something we could do outside the boundaries of WebExtensions that would be an important and valuable product direction? The data-driven process we want to do requires us to actually ship those ideas. And then be ready to un-ship them as well, ideally without making the WebExtension team run around for our capricious and unfounded requests.

Making System Add-ons less stressful to ship would be great. It's an important goal, because stress leads to avoidance or other maladaptive behavior.  But I'm skeptical about doing it through code modularity.

[Andrew McKay]

On 12 October 2016 at 12:57, Ian Bicking <ibic...@mozilla.com> wrote:
> I also think exactly the kind of pacing of these ideas makes WebExtensions a
> hard fit, though Hybrid WebExtensions would be a good fit. Page Shot could
> be rewritten fairly easily to be 95% WebExtension. The last 5% could be
> fixed with probably-acceptable product changes, but we'd be left
> wondering... is there something we could do outside the boundaries of
> WebExtensions that would be an important and valuable product direction? The
> data-driven process we want to do requires us to actually ship those ideas.
> And then be ready to un-ship them as well, ideally without making the
> WebExtension team run around for our capricious and unfounded requests.

I'm thinking it would be good to make the main Test Pilot a
WebExtension Experiment [1]. When a new project comes into Test Pilot,
if the WebExtensions APIs don't exist then they get added to the main
Test Pilot add-on as WebExtension APIs [2].

In this case you could possibly write that remaining 5% as a
WebExtension API (or not, the choice is yours). If then Page Shot
graduates, we could graduate the API, or make whatever decision we'd
like at that time.

This way we get to use the same data pipeline and use that to guide the APIs.

I've been wanting to post more about WebExtension Experiments but
wanted to get the documentation and flow more in order, so I know
there's still some work to be done there.

[1] https://webextensions-experiments.readthedocs.io/en/latest/
[2] This would require us to allow Test Pilot as a WebExtension
Experiment to exist on release.

[Axel Hecht]

On 12/10/16 17:09, Alexandre poirot wrote:

1. XPCOM / XPIDL, for better or worse, offered a boundary and interface system. I think those interfaces were originally intended to be pretty stable. Down the line, I seem to recall us deciding that having frozen interfaces there was slowing us down, and so now those interfaces are changed at will. I understand that we were more constrained because more applications were relying on the underlying interfaces, but this all starts to feel kinda familiar. Are we at risk of creating a new set of interfaces and silos that get holes poked through them over time because they slow us down instead of speeding us up? Are we repeating ourselves? If so, what did we learn last time that will make this time better?
   

We do have some boundaries with XPCOM, some others with JSM, yet some others with message managers.

But that's just too many different kind of boundaries/interfaces. If we only had one border, everything would have been much better.

If you look, again, at session restore you will have to go through all of them and it is far from being obvious and will feel very hard to approach to anyone except mozilla employees who have to learn all these things. Web extension abstracts all that. The difference between a content script and a background page is very small.

I have to disagree on this.

XPCOM boundaries are per-instance boundaries, JSM is more-clearly-than-xpcom service boundaries, message managers are on-change notification boundaries.

Those things are just realities in software design, and need to be reflected.

I think it's an element of good software architecture that the difference between these three concepts manifests in the APIs in which you interact with them.

Axel

[Axel Hecht]

I think we need to have a bigger conversation about React.

IMHO, React makes a lot of statements that are very related to the statements that XUL made 15 years ago.

The core statement might be "HTML can't fly". Back then, mozilla's answer was XML. Facebook's answer today is javascript.

The impact on HTML of both is the same though, it constrains HTML from winning.

Axel

[Alexandre poirot]

2016-10-12 18:04 GMT+02:00 Gijs Kruitbosch <gijskru...@gmail.com>:

2016-10-12 8:26 GMT-07:00 Gijs Kruitbosch <gijskru...@gmail.com>:

Session restore?
http://blog.techno-barje.fr/post/2016/03/14/session-restore-web-extension/

The big advantage to this particular feature is that it doesn't involve much UI, which is where WebExtension is the most limited.

... but then, is avoiding the UI problem really going to be helpful in the future?

That's just being iterative. This experiment is here, available for anyone to test it, compare the performances, and see for real how that would look like.

We should also be iterative about integrating within browser.xul UI. The awesomebar for example could be a good candidate of a self contained UI. I have a prototype where I rip off browser.xul completely and implement everything out of web extensions (I described in my first reply in this thread). But that's just a showoff to help drawing a path between today and a fully addonified browser.
Speaking about mochitest-plain, I got them running in a completely new browser, in full HTML, on try. It wasn't green as some web features were obviously broken, but the test harness was running just fine.

Aligning ourselves with the web is a good goal, IMO, but seems problematic with WebExtensions. WebExtensions, its async-ness, and its lacking support for actual UI creation, and interfacing with related UI on a single main thread with multiple async components, is (to the best of my knowledge) not a solved problem**. Then there's all the stuff about exposed vs. internally used API surfaces. Using React for the internal implementation of the frontend components would be better-understood (even if I'm personally not a massive React fan...).

Using React is ortogonal to the extension proposal. You may use whatever web feature you prefer in it.
The benefit of building your app with addons is that you can replace each of them and use completely different web technologies. Whereas if you just do one big document implementing everyhing, like browser.xul, it won't be possible.

 

Those are all solvable problems, but there would need to be a commitment that people want to invest in this, and broader agreement on tools and direction.

I think some experiments would help taking any decision and help everyone understand how that would really look like.

I've been working on this very narrow topic since January as a side project and I do have a interesting feedback to share:
* I wasn't expecting to be able to reimplement most of Session restore features as a web extension. I'm missing some really small tweaks to chrome.tabs API.

* I was really surprised to be able to reimplement the core browser features (tabs, urlbar, content) by just using chrome.tabs API, as-is, with really dumb tweaks made to the official API, which I'm not sure have to be privileged/internal and could just be exposed to all addons!

[Sylvestre Ledru]

Le 06/10/2016 à 17:08, Benjamin Smedberg a écrit :
> I spent a week writing a thing about modularity, webextensions, and going faster. I think it's important for us to decide the module structure of our code especially as we start shipping independent modules/going faster. And I believe that having better module structure, boundaries, and
> documentation is critical to our teams being more agile and also attracting contributors to the project.
>
> http://benjamin.smedbergs.us/blog/2016-09-03/modularity-and-webextensions/
>
> I personally think that we should double down on WebExtensions as a model and start using that for large parts of Firefox. But Andy McKay and Rob Helper had some good counter-thoughts and I've asked them to post here to elaborate.
>
> In the post I asked everyone to send followups to firefox-dev, so I wanted to start a thread here to collect responses. Over the next months I'd like this to turn into a firm decision about how we're going to build system addons; but I'd like to start by seeing what feedback people have and even
> whether I've framed the problem correctly.
I like the idea of having more flexibility and modularity.
However, from my perspective, I don't have the feeling that moving faster is what our users are asking. I believe they are asking for stability and no regression. And, for now, I don't think we have been able to achieve that.
I am concerned that not having features riding the train might lead with poorer quality as we need time to have the feature backing properly and gather user feedback.

In parallel, we will need to streamline our workflows once/if we start using this capability to ship outside of the release cycle. The Go Faster is already adding a lot of complexity for release managers.

Sylvestre

[David Teller]

That's a good question. Benjamin, I remember that, a few years back,
you had user research numbers with "what users want". At the time, if my
memory serves, it was largely about video stability.

Do you have updated numbers?

Note that, as far as I can tell, re-thinking our APIs and API boundaries
is useful not just for Go Faster but for all front-end works.

Cheers,
David