Handling execution dependencies

[David Rajchenbach-Teller]

Dear everybody,

As Firefox moves each day more to an asynchronous execution model, we
keep facing issues related to dependencies between tasks. Here are a few
examples:
1. component A can only be initialized one or more other components have
reached some state;
2. off main thread I/O should only be shutdown once all components that
can use it have reached a state in which they don't need it anymore;
3. private browsing should be displayed only once the state of the
session has been (asynchronously) saved.

In our codebase, most of the synchronization is done through the
observer service. While this service has proved very useful, it is also
quite limited. For instance, using this service to implement a
dependency for scenario 2. above is rather tricky.

So, perhaps it is time to start working on a library (or at least a
design guideline) that will simplify our work refactoring Firefox into
asynchronicity.

As a base for discussion, I have put together a small RFC based on
promises: https://wiki.mozilla.org/RFC/TaskDependencies

Cheers,
David

--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla

[Ehsan Akhgari]

Can you please add code snippets on that page to show how this API can
be used to solve some example problems? It would really help if those
problems are the sort of things that actually come up in the Mozilla
code base. I have a really hard time inferring how the API is supposed
to be used by just reading that page (and I have to admit that I don't
know a lot about the concept of promises.)

Cheers,
Ehsan

[Gregory Szorc]

I love the nsIDependencyService API. Gated waiting on multiple async
events with timeouts is a *very* useful tool to have. I wish we had that
API in the tree! As it is, that API is more generic than a "dependency
service." But that's bikeshedding over naming.

While that API is generally useful, it only solves the "service
dependency" problem for services which are *always* initialized. If you
look at a more feature complete service initialization system such as
Upstart [1] or Service Management Facility [2], you'll find the current
proposal falls short in a number of areas, notably optional services.

If I were writing Gecko from scratch today and took the world view that
Gecko is an operating system, I would certainly be tempted to design
XPCOM/Gecko services as Upstart/SMF-like services. Each service would
list dependencies, runlevels, etc. Initializing a Gecko application
would involve bootstrapping into the service controller (the init/pid-1
process in *NIX speak) where you would request the start of a single
high-level "main" application/service (e.g. the "desktop-browser"
service for Firefox). The service controller would create a dependency
tree of all the required services (network, Places, etc) and proceed to
initialize them automagically.

Now, such a solution is probably not feasible in the short term. But, I
believe we would like to borrow some of its features.

Optional services come to mind. We currently have a number of features
that behave like services but aren't. Sync is definitely one. Addons
Manager /might/ be another. Their initialization is primarily controlled
by performing a Cu.import() where the side-effect is some
singleton/service is instantiated. This results in ugly hackiness like
lazy module getters and importers. This often results in inadvertent
misuse of a module by loading it too early. This results in memory bloat.

What would be really awesome is to capture service dependency metadata
and to declare services as optional. If nothing is relying on a service,
don't load it until it is needed! Sure, you'll probably want to pre-load
commonly used services as an optimization- nothing says you can't. The
real benefit is the ability to unload unused services, resulting in
memory recovery. Think Android and iOS and how they kill previously-used
applications during memory pressure. Don't need Addons Manager around
after installing that new add-on? Don't worry, it will be unloaded
automatically!

Would desktop Firefox benefit from such an advanced service management
system? I'm not sure. The real beneficiaries are mobile, especially
Firefox OS. Who knows, perhaps they invented something already!

I guess what I'm trying to say is: what problem are you trying to solve?
If it's for services as they exist in Gecko today, I think what you have
is a good start. But if you are looking to shed constraints from
decisions made 10+ years ago, you could certainly dream bigger.

[1] http://upstart.ubuntu.com/
[2] https://en.wikipedia.org/wiki/Service_Management_Facility

[Robert O'Callahan]

On Tue, Jan 15, 2013 at 2:26 PM, Gregory Szorc <g...@mozilla.com> wrote:

> If I were writing Gecko from scratch today and took the world view that
> Gecko is an operating system, I would certainly be tempted to design
> XPCOM/Gecko services as Upstart/SMF-like services.
>

If I were, I would identify all the XPCOM services that we're sure to need
almost immediately (which is most of them, for Gecko core), and just
hardcode their initialization and dispense with XPCOM service machinery
altogether.

The system you describe may be great, but you really should limit it to the
optional services only. For everything else it's just overhead. Like the
XPCOM overhead that we're gradually getting rid of.

Rob
--
Jesus called them together and said, “You know that the rulers of the
Gentiles lord it over them, and their high officials exercise authority
over them. Not so with you. Instead, whoever wants to become great among
you must be your servant, and whoever wants to be first must be your
slave — just
as the Son of Man did not come to be served, but to serve, and to give his
life as a ransom for many.” [Matthew 20:25-28]

[Anthony Jones]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David,

On 13/01/13 04:46, David Rajchenbach-Teller wrote:
> As a base for discussion, I have put together a small RFC based on
> promises: https://wiki.mozilla.org/RFC/TaskDependencies

You seem to be on the right track. You've probably already worked out
that you'll need to write some dependency code to make sure you
execute the then() and run() calls in the right order. There will also
be some threading considerations to deal with.

Reactive Programming could be viewed as a more general case of what
you're intending to do. You could consider RP to be an enhanced
version of the observer pattern that deals with more system wide
issues and therefore scales better. If you consider your proposal in
terms of RP then you would say:

* A Promise is a Behavior of boolean. It starts as false and becomes
true when something happens, which could be a timer. Behaviors can
contain arbitrary types.
* A Promise can be composed of other promises using boolean operators.
Behaviors can be composed with arbitrary functions
* A Barrier is an Event that fires when a promise changes value.
Events can be triggered when a Behavior changes value and may either
be used to create other Events, or they can have ordinary listeners.

You could trivially write your promise API in terms of Sodium[1].

https://github.com/kentuckyfriedtakahe/sodium

Anthony


[1] The C++ version of Sodium uses C++11 closure syntax so it requires
gcc >= 4.5 to build.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJQ9benAAoJECKjmZddR/QLtaIIAMtKA67+WIHD3wFTVO0dcVaC
qBrLq+I25TUjn8KsZO7NJ1m6nZcuhOU8JgkX2hDyK03vtoBnGefXqH1RM5IE41Wl
iBnulV9OvdTIhN0SRGmaI3hR+VrRnXGVi41DCFPjMXiLmnNdmdpHF9P/2t/sMZX2
Ve/Ti/YfUZgUqprS8dut4zEnT8wZ8OUKNSHYW6NKWwT9a8ubpesNSn4+72ifJyJa
ZeJbM33JMOFzp1Y56MVl2f6Zn7RcreJhE7Bm4tQ+U037/1oEvD68Qh3vAKksHF9X
vfQnbjcUfMdkPSLoxOCH2eNVReOnxf8p666hIikOBMO4/QigLcAmOKDxQZn8VU4=
=v9am
-----END PGP SIGNATURE-----