Intent to Experiment: CSS Houdini Paint API Level 1

[Jet Villegas]

Spec: https://drafts.css-houdini.org/css-paint-api/

Summary: The CSS Paint API is the first of several Web Rendering proposals
from the CSS Houdini Task Force. The CSS Paint API allows Web authors to
define and register a custom Paint method to be executed by the Layout
engine as other elements are rendered.

Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1302328

Link to standard: https://drafts.css-houdini.org/css-paint-api/

Platform coverage: Android, Desktop

Estimated or target release: TBD

Preference behind which this will be implemented: TBD

DevTools bug: TBD

Tests - TBD

Implementation Details:

CSS Paint API depends on the implementation of the following features:
CSS Houdini Properties & Values - https://bugzilla.mozilla.org/
show_bug.cgi?id=1273706
Houdini "Worklets" - https://bugzilla.mozilla.org/show_bug.cgi?id=1290021

The planned implementation in Gecko builds upon the HTML5 Canvas2D API to
provide the rendering surface for CSS Paint.

Risks:

The experimental implementation has progressed enough to warrant this
public Intent to Implement. However, significant risks will need to be
mitigated by careful design and execution if these features are to pass the
experimental stage and ship in Firefox, including:

1. Use Cases. It's not clear that the use cases proposed in the
specification warrant the additional Rendering System complexity. Apart
from conical gradients, we haven't seen many author requests for the other
use cases. If the existing Canvas2D feature set is lacking, what are the
compelling use cases and maximally useful API for such use cases? It's not
clear that the proposed Canvas2D-based API is desirable over a different
API design (eg., WebGL) if most use cases need to directly manipulate
pixels.

2. Rendering Performance. The planned Canvas2D backing store approach may
be too slow for real-world usage of the API. In the future, we may replace
the Canvas2D approach to have the custom paint methods create Layout
(displayList) nodes for direct rendering by the Layout engine, bypassing
the need for a Canvas2D backing store. It's worth noting that the Paint API
isn't directly compatible with existing displayList nodes (e.g., support
for raw paths, funny shapes, & pixel manipulation.)

There may also be other performance issues that arise with the API's usage
in combination with existing CSS features (e.g., CSS Masking, Filters,
etc.) The displayList vs. canvas bitmap implementation would probably look
a good bit different in WebRender. It's also worth noting that multiple
implementations shipping a bitmap-based version can create dependencies
that prevent us from switching to a faster alternative version in the
future.

3. Integration with Gecko Architecture. The Quantum Project <
https://wiki.mozilla.org/Quantum> is a major overhaul of the Firefox
Rendering Engine. Implementing the CSS Paint API while that effort is in
progress may add significant impedance. However, a counter-argument is that
we should design Quantum to allow for such extensibility in the future.
Duplication of work ( writing code that would need to be rewritten for
Quantum ) is not desirable and should be avoided.

For WebRender/Quantum, we could initially push this through the same path
that SVG will go through (which will be rasterized on the CPU and then
cached in a GPU texture atlas). It does seem like Houdini Paint could
reduce the amount of acceleration we can do on the GPU (at least in the
short term), but we won't be any worse off than other browsers in that
regard.

4. Dependency on incomplete implementations/specifications. The dependency
creates a chicken/egg scenario where we can't sufficiently evaluate the
dependent specifications (e.g, Worklets, and Properties & Values) without
also implementing an initial key use case (e.g., CSS Paint or Worklets for
Web Audio.) This is somewhat mitigated by getting the implementation far
enough along to formulate informed opinions on all the specifications.

The Houdini Task Force is meeting next week in Seattle to discuss this and
other specifications.

[Xidorn Quan]

Great news!

I guess that may also mean billm's bug 1308039 [1] which tried to do
painting during GC (which will be backed out soon [2] because of lots of
breakage) may never get chance to land anymore, since this API means it
is expected to run JS code during painting.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1308039
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1328423

- Xidorn

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[smaug]

On 01/05/2017 06:00 PM, Jet Villegas wrote:
> Spec: https://drafts.css-houdini.org/css-paint-api/
>
> Summary: The CSS Paint API is the first of several Web Rendering proposals
> from the CSS Houdini Task Force. The CSS Paint API allows Web authors to
> define and register a custom Paint method to be executed by the Layout
> engine as other elements are rendered.
>
> Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1302328
>
> Link to standard: https://drafts.css-houdini.org/css-paint-api/
>
> Platform coverage: Android, Desktop
>
> Estimated or target release: TBD
>
> Preference behind which this will be implemented: TBD
>
> DevTools bug: TBD
>
> Tests - TBD
>
> Implementation Details:
>
> CSS Paint API depends on the implementation of the following features:
> CSS Houdini Properties & Values - https://bugzilla.mozilla.org/
> show_bug.cgi?id=1273706

As you probably know, baku is working on to get support for proper worklets, not just
a prototype, supporting running in different threads and so (AudioWorklets really need that).
https://bugzilla.mozilla.org/show_bug.cgi?id=1328964

Couldn't we add some new types of worklets for different APIs?
Or, has it been discussed to have a method to access the worklet?
So, not window.paintWorklet but
window.getPaintWorklet("2d")
which could be extended later, similar to canvas context, to support
different types of paint worklets.

[Boris Zbarsky]

On 1/6/17 5:53 AM, Xidorn Quan wrote:
> I guess that may also mean billm's bug 1308039 [1] which tried to do
> painting during GC (which will be backed out soon [2] because of lots of
> breakage) may never get chance to land anymore, since this API means it
> is expected to run JS code during painting.

That should be OK, I think, as long as the paint worklets are on a
separate thread (and hence in a separate runtime) from the JS which is
doing GC when we decide to paint.

In-thread worklets would be a problem for paint-during-gc, though.

-Boris

[Bill McCloskey]

Are in-thread worklets a thing? The patch that allows tab switch painting
while content JS runs isn't going to be backed out. If the tab switch
paints allow JS to run, and that JS is from the same context as the content
JS that was interrupted, then that would be bad.

-Bill

[Boris Zbarsky]

On 1/6/17 2:34 PM, Bill McCloskey wrote:
> Are in-thread worklets a thing?

Unclear.

The spec is agnostic on where worklets run. I know Chrome was running
some of them in-thread (but not the details). For us, it's basically an
implementation decision.

> The patch that allows tab switch painting
> while content JS runs isn't going to be backed out. If the tab switch
> paints allow JS to run, and that JS is from the same context as the content
> JS that was interrupted, then that would be bad.

What does "same context" mean here? An in-thread worklet would be
same-JSContext/JSRuntime/whatever, but different-global and
shared-nothing (at least observably from JS) with any web pages.

Is the badness you refer to page-observable JS-level reentry, or
something deeper in our engine internals? Because the former is not an
issue with worklets, I expect.

-Boris

[Bill McCloskey]

On Fri, Jan 6, 2017 at 12:19 PM, Boris Zbarsky <bzba...@mit.edu> wrote:

> On 1/6/17 2:34 PM, Bill McCloskey wrote:The patch that allows tab switch

> painting
>
>> while content JS runs isn't going to be backed out. If the tab switch
>> paints allow JS to run, and that JS is from the same context as the
>> content
>> JS that was interrupted, then that would be bad.
>>
>
> What does "same context" mean here? An in-thread worklet would be
> same-JSContext/JSRuntime/whatever, but different-global and
> shared-nothing (at least observably from JS) with any web pages.
>
> Is the badness you refer to page-observable JS-level reentry, or something
> deeper in our engine internals? Because the former is not an issue with
> worklets, I expect.

The badness is just page-observable JS-level reentry (that I know of at
least). So that sounds good. Thanks.

-Bill

[Andrea Marchesini]

On Fri, Jan 6, 2017 at 9:19 PM, Boris Zbarsky <bzba...@mit.edu> wrote:

> On 1/6/17 2:34 PM, Bill McCloskey wrote:
>
>> Are in-thread worklets a thing?
>>
>
> Unclear.
>

Right. So far, I'm concentrating on having Worklet able to run out of
main-thread.
My goal is to have something ready for audioWorklet, then, I want to have
something able to run on any thread.
The code is not ready yet, but the main idea is to have worklet thread
agnostic.

[tombouc...@gmail.com]