Question on getting pixels from blink/renderer/modules/accessibility.

[Ramin Halavati]

Hi,

I am trying to find a way to get pixels for any general node in third_party/blink/renderer/modules/accessibility. We already have AXNodeObject::ImageDataUrl function to return the bitmap of a certain node, but it is only supporting HTML element types that already include a picture (namely image, canvas, video) and I am trying to expand to to any general element type.

So far I've been looking for a way to get access to a painter or compositor, but I did not find any promising solution yet. Can you point me in the right direction? Is it even expected at this abstraction level?

Thanks,

Ramin

P.S., Sorry for naive questions and please point me to the right email thread if this is not.

[Philip Rogers]

This is a deceptively difficult problem in general. Our paint system uses the CSS definition of paint order which starts painting at stacking contexts and it is challenging to exclude non-stacked siblings of a non-stacked target element. Some other complications are whether ancestor effects/clips should be applied, and whether content behind the target should be included.

DataTransfer::NodeImage is an attempt at this. Another option is to get the entire page painted (examples: devtools uses PageHandler::CaptureScreenshot and printing uses SpoolSinglePage), then crop it to the target element.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQi_LSUFXWxpfAKQKKVhRzK1UvGVz%2B4unQUN2qskFSSafw%40mail.gmail.com.

[Ramin Halavati]

Thank you Philip.

Since this accessibility support function is called for all renderer types (in ChromeOS including Arc++ and Ash), there would be a better consistency if we have the snapshotting code in the renderer. Therefore I am trying to avoid using the browser side ui::GrabViewSnapshot (as in DevTools), unless the alternative approach proves to be too complicated or unnecessarily heavy.

I tried using DataTransfer::NodeImage in this draft CL, but my understanding is that since it triggers a repaint, we should make sure it is not in the middle of a lifecycle update to avoid conflicts. Is there a notification of a signal that tells when it's safe to request it? I assume it's the same for the SpoolSinglePage as it also redraws the image. And performance wise, can we say that as this approach triggers a repaint, it's heavier than the browser based snapshotting (using ui::GrabViewSnapshot), as the latter is using the already painted output and does not require a repaint?

Best,

Ramin

[Philip Rogers]

On Mon, Jan 10, 2022 at 6:51 AM Ramin Halavati <rhal...@google.com> wrote:

Thank you Philip.

Since this accessibility support function is called for all renderer types (in ChromeOS including Arc++ and Ash), there would be a better consistency if we have the snapshotting code in the renderer. Therefore I am trying to avoid using the browser side ui::GrabViewSnapshot (as in DevTools), unless the alternative approach proves to be too complicated or unnecessarily heavy.

I tried using DataTransfer::NodeImage in this draft CL, but my understanding is that since it triggers a repaint, we should make sure it is not in the middle of a lifecycle update to avoid conflicts. Is there a notification of a signal that tells when it's safe to request it?

It is important that we do not introduce lifecycle violations but I think that may already be guaranteed from when the accessibility code runs. For example, AxObject::GetRelativeBounds cannot be called from within a lifecycle update either. You can probably just synchronously call the paint function you need.

 

I assume it's the same for the SpoolSinglePage as it also redraws the image. And performance wise, can we say that as this approach triggers a repaint, it's heavier than the browser based snapshotting (using ui::GrabViewSnapshot), as the latter is using the already painted output and does not require a repaint?

The paint code in NodeImage is really just a walk of layout datastructures and isn't more expensive than things like layout, style, etc (i.e., it's O(|layout|)). Doing this for each object would be wildly inefficient though.

Expanding on the thing about this being an ambiguous problem:

data:text/html,<body style="background: rebeccapurple; filter: blur(1px)"><div style="height: 100px; width: 100px; border: 1px solid white; color:white;" draggable=true>hello world</div>

The drag image for the div will only contain white and transparent pixels, and will not be blurred. If you want the purple background too, you'll want to do the full-page-paint+crop approach.

[Ramin Halavati]

Thank you very much Philip,

About the concern that a repaint based approach (similar to NodeImage) will not have the backgrounds or other higher level effects, can we say that on the other hand (specially since this function is explicitly querying the image for a certain node) the full-page-paint+crop approach also suffers from the possibility that the node maybe be partially covered by another node, or being out of the screen's visible area (needing scroll), or even being covered by a dialog box?

[Philip Rogers]

On Tue, Jan 11, 2022 at 7:42 AM Ramin Halavati <rhal...@google.com> wrote:

Thank you very much Philip,

About the concern that a repaint based approach (similar to NodeImage) will not have the backgrounds or other higher level effects, can we say that on the other hand (specially since this function is explicitly querying the image for a certain node) the full-page-paint+crop approach also suffers from the possibility that the node maybe be partially covered by another node, or being out of the screen's visible area (needing scroll), or even being covered by a dialog box?

That's correct: the full-page-crop approach will include content on top of the target content, will not include the target if it is scrolled offscreen, etc.

[Ramin Halavati]

Thank you very much Philip. I will write a 1-pager on this discussion for posteriority and will proceed with the inside blink approach.

Just as the concluding question, I have a draft CL on it (crrev.com/c/3370381). Would you suggest that I take a (possibly simplified) copy of the DataTransfer::NodeImage in third_party/blink/renderer/modules/accessibility, or refactor the existing code in DataTransfer to fit the existing and the new use case. I prefer the latter as it is a very delicate and complex process. If you agree with that, do you have any suggestions on the refactoring or I would just go with it and ask you to take a look after?

Best,

Ramin

[Philip Rogers]

On Wed, Jan 12, 2022 at 8:06 AM Ramin Halavati <rhal...@google.com> wrote:

Thank you very much Philip. I will write a 1-pager on this discussion for posteriority and will proceed with the inside blink approach.

Just as the concluding question, I have a draft CL on it (crrev.com/c/3370381). Would you suggest that I take a (possibly simplified) copy of the DataTransfer::NodeImage in third_party/blink/renderer/modules/accessibility, or refactor the existing code in DataTransfer to fit the existing and the new use case. I prefer the latter as it is a very delicate and complex process. If you agree with that, do you have any suggestions on the refactoring or I would just go with it and ask you to take a look after?

If the accessibility and drag-image code are unlikely to diverge too much, I think it makes sense to factor out the code that creates a PaintArtifact for a Node out of DraggedNodeImageBuilder and into a helper. This helper could include a comment about how the painting starts from the containing stacking context.