Best point to detect regions of content area that are updated.

[Ramin Halavati]

Hi!

I am working on a proposal for an accessibility module that takes snapshots of the screen and annotates them through a local image processing module.

To do so, I need to have triggers that say "this region of the screen is updated". I've looked into several positions for the trigger (like ui::Window::SchedulePaintInRect oand cc::AsyncLayerTreeFrameSink::SubmitCompositorFrame) but could not find a place that gives me the content area rectangles that are updated. I would appreciate any pointers on where to look.

Thank you,

Ramin

[Khushal Sagar]

Are you looking for the precise damaged region (list of rectangles) or would a unioned rect suffice? Damage tracking in the compositing stack tracks the latter.

Also, how is the snapshot getting triggered? That'll help in pointing out the easiest way to get this information.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMgMY8wzFGBpai2sX1bE5rjyZtM2B-NKtiACzr0V3GDRn9h6WQ%40mail.gmail.com.

[Khushal Sagar]

On Mon, Jan 3, 2022 at 10:04 PM Ramin Halavati <rhal...@google.com> wrote:

I don't need the precise damaged region and unioned rect suffices.

I'm still considering possibilities for the triggering mechanism and have not found a good one yet. It would be great if you suggest options that I take a closer look at.

Which process is managing the image processing module? I'm assuming it's the browser so you'll need a mechanism for the browser process to be notified when the web content updates and a new snapshot is available.

By default the browser doesn't receive such notifications, frames go directly from the renderer to Viz/GPU and then the display. You'd need to attach a listener to a FrameSink whose frames you want to capture. The sink could be 1:1 with the web content for a particular tab, a browser window or the whole display (in the case of CrOS). The closest use-case to this is FrameSinkVideoCapturer. It's worth looking through the source code for how it's used.

cc-ing some GPU folks who might have additional pointers.

 

Thanks,

Ramin

[Ramin Halavati]

Thank you. The image processing module will be a service in a sandboxed process. So I think triggers from any other process can send images to it and the result can be sent through another mojo interface to the process that applies them.

I will look into FrameSink.

Best,

Ramin

[Ramin Halavati]

I don't need the precise damaged region and unioned rect suffices.

I'm still considering possibilities for the triggering mechanism and have not found a good one yet. It would be great if you suggest options that I take a closer look at.

Thanks,

Ramin

On Mon, Jan 3, 2022 at 5:00 PM Khushal Sagar <khusha...@chromium.org> wrote: