Request for adding a new service.

[Ramin Halavati]

Dear Service Owners!

Sorry for sending this permission request late, I was not aware of the process.

I would like to add a new service for using a Google3 based image processing library to analyze Chrome screenshots and improve the accessibility tree. The library is downloaded via component updater for users who enable this feature.

Design Doc: go/chrome-screen-ai

Prototype CL: crrev.com/c/3358073

We have discussed the issue with the security team and concluded that we need a sandboxed process for it, in short because:

• The library in Google3 has a lot of dependencies and transferring the code to Chrome and compiling it with Chrome's hardened build requirements requires a huge maintenance cost.
• The input to the library is screenshots of the websites (partial or full screen) and so the websites can control the input. Therefore if issues exist in the library that can be exploited, websites can trigger it.

Please let me know if I need to add any extra information for the request.

Best,

Ramin

[Colin Blundell]

Hi Ramin,

Thanks for reaching out! The rationale on having this code run OOP makes sense to me.

Can you detail what the intended consumers of this service are? That can help us determine where it makes sense for it to live in the tree.

Best,

Colin

[Ramin Halavati]

Hi Colin,

The primary target is users who have enabled screen-readers or have low vision, but in future it can help other assistive technologies as well.

Here is a list of first use cases:

1. Converting scanned PDFs into texts that are readable/navigable by screen readers.
   
2. On ChromeOS (and also maybe on desktop), extracting the accessibility tree of renderers which don't provide it. Examples are Chrome Remote Desktop, Parallels on ChromeOS, STEAM, Linux Apps, Android apps.
3. Extracting document structure or heading levels from canvases.
4. Identifying the main content on a page.
5. Detecting precise visual focus bound on ChromeVox.
6.  Adding labels to page elements that are not [correctly] labeled by the web developers.

There is a more detailed explanation in this link.

Best,

Ramin

[Colin Blundell]

Thanks, Ramin! This is definitely useful info (and the feature sounds awesome!).

What I meant was actually the following: On a technical level, where do the intended consumers live in the codebase? e.g., if all the intended consumers are in //chrome, then it likely makes sense for the service to live in //chrome/services. If the service is an internal detail of //content, then it likely makes sense for it to live in //content/services. etc.

[Ramin Halavati]

Oh sorry, I got the question wrong!

I may be a little off on this (dtseng@ please correct me if I am wrong), but here are the first consumers that we have in mind:

• 1st milestone:
• Content Renderer (/content/renderer/accessibility)
• PDF( /pdf or maybe triggered from elsewhere)
• 2nd milestone:
  
• Chrome OS Ash Aura (/ui/views/accessibility)
• ARC (/chrome/browser/ash/arc)

[Colin Blundell]

Thanks again!

Is the model that you envision the following: For each use case for generating an accessibility tree (e.g., generating an accessibility tree for a PDF or for a webpage), the code that wants to generate the accessibility tree will communicate with the service and pass it the input in question, with the service doing its magic and returning the accessibility tree?

Or is it more like the following: Those use cases get an accessibility tree via some common API surface (that e.g. lives in //content), with the implementation of that API surface talking to the service to get the optimized-by-ML accessibility tree?

Or is it something else :)?

Best,

Colin

[Ramin Halavati]

 think it's closer to the first one, but as I have not started the design for other cases, it's hard to give a general answer.

What I expect is that in all cases the code that is in charge of generating the AX tree sends a snapshot of what the user is really seeing to the library, and gets some annotations that help in improving (or in some cases creating) the tree.

For the /content/renderer case, this call is from inside the content layer, but I think for the other use cases (PDF, Aura, and Arc), it will be from outside /content.

Did I answer your question?

[Colin Blundell]

Thanks, Ramin!

This was all very helpful. Based on your answers I don't think that it makes sense for the service to live in //content (e.g., for the PDF use case). It doesn't make sense for it to live in //chrome either as it's targeted for use from lower layers.

I would suggest that //components/services is a more appropriate location for this service than //services (see the text in //services/README.md about distinctions between the two). Does that make sense to you? Any other //services/OWNERS have opinions?

Best,

Colin

[Ramin Halavati]

Hi Colin,

Thank you very much, it sounds good to me. I will move it and get back to you if I find any problems.

Best,

Ramin

[Colin Blundell]

Ramin pointed out off-thread that since //content will depend on the ScreenAI service, putting the service in //components would require adding a dependency from //content to //components/services/screen_ai. Such dependencies are already allowed for shared code between //third_party/blink and //content. I think that shared code between //pdf and //content effectively falls into the same case, given my understanding that we're trying to remove the dependency of //pdf on //content. +John Abd-El-Malek Does that make sense?

[Ramin Halavati]

Thank you Colin, I will wait for John's reply, but I am not quite sure that I got your point. Do I understand correctly that you are suggesting that it's fine to move the ScreenAI service to components, and then add a dependency from content to components/services/screen_ai?

[Colin Blundell]

On Mon, Feb 21, 2022 at 4:30 PM Ramin Halavati <rhal...@google.com> wrote:

Thank you Colin, I will wait for John's reply, but I am not quite sure that I got your point. Do I understand correctly that you are suggesting that it's fine to move the ScreenAI service to components, and then add a dependency from content to components/services/screen_ai?

Yes: I think the fact that the code will be shared between //pdf and //content should make it OK to add the dependency from //content to //components/services/screen_ai.

[Ramin Halavati]

Thank you Colin, I updated the CL accordingly.

What do you think +John Abd-El-Malek ? (Do I need to start another permission request thread for adding a component dependency to contents as suggested in content/renderer/DEPS)?

Best,

Ramin

[John Abd-El-Malek]

Sorry for the delay, this all sounds good.

[Ramin Halavati]

Great! Thank you John and Colin.