What does screen_ai do?

[guest271314]

Chromium includes what appears to be some kind of extension, screen_ai. 

There's a file libchromescreenai.so in the folder that is 288 MB alone.

It is not clear at all what the extension does.

Here's the manifest.json

{
  "manifest_version": 2,
  "name": "Chrome Screen AI",
  "version": "122.1"
}

What does screen_ai do?

[Ramin Halavati]

It's a library that provides some AI tools, including main content extraction for reading mode and OCR for PDF accessibility.

[guest271314]

Thanks! 

Some questions:

- Why isn't that description in the manifest.json?

- Why and how is "AI" used for the tasks described?

TBH I ran > /home/user/.config/chromium/screen_ai/122.1/libchromescreenai.so to truncate the file to 0 size because I had no idea what "screen_ai" was supposed to be doing.

I was rather surprised to find a 288 MB shared object file re a screen_ai extension when Chrome still sends remote network requests for Google voices for SpeechSynthesisUtterance and for all STT for webkitSpeechRecognition for Web Speech API. I would think TTS and STT would be shipped in the browser before including "AI" code in the developer release.

[Ramin Halavati]

I will look into how I can update the manifest.json to include a description of the services that this component gives.

The size is a bit large on Linux and it's much smaller on other platforms, that also needs looking into.

Both tasks are entirely done on device (no network access).

For main content extraction, an AI module tries to find the main part of the page and present it in a sidebar (Reading Mode). That can be used by users who have attention or reading difficulties or for any other reason want to focus on the main text.

For OCR, if a PDF doesn't have accessible text for screen reader users, the images inside the PDF are sent to this module and the extracted text is injected into the PDF.

Ramin Halavati

Senior Software Engineer

rhal...@google.com
+49 173 2614796

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

     

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[guest271314]

I don't understand how and why this leapfrogged Web Speech API implementation on-device. 

Web Speech API webkitSpeechSynthesisRecognition() is broken on Chromium the last time I checked. E.g., see https://github.com/mdn/dom-examples/issues/219 which consistently logs network error on Chromium 122. Because there is no in-browser STT implementation. And if I remember correctly I had to get Google API kets from net-log or built-in extension page in order to manually request the remote Google SST services.

When Google voices on Chrome are used a remote request is made to Google servers. I think PATTS is for "Googlers" only.

I just don't get how "A.I." can be shipped before TTS/STT in the browser. I could cynically just cast the decision off as "A.I." being the trending thingamajig to sell, no matter whatever else is left on the back-burner for years.

It would be very helpful, given the idea is for accessibility, to implement TTS and STT in the browser, using WebAssembly, or "A.I" if you prefer.

[Slade Watkins]

On Jan 18, 2024, at 5:21 AM, 'Ramin Halavati' via Chromium-discuss <chromium...@chromium.org> wrote:

The size is a bit large on Linux and it's much smaller on other platforms, that also needs looking into.

Okay, out of curiosity: how much larger?? I assume that’s due to needing to build more into the Linux variant(s)?

confused,

Slade

[Ramin Halavati]

It's around 289MB for Linux and 50MB for Mac.

The OCR code includes many modules for training and performance optimization. These modules are dropped from ChromeOS, Mac and Windows build, but probably not as good for Linux and need more work.

Best,

Ramin

[Ramin Halavati]

Hi,

No, unfortunately there isn't any public documentation on OCR models and performance comparison.

Best,

Ramin

On Fri, Feb 27, 2026 at 12:20 AM sp1d3r <aspy...@gmail.com> wrote:

Is there any documentation around the architecture of the OCR models?

I just want to benchmark it against other OCRs out of curiosity.

Thanks

[sp1d3r]

Is there any documentation around the architecture of the OCR models?

I just want to benchmark it against other OCRs out of curiosity.

Thanks

On Tuesday, January 23, 2024 at 2:58:29 PM UTC+5:30 Ramin Halavati wrote:

[guest271314]

Here's the contents of "files_list_ocr.txt" from "screen_ai" version 140.19.

gocr_mobile_chrome_multiscript_2024_q4_engine.binarypb
aksara/aksara_page_layout_analysis_rpn_gro_2024_q4.binarypb
aksara/aksara_page_layout_analysis_ti_rpn_gro_2024_q4.binarypb
gocr/gocr_models/gocr_line_recognition_omni_mobile_chrome_multiscript_2024_q4.binarypb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/arab.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/arab_fst_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/arab_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/arab_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/arab_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/arab_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/bede.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/bede_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/bede_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/beng_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/beng_deva_gujr_guru.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/beng_deva_gujr_guru_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/cyrl.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/cyrl_fst_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/cyrl_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/cyrl_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/cyrl_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/cyrl_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/deva_fst_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/deva_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/deva_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/geor.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/geor_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/geor_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/gocr_mobile_und.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/gocr_mobile_und_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/gocr_mobile_und_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/grek.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/grek_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/grek_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/gujr.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/gujr_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/gujr_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/guru_2024_q3_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hani_fst_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hani_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hani_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hanijpan.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hanijpan_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hanijpan_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hebr.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hebr_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/hebr_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/jpan_fst_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/jpan_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/jpan_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/khmr_2024_q3_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/khmr_laoo_thai_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/khmr_laoo_thai.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/khmr_laoo_thai_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/knda_fst_2024_q3_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/knda_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/knda_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/knda_sinh_telu.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/knda_sinh_telu_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/knda_sinh_telu_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/kore.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/kore_fst_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/kore_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/kore_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/kore_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/kore_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/laoo_2024_q3_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/mlym.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/mlym_fst_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/mlym_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/mlym_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/mlym_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/mlym_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/sinh_2024_q3_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/taml.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/taml_fst_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/taml_label_map.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/taml_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/taml_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/taml_prior.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/telu_fst_2024_q3_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/telu_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/telu_lm.syms
gocr/gocr_models/line_recognition_mobile_convnext320_omni/tflite_langid.tflite
gocr/gocr_models/line_recognition_mobile_convnext320_omni/thai_fst_2024_q3_config.pb
gocr/gocr_models/line_recognition_mobile_convnext320_omni/thai_lm.fst
gocr/gocr_models/line_recognition_mobile_convnext320_omni/thai_lm.syms
gocr/gocr_models/detection/gocr_group_rpn_text_detection_config_2024_q4_chrome.binarypb
gocr/gocr_models/detection/gocr_group_rpn_text_detection_model_2024_q4.tflite
gocr/layout/cluster_sort/model_v2.tflite

Here's the contents of README.md

 # Chrome Screen AI Library

## Purpose
Chrome Screen AI library provides two on-device functionalities for Chrome and
ChromeOS:
* **Main Content Extraction:** Intelligently isolates the main content of a web
   page, improving its readability by stripping distracting elements (based on
   the accessibility tree).
* **Optical Character Recognition:** Extracts text from image.
These functionalities are entirely on device and do not send any data to
network or store on disk. 

Please see https://source.chromium.org/chromium/chromium/src/+/main:services/screen_ai/README.md

 Here's the contents of "screen2x_config.pbtxt"

# proto-file: knowledge/cerebra/sense/im2query/screenai/screen2x/lib/screen2x.proto
# proto-message: screenai.screen2x.Screen2xOptions

model_path: "knowledge/cerebra/sense/im2query/screenai/screen2x/lib/data/chrome_a11y/v011/model.tflite"
model_config {
  mpnn_config {
    signature: "serving_default"
    senders_node_name: "input_1_senders"
    receivers_node_name: "input_1_receivers"
    nodes_node_name: "input_1_nodes"
    edges_node_name: "input_1_edges"
    globals_node_name: "input_1_globals"
    n_node_node_name: "input_1_n_node"
    n_edge_node_name: "input_1_n_edge"
    numeric_features_node_name: "input_2_numeric"
    chrome_role_node_name: "input_2_chrome_role"
    chrome_role_description_node_name: "input_2_chrome_role_description"
    output_node_name: "output_1"
    inference_config {
      num_threads: 1
      class_thresholds {
        label: CONTENT_NON_ESSENTIAL
        prediction_threshold: 0.65
      }
      class_thresholds {
        label: CONTENT_TEXT_HEADLINE
        prediction_threshold: 0.0
      }
      class_thresholds {
        label: CONTENT_TEXT_MAIN
        prediction_threshold: 0.0
      }
    }
    model_type: CHROME_A11Y
  }
}
post_process_options {
  chrome_post_process: true
}

 There's a couple folders, "aksara" and "gocr" which appear to contain the "models".

WHen Chromium is loaded the "screen_ai" folder eventually gets generated/written to ~/.config/chromium on Linux.

I usually just delete the folder, along with deleting "Google Safe Browsing" folder. I saved the folder once. Thus what I can quote above.

--
--
Chromium Discussion mailing list: chromium...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-discuss

To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/chromium-discuss/S3RBdCHzUI8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chromium-discu...@chromium.org.