[Chrome ATL approval] Inclusion of a new third party llguidance Rust library

[Frank Li (IE)]

Hi Chrome ATL approvers,

 

Per @Clark Duvall’s pointer, I am sending this email to start the approval process of including a new third party llguidance Rust library into Chromium.

 

• Motivation of your project

Support for guidance/structured output with prompt API · Issue #35 · webmachinelearning/prompt-api

 

• Design docs
• Guidance/Structured output with Prompt API - Google Docs
• Blink side API for structured outputs - Google Docs
• Additional checkout size

Source size is ~575KB from the checkout of third_party\rust\chromium_crates_io\vendor\llguidance-0.6.27

 

• Build time increase
• This refers to building chrome or test targets in the critical development path.
• If the increase is significant (e.g., 30+ seconds), can we consider making this an optional build target?

TODO(FrankLi): I will find out from build log.

 

• Binary size increase on Android (official builds)

It is not build for Android because of enable_ml_internal build flag.

 

• Binary size increase on Windows

1,858,560 Byes (~18.5MB) on Windows x64 release build (unofficial). Official build should be smaller in binary size.

 

• Is this library maintained on all platforms that we will use it on?

Yes.

 

• Does it have any performance / memory implications (esp. on Android)? Was the library designed with intended use on Android?
• enable_ml_internal build flag does not build for Android.
• Per the design document above: “…In our testing, for a set of 10k JSON schemas with instances totaling about 2M tokens, we have found the initial pre-computation time to be over 3 seconds in 10% of cases, and over 120s in 0.1% of cases. While mask computation is often very fast (a few us), 1% of masks take longer than 100ms to compute and 0.1% take longer than 2s, with some hitting minutes…”.
• Per GitHub - guidance-ai/llguidance: Super-fast Structured Outputs, “…This library implements constrained decoding (also called constrained sampling or structured outputs) for Large Langauge Models (LLMs). It can enforce arbitrary context-free grammar on the output of LLM and is fast - on the order of 50μs of CPU time per token (for 128k tokenizer) with negligible startup costs….”
  
  
• Do we really need the library? Is there any alternative such as an existing library already in Chromium? If introducing a library with similar functionality as existing, will it be easy for another developer to understand which should be used where? Will you commit to consolidating uses in Chromium and remove the alternative libraries?

There are a few other libraries that provide constrained sampling (aka constrained decoding) apart from llguidance, based on conversations with researchers who work on llguidance we believe this library provides the best in terms of coverage, performance, and support.

 

• For desktop (Win/Mac/Linux/ChromeOS), does the dependency introduce closed source components (e.g., binaries, WASM binaries, obfuscated code)? If yes, please reach out to Chrome ATLs.

No.

 

[References]
- [ChromeML] Import of llguidance Rust crate (6272323) · Gerrit Code Review
- [ChromeML] use of llguidance to support structured outputs from LLM (6232561) · Gerrit Code Review

Best regards,

Frank

 

[Michal Moskal]

Some quick notes:

• Should be "1,858,560 Byes (~1.85MB)" not 18.5MB. 
• Also "In our testing, for a set of 10k JSON" from the design document refers to xgrammar not llguidance. You can see performance results for llguidance and other libraries here https://github.com/guidance-ai/jsonschemabench/tree/main/maskbench

---

From: Frank Li (IE) <fra...@microsoft.com>
Sent: Friday, February 28, 2025 5:14 PM
To: chrome-at...@google.com <chrome-at...@google.com>; Clark Duvall <cdu...@google.com>
Cc: Sushanth Rajasankar <Sushanth....@microsoft.com>; Michal Moskal <Michal...@microsoft.com>; rust...@chromium.org <rust...@chromium.org>
Subject: [Chrome ATL approval] Inclusion of a new third party llguidance Rust library

 

[Frank Li (IE)]

Thanks @Michal Moskal for the corrections! I have also corrected inline below.

 

Best regards,

Frank

1,858,560 Byes (~1.85MB) on Windows x64 release build (unofficial). Official build should be smaller in binary size.

 

• Is this library maintained on all platforms that we will use it on?

Yes.

 

• Does it have any performance / memory implications (esp. on Android)? Was the library designed with intended use on Android?

• enable_ml_internal build flag does not build for Android.

• performance results for llguidance and other libraries: jsonschemabench/maskbench at main · guidance-ai/jsonschemabench · GitHub

• Per GitHub - guidance-ai/llguidance: Super-fast Structured Outputs, “…This library implements constrained decoding (also called constrained sampling or structured outputs) for Large Langauge Models (LLMs). It can enforce arbitrary context-free grammar on the output of LLM and is fast - on the order of 50μs of CPU time per token (for 128k tokenizer) with negligible startup costs….”

[Sushanth Rajasankar]

Gentle ping. + Rick Byers.

Hi All,

We are working towards contributing to Chromium - JSON output support to the chromium built in AI prompt APIs.

To this end, we landed our proposed API modifications to Google's explainer Add structured outputs by sushraja-msft · Pull Request #78 · webmachinelearning/prompt-api.

Some context, for the library we would like to use to bring structured outputs to Chromium.

LLGuidance is refactoring of guidance-ai/guidance: A guidance language for controlling large language models, which extracts the core parts of guidance into a smaller rust library and now is a dependency of guidance. 

The parent library guidance and llguidance are open source projects, with cross company contributors to guidance. Guidance in particular has top contributors from Google DeepMind researchers slundberg, marcotcr apart from researchers from Microsoft mmoskal, Harsha-Nori.

Llguidance has been integrated into other inferencing frameworks such as llama.ccp , ort-genai.

Looking forward to hearing your thoughts or concerns about this project.

Thanks,

Sushanth

---

From: Frank Li (IE) <fra...@microsoft.com>

Sent: Friday, February 28, 2025 5:31 PM
To: Michal Moskal <Michal...@microsoft.com>; chrome-at...@google.com <chrome-at...@google.com>; Clark Duvall <cdu...@google.com>
Cc: Sushanth Rajasankar <Sushanth....@microsoft.com>; rust...@chromium.org <rust...@chromium.org>
Subject: RE: [Chrome ATL approval] Inclusion of a new third party llguidance Rust library

 

[Rick Byers]

Hi Sushanth,

Sorry for the delay! Somehow we didn't get the original message from Frank on Feb 28 (this may have been when we were struggling with spam filtering on this group), but we did get yours on Mar 3 and have just been slow to review.

This all sounds reasonable from a third_party perspective, so LGTM for ATLs. You will still need a security and license review, and of course any approval from OWNERS of the code in which you are integrating. 

Thanks,

   Rick

[Frank Li (IE)]

>… so LGTM for ATLs. … need a security and license review, and of course any approval from OWNERS of the code…
Thank you all for looking into this. @Clark Duvall has also guided us on the next steps to follow.

 

Best regards,

Frank

 

From: Rick Byers <rby...@chromium.org>
Sent: Monday, March 10, 2025 7:18 AM
To: Sushanth Rajasankar <Sushanth....@microsoft.com>
Cc: Frank Li (IE) <fra...@microsoft.com>; Michal Moskal <Michal...@microsoft.com>; chrome-at...@google.com; Clark Duvall <cdu...@google.com>; rust...@chromium.org; Etienne Noël <etien...@google.com>; tha...@chromium.org; Erik Anderson <Erik.A...@microsoft.com>; Matthew Riley <mat...@google.com>
Subject: [EXTERNAL] Re: [Chrome ATL approval] Inclusion of a new third party llguidance Rust library

 

You don't often get email from rby...@chromium.org. Learn why this is important

[Sushanth Rajasankar]

Thank you, Rick, Chrome ATLs. Looking forward to the next discussions.

---

From: Frank Li (IE) <fra...@microsoft.com>

Sent: Monday, March 10, 2025 9:10 AM
To: Rick Byers <rby...@chromium.org>; Sushanth Rajasankar <Sushanth....@microsoft.com>; Clark Duvall <cdu...@google.com>
Cc: Michal Moskal <Michal...@microsoft.com>; chrome-at...@google.com <chrome-at...@google.com>; rust...@chromium.org <rust...@chromium.org>; Etienne Noël <etien...@google.com>; tha...@chromium.org <tha...@chromium.org>; Erik Anderson <Erik.A...@microsoft.com>; Matthew Riley <mat...@google.com>
Subject: RE: [EXTERNAL] Re: [Chrome ATL approval] Inclusion of a new third party llguidance Rust library