Showcase: Zero-Cloud BSL In-Vision Interpreter driven by Gemini Nano & S-Expression AST Vectors

[joseph brewerton]

Hi everyone,

I wanted to share a real-world implementation of Chrome's Built-in AI (window.ai.languageModel / Gemini Nano) that moves beyond conversational chatbots or text summarisation, using the model instead as a local symbolic compiler and real-time physical choreographer for an open-source educational runtime.

(Note: This is an active work-in-progress / beta build—I'm currently cleaning up a few minor edge-case cartridge loading quirks in the dev pipeline, but the core engine, on-device AI integration, and live rendering are fully operational.)

The Context: Replacing 25 MB Simulations and Heavy Video Streams

Modern STEM digital education faces two massive architectural bottlenecks in real-world classrooms:

1. Bloated Simulation Runtimes: Legacy tools (PhET, HTML5 canvas suites) bundle 10–25 MB of monolithic JavaScript per simulation, imposing 1.5–3 second cold starts and 40–90 MB RAM footprints on student devices.

2. The "Video Tax" on Accessibility: Providing British Sign Language (BSL) traditionally requires streaming 50+ MB MP4 video overlays per lesson, which chokes bandwidth-constrained school networks and creates prohibitive hosting costs.

The Architecture: A Decoupled <35 KB Vector Engine Driven by AST S-Expressions

To solve this, we decoupled the runtime into a lightweight, zero-dependency browser-native stack executing at 60 FPS in standard SVG DOM:

1. The Headless AST Vector Player Core (<35 KB gzipped):

   Replacing heavy canvas or WebGL engines, simulations (like Bohr atomic electron shells, Boyle's law gas pistons, and Keplerian orbits) are codified as declarative Lisp-like S-Expressions (.ast) paired with pure SVG markup. It achieves a sub-15 ms cold start and runs within an <8 MB RAM ceiling.

2. Gemini Nano as an On-Stage Co-Pilot (window.ai):

   Rather than outputting conversational prose, Chrome's Prompt API acts as a real-time copilot. When a student manipulates a slider or makes an error, Nano evaluates telemetry snapshots and emits structured, token-dense action tuples (e.g., (:act :seek 0.50 :highlight "#nucleus" :say "...")) to scrub the simulation timeline, pulse SVG elements, and guide the student visually.

3. Autonomous "In-Vision" BSL Vector Interpreter:

   The player synchronises speech boundaries and subtitles directly into an on-stage 3D BSL interpreter. Nano decomposes English grammar into BSL spatial syntax, which is solved at 60 FPS via Hermite splines and WASM to articulate the two-handed BANZSL manual alphabet and curriculum signs. Each sign requires just ~178 bytes instead of 10 MB video clips.

4. Zero-Egress Statutory Safeguarding:

   Because all neural inference (Gemini Nano) and vector maths execute 100% on-device with zero cloud data egress, it complies by design with UK GDPR (Article 25) and Keeping Children Safe in Education (KCSIE) mandates—completely eliminating data privacy friction for schools.

Technical Takeaways for the Built-in AI Team:

• High Efficiency via Declarative S-Expressions: Instructing Nano to output strict, domain-specific S-Expressions rather than verbose JSON cut token usage drastically and eliminated JSON schema hallucinations.

• Separation of Reasoning vs Mechanical Execution: Offloading mathematical interpolation (splines, perspective matrices) to WASM/JS while using Nano purely for semantic intent kept performance locked at 60 FPS on low-end hardware.

• W3C Document PiP Integration: Running both the AST Player stage and the BSL In-Vision overlay with native documentPictureInPicture allows students to pop out lessons over smartboard software or secondary monitors seamlessly.

You can see the engine running live (Bohr atomic shells simulation with synchronized real-time 3D BSL interpretation) here:

[https://josephbrewerton-oss.github.io/wwwebucation/simulator?preset=atom](https://josephbrewerton-oss.github.io/wwwebucation/simulator?preset=atom)[cite: 2, 4]

Would love to hear the Chromium team's thoughts on providing native GGML/GBNF grammar constraints for the Prompt API to enforce rigid S-Expression output formatting directly during sampling.

Best regards,

Joseph Brewerton

[Kenji Baheux]

Hi Joseph,

Thanks for sharing this, including making the code and live player available to explore. 

Pairing a lightweight, deterministic SVG/Wasm simulation engine with the Prompt API acting as a bounded UI choreographer (:seek, :highlight, :var, :say) is a smart architectural split for compact on-device models.

On your question about native GGML/GBNF grammar constraints for S-Expressions, here is where things stand and where we would love your hands-on feedback:

• RegExp Constraints (Available to test today): Many developers only use responseConstraint with JSON Schema, but the Prompt API also accepts a standard JavaScript RegExp object (alongside omitResponseConstraintInput: true if you want to keep the constraint from eating into your context window). In theory, your single-level Co-Pilot action tuples ((:act :seek ... :highlight ... :say ...)) look regular enough that a RegExp constraint could enforce the syntax directly during sampling. That said, we haven’t seen many folks stress-test RegExp. If you give it a spin, I’d be very curious whether it holds up cleanly for your :act commands or if you run into edge cases.
• JSON Schema Overhead vs. Upcoming Runtime Optimizations: Your point about S-Expressions cutting token bloat compared to verbose JSON makes a lot of sense on small models. We are actively working on speculative decoding optimizations (as well as native tool-calling support) that should help reduce the latency tax of structured boilerplate tokens. We’ll share testing details with the EPP group soon so you can benchmark whether the gap narrows in practice.
• Native GBNF / Context-Free Grammars: For deeply nested AST scenes (where RegExp is inadequate), a full context-free grammar like GBNF is the only way to get strict non-JSON guarantees. While local inference runtimes compile constraints into grammar masks under the hood anyway, our main hesitation at the web platform level is cross-browser standardization complexity and guarding against pathological recursive grammars hanging the browser process. Real-world data on where RegExp + JSON Schema fall short for use cases like yours would be super helpful as we weigh that trade-off in the W3C group.

(One quick housekeeping note: I noticed aicaller.ts already checks window.LanguageModel first, which is great—just a heads-up to update window.ai.languageModel in your docs/posts when you get a chance, as the legacy window.ai namespace is retired.)

If you end up testing responseConstraint with a RegExp on the :act tuples, please let us know how the latency and reliability compare!

Best, Kenji

--
You received this message because you are subscribed to the Google Groups "Chrome Built-in AI Early Preview Program Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ai-dev-previe...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ai-dev-preview-discuss/eebd5258-389e-4563-b2ea-f5d61e6c2804n%40chromium.org.

[joseph brewerton]

Subject: Empirical Prompt API Feedback: responseConstraint (RegExp) Ignored in session.prompt()

Hi Kenji,

We conducted an empirical test using Chrome’s on-device LanguageModel interface to test structured output constraints for choreographing educational physics simulations (using S-expression action tuples).

Here is the exact test run and observed behavior:

• Target Constraint: Flat tuple regex:

  JavaScript

  const NANO_ACT_TUPLE_REGEX = /^\(:act\s+:(seek\s+[0-9.]+|var\s+"[a-zA-Z0-9_-]+"\s+[0-9.]+|highlight\s+"#[a-zA-Z0-9_-]+"|say\s+"[^"]*")(\s+:(seek\s+[0-9.]+|var\s+"[a-zA-Z0-9_-]+"\s+[0-9.]+|highlight\s+"#[a-zA-Z0-9_-]+"|say\s+"[^"]*"))*\)$/;

• Invocation:

  JavaScript

  const session = await LanguageModel.create({ systemPrompt: "You are an on-device simulation choreographer. Emit only an S-expression (:act ...) tuple.", expectedOutputLanguages: ["en"] }); const result = await session.prompt("Compress the cylinder to show Boyle's Law.", { responseConstraint: NANO_ACT_TUPLE_REGEX, omitResponseConstraintInput: true });

• Observed Output:

  Plaintext

  "...has to exert more force to maintain the same moles when the volume shrinks. **Important Note:** Since I can't physically *draw* the figure here, I rely on descriptions/visual metaphors. To truly grasp Boyle's Law, it's helpful to use a simple diagram..."

• Result: RegExp Match Verified: false (no SyntaxError DOMException was thrown).

Key Takeaways for W3C / Chrome Prompt API:

1. RegExp Masking Silently Bypassed: Passing a RegExp via responseConstraint in session.prompt() failed to enforce logit masking during decoding. Rather than constraining the tokens or rejecting the promise with a SyntaxError DOMException, the model defaulted to conversational prose.

2. Compute & Token Waste on Low-Spec Hardware: Because the constraint was bypassed, the on-device model generated hundreds of unnecessary conversational tokens. On low-spec consumer laptops, this resulted in repeated [Violation] 'message' handler took ~200ms+ main thread stalls, when a strict parser/mask should have terminated generation within 20 tokens.

3. The Case for CFG / GBNF: If even flat Regular Expression logit masking is unreliable or easily bypassed in current builds, structured hierarchical grammars (like AST S-expressions or complex JSON schemas) will require native context-free grammar (CFG/GBNF) enforcement in the underlying inference engine.

"P.S. I use Gemini to help structure and draft my technical write-ups due to autism, but the test harness, repo, and console output above are from real code I'm running locally on my machine."

Regards

Joe

[Kenji Baheux]

Hi Joseph,

Thanks for sharing this empirical test and the zero-cloud BSL In-Vision Interpreter showcase.

You are correct that passing a RegExp to responseConstraint in session.prompt() is supported by the API bindings. Silently bypassing the constraint without throwing an error is a bug on our end.

We’ve filed a Chromium issue to track and triage this: https://issues.chromium.org/issues/576432037

As a tiny side detail in case it’s helpful: in LanguageModel.create(), system instructions should be passed via initialPrompts: [{ role: 'system', content: '...' }] rather than systemPrompt.

Best, Kenji

---

How was this email? 😵‍💫 Unclear · 🧱 Too dense · 👍 Clear · 🤩 Awesome · 💬 Something else

[joseph brewerton]

Hi Kenji,

Thanks for confirming the issue and for the tip on initialPrompts.

I’ve updated the implementation to use initialPrompts: [{ role: 'system', ... }] and added client-side validation to catch the bypass until the Chromium fix lands. Appreciate the swift triage on Issue 576432037.

Best,

Joseph

[joseph brewerton]

Hi Kenji and EPP team,

Following up on our Prompt API testbed: while the Chromium team works on the responseConstraint RegExp triage (Issue 576432037), we’ve reached a major milestone with our symbolic interpreter that pushes Gemini Nano's spatial reasoning significantly further.

We’ve completed the upgrade of our In-Vision Interpreter from Level 1 fingerspelling/SSE to Signature BSL Level 2 / Level 3 Natural Grammar & 3D Spatial Loci:

1. Linguistic & Spatial Reordering via Nano:

   • Timeline Primacy (Time-First): Temporal markers are extracted and post-positioned at index 0 to anchor the chronological timeline.

   • Topic-Comment (OSV) Syntax: Sentence structures are mapped to authentic visual grammar while suppressing redundant English copula/particles (is, are, was, the, an).

   • 3D Spatial Loci Anchors: Referents are anchored to discrete 3D spatial coordinates (1P, 2P, Locus A, Locus B), enabling directional agreement verbs (GIVE, TELL, ASK, SHOW, HELP, EXPLAIN) to modulate their 3D spline sweep trajectories dynamically based on sender and recipient loci.

   • Syntactic Non-Manual Features (NMF): Facial grammar is synchronized directly to clause structure (eyebrows: 'furrowed' on WH-questions post-positioned to clause-end, affirmative/polar eyebrows, and headTilt: 'shake' on negations).

2. Hardened Vector Performance & Invariants:

   • Payload: The full expanded 100-sign 3D multi-keyframe lexicon compresses to 16.08 KB gzipped (static/lexicons/bsl-signs.ast), averaging 164 bytes per sign.

   • Kinematics: Hermite spline trajectory calculations execute at 0.43 µs per operation on the CPU, locking performance at 60 FPS in DOM-native SVG without WebGL dependencies or garbage collection pauses.

   • Verification: The engine passes 394 / 394 automated deterministic invariant tests.

3. Live ⚖️ Syntax & Spatial Loci Inspector: We added a live syntax drawer to BslInterpreterOverlay.tsx (accessible via the ⚖️ button in the player header). It visually displays Nano’s sentence classification, live decomposed BSL gloss sequence badges ([TIME] ➔ [TOPIC] ➔ [SUBJ(A)] ➔ [DIR-VERB(A➔B)] ➔ [OBJ(B)]), active NMF states, and a real-time Spatial Loci Compass showing entity coordinate mappings.

The live build with the new Level 2/3 inspector is up at:

https://josephbrewerton-oss.github.io/wwwebucation/simulator?preset=atom

Whenever a Canary build or flag is ready for Issue 576432037, we have our test harness primed to benchmark RegExp constraints against these structured spatial S-Expression tuples.

Best regards,

Joseph Brewerton