Summarizer requires a user gesture

[Raymond Camden]

I'm testing a demo where I can drag/drop an electronic comic book into the web app and the prompt API is used to analyze the pictures inside. This works (well, mostly). For each image, I generate a summary string. 

I then wanted to use the summary API to summarize the summaries. 

let availability = await Summarizer.availability();
console.log('summarizer availability', availability);
let summarizer = await Summarizer.create({
format:'plain-text',
length:'long',
monitor(m) {
     m.addEventListener('downloadprogress', (e) => {
      console.log(`Downloading summarizer: ${e.loaded * 100}%`);
    });
  }
});

I get `downloadable` for status. But when create is run, I get:

Uncaught (in promise) NotAllowedError: Requires a user gesture when availability is "downloading" or "downloadable".

There was an earlier user gesture, afaik, earlier when I dropped the comic book onto the web page. As a test, I also interacted with the page while the initial process (analyze images) was working. What else can I do here?

[Roland Bouman]

I ran into similar situations. 

Unfortunately, if the Summarizer (with those options) isn't already in the status "available", then you won't be able to escape this.

If I understand correctly,

navigator.userActivation.isActive

must be true for the create call to succeed. 

So you could check explicitly for userActivation, but this will only help you predict whether the call to create will fail - I can't help directly not help to create a situation where it will succeed.

What should work, is check for user activation directly before the call to create(), and when you find isActive is not true, exit the current workflow and pop up a modal dialog to inform the user the action is taking a little longer.

The dialog should have a "Ok" button which would close the dialog, and then inside the click event handler for that button you could you try to create the Summarizer.

(Once you created the Summarizer, you can cache it somewhere for later, repeated use)

I think that in theory, it's possible that before the call to availability, the userActivation could be true, while it might not necessarily be true anymore when availability resolves, even if it returns the status available.

(This is based on my understanding that userActivation uses a timeout-mechanism)

[Raymond Camden]

Ugh, that sounds... crazy. :) I mean, I only start my process after a user click, so how could it _not_ be active?

I'm running into this in a second demo as well, with Rewriter. :\

--

===========================================================================
Raymond Camden

Email : raymon...@gmail.com
Blog : www.raymondcamden.com
Twitter: raymondcamden

[François Beaufort]

Thanks for reporting those issues!

Raymond, can you share your demo URL so that I can try to reproduce locally?
Which Chrome version did you use by the way?

--
You received this message because you are subscribed to the Google Groups "Chrome Built-in AI Early Preview Program Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ai-dev-previe...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ai-dev-preview-discuss/CAK2avF9wRjPNf39ER9ZQgqxUGWXAPhP3HsQ1_PpK1uGYE%3DFeKg%40mail.gmail.com.

[Raymond Camden]

I can share - but keep in mind this is a WIP and a bit ugly. Also, if you need a comic book file, send me a direct email. 

https://cfjedimaster.github.io/ai-testingzone/comic_web_ai/index.html

Use console of course. 

[François Beaufort]

I nuked my Chrome Canary user data dir to simulate a fresh install, enabled the Prompt API flag, dropped a cbz file into https://cfjedimaster.github.io/ai-testingzone/comic_web_ai/index.html and waited for the AI model to be downloaded. Note that I had to wait a bit to get from 90% to 100% as expected since it's uncompression time.
After that, I had to refresh the page, dropped the cbz file again, and it said "Starting work on AI Summary." Note that I had another error which doesn't seem to related:

zip.min.js:1 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'size')
    at new Mt (zip.min.js:1:21728)
    at getBlob (app.js:94:12)
    at doAISummary (app.js:193:34)
    at handleAISupport (app.js:180:9)

In other words, I could not reproduce the "NotAllowedError: Requires a user gesture when availability is "downloading" or "downloadable"."  error sadly. 

There's one issue I've noticed though in this code.
You checked the previously set "status" variable after "await LanguageModel.create()". You should check "await LanguageModel.availability()" again after since  "await LanguageModel.create()" will actually trigger model download if status is 'downloadable'.

[François Beaufort]

Something like that:

let status = await LanguageModel.availability();
if (status === "unavailable") {
  $aiSummary.innerHTML = "<p>AI support is unavailable, sorry.</p>";
  return;
}

session = await LanguageModel.create(...);

status = await LanguageModel.availability();
console.log("status", status);
if (status === "downloadable" || status == "downloading") {
  $aiSummary.innerHTML =
    "<p>AI support is enabled, but must be downloaded. Please stand by.</p>";
  return;
} else doAISummary(pages, reader, binreader);

[Raymond Camden]

Thanks - I noticed some issues with my download support (I'm using similar code in another demo) - will hack at it.