Getting "unsupported language" and "NotReadableError" errors from the model during summarization?

[Robert Oschler]

NOTE: I get these same errors from either my YouTube Summarizer Chrome extension contest or from the Summarization API Playground page.  Also, for the NotReadableError, I found this web page from July 2024 where the author had the same problem.  However, I do not believe in my case it is due to context window overage:

https://ainoya.dev/posts/web-distiller-ai-chrome-extension/

MAIN MESSAGE

I am getting errors frequently during summarization attempts of these two types:

1)  The model attempted to output text in an untested language, and was prevented from doing so.

Text:

bad woman does Kamala Inspire hope that Dems say nope Kamala Harris you remember her the vice president she's on the left she's losing support even among her strongest Democrat backers it's so bad George Santos is thinking of divorcing her as the New York Times points out our allies are tired of waiting for her to Define her vice presidency the New York Times calls it a painful reality that dozens of Dems including those who helped elect her don't see her as the future leader of their party much less the country you know you're dead in the water when even the times is against you that or you're on a date with Ted Kennedy how dare you laugh it's too soon I know even Hillary Clinton thinks Kamala doesn't have the political instincts to win and she murders people with a serpic wit and a knife made of happiness which is the name of her gun so I'd listen to her for sure where was I for more on all of this we've got a surprise special guest joining me live from Washington vice president Kamala Harris hello hello Greg according to the times your own advisors have referred reporters to democrats for quotes praising you and even they say they've lost hope in you what do you have to say about that well in order to Galvanize everything that we have already discussed and everything that we've already brought down to America okay I think we need to understand that everything that we have uncovered and unburdened we will continue to uncover and unburden even more going further you know Hillary has a history of supporting all liberal women running for office but now she's saying you're doomed what are your thoughts well you know when it comes to Hillary and you know I can make a list and send it to her on her private email but I think she's well aware of everything that we have tried to bring and try to undo more than what she has done and accomplished as her time as first lady kill me Hillary James kleiberg South Carolina rep who often backs people of color was asked if you should still be Biden's running mate and he'll said he'll leave that up to Joe what do you think look I mean look we've galvanized a lot with electricity and look at how much we've done with yellow school buses okay think about the electric school buses for all the children and they know that everything that we have unburdened we will soon burden even further despite the fact that we have yet to do anything positive for the country and bring it to its need do you uh last question were you jealous when you saw your husband make out with Dr Jill at the State of the Union oh great Joe's half dead let well thank you vice president Kamala Harris you're welcome back to America.

main.js:203 Uncaught (in promise) NotSupportedError: The model attempted to output text in an untested language, and was prevented from doing so.

2) NotReadableError: The execution yielded a bad response.

Text:

GREAT JOB AS ALWAYS I ALWAYS APPRECIATE IT. I ALWAYS APPRECIATE THIS NEXT GENTLEMAN COMING UP KEVIN O'LEARY SHARK TANK INVESTOR, SO MUCH MORE, WHAT IS YOUR GUT TELL YOU HOW TOMORROW GOES. >> BINARY GOING TO BE DECIDED STOP BY FOR THE MORNING LIKE EVERYBODY ELSE IS GUESSING I'M IN THE CAMP LUCIDLY GO TO ONE SIDE OR THE OTHER I DON'T TRUST PULLING ANYMORE I'VE EVEN LOOKED AT PRIVATE PULLING FROM THE PROBLEM I LEARNED AS AN INVESTOR INVESTIGATED SOME OF THE STUFF IS ACTUAL PANELS THAT ARE DONE EITHER ONLINE OR BY CELL PHONE ARE VERY SMALL. IF YOU ARE PULLING A COUNTY AND ONLY GETTING 1200 PEOPLE, YOUR MARGIN FOR AIR IS ONLY 4% AND NO CANDIDATE IS MORE THAN 2% AHEAD. IT IS USELESS DATA. MY GUT TELLS ME PEOPLE HAVE DECIDED YOU HAVE A DIFFERENT WAY TO LOOK AT THE TWO CANDIDATES IN TERMS OF THE DIRECTION THEY WANT TO TAKE THE COUNTRY ON SOMEBODY DIFFERENT ISSUES. I THINK YOU WILL FIND OUT TOMORROW BETWEEN ONE AND THREE IN THE MORNING IT'S ALL ONE SIDE OR THE OTHER AND I REALLY BELIEVE THAT TO BE THE CASE. POLLSTERS HAVE TO FIND A NEW CAREER, CALL ME AND I GIVE YOU A JOB SOMEWHERE. >> LET ME ASK YOU WHAT'S HAPPENING ON THE MARKETS THE DOW IS DOWN 360 SOME ODD POINTS THERE ARE A LOT OF FACTORS THAT PLAY IOWA TO GET INTO THE NITTY-GRITTY BUT WE SEE INTEREST RATES WHICH ARE BACKED UP PRETTY PROHIBITIVELY THAT'S THE HIGHEST 36% WE ARE UNDER 4. 3% ON THE TENURE I DON'T THAT'S TELLING YOU ANYTHING BUT IS NOT TELLING YOU ANYTHING? >> HERE'S WHAT IT'S TELLING US ALL NUMBER ONE IS VERY RARE TIMES WHEN THE FED IS OVER SHOWERED BY ANOTHER EVENT AND THAT'S WHAT'S HAPPENED HERE, THE ELECTION IS OVERSHADOWING EVERYTHING AND THERE'S A LOT OF UNCERTAINTY AND WE'VE BEEN TALKING ABOUT FOR DAYS MARKETS INCLUDED THE TREASURY MARKET DON'T LIKE UNCERTAINTY AND WE'VE GOT IT SO I THINK THINGS WILL SETTLE DOWN THE JOB NUMBER THAT JUST CAME OUT SOFTER THAN MANY ANTICIPATED INCLUDING ME AND A LOT OF BLOWING NOISE IN THEIR. AS PEOPLE THINK ABOUT THE FED GOING TO DO I'M NOT SURE THAT EVERYBODY'S IN THE 50 BASIS POINT CAMP, I THINK WERE LIKE 25 MAYBE OR MAYBE EVEN NOTHING FOR NOW WHILE THEY TRY TO FIGURE IT OUT. ALL OF THIS IS UNCERTAINTY, I BROUGHT A JET BOMBER T-BILLS AND TOOK ADVANTAGE AND SAID I LIKE T-BILLS DURING UNCERTAIN TIMES A LITTLE CASH OR RENO, 4. 62%, I WILL TAKE IT. LET ME GET A QUICK TAKE ON WHAT DONALD TRUMP HAD SAID IF HE LOSES THAT THERE WILL BE A DEPRESSION IN A MARKET CRASH, WHAT DO YOU THINK ABOUT. >> DONALD TRUMP BEING THE DONALD YOU GOT ABOUT ONE WAY OR ANOTHER BASED ON POLICY BECAUSE BOTH SIDES ARE BEING ABSOLUTELY CRAZY, THE THING THAT I TAKE SO LISTEN IF TRUMP WINS IF HE SURROUNDED HIMSELF IF THERE IS A TRUMP 2. 0, BETTER PEOPLE AROUND HIM. THE TRANSITION TEAM HEADED BY CANTOR FITZ-GERALD GUY I HAD A CHANCE TO SPEAK WITH HIM, HE IS REALLY FOCUSING ON AAA PLAYERS THAT ARE POLICY FOCUSED, THAT IS GREAT AND IF EVERYBODY TRUST HIM AS A BIPARTISAN GUY THE RHETORIC HAS BEEN INSANE THE NAZI HUNTING IN THE FIRING SQUAD YOU HAVE TO LET ALL OF THAT GO INTO SIDE WHICH BASKET POLICY DO YOU WANT HE WANTED TOP-DOWN ECONOMY WHERE HARRIS IS GOING TO SAY I'M GOING TO TAX EVERYBODY AT 28% IN THE PICK THE WINNERS LIKE THE CHIPS IN SCIENCE ACT OR ALL DECIDE WHO GETS THE MONEY OR DO YOU WANT A JUMP BALL WHERE THE CORPORATE TAX RATES ARE LOWER AT EVERY 11 SECTOR, THAT IS A TRUMP CAMP, LESS REGULATION AND THE REST EVERYBODY TO LAX, TRUMP IS NOT GOING TO STOP TALKING THE WAY HE TALKS HE'S BEEN DOING IT FOR NINE YEARS HE'S NEVER GOING TO CHANGE, GET OVER IT. NEIL: I THINK WHEN YOU'RE REFERRING TO HOWARD IT WAS HOWARD LUCK NICK OF CANTOR FITZ-GERALD. A GOOD GUY. THIS TIME HE IS THE RIGHT GUY. EVERYBODY LIKES THAT GUY AND EVERYBODY HE CALLS IS SAYING I WOULD LIKE TO BE A POLICY-BASED GUY HE IS HANDPICKING THESE PEOPLE. IT IS NOT THE CRAZY STUFF THAT HAPPENED LAST TIME. YOU COULD GET A BETTER TEAM IT IS GOING TO BE A LOT MORE THOUGHTFUL. IF YOU GO WITH TRUMP, TRUMP 2. 0 WILL NOT BE.

main.js:203 Uncaught (in promise) NotReadableError: The execution yielded a bad response.

[Thomas Steiner]

Hi Robert,

For the first issue, given your input text, this looks unexpected. Could you please file a new Chromium bug (https://new.crbug.com) and send the URL here?

For the second, issue, It might be related to an overflow actually. Does it work with shorter text? If so, can you paste your long text in the input field of https://chrome.dev/web-ai-demos/prompt-api-playground/ and share the number of tokens? (If you enter text, the token count is updated in real time.)

Thanks,

Tom

--
You received this message because you are subscribed to the Google Groups "Chrome Built-in AI Early Preview Program Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ai-dev-previe...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ai-dev-preview-discuss/2916b7a9-2b3a-4ba1-8ba8-320883b2e4den%40chromium.org.

[Robert Oschler]

Hello Thomas,

1) Here is the bug report:

https://issues.chromium.org/issues/377366940

2) The input text according to the Prompt API Playground is only 1052 tokens in length, so I don't think that's it?  It does not work with shorter text.  I've done a fair amount of testing because the length of YouTube video transcripts can be quite long and I am currently working on chunkifying text into 5000 word or less chunks since that should be within the ~ 7300 token limit the Summarization API Playground warns about.  But in any case, the example text for the error is as I said, only 1052 tokens.

My hunch is that the LLM doesn't like text with odd punctuation or all caps, but that's a pretty weak hunch.  Tomorrow, when I get up, I'll run the same text through some code I have the uses an LLM to properly repunctuate text and see if I still get the error.  Note, I don't typically get this error on normal document text like that taken from a properly written web page, etc.

[Robert Oschler]

Also, see the attached screenshot of the Prompt API Playground.  You can see the tokens and at the bottom, you can see the "The execution yielded a bad response" error.

On Tuesday, November 5, 2024 at 1:57:52 AM UTC-5 Thomas Steiner wrote:

[Thomas Steiner]

On Tue, Nov 5, 2024 at 8:18 AM Robert Oschler <robert....@gmail.com> wrote:

Hello Thomas,

1) Here is the bug report:

https://issues.chromium.org/issues/377366940

Thank you, triaged this to the right component, so it shows up for the engineers.

 

2) The input text according to the Prompt API Playground is only 1052 tokens in length, so I don't think that's it?  It does not work with shorter text.  I've done a fair amount of testing because the length of YouTube video transcripts can be quite long and I am currently working on chunkifying text into 5000 word or less chunks since that should be within the ~ 7300 token limit the Summarization API Playground warns about.  But in any case, the example text for the error is as I said, only 1052 tokens.

So internally the Summarization API calls the Prompt API with a special internal system prompt. At the moment, there is a per prompt limit of 1024 tokens, which means your text alone already hits the limit. I suggest breaking down the text further. 

--

Thomas Steiner, PhD—Developer Relations Engineer (blog.tomayac.com, toot.cafe/@tomayac)

Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany
Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891

----- BEGIN PGP SIGNATURE -----

Version: GnuPG v2.4.3 (GNU/Linux)

iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck

0fjumBl3DCharaCTersAttH3b0ttom.xKcd.cOm/1181.

----- END PGP SIGNATURE -----

[Robert Oschler]

Hi Thomas,

I am still getting the error and the text below is 977 tokens according to the Prompt API playground.  Note, every now and then the exact same call works and I get a summary.  But the error happens more often than not:

"GREAT JOB AS ALWAYS I ALWAYS APPRECIATE IT. I ALWAYS APPRECIATE THIS NEXT GENTLEMAN COMING UP KEVIN O'LEARY SHARK TANK INVESTOR, SO MUCH MORE, WHAT IS YOUR GUT TELL YOU HOW TOMORROW GOES. >> BINARY GOING TO BE DECIDED STOP BY FOR THE MORNING LIKE EVERYBODY ELSE IS GUESSING I'M IN THE CAMP LUCIDLY GO TO ONE SIDE OR THE OTHER I DON'T TRUST PULLING ANYMORE I'VE EVEN LOOKED AT PRIVATE PULLING FROM THE PROBLEM I LEARNED AS AN INVESTOR INVESTIGATED SOME OF THE STUFF IS ACTUAL PANELS THAT ARE DONE EITHER ONLINE OR BY CELL PHONE ARE VERY SMALL. IF YOU ARE PULLING A COUNTY AND ONLY GETTING 1200 PEOPLE, YOUR MARGIN FOR AIR IS ONLY 4% AND NO CANDIDATE IS MORE THAN 2% AHEAD. IT IS USELESS DATA. MY GUT TELLS ME PEOPLE HAVE DECIDED YOU HAVE A DIFFERENT WAY TO LOOK AT THE TWO CANDIDATES IN TERMS OF THE DIRECTION THEY WANT TO TAKE THE COUNTRY ON SOMEBODY DIFFERENT ISSUES. I THINK YOU WILL FIND OUT TOMORROW BETWEEN ONE AND THREE IN THE MORNING IT'S ALL ONE SIDE OR THE OTHER AND I REALLY BELIEVE THAT TO BE THE CASE. POLLSTERS HAVE TO FIND A NEW CAREER, CALL ME AND I GIVE YOU A JOB SOMEWHERE. >> LET ME ASK YOU WHAT'S HAPPENING ON THE MARKETS THE DOW IS DOWN 360 SOME ODD POINTS THERE ARE A LOT OF FACTORS THAT PLAY IOWA TO GET INTO THE NITTY-GRITTY BUT WE SEE INTEREST RATES WHICH ARE BACKED UP PRETTY PROHIBITIVELY THAT'S THE HIGHEST 36% WE ARE UNDER 4. 3% ON THE TENURE I DON'T THAT'S TELLING YOU ANYTHING BUT IS NOT TELLING YOU ANYTHING? >> HERE'S WHAT IT'S TELLING US ALL NUMBER ONE IS VERY RARE TIMES WHEN THE FED IS OVER SHOWERED BY ANOTHER EVENT AND THAT'S WHAT'S HAPPENED HERE, THE ELECTION IS OVERSHADOWING EVERYTHING AND THERE'S A LOT OF UNCERTAINTY AND WE'VE BEEN TALKING ABOUT FOR DAYS MARKETS INCLUDED THE TREASURY MARKET DON'T LIKE UNCERTAINTY AND WE'VE GOT IT SO I THINK THINGS WILL SETTLE DOWN THE JOB NUMBER THAT JUST CAME OUT SOFTER THAN MANY ANTICIPATED INCLUDING ME AND A LOT OF BLOWING NOISE IN THEIR. AS PEOPLE THINK ABOUT THE FED GOING TO DO I'M NOT SURE THAT EVERYBODY'S IN THE 50 BASIS POINT CAMP, I THINK WERE LIKE 25 MAYBE OR MAYBE EVEN NOTHING FOR NOW WHILE THEY TRY TO FIGURE IT OUT. ALL OF THIS IS UNCERTAINTY, I BROUGHT A JET BOMBER T-BILLS AND TOOK ADVANTAGE AND SAID I LIKE T-BILLS DURING UNCERTAIN TIMES A LITTLE CASH OR RENO, 4. 62%, I WILL TAKE IT. LET ME GET A QUICK TAKE ON WHAT DONALD TRUMP HAD SAID IF HE LOSES THAT THERE WILL BE A DEPRESSION IN A MARKET CRASH, WHAT DO YOU THINK ABOUT. >> DONALD TRUMP BEING THE DONALD YOU GOT ABOUT ONE WAY OR ANOTHER BASED ON POLICY BECAUSE BOTH SIDES ARE BEING ABSOLUTELY CRAZY, THE THING THAT I TAKE SO LISTEN IF TRUMP WINS IF HE SURROUNDED HIMSELF IF THERE IS A TRUMP 2. 0, BETTER PEOPLE AROUND HIM. THE TRANSITION TEAM HEADED BY CANTOR FITZ-GERALD GUY I HAD A CHANCE TO SPEAK WITH HIM, HE IS REALLY FOCUSING ON AAA PLAYERS THAT ARE POLICY FOCUSED, THAT IS GREAT AND IF EVERYBODY TRUST HIM AS A BIPARTISAN GUY THE RHETORIC HAS BEEN INSANE THE NAZI HUNTING IN THE FIRING SQUAD YOU HAVE TO LET ALL OF THAT GO INTO SIDE WHICH BASKET POLICY DO YOU WANT HE WANTED TOP-DOWN ECONOMY WHERE HARRIS IS GOING TO SAY I'M GOING TO TAX EVERYBODY AT 28% IN THE PICK THE WINNERS LIKE THE CHIPS IN SCIENCE ACT OR ALL DECIDE WHO GETS THE MONEY OR DO YOU WANT A JUMP BALL WHERE THE CORPORATE TAX RATES ARE LOWER AT EVERY 11 SECTOR, THAT IS A TRUMP CAMP, LESS REGULATION AND THE REST EVERYBODY TO LAX, TRUMP IS NOT GOING TO STOP TALKING THE WAY HE TALKS HE'S BEEN DOING IT FOR NINE YEARS HE'S NEVER GOING TO CHANGE, GET OVER IT. NEIL: I THINK WHEN YOU'RE REFERRING TO HOWARD IT WAS HOWARD LUCK NICK OF CANTOR FITZ-GERALD. A GOOD GUY. THIS TIME HE IS THE RIGHT GUY."

[Thomas Steiner]

I this case, please also file another bug and paste the link here. Please include the text, so there's a chance the problem reproduces for the engineering team once they try. Thanks for your help!

Cheers,

Tom

[Robert Oschler]

Hello Thomas,

Here is the new bug report:

https://issues.chromium.org/issues/377531190

re: component category

I searched carefully the component category drop-down box for the category I saw in your edit to my first bug report and did not find it.  Oddly enough, I just went back to that first bug report and the component category seems to have regressed to "I don't know", even though a day or so ago, I did see that "summarize API" specific category you switched it to.  Der Teufel hat den Schnaps gemacht.

https://issues.chromium.org/issues/377366940

[Thomas Steiner]

Thanks, Robert, Chromium > Blink > AI > Summarization is perfect.