Prompt API: Bulk Inference Recipes

[Nitin Pasumarthy]

Hi team,

I'm new to the API. And I'm super excited to try it out and possibly put up a prod proposal for use on linkedin.com

What are the best practices to use the prompt API for bulk inference? I have use case pipeline where I want to process 100 text blocks in parallel and then show the results to the user. 

Also what am I missing with tool use? I ran the https://github.com/webmachinelearning/prompt-api/blob/main/README.md#tool-use example in console and get
```tool_code
print(google_search.search(queries=["weather in Seattle"]))
```

instead of the final result after calling the API.

[Kenji Baheux]

Hi Nitin,

Individual calls to the prompt  / promptStreaming are queued as most devices are not capable of handling parallel LLM workloads.

An alternative would be to engineer the system prompt and user input so that it can process multiple inputs in one call, as long as everything fits within the context window.

As for function calling, it's not yet supported in the API and the current Nano model hasn't been specifically trained for that use case. 

That said, there is a version of the built-in AI provider for Vercel AI SDK that has a polyfilled support for function calling with the current Gemini Nano.

Anecdotal feedback suggests that it's viable enough for prototyping / exploring ideas. 

In case that's useful please check my modified version of Ilya's AgentBoard where I take advantage of this built-in AI provider.

Best,

--
You received this message because you are subscribed to the Google Groups "Chrome Built-in AI Early Preview Program Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ai-dev-previe...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ai-dev-preview-discuss/4d14cbcd-c2bd-4e47-a45e-4258d5977094n%40chromium.org.

--

Kenji BAHEUX (my how-to)

Product Manager - Chrome

Google Japan