| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
enum LanguageModelMessageRole { "system", "user", "assistant" };Maybe I misremembered, but I thought we wanted to add the tool roles to match huggingface API, which uses "assistant" for tool-call, and "tool" for tool-response? (I don't have a strong opinion on this, since ultimately it's up to the model specific impl to convert them into text prompt)
Just a clarifying qq: Without the tool roles, we will always require tool response content in "user" role, and tool call content in "assistant" role, correct?
interface LanguageModelToolSuccess {
[CallWith=ExecutionContext, RaisesException] constructor(LanguageModelToolSuccessInit init);
readonly attribute DOMString callID;
readonly attribute DOMString name;
readonly attribute FrozenArray<LanguageModelToolResultContent> result;
};
dictionary LanguageModelToolSuccessInit {
required DOMString callID;
required DOMString name;
required sequence<LanguageModelToolResultContent> result;
};
// Failed tool execution result.
[
Exposed(Window AIPromptAPI, Worker AIPromptAPIForWorkers),
RuntimeEnabled=AIPromptAPI,
SecureContext
]
interface LanguageModelToolError {
[CallWith=ExecutionContext, RaisesException] constructor(LanguageModelToolErrorInit init);
readonly attribute DOMString callID;
readonly attribute DOMString name;
readonly attribute DOMString errorMessage;
};Thank you for discovering and fixing the IDL union issue!
Making ToolCall an interface LG. For ToolResponse, have we considered using a single type to hold the state? e.g:
```
dictionary LanguageModelToolResponse {
required DOMString callID;
required DOMString name;
LanguageModelToolResultStatus status = "success";
// Populated if status is "success". If populated when Status is "error" then this field is ignored.
sequence<LanguageModelToolResultContent> result;
// Populated if status is "error". If populated when Status is "success" then this field is ignored.
DOMString errorMessage;
};
enum LanguageModelToolResultStatus { "success", "error" };
```
Rationale:
Similar examples in chromium:
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
enum LanguageModelMessageRole { "system", "user", "assistant" };Maybe I misremembered, but I thought we wanted to add the tool roles to match huggingface API, which uses "assistant" for tool-call, and "tool" for tool-response? (I don't have a strong opinion on this, since ultimately it's up to the model specific impl to convert them into text prompt)
Just a clarifying qq: Without the tool roles, we will always require tool response content in "user" role, and tool call content in "assistant" role, correct?
Mike raised the concern (see above link) that existing roles should be sufficient.
>... require tool response content in "user" role, and tool call content in "assistant" role, correct?
I think so.
I feel like separate them is clearer/concise and type safe (enforced by binding generated code).
Since we had them separated in the last edit, I thought there was no concern on this.
@Mike/@Jingyun, please let me know if we need to change.
| Code-Review | +1 |
// This header exists to satisfy the bindings generator which expectsWe could split the interfaces into separate idl files, but that shouldn't block this CL.
[CallWith=ExecutionContext, RaisesException] constructor(LanguageModelToolCallInit init);Is the context needed? It doesn't seem to be used?
One more bit of precedent is OpenAI's Responses API taking an arbitrary string for function responses, which may be "JSON, error codes, plain text, etc.": https://platform.openai.com/docs/guides/function-calling#incorporating-results-into-response
Relatedly, that API suggests: "If your function has no return value (e.g. send_email), simply return a string that indicates success or failure. (e.g. "success")".
My natural inclination is also to choose simplicity of a single type.
Even so, it's fine to proceed with them separated if you have a strong preference for that design.
EXPECT_TRUE(args_value->IsObject());optional nit: check the value content if it's easy enough
EXPECT_TRUE(args_value.IsEmpty() || args_value->IsNull());nit: Could this more specifically check just `IsNull`?
EXPECT_EQ(tool_success->result()[0]->type().AsString(), "text");Is it easy to check the value here too?
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
// This header exists to satisfy the bindings generator which expectsWe could split the interfaces into separate idl files, but that shouldn't block this CL.
Acknowledged
[CallWith=ExecutionContext, RaisesException] constructor(LanguageModelToolCallInit init);Is the context needed? It doesn't seem to be used?
Good catch!
removed.
interface LanguageModelToolSuccess {
[CallWith=ExecutionContext, RaisesException] constructor(LanguageModelToolSuccessInit init);
readonly attribute DOMString callID;
readonly attribute DOMString name;
readonly attribute FrozenArray<LanguageModelToolResultContent> result;
};
dictionary LanguageModelToolSuccessInit {
required DOMString callID;
required DOMString name;
required sequence<LanguageModelToolResultContent> result;
};
// Failed tool execution result.
[
Exposed(Window AIPromptAPI, Worker AIPromptAPIForWorkers),
RuntimeEnabled=AIPromptAPI,
SecureContext
]
interface LanguageModelToolError {
[CallWith=ExecutionContext, RaisesException] constructor(LanguageModelToolErrorInit init);
readonly attribute DOMString callID;
readonly attribute DOMString name;
readonly attribute DOMString errorMessage;
};I am going to stick with explicit types for success and error. Maybe you will change mind (or not) when I post the complete blink layer CL.
- in renderer process:
`
LanguageModelPromptBuilder::ProcessEntry() {
...
// Use binding generated code:
// - to check if it is 'success' or 'error'
// - to get the interfaces
if (content_value->IsLanguageModelToolSuccess()) {
tool_success = content_value->GetAsLanguageModelToolSuccess();
...
} else if (content_value->IsLanguageModelToolError()) {
tool_error = content_value->GetAsLanguageModelToolError();
...
}
...
}
`
- When we convert it to mojom for browser process to consume internally,
it will be just a single field type of `mojo_base.mojom.Value tool_response;` in
AILanguageModelPromptContent as you would like to see.
optional nit: check the value content if it's easy enough
Done
EXPECT_TRUE(args_value.IsEmpty() || args_value->IsNull());nit: Could this more specifically check just `IsNull`?
Done
EXPECT_EQ(tool_success->result()[0]->type().AsString(), "text");Is it easy to check the value here too?
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Thank you for writing out the blink conversion code... The difference is probably smaller than I thought. Let's proceed with your design!
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Thanks @Jingyun for the CR and discussions.
I am trying to satisfy the "Review-Enforcement" for this CL.
It worked last time with @Mike's +1 but not this time.
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
+@Steven to help me to get pass the "Review-Enforcement" due to re-auth issue in Microsoft workflow.
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
| Code-Review | +1 |
+@Steven to help me to get pass the "Review-Enforcement" due to re-auth issue in Microsoft workflow.
Is the ReAuth issue something we can raise with Chromium's infra team?
Cc:
+ Evan Stade (evan...@microsoft.com) who had indicated a conversation with Google infra wrt re-auth. I think the plan is for Microsoft contributors to continue to use LUCI_BYPASS_REAUTH=1
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
[PromptAPI] Revise tool types for `Tool use` IDL
- Convert LanguageModelToolCall, LanguageModelToolSuccess, and
LanguageModelToolError from dictionaries to ScriptWrappable
interfaces to enable proper union type discrimination in the Prompt API.
- Remove `tool-call` and `tool-response` from LanguageModelMessageRole
per previous comment that `user` and `assistant` roles are sufficient.
LanguageModelMessageType enum values of `tool-call` and `tool-response`
can be used to represent the message content types.
Test: blink_unittests --gtest_filter="LanguageModelToolTypeTest.*"
NO_IFTTT=It is not added by this CL.
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |