[Built-In API] Add kInternal InputSource for metric attribution. [chromium/src : main]
Robbie McElrath (Gerrit)
Robbie McElrath added 1 comment![]( "Open in Gerrit")
Isaac AhoumaAre these actually all user input, or are some of them things like system prompts or model output?
Isaac AhoumaI checked with Gemini CLI and this is what I got:
```
I have verified that all three call sites correctly use kUserInput based on the available InputSource enum values:1. OnDeviceContext::Append: This is used to feed the conversation history and any feature-specific context into the model session. Since this data is
provided as part of the user's interaction session, kUserInput is the most appropriate attribution.
2. OnDeviceExecution::BeginExecution: This appends the user's actual prompt right before generation. This is direct user input, making kUserInput the
correct choice.
3. OnDeviceModelValidator::ValidateNextPrompt: This sends pre-defined validation prompts to the model. While these are synthetic, they represent the "user
side" of the interaction, and kUserInput is the most accurate existing attribution (as they are certainly not model feedback or unknown).None of these calls involve feeding the model's own generated output back into itself, which is what kModelOutputFeedback is reserved for. Additionally,
the current InputSource enum does not have a specific value for system-generated prompts, so kUserInput is the standard for all non-feedback inputs in this
context.
```
Steven HolteI guess, one thing we can consider here is adding a separate enum value for system prompts (or rename kModelOutputFeedback). WDYT?
Isaac AhoumaI'm confused about how this is different from what the control tokens are supposed to tell you? I think "UserInput" defined as "input provided directly by the user" is promising a distinction you can't actually make, since it suggests things like tokens coming from the prompt template are distingushed from tokens in the actual text inserted in the template.
Isaac AhoumaThanks for the feedback! I'm not deeply familiar with the Token enum (control tokens), but my understanding is that Token and InputSource operate at different levels:
- Token: Seems to work at the piece-level. It helps the model distinguish structural roles within the input text, like separating system instructions from user text in a template.
- InputSource: This is used at the action-level (via AppendOptions). It was introduced specifically for accurate performance metric attribution (like tokens/sec and processing time) by tagging the origin of the entire task.
The main reason for adding InputSource (in https://crrev.com/c/7577516) was to get a clear signal for bucketed logging. This allows the backend to separate metrics for new user inputs versus "feedback" inputs, where the model's previous output is used as the next input. The histograms were updated in https://crrev.com/i/9029232 to use this, adding suffixes like .ModelFeedback to fix http://b/483729567.
Regarding the scope of InputSource, IIUC an input should be tagged kUserInput if it originates from the active user interaction session, even if that input string contains system-provided template elements (which would be marked by Token enums).
I'm open to suggestions, so please let me know if this interpretation/approach is off and how we should make it better.
Steven HolteUnresolving
Robbie McElrathMy main issue is that InputSource and input_source field are documented in such a way that they suggest they are authoritative fields about the provencance of the data, but we don't actually even expect all of the things the pass input here to break down their inputs in such a way "UserInput" is in a seperate Append.
I think we should change the semantics here to clarify that this field is *just* for distinguishing metrics. If we don't actually need these metrics broken down for these features (which this CL wouldn't do, since it's assigning the same label to *all* of the append calls for those features), then it seems reasonable they just continue to log as Unknown type input.
If we wanted to distinguish say, context vs execution for them, it would be reasonable to add new labels, but it's not clear that's useful here?
I thought InputSource was meant to reflect provenance as well. If it's not supposed to do that then renaming kUserInput might be all we need here, though populating the field is probably still a good idea.
Related details
- Isaac Ahouma
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Isaac Ahouma (Gerrit)
Isaac Ahouma added 1 comment![]( "Open in Gerrit")
Yes, InputSource is used to indicate provenance as well for metric attribution purposes. I have updated the documentation. I was actually thinking about adding a `kInternal` value (although maybe this is not needed for this cl and I can remove this change if so) and adding a proper variant for that in the histogram.
Related details
- Robbie McElrath
- Steven Holte
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Isaac Ahouma (Gerrit)
Isaac Ahouma has uploaded the change for review![]( "Open in Gerrit")
Isaac Ahouma removed francisjp...@google.com, josiah...@chromium.org, feature-me...@chromium.org, ios-r...@chromium.org, feature-v...@chromium.org, kyungjunle...@google.com, max+watc...@igalia.com, eme-r...@chromium.org, gcasto+w...@chromium.org, katie...@chromium.org and nektar...@chromium.org from reviewers of this change.
Commit message
[Built-In API] Add kInternal InputSource for metric attribution.
This CL updates the InputSource enum to include a new kInternal value
for tasks like validation and clarifies its purpose for performance
metric attribution.
Specifically:
- Updated on_device_model.mojom with kInternal and improved docs.
- Updated OnDeviceModelValidator to use kInternal.
- Added .Internal metric variant and updated C++ mapping.
- Added unit tests to verify InputSource attribution.
Change diff
Change information
- M components/optimization_guide/core/model_execution/on_device_context.cc
- M components/optimization_guide/core/model_execution/on_device_execution.cc
- M components/optimization_guide/core/model_execution/on_device_model_validator.cc
- M components/optimization_guide/core/model_execution/on_device_model_validator_unittest.cc
- M services/on_device_model/public/cpp/test_support/fake_service.cc
- M services/on_device_model/public/mojom/on_device_model.mojom
- M tools/metrics/histograms/metadata/on_device_model/histograms.xml
Related details
- Robbie McElrath
- Steven Holte
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |