[experimental-ci-only-validator] Introduce ci_only validator skill [chromium/src : main]
Aadarsh Battula (Gerrit)
Aadarsh Battula has uploaded the change for review![]( "Open in Gerrit")
Commit message
[experimental-ci-only-validator] Introduce ci_only validator skill
This CL introduces the experimental-ci-only-validator skill
which uses the underlying Python helper script to help triaging
missing coverage bugs on Gerrit.
Bug: 519590223
Test: None
TAG=agy
CONV=19f2d950-e07e-427c-a7a3-de836354df62
[experimental-ci-only-validator] Add ci_only validator tool
This CL introduces the ci_only validator tool and its tests.
The tool checks if a Chromium test suite is barred by `ci_only = True`
on both the coverage-specific trybots and standard CQ coverage trybots.
Bug: 519590223
Test: vpython3 tools/code_coverage/ci_only_validator_test.py
TAG=agy
CONV=19f2d950-e07e-427c-a7a3-de836354df62
Change diff
Change information
- A agents/prompts/eval/experimental-ci-only-validator/basic_check.promptfoo.yaml
- A agents/skills/experimental-ci-only-validator/SKILL.md
- A tools/code_coverage/ci_only_validator.py
- A tools/code_coverage/ci_only_validator_test.py
Related details
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
#### 2. Checking Historical Configuration (at a revision)I chose to not include the time travel code explorer skill since it requires multiple tool calls to explore all the try builder configs at a specific revision.
Related details
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Brian Sheedy (Gerrit)
Brian Sheedy added 2 comments![]( "Open in Gerrit")
- type: contains
value: "ci_only = True"This seems liable to fail because the agent won't report this exact string.
Expecting certain builder names to be reported makes sense, but I don't think this does.
#### 2. Checking Historical Configuration (at a revision)I chose to not include the time travel code explorer skill since it requires multiple tool calls to explore all the try builder configs at a specific revision.
Acknowledged
Related details
- Aadarsh Battula
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
- type: contains
value: "ci_only = True"This seems liable to fail because the agent won't report this exact string.
Expecting certain builder names to be reported makes sense, but I don't think this does.
Would it be better to define what the model should output in the query? Ex. Output "ci_only: True" if the configs contain it. I am pending access to google cloud resources, so I am unable to run evals locally still.
Related details
- Brian Sheedy
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Brian Sheedy (Gerrit)
Brian Sheedy added 1 comment![]( "Open in Gerrit")
- type: contains
value: "ci_only = True"Aadarsh BattulaThis seems liable to fail because the agent won't report this exact string.
Expecting certain builder names to be reported makes sense, but I don't think this does.
Would it be better to define what the model should output in the query? Ex. Output "ci_only: True" if the configs contain it. I am pending access to google cloud resources, so I am unable to run evals locally still.
I think the better option would be to update prompt to ask for what we really want. We don't really care about the output format that the agent uses, we only care about the information included in its response.
In this particular case, a prompt like
`Use the experimental-ci-only-validator skill to find all CI-only code coverage LUCI builders that run the 'chrome_junit_tests' suite` seems more fitting and then the asserts can be that the substrings of the builders that we expect.
Related details
- Aadarsh Battula
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
- type: contains
value: "ci_only = True"Aadarsh BattulaThis seems liable to fail because the agent won't report this exact string.
Expecting certain builder names to be reported makes sense, but I don't think this does.
Brian SheedyWould it be better to define what the model should output in the query? Ex. Output "ci_only: True" if the configs contain it. I am pending access to google cloud resources, so I am unable to run evals locally still.
I think the better option would be to update prompt to ask for what we really want. We don't really care about the output format that the agent uses, we only care about the information included in its response.
In this particular case, a prompt like
`Use the experimental-ci-only-validator skill to find all CI-only code coverage LUCI builders that run the 'chrome_junit_tests' suite` seems more fitting and then the asserts can be that the substrings of the builders that we expect.
- Brian Sheedy
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Brian Sheedy (Gerrit)
Brian Sheedy voted and added 1 comment![]( "Open in Gerrit")
Votes added by Brian Sheedy
| Code-Review | +1 |
1 comment
LGTM
+CC bryankim since this adds another eval
Related details
- Aadarsh Battula
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Prakhar Asthana (Gerrit)
Prakhar Asthana voted Code-Review+1![]( "Open in Gerrit")
- Aadarsh Battula
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula voted Commit-Queue+2![]( "Open in Gerrit")
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Chromium LUCI CQ (Gerrit)
Chromium LUCI CQ submitted the change![]( "Open in Gerrit")
Unreviewed changes
19 is the latest approved patch-set.
No files were changed between the latest approved patch-set and the submitted one.
Change information
Introduce ci_only validator skill
This CL introduces the experimental-ci-only-validator skill
which uses the underlying Python helper script to help triaging
missing coverage bugs on Gerrit.
TAG=agy
CONV=19f2d950-e07e-427c-a7a3-de836354df62
- A agents/prompts/eval/experimental-ci-only-validator/basic_check.promptfoo.yaml
- A agents/skills/experimental-ci-only-validator/SKILL.md
Code-Review: +1 by Brian Sheedy, +1 by Prakhar Asthana
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |