Introduce SwarmingOutputValidator agent skill [chromium/src : main]
Brian Sheedy (Gerrit)
Brian Sheedy added 1 comment![]( "Open in Gerrit")
- Locate `CAS Inputs` digest hash from the Swarming task summary page.This seems like a potentially huge context hog since the task page contains the entire stdout/stderr log from the test, which is potentially massive. This should probably either be provided as input or a helper script should be used to retrieve the task page and extract just the input isolate digest.
Related details
- Aadarsh Battula
- David Hinojosa
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
- Locate `CAS Inputs` digest hash from the Swarming task summary page.This seems like a potentially huge context hog since the task page contains the entire stdout/stderr log from the test, which is potentially massive. This should probably either be provided as input or a helper script should be used to retrieve the task page and extract just the input isolate digest.
- Brian Sheedy
- David Hinojosa
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Brian Sheedy (Gerrit)
Brian Sheedy added 4 comments![]( "Open in Gerrit")
- Locate `CAS Inputs` digest hash from the Swarming task summary page.Aadarsh BattulaThis seems like a potentially huge context hog since the task page contains the entire stdout/stderr log from the test, which is potentially massive. This should probably either be provided as input or a helper script should be used to retrieve the task page and extract just the input isolate digest.
Marked as resolved.
Ah, looking at the script, you're actually pulling the information from Buildbucket, which should be a lot smaller than the full Swarming task page.
I think keeping the helper script is still useful so we can rely on that part of the skill being deterministic, though.
import requests # pylint: disable=import-outside-toplevelImport at the module level.
"""Comprehensive unit tests for get_task_digest.py achieving 100% coverage."""`Unit tests for get_task_digest.py` is sufficient.
In general, remove/update LLM-like comments/docstrings.
"""Tests 100% of statements and branches across all helper functions."""Ditto.
Related details
- Aadarsh Battula
- David Hinojosa
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 3 comments![]( "Open in Gerrit")
import requests # pylint: disable=import-outside-toplevelImport at the module level.
Marked as resolved.
"""Comprehensive unit tests for get_task_digest.py achieving 100% coverage."""`Unit tests for get_task_digest.py` is sufficient.
In general, remove/update LLM-like comments/docstrings.
Acknowledged
"""Tests 100% of statements and branches across all helper functions."""Ditto.
Marked as resolved.
- Brian Sheedy
- David Hinojosa
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
David Hinojosa (Gerrit)
David Hinojosa added 1 comment![]( "Open in Gerrit")
Related details
- Aadarsh Battula
- Brian Sheedy
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Brian Sheedy (Gerrit)
Brian Sheedy voted Code-Review+1![]( "Open in Gerrit")
- Aadarsh Battula
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
Related details
- Brian Sheedy
- David Hinojosa
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
David Hinojosa (Gerrit)
David Hinojosa added 1 comment![]( "Open in Gerrit")
> [!WARNING] **Context Clogging Trap (Prohibit Web Page Scraping)** Do NOT
> scrape raw HTML task summary pages (`chromium-swarm.appspot.com` or
> `ci.chromium.org`). Swarming embeds massive stdout/stderr chunks directly in
> DOM trees, which will overwhelm context limits.nit: This rule kinda seems relevant for all the skills in the Coverage Debugging Agent. How about defining this in the orchestrator agent to pass on to all subagents or as a global rule for all the agents to read at startup?
Related details
- Aadarsh Battula
- Brian Sheedy
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
> [!WARNING] **Context Clogging Trap (Prohibit Web Page Scraping)** Do NOT
> scrape raw HTML task summary pages (`chromium-swarm.appspot.com` or
> `ci.chromium.org`). Swarming embeds massive stdout/stderr chunks directly in
> DOM trees, which will overwhelm context limits.nit: This rule kinda seems relevant for all the skills in the Coverage Debugging Agent. How about defining this in the orchestrator agent to pass on to all subagents or as a global rule for all the agents to read at startup?
- Brian Sheedy
- David Hinojosa
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Prakhar Asthana (Gerrit)
Prakhar Asthana added 2 comments![]( "Open in Gerrit")
name: experimental-swarming-output-validatorlets mention "code coverage" in there, so that it is not mistaken for any generic swarming output validator
Validates Swarming execution logs, CAS input/output artifacts, and profile
generation environments to isolate remote trybot failures.Similar to above, restrict it to code coverage
Related details
- Aadarsh Battula
- Brian Sheedy
- David Hinojosa
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 2 comments![]( "Open in Gerrit")
lets mention "code coverage" in there, so that it is not mistaken for any generic swarming output validator
Marked as resolved.
Validates Swarming execution logs, CAS input/output artifacts, and profile
generation environments to isolate remote trybot failures.Similar to above, restrict it to code coverage
- Brian Sheedy
- David Hinojosa
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Prakhar Asthana (Gerrit)
Prakhar Asthana voted Code-Review+1![]( "Open in Gerrit")
- Aadarsh Battula
- Brian Sheedy
- David Hinojosa
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Prakhar Asthana (Gerrit)
Prakhar Asthana voted Code-Review+1![]( "Open in Gerrit")
| Code-Review | +1 |
Related details
- Aadarsh Battula
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Struan Shrimpton (Gerrit)
Struan Shrimpton added 3 comments![]( "Open in Gerrit")
## InfrastructureI don't think these belong in the common templates. Looking at swarming/luci is a comparatively bespoke thing but it's also creating an unnecessary dependency between the existing skill and these templates
First, extract the Buildbucket build ID (`build_id`) from `build_url` (e.g.,The build url is not always addressed via the build id. e.g. https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/2772707/infra
swarming collect -S chromium-swarm.appspot.com \I'm not sure you can take for granted swarming is on PATH?
Related details
- Aadarsh Battula
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 3 comments![]( "Open in Gerrit")
I don't think these belong in the common templates. Looking at swarming/luci is a comparatively bespoke thing but it's also creating an unnecessary dependency between the existing skill and these templates
Marked as resolved.
First, extract the Buildbucket build ID (`build_id`) from `build_url` (e.g.,The build url is not always addressed via the build id. e.g. https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/2772707/infra
Marked as resolved.
I'm not sure you can take for granted swarming is on PATH?
- Prakhar Asthana
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Struan Shrimpton (Gerrit)
Struan Shrimpton voted and added 1 comment![]( "Open in Gerrit")
Votes added by Struan Shrimpton
| Code-Review | +1 |
1 comment
First, resolve the numeric Buildbucket build ID (`build_id`) from `build_url`. Note that LUCI URLs may use direct Buildbucket IDs (`/b/8679...`) or sequential builder paths (`.../builders/try/linux-rel/2772707/...`). When given a builder path URL, pass the canonical builder path (`chromium/try/linux-rel/2772707`) directly to `bb get` or use the `buildbucket` skill to resolve the numeric build ID.nit: word wrap
Related details
- Aadarsh Battula
- Prakhar Asthana
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
First, resolve the numeric Buildbucket build ID (`build_id`) from `build_url`. Note that LUCI URLs may use direct Buildbucket IDs (`/b/8679...`) or sequential builder paths (`.../builders/try/linux-rel/2772707/...`). When given a builder path URL, pass the canonical builder path (`chromium/try/linux-rel/2772707`) directly to `bb get` or use the `buildbucket` skill to resolve the numeric build ID.nit: word wrap
Marked as resolved.
- Brian Sheedy
- Prakhar Asthana
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Brian Sheedy (Gerrit)
Brian Sheedy added 2 comments![]( "Open in Gerrit")
### 3. Pre-Run CAS Input Audit (Combination 4)Might need re-numbering/re-ordering? There's combination 1, 4, and 2 in that order. I'm not quite sure what `Combination` means in this context, though. Is `Scenario` maybe a better descriptor?
> [!WARNING] **Context Clogging Trap (Prohibit Web Page Scraping)** Do NOT
> scrape raw HTML task summary pages (`chromium-swarm.appspot.com` or
> `ci.chromium.org`). Swarming embeds massive stdout/stderr chunks directly in
> DOM trees, which will overwhelm context limits.nit: This rule kinda seems relevant for all the skills in the Coverage Debugging Agent. How about defining this in the orchestrator agent to pass on to all subagents or as a global rule for all the agents to read at startup?
Brian SheedyMarked as resolved.
This appears to still be present in the most recent patchset, but marking this as resolved implies that it should have been moved somewhere else.
Related details
- Aadarsh Battula
- Prakhar Asthana
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Prakhar Asthana (Gerrit)
Prakhar Asthana voted Code-Review+1![]( "Open in Gerrit")
- Aadarsh Battula
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
Might need re-numbering/re-ordering? There's combination 1, 4, and 2 in that order. I'm not quite sure what `Combination` means in this context, though. Is `Scenario` maybe a better descriptor?
Removed the combination numbering. Combination was supposed to represent the different ways we can test out the coverage generation locally.
Related details
- Brian Sheedy
- David Hinojosa
- Prakhar Asthana
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
> [!WARNING] **Context Clogging Trap (Prohibit Web Page Scraping)** Do NOT
> scrape raw HTML task summary pages (`chromium-swarm.appspot.com` or
> `ci.chromium.org`). Swarming embeds massive stdout/stderr chunks directly in
> DOM trees, which will overwhelm context limits.Aadarsh Battulanit: This rule kinda seems relevant for all the skills in the Coverage Debugging Agent. How about defining this in the orchestrator agent to pass on to all subagents or as a global rule for all the agents to read at startup?
Brian SheedyMarked as resolved.
This appears to still be present in the most recent patchset, but marking this as resolved implies that it should have been moved somewhere else.
I removed it because of Struan's comment, which said that the rule should not to be invoked for every chromium agent. I just added a code_coverage.md file that includes the rule in a new CL. This file will be imported in src/GEMINI.md during the code coverage installer script. It will be removed at the end of orchestrator skill as well.
https://chromium-review.git.corp.google.com/c/chromium/src/+/8067119
Related details
- Brian Sheedy
- David Hinojosa
- Prakhar Asthana
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Brian Sheedy (Gerrit)
Brian Sheedy voted and added 3 comments![]( "Open in Gerrit")
Votes added by Brian Sheedy
| Code-Review | +1 |
3 comments
> [!WARNING] **Context Clogging Trap (Prohibit Web Page Scraping)** Do NOT
> scrape raw HTML task summary pages (`chromium-swarm.appspot.com` or
> `ci.chromium.org`). Swarming embeds massive stdout/stderr chunks directly in
> DOM trees, which will overwhelm context limits.Aadarsh Battulanit: This rule kinda seems relevant for all the skills in the Coverage Debugging Agent. How about defining this in the orchestrator agent to pass on to all subagents or as a global rule for all the agents to read at startup?
Brian SheedyMarked as resolved.
Aadarsh BattulaThis appears to still be present in the most recent patchset, but marking this as resolved implies that it should have been moved somewhere else.
I removed it because of Struan's comment, which said that the rule should not to be invoked for every chromium agent. I just added a code_coverage.md file that includes the rule in a new CL. This file will be imported in src/GEMINI.md during the code coverage installer script. It will be removed at the end of orchestrator skill as well.
https://chromium-review.git.corp.google.com/c/chromium/src/+/8067119
- Aadarsh Battula
- David Hinojosa
- Prakhar Asthana
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula added 1 comment![]( "Open in Gerrit")
Aadarsh BattulaWrap at 80 characters.
Meant to remove the global rule file from this CL and include it in this one: https://chromium-review.git.corp.google.com/c/chromium/src/+/8067119
Just addressed this comment in the other CL.
Related details
- Brian Sheedy
- David Hinojosa
- Prakhar Asthana
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Brian Sheedy (Gerrit)
Brian Sheedy voted Code-Review+1![]( "Open in Gerrit")
- Aadarsh Battula
- David Hinojosa
- Prakhar Asthana
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Prakhar Asthana (Gerrit)
Prakhar Asthana voted Code-Review+1![]( "Open in Gerrit")
| Code-Review | +1 |
Related details
- Aadarsh Battula
- David Hinojosa
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Aadarsh Battula (Gerrit)
Aadarsh Battula voted Commit-Queue+2![]( "Open in Gerrit")
- David Hinojosa
- Struan Shrimpton
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Chromium LUCI CQ (Gerrit)
Chromium LUCI CQ submitted the change![]( "Open in Gerrit")
Change information
Introduce SwarmingOutputValidator agent skill
This CL introduces the `experimental-swarming-output-validator` agent
skill describing how to audit remote Swarming shard executions on
Chromium tryjobs.
Includes:
1. SKILL.md defining step-by-step workflows for Fast Remote Output
Audits (swarming collect), Pre-Run CAS Input Audits (cas download),
and Hermetic Container Replays (swarming reproduce).
2. Standalone helper tool to retrieve trigger step output logs and
extract CAS isolate digest hashes.
TAG=agy
CONV=2771ec5f-7b9c-40c6-b3fd-85d1265eece0
- A agents/skills/experimental-code-coverage-swarming-output-validator/SKILL.md
- A tools/code_coverage/get_task_digest.py
- A tools/code_coverage/get_task_digest_test.py
Code-Review: +1 by Prakhar Asthana, +1 by Brian Sheedy
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |