Repository navigation
Expose flaky test failures for jobs - #1021
Merged
Merged
Conversation
List failed tests currently labeled flaky by Test Engine for a job. Keep empty results inconclusive rather than asserting a causal verdict. Amp-Thread-ID: T-a1c01a67-e152-4121-8206-9bf76c88dbc4
![buildsworth-bk-app[bot]](https://avatars.githubusercontent.com/in/3010600?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The CLI does not expose Test Engine's flaky labels against a failed job, so agents cannot use that evidence when deciding whether to retry a failure.
Add
bk job flaky-tests <job-uuid>to list tests with failed executions for that job that Test Engine currently labels flaky. This is not a verdict that flakiness caused the job failure; empty results remain inconclusive when data is missing or incomplete.Related: PB-2564.
Changes
Testing
env -u BUILDKITE_API_TOKEN mise run test); the orb's token override otherwise interferes with existing credential tests.mise run format).mise run lintreports 0 issues. Command tests cover pagination, JSON/YAML output, missing data, and API errors. Live Test Engine data was not exercised.Disclosures / Credits
Amp implemented the command, tests, and documentation and prepared this description. Amp's Oracle reviewed the changes and reported no actionable correctness findings.