Repository navigation
[camera_android_camerax] Enforce CHANGELOG backticks and add eval commit author check to pre-push-skill - #12624
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the pre-push skill documentation, evaluation tests, and setup scripts to verify CHANGELOG.md formatting and version bumps. It adds a new evaluation case for user-facing documentation changes, updates the success setup script to bump versions and update the changelog, and introduces a new setup script for user-facing documentation changes. Feedback suggests checking the exit codes of git add and git commit commands in both setup scripts to ensure they fail visibly upon failure.
c9ae625 to
c95fae1
Compare
…, and document parameters
| stdout.writeln('Error: Cannot run setup scripts on main branch.'); | ||
| exit(1); | ||
| } | ||
| ensureNotMainBranch(); |
There was a problem hiding this comment.
This was pre existing code that I extracted.
… exit codes, and auto-detect upstream remote
…nd isolate test author credentials
…asing in check_eval_commits.dart
|
There was a problem hiding this comment.
Code Review
This pull request introduces a pre-push validation check to detect and prevent commits authored or committed with evaluation test credentials. It adds a check_eval_commits.dart tool, helper utilities in test_utils.dart, and corresponding evaluation test scenarios, while updating the SKILL.md documentation and pubspec.yaml dependencies. Feedback on the changes suggests refining the git log format to avoid false positives from commit messages and making the remote URL regex more robust by handling optional trailing slashes.
…e> | %s' and support trailing slash in remote URL
| '-m', | ||
| 'Add DummyEvalFeature.java without tests', | ||
| ]); | ||
| commitFiles(<String>[javaFile.path], 'Add DummyEvalFeature.java without tests'); |
There was a problem hiding this comment.
Mostly just thinking aloud here but wanted to confirm your thinking:
I'm conflicted because I would think that this one and setup_mixed_native_test.dart should have the override to ensure the code is ready to push even when the author is valid. However, the eval still technically checks that the test check would fail individually, it just doesn't isolate that as the reason for determining that the code is not ready to push. I could see overriding the author in these cases being a slippery slope for doing the same for similar evals we'll likely add in the future that also modify code, and I'm not sure that it's a major value loss to not isolate reasons for the check failing. Wdyt?
There was a problem hiding this comment.
"even when the author is valid" Or did you mean even when the author is invalid?
What I am trying to protect here is eval code getting pushed. Skills that modify code are totally fine and they would use the local git config.
FWIW this was a not a theoretical issue. Jetski ide changed the rules around when it can branch into a subagent with isolated code and this pr committed eval code and pushed it. See https://github.com/flutter/packages/compare/c9ae62501ef56d4382ea2047b14a3c3926111c71..c95fae1cbc1b23b2c98335f59dfaafe37488ffa3
There was a problem hiding this comment.
Ah ok yes. What I was describing is the case where a pre-push eval results in "not ready to push" but that is now because (1) the original subject matter that we are actually evaluating and (2) the subagent pushed code to test the original subject matter. So now the conclusion "The eval failed because the original subject matter" is not technically true anymore.
At the end of the day, we need to protect subagents from pushing though so SGTM!
There was a problem hiding this comment.
We talked offline. The issue is that my check for invalid author in evals should not be part of this pr and should be part of a pre push git hook. Will remove that part of the pr and get rereview.
There was a problem hiding this comment.
Removed the commit author check from pre-push-skill and all corresponding evals. All 5 remaining evals and repo presubmit checks pass.
…ness skill (#12628) This came up as part of #12624 I noticed that the pre-push skill passed my code but linux analyze failed. It was because I didnt realize that this checkout (I have 2) didnt have git hooks installed. - @reidbaker --- Agent authored description Updates the `check-readiness` skill to verify that Git pre-commit hooks (`script/githooks`) are configured in the repository (`core.hooksPath`). If unconfigured or pointing to an unexpected path, it provides actionable instructions to run `script/githooks/bin/install_hooks.dart`. Includes unit test coverage in `check_test.dart`. Part of flutter/flutter#188870 ## Pre-Review Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] page, which explains my responsibilities. - [x] I read and followed the [relevant style guides] and ran [the auto-formatter]. - [x] I signed the [CLA]. - [x] The title of the PR starts with the name of the package surrounded by square brackets, e.g. `[shared_preferences]` - [x] I [linked to at least one issue that this PR fixes] in the description above. - [x] I followed [the version and CHANGELOG instructions], using [semantic versioning] and the [repository CHANGELOG style], or I have commented below to indicate which documented exception this PR falls under[^1]. - [x] I updated/added any relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or I have commented below to indicate which [test exemption] this PR falls under[^1]. - [x] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/packages/blob/main/CONTRIBUTING.md [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md [relevant style guides]: https://github.com/flutter/packages/blob/main/CONTRIBUTING.md#style [the auto-formatter]: https://github.com/flutter/packages/blob/main/script/tool/README.md#format-code [CLA]: https://cla.developers.google.com/ [Discord]: https://github.com/flutter/flutter/blob/master/docs/contributing/Chat.md [linked to at least one issue that this PR fixes]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#overview [the version and CHANGELOG instructions]: https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#version-and-changelog-updates [semantic versioning]: https://dart.dev/tools/pub/versioning#semantic-versions [repository CHANGELOG style]: https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#changelog-style [test exemption]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#tests --------- Co-authored-by: Reid Baker <[email protected]>
86b1070 to
d5ece3f
Compare
…er#191965) flutter/packages@bd3cbc1...cd4cdd0 2026-08-28 [email protected] [material_ui] Add all M3 templates and generated code to `temporarily_excluded/` before migration (flutter/packages#12661) 2026-08-28 [email protected] Roll Flutter from 15d8908 to e8dca90 (58 revisions) (flutter/packages#12660) 2026-08-28 [email protected] [various] Update pigeon dev_dependency to ^27.3.2 (flutter/packages#12615) 2026-08-27 [email protected] [camera_android_camerax] Enforce CHANGELOG backticks and add eval commit author check to pre-push-skill (flutter/packages#12624) 2026-08-27 [email protected] [material_ui] Prevent stale async suggestions in SearchAnchor (flutter/packages#12478) 2026-08-27 [email protected] [go_router_builder] Support analyzer 14 (flutter/packages#12614) 2026-08-27 [email protected] [cupertino_ui] Use super parameters in more places (flutter/packages#12459) 2026-08-27 [email protected] [material_ui] Remove no-shuffle from progress indicator test (flutter/packages#12505) 2026-08-27 [email protected] [two_dimensional_scrollables] Activate leak testing and fix memory leaks (flutter/packages#11653) 2026-08-27 [email protected] [material_ui] Add helper methods in gen_defaults template (flutter/packages#12637) 2026-08-27 [email protected] [cupertino_ui] fix CupertinoIcons font not being included in examples and fix TextEditingController leaks (flutter/packages#12228) 2026-08-27 [email protected] [camera_android_camerax] Check Git hooks configuration in check-readiness skill (flutter/packages#12628) 2026-08-27 [email protected] [ci] sync back pr for branch release only run when release succeeds (flutter/packages#12581) 2026-08-27 [email protected] [google_maps_flutter] Convert overlay controllers to Swift (flutter/packages#12638) 2026-08-27 [email protected] [go_router_builder] Fixes text golden test to ignore platform specific newline (flutter/packages#12652) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
…ness skill (flutter#12628) This came up as part of flutter#12624 I noticed that the pre-push skill passed my code but linux analyze failed. It was because I didnt realize that this checkout (I have 2) didnt have git hooks installed. - @reidbaker --- Agent authored description Updates the `check-readiness` skill to verify that Git pre-commit hooks (`script/githooks`) are configured in the repository (`core.hooksPath`). If unconfigured or pointing to an unexpected path, it provides actionable instructions to run `script/githooks/bin/install_hooks.dart`. Includes unit test coverage in `check_test.dart`. Part of flutter/flutter#188870 ## Pre-Review Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] page, which explains my responsibilities. - [x] I read and followed the [relevant style guides] and ran [the auto-formatter]. - [x] I signed the [CLA]. - [x] The title of the PR starts with the name of the package surrounded by square brackets, e.g. `[shared_preferences]` - [x] I [linked to at least one issue that this PR fixes] in the description above. - [x] I followed [the version and CHANGELOG instructions], using [semantic versioning] and the [repository CHANGELOG style], or I have commented below to indicate which documented exception this PR falls under[^1]. - [x] I updated/added any relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or I have commented below to indicate which [test exemption] this PR falls under[^1]. - [x] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/packages/blob/main/CONTRIBUTING.md [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md [relevant style guides]: https://github.com/flutter/packages/blob/main/CONTRIBUTING.md#style [the auto-formatter]: https://github.com/flutter/packages/blob/main/script/tool/README.md#format-code [CLA]: https://cla.developers.google.com/ [Discord]: https://github.com/flutter/flutter/blob/master/docs/contributing/Chat.md [linked to at least one issue that this PR fixes]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#overview [the version and CHANGELOG instructions]: https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#version-and-changelog-updates [semantic versioning]: https://dart.dev/tools/pub/versioning#semantic-versions [repository CHANGELOG style]: https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#changelog-style [test exemption]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#tests --------- Co-authored-by: Reid Baker <[email protected]>
…mit author check to pre-push-skill (flutter#12624) This work started by adding evals to make sure the changlog backticks work did not regress. Then I discovered that the agents.md file I updated didn't work if the camera_android_camerax folder was opened which was the documented project-oneshot plan. When authoring those evals I ran into issues branching the workspace. google/skills_lint.dart#3 addresses updating the skill to work in more situations. This pr also adds some shared utilities and a script to verify that eval commits are not pushed. - @reidbaker --- Agent authored description. Enforces the repository-wide requirement that code references in `CHANGELOG.md` must be enclosed in backticks, and adds verification tooling to prevent pushing evaluation test commits. ## Changes ### 1. CHANGELOG Backtick Enforcement - Updated `pre-push-skill/SKILL.md` to verify `CHANGELOG.md` formatting following the ecosystem contributing guidelines. - Linked package `AGENTS.md` to repository root `AGENTS.md`. - Added Eval 5 (`setup_user_facing_docs_change.dart`) to evaluate and assert detection of unquoted code references in `CHANGELOG.md`. ### 2. Eval Commit Prevention & Tooling - Created `evals/tool/check_eval_commits.dart` CLI tool using `package:args` and idiomatic exit code handling to verify that outgoing branch commits contain no forbidden evaluation author credentials (`[email protected]`). - Added Step 3 ("Check for Evaluation Test Commits") to `pre-push-skill/SKILL.md`. - Extracted shared author credentials and commit helpers into `evals/tool/test_utils.dart` and migrated all eval setup scripts to use it. - Added Eval 6 (`setup_eval_commit_failure.dart`) to evaluate and assert rejection of evaluation test commits. ## Verification - `flutter test test/skills_evals_test.dart` - `flutter test test/validate_skills_test.dart` - `dart analyze .agents/skills/pre-push-skill/ evals/` - `dart run evals/tool/check_eval_commits.dart --remote=origin` - Ran full evaluation suite (Evals 1–6) with subagents in isolated worktrees.
…ness skill (flutter#12628) This came up as part of flutter#12624 I noticed that the pre-push skill passed my code but linux analyze failed. It was because I didnt realize that this checkout (I have 2) didnt have git hooks installed. - @reidbaker --- Agent authored description Updates the `check-readiness` skill to verify that Git pre-commit hooks (`script/githooks`) are configured in the repository (`core.hooksPath`). If unconfigured or pointing to an unexpected path, it provides actionable instructions to run `script/githooks/bin/install_hooks.dart`. Includes unit test coverage in `check_test.dart`. Part of flutter/flutter#188870 ## Pre-Review Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] page, which explains my responsibilities. - [x] I read and followed the [relevant style guides] and ran [the auto-formatter]. - [x] I signed the [CLA]. - [x] The title of the PR starts with the name of the package surrounded by square brackets, e.g. `[shared_preferences]` - [x] I [linked to at least one issue that this PR fixes] in the description above. - [x] I followed [the version and CHANGELOG instructions], using [semantic versioning] and the [repository CHANGELOG style], or I have commented below to indicate which documented exception this PR falls under[^1]. - [x] I updated/added any relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or I have commented below to indicate which [test exemption] this PR falls under[^1]. - [x] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/packages/blob/main/CONTRIBUTING.md [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md [relevant style guides]: https://github.com/flutter/packages/blob/main/CONTRIBUTING.md#style [the auto-formatter]: https://github.com/flutter/packages/blob/main/script/tool/README.md#format-code [CLA]: https://cla.developers.google.com/ [Discord]: https://github.com/flutter/flutter/blob/master/docs/contributing/Chat.md [linked to at least one issue that this PR fixes]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#overview [the version and CHANGELOG instructions]: https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#version-and-changelog-updates [semantic versioning]: https://dart.dev/tools/pub/versioning#semantic-versions [repository CHANGELOG style]: https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#changelog-style [test exemption]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#tests --------- Co-authored-by: Reid Baker <[email protected]>
…mit author check to pre-push-skill (flutter#12624) This work started by adding evals to make sure the changlog backticks work did not regress. Then I discovered that the agents.md file I updated didn't work if the camera_android_camerax folder was opened which was the documented project-oneshot plan. When authoring those evals I ran into issues branching the workspace. google/skills_lint.dart#3 addresses updating the skill to work in more situations. This pr also adds some shared utilities and a script to verify that eval commits are not pushed. - @reidbaker --- Agent authored description. Enforces the repository-wide requirement that code references in `CHANGELOG.md` must be enclosed in backticks, and adds verification tooling to prevent pushing evaluation test commits. ## Changes ### 1. CHANGELOG Backtick Enforcement - Updated `pre-push-skill/SKILL.md` to verify `CHANGELOG.md` formatting following the ecosystem contributing guidelines. - Linked package `AGENTS.md` to repository root `AGENTS.md`. - Added Eval 5 (`setup_user_facing_docs_change.dart`) to evaluate and assert detection of unquoted code references in `CHANGELOG.md`. ### 2. Eval Commit Prevention & Tooling - Created `evals/tool/check_eval_commits.dart` CLI tool using `package:args` and idiomatic exit code handling to verify that outgoing branch commits contain no forbidden evaluation author credentials (`[email protected]`). - Added Step 3 ("Check for Evaluation Test Commits") to `pre-push-skill/SKILL.md`. - Extracted shared author credentials and commit helpers into `evals/tool/test_utils.dart` and migrated all eval setup scripts to use it. - Added Eval 6 (`setup_eval_commit_failure.dart`) to evaluate and assert rejection of evaluation test commits. ## Verification - `flutter test test/skills_evals_test.dart` - `flutter test test/validate_skills_test.dart` - `dart analyze .agents/skills/pre-push-skill/ evals/` - `dart run evals/tool/check_eval_commits.dart --remote=origin` - Ran full evaluation suite (Evals 1–6) with subagents in isolated worktrees.
This work started by adding evals to make sure the changlog backticks work did not regress. Then I discovered that the agents.md file I updated didn't work if the camera_android_camerax folder was opened which was the documented project-oneshot plan.
When authoring those evals I ran into issues branching the workspace. google/skills_lint.dart#3 addresses updating the skill to work in more situations.
This pr also adds some shared utilities and a script to verify that eval commits are not pushed.
Agent authored description.
Enforces the repository-wide requirement that code references in
CHANGELOG.mdmust be enclosed in backticks, and adds verification tooling to prevent pushing evaluation test commits.Changes
1. CHANGELOG Backtick Enforcement
pre-push-skill/SKILL.mdto verifyCHANGELOG.mdformatting following the ecosystem contributing guidelines.AGENTS.mdto repository rootAGENTS.md.setup_user_facing_docs_change.dart) to evaluate and assert detection of unquoted code references inCHANGELOG.md.2. Eval Commit Prevention & Tooling
evals/tool/check_eval_commits.dartCLI tool usingpackage:argsand idiomatic exit code handling to verify that outgoing branch commits contain no forbidden evaluation author credentials ([email protected]).pre-push-skill/SKILL.md.evals/tool/test_utils.dartand migrated all eval setup scripts to use it.setup_eval_commit_failure.dart) to evaluate and assert rejection of evaluation test commits.Verification
flutter test test/skills_evals_test.dartflutter test test/validate_skills_test.dartdart analyze .agents/skills/pre-push-skill/ evals/dart run evals/tool/check_eval_commits.dart --remote=origin