Skip to content

Android_hardware_smoke_test: clean up golden copy in CI - #189948

Merged
auto-submit[bot] merged 8 commits into
flutter:masterfrom
andywolff:devicelab-skia-gold-simplification
Jul 27, 2026
Merged

auto-submit[bot] merged 8 commits into
flutter:masterfrom
andywolff:devicelab-skia-gold-simplification

Conversation

@andywolff

Copy link
Copy Markdown
Contributor

In PR #189390 (part of issue #187538) we configured some new CI test shards which run the instrumented tests subsequently after the driver tests. As part of that, we had to copy the goldens to the asset directory so they would be bundled into the instrumented test APK. This PR proposes to skip the copy by configuring the skia gold comparator to write directly to the desired directory, via a new optional localOutputDir parameter.

Pre-launch Checklist

…e tests

Introduce an optional `localOutputDir` configuration parameter to the Skia Gold comparator. When provided, this writes local screenshots directly into the specified assets directory on the host runner, instead of using the legacy prefix-folder resolution.

This preserves backward compatibility for all other engine integration tests that share the skia_gold.dart helper, while completely eliminating the file-copy phase from the smoke tests runner.
@andywolff
andywolff force-pushed the devicelab-skia-gold-simplification branch from 4d6527d to d0e5dd2 Compare July 23, 2026 21:00
@andywolff

Copy link
Copy Markdown
Contributor Author

local and staging runs passed
Vulkan https://ci.chromium.org/b/8675450318281279297
OpenGL ES https://ci.chromium.org/b/8675450303955562721

@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@andywolff andywolff added the CICD Run CI/CD label Jul 23, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Android hardware smoke tests by removing manual directory copying and cleanup of golden images, replacing it with a new localOutputDir parameter in enableSkiaGoldComparator to write goldens directly to the target directory. Feedback highlights that the specified localOutputDir in driver_test.dart resolves incorrectly and should be changed to 'goldens'. Additionally, the new parameter should be documented, and path resolution should be updated to ensure cross-platform compatibility on Windows.

await enableSkiaGoldComparator(namePrefix: 'android_hardware_smoke_test$activeGoldenVariant');
await enableSkiaGoldComparator(
namePrefix: 'android_hardware_smoke_test$activeGoldenVariant',
localOutputDir: 'test_driver/goldens',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Since baseDir in _SkiaGoldComparator defaults to the directory of the running script (which is test_driver/), passing 'test_driver/goldens' as localOutputDir will resolve to test_driver/test_driver/goldens. This will cause the goldens to be written to the wrong directory, meaning they won't be packaged into the APK and won't be cleaned up properly. It should be set to 'goldens' instead.

Suggested change
localOutputDir: 'test_driver/goldens',
localOutputDir: 'goldens',

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

baseDir is parsed without a trailing slash, causing URI resolution to strip the last directory segment (test_driver). To make the behavior match developer intuition, I have updated baseDir assignment to use Uri.directory which guarantees a trailing slash. Now we can pass goldens.

Comment thread dev/tools/android_driver_extensions/lib/skia_gold.dart
Comment on lines +161 to +164
if (localOutputDir != null) {
final String fileName = path.basename(uri.path);
return io.File.fromUri(baseDir.resolve('$localOutputDir/$fileName'));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To ensure cross-platform compatibility (especially on Windows), we should normalize any potential backslashes in localOutputDir to forward slashes before resolving, as standard URI resolution does not treat backslashes as directory separators. Additionally, using uri.pathSegments.last is a cleaner and more platform-independent way to extract the filename from a URI than path.basename(uri.path).

    if (localOutputDir != null) {
      final String fileName = uri.pathSegments.last;
      final String normalizedOutputDir = localOutputDir!.replaceAll('\\', '/');
      return io.File.fromUri(baseDir.resolve('$normalizedOutputDir/$fileName'));
    }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@andywolff
andywolff force-pushed the devicelab-skia-gold-simplification branch from c9ed1b4 to 124a92a Compare July 24, 2026 17:34
@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Android hardware smoke tests by removing manual golden file copying and cleanup in the suite runner, instead configuring the Skia Gold comparator to output local screenshots directly to a specified directory via a new localOutputDir parameter. Feedback on these changes suggests using io.Platform.script.toFilePath() to prevent path resolution issues on Windows, and utilizing path.join with baseDir.toFilePath() for more robust and idiomatic file path construction.

this.namePrefix,
this.localOutputDir,
Uri? baseDir,
}) : baseDir = baseDir ?? Uri.directory(path.dirname(io.Platform.script.path));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using io.Platform.script.path can lead to issues on Windows because it returns a URI path (e.g., /C:/foo/bar), whereas path.dirname and Uri.directory expect a platform-native path. Using io.Platform.script.toFilePath() ensures a correct platform-native path is used.

  }) : baseDir = baseDir ?? Uri.directory(path.dirname(io.Platform.script.toFilePath()));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines +168 to +172
if (localOutputDir != null) {
final String fileName = uri.pathSegments.last;
final String normalizedOutputDir = localOutputDir!.replaceAll(r'\', '/');
return io.File.fromUri(baseDir.resolve('$normalizedOutputDir/$fileName'));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using path.join with baseDir.toFilePath() is more robust and idiomatic than manual string replacement and URI resolution. It automatically handles platform-specific path separators and avoids potential issues with leading/trailing slashes in localOutputDir.

    if (localOutputDir != null) {
      final String fileName = uri.pathSegments.last;
      return io.File(path.join(baseDir.toFilePath(), localOutputDir!, fileName));
    }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@andywolff
andywolff marked this pull request as ready for review July 24, 2026 20:22
@andywolff
andywolff requested a review from jtmcdole July 24, 2026 20:23

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request simplifies the Android hardware smoke tests by removing manual directory copying and cleanup of temporary golden files. It introduces a localOutputDir parameter to enableSkiaGoldComparator to specify where Skia Gold screenshots should be stored. Feedback on these changes suggests preserving the relative path structure of the golden file URI in _getGoldenFile when localOutputDir is provided, as the current implementation discards subdirectories by only using the last path segment.

Comment thread dev/tools/android_driver_extensions/lib/skia_gold.dart
jtmcdole
jtmcdole previously approved these changes Jul 24, 2026
Comment on lines +36 to +38
activeGoldenVariant = (replyVariant != null && replyVariant.isNotEmpty)
? '.$replyVariant'
: '';

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

activeGoldenVariant = switch (replyVariant) {
      String s when s.isNotEmpty => '.$s',
      _ => '',
    };

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. I always felt the ternary operator was a little cryptic, I like this switch semantic a lot better

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, that string needed to be final

jtmcdole
jtmcdole previously approved these changes Jul 24, 2026
@andywolff andywolff added the autosubmit Merge PR when tree becomes green via auto submit App label Jul 24, 2026
@auto-submit auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Jul 24, 2026
@auto-submit

auto-submit Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

autosubmit label was removed for flutter/flutter/189948, because - The status or check suite Tree_analyze has failed. Please fix the issues identified (or deflake) before re-applying this label.

@jtmcdole jtmcdole left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

printer2

@andywolff andywolff added the autosubmit Merge PR when tree becomes green via auto submit App label Jul 24, 2026
@auto-submit auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Jul 24, 2026
@auto-submit

auto-submit Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

autosubmit label was removed for flutter/flutter/189948, because - The status or check suite Dashboard Checks has failed. Please fix the issues identified (or deflake) before re-applying this label.

@andywolff andywolff added the autosubmit Merge PR when tree becomes green via auto submit App label Jul 27, 2026
@auto-submit
auto-submit Bot added this pull request to the merge queue Jul 27, 2026
Merged via the queue into flutter:master with commit be9b204 Jul 27, 2026
26 checks passed
@flutter-dashboard flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Jul 27, 2026
auto-submit Bot pushed a commit to flutter/packages that referenced this pull request Jul 28, 2026
flutter/flutter@9988960...0f02463

2026-07-28 [email protected] Roll Fuchsia Test Scripts from E8hJ1AfK8CtGtaES0... to 1frGe_KltAJKkeyPg... (flutter/flutter#190134)
2026-07-28 [email protected] iOS: Reject merged-platform-ui-thread=mergeAfterLaunch (flutter/flutter#190051)
2026-07-28 [email protected] iOS: Migrate TaskRunner tests to Swift Testing (flutter/flutter#190055)
2026-07-28 [email protected] Run Mac golden tests on ARM bots (flutter/flutter#189465)
2026-07-28 [email protected] iOS,macOS: Rename Swift test files to end in Tests.swift (flutter/flutter#190063)
2026-07-28 [email protected] Fix hcpp cliprect being behind by 1 frame when scrolling (flutter/flutter#189946)
2026-07-28 [email protected] Roll Fuchsia Linux SDK from vpboK5fPPIoFteqRq... to OZkZC_2CZ_G5rbMIS... (flutter/flutter#190115)
2026-07-27 [email protected] Add Ishaq Hassan to AUTHORS (flutter/flutter#190064)
2026-07-27 [email protected] [wimp] fixes ubo padding size issue (flutter/flutter#189958)
2026-07-27 [email protected] Roll pub packages (flutter/flutter#189872)
2026-07-27 [email protected] Move tool host_cross_arch tests into different shards (flutter/flutter#189470)
2026-07-27 49699333+dependabot[bot]@users.noreply.github.com Bump actions/labeler from 6.2.0 to 7.0.0 in the all-github-actions group (flutter/flutter#190099)
2026-07-27 [email protected] [ios]do not nuke user input path when running uiscene integration test (flutter/flutter#186436)
2026-07-27 [email protected] ci: verify_binaries_pre_codesigned part 2 (flutter/flutter#190078)
2026-07-27 [email protected] Roll Abseil to ff6e8ce3e932 (flutter/flutter#189998)
2026-07-27 [email protected] Android_hardware_smoke_test: clean up golden copy in CI (flutter/flutter#189948)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants