[Tooling] Add google owned cache for dependencies as an option in ci by reidbaker · Pull Request #4567 · flutter/packages

reidbaker · 2023-07-26T15:49:34Z

Proof of concept of a package using artifact hub.
Artifact hub is a google owned and managed cache of google/maven dependencies.
Using this cache should decrease the number of flakes related to downloading dependencies along with future benefits of licence analysis and security alerts.

Read more at go/artifact-hub#maven
flutter/flutter/issues/120119

Next steps:
Enable a new env variable on CI servers. https://flutter-review.googlesource.com/c/recipes/+/48260 cl/551888350
Merge this PR.
Apply this change to all packages
Add enforcement as part of the gradle check.

Developers have a new dependency for buildscript but it is publicly available, so I added a next changelog.

Pre-launch Checklist

Requried to be an authenticated googler `gcloud auth application-default login` Useful links https://g3doc.corp.google.com/company/teams/artifact-hub/index.md?cl=head#maven https://github.com/GoogleCloudPlatform/artifact-registry-maven-tools/blob/master/README.md Useful command ``` gcloud artifacts print-settings mvn \ --project=artifact-foundry-prod \ --repository=maven-3p \ --location=us ```

…epository addition

flutter-dashboard · 2023-07-26T15:49:37Z

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

ghost

LGTM!

ghost · 2023-07-28T14:32:48Z

@@ -0,0 +1,16 @@
+# Gradle Structure


Having seen the details here, I think a wiki page referenced from https://github.com/flutter/flutter/wiki/Understanding-Packages-tests#external-build-dependencies and maybe https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure would probably be better than an md file.

ok in that case consider the md files content for review and I will remove and update the wiki before merging.

ghost · 2023-07-28T14:32:57Z

@@ -0,0 +1,16 @@
+# Gradle Structure
+
+`package/example/android/settings.gradle` imports the flutter tooling, includes the app directory, and configures GoogleCloudPlatform/artifact-registry-maven-tools for use in ci.


ghost · 2023-07-28T14:33:19Z

+
+`package/example/android/settings.gradle` imports the flutter tooling, includes the app directory, and configures GoogleCloudPlatform/artifact-registry-maven-tools for use in ci.
+
+This repo has a GCP instance that mirrors dependencies available from `google()` and `mavenCentral()` used by ci (or googlers). This gives us redundant uptime for dependency availability.


Nits: CI, Googlers

ghost · 2023-07-28T14:33:45Z

+
+Using this cache is not intended or avaiable for contributors outside of CI. We protect that execution with an environment variable `ARTIFACT_HUB_REPOSITORY` to ensure that by default users do not see rejected cloud credentials or errors in builds.
+
+Googlers can debug locally by setting `ARTIFACT_HUB_REPOSITORY` to the valid artifact hub value and authenticating with GCP. To authenticate run `gcloud auth application-default login`. To find artifact hub url use `<url>` section of go/artifact-hub#maven or inspect the value on CI servers. CI uses a service account for billing. That is defined in go/artifact-hub-service-account (googler access only).


Nit: Googler access

ghost · 2023-07-28T14:34:38Z

@@ -1,3 +1,7 @@
+## NEXT
+
+* Updates android example to be able to use custom use artifactregistery.


Since nobody outside the Flutter team can use this, let's not put it in the CHANGELOG.

They can if they setup a artifactrepository then setup the env variables on their local machine (I just updated the file to reflect this). I removed the google specific access parts to configuration. Also mechanically there is a new dependency and the changelog thing scolded me.

I can remove this if you want even though there is technically a developer facing change that should not impact the.

removed and replaced with https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure#gradle-structure

ghost · 2023-07-28T14:36:13Z

Overriding changelog check: see #4567 (comment)

ghost · 2023-07-28T15:07:32Z

Approved by artifact hub team via email by sfelix@

Replaced with https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure#gradle-structure

ghost · 2023-08-01T14:39:37Z

Confirmed the environment variable was set correctly https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8773957271362564241/+/u/Run_package_tests/download_Dart_and_Android_deps/execution_details and confirmed the artifact hub variable was used
Using artifact hub
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8773957271362564241/+/u/Run_package_tests/download_Dart_and_Android_deps/stdout

ghost · 2023-08-01T16:14:55Z

Confirmed the environment variable was set correctly https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8773957271362564241/+/u/Run_package_tests/download_Dart_and_Android_deps/execution_details and confirmed the artifact hub variable was used Using artifact hub https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8773957271362564241/+/u/Run_package_tests/download_Dart_and_Android_deps/stdout

https://critique.corp.google.com/cl/552818879 had not landed. Links above show our configuration is correct and that if we lose access to artifact hub our builds will not go red. I do not know of a way to verify we are getting the dependencies from the artifact hub server without ssh access to the machine while they are being downloaded.

In a future pr I could make ci only use the artifact hub instance but I am not confident enough yet to make that call.

reidbaker added 2 commits July 25, 2023 18:18

Update documentation, split out plugin apply into setting, simplify r…

befa254

…epository addition

reidbaker requested a review from camsim99 as a code owner July 26, 2023 15:49

github-actions Bot added p: camera platform-android labels Jul 26, 2023

reidbaker changed the title y [Tooling] Add google owned cache for dependencies as an option in ci Jul 26, 2023

reidbaker requested review from godofredoc, johnmccutchan and stuartmorgan-g July 26, 2023 15:55

johnmccutchan approved these changes Jul 26, 2023

View reviewed changes

johnmccutchan reviewed Jul 26, 2023

View reviewed changes

Comment thread packages/camera/camera_android_camerax/example/android/build.gradle Outdated

Update the env variable to be more specific

73aea86

gmackall reviewed Jul 26, 2023

View reviewed changes

Comment thread GRADLE_STRUCTURE.md Outdated

Comment thread GRADLE_STRUCTURE.md Outdated

Comment thread packages/camera/camera_android_camerax/example/android/build.gradle Outdated

Speling and phrasing updates, make key a variable

a2e87fb

gmackall approved these changes Jul 26, 2023

View reviewed changes

gmackall reviewed Jul 26, 2023

View reviewed changes

Comment thread GRADLE_STRUCTURE.md Outdated

Reid Baker added 4 commits July 26, 2023 16:02

formatting

474f547

Merge branch 'main' into i120119-artifact-hub-poc

4747b49

Update changelog

797e066

Merge branch 'main' into i120119-artifact-hub-poc

f249971

stuartmorgan-g reviewed Jul 28, 2023

View reviewed changes

stuartmorgan-g added the override: no changelog needed Override the check requiring CHANGELOG updates for most changes label Jul 28, 2023

Reid Baker added 4 commits July 28, 2023 10:58

Update GRADLE_STRUCTURE.md to use CI instead of ci

81b0406

Update GRADLE_STRUCTURE.md

a996c06

Merge branch 'main' into i120119-artifact-hub-poc

2adfc43

Update CHANGELOG.md to not have comment

3ce0495

Delete GRADLE_STRUCTURE.md

6fc5753

Replaced with https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure#gradle-structure

stuartmorgan-g approved these changes Jul 31, 2023

View reviewed changes

Merge branch 'main' into i120119-artifact-hub-poc

77266fd

reidbaker added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 1, 2023

Merge branch 'main' into i120119-artifact-hub-poc

23e9840

auto-submit Bot merged commit 11b79b5 into flutter:main Aug 2, 2023

reidbaker deleted the i120119-artifact-hub-poc branch August 3, 2023 14:15

engine-flutter-autoroll mentioned this pull request Aug 3, 2023

Roll Packages from 4e4961a24b3d to 9323e33ed9d8 (8 revisions) flutter/flutter#131852

Closed

engine-flutter-autoroll mentioned this pull request Aug 3, 2023

Roll Packages from 4e4961a24b3d to 94ec57fe2583 (9 revisions) flutter/flutter#131855

Closed

engine-flutter-autoroll mentioned this pull request Aug 3, 2023

Roll Packages from 4e4961a24b3d to d00c1f9eb01c (10 revisions) flutter/flutter#131866

Merged

reidbaker mentioned this pull request Aug 4, 2023

[All] Expand artifact hub to all plugins #4645

Merged

10 of 11 tasks

gmackall mentioned this pull request Oct 12, 2023

Mac module_custom_host_app_name_test is 2.02% flaky flutter/flutter#136327

Closed

jmagman mentioned this pull request Jan 7, 2025

Android engine test flaking on failure to fetch dependencies flutter/flutter#120119

Open

		@@ -0,0 +1,16 @@
		# Gradle Structure

		`package/example/android/settings.gradle` imports the flutter tooling, includes the app directory, and configures GoogleCloudPlatform/artifact-registry-maven-tools for use in ci.


		`package/example/android/settings.gradle` imports the flutter tooling, includes the app directory, and configures GoogleCloudPlatform/artifact-registry-maven-tools for use in ci.

		This repo has a GCP instance that mirrors dependencies available from `google()` and `mavenCentral()` used by ci (or googlers). This gives us redundant uptime for dependency availability.


		Using this cache is not intended or avaiable for contributors outside of CI. We protect that execution with an environment variable `ARTIFACT_HUB_REPOSITORY` to ensure that by default users do not see rejected cloud credentials or errors in builds.

		Googlers can debug locally by setting `ARTIFACT_HUB_REPOSITORY` to the valid artifact hub value and authenticating with GCP. To authenticate run `gcloud auth application-default login`. To find artifact hub url use `<url>` section of go/artifact-hub#maven or inspect the value on CI servers. CI uses a service account for billing. That is defined in go/artifact-hub-service-account (googler access only).

		@@ -1,3 +1,7 @@
		## NEXT

		* Updates android example to be able to use custom use artifactregistery.

Conversation

reidbaker commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

flutter-dashboard Bot commented Jul 26, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

ghost commented Jul 28, 2023

Uh oh!

ghost commented Jul 28, 2023

Uh oh!

ghost commented Aug 1, 2023

Uh oh!

ghost commented Aug 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

reidbaker commented Jul 26, 2023 •

edited

Loading