Skip to content

Release fails to codesign new binaries #188075

Closed

Description

@vashworth

https://ci.chromium.org/ui/p/dart-internal/builders/flutter/Mac%20Production%20Engine%20Drone/28563/overview

[SEVERE] codesign: The binary file analysis_server_aot.dart.snapshot is causing an issue. 
This file is located at dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot in the flutter engine artifact.
[SEVERE] codesign: The system has detected a binary file at dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot. But it is not in the codesigning configuration files you provided. If this is a new engine artifact, please add it to one of the codesigning config files.
Unhandled exception:
Exception: Codesign test failed.

We compared binary files in engine artifacts with those listed in
* entitlements.txt
* without_entitlements.txt
* unsigned_binaries.txt
and the binary files do not match.

These are the configuration files encoded in engine artifact zip that detail
the code-signing requirements of each of the binaries in the archive.
Either an unexpected binary was listed in these files, or one of the expected
binaries listed in these files was not found in the archive.

This usually happens during an engine roll.

If this is a valid change, then the BUILD.gn or the codesigning configuration
files need to be changed. Binaries that will run on a macOS host require
entitlements, and binaries that run on an iOS device must NOT have entitlements.
For example, if this is a new binary that runs on macOS host, add it
to entitlements.txt file inside the zip artifact produced by BUILD.gn.
If this is a new binary that needs to be run on iOS device, add it to
without_entitlements.txt. If there are obsolete binaries in entitlements
configuration files, please delete or update these file paths accordingly.
[SEVERE] codesign: The binary file frontend_server_aot.dart.snapshot is causing an issue. 
This file is located at frontend_server_aot.dart.snapshot in the flutter engine artifact.
[SEVERE] codesign: The system has detected a binary file at frontend_server_aot.dart.snapshot. But it is not in the codesigning configuration files you provided. If this is a new engine artifact, please add it to one of the codesigning config files.
...

I believe this is caused by #186754, which added new binaries.

However, the entitlement file was not updated.

I think the dart-sdk ones need to be added here:

generated_file("dart_sdk_entitlement_config") {
outputs = [ "$target_gen_dir/dart_sdk_entitlements.txt" ]
# Dart SDK is a prebuilt archive. The rule for building
# Dart SDK is https://github.com/flutter/flutter/blob/main/engine/src/flutter/BUILD.gn#L61
contents = [
"dart-sdk/bin/dart",
"dart-sdk/bin/dartaotruntime",
"dart-sdk/bin/dartvm",
"dart-sdk/bin/utils/gen_snapshot",
"dart-sdk/bin/utils/wasm-opt",
]
deps = []
}

The frontend_server_aot.dart.snapshot, may need to do something like this:

metadata = {
entitlement_file_path = [ "flutter_tester" ]
}

I'm not 100% sure on how this all works

Activity

  1. vashworth commented on Jun 16, 2026

    @vashworth
    ContributorAuthor

    This is blocking the release: #187787

  2. added
    P1High-priority issues at the top of the work list
    on Jun 16, 2026
  3. vashworth commented on Jun 16, 2026

    @vashworth
    ContributorAuthor

    @jason-simmons Can you take a look? Looks like you added these artifacts?

  4. added
    P0Critical issues such as a build break or regression
    and removed
    P1High-priority issues at the top of the work list
    on Jun 16, 2026
  5. jason-simmons commented on Jun 16, 2026

    @jason-simmons
    Member

    Dart changed the format of AOT snapshots on macOS so that these files are now recognized as binaries.

    I landed a change to a code signing test that updates the list of binaries to match this (1924b79). It looks like that change was cherry-picked into flutter-3.45-candidate.0.

    But I'm not familiar with code signing process used to build releases.

    The code signing error log says:

    We compared binary files in engine artifacts with those listed in
    * entitlements.txt
    * without_entitlements.txt
    * unsigned_binaries.txt
    and the binary files do not match.
    

    Where does the entitlements.txt file come from?
    Do the binaries that were added to the binariesWithEntitlements list in 1924b79 also need to be added there?

  6. jason-simmons commented on Jun 16, 2026

    @jason-simmons
    Member

    The binaries added in 1924b79 may also need to be added to the dart_sdk_entitlement_config list in https://github.com/flutter/flutter/blob/master/engine/src/flutter/build/archives/BUILD.gn

    That list generates the entitlements.txt file that is added to the zip_archives/dart-sdk-darwin-arm64.zip artifact output by the Flutter engine build.

  7. added
    P1High-priority issues at the top of the work list
    and removed
    P0Critical issues such as a build break or regression
    on Jun 17, 2026
  8. LongCatIsLooong commented on Jun 17, 2026

    @LongCatIsLooong
    Contributor

    https://ci.chromium.org/ui/p/dart-internal/builders/flutter/Linux%20flutter_release_builder/1350/overview codesign check failure on the release branch seems resolved. Will apply a similar fix on master.

  9. LongCatIsLooong commented on Jun 22, 2026

    @LongCatIsLooong
    Contributor

    Turns out the issue isn't really fixed. There is a new entitlement-related failure blocking the 3.46 beta release: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64%20verify_binaries_codesigned/712/overview

    Error message
    Verifying the code signature of /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot
    Verifying entitlements of /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot" does not have expected entitlement com.apple.security.cs.allow-jit.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot" does not have expected entitlement com.apple.security.cs.allow-unsigned-executable-memory.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot" does not have expected entitlement com.apple.security.cs.allow-dyld-environment-variables.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot" does not have expected entitlement com.apple.security.network.client.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot" does not have expected entitlement com.apple.security.network.server.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot" does not have expected entitlement com.apple.security.cs.disable-library-validation.
    Verifying the code signature of /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot
    Verifying entitlements of /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot" does not have expected entitlement com.apple.security.cs.allow-jit.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot" does not have expected entitlement com.apple.security.cs.allow-unsigned-executable-memory.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot" does not have expected entitlement com.apple.security.cs.allow-dyld-environment-variables.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot" does not have expected entitlement com.apple.security.network.client.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot" does not have expected entitlement com.apple.security.network.server.
    File "/Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot" does not have expected entitlement com.apple.security.cs.disable-library-validation.
    .......
    ╔═╡ERROR #1╞════════════════════════════════════════════════════════════════════
    ║ UNEXPECTED ERROR!
    ║ Exception: Test failed because files found with the wrong entitlements:
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/artifacts/engine/darwin-x64/frontend_server_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/dartdevc_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/dart_tooling_daemon_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/kernel_worker_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/frontend_server_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/dds_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/dart2wasm_product.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/dart2bytecode.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/dart2js_aot.dart.snapshot
    ║ /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/snapshots/dartdev_aot.dart.snapshot
    ║ #0      verifySignatures (file:///Volumes/Work/s/w/ir/x/w/flutter/dev/bots/suite_runners/run_verify_binaries_codesigned_tests.dart:343:5)
    ║ <asynchronous suspension>
    ║ #1      verifyCodesignedTestRunner (file:///Volumes/Work/s/w/ir/x/w/flutter/dev/bots/suite_runners/run_verify_binaries_codesigned_tests.dart:25:3)
    ║ <asynchronous suspension>
    ║ #2      _runFromList (file:///Volumes/Work/s/w/ir/x/w/flutter/dev/bots/utils.dart:727:7)
    ║ <asynchronous suspension>
    ║ #3      main (file:///Volumes/Work/s/w/ir/x/w/flutter/dev/bots/test.dart:124:5)
    ║ <asynchronous suspension>
    ║ 
    ║ The test.dart script should be corrected to catch this error and call foundError().
    ║ Some tests are likely to have been skipped.
    ╚═══════════════════════════════════════════════════════════════════════════════
    

    However in this task, those snapshots should have been signed and uploaded:

    [INFO] codesign: Signing file at path /Volumes/Work/s/w/ir/x/t/conductor_codesignRTIt5R/single_artifact/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot
    [INFO] codesign: The virtual entitlement path associated with file is dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot
    [INFO] codesign: The decision to sign with entitlement is true
    [INFO] codesign: Executing: /usr/bin/codesign --keychain build.keychain -f -s Developer ID Application: FLUTTER.IO LLC /Volumes/Work/s/w/ir/x/t/conductor_codesignRTIt5R/single_artifact/dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot --timestamp --options=runtime --entitlements /Volumes/Work/s/w/ir/x/t/conductor_codesignRTIt5R/Entitlements.plist
    
    [INFO] codesign: Child file of directory snapshots is analysis_server_aot.dart.snapshot
    

    I downloaded the archive from https://storage.googleapis.com/flutter_infra_release/flutter/ca13a7eb9727f9182ac0d41604d39307ea9a64c7/dart-sdk-darwin-x64.zip and ran codesign -d --entitlements , executable binaries such as wasm-opt have the correct entitlements, but the aot snapshots don't have the entitlements (but they are signed).

    Could it be that the codesign actually doesn't attach the entitlements to the snapshot binaries even if told to (because it's not necessary)?

  10. 3 remaining items

  11. gaaclarke commented on Jun 22, 2026

    @gaaclarke
    Member

    Notes from triage: My understanding is that @LongCatIsLooong is looking into this right now. Is that correct?

  12. LongCatIsLooong commented on Jun 22, 2026

    @LongCatIsLooong
    Contributor

    Notes from triage: My understanding is that [@LongCatIsLooong](https://github.com/LongCatIsLooong) is looking into this right now. Is that correct?

    I worked on this last week as the release engineer. I'm planning to test out my theory (the codesign utility doesn't add entitlements to dylibs) on the obsolete 3.45 branch (is there a better way to test this?), if that doesn't work then someone more knowledgable about code signing should take over.

  13. vashworth commented on Jun 22, 2026

    @vashworth
    ContributorAuthor

    We confirmed that the snapshots, even when codesigned with entitlements, are not actually assigned entitlements and therefore the check fails.

    Unfortunately, this is not straightforward fix and will require a bunch of gn changes.

    Here's what needs to be done (from my limited understanding):

    1. The snapshots need to be moved to binariesWithoutEntitlements

    'artifacts/engine/darwin-x64/frontend_server_aot.dart.snapshot',

    'dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot',
    'dart-sdk/bin/snapshots/dart2bytecode.dart.snapshot',
    'dart-sdk/bin/snapshots/dart2js_aot.dart.snapshot',
    'dart-sdk/bin/snapshots/dart2wasm_product.snapshot',
    'dart-sdk/bin/snapshots/dart_tooling_daemon_aot.dart.snapshot',
    'dart-sdk/bin/snapshots/dartdev_aot.dart.snapshot',
    'dart-sdk/bin/snapshots/dartdevc_aot.dart.snapshot',
    'dart-sdk/bin/snapshots/dds_aot.dart.snapshot',
    'dart-sdk/bin/snapshots/frontend_server_aot.dart.snapshot',
    'dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot',
    'dart-sdk/bin/snapshots/kernel_worker_aot.dart.snapshot',

    List<String> binariesWithoutEntitlements(String flutterRoot) {

    1. Replace entitlement_file_path with without_entitlement_file_path:

    entitlement_file_path = [ "frontend_server_aot.dart.snapshot" ]

    1. Create a new generated file artifacts_without_entitlement_config and add to artifacts zip_bundle

    Similar to artifacts_entitlement_config:

    generated_file("artifacts_entitlement_config") {
    outputs = [ "$target_gen_dir/entitlements.txt" ]
    data_keys = [ "entitlement_file_path" ]
    # You may also need to update
    # dev/bots/suite_runners/run_verify_binaries_codesigned_tests.dart
    # when adding/removing binaries from the entitlement config.
    deps = [ "//flutter/lib/snapshot:generate_snapshot_bin" ]
    if (flutter_runtime_mode == "debug") {
    deps += [
    "//flutter/flutter_frontend_server:frontend_server",
    "//flutter/impeller/compiler:impellerc",
    "//flutter/impeller/tessellator:tessellator_shared",
    "//flutter/shell/testing",
    "//flutter/tools/path_ops",
    ]
    }
    }

    Probably something like this:

    generated_file("artifacts_without_entitlement_config") {
      outputs = [ "$target_gen_dir/without_entitlements.txt" ]
    
      data_keys = [ "without_entitlement_file_path" ]
    
      # You may also need to update
      # dev/bots/suite_runners/run_verify_binaries_codesigned_tests.dart
      # when adding/removing binaries from the without_entitlements config.
      if (flutter_runtime_mode == "debug") {
        deps += [
          "//flutter/flutter_frontend_server:frontend_server",
        ]
      }
    }

    And added here:

    if (host_os == "mac") {
    deps += [ ":artifacts_entitlement_config" ]
    files += [
    {
    source = "$target_gen_dir/entitlements.txt"
    destination = "entitlements.txt"
    },
    ]

    1. Create a new generated file dart_sdk_without_entitlement_config and add to dart_sdk_archive zip_bundle and dart_sdk_archive zip_bundle_from_file.

    "dart-sdk/bin/snapshots/analysis_server_aot.dart.snapshot",
    "dart-sdk/bin/snapshots/dart2bytecode.dart.snapshot",
    "dart-sdk/bin/snapshots/dart2js_aot.dart.snapshot",
    "dart-sdk/bin/snapshots/dart2wasm_product.snapshot",
    "dart-sdk/bin/snapshots/dart_tooling_daemon_aot.dart.snapshot",
    "dart-sdk/bin/snapshots/dartdev_aot.dart.snapshot",
    "dart-sdk/bin/snapshots/dartdevc_aot.dart.snapshot",
    "dart-sdk/bin/snapshots/dds_aot.dart.snapshot",
    "dart-sdk/bin/snapshots/frontend_server_aot.dart.snapshot",
    "dart-sdk/bin/snapshots/gen_kernel_aot.dart.snapshot",
    "dart-sdk/bin/snapshots/kernel_worker_aot.dart.snapshot",

    zip_bundle("dart_sdk_archive") {
    deps = [ ":dart_sdk_entitlement_config" ]
    if (dart_dynamic_modules) {
    output = "dart-sdk-$full_target_platform_name-ddm.zip"
    } else {
    output = "dart-sdk-$full_target_platform_name.zip"
    }
    files = [
    {
    source = prebuilt_dart_sdk
    destination = "dart-sdk"
    },
    ]
    if (is_mac) {
    # Mac artifacts sometimes use mac and sometimes darwin. Standardizing the
    # names will require changes in the list of artifacts the tool is downloading.
    output = "dart-sdk-darwin-$target_cpu.zip"
    files += [
    {
    source = "$target_gen_dir/dart_sdk_entitlements.txt"
    destination = "entitlements.txt"
    },
    ]
    }
    }
    }

    zip_bundle_from_file("dart_sdk_archive") {
    deps = [ "$dart_src:create_sdk" ]
    if (dart_dynamic_modules) {
    output = "dart-sdk-$full_target_platform_name-ddm.zip"
    } else {
    output = "dart-sdk-$full_target_platform_name.zip"
    }
    files = [
    {
    source = rebase_path("$root_build_dir/dart-sdk")
    destination = "dart-sdk"
    },
    ]
    if (is_mac) {
    output = "dart-sdk-darwin-$target_cpu.zip"
    deps += [ ":dart_sdk_entitlement_config" ]
    files += [
    {
    source = "$target_gen_dir/dart_sdk_entitlements.txt"
    destination = "entitlements.txt"
    },
    ]
    }
    }

  14. gaaclarke commented on Jun 22, 2026

    @gaaclarke
    Member

    I worked on this last week as the release engineer. I'm planning to test out my theory (the codesign utility doesn't add entitlements to dylibs) on the obsolete 3.45 branch (is there a better way to test this?), if that doesn't work then someone more knowledgable about code signing should take over.

    Okay, I don't want this to get lost. @LongCatIsLooong why don't you try your experiment and report back here. @vashworth if that doesn't pan out we should figure out how to get this addressed and make sure it gets to the attention of the right people.

  15. LongCatIsLooong commented on Jun 24, 2026

    @LongCatIsLooong
    Contributor
  16. github-actions commented on Jul 14, 2026

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  17. locked as resolved and limited conversation to collaborators on Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P0Critical issues such as a build break or regressionteam-engineOwned by Engine teamtriaged-engineTriaged by Engine team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions