Skip to content

linux_fuchsia builder is failing often due to error in "Upload debug symbols to CIPD" #187806

Description

@jason-simmons

The majority of runs of the linux_fuchsia builder have been failing over the past day. The error occurs in the Upload debug symbols to CIPD step:

Directory: /b/s/w/ir/x/w/rc is not empty, deleting it.
CIPD package flutter/fuchsia-debug-symbols-arm64 tag git_revision:HEAD already exists!
Enumerating files to zip...
Error: dir "3": scanning input directory: lstat /b/s/w/ir/x/w/rc/3: no such file or directory.
Failed 1 times.
Error was: Command '['cipd', 'pkg-build', '-pkg-def', '/b/s/w/ir/x/w/rc/debug_symbols.cipd.yaml', '-out', '/b/s/w/ir/x/w/rc/fuchsia-debug-symbols-arm64.cipd']' returned non-zero exit status 1.
Enumerating files to zip...
Error: dir "3": scanning input directory: lstat /b/s/w/ir/x/w/rc/3: no such file or directory.
Failed 2 times.
Error was: Command '['cipd', 'pkg-build', '-pkg-def', '/b/s/w/ir/x/w/rc/debug_symbols.cipd.yaml', '-out', '/b/s/w/ir/x/w/rc/fuchsia-debug-symbols-arm64.cipd']' returned non-zero exit status 1.
Enumerating files to zip...
Error: dir "3": scanning input directory: lstat /b/s/w/ir/x/w/rc/3: no such file or directory.
Failed 3 times.
Error was: Command '['cipd', 'pkg-build', '-pkg-def', '/b/s/w/ir/x/w/rc/debug_symbols.cipd.yaml', '-out', '/b/s/w/ir/x/w/rc/fuchsia-debug-symbols-arm64.cipd']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/b/s/w/ir/cache/builder/engine/src/flutter/tools/fuchsia/merge_and_upload_debug_symbols.py", line 260, in <module>
    sys.exit(main())
             ^^^^^^
  File "/b/s/w/ir/cache/builder/engine/src/flutter/tools/fuchsia/merge_and_upload_debug_symbols.py", line 255, in main
    ProcessCIPDPackage(should_upload, cipd_def, engine_version, content_hash, out_dir, arch)
  File "/b/s/w/ir/cache/builder/engine/src/flutter/tools/fuchsia/merge_and_upload_debug_symbols.py", line 111, in ProcessCIPDPackage
    RunCIPDCommandWithRetries(command, _packaging_dir)
  File "/b/s/w/ir/cache/builder/engine/src/flutter/tools/fuchsia/merge_and_upload_debug_symbols.py", line 146, in RunCIPDCommandWithRetries
    subprocess.check_call(command, cwd=working_dir)
  File "/b/s/w/ir/cipd_bin_packages/cpython3/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cipd', 'pkg-build', '-pkg-def', '/b/s/w/ir/x/w/rc/debug_symbols.cipd.yaml', '-out', '/b/s/w/ir/x/w/rc/fuchsia-debug-symbols-arm64.cipd']' returned non-zero exit status 1.

Examples:
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_fuchsia/54800/overview
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_fuchsia/54799/overview
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_fuchsia/54796/overview

Activity

  1. jason-simmons commented on Jun 10, 2026

    @jason-simmons
    MemberAuthor

    This happened because the merge_and_upload_debug_symbols.py script was not generating the proper YAML file when building a CIPD package of the Fuchsia debug symbols.

    merge_and_upload_debug_symbols.py is creating an archive of debug symbols taken from the .build-id directory in each Fuchsia engine build output directory.

    .build-id contains a collection of binaries indexed by each binary's hash. For each hash, .build-id will contain a subdirectory whose name is the first two hex digits of the hash.

    merge_and_upload_debug_symbols.py writes a YAML file that contains entries for each build_id subdirectory.

    For example:

      - dir: 7d
      - dir: cb
      - dir: 6c
      - dir: 01
    

    If the directory name contains only digits starting with a zero such as dir: 01, then YAML interprets the value as the integer 1. That does not match the actual directory name of 01, and the CIPD package build fails.

    The YAML file should instead quote the directory name as a string.

    I don't know why this problem only started happening recently though. I'm wondering if something changed in the format of the files that are output by the Fuchsia build.

  2. jason-simmons commented on Jun 11, 2026

    @jason-simmons
    MemberAuthor

    This should be fixed now. Pending PRs may need to be rebased to include 312bf19

  3. github-actions commented on Jul 14, 2026

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  4. locked as resolved and limited conversation to collaborators on Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    team-infraOwned by Infrastructure team

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions