Skip to content

CI: Migrate from tox + micromamba to pixi with pixi-build-cmake - #3912

Merged
MridulS merged 16 commits into
scipp:mainfrom
MridulS:pixi-build-cmake-v2
Aug 14, 2026
Merged

MridulS merged 16 commits into
scipp:mainfrom
MridulS:pixi-build-cmake-v2

Conversation

@MridulS

@MridulS MridulS commented Jun 22, 2026 •

Copy link
Copy Markdown
Member

Replaces the tox + micromamba + requirements/*.txt + .buildconfig/*.yml development and CI tooling with pixi, using the (preview) pixi-build feature with the pixi-build-cmake backend to build the C++ extension as a conda package.

What changes

One manifest (pixi.toml) + one lock file (pixi.lock) replace:

  • tox.ini (all envs/tasks)
  • requirements/*.in / *.txt (pip-compile-multi)
  • .buildconfig/ci-*.yml (conda envs for CI C++ builds)
  • docs/environments/developer.yml

Environments

Environment Purpose
default All features, for bare pixi run <task> convenience
dev Incremental C++ development: manual cmake + Ninja via the existing ci-linux/-macos/-windows presets, PYTHONPATH activation. Toolchain: conda cxx-compiler on Linux (runner g++ too old for scipp's C++23) and macOS; MSVC via vs-shell on Windows (the conda vs2022 activation clashes with it). Own solve-group so pixi install -e dev never triggers a slow full pixi-build rebuild. Builds the C++ test suite (which pixi-build-cmake does not). Deliberately excludes the plot feature: plopp requires scipp, which would install the published conda-forge scipp whose libscipp-* shadow the freshly built dev libraries at runtime (conda compiler LDFLAGS put $CONDA_PREFIX/lib on the rpath) — this exact failure showed up as undefined symbol: scipp::core::madvise_hugepage when #3937 added new symbols.
test, test-py312/313/314 pixi-build-cmake builds and installs scipp automatically; Python matrix via [workspace.build-variants]
test-py314t Free-threaded CPython 3.14. scipp gets a dedicated cp314t build-variant ("3.14.* *_cp314t"). Plotting deps (matplotlib/plopp) are split into a plot feature because they have no cp314t conda-forge builds yet; scipp's own test suite does not import them.
docs, type-check, lint Docs build + doctests, mypy, pre-commit (lint has no default feature)

CI

  • pr_and_main.yml: unified matrix (ubuntu / macos / windows) — C++ tests via the dev env, Python tests via the test env; docs, mypy, and linkcheck on ubuntu only. The Windows leg was validated separately on a unified dev+test experiment run before this PR.
  • [email protected] everywhere with a pinned pixi-version and frozen: true; the formatting job runs pixi lock --check so a stale pixi.lock fails loudly instead of silently installing old pins.
  • Coverage keeps using system g++ (matching system gcov) via an explicit -DCMAKE_CXX_COMPILER, since the dev env's conda cxx-compiler activation would otherwise override CXX.
  • The wheel (cibuildwheel) and conda (conda-build) release paths are unchanged.
  • Nightly-deps testing still installs the frozen envs, then upgrades numpy/scipy/h5py/pandas/xarray to nightlies with pip on top.

Conda packaging test path (second commit)

Real conda releases happen in conda-forge/scipp-feedstock; the in-repo recipe exists only to catch feedstock breakage early. It is converted from conda-build (conda/meta.yaml + setup-miniconda) to a rattler-build v1 recipe (conda/recipe.yaml) built via the pixi package environment against conda-forge-pinning's global pins — so all of CI uses one toolchain. Details:

  • The workspace gains linux-aarch64/osx-64 for the package env only (weekly builds run on native arm-linux and intel-mac runners); every other env stays restricted to the three dev platforms via feature-level platform lists, keeping the lock lean and avoiding unsolvable combinations (e.g. the free-threaded stack on osx-64).
  • The 3.14t variant now selects the free-threaded interpreter explicitly (3.14.* *_cp314t). The old conda-build variant (3.14.* *_cp314 + scalar is_freethreading) silently resolved a regular cp314 python, so the "free-threaded" conda test wasn't actually free-threaded.
  • ipykernel is dropped from the recipe's test requirements (scipp's tests never import it; it has no win-64 cp314t build — surfaced by the now genuinely free-threaded Windows leg). The feedstock may want the same change.
  • weekly.yml matrices run with fail-fast: false so one target's failure doesn't cancel the rest.

Validated by dispatching the weekly workflow on the fork: all 15 conda targets (5 platforms × py3.11/3.14/3.14t) and all 15 cibuildwheel legs green, plus a full local build + embedded package tests (5204 passed) on osx-arm64.

Notes

  • preview = ["pixi-build"] is still required; the backend is pinned to 0.* because the backend version is not captured in pixi.lock.
  • requires-pixi = ">=0.74" guards against older pixi versions with incompatible source-package behavior; keep in sync with pixi-version in CI.
  • cppcheck is pinned to 2.18.3 (same as main used before this PR) — it fixes internalError crashes in cppcheck's whole-program analysis directory.
  • TBB 2023.0.0 carried over from MAINT: TBB bump to 2023.0.0 #3916. [package] version is a deliberately inert 9999.0.0: it only names locally built dev packages (wheels/conda releases derive the real version from git via CMake), and it cannot be 0.0.0 because plopp in the workspace requires scipp >=25.11.0.
  • Validated locally on osx-arm64 with pixi 0.76.0: full pixi lock; pixi-build-cmake builds on both the test (cp311) and test-py314t (cp314t) environments with the full pytest suite green on both (5321 passed each); and pixi run -e dev cpp-test with the conda clang toolchain — 5/5 C++ test suites pass (the runtime symbol shadowing above reproduced and is gone after dropping plot from dev).

Review round 1 (thanks @SimonHeybrock)

  • mypy restored to real coverage: the task now runs python -m mypy -p scipp (checks the installed package, 91 files, hard error if the package is absent — the check cannot silently collapse again) plus a second pass over repo-level python. This surfaced four latent type errors under current mypy/numpy stubs; all four are fixed in source (positional arange call, float64 coercion for np.diag, an annotated wrapper local in curve_fit, one stale ignore removed) — behavior-preserving, full test suite green.
  • import-minimal is a real check again: it now runs in a dedicated minimal environment (scipp + python/numpy only, sharing the default solve group) on all three platforms in CI, restoring the old tox semantics of catching accidental hard imports of optional dependencies.
  • check-release dropped entirely (rather than porting prepare-release): git history shows the RELEASE_PLACEHOLDER machinery was last used for the 23.05 release (2023-05); every release since shipped without it.
  • Release docs are now built from the source tree, not from the freshly published PyPI wheel (the old --skip-pkg-install + sleep 120 dance). The unused manage-versions/target-folder options and the version input were removed — dead since docs/version.py was deleted in c7eb88a.
  • Docs notebook cleanup ported: the published-site size optimisation (delete generated .ipynb except under _sources/) is back, as a cross-platform python one-liner.
  • pybind11 pinned ==3.0.* for wheel builds: pybind11 3.1.0 (2026-08-06) changes overload resolution so python ints reach float64 operator overloads, breaking int-scalar arithmetic (22 wheel-test failures; conda unaffected — conda-forge pins 3.0.x). Adopting 3.1 is follow-up work.
  • ccache honesty: the pixi-build-cmake package build is not ccache-cacheable today. The build sandbox scrubs outer env vars (verified: a full build with CCACHE_DIR pointed at a probe dir left it empty) and denies writes outside its build tree (CCACHE_DIR injected via [package.build.config] env fails with ccache: error: Read-only file system). The earlier launcher env var and comments claiming otherwise are reverted; the ccache-action reorder and 1G cache size stay (they serve the dev build). Making the package build cacheable is follow-up work, likely needing backend support.
  • IPO flag dropped from the package build: check_ipo_supported() fails with the conda compilers (same on main) and lib/cmake/IPO.cmake FORCE-overrides the cache variable, so -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON was silently ignored.

🤖 Generated with Claude Code

@MridulS
MridulS force-pushed the pixi-build-cmake-v2 branch from 46c48e5 to 6e19adf Compare August 6, 2026 08:47
MridulS and others added 5 commits August 6, 2026 13:45
The real conda releases happen in conda-forge/scipp-feedstock; the in-repo
recipe exists only to catch feedstock breakage early. Convert it from
conda-build (meta.yaml + setup-miniconda) to a rattler-build v1 recipe
(conda/recipe.yaml) built from the pixi `package` environment against
conda-forge-pinning's global pins -- one toolchain (pixi) for all of CI.

- The workspace gains linux-aarch64/osx-64 for the `package` env only (weekly
  builds on native arm-linux and intel-mac runners); all other envs stay
  restricted to the three dev platforms via feature-level platform lists.
- The 3.14t variant now selects the free-threaded interpreter explicitly
  (3.14.* *_cp314t); conda-build's variant silently resolved regular cp314.
- ipykernel dropped from the recipe's test requirements: tests never import
  it and it has no win-64 cp314t build.
- weekly.yml matrices no longer fail-fast.

Validated on the fork: all 15 conda targets (5 platforms x py3.11/3.14/3.14t)
plus 15 wheel legs green.

Co-Authored-By: Claude Fable 5 <[email protected]>
Pin pybind11 to the same major version the conda builds use, and drop
matplotlib from the test extra -- the test suite does not import it (it stays
in the `extra` extra for users).

Co-Authored-By: Claude Fable 5 <[email protected]>
…p sh -c

- Shared C++ pins (eigen/boost/pybind11/tbb/gtest) move to
  [workspace.dependencies], inherited with { workspace = true } by the
  package host/run deps and the dev feature -- each version declared once.
- The osx-arm64 build config no longer duplicates the base extra-args: the
  deployment target is set via MACOSX_DEPLOYMENT_TARGET in the platform env,
  which merges with the base config (verified: built extension has minos 14.0).
- docs-clean/static tasks drop their sh -c wrappers; pixi's task shell handles
  the constructs natively, which also makes them work on Windows.

No lockfile change: identical solve, just single-sourced declarations.

Co-Authored-By: Claude Fable 5 <[email protected]>
conda-forge's macosx_deployment_target_osx-arm64 meta-package exports
MACOSX_DEPLOYMENT_TARGET during build-env activation (13.0 on CI), overriding
the backend config env and breaking C++23 std::to_chars (needs >= 13.3). The
-DCMAKE_OSX_DEPLOYMENT_TARGET cache variable is immune; keep the duplicated
extra-args list with a comment explaining why.

Co-Authored-By: Claude Fable 5 <[email protected]>
@MridulS
MridulS marked this pull request as ready for review August 6, 2026 14:58
@MridulS

MridulS commented Aug 6, 2026

Copy link
Copy Markdown
Member Author

I think this is ready now, if you ignore the pixi.lock file this is +659 and -1226

@MridulS
MridulS requested a review from SimonHeybrock August 6, 2026 15:04
MridulS and others added 3 commits August 6, 2026 17:07
The workflow rewrite accidentally downgraded it to v4; main had already
merged the dependabot bump to v6 for this file.

Co-Authored-By: Claude Fable 5 <[email protected]>
conda-forge python does not bundle pip, so the bare `pip install` fell
through PATH to the runner's system pip -- the nightlies never reached the
test environment and the workflow green-lit the regular locked deps. Add pip
to the test feature, invoke it as `python -m pip`, and print the installed
versions so the log shows what was actually tested.

Co-Authored-By: Claude Fable 5 <[email protected]>
Weekly coverage has been failing on main for weeks: the hardcoded g++-11
cannot compile scipp's C++23 (std::views::zip needs GCC >= 13). The default
toolchain on ubuntu-24.04 is g++-13 with a matched gcov, which is all lcov
needs.

Co-Authored-By: Claude Fable 5 <[email protected]>

@SimonHeybrock SimonHeybrock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this is a big migration and the inline comments explaining why each workaround exists (deployment target, plot exclusion from dev, backend pinning) make it far easier to review than it could have been. Most of what follows is about behaviour that quietly disappeared in the port rather than about pixi itself.

I compared the CI logs of this branch (run 31114682031) against a recent main run (30547746691) — details in the inline comments.

Blocking

Two checks stop checking anything: mypy (96 → 5 source files) and import-minimal. See inline.

Docs publishing

Not mentioned in the PR description, but worth being explicit about since it only shows up on a real release:

  • Release docs used to be built against the freshly published PyPI wheel (tox --skip-pkg-install -e docs -- scipp=="$VERSION", after sleep 120 to wait for PyPI). They are now built from source. That is an improvement, but please call it out in the description.
  • Dropping manage-versions and target-folder is correct — docs/version.py was deleted in c7eb88a, so both were already dead — but it is an unrelated cleanup riding along in a pixi PR. Please mention it.
  • docs.yml still declares the version input and env.VERSION, and release.yml still passes version:. All unused now; drop them.

CI performance

setup-pixi runs before hendrikmuhs/ccache-action, and setup-pixi is where pixi-build-cmake compiles scipp: 15:17:38 → 15:38:27, 21 minutes, with no ccache available. The later pixi run -e dev cpp-test build takes 4 minutes off a warm cache. Net job time is ~28 min vs ~25 min on main, so this is not a regression — but moving ccache-action ahead of setup-pixi and wiring CMAKE_CXX_COMPILER_LAUNCHER into [package.build.config] extra-args should reclaim most of that 20 minutes.

Smaller things

  • PIXI_FROZEN is not set for the workflows, so frozen: true only covers the setup-pixi install step; the subsequent pixi run steps can still re-solve and touch the lock. Setting it at workflow env: level would make the guarantee complete.
  • tools/metatoenv.py generates conda environment files from conda/meta.yaml and is orphaned by its deletion. Nothing else references it — delete it too?
  • requirements/asv.txt is gone but asv.conf.json stays. Is anything still running the benchmarks, and does it need a pixi environment?
  • The dev environment pulls in the docs and type-check features, so incremental C++ work installs sphinx/mypy/pythreejs/ipykernel. In CI those environments are installed separately anyway, so dev's copies are never used. Intentional?

Comment thread pixi.toml Outdated
pandas-stubs = "*"

[feature.type-check.tasks]
mypy = { cmd = "python -m mypy . --exclude '^src/' --exclude '^install/'", description = "Run mypy type checking" }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this makes the mypy job a no-op.

main: Success: no issues found in 96 source files.
This branch (run 31114682031, ubuntu leg): Success: no issues found in 5 source files.

Those 5 are benchmarks/*.py plus lib/core/test/generate_arithmetic_parameters.py — no scipp code is type-checked at all.

Two causes compound. The type-check environment gets scipp from pixi-build-cmake into site-packages, which mypy . does not descend into; and --exclude '^install/' removes the one directory the tox env deliberately aimed mypy at (its comment: "Exclude src because we test the installed package, not the source tree").

python -m mypy -p scipp . would probably restore it. Whatever the fix, it would be good to have something that fails loudly if the checked-file count collapses again — this was silent.

Comment thread pixi.toml Outdated
ipywidgets = "*"

[feature.test.tasks]
import-minimal = { cmd = "python -c 'import scipp'", description = "Verify scipp can be imported" }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: import-minimal no longer tests what it existed for, and no longer runs.

In tox it had deps = -r requirements/base.txt, i.e. numpy + psutil only. The point was to catch scipp accidentally growing a hard import of an optional dependency. Here it lives in feature.test, so it runs in an environment with the full scientific stack and can never fail for that reason — and pr_and_main.yml dropped the step anyway.

Either give it a minimal environment (build + py311, no test feature) and keep it in CI, or delete the task and say in the description that we are dropping the check.

Comment thread pixi.toml Outdated
requests = "*"

[feature.docs.tasks]
docs = { cmd = "python -m sphinx -v -b html -d doctrees docs html && python -m sphinx -v -b doctest -d doctrees docs html", description = "Build HTML docs and run doctests" }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tox docs and docs-clean commands both ended with:

find html -type f -name "*.ipynb" -not -path "html/_sources/*" -delete

Neither pixi task does. The rationale is from a346925 (originally in docs/build_and_test.py): "Remove Jupyter notebooks used for documentation build, they are not accessible and create size bloat. However, keep the ones in the _sources folder, as the download buttons link to them."

So it is a deliberate size optimisation for the published site, and the _sources carve-out keeps the download buttons working. Please port it over.

Comment thread pixi.toml Outdated
# Runs twice on failure so auto-fixed files leave the second run green.
# (pixi's task shell handles `||` and subshells natively on all platforms.)
static = { cmd = "pre-commit run -a || (echo && pre-commit run -a)", description = "Run all pre-commit checks" }
check-release = { cmd = "sh -c '! git grep -rl RELEASE_PLACEHOLDER docs/ src/scipp'", description = "Check no release placeholders remain" }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things here:

  1. prepare-release is gone, along with its section in getting-started.rst, while check-release still gates release.yml. That leaves a gate with no tool to satisfy it. Honestly I doubt we have ever used either — in which case the cleaner resolution is to drop check-release too rather than port prepare-release. Either way it should be a decision, not a side effect.

  2. This is the only task still wrapped in sh -c after 421c47b removed the wrappers elsewhere. If that is because the task shell cannot handle the leading !, a short comment saying so would stop someone "fixing" it later and silently breaking the Windows lint environment.

Comment thread pixi.toml Outdated
# Only used as the conda package version for locally built dev packages; wheels
# and release conda packages derive the real version from git via CMake.
# Bump on release (latest tag).
version = "26.7.0"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already stale — 26.8.0 is tagged. A hand-bumped number that nothing enforces and that only affects a local conda package filename will keep drifting. I would rather see an obviously inert 0.0.0 here, with the comment explaining that the real version comes from git, than one that looks like it tracks releases but does not.

Comment thread .github/workflows/pr_and_main.yml Outdated
cache-write: ${{ github.event_name == 'push' && github.ref_name == 'main' }}

- name: ccache
uses: hendrikmuhs/[email protected]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ordering: this runs after setup-pixi, but setup-pixi is where the expensive build happens — pixi-build-cmake compiles scipp during environment installation, 21 minutes in run 31114682031, entirely without ccache. Only the much cheaper pixi run -e dev cpp-test build (4 min) benefits from the cache.

Moving this step above setup-pixi and passing -DCMAKE_CXX_COMPILER_LAUNCHER=ccache through [package.build.config] extra-args should make the big build cacheable too.

create-args: python=${{ matrix.python-version }}
pixi-version: v0.76.0
environments: ${{ contains(matrix.variant.os, 'ubuntu') && 'dev test docs type-check' || 'dev test' }}
frozen: true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

frozen: true only applies to the install setup-pixi performs. The pixi run steps below are not frozen and may re-solve or rewrite pixi.lock. Setting PIXI_FROZEN: true in the workflow-level env: would cover the whole job (same for the other workflows).

Comment thread .gitignore
.pixi/
conda/package
# Generated python-variant selection for the conda-package task
dynamic_var.yaml

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new configure-debug / build-debug tasks write to build-debug/<env> and install-debug/<env>, neither of which is ignored (build/ and install/ are, and cmake-build-* does not match). Worth adding here.

Comment thread cppcheck-suppressions.txt Outdated
// Match any `_deps` directory at any depth so it works regardless of the build
// directory name (e.g. build/, build/dev, build/test). `**` crosses path
// separators; a plain `*` does not under cppcheck >= 2.18.
*:**/_deps/** No newline at end of file

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still no trailing newline (pre-existing, but the file is being touched anyway).

environments: docs
frozen: true
cache: true
- run: pixi run -e docs docs

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the publish branch gone, inputs.version and the env.VERSION above are unused, yet release.yml still passes version:. Drop both.

Also note this changes what release documentation is built from: previously the freshly published PyPI wheel, now the source tree. Better, but it deserves a line in the PR description.

MridulS and others added 5 commits August 7, 2026 11:14
- ccache-action now runs BEFORE setup-pixi and CMAKE_CXX_COMPILER_LAUNCHER=ccache
  is set at job level (non-Windows), so the pixi-build-cmake package build --
  the most expensive compile in each job -- is cacheable; ccache max-size
  raised to 1G for the two builds per job.
- PIXI_FROZEN=true at workflow level in all pixi workflows: setup-pixi's
  frozen option only covers its own install; this freezes the pixi run steps.
- docs/docs-clean tasks re-gain the published-site size optimisation (delete
  generated notebooks except under _sources), as a cross-platform python
  one-liner instead of unix find.
- docs.yml: drop the now-unused version input and env.VERSION; release.yml no
  longer passes it.
- Spell out build/dev, install/dev etc. instead of templating on the
  environment name; only the dev environment has these tasks.
- Delete tools/metatoenv.py (generated env files from the removed
  conda/meta.yaml; orphaned).
- gitignore build-debug/ and install-debug/; trailing newline in
  cppcheck-suppressions.txt.

Co-Authored-By: Claude Fable 5 <[email protected]>
git history shows RELEASE_PLACEHOLDER was last used in the 23.05 release
(2023-05); every release since shipped without it. Remove the check-release
gate rather than porting the equally dead prepare-release counterpart.

Co-Authored-By: Claude Fable 5 <[email protected]>
- import-minimal now runs in a dedicated minimal environment (scipp + the
  default python/numpy feature only, sharing the default solve group so the
  same scipp build is reused) and is back in per-platform CI. In the full test
  environment it could never catch scipp growing a hard import of an optional
  dependency, and CI had dropped the step entirely.
- [package] version becomes an inert 9999.0.0 instead of a hand-tracked copy
  of the latest tag that nothing enforces. It cannot be 0.0.0: plopp in the
  workspace requires scipp >=25.11, and the source package must satisfy that.

Co-Authored-By: Claude Fable 5 <[email protected]>
The mypy task now checks the installed package with -p scipp (91 files; fails
loudly if the package is absent) plus repo-level python separately. Restoring
coverage surfaced four latent errors under current mypy/numpy stubs, fixed
properly: positional arange call (numpy stubs declare start/stop/step
positional-only; runtime-equivalent, verified), concrete float64 array into
np.diag for linear_transform, an explicitly annotated wrapper callable in
curve_fit, and one now-unused type-ignore removed.

Co-Authored-By: Claude Fable 5 <[email protected]>
pybind11 3.1.0 (released 2026-08-06) changes overload resolution so python
ints reach float64 operator overloads, breaking int-scalar arithmetic in
wheels (22 test failures; conda builds are unaffected because conda-forge
pins 3.0.x). Align the wheel build-system pin with pixi.toml and conda-forge;
adopting pybind11 3.1 is follow-up work.

Co-Authored-By: Claude Fable 5 <[email protected]>
@MridulS

MridulS commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

addressed the reviews in the last commits.

@SimonHeybrock SimonHeybrock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — most of this is properly addressed, and the mypy restoration is done the right way: -p scipp checks the installed package (91 files) and fails loudly if it is absent, so the check cannot silently collapse to zero again.

I also verified the four source changes that restoring coverage surfaced, and all four are behaviour-preserving: _normalize_range_args always returns all three keys (None included), and np.arange(4, None, None) is identical to the old keyword form; f is not read after the fn rebind in _curve_fit_chunk; np.diag(np.asarray(..., float64)) only coerces a dtype linear_transforms requires anyway.

Confirmed working: import-minimal (lock shows numpy + scipp only, and it runs on all three platforms), the notebook cleanup including the _sources carve-out — and it is on the publish path, since docs.yml runs the docs task — check-release removed with no leftover references, the docs.yml version input, PIXI_FROZEN in all five workflows, .gitignore, the cppcheck newline, and tools/metatoenv.py.

Blocking: the ccache change does not do anything

The step reordering landed with comments asserting the pixi-build-cmake package build is now cached. It is not. Compile counts against the ccache statistics from the same job:

compiles in ccache -s
pixi-build package (inside setup-pixi) 327 absent
pixi run -e dev cpp-test 464 464 hits / 471 calls

Both post-reorder runs show the same picture (31167406260: 459/471; 31169836817: 464/471), and setup-pixi still takes 16m48s and 22m23s respectively — the same ~21 minutes as before.

The cause is that lib/CMakeLists.txt already sets CMAKE_CXX_COMPILER_LAUNCHER whenever ccache is on PATH, so the new workflow-level env var is redundant; ccache is invoked during the package build, but the rattler-build sandbox does not see ccache-action's config ($HOME/.config/ccache/ccache.conf pointing cache_dir at $GITHUB_WORKSPACE/.ccache), so it writes to a different cache that is never persisted.

Either fix it properly — export CCACHE_DIR into the build environment via [package.build.config] env and confirm the call count rises to ~800 — or revert the env var and the comments and open a follow-up. A comment claiming a benefit that measurably is not there is worse than leaving the ordering alone.

The description is now wrong, not just incomplete

It still says "[package] version bumped to match the latest release (26.7.0)", which contradicts the 9999.0.0 in the manifest. Still missing, from the previous round: release docs are now built from source rather than from the freshly published PyPI wheel, and the manage-versions / target-folder removal. Please also add the check-release removal, the new import-minimal environment, and the pybind11 pin.

pybind11 ==3.0.*

The reasoning is convincing and pinning is the right short-term call, but it is a product-level constraint arriving in a CI-migration PR, and no leg in this PR's CI exercises the wheel build that motivated it. It also makes pybind11 = "3.0.*" in [workspace.dependencies] load-bearing: the conda path breaks the same way as soon as conda-forge's global pinning moves to 3.1. Could you open a follow-up issue for adopting pybind11 3.1 and reference it from the comment?

Still open from the last round

  • requirements/asv.txt is gone but asv.conf.json stays. Is anything still running the benchmarks, and does it need a pixi environment?
  • dev pulling in the docs and type-check features, whose copies are never used in CI. Intentional?

Minor: IPO is off in the package build

The pixi build logs IPO is not supported: CMake doesn't support IPO for current C compiler on all three platforms, and lib/cmake/IPO.cmake responds with set(CMAKE_INTERPROCEDURAL_OPTIMIZATION FALSE CACHE BOOL ... FORCE) — the FORCE overrides the -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON passed through [package.build.config] extra-args. So that argument is silently negated and the package is built without IPO.

The check_ipo_supported() failure itself is pre-existing (main logs the same warning), and the pixi-built package is CI-only, so nothing shipped changes. But the extra-args entry currently asserts something that cannot hold, so it should either be dropped or made to work.

@SimonHeybrock

Copy link
Copy Markdown
Member

Followed up on the IPO point from my review: #3942. It turns out IPO has been off in every build — wheels and conda included — since #3777 landed CPM in October, because check_ipo_supported() is called without LANGUAGES and llnl-units enables C. Nothing for you to fix here, but it does mean the -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON in [package.build.config] extra-args cannot take effect today. Simplest is to drop it and let #3942 turn IPO back on everywhere at once; keeping it is fine too if you add a reference to the issue so the next reader knows it is inert.

The pixi-build-cmake package build cannot use the persisted ccache today, and
the workflow env var claiming to enable that was measurably doing nothing
(package-build compiles never appeared in ccache stats). Verified locally why
both routes fail: the build sandbox scrubs outer environment variables (a full
build with CCACHE_DIR pointed at a probe directory left it empty), and
injecting CCACHE_DIR via [package.build.config] env fails the build with
'ccache: error: Read-only file system' (the sandbox denies writes outside its
build tree). Revert the env var and the comments; keep the ccache-action
reorder and the 1G cache size, which serve the dev build. Making the package
build cacheable is follow-up work, likely needing backend support.

Also drop -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON from the package build:
check_ipo_supported() fails with the conda compilers (as on main) and
lib/cmake/IPO.cmake FORCE-overrides the cache variable to FALSE, so the flag
was silently ignored.

Co-Authored-By: Claude Fable 5 <[email protected]>

@SimonHeybrock SimonHeybrock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@SimonHeybrock SimonHeybrock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@MridulS
MridulS merged commit be5e045 into scipp:main Aug 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants