Update spack-stack-dev from spack develop 2026/05/20 - #577

Fixes an issue that has bothered me for years. If you do

```
spack buildcache push --base-image foo --<tab>
```

you get nothing, because Spack looks for a completion function

```
_spack_buildcache_push_foo
```

cause it doesn't know that `foo` is a value.

This fixes that by keeping track of the last function that exists and
use that for tab completion instead, which in this case is

```
_spack_buildcache_push
```

Signed-off-by: Harmen Stoppels <[email protected]>

The failure message referenced s.package.log_path, a symlink that may
not exist if the build fails before the stage is created. Now the parent
creates the log file via tempfile.mkstemp, passes the path to the child,
and the failure message always references a real, existing file.

Signed-off-by: Harmen Stoppels <[email protected]>

It probably doesn't matter much for performance,
but the dimension is reduced like:

O(Package x Virtual) -> O(Virtual)

Also, integrity constraint should help the USC strategy
to learn clauses faster.

Signed-off-by: Massimiliano Culpo <[email protected]>

Signed-off-by: Harmen Stoppels <[email protected]>

This makes the installer a bit snappier in the UI, and reduces the latency 
for short-running installs (e.g. python packages and installs from build cache).

The idea is to save installed specs to the database only if in the last
5 seconds no other specs finished to build.

Builds of parents are now started before the child is stored in the database. 
This reduces latency significantly; previously builds of parents were only 
started when the database was updated (a few hundred milliseconds later).

For multi-spack-process parallel builds, this will mean that the Spack process
continues to hold a write lock after the spec has been installed, until it's 
persisted to the database.

Signed-off-by: Harmen Stoppels <[email protected]>

)

* Include parent scopes: tweak validation, add unit test

Signed-off-by: tldahlgren <[email protected]>

Signed-off-by: Harmen Stoppels <[email protected]>

Currently, there are scenarios and use cases that are difficult to express with 
a single `spack.yaml` file. For instance:

1. Start with an old system compiler to bootstrap a new toolchain, 
    and then software on top of that
2. Deploy a highly heterogeneous stack (different compilers, different options
    for the same set of specs)  while ensuring a fine control over dependencies

These use cases are usually dealt with an iterative approach or multiple 
environments. Both methods are sub-optimal and error prone.

In this commit we solve these kind of issues by allowing named group of specs, 
as shown in spack#49097. Each named group of specs:

- Can have dependencies on other groups. If that happens, the dependency groups
  are concretized before the current group, and their roots specs are always available
  for reuse in the current concretization.

- Can override configuration scopes with details that matter only for the current group,
  since the override is then used _only_ for the concretization of the current group

Signed-off-by: Massimiliano Culpo <[email protected]>

* Lookup open file handles by (ino, dev) key
* Drop `pid` from the key: when forking, fds are inherited and all cached file
  handles are valid; when forkserver/spawn, the cache is empty. In neither case
  `pid` matters.
* Do not close a file handle when the last lock is released
* Use EAFP pattern:
  * directly open lock file in read/write mode
  * if it fails, open in read mode
  * if it fails, try to make the relevant dirs and open in read/write

This makes the number of syscalls minimal in the most common use cases:

1. the parent dirs and lock files already exists
2. the lock is repeatedly locked and unlocked

In repeated lock calls, the happy path is just a stat call before the fcntl syscall.
If the lock file has been unlinked from the dir and re-created by a different process,
we invalidate our cache and re-open the new lock file: this is guaranteed because
we maintain an open fd, meaning that the new lock file must have a different inode.

This commit adds support for group of specs to `spack find`.
Includes a unit test to avoid regressions.

Signed-off-by: Massimiliano Culpo <[email protected]>

Signed-off-by: tldahlgren <[email protected]>

Show [+]/[x]/[e]/[ ] indicators and print the install prefix for
finished builds, matching the TTY display format.

Signed-off-by: Harmen Stoppels <[email protected]>

…pack#51960)

Previously the `--overwrite` flag was passed on all the way to the build
process, where it was used to decide whether to error if the install
prefix exists prior to the build. That would lead to hard to overcome
install failures if an initial build didn't clear its prefix on failure (which
was the case due to `SIGTERM` exiting the interpreter immediately, see
spack#51967)

With this change, `--overwrite` is only used to determine what specs will
get rebuild. The build process always does an "overwrite install" in the
sense that if a prefix exists, it's moved out of the way.

* On success and generally with `--keep-prefix`, the old prefix is removed.
* Otherwise, on failure, the old prefix is moved back in place.

Also fix the weird alias `pathlib as pathlb`, probably an artifact from
auto-import in my editor at some point in time.

Allow multi-process concurrency, additionally to per-process package
parallelism.

Basic idea is as follows. If there are pending builds in this Spack
process:

* Take a read lock on the database (if this fails, re-enter event loop)
* Take a prefix write lock on the to-be-installed spec (if this fails,
  try the next pending build)
* If the spec is installed in the meantime: drop the prefix write lock,
  remove the pending build, enqueue its parents, continue
* If the spec is not installed, acquire a jobserver token if needed (if
  this fails, re-enter the event loop)
* If all succeeds, schedule the build, try to schedule more.
* Finally release the read lock on the db.

If it's not possible to obtain any write lock on the prefix lock, inform
the event loop that it shouldn't wake up on available jobserver tokens.
This is a measure to avoid a busy wait: locally there are jobserver
tokens available, but all pending builds are claimed by another process.

What this commit does not yet do:

* Take read locks on build dependencies and their link/run deps. So, it
  does not guard against concurrent uninstalls of (dependencies of) build
  deps.
* Avoid scheduling builds of packages that failed to install in another
  Spack process.

…41710)

Signed-off-by: Wouter Deconinck <[email protected]>

* Add --json option to spack repo list

This enhancement adds machine-readable JSON output format to the
'spack repo list' command, similar to other Spack commands like 'find'.

The JSON output includes detailed information about each repository:
- name: Configuration name
- namespace: Repository namespace
- path: Path to the repository
- api_version: Package API version
- status: Repository status (installed, uninitialized, or error)
- error: Error message if the repository has issues

Added tests to verify the functionality and updated documentation.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Gregory Becker <[email protected]>

* style

Signed-off-by: Gregory Becker <[email protected]>

* completions

Signed-off-by: Gregory Becker <[email protected]>

* refactor test for pythonic clarity

Signed-off-by: Gregory Becker <[email protected]>

---------

Signed-off-by: Gregory Becker <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>

* solver: add virtuals to the correct unification sets

fixes spack#51995

There are cases where in a unified environment one root depends
on e.g. llvm as a provider for libllvm, but is compiled with gcc,
and another root is instead compiled with llvm.

In those cases we have:
```
provider(node(0, gcc), node(0, c)).
provider(node(0, llvm), node(1, c)).
provider(node(0, llvm), node(0, libllvm)).
```
and we have to add the virtual node that is being provided to
the correct unification set. The rule:
```
unification_set(SetID, VirtualNode)
  :- provider(PackageNode, VirtualNode),
     unification_set(SetID, PackageNode).
```
is therefore wrong in those cases, and so are other similar
simplifying assumptions.

In this commit we fix the issue by adding virtuals to the
correct unification sets.

Signed-off-by: Massimiliano Culpo <[email protected]>

Several users have requested the ability to have `python` or other non-virtual
packages as elements in the lmod hierarchy, to better isolate builds with 
different versions / features.

This PR eliminates the restriction that hierarchy components must be virtual 
packages or compilers.

Includes test and documentation modifications.

Signed-off-by: Gregory Becker <[email protected]>

The `opengl` package fails at runtime with a message,
saying it's a placeholder for external libraries. Make
it fail at concretization time instead by default.

Signed-off-by: Massimiliano Culpo <[email protected]>

* solver: fix rule for virtuals that are provided together

fixes spack#51512

Sometimes a unified environment has roots compiled
with different compilers. In those cases the rule
to ensure that virtuals that need to be provided
together ARE provided together was wrong.

In this PR we fix it, and we add a test case
to avoid regression.


Signed-off-by: Massimiliano Culpo <[email protected]>

This PR adds support for specifying the name of a git include. It also changes paths for 
SingleFileScopes reported using spack config scopes -p so they do NOT end with the 
path separator.

For example, given

include:
- name: site
  git: https://github.com/spack/spack-configs.git
  branch: main
  paths:
  - USC/config
the scope names would be:

$ spack config scopes -p
Scope               Path
command_line
spack               $spack/etc/spack/
user                $HOME/.spack/
site                $HOME/.spack/includes/nncrh7v/USC/config/
system              /etc/spack/
defaults            $spack/etc/spack/defaults/
defaults:darwin     $spack/etc/spack/defaults/darwin/
defaults:base       $spack/etc/spack/defaults/base/
_builtin
Consequently the default site configuration is overridden.

Suppose you only want two of the current five `USC/config` configuration files. If you provide multiple explicit paths, then the name will be prepended to each path entry to ensure uniqueness.

Given you only want the config.yaml and packages.yaml files from the repository:

include:
- name: site
  git: https://github.com/spack/spack-configs.git
  branch: main
  paths:
  - USC/config/config.yaml
  - USC/config/packages.yaml

then the scope names are:

$ spack config scopes -p
Scope               Path
command_line
spack                         $spack/etc/spack/
user                           $HOME/.spack/
site:config.yaml        $HOME/.spack/includes/nncrh7v/USC/config/config.yaml
site:packages.yaml  $HOME/.spack/includes/nncrh7v/USC/config/packages.yaml
site                            $HOME/github/prs/spack/etc/spack/site/
system                      /etc/spack/
defaults                    $spack/etc/spack/defaults/
defaults:darwin       $spack/etc/spack/defaults/darwin/
defaults:base          $spack/etc/spack/defaults/base/
_builtin

Which means those configuration files do NOT override the default site configuration but their contents do have higher precedence.

---------

Signed-off-by: tldahlgren <[email protected]>

…1914)

* Bugfix/cmd list: return proper path for json and html output

Signed-off-by: tldahlgren <[email protected]>

* spack list: improve handling of local, spack, and non-spack package repos.

Signed-off-by: tldahlgren <[email protected]>

* Don't use os.sep for checking the file URL (on windows)

Signed-off-by: tldahlgren <[email protected]>

* test_list_format_non_github_repo: Use 'as_uri' to ensure have a file URI

Signed-off-by: tldahlgren <[email protected]>

* Add test_list_github_url_fails

Signed-off-by: tldahlgren <[email protected]>

---------

Signed-off-by: tldahlgren <[email protected]>

* solver: don't assume max_dupes = 1 for possible link/run deps

If a virtual is a possible link/run dependency, don't make
assumption on its number of duplicates based on whether it
may appear in the link/run closure.

That may cause issues if e.g. packages have typos and use:
```
depends_on("c")
```
since the max_dupes for that language will be 1 instead of
the configured default of 2.

Signed-off-by: Massimiliano Culpo <[email protected]>

* solver: fix an issue with virtuals when max_dupes > 1

When a virtual is depended on with multiple edge types,
we must ensure that the same package gets the same
provider on each of the edge types.

E.g. it can't happen that a:
```
depends_on("lapack", type=("build","link"))
```
is satisfied by openblas on the link part,
and by netlib-lapack on the build part.

Signed-off-by: Massimiliano Culpo <[email protected]>

---------

Signed-off-by: Massimiliano Culpo <[email protected]>

Fix accidental quadratic complexity issue when iterating
DisjointSetsOfValues objects.

Signed-off-by: Harmen Stoppels <[email protected]>

This option works only if an environment is active
and accounts for group overrides when displaying
the configuration.

Signed-off-by: Massimiliano Culpo <[email protected]>

…pack#52031)

Bumps [sphinxcontrib-svg2pdfconverter](https://github.com/missinglinkelectronics/sphinxcontrib-svg2pdfconverter) from 2.0.0 to 2.1.0.
- [Commits](missinglinkelectronics/sphinxcontrib-svg2pdfconverter@v2.0.0...v2.1.0)

---
updated-dependencies:
- dependency-name: sphinxcontrib-svg2pdfconverter
  dependency-version: 2.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Peter Scheibel <[email protected]>

Currently we do an ioctl syscall on every redraw to get the terminal
size. Instead use the self-pipe trick to handle `SIGWINCH` in the event
loop, so we only query the terminal size if changed. This is only
enabled in TTY mode.

Signed-off-by: Harmen Stoppels <[email protected]>

Signed-off-by: Brian Vanderwende <[email protected]>

Signed-off-by: Abele, Daniel <[email protected]>

…k#52392)

Bumps [julia-actions/setup-julia](https://github.com/julia-actions/setup-julia) from 3.0.1 to 3.0.2.
- [Release notes](https://github.com/julia-actions/setup-julia/releases)
- [Commits](julia-actions/setup-julia@f6f565d...fa02766)

---
updated-dependencies:
- dependency-name: julia-actions/setup-julia
  dependency-version: 3.0.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

spack#52393)

Bumps [coverage](https://github.com/coveragepy/coveragepy) from 7.13.5 to 7.14.0.
- [Release notes](https://github.com/coveragepy/coveragepy/releases)
- [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst)
- [Commits](coveragepy/coveragepy@7.13.5...7.14.0)

---
updated-dependencies:
- dependency-name: coverage
  dependency-version: 7.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Adds support for sandboxing builds in the new installer. Works on Linux using Landlock.

The goal is primarily build isolation and reproducibility. This sandbox does not guard against malicious python code in `package.py`, which runs before the build on module import.

When enabled in config, the following dirs are readable and executable:

* Prefixes of Spack installed dependencies (excluding externals)
* User configured dirs

The following are writeable and executable:

* Install prefix
* Stage dir
* System temp dir (POSIX `/tmp` or `TMPDIR`)
* `/dev/null` (POSIX)
* User configured dirs

The sandbox applies to the build process (and sub-processes) after sources are fetched and extracted, and before running the install phases.

Signed-off-by: Harmen Stoppels <[email protected]>

…2388)

Toolchains are currently not expanded in the condition
of a conditional preference. Here we fix the issue and
add a regression test.

Signed-off-by: Massimiliano Culpo <[email protected]>

This closes the sentinel file descriptors, which would otherwise only happen on exit, resulting in an fd leak.

Signed-off-by: Harmen Stoppels <[email protected]>

* add audit check for placeholder maintainers

spack/spack-packages#4821 is the only package
that has this issue; this will ensure it can't get past CI.

Signed-off-by: Caetano Melone <[email protected]>

* update package hash to be valid sha256

Signed-off-by: Caetano Melone <[email protected]>

* update mock package to avoid boilerplate url error

Signed-off-by: Caetano Melone <[email protected]>

* fix unit tests

Signed-off-by: Caetano Melone <[email protected]>

* set intersections for lookup rather than loops

Signed-off-by: Caetano Melone <[email protected]>
Co-authored-by: Alec Scott <[email protected]>

---------

Signed-off-by: Caetano Melone <[email protected]>
Co-authored-by: Alec Scott <[email protected]>

…52355)

* Fix setup failure when nonexist directory has '.' in basename

I encountered a nasty failure that happens before the `-v`/`-d`
arguments are parsed in spack:
```
==> Error: File-based scope does not exist yet: should have a .yaml/.yml extension for file scopes, or no extension for directory scopes (currently .04)
```
This happened because (the FNAL fork of) spack was looking for a missing directory `.../etc/spack/linux/ubuntu24.04`. The logic in place assumed that having an extension meant it was a file.

Signed-off-by: Seth R Johnson <[email protected]>

* Print a warning

Signed-off-by: Seth R Johnson <[email protected]>

* Improve nomenclature

Signed-off-by: Seth R Johnson <[email protected]>

* Redo logic

Signed-off-by: Seth R Johnson <[email protected]>

* Fix typos

Signed-off-by: Seth R Johnson <[email protected]>

* Style

Signed-off-by: Seth R Johnson <[email protected]>

---------

Signed-off-by: Seth R Johnson <[email protected]>

Spack-on-Windows has long had a dependency on a pip-installed pywin32
module as that module typically vendors a lot of the win32 api
interactions missing from the Python standard library.

This PR removes this dependency and reimplements the functions we used
it for.

Signed-off-by: John Parent <[email protected]>

With Python 3.14 using forkserver, and macOS using spawn, spack
buildcache push is sequential. This PR makes it parallel.

That's beneficial if we can avoid expensive pickling of environments, so
make that opt-in.

Further, improve fork-safety by clearing urlopen related state, similar
to what the new installer does.

Signed-off-by: Harmen Stoppels <[email protected]>

Previously scripts assumed you were working in an established spack shell. This allows them to be invoked directly like bin/spack (or bin\spack for cmd)

Signed-off-by: John Parent <[email protected]>

…pack#52417)

Signed-off-by: Alec Scott <[email protected]>

Signed-off-by: John Parent <[email protected]>

…m-develop