environment groups: add "explicit" attribute - #52244

* solver: choose `virtual_on_edge` and derive `depends_on` (#52003)

* solver: choose attr("virtual_on_edge")

Signed-off-by: Massimiliano Culpo <[email protected]>

* solver: simplify virtual_edge_needed

Signed-off-by: Massimiliano Culpo <[email protected]>

---------

Signed-off-by: Massimiliano Culpo <[email protected]>

* spack-completion.bash: completion after --foo bar (#51974)

Fixes an issue that has bothered me for years. If you do

```
spack buildcache push --base-image foo --<tab>
```

you get nothing, because Spack looks for a completion function

```
_spack_buildcache_push_foo
```

cause it doesn't know that `foo` is a value.

This fixes that by keeping track of the last function that exists and
use that for tab completion instead, which in this case is

```
_spack_buildcache_push
```

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: parse build log on failure (#52000)

The failure message referenced s.package.log_path, a symlink that may
not exist if the build fails before the stage is created. Now the parent
creates the log file via tempfile.mkstemp, passes the path to the child,
and the failure message always references a real, existing file.

Signed-off-by: Harmen Stoppels <[email protected]>

* solver: turn a choice rule into an integrity constraint (#52001)

It probably doesn't matter much for performance,
but the dimension is reduced like:

O(Package x Virtual) -> O(Virtual)

Also, integrity constraint should help the USC strategy
to learn clauses faster.

Signed-off-by: Massimiliano Culpo <[email protected]>

* new_installer.py: small log file improvements (#51998)

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: delayed database write (#51951)

This makes the installer a bit snappier in the UI, and reduces the latency 
for short-running installs (e.g. python packages and installs from build cache).

The idea is to save installed specs to the database only if in the last
5 seconds no other specs finished to build.

Builds of parents are now started before the child is stored in the database. 
This reduces latency significantly; previously builds of parents were only 
started when the database was updated (a few hundred milliseconds later).

For multi-spack-process parallel builds, this will mean that the Spack process
continues to hold a write lock after the spec has been installed, until it's 
persisted to the database.

Signed-off-by: Harmen Stoppels <[email protected]>

* Include parent scopes: consistent validation, add unit test (#51966)

* Include parent scopes: tweak validation, add unit test

Signed-off-by: tldahlgren <[email protected]>

* tests: isolate binary index cache in some tests (#52007)

Signed-off-by: Harmen Stoppels <[email protected]>

* environment: allow group of specs with dependencies (#51891)

Currently, there are scenarios and use cases that are difficult to express with 
a single `spack.yaml` file. For instance:

1. Start with an old system compiler to bootstrap a new toolchain, 
    and then software on top of that
2. Deploy a highly heterogeneous stack (different compilers, different options
    for the same set of specs)  while ensuring a fine control over dependencies

These use cases are usually dealt with an iterative approach or multiple 
environments. Both methods are sub-optimal and error prone.

In this commit we solve these kind of issues by allowing named group of specs, 
as shown in #49097. Each named group of specs:

- Can have dependencies on other groups. If that happens, the dependency groups
  are concretized before the current group, and their roots specs are always available
  for reuse in the current concretization.

- Can override configuration scopes with details that matter only for the current group,
  since the override is then used _only_ for the concretization of the current group

Signed-off-by: Massimiliano Culpo <[email protected]>

* lock.py: reduce syscalls to a minimum (#51996)

* Lookup open file handles by (ino, dev) key
* Drop `pid` from the key: when forking, fds are inherited and all cached file
  handles are valid; when forkserver/spawn, the cache is empty. In neither case
  `pid` matters.
* Do not close a file handle when the last lock is released
* Use EAFP pattern:
  * directly open lock file in read/write mode
  * if it fails, open in read mode
  * if it fails, try to make the relevant dirs and open in read/write

This makes the number of syscalls minimal in the most common use cases:

1. the parent dirs and lock files already exists
2. the lock is repeatedly locked and unlocked

In repeated lock calls, the happy path is just a stat call before the fcntl syscall.
If the lock file has been unlinked from the dir and re-created by a different process,
we invalidate our cache and re-open the new lock file: this is guaranteed because
we maintain an open fd, meaning that the new lock file must have a different inode.

* spack find: display group of specs in environment (#52009)

This commit adds support for group of specs to `spack find`.
Includes a unit test to avoid regressions.

Signed-off-by: Massimiliano Culpo <[email protected]>

* config.py: fix invalid scope error message (#52010)

Signed-off-by: tldahlgren <[email protected]>

* new_installer.py: improve non-TTY output format (#52005)

Show [+]/[x]/[e]/[ ] indicators and print the install prefix for
finished builds, matching the TTY display format.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: always pivot prefix, decouple from `--overwrite` (#51960)

Previously the `--overwrite` flag was passed on all the way to the build
process, where it was used to decide whether to error if the install
prefix exists prior to the build. That would lead to hard to overcome
install failures if an initial build didn't clear its prefix on failure (which
was the case due to `SIGTERM` exiting the interpreter immediately, see
#51967)

With this change, `--overwrite` is only used to determine what specs will
get rebuild. The build process always does an "overwrite install" in the
sense that if a prefix exists, it's moved out of the way.

* On success and generally with `--keep-prefix`, the old prefix is removed.
* Otherwise, on failure, the old prefix is moved back in place.

Also fix the weird alias `pathlib as pathlb`, probably an artifact from
auto-import in my editor at some point in time.

* new_installer.py: initial multi-process support (#51985)

Allow multi-process concurrency, additionally to per-process package
parallelism.

Basic idea is as follows. If there are pending builds in this Spack
process:

* Take a read lock on the database (if this fails, re-enter event loop)
* Take a prefix write lock on the to-be-installed spec (if this fails,
  try the next pending build)
* If the spec is installed in the meantime: drop the prefix write lock,
  remove the pending build, enqueue its parents, continue
* If the spec is not installed, acquire a jobserver token if needed (if
  this fails, re-enter the event loop)
* If all succeeds, schedule the build, try to schedule more.
* Finally release the read lock on the db.

If it's not possible to obtain any write lock on the prefix lock, inform
the event loop that it shouldn't wake up on available jobserver tokens.
This is a measure to avoid a busy wait: locally there are jobserver
tokens available, but all pending builds are claimed by another process.

What this commit does not yet do:

* Take read locks on build dependencies and their link/run deps. So, it
  does not guard against concurrent uninstalls of (dependencies of) build
  deps.
* Avoid scheduling builds of packages that failed to install in another
  Spack process.

* setup-env.sh: if exe contains qemu, use /proc/$$/comm instead (#41710)

Signed-off-by: Wouter Deconinck <[email protected]>

* spack repo list: machine readable output with --json (#51950)

* Add --json option to spack repo list

This enhancement adds machine-readable JSON output format to the
'spack repo list' command, similar to other Spack commands like 'find'.

The JSON output includes detailed information about each repository:
- name: Configuration name
- namespace: Repository namespace
- path: Path to the repository
- api_version: Package API version
- status: Repository status (installed, uninitialized, or error)
- error: Error message if the repository has issues

Added tests to verify the functionality and updated documentation.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Gregory Becker <[email protected]>

* style

Signed-off-by: Gregory Becker <[email protected]>

* completions

Signed-off-by: Gregory Becker <[email protected]>

* refactor test for pythonic clarity

Signed-off-by: Gregory Becker <[email protected]>

---------

Signed-off-by: Gregory Becker <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>

* solver: add virtuals to the correct unification sets (#52015)

* solver: add virtuals to the correct unification sets

fixes #51995

There are cases where in a unified environment one root depends
on e.g. llvm as a provider for libllvm, but is compiled with gcc,
and another root is instead compiled with llvm.

In those cases we have:
```
provider(node(0, gcc), node(0, c)).
provider(node(0, llvm), node(1, c)).
provider(node(0, llvm), node(0, libllvm)).
```
and we have to add the virtual node that is being provided to
the correct unification set. The rule:
```
unification_set(SetID, VirtualNode)
  :- provider(PackageNode, VirtualNode),
     unification_set(SetID, PackageNode).
```
is therefore wrong in those cases, and so are other similar
simplifying assumptions.

In this commit we fix the issue by adding virtuals to the
correct unification sets.

Signed-off-by: Massimiliano Culpo <[email protected]>

* lmod layout: support non-virtuals in hierarchy (#51824)

Several users have requested the ability to have `python` or other non-virtual
packages as elements in the lmod hierarchy, to better isolate builds with 
different versions / features.

This PR eliminates the restriction that hierarchy components must be virtual 
packages or compilers.

Includes test and documentation modifications.

Signed-off-by: Gregory Becker <[email protected]>

* packages.yaml: mark opengl as buildable:false (#52019)

The `opengl` package fails at runtime with a message,
saying it's a placeholder for external libraries. Make
it fail at concretization time instead by default.

Signed-off-by: Massimiliano Culpo <[email protected]>

* solver: fix rule for virtuals that are provided together (#52021)

* solver: fix rule for virtuals that are provided together

fixes #51512

Sometimes a unified environment has roots compiled
with different compilers. In those cases the rule
to ensure that virtuals that need to be provided
together ARE provided together was wrong.

In this PR we fix it, and we add a test case
to avoid regression.


Signed-off-by: Massimiliano Culpo <[email protected]>

* Support named git includes (#51939)

This PR adds support for specifying the name of a git include. It also changes paths for 
SingleFileScopes reported using spack config scopes -p so they do NOT end with the 
path separator.

For example, given

include:
- name: site
  git: https://github.com/spack/spack-configs.git
  branch: main
  paths:
  - USC/config
the scope names would be:

$ spack config scopes -p
Scope               Path
command_line
spack               $spack/etc/spack/
user                $HOME/.spack/
site                $HOME/.spack/includes/nncrh7v/USC/config/
system              /etc/spack/
defaults            $spack/etc/spack/defaults/
defaults:darwin     $spack/etc/spack/defaults/darwin/
defaults:base       $spack/etc/spack/defaults/base/
_builtin
Consequently the default site configuration is overridden.

Suppose you only want two of the current five `USC/config` configuration files. If you provide multiple explicit paths, then the name will be prepended to each path entry to ensure uniqueness.

Given you only want the config.yaml and packages.yaml files from the repository:

include:
- name: site
  git: https://github.com/spack/spack-configs.git
  branch: main
  paths:
  - USC/config/config.yaml
  - USC/config/packages.yaml

then the scope names are:

$ spack config scopes -p
Scope               Path
command_line
spack                         $spack/etc/spack/
user                           $HOME/.spack/
site:config.yaml        $HOME/.spack/includes/nncrh7v/USC/config/config.yaml
site:packages.yaml  $HOME/.spack/includes/nncrh7v/USC/config/packages.yaml
site                            $HOME/github/prs/spack/etc/spack/site/
system                      /etc/spack/
defaults                    $spack/etc/spack/defaults/
defaults:darwin       $spack/etc/spack/defaults/darwin/
defaults:base          $spack/etc/spack/defaults/base/
_builtin

Which means those configuration files do NOT override the default site configuration but their contents do have higher precedence.

---------

Signed-off-by: tldahlgren <[email protected]>

* Bugfix/cmd list: return proper path for json and html output (#51914)

* Bugfix/cmd list: return proper path for json and html output

Signed-off-by: tldahlgren <[email protected]>

* spack list: improve handling of local, spack, and non-spack package repos.

Signed-off-by: tldahlgren <[email protected]>

* Don't use os.sep for checking the file URL (on windows)

Signed-off-by: tldahlgren <[email protected]>

* test_list_format_non_github_repo: Use 'as_uri' to ensure have a file URI

Signed-off-by: tldahlgren <[email protected]>

* Add test_list_github_url_fails

Signed-off-by: tldahlgren <[email protected]>

---------

Signed-off-by: tldahlgren <[email protected]>

* solver: fix issues with the computation of `max_dupes` (#52027)

* solver: don't assume max_dupes = 1 for possible link/run deps

If a virtual is a possible link/run dependency, don't make
assumption on its number of duplicates based on whether it
may appear in the link/run closure.

That may cause issues if e.g. packages have typos and use:
```
depends_on("c")
```
since the max_dupes for that language will be 1 instead of
the configured default of 2.

Signed-off-by: Massimiliano Culpo <[email protected]>

* solver: fix an issue with virtuals when max_dupes > 1

When a virtual is depended on with multiple edge types,
we must ensure that the same package gets the same
provider on each of the edge types.

E.g. it can't happen that a:
```
depends_on("lapack", type=("build","link"))
```
is satisfied by openblas on the link part,
and by netlib-lapack on the build part.

Signed-off-by: Massimiliano Culpo <[email protected]>

---------

Signed-off-by: Massimiliano Culpo <[email protected]>

* variant.py: Sequence -> Iterable (#51859)

Fix accidental quadratic complexity issue when iterating
DisjointSetsOfValues objects.

Signed-off-by: Harmen Stoppels <[email protected]>

* spack config: add --group option (#52025)

This option works only if an environment is active
and accounts for group overrides when displaying
the configuration.

Signed-off-by: Massimiliano Culpo <[email protected]>

* build(deps): bump sphinxcontrib-svg2pdfconverter in /lib/spack/docs (#52031)

Bumps [sphinxcontrib-svg2pdfconverter](https://github.com/missinglinkelectronics/sphinxcontrib-svg2pdfconverter) from 2.0.0 to 2.1.0.
- [Commits](https://github.com/missinglinkelectronics/sphinxcontrib-svg2pdfconverter/compare/v2.0.0...v2.1.0)

---
updated-dependencies:
- dependency-name: sphinxcontrib-svg2pdfconverter
  dependency-version: 2.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* tests: print autocompletion diff when it needs updating (#52032)

Signed-off-by: Peter Scheibel <[email protected]>

* new_installer.py: event-driven terminal resize (#52011)

Currently we do an ioctl syscall on every redraw to get the terminal
size. Instead use the self-pipe trick to handle `SIGWINCH` in the event
loop, so we only query the terminal size if changed. This is only
enabled in TTY mode.

Signed-off-by: Harmen Stoppels <[email protected]>

* Support cxx and fortran for Lmod compiler hierarchy (#52018)

Signed-off-by: Brian Vanderwende <[email protected]>

* color.py: fix @@ unescaping (#52034)

Move @@ -> @ replacement into match_to_ansi's text branch so it applies
inside `{...}` blocks; top-level `@@` is already handled by the regex.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: read locks on installed deps (#52012)

After writing an installed spec to the DB, downgrade the prefix write lock to a                                                                                                     
read lock instead of releasing it outright. This prevents another process from                                                                                                      
uninstalling the spec while the current process still depends on it. The read                                                                                                       
locks are collected in `retained_read_locks` and released on exit.                                                                                                                  
                                                                                                                                                                                    
Locks and resources are released best-effort under a single `finally` block
after the event loop finishes.

* config.py: atomic writes to prevent parallel test races (#52041)

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: concurrent overwrite installs (#52013)

If two concurrent install processes do an overwrite install of the same
spec, the second one now realizes that the first one has overwritten the
install by comparing against the timestamp at which the installer was
started.

Mimics the older installer.

Signed-off-by: Harmen Stoppels <[email protected]>

* ci: try new installer in bootstrap.yml (#51784)

Signed-off-by: Harmen Stoppels <[email protected]>

* Fix some typos found with codespell (#52038)

Signed-off-by: Xavier Delaruelle <[email protected]>

* new_installer.py: archive build provenance metadata (#51999)

Archive spack-build-out.txt.gz, spack-build-env.txt,
spack-configure-args.txt, install-time-test-log.txt, repos/, and
archived-files/ into the .spack metadata directory after a successful
build, matching what the old installer writes.

Signed-off-by: Harmen Stoppels <[email protected]>

* mailmap: add Harmen Stoppels (#52035)

Signed-off-by: Harmen Stoppels <[email protected]>

* lock.py: re-affirm POSIX lock to support forking (#52004)

* lock.py: re-affirm lock to support forking

When doing acquire_write, fork, acquire_write, the forked process
currently does not error, because the `_writes` counter is `1` after the
fork. But forking does not inherit locks, because it's a different pid.

With this change, on a nested lock, we always do the lock syscall, which
is a no-op for the process that acquired the lock, but an error for the
forked process.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: reset stdout/stderr on Tee exit (#52053)

Previously the build subprocess would close stdout/stderr, which in turn
would unblock the Tee thread so it could be `.join`ed and then exit the
build process.

That does not work in certain hard to troubleshoot edge cases witnessed
on macOS + pytest-xdist + coverage, where presumably stdout/stderr is
flushed between sys.exit and end-of-process.

Since Python already makes it hard to close stdout/stderr
(sys.stdout.close() is a no-op), the approach taken here is to dup the
file descriptor and restore it afterwards to not break possible atexit
handlers or cpython internals.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: no log file for externals (#52055)

Signed-off-by: Harmen Stoppels <[email protected]>

* spack buildcache update-index: add timing when verbose is active (#52051)

When verbose, displays a summary of where the time was spent
at the end of the execution.

Signed-off-by: Massimiliano Culpo <[email protected]>

* new_installer.py: integration testing (#52049)

* new_installer.py: integration testing

* Mimic GlobalStateMarshaller, but (a) drop unnecessary package related
  serialization bits and (b) do not serialize the active env
* Add support for `--fake` used in tests.
* Enable some unit tests for the new installer by parametrizing the
  installer config (old/new).

Signed-off-by: Harmen Stoppels <[email protected]>

* GlobalState: dynamic instead of at a class definition

Signed-off-by: Harmen Stoppels <[email protected]>

---------

Signed-off-by: Harmen Stoppels <[email protected]>

* Revert "variant.py: Sequence -> Iterable (#51859)" (#52056)

This reverts commit 341772bbab8673059dfc10913bd3ad5092050c8f.

* package_base.py: unit_test_check is not public API (#52060)

This function exists for the purpose of unit tests; it looks a lot like
a hook to be used in spack-packages, which is rather unfortunate.

Make it private.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: add --test support (#52047)

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: --verbose in non-TTY mode (#52048)

This adds support for `spack install --verbose` in non-TTY mode.

It balances readability and troubleshootability:

* Follow logs of one build and one build only
* Always follow a build from start to finish, never half-way the process
* For concurrent builds: only print install phase state changes as a
  single line, interleaved with the active build's logs.

In the case of `spack install --verbose -p1` this choice results in the
same output as people were used to from the old installer before package
parallelism

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: reporters (#52058)

* new_installer.py: reporters

Signed-off-by: Harmen Stoppels <[email protected]>

* communicate install from build cache

Signed-off-by: Harmen Stoppels <[email protected]>

* debug message on report failure

Signed-off-by: Harmen Stoppels <[email protected]>

* exitcode

Signed-off-by: Harmen Stoppels <[email protected]>

---------

Signed-off-by: Harmen Stoppels <[email protected]>

* installer.py: pass debug=True to logger for clarity (#52067)

Signed-off-by: Gregory Becker <[email protected]>

* new_installer.py: enable debug like old installer (#52059)

* Set debug=1 for the tty during phase execution
* Force line buffering for sys.stdout and sys.error to ensure correctly ordered output

Signed-off-by: Harmen Stoppels <[email protected]>

* concretize.lp: avoid imposing transitive link deps of compiler run deps (#52029)

When a compiler is reused as a pure build dependency, its run-reachable
transitive deps are unified in the build environment (they appear in
PATH etc.), but their pure link-type dependencies are local to the
compiler's toolchain and must not be forced onto the package being
built.

Previously, avoid_link_dependency only suppressed direct link-only deps
of the compiler itself. This missed the transitive case: if the compiler
has a run+link dep on binutils and binutils has a pure link dep on zlib,
the imposed hash on zlib from binutils was propagated to the package,
causing an UnsatisfiableSpecError when the package requested a different
zlib version.

Fix: add compiler_non_lib_run_dep/2 rules that compute the full
transitive run-dep closure of non-library compilers, then extend
avoid_link_dependency to suppress hash/edge constraints for pure link
deps at every level of that closure.

asp.py also now includes transitive link deps of reusable compilers as
reusable candidates (not just run deps), so their hash_attr facts are
available for the new rules.  When a compiler is used as a library, or
for a fresh (--fresh) concretization, the existing behavior is
unchanged.

Signed-off-by: Harmen Stoppels <[email protected]>

* Deprecate `include_concrete:` in favor of `include:`

The `include_concrete` key in spack.yaml is deprecated. Concrete
environments (lock files) should now be listed under the standard
`include` key with an explicit `spack.lock` path:

# Old (deprecated)
include_concrete:
- /path/to/env

# New
include:
- /path/to/env/spack.lock

Using `include_concrete` still works but emits a deprecation warning.
Users can run `spack env update <env>` to automatically migrate the 
manifest to the new format.

Signed-off-by: tldahlgren <[email protected]>
Signed-off-by: Tamara Dahlgren <[email protected]>
Co-authored-by: Massimiliano Culpo <[email protected]>

* modules: ease tcl template & modulefile readability (#52046)

Update "tcl" modulefile template to ease its readability and readability
of generated modulefiles:

* always use "depends-on" command to auto load dependencies and define
  this command if it is not found (when Environment Modules <5.1 is
  used)
* always specify a path delimiter when using append-path, prepend-path
  and remove-path

Signed-off-by: Xavier Delaruelle <[email protected]>

* new_installer.py: log filtering of path padding (#52071)

Do log filtering in `BuildStatus`, which is a class that handles the UI of the
new installer. The Tee class in the build process still duplexes build output
verbatim to the log file and to the parent process; this is just a UI
enhancement, and therefore part of the parent process.

* Support both `str` and `bytes` in `path.py` log filtering utility
* Use bytes based log filtering in `new_installer.py` (it treats stdout/stderr as raw bytes)
* Parametrize existing tests over `str` and `bytes`.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: drain logs pipe before close (#52080)

* new_installer.py: always try build cache (#52073)

* new_installer.py: better color support (and a small path filtering fix) (#52072)

* In non-TTY mode use colors when forced (`spack --color=always` or
  `SPACK_COLOR=always`)
* In TTY mode do not use colors when explicitly disabled with the flag
  `spack --color=never` or `SPACK_COLOR=never`.
* Use single code path for status line whether TTY or non-TTY.
* While at it: apply path filtering of the install prefix also in the status line to `[+] ... /path/to/__spack_path_placeholder__/...`.



Signed-off-by: Harmen Stoppels <[email protected]>

* fetch: retry urllib downloads with back-off (#52081)

In CI, transient fetch errors (S3 rate limits, server errors, mid-stream
disconnects) cause spurious failures. To handle this uniformly:

- Add `web_util.is_transient_error(e)` covering HTTP 5xx/429, socket
  timeouts, and botocore ResponseStreamingError.
- Rewrite `_fetch_urllib` to retry up to `_FETCH_RETRIES` times with
  exponential back-off (1s, 2s, 4s, 8s), writing to a `.part` file
  for atomicity (consistent with `_fetch_curl`).
- Simplify `oci/opener.py`'s retry logic to delegate to
  `is_transient_error`.

Signed-off-by: Harmen Stoppels <[email protected]>

* spec_parser.py: extract toolchain expansion from SpecParser (#52076)

Toolchain references are now parsed as regular dependencies and expanded 
in a separate post-parse step via `expand_toolchains`.

Toolchain expansion now happens in these entry points:

* SpecList/SpecListParser for environment YAML
* RequirementParser for packages.yaml requirements
* spack.cmd.parse_specs for CLI

Signed-off-by: Massimiliano Culpo <[email protected]>

* Add test for resolving git-based relative includes in environments (#52069)

The test ensures that `spack.lock` entries
inside git-based includes are correctly resolved
using the clone destination as the base directory,
rather than the manifest directory.

Signed-off-by: Massimiliano Culpo <[email protected]>

* new_installer.py: join timeout in finally block (#52082)

If a grandchild process ignores SIGTERM, the cleanup path hangs forever,
leaving locks, jobserver, and terminal unreleased. Add a 30-second
timeout and escalate to SIGKILL.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: clean up state_buffers on exit (#52083)

When a child exits, if the sentinel fires before the state pipe EOF,
state_buffers retains a partial JSON fragment. If the OS reuses that fd
number for a new child's state pipe, the stale data is prepended,
corrupting the JSON stream. Pop the buffer before cleanup closes the fd.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: use longer timeout for non-TTY (#52085)

* new_installer.py: fix non-blocking I/O (#52086)

* new_installer.py: ensure newlines between logs (#52087)

If a log line does not end with a newline, the UI should add one when
switching to overview mode or the new log. If a log line did end with a
newline, we shouldn't add double newlines.

This prevents things like:

```
checking for foo.h header... ==> following log of xyz
...
```

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: catch JSONDecodeError (#52084)

Malformed data from a crashing child causes an uncaught exception that
crashes the event loop. Catch JSONDecodeError and skip the malformed
line instead. In practice this should never happen, but it does not hurt
to be a bit more defensive.

Signed-off-by: Harmen Stoppels <[email protected]>

* ci: only use --verbose for root install, not deps (#52090)

When using the new installer in CI, it's not particularly pretty to do
`--verbose` when installing dependencies. Only apply --verbose to the
root package to see build logs.

Also drop --backtrace. In the new installer the child process always
prints the full backtrace to the build log on failure. The parent
process shouldn't show a backtrace to the installer event loop on
install failure, that suggests the problem is with Spack instead of
with the build, causing confusion.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: fork safety of ssl/boto3/urllib3 (#52089)

It turns out Gitlab CI was failing with parallel downloads not 
because of concurrent requests that need retry, but due to 
fork-safety issues of SSL context objects.

The likely explanation is that shared resources are mutated 
by forked process A that invalidate them for concurrent 
forked process B.

Signed-off-by: Harmen Stoppels <[email protected]>

* install_test.py: break circular import (#52092)

Signed-off-by: Harmen Stoppels <[email protected]>

* urlopen: clean up resources (#52091)

Add read_text() and read_json() helpers to spack.util.web that
encapsulate the fetch-decode-close cycle, and use them to replace
repeated read_from_url + TextIOWrapper boilerplate.

Signed-off-by: Harmen Stoppels <[email protected]>

* environment: reconstruct groups from lockfiles (#52095)

When a spack.yaml is created from a lockfile, the groups are
now preserved. `needs` dependencies are not preserved yet.

Signed-off-by: Massimiliano Culpo <[email protected]>

* new_installer.py: build failure ui improvements (#52093)

* Parse logs on error to get a log summary
* Allow users to navigate through logs with `n`/`p`
* Print the full log path in overview mode: `[x] ... failed: <log path>`
* Print the full log path in log mode after the log summary.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: two log fixes (#52096)

1. Unlink the log file from the stage dir if successful and not
   --keep-stage.
2. Drop second call to install test logs.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: no status line at the end (#52098)

Signed-off-by: Harmen Stoppels <[email protected]>

* Remove amdblis and amdlibflame as defaults (#52099)

The two packages are not very well maintained, and vendor-specific.

So it may be surprising to see `amdblis` selected when `openblas` is
not possible. Let Spack users decide the defaults instead.

Signed-off-by: Massimiliano Culpo <[email protected]>

* git: ensure git server configured for tests (#51989)

Our full clone test is failing to pull specific commits on ubuntu containers.
This is because the git version on the container supports `--filter=blob:none` 
but does not turn it on by default.

This change ensures that filter is configured before fetching when it will be 
used in the option list. Note that if the server is not configured to support 
filters things still proceed without erroring.

Git logs a message that the server isn't configured and won't apply the filter.
This is the expected behavior.

Signed-off-by: psakievich <[email protected]>

* Fix fetch logic error

Signed-off-by: Phil Sakievich <[email protected]>

* Clarify code per review

Signed-off-by: psakievich <[email protected]>

---------

Signed-off-by: psakievich <[email protected]>
Signed-off-by: Phil Sakievich <[email protected]>

* new_installer.py: clear progress line on ^C (#52103)

Very minor: on ^C the status line "Progress: ..." is sometimes cleared
and sometimes not, depending on what moment ^C was pressed.

If right after a spinner tick it would be removed, else it would not
because then the UI was not "dirty".

Signed-off-by: Harmen Stoppels <[email protected]>

* test/cmd/install.py: enable new installer more (#52097)

* test/cmd/install.py: enable new installer more

* fix case of forking under pytest

Signed-off-by: Harmen Stoppels <[email protected]>

* detection: deduplicate specs at different prefixes (#52100)

When the same spec (e.g. [email protected]) is discovered at
two different prefixes, only the first entry is kept.

A warning is printed to users for any successive
duplicate found in other prefixes.

Signed-off-by: Massimiliano Culpo <[email protected]>

* solver: prefer best provider above one with no penalty on variants (#52070)

Before this commit there were cases where a default compiler was not selected 
because it had penalty on variants. Instead, the second-best provider was selected.

Here we tweak priorities to ensure that the compiler choice is above variant penalty.

Signed-off-by: Massimiliano Culpo <[email protected]>

* requirements.py: fix toolchain expansion for preferences (#52106)

Signed-off-by: Massimiliano Culpo <[email protected]>

* Revert "solver: prefer best provider above one with no penalty on variants (#…" (#52108)

This reverts commit 1dd8c8167ae7af42b4400dc5220358cba8f9f392.

Signed-off-by: Harmen Stoppels <[email protected]>

* Add Changelog to Package API section (#52110)

Signed-off-by: Massimiliano Culpo <[email protected]>

* audit: catch propagation in directives (#52113)

Variant and dependency propagation shouldn't be used in package.py files. This PR adds checks to catch such uses and block them in CI.

Signed-off-by: Massimiliano Culpo <[email protected]>

* urlopen: improved retry on transient error (#52088)

* Use single retry-on-transient-error mechanism throughout.
* Add exponential back-off also to `url_exists` in case of transient errors
* Widen retry mechanism exception `OSError -> Exception`, because botocore/urllib3 exceptions do not subtype `OSError` but `Exception` directly

* solver: rename `NodeArgument` to `NodeId`. (#52116)

`NodeArgument` is used as an argument when reconstructing specs,
but I think it's clearer to call it a `NodeId`, since its real
function is to disambiguate "dupes", which is what we call different
configurations of the same package.

- [x] Rename `NodeArgument` to `NodeId` in `core.py` and `asp.py`
- [x] Call `NodeID` `ID` in `concretize.lp` for consistency with `asp.py`

Signed-off-by: Todd Gamblin <[email protected]>

* `spack_json`: add `pretty` option to `to_json()` (#52118)

Comparing JSON specs in tests is not useful unless `pytest` can show us a diff.

- [x] Introduce a `pretty=True` option to Spack JSON dumps that prints
      with newlines and indentation.
- [x] Modify tests to show a detailed diff using the new json output.

Signed-off-by: Todd Gamblin <[email protected]>

* executable.py: refactor to improve static type analysis (#52111)

* The use of TextIO in Executable.__call__ was a bug, only BinaryIO was allowed.

* input= doesn't support str or str.split values.

Signed-off-by: Harmen Stoppels <[email protected]>

* `.gitignore`: ignore coding agents state directories (#52125)

Claude, Gemini, and Codex all leave a hidden top-level directory in the project
for their own saved state. Ignore these for now.

If there are helpful things in here that make sense to check in, we can modify
this to allow them. For now, just prevent people from inadvertently checking
these directories in.

Signed-off-by: Todd Gamblin <[email protected]>

* new_installer.py: support --source (#52122)

* new_installer.py: dynamic --jobs (#51997)

Makes the number of build jobs (-j) dynamic with `+` and `-` keyboard input,
provided that we are the owner of the jobserver.

* On `+`: write a byte to `jobserver.w`
* On `-`: eventually read a byte from `jobserver.r`

So, `+` applies immediately, while `-` happens at some point in the event loop.
If the target number of jobs is less than the effective number of jobs because
the Spack process didn't get to read from the jobserver pipe, the terminal UI
renders it as `8=>4` meaning it's in the process of decreasing from 8 to 4 jobs.

There are two ways in which parallelism is reduced:

1. A parallel build finishes: do not write back the Spack-acquired token
2. The `jobserver.r` becomes readable: read at most `num_jobs - target_jobs` bytes.

Signed-off-by: Harmen Stoppels <[email protected]>

* lock.py: add non-blocking api (#52126)

Add `Lock.try_acquire_write()` and `Lock.try_acquire_read()` to simplify
the code in the new installer which only uses non-blocking lock calls in
the event loop.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: improve SIGTERM handler of build (#52102)

The previous implementation of SIGTERM was somewhat brittle. The
assumption was that if we're stuck in `Executable("make")(...)` and
receive SIGTERM, we can just forward the signal and then return to
waiting for `make` to exit with nonzero exite code, which would trigger
an exception and ultimately build failure.

Apart from the fact that the build *could* run
`Executable("make")(fail_on_error=False)`, it could also just be running
Python code, which would only continue.

So, instead of assuming we're stuck in `waitpid`, do an explicit
`waitpid` until all children have exited, and then raise
`KeyboardInterrupt`. This ensures we hit an exception in all cases
mentioned above.

Signed-off-by: Harmen Stoppels <[email protected]>

* solver: prefer best compiler above one with no penalty on variants (take 2) (#52109)

Before this commit there were cases where a default compiler was not selected 
because it had penalty on variants. Instead, the second-best provider was selected.

Here we tweak priorities to ensure that the compiler choice is above variant penalty.

Signed-off-by: Massimiliano Culpo <[email protected]>

* installer: improve spawn/forkserver (#52127)

* In the forkserver case, preload a few modules needed in the installer
* Avoid the 8k stat calls in all build processes

Signed-off-by: Harmen Stoppels <[email protected]>

* solver: remove unused `asp` attribute from `Result` (#52136)

Signed-off-by: Massimiliano Culpo <[email protected]>

* debug.py: avoid pdb import, simplify (#52139)

Importing pdb is problematic cause it pulls in readline, which makes
it impossible to run `spack install &` in the background on macOS, as
it queries stdin and gets immediately suspended as a result.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer: mark explicit if already installed but implicit in db (#52120)

Signed-off-by: Harmen Stoppels <[email protected]>

* solver: switch version_constraints from set of tuples to dictionary (#52140)

Signed-off-by: Massimiliano Culpo <[email protected]>

* Mirror all skip placeholder packages (#51991)

Placeholder (or stub) packages for externals do not have a url or vcs to fetch from. They should not error out for `spack mirror -a`

* Skip placeholder packages

Signed-off-by: Angelica Loshak <[email protected]>

* unit tests skipping placeholder pkgs

Signed-off-by: Angelica Loshak <[email protected]>

* Placeholder mock pkg

Signed-off-by: Angelica Loshak <[email protected]>

---------

Signed-off-by: Angelica Loshak <[email protected]>

* build(deps): bump black in /.github/workflows/requirements/style (#52068)

Bumps [black](https://github.com/psf/black) from 25.12.0 to 26.3.1.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/25.12.0...26.3.1)

---
updated-dependencies:
- dependency-name: black
  dependency-version: 26.3.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Revert "build(deps): bump black in /.github/workflows/requirements/style (#52068)" (#52147)

This reverts commit 0875e9be639bc4c8cfafeb74bfe6d7e678d1bbef.

Signed-off-by: Harmen Stoppels <[email protected]>

* config.yaml: default to `installer: new` (#52149)

Use the new installer by default on Linux and Darwin.

Dispatch to old installer when using unsupported features (splicing, --until).

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: support --until (#52123)

Signed-off-by: Harmen Stoppels <[email protected]>

* remote include path: cache under enclosing scope ({spack|include}.yaml) path or tempdir (from #50207) (#51896)

* remote include path: cache under enclosing scope

Cache remote include paths under the enclosing scope's path. If it
has no path, the file will be cached in tmpdir

* Update include yaml docs

* cleanup and add unit tests for include's destination directory

* config: add extra subdirectory to support multiple remote includes

subdir naming order: name, git repo name, or hash

---------

Signed-off-by: tldahlgren <[email protected]>

* languages: add hip-lang, cuda-lang (#52145)

Signed-off-by: Harmen Stoppels <[email protected]>

* Document correct practices for specifying dependency version ranges (#52022)

* Document version range practice

Signed-off-by: Seth R Johnson <[email protected]>

* Tweak verbiage and examples

Signed-off-by: Seth R Johnson <[email protected]>

* Fix typo

Signed-off-by: Seth R Johnson <[email protected]>

* Incorporate feedback

Signed-off-by: Seth R Johnson <[email protected]>

* Incorporate comments

Signed-off-by: Seth R Johnson <[email protected]>

---------

Signed-off-by: Seth R Johnson <[email protected]>

* new_installer.py: suspend, background, foreground (#52101)

* new_installer.py: Ctrl+Z, bg, and fg

When the user presses Ctrl+Z, the parent process was stopped by SIGTSTP
but its build subprocesses kept running, consuming CPU and filling log
pipes until they blocked on write.

Fix this by broadcasting SIGSTOP to every child process group just before
the parent suspends, and SIGCONT when it resumes. Because children call
os.setsid(), one killpg() per child stops the entire tree of compiler
and build-system subprocesses.

The changes are structured as follows:

* Add TerminalState: a new class that owns all terminal-related setup
  and teardown: cbreak mode, stdin selector registration, SIGWINCH
  self-pipe, and the SIGTSTP handler. Two optional hooks, on_suspend and
  on_resume, let callers register side-effects without coupling
  TerminalState to child process management. The installer wires these
  up to _signal_children().

* Add headless mode to the UI class: when True, update(), print_logs(),
  and the non-TTY path of update_state() are all suppressed.
  TerminalState sets this flag on suspend and clears it on resume; it
  also cleared when the process transitions from background to
  foreground. When in headless mode, the event loop fires only once a
  second since there are no spinners to update; but we *do* have to
  check every now and then whether we went to the foreground, which I
  think is only possible through polling. So, once a second we detect
  whether we go from headless back to ... headful?

* Reset SIGTSTP to SIG_DFL in worker_function so child processes do not
  inherit the parent's handler after forking.

* Add TestHeadlessMode unit tests covering the three suppression paths.

Signed-off-by: Harmen Stoppels <[email protected]>

* fix for zsh: output stdout = suspend, so do not print

Signed-off-by: Harmen Stoppels <[email protected]>

---------

Signed-off-by: Harmen Stoppels <[email protected]>

* conf.py: do not document re-export of spack.package classes (#52169)

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: elapsed time (#52163)

* new_installer.py: print full line on finish

If a build finishes, we can print the full status line instead of
truncating it so it fits the terminal width.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: duration

Print

```
[+] gzo2zcw [email protected] /tmp/x/darwin-m4/pkgconf-2.5.1-gzo2zcwlb4xijoa55hvqdllvwp6l4z6n (5s)
[+] 4b4piwd [email protected] /tmp/x/darwin-m4/berkeley-db-18.1.40-4b4piwdndd6dqu35duc2lipafivfhfvy (23s)
[+] uybprz4 [email protected] /tmp/x/darwin-m4/libiconv-1.18-uybprz4tzlcwzxeebqzk2nh5znuagvth (24s)
[+] koh6kwu [email protected] /tmp/x/darwin-m4/ncurses-6.6-koh6kwuju3yay4r47ol6abfy7lpc434a (1m04s)
[+] fr3f5o6 [email protected] /tmp/x/darwin-m4/readline-8.3-fr3f5o6i2ubrkpnw4ia64qpi2uqnvtsz (11s)
[+] lu4rfdj [email protected] /tmp/x/darwin-m4/diffutils-3.12-lu4rfdjlepbltqqouzjmfyzd4nuyt36p (1m02s)
[+] 6thd5gj [email protected] /tmp/x/darwin-m4/bzip2-1.0.8-6thd5gjm42tbtq5mqbuqspmgfrdsstia (2s)
[+] juwuxzu [email protected] /tmp/x/darwin-m4/gdbm-1.26-juwuxzudiwkbuvdj2im6uzff2qr3jjl3 (11s)
[+] jm5dx2t [email protected] /tmp/x/darwin-m4/nghttp2-1.67.1-jm5dx2tciwjf7gey23cbqouypgmlz7n4 (14s)
[+] 4tnhqsm [email protected] /tmp/x/darwin-m4/perl-5.42.0-4tnhqsmre6iamiwcg6hn6nbowvse6332 (1m32s)
[+] ku62zd5 [email protected] /tmp/x/darwin-m4/openssl-3.6.1-ku62zd5nsontauvl5vw7o27cfofzp42a (39s)
[+] lkzwnfg [email protected] /tmp/x/darwin-m4/curl-8.18.0-lkzwnfgfvjff6ve6lpfc2w6laff42zjm (49s)
```

Signed-off-by: Harmen Stoppels <[email protected]>

---------

Signed-off-by: Harmen Stoppels <[email protected]>

* docs: pygments v2.20.0 (#52171)

* new_installer.py: fix cursor movement compatibility (#52173)

`\033[<n>A` (Cursor Up) and `\033[<n>B` (Cursor Down) have been part of
the original VT100 spec since 1978, so every terminal emulator should
support them, including JuiceSSH.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: docs (#52164)

Signed-off-by: Harmen Stoppels <[email protected]>

* ci: print spec in each rebuild job (#49021)


Signed-off-by: Gregory Becker <[email protected]>

* Update Completion: turn off argparse color for 3.14 and newer (#52174)

Argparse color adds of color control characters in Python >= 3.14,
which breaks "spack commands --update-completion". Add logic to
check whether output is a terminal and turn off color when it is not.

---------

Signed-off-by: John Parent <[email protected]>

* docs: minor modification on the example (#52180)

* repo: improve patch lookup speed (#52157)

This commit avoids hitting the 8k stat call penalty when looking up
patch files in most cases.

Previously the strategy was:

* check cache validity (8k stat calls)
* lookup the patch by shasum in cache

With this commit we add an initial "optimistic" lookup in possibly stale
cache. It's expected that we hit this code path a lot as patches directives
are rarely changed.

* lookup the patch by shasum in possibly stale cache
* validate the entry by comparing with package class metadata
* early return if the same
* otherwise check ache validity (8k stat calls)
* lookup the patch by shasum in cache

This helps with the installer in the forkserver and spawn during staging
of patches, which can now be done without validating freshness of the
patch index.

Signed-off-by: Harmen Stoppels <[email protected]>

* spec.py: avoid cache validation in satisfies (#52176)

The goal is to avoid consulting the virtual provider lookup during `Spec.satisfies`, in particular in the common case of concrete lhs.

This is accomplished as follows:

1. The left-hand side provides edge attributes to which the right-hand side nodes can be matched.
2. If the right hand side is a known virtual and merely mentions a name `%mpi`, exit early
3. The only virtual attribute to (very rarely) check for satisfaction is the version `mpi@3`; here lookup the package metadata instead of the provider cache in case the left-hand side is concrete.

On top of that, various code paths are unified that were incorrectly considered "exceptions to the rule". There is no branching on concrete/abstract and direct/transitive deps.

The assumption in this PR is that in the case of right-hand side abstract, its size is O(1) so that it's worst case linear time (early return). The previous implementation was best case linear time in the `^pkg` case.

The behavior of `satisfies(%possible-provider, ^virtual)` is defined in a test. That statement is false since you can depend on a package that could provide a virtual without depending on the virtual.

A bug in a test was fixed, which asserted `s.satisfies("%c,cxx,fortran=gcc")`, even though `s` did not depend on `fortran`.

Signed-off-by: Massimiliano Culpo <[email protected]>
Signed-off-by: Harmen Stoppels <[email protected]>
Co-authored-by: Massimiliano Culpo <[email protected]>

* tags.py: fix integration test (#52184)

Signed-off-by: Harmen Stoppels <[email protected]>

* environment.py: use tag from pkg for view regen (#52182)

The logic to exclude packages tagged "runtime" from views triggers cache
validation, resulting in 8k stat calls.

Avoid this by querying the package classes.

This is needed to make `spack install` not trigger many stat calls on
package.py files for a concrete environment.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: remove pkg->mtime dict ipc (#52183)

With recent changes there is no need anymore to communicate the package
to mtime dictionary to build subprocesses. Remove this to reduce
subprocess startup latency by ~15ms per process.

Signed-off-by: Harmen Stoppels <[email protected]>

* main.py: cache spack commit (#52185)

Signed-off-by: Harmen Stoppels <[email protected]>

* build(deps): bump mypy in /.github/workflows/requirements/style (#52178)

Bumps [mypy](https://github.com/python/mypy) from 1.19.1 to 1.20.0.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.19.1...v1.20.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-version: 1.20.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* new_installer.py: new process group instead of session (#52192)

In build subprocesses, create a new process group instead of a new session.
The benefit is that tools like pstree neatly show the spack install process
with all its concurrent builds as a tree, making it easier to verify whether
the jobserver is working correctly: -j N <--> N leaf nodes.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer: make build cache index optional (#52188)

Signed-off-by: Victor Brunini <[email protected]>

* schema: reduce schema size with $ref and definitions (#52206)

Use jsonschema $ref / definitions to deduplicate repeated sub-schemas
(env modifications, projections, module file configuration, CI job
attributes) across standalone and merged schemas.

This reduces the size as JSON by a factor 3 to 4.

Signed-off-by: Harmen Stoppels <[email protected]>

* environment.py: minor comment fix  (#52213)

Remove em-dash from comment

Signed-off-by: Harmen Stoppels <[email protected]>

* docs: cleanup installer output (#52204)

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: defensive prefix checks (#52146)

* Ensure prefix uniqueness
* Ensure no install in upstream

Signed-off-by: Harmen Stoppels <[email protected]>

* Pin remaining versioned GHAs to SHAs (#52218)

* Pin julia actions to shas

Signed-off-by: John Parent <[email protected]>

* Better julia cache pin

Signed-off-by: John Parent <[email protected]>

* Better setup julia pin

Signed-off-by: John Parent <[email protected]>

---------

Signed-off-by: John Parent <[email protected]>

* display_specs: display abstract hash if present (#52219)

* display_specs: display abstract hash if present

Abstract specs may have an abstract hash and no name, even in contexts
in which anonymous specs are generally not allowed.

This commit updates the default format string for `display_specs` to
display the abstract hash if one exists. Otherwise, we pass empty
strings to `tty.colify` and get a division by 0 error.

---------

Signed-off-by: Gregory Becker <[email protected]>

* lock.py: fix release_write when _read > 0 (#52202)

Fix a bug where

```
lock.acquire_read()
lock.acquire_write()
lock.release_write()
```

would hold an exclusive lock afterwards; it should be a shared lock.

Signed-off-by: Harmen Stoppels <[email protected]>

* database.py: fix toctou bugs (#52200)

Use try-open-except instead of if-exists-open, etc.

Signed-off-by: Harmen Stoppels <[email protected]>

* file_cache.py: single-file byte-range locking (#52199)

Replace the multiple .lock file approach with fcntl byte-range locking
on a single global lockfile.

Works well for the installer, and reduces the number of open files a
bit.

Signed-off-by: Harmen Stoppels <[email protected]>

* installer.py: avoid chmod when --fake (#52196)

Signed-off-by: Harmen Stoppels <[email protected]>

* installer.py: --until <last phase> (#52195)

Currently `--until install` is handled as an exceptional case in the old
installer where it simplify continues installation as if no flag was
passed.

This makes it impossible to run until install without registering the
spec in the database, which could be a valid use case. Also makes it
hard to troubleshoot post-install hooks.

This change removes the exceptional case so that the builds stops after
the last phase. It adapts integrations tests accordingly, ensuring the
behavior is the same for the old and new installer.

Signed-off-by: Harmen Stoppels <[email protected]>

* spack env create: do not call regen views (#52190)

When creating an empty view, do just that. The view regenerate may
trigger an unnecessary database read, which is just overhead.

Only generate views if the env is initialized from a spack.lock file.

Signed-off-by: Harmen Stoppels <[email protected]>

* environment: replace included environment list attributes with ConcretizedRootInfo (#52217)

`included_concretized_user_specs` and `included_concretized_order` were two parallel
`Dict[str, List[...]]` that had to be kept in sync by hand, resulting in a `zip()`
that could silently truncate if they diverged.

Replace both with a single `included_concretized_roots: Dict[str, List[ConcretizedRootInfo]]`,
mirroring the pattern already used for the main environment's `concretized_roots`.

Signed-off-by: Massimiliano Culpo <[email protected]>

* Don't drop compiler flags when `packages:all:require` sets a target (#52133)

Compiler flags added to a compiler definition were not applied 
in the following circumstances:

- The user sets packages:all:require:[target=x]
- The default target for a system does not match x
- The compiler definition does not include a target

Signed-off-by: Peter Josef Scheibel <[email protected]>
Signed-off-by: Peter Scheibel <[email protected]>

* new_installer.py: sub_process < /dev/null (#52221)

In the Python build subprocess, redirect stdin to /dev/null. This
regressed after #52192 as that change attaches the user's stdin.

Signed-off-by: Harmen Stoppels <[email protected]>

* spack buildcache push: add --group argument (#52224)

When an environment is active, --group can be used
to push only the specs from that group. The option
can be given multiple times.

Signed-off-by: Massimiliano Culpo <[email protected]>

* new_installer.py: fix line wrap issues (#52193)

In the "active area" of the terminal UI, downwards cursor movement is
used instead of `\n` to prevent flickering. The variable
`active_area_rows` keeps track on how many lines cursor movement can be
used, and any new builds would use `\n` instead (to scroll the terminal
when needed).

However, after #52163 the finished builds were printed *without*
truncating to terminal width, and typically exceed it. This means they
can consume multiple lines from the "active area" of running builds, and
as a result the new "active area" is actualy fewer lines.

To fix this, reset `active_area_rows` to 0, which forces `\n` instead of
cursor movement.

Use the same trick when the terminal resizes. A narrower/wider terminal
can result in line wrapping, so better not to rely on cursor movement as
much.

Signed-off-by: Harmen Stoppels <[email protected]>

* file_cache: drop init_entry/mtime and simplify (#52201)

Remove `FileCache.init_entry` and `FileCache.mtime` because they were
TOCTOU-prone (check existence/permissions, then open later) and required
every call site to remember a separate initialization step.

Instead, `read_transaction` and `write_transaction` are now
`@contextmanager` generators that acquire locks and open files directly:

- `read_transaction` yields an open file or `None` if missing.
- `write_transaction` yields `(old_file_or_none, new_file)` and creates
  parent directories on demand.

Errors (permission, not-a-file) are raised as `CacheError` at the point
of use rather than checked upfront with `os.access`.

Call sites in `binary_distribution`, `compilers/libraries`, `repo`, and
`git_ref_lookup` are updated to drop `init_entry` calls and handle the
`None` case from `read_transaction`.

Signed-off-by: Harmen Stoppels <[email protected]>

* Add more tests for error messages (#52228)

Add three parametrized tests for error messages, partitioned
by the "main" cause of the error:
1. User input on the CLI
2. User configuration
3. package.py directives

These tests capture the status quo, and check that the
error messages contain specific strings.

Signed-off-by: Massimiliano Culpo <[email protected]>

* LockTransaction: fix typing, context-manager path (#52203)

`LockTransaction.__init__` accepted acquire as

```
Union[ReleaseFnType, ContextManager]
```

and `__enter__` detected whether acquire returned a context manager to
nest into. No caller uses this: all pass plain callables or `None`.

- Remove context-manager detection from __enter__/__exit__ and the
  self._as instance variable
- Type acquire as `Optional[Callable[[], None]]`
- Add `ExitFnType` alias and type release as `Optional[ExitFnType]`,
  fixing the previous incorrect typing.
- Fix `ReleaseFnType` return to `Optional[bool]` and coerce with
  `bool()` in release_read/release_write to satisfy mypy
- Raise `NotImplementedError` instead of returning NotImplemented in
  _enter/_exit
- Remove `test_transaction_with_context_manager` which exclusively
  tested the removed feature; exception suppression via release
  returning True is still covered by `test_transaction_with_exception`

Signed-off-by: Harmen Stoppels <[email protected]>

* reporters: fix hard failure with non-UTF-8 logs (#52235)

Use `errors="replace"` to avoid exceptions when creating report entries.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: use buffered io in log (#52231)

The Tee thread used `os.write`, which returns the number of bytes
written, possibly less than the input. That's a data loss risk fixed by
using buffered io.

Presumably we wouldn't hit this, cause the log is on the same file
system as the build, but still good to fix. The original consideration
was that we should not block a build over logs, but arguably having
broken logs is worse.

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: fix stdin double buffering bug (#52234)

Fix a bug where pasting a dag hash or package name in `/` mode of the
TUI would result in only a single character being added to the filter.

The cause of this is double buffering (kernel + TextIOWrapper). The
idea was to read one character per event loop iteration. But after
`sys.stdin.read(1)` multiple bytes are `os.read` from the stdin fd,
draining the pipe and moving the data into the TextIOWrapper.
Another key press is needed for that to be read in a later iteration
of the event loop.

The fix does a bit more:

* handle multiple characters per event loop iteration so pasting text
  isn't laggy
* filter out ansi escape characters for cursor movement

The only buffering now is for multi-byte UTF-8 chars, which is
desirable.

* core: fix two typos (#52240)

Signed-off-by: Harmen Stoppels <[email protected]>

* new_installer.py: extract two functions (#52241)

Signed-off-by: Harmen Stoppels <[email protected]>

* environment groups: add "explicit" attribute (#52244)

With this attribute users can control whether root
specs from groups are "explicit" or not (default is True).

Root specs from `explicit: False` groups are eligible
for garbage collection.

Signed-off-by: Massimiliano Culpo <[email protected]>

* setup-env.sh: speed up when no module command (#52245)

The `setup-env.sh` script was slow because it triggers a database search
for `environment-modules`, that's bad UX and an unnecessary implicit
preference over `lmod`.

This commit speeds up `source setup-env.sh` for users who do not have
`module` available as a shell function.

The user is responsible for making `module` available in their shell, and if
so, there's a small startup cost as Spack is queried for MODULEPATH.

Signed-off-by: Harmen Stoppels <[email protected]>

* file_cache.py: use mkstemp instead of .tmp suffix (#52238)

Signed-off-by: Ryan Krattiger <[email protected]>

* ctest_log_parser.py: respect configured concurrency level (#52215)

Respects the general concurrency level of Spack

Prevents a bug on large Windows machines where we create a pool of > 63 thread/process handles to wait on and trip an internal win32 api limit

Signed-off-by: John Parent <[email protected]>

* Fix Dependabot config and add coverage requirements (#52253)

Signed-off-by: Alec Scott <[email protected]>

* Update actions/checkout to v6 (#52252)

* Update actions/checkout to v6
GHA is deprecating nodejs 20, new minimum require is nodejs 24.
checkout action requires at least v5, updating to latest release (6.0.2)

Signed-off-by: Ryan Krattiger <[email protected]>

* Remove labels to avoid confusing dependabot going forward

Signed-off-by: Ryan Krattiger <[email protected]>

---------

Signed-off-by: Ryan Krattiger <[email protected]>

* build(deps): bump mypy in /.github/workflows/requirements/style (#52254)

Bumps [mypy](https://github.com/python/mypy) from 1.20.0 to 1.20.1.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.20.0...v1.20.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-version: 1.20.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump julia-actions/cache from 2.1.0 to 3.0.2 (#52255)

Bumps [julia-actions/cache](https://github.com/julia-actions/cache) from 2.1.0 to 3.0.2.
- [Release notes](https://github.com/julia-actions/cache/releases)
- [Commits](https://github.com/julia-actions/cache/compare/d10a6fd8f31b12404a54613ebad242900567f2b9...9a93c5fb3e9c1c20b60fc80a478cae53e38618a4)

---
updated-dependencies:
- dependency-name: julia-actions/cache
  dependency-version: 3.0.2
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump docker/setup-buildx-action from 3.8.0 to 4.0.0 (#52256)

Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.8.0 to 4.0.0.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Comm…