Fortran dependency can cause errors in mixed compiler cases with unify:true

The mere existence of a fortran dependency in two different packages can cause havok when concretizing mixed compiler cases. This was observed with a BLAS/LAPACK library depending on fortran and another package also adding a fortran dependency. Note the reproducer shows that there doesn't even have to be a direct depending between the two packages involved.

Steps to reproduce

1. Create two simple Spack packages

$ SPACK_PACKAGES_DIR=$(dirname $(spack location -p zlib))
$ mkdir -p $SPACK_PACKAGES_DIR/{my_package,other_package}

$ cat > $SPACK_PACKAGES_DIR/my_package/package.py <<EOF
from spack_repo.builtin.build_systems.generic import Package
from spack.package import *

class MyPackage(Package):
    version("1.2.4")

    depends_on("c", type="build")
    depends_on("cxx", type="build")
    # depends_on("fortran", type="build")  # fails to concretize if I add this
EOF

$ cat > $SPACK_PACKAGES_DIR/other_package/package.py <<EOF
from spack_repo.builtin.build_systems.generic import Package
from spack.package import *

class OtherPackage(Package):
    version("1.2.4")

    depends_on("c", type="build")
    depends_on("fortran", type="build")
EOF

2. Create test environment

$ mkdir testenv
$ cat > testenv/spack.yaml <<EOF
spack:
  specs:
  - my-package %c,cxx=clang
  - other-package %c,fortran=gcc
  view: true
  concretizer:
    unify: true
EOF

3. Activate and concretize the environment without the fortran dependency in my-package

$ spack env activate -d testenv
$ spack concretize -f --fresh
==> Concretized 2 specs:
 -   l7jy6na  [email protected] build_system=generic platform=linux os=ubuntu24.04 target=sapphirerapids %c,[email protected]
[+]  7wntybw      ^[email protected] build_system=generic platform=linux os=ubuntu24.04 target=sapphirerapids 
[e]  4bfivdk      ^[email protected] build_system=autotools platform=linux os=ubuntu24.04 target=x86_64 
[e]  3v4noyf      ^[email protected]+clang~cuda~flang+gold+libomptarget~libomptarget_debug~link_llvm_dylib+lld+lldb+llvm_dylib+lua~mlir+polly~python~split_dwarf~utils~z3~zstd build_system=cmake build_type=Release compiler-rt=runtime generator=ninja libcxx=runtime libunwind=runtime openmp=runtime shlib_symbol_version=none targets:=all version_suffix=none platform=linux os=ubuntu24.04 target=x86_64 
 -   2cmunxl  [email protected] build_system=generic platform=linux os=ubuntu24.04 target=sapphirerapids %c,[email protected]
[e]  pn4nide      ^[email protected]~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=ubuntu24.04 target=sapphirerapids 
 -   qvnny37      ^[email protected] build_system=generic platform=linux os=ubuntu24.04 target=sapphirerapids

4. Add the fortran dependency to my-package and try to reconcretize

$ spack edit my-package # and add dependency
$ spack concretize -f --fresh
==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    [my-package %c,cxx=clang, other-package %c,fortran=gcc] is unsatisfiable. You could consider setting `concretizer:unify` to `when_possible` or `false` to allow multiple versions of some packages.

With unify:when_possible you get:

$ spack concretize -f --fresh
==> Concretized 2 specs:
 -   4c2tmr4  [email protected] build_system=generic platform=linux os=ubuntu24.04 target=sapphirerapids %[email protected] %c,[email protected]
[+]  7wntybw      ^[email protected] build_system=generic platform=linux os=ubuntu24.04 target=sapphirerapids 
[e]  pn4nide      ^[email protected]~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=ubuntu24.04 target=sapphirerapids 
 -   qvnny37      ^[email protected] build_system=generic platform=linux os=ubuntu24.04 target=sapphirerapids 
[e]  4bfivdk      ^[email protected] build_system=autotools platform=linux os=ubuntu24.04 target=x86_64 
[e]  3v4noyf      ^[email protected]+clang~cuda~flang+gold+libomptarget~libomptarget_debug~link_llvm_dylib+lld+lldb+llvm_dylib+lua~mlir+polly~python~split_dwarf~utils~z3~zstd build_system=cmake build_type=Release compiler-rt=runtime generator=ninja libcxx=runtime libunwind=runtime openmp=runtime shlib_symbol_version=none targets:=all version_suffix=none platform=linux os=ubuntu24.04 target=x86_64 
 -   2cmunxl  [email protected] build_system=generic platform=linux os=ubuntu24.04 target=sapphirerapids %c,[email protected]

Also note, just adding %fortran=gcc also doesn't help:

[my-package %fortran=gcc %c,cxx=clang, other-package %c,fortran=gcc] is unsatisfiable. You could consider setting `concretizer:unify` to `when_possible` or `false` to allow multiple versions of some packages.

Error message

$ spack --debug --stacktrace concretize -f --fresh
lib/spack/spack/cmd/__init__.py:126 ==> [2025-11-05-10:32:38.871477] Imported concretize from built-in commands
lib/spack/spack/cmd/__init__.py:126 ==> [2025-11-05-10:32:38.871987] Imported concretize from built-in commands
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:38.875121] Reading config from file /home/rberger/projects/devops/testenv/spack.yaml
lib/spack/spack/environment/environment.py:257 ==> [2025-11-05-10:32:38.876299] Deactivated environment '/home/rberger/projects/devops/testenv'
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:38.879629] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/repos.yaml
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:38.880732] Reading config from file /home/rberger/projects/devops/spack/etc/spack/repos.yaml
lib/spack/spack/environment/environment.py:230 ==> [2025-11-05-10:32:38.882691] Using environment '/home/rberger/projects/devops/testenv'
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:38.882960] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/concretizer.yaml
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:38.890185] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/packages.yaml
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:38.905286] Reading config from file /home/rberger/projects/devops/spack/etc/spack/packages.yaml
lib/spack/spack/compilers/config.py:277 ==> [2025-11-05-10:32:38.990412] [/home/rberger/projects/devops/spack/lib/spack/spack/compilers/config.py] The external spec '[email protected]' cannot be used as a compiler: missing the 'extra_attributes' key
lib/spack/spack/database.py:610 ==> [2025-11-05-10:32:39.193632] DATABASE LOCK TIMEOUT: 60s
lib/spack/spack/store.py:179 ==> [2025-11-05-10:32:39.193774] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/compilers/config.py:277 ==> [2025-11-05-10:32:39.197561] [/home/rberger/projects/devops/spack/lib/spack/spack/compilers/config.py] The external spec '[email protected]' cannot be used as a compiler: missing the 'extra_attributes' key
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:39.705997] [spack.solver.input_analysis] apple-clang is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:39.707376] [spack.solver.input_analysis] msvc is not for this platform
lib/spack/spack/database.py:610 ==> [2025-11-05-10:32:39.711157] DATABASE LOCK TIMEOUT: 60s
lib/spack/spack/store.py:179 ==> [2025-11-05-10:32:39.711296] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:39.863979] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/environment/environment.py:257 ==> [2025-11-05-10:32:39.867179] Deactivated environment '/home/rberger/projects/devops/testenv'
lib/spack/spack/bootstrap/config.py:128 ==> [2025-11-05-10:32:39.867617] [BOOTSTRAP CONFIG SCOPE] name=_builtin
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:39.868057] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/bootstrap/config.py:137 ==> [2025-11-05-10:32:39.870846] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/home/rberger/projects/devops/spack/etc/spack/defaults
lib/spack/spack/bootstrap/config.py:137 ==> [2025-11-05-10:32:39.870955] [BOOTSTRAP CONFIG SCOPE] name=bootstrap, path=/home/rberger/projects/devops/.spack/bootstrap/config
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:39.871125] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/include.yaml
lib/spack/spack/config.py:983 ==> [2025-11-05-10:32:39.872200] Creating DirectoryConfigScope defaults:linux for '/home/rberger/projects/devops/spack/etc/spack/defaults/linux'
lib/spack/spack/config.py:983 ==> [2025-11-05-10:32:39.872360] Creating DirectoryConfigScope defaults:base for '/home/rberger/projects/devops/spack/etc/spack/defaults/base'
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:39.872955] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/config.yaml
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:39.880196] Reading config from file /home/rberger/projects/devops/.spack/bootstrap/config/config.yaml
lib/spack/spack/database.py:610 ==> [2025-11-05-10:32:39.884219] DATABASE LOCK TIMEOUT: 60s
lib/spack/spack/store.py:179 ==> [2025-11-05-10:32:39.884346] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:39.891402] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/packages.yaml
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:39.905797] Reading config from file /home/rberger/projects/devops/.spack/bootstrap/config/packages.yaml
lib/spack/spack/bootstrap/_common.py:239 ==> [2025-11-05-10:32:40.046695] [BOOTSTRAP ROOT SPEC] clingo-bootstrap@spack+python platform=linux target=x86_64
lib/spack/spack/bootstrap/core.py:358 ==> [2025-11-05-10:32:40.046932] [BOOTSTRAP MODULE clingo] Try importing from Python
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:40.047470] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/bootstrap/_common.py:97 ==> [2025-11-05-10:32:40.061601] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python platform=linux target=x86_64 ^[email protected]/et3twemsecqryfzl23e3cmsbca534dlo" provides the "clingo" Python module
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:40.062269] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/config.yaml
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:40.069354] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/repos.yaml
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:40.070463] Reading config from file /home/rberger/projects/devops/spack/etc/spack/repos.yaml
lib/spack/spack/environment/environment.py:230 ==> [2025-11-05-10:32:40.072302] Using environment '/home/rberger/projects/devops/testenv'
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:40.079268] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/packages.yaml
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:40.094315] Reading config from file /home/rberger/projects/devops/spack/etc/spack/packages.yaml
lib/spack/spack/compilers/config.py:277 ==> [2025-11-05-10:32:40.107339] [/home/rberger/projects/devops/spack/lib/spack/spack/compilers/config.py] The external spec '[email protected]' cannot be used as a compiler: missing the 'extra_attributes' key
lib/spack/spack/solver/asp.py:3502 ==> [2025-11-05-10:32:40.302621] the C compiler /home/rberger/projects/devops/spack/opt/spack/linux-sapphirerapids/compiler-wrapper-1.0-yhgl6mgngb2y5ndqfoimbn7gyqgue5ia/libexec/spack/clang/clang does not exist, or does not run correctly. The compiler [email protected]/bxspl2eebm7d5ksssvzojgza3xigylpd will not be used during concretization.
lib/spack/spack/config.py:1557 ==> [2025-11-05-10:32:40.316659] Reading config from file /home/rberger/projects/devops/spack/etc/spack/defaults/base/concretizer.yaml
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:40.387464] [spack.solver.input_analysis] apple-clang is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:40.387625] [spack.solver.input_analysis] msvc is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:40.459203] [spack.solver.input_analysis] apple-libuuid is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:40.521196] [spack.solver.input_analysis] apple-gl is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:40.522707] [spack.solver.input_analysis] wgl is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:40.814945] [spack.solver.input_analysis] msmpi is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:41.162029] [spack.solver.input_analysis] veclibfort is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:41.270677] [spack.solver.input_analysis] nvpl-blas is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:41.274462] [spack.solver.input_analysis] nvpl-lapack is not for this platform
lib/spack/spack/solver/input_analysis.py:95 ==> [2025-11-05-10:32:41.776193] [spack.solver.input_analysis] apple-libunwind is not for this platform
lib/spack/spack/compilers/config.py:277 ==> [2025-11-05-10:32:41.782469] [/home/rberger/projects/devops/spack/lib/spack/spack/compilers/config.py] The external spec '[email protected]' cannot be used as a compiler: missing the 'extra_attributes' key
lib/spack/spack/compilers/config.py:277 ==> [2025-11-05-10:32:41.928608] [/home/rberger/projects/devops/spack/lib/spack/spack/compilers/config.py] The external spec '[email protected]' cannot be used as a compiler: missing the 'extra_attributes' key
lib/spack/spack/main.py:1131 ==> [2025-11-05-10:32:51.678478] SolverError: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    [other-package %c,fortran=gcc, my-package %c,cxx=clang] is unsatisfiable. You could consider setting `concretizer:unify` to `when_possible` or `false` to allow multiple versions of some packages.
lib/spack/spack/error.py:58 ==> [2025-11-05-10:32:51.678661] Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    [other-package %c,fortran=gcc, my-package %c,cxx=clang] is unsatisfiable. You could consider setting `concretizer:unify` to `when_possible` or `false` to allow multiple versions of some packages.
Traceback (most recent call last):
  File "/home/rberger/projects/devops/spack/lib/spack/spack/main.py", line 1124, in main
    return _main(argv)
           ^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/main.py", line 1076, in _main
    return finish_parse_and_run(parser, cmd_name, args, env_format_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/main.py", line 1107, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/main.py", line 651, in _invoke_command
    return_val = command(parser, args)
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/cmd/concretize.py", line 44, in concretize
    concretized_specs = env.concretize(tests=tests)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/environment/environment.py", line 1536, in concretize
    return self._concretize_together(tests=tests)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/environment/environment.py", line 1641, in _concretize_together
    concretized_specs = spack.concretize.concretize_together(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/concretize.py", line 54, in concretize_together
    concrete_specs = _concretize_specs_together(to_concretize, tests=tests)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/concretize.py", line 37, in _concretize_specs_together
    result = Solver().solve(abstract_specs, tests=tests, allow_deprecated=allow_deprecated)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/solver/asp.py", line 4158, in solve
    result, _, _ = self.solve_with_stats(specs, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/solver/asp.py", line 4146, in solve_with_stats
    result = self.driver.solve(
             ^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/solver/asp.py", line 1257, in solve
    result = self._run_clingo(specs, setup, problem_repr, control_file_paths, timer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rberger/projects/devops/spack/lib/spack/spack/solver/asp.py", line 1156, in _run_clingo
    result.raise_if_unsat()
  File "/home/rberger/projects/devops/spack/lib/spack/spack/solver/asp.py", line 447, in raise_if_unsat
    raise SolverError(constraints, conflicts=conflicts)
spack.solver.asp.SolverError: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    [other-package %c,fortran=gcc, my-package %c,cxx=clang] is unsatisfiable. You could consider setting `concretizer:unify` to `when_possible` or `false` to allow multiple versions of some packages

Information on your system

Spack: 1.1.0.dev0 (75611a9)
Builtin repo: spack/spack-packages@4b3ea86
Python: 3.12.3
Platform: linux-ubuntu24.04-sapphirerapids

General information

I have run spack debug report and reported the version of Spack/Python/Platform
I have searched the issues of this repo and believe this is not a duplicate
I have run the failing commands in debug mode and reported the output

Confirmed. Thanks for the reproducer, I'm looking into it right now. Will post any new findings in comments.

Possibly solved by #51923, except it takes a very long time to find an optimal solution for me. I'll check if we can speed this case up easily.

@alalazo

@alalazo I'm observing very long concretizations in the reproducer, even in the non-error case (10min+). For all I know the concretizer could be bitcoin mining in the meantime. It's also faster to build a fresh LLVM than having this simple case concretized, which is a bit concerning. 😉

Finished a bugfix for #51995 Now looking into this one.

@alalazo

@alalazo I've also tried to see if #52015 fixes this one. Interestingly, the non-error case is fast again. Tested both on Linux and macOS Tahoe. So at least we're only dealing with the slow solve for the error case now.

I found the issue here. The cause is similar to the other issue: we use provider where we should instead use a more accurate information on edges. I think if you apply this diff:

diff --git a/lib/spack/spack/solver/concretize.lp b/lib/spack/spack/solver/concretize.lp
index 8267687497..392aa21df8 100644
--- a/lib/spack/spack/solver/concretize.lp
+++ b/lib/spack/spack/solver/concretize.lp
@@ -981,19 +981,6 @@ error(1, Msg) :-
 %-----------------------------------------------------------------------------
 
 % Enforces all virtuals to be provided, if multiple of them are provided together
-error(100, "Package '{0}' needs to provide both '{1}' and '{2}' together, but provides only '{1}'", Package, Virtual1, Virtual2)
-:- % This package provides 2 or more virtuals together
-   condition_holds(ID, node(X, Package)),
-   pkg_fact(Package, provided_together(ID, SetID, Virtual1)),
-   pkg_fact(Package, provided_together(ID, SetID, Virtual2)),
-   Virtual1 != Virtual2,
-   % One node depends on those virtuals AND on this package
-   node_depends_on_virtual(ClientNode, Virtual1),
-   node_depends_on_virtual(ClientNode, Virtual2),
-   depends_on(ClientNode, node(X, Package)),
-   % But this package is a provider of only one of them
-   provider(node(X, Package), node(_, Virtual1)),
-   not provider(node(X, Package), node(_, Virtual2)).
 
 % if a package depends on a virtual, it's not external and we have a
 % provider for that virtual then it depends on the provider

you should be able to concretize this case (meaning that is the rule to fix).

solver: fix rule for virtuals that are provided together

solver: fix rule for virtuals that are provided together (spack#52021)

Update spack-stack-dev from spack develop 2026/05/20 (#577)

Navigation Menu

Repository navigation

Fortran dependency can cause errors in mixed compiler cases with `unify:true` #51512

Steps to reproduce

1. Create two simple Spack packages

2. Create test environment

3. Activate and concretize the environment without the fortran dependency in my-package

4. Add the fortran dependency to my-package and try to reconcretize

Error message

Information on your system

General information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Fortran dependency can cause errors in mixed compiler cases with unify:true #51512

Description

Steps to reproduce

1. Create two simple Spack packages

2. Create test environment

3. Activate and concretize the environment without the fortran dependency in my-package

4. Add the fortran dependency to my-package and try to reconcretize

Error message

Information on your system

General information

Activity

alalazo commented on Feb 3, 2026

alalazo commented on Feb 9, 2026

rbberger commented on Feb 27, 2026

alalazo commented on Mar 2, 2026

rbberger commented on Mar 2, 2026

alalazo commented on Mar 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Fortran dependency can cause errors in mixed compiler cases with `unify:true` #51512