Repository navigation
upgrade removes non-automatic package to upgrade its dependencies #2740
Description
I had a virtual machine with 14.1 that I freebsd-upgrade'd to 14.4 then 15.1 and now when trying to upgrade packages it upgraded pkg from 1.21.3 to 2.8.4, it then mentions no removal in what will be done and downloads packages then re-mentions what will be done which includes removal of a non-automatic package (linux-dwarffortress) to which I assume is because it is installing upgrades of linux-c7 ports. Manually specifying all non-automatic packages on a pkg upgrade line gives the more expected result of removing linux-c7 ports to install linux-rl7 and no longer attempts to remove the non-automatic packages.
Over the years I've been thinking some package removals were not a correct step but its normally happened on my main system with thousands of packages installed and I assumed it was probably related to times that I ran upgrades by manually listing packages to upgrade instead of upgrading all. This time I have a smaller install of just devel/git, emulators/virtualbox-ose-additions, games/linux-dwarffortress, ports-mgmt/pkg, ports-mgmt/portconfig, and x11/xorg currently listed as manually installed and autoremove says there is nothing to do. I suspect the full packageset can be pruned but haven't tried to change anything in case I can provide any feedback that leads to a fix for the solver.
If its small quantities of content to reproduce then I will likely manually type it but for big things I may need to finally figure out a way to pass data between that machine and its host as I never got around to that back when I had set it up.
If I can provide something useful then please let me know. System has boot environments so I can likely do some things including going back to the original 14.1 state.
Activity
arrowd commentedon Aug 21, 2026
You can use
pkg set -v 1to mark a package vital. Vital packages can't be removed, only upgraded.It is arguable that non-automatic should imply vital, I tend to think that it shouldn't.
bdrewery commentedon Aug 21, 2026
A problem with
pkg set -v 1is it gets cleared on upgrade if the package was not built with it, doesn't it? So it is an opinion of the package maintainer.If I install something manually why should it be eligible to be removed? Why shouldn't vital be implied? The user's opinion is they needed the package. Two conflicting perspectives here. Packager has more control than the user.
arrowd commentedon Aug 21, 2026
Ouch, indeed. That needs to be fixed, I think.
A package might get removed in the repo and it would not be possible to upgrade its dependencies. This might result in "No updates available" plan. I imagine it would happen quite often for people who rarely run
poudriere bulk -aand instead build a selected set of packages.mirror176 commentedon Aug 21, 2026
This isn't just that pkg is trying to remove a manually installed package, but that it is trying to do it to upgrade an automatic package that is only used as a dependency of the package to be removed instead of upgrading the manually installed package and blowing away auto-installed dependencies to make room for conflicting dependencies that will also be auto-installed.
I gave a shot at narrowing it down. I now have only pkg and linux-dwarffortress installed manually (39 packages total). Running
pkg upgradetries to remove linux-dwarffortress while runningpkg upgrade linux-dwarffortress pkgdoes not.It seems out of place that users currently expecting to run
pkg upgradeneed to runpkg upgrade `pkg noauto`or a non-alias command likepkg upgrade `pkg query -e "%a=0" %n`to force pkg to favor their manually installed packages over the dependencies that they need.I could see failing when upgrades will remove manually installed packages but I'm okay with it prompting. That being said, it should be 'very' clear to the user when that is happening. The question of failing or not on an upgrade is the same as the question of failing or not on an install when only part of the work can be done as the user expects;
pkg install a xorgerrors out because it cannot install 'a' (50% user expectations could succeed, run aborted) but in this casepkg upgradedoes not error out even though it is removing the only manually installed package 'linux-dwarffortress' (=100% user expectations fail, run proceeds).Alternatively I still think there is value in having the ability to perform only upgrades that meet the user's expectations without requiring the user manually go back and re-specify that packages they manually installed should be vital or locked or manually trial+error their way through individual package upgrades command to be able to abort any time an unsatisfactory result is reached. Basically a 'do what you can do and stop before you break stuff'. I understand that except when dependency trees are completely isolated that 'some' dependencies could be viewed as upgradeable when doing so would cause problems so a warning would be in order before upgrading dependencies that are not on branches isolated from the dependent packages.
Anyways, I assume my initial post is likely the result of the solver struggling with conflicts between -c7 and -rl9 linux packages and for some reason it is trying to solve those by upgrading the -c7 versions instead of wiping them out to add -rl9 which is what newer linux-dwarffortress packages now depends on.
Basic output includes stuff like
Checking integrity... done (71 conflicting)
65 when manually specifying what to upgrade[snip as I'm manually typing but there are a lot of rl9 [FreeBSD-ports] conflicting with c7 [installed] (correct) and c7 [FreeBSD-ports] (irrelevant if work puts priority on manual packages only; not present on the upgrade command that specifies only manually installed packages) which happens on a file by file conflict.]
Cannot solve problems using SAT solver, trying another plan
I usually see this when pkg seems to do unexpected thingsChecking integrity... done (0 conflicting)
then it lists things to be installed, upgraded, and reinstalled due to ABI change which ends with removing a series of linux-c7 ports and linux-dwarffortress.
I also notice that pkg now says
WARNING: 2 package(s) have known vulnerabilities. Is that is the after-upgrade counter rather than before? It could be helpful to know how many vulnerabilities are both present and will be removed and added by upgrading. Its good to be told/reminded of issues that currently exist but similarly it would be just as good to know what state you will be in after upgrading since some vulnerabilities only appear after a certain version or only once a new dependency comes in.arrowd commentedon Aug 22, 2026
Imagine the following dep graph:
Then the
libDepport gets updated to a new version, which results inlibDep-1package to be removed andlibDep-2package to be created in the repo. In consequence, bothappAandappBports get rebuilt against the newlibDep-2, but for example onlyappAsucceeds.appBfails because it is not prepared for the new version oflibDep.Now, from the client's perspective we have only 2 choices when doing an upgrade:
appAstays. This means upgradinglibDepto2, refreshingappA, and removinglibB.appBstays. This means leavinglibDep-1 in place, as well as all other packages.There is no way pkg could figure the "user expectations" here, regardless of auto and vital flags. Sometimes it is better to postpone the upgrade (pt.2, update nothing), sometimes the plan with removals (pt.1) is a correct one, because packages being removed are actually deprecated and should be replaced with different ones. Running
pkg upgrade appAorpkg upgrade appBgives the same effect as you require from the auto flag - it hints the solver that this package should stay. This is why you see different results betweenpkg upgradeandpkg upgrade dwarffortresspkg already tries to update everything without removals, so if you see removals in the plan, it means that there are conflicts and pkg proposed you one of the possible solutions. It can't magically know exactly which solution of many you'd like to accept, so you have to help it.
If you provide an exact output for
pkg upgradeandpkg upgrade dwarffortress, I can look into it and try to figure out what's going on.mirror176 commentedon Aug 22, 2026
The starting state had rl9 already in the ports tree but I think it predated when many of the ports were merged into few. The following were generated with only pkg and linux-dwarffortress in a manually installed state and autoremove said there is no work to be done. If I can gather any other useful output just let me know what commands to run or files to provide. I got a shared folder working in the VM now though I assumed the virtualbox-ose-additions port is necessary so I have added that back in after I collected this output.
pkg upgrade
pkg upgrade linux-dwarffortress
arrowd commentedon Aug 22, 2026
Is
linux_base-c7automatic in your case?mirror176 commentedon Aug 23, 2026
Yes. Rather than my usual routine of following the handbook instructions to get it first with a separate command, I had used the VM to try to create a clean and completely slimmed down test environment so just started with the basic pkg install command on what I wanted. I don't think I left the installed packages in any weird incomplete or modified state at that point but it was a while since I last touched the VM.
A branch of the dependency tree (don't know how to formally generate a complete one) linux_base-c7 -> linux-c7-elfutils-libelf -> linux-c7-dri -> linux-dwarffortress
arrowd commentedon Aug 23, 2026
Just to be sure, could you post the output of
pkg query -e %a=0 %n?mirror176 commentedon Aug 23, 2026
As I stated earlier, it now has a 3rd line because I reinstalled that package to do the data transfer though I didn't test if it is required for virtualbox shared folders. When I captured the above 2 pkg upgrade outputs I only had the first 2 installed. Output captured is:
versions are:
0.47.05
2.8.4
6.1.50.1500068_4
I'm guessing that the last version is a reminder that the port should be considered for inclusion in FreeBSD-ports-kmods builds but that is a separate issue from this bug report.
arrowd commentedon Aug 25, 2026
All right, maybe the solver should try to prefer plans that keep non-automatic packages. But that's not as easy as it seems - we somehow need to avoid getting into the situation I described above, when the decision for keeping a package prevents us from upgrading others. We should not create an illusion that "Your packages are up to date" while in reality there are actually package that could be updated at the cost of removing some non-automatic ones.
Maybe something like this:
@bdrewery @bapt What do you think?
mirror176 commentedon Aug 28, 2026
I totally agree that users should not be left with outdated packages + no information about it.
If 'something' needs to be removed, shouldn't the solver let the automatic ones go before letting go of non-automatic ones when such a choice exists? This bug is reporting that specific case where the opposite is what is done.
I understand that such a conflict will get complicated on real world scenarios once ports have conflicting needs during an upgrade as I've ran into those over the years. Sometimes its hard to fix by altering the port to have selections to alter dependencies and worse when it requires porting effort just to make the software function with the altered dependency choice at all. Sometimes ports artificially force a requirement where its juts a little port alteration and it avoids conflicts but until then users get to deal with choosing which package(s) to lose.
I assume that if a package cannot use a newer version of a dependency and that there isn't an older version provided in another package then the port would fail to build the program at that point. Unfortunately pkg doesn't have any notes about failed attempts to build software in the repository that I'm aware of so at that point pkg cannot inform users that a dependency has an upgrade but one of its consumers is not yet updated to use it beyond users reviewing removed+installed ports to try to determine if their desired program went away completely, is temporarily broken, or got an entry in 'removed' and separately in 'installed' to move things around instead of just remove them. Removed and installed lines don't normally say why the change is happening. It would be good if pkg said why that is happening and it would be good if removed items were checked for non-automatic, locked, and vital status so more attention can be brought to the user about the conflicting state that likely has undesired results.
With discussion earlier about marking a package as vital, marking linux-dwarffortress as vital leads to a general
pkg upgradeto upgrade 4 and reinstall 23. It will not upgrade or reinstall linux-dwarffortress in that run but will do so when it is added to the command. At least according to the prompt before it is ran; haven't yet had it proceed in case there is anything needed from the machine in its current state to look into this issue.Marietto2008 commentedon Sep 23, 2026
I told to Claude (the new OPUS 5.5) to read the full discussion and it comes up with this "solution". I hope it can be useful in some way.
pkg #2740 — root cause in the solver fallback, and a proposal
Analysis against
freebsd/pkgmain(libpkg/pkg_solve.c,libpkg/pkg_jobs.c), using the two outputs posted by @mirror176.1. What actually happens
Why the first plan shows no removals. The first solve runs before fetch, when remote file lists are not loaded. The c7↔rl9 exclusion is only a file conflict, so it is invisible at that point. After fetch,
pkg_jobs_check_conflicts()finds the 71 file conflicts, registers them, and re-solves. This is why the plan changes after the download.Why
pkg upgradeandpkg upgrade linux-dwarffortressdiverge.jobs_solve_full_upgrade()turns every upgrade candidate (31 here) into arequest_add.pkg_solve_add_request_rule()makes each one aPKG_VAR_TOPvariable and apicosat_assume(). At SAT level, "upgradelinux_base-c77.9.2009_3→_5" (automatic) and "reinstalllinux-dwarffortress(new build, rl9 deps)" (non-automatic) are indistinguishable hard assumptions.pkg_solve_sat_problem()then takespicosat_failed_assumptions()and picks the last element ("by experience the culprit seems to always be the latest"). It marks itPKG_VAR_FAILED. On reiterate, TOP vars withFAILEDgetPKG_VAR_INSTALLflipped and are re-assumed with the opposite sign. So the dropped request is not relaxed, it is negated: "the new dwarffortress must not be installed".picosat_set_default_phase_lit+more_important). That is the same treatment every installed automatic package gets. Keeping it is incompatible with the c7 upgrade, wherec7-jpeg/libpng/libtiff/jasper/jbigkitare replaced byc7-imageformats-libs. So it is removed.pkg upgrade linux-dwarffortressthere is only one request. The c7 packages are not TOP, only phase-preferred, so the solver drops them freely. The correct plan comes out in one shot, without "trying another plan".Conclusion. "pkg proposed you one of the possible solutions" is accurate. But which solution is chosen depends on the position of a literal in the failed-assumption array, which follows variable numbering. It does not depend on the automatic, vital or locked flags, or on any cost.
Also note that
picosat_failed_assumptions()is documented as an over-approximation. It is not a minimal core, so "the last one" may not even be necessary to the conflict.2. Why the two-pass proposal does not fix this report
Proposal: normal plan → if it removes non-automatic packages, re-solve with non-automatic packages treated as vital → use that plan if it succeeds.
@mirror176 has already run pass 2 by hand: he marked dwarffortress vital. The result was 4 upgrades and 23 reinstalls, with dwarffortress neither upgraded nor reinstalled. The plan is SAT, so the proposal would adopt it. That plan:
FreeBSD:14build with the c7 stack;pkg upgrade linux-dwarffortressplan, which keeps the same non-automatic set and has everything newer;The reason is structural. The vital rule
(L ∨ R)only restricts which assumption can be negated. It does not change the fact that upgrades of automatic packages compete as hard requests against upgrades of user packages. The fallback still negates "the last failed assumption", which is now the dwarffortress upgrade.The proposal also doubles the solve cost in exactly the case that already loops through fetch and integrity checks (up to 100 attempts).
On the appA/appB example: I agree that intent cannot always be inferred. That argues for an explicit priority order plus explicit reporting of what was held back. It does not argue for an order-dependent choice.
3. On
pkg set -v 1(confirming @bdrewery)pkg_jobs_need_upgrade()contains:A user-set vital flag on a package not built vital therefore does two things:
automaticis local user state, propagated bypkg_jobs_propagate_automatic().vitalis compared as package metadata, and both kinds of vital share one field.Fix: store user-vital separately (a column or annotation), OR it into the effective flag, and exclude it from the
need_upgradecomparison. This is independent of #2740.4. Proposal: priority tiers + core-guided relaxation (drop, never negate)
This is a contained change in
pkg_solve.c. Each assumption gets a priority:s(picosat_inc_max_var), clause(¬s ∨ L ∨ R1 … ∨ Rn), assumesThe loop replaces the current
reiteratefallback:Key properties:
activebecomes unconstrained. It does not become "must not be installed". Negation forces removals the problem never required.On this report. The core contains P3 (c7 upgrades) and P2 (dwarffortress). The P3 literals are dropped. The dwarffortress assumption pulls in rl9, the c7 variables become free and are set false. The
alsa-libupgrade is not in the core and stays. The result is thepkg upgrade linux-dwarffortressplan plusalsa-lib, which is what the reporter expected, from a plainpkg upgrade.On appA/appB.
If libDep-2 is the only conflict, P3 is dropped first; that is not enough.
The core is then {s_appB (P1), appA-2 (P2)}. What happens next is a policy decision:
UPGRADE_NONAUTO_REMOVAL=hold(default): drop P2. appA is held back, and the output says so:Held back: appA 1 -> 2 (would remove non-automatic appB; run 'pkg upgrade appA' to accept).This is
apt-get upgrade"kept back" semantics.=allow: drop P1. appB is removed, flagged(non-automatic)in the REMOVED list, and-ydoes not auto-confirm it. This is comparable to dnf's--allowerasingopt-in.Nothing is silent in either mode, which answers the "Your packages are up to date" concern.
An explicit
pkg upgrade appApromotes appA to P0, as the partial-upgrade path already effectively does.5. Regression test (atf, local repos)
app-1(non-automatic) depends onlibold-1(automatic).libold-2, which file-conflicts withlibnew-1;libnew-1;app-1rebuilt with a changed ABI or deps, depending onlibnew.pkg upgrade -n: installlibnew, removelibold, reinstallapp, and noappremoval.aaa-lib*vszzz-lib*). The current heuristic should produce different plans across the two orderings. The tiered loop must produce the same plan for both.6. Side notes
bapt commentedon Sep 25, 2026
@Marietto2008 thanks for trying to help here, but the analysis from claude here is adding more noise then being helpful as I tried to explain on the mailing lists, it may drive people in wrong directions and will actually fail at doing what is your initial intention. (Again I understand and i am thank full that you are trying to help).
bapt commentedon Sep 25, 2026
for @bdrewery and @arrowd the vital flag is not supposed to be end user, that said apparently people wants to be able to use it, so I will add a local variation of it for users to be able to override.
arrowd commentedon Sep 25, 2026
What about my idea from #2740 (comment) ? It would result in the following behavior:
pkg upgradefor the first time.pkg upgradefor the second time, the solver will detect that nothing could be upgraded without removing non-automatic packages and will propose the plan with deletions.This does not break any behaviour as far as I can tell, only makes it easier to detect dangerous upgrades.
bapt commentedon Sep 25, 2026
I will push in a couple of hours new code which adresses the best I found all the issues here:
1/ it shows why something is requested to be removed
2/ there is a mode which is now able to forbid (aka make it fail) if any non automatic package is getting removed
3/ the solve has been reworked deeply (and it hurt my brain quite deeply) so we have a "smarter" replay of the solver with the assumption we should try to preserve as much as possible what has been installed.
4/ I have now a local override table to be able to store the user set vital flag to is is preserved and prefered over the package defined version.
mirror176 commentedon Sep 25, 2026
I assume in the case of this filed bug I'd likely have seen -c7 packages still get upgraded but in lesser number as opposed to seeing the manual package get upgraded at the cost of scheduling the automatics for deletion? If it purges any upgrade step that would remove a non-automatic and allows the solver to reach the non-automatic and rework the list according then we definitely have something.
I assume user is prompted that some upgrades cannot be performed at least numerically and (may be a future goal) preferably with the list of details of why items are skipped?
That makes it easier to do some non-dangerous upgrades. Detecting dangerous upgrades requires more attention be drawn to the fact that something is being removed and I think there is value in telling the user that it is a non-automatic that is slated for removal which needs to be separated from an automatic being removed to reduce the noise. Users may hit a point where they have an automatic that came in as a dependency of a package or metapackage which they actively want but they can always 1. mark it non-automatic if its important and 2. read the deletions of automatics but otherwise I assume automatics should be something users will not care about most of the time which is why autoremove is usually but not always safe to blindly consider.
This will definitely save a number of users from at least some headache, especially if the default can . I could see holding off changing default action until a major version bump that is accompanied with documentation warning users the workflow is different. Once we gain this workflow I don't see as much value in keeping the old layout.
One thing about the prompt for deletions is the user never had a way to be given a choice to remove A vs remove B and the solver just jumps to a conclusion. My other solver bug report #2752, if we go off topic from the actual bug I tried to show with it, showed removing kde but a general
pkg upgradewith the system's installed packages causes xlibre to be the removal candidate because kde needs to be updated and that depends on the conflicting xorg. I also lose the xlibre-nvidia-driver without gaining the xorg nvidia driver; probably no good logical way to handle nvidia unless solver searches flavors for a reinstall. In any case, I am not given a choice of if I want kde vs xlibre but rather the solver picks one for me. There are times that I will actively uninstall/reinstall packages to use one one moment and use a conflicting one another moment but I know of no way to be able to expect what the solver will do next. We could further pollute the database with a user-adjustable scoring or score-based categorizing so I can say that I want to keep some cad software for doing work vs keep a game for entertainment, but without any such thing I feel like its like a coinflip what pkg will do next and since the coin doesn't always flip the way I want I'm left to intervene but with manual + smaller commands.I don't know of a good command line way to interact with the user to ask what to keep vs lose but cannot imagine something better than a list of what to keep and what gets lost due to conflict and let the user add/remove from the keep list until they like the outcome. Definitely getting into something beyond passing a command line list and following with a simple yes/no prompt as the user would need to edit the list live. Abort with a list of this non-automatic conflicts with that non-automatic(s) or the dependency xyz of the non-automatic and leaving it up to the user to build a smaller command from the output seems the most logical and simple with a dynamic list control being something to develop later and maybe separately.
Thank you very much for your time working on this.