Repository navigation
KVM: destroy with vm.destroy.forcestop leaves the instance running when its host is briefly disconnected #14232
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
engine-orchestration, destroyVirtualMachine, KVM
CLOUDSTACK VERSION
4.22
CONFIGURATION
vm.destroy.forcestop=true. KVM hosts. Any rolling restart of the agents or the management servers while instances are being destroyed.
OS / ENVIRONMENT
KVM / libvirt.
SUMMARY
With vm.destroy.forcestop=true, destroying an instance whose host is briefly disconnected releases the instance's NICs, IP addresses and volumes without stopping it. The domain keeps running on the host with no record in CloudStack. Its IP is handed to the next instance, so two live instances answer for the same address, and its root volume stays in Destroy because the delete fails while the domain holds the image.
Same end state as #14206, different trigger. #14206 is a stale power report during start; this one is the destroy path itself.
STEPS TO REPRODUCE
- Set
vm.destroy.forcestop=true. - Restart
cloudstack-agenton a KVM host (or a management server, which disconnects the agents connected to it while they rebalance). - While the host is
Disconnected, destroy a running instance on it withexpunge=true.
Any client that destroys instances routinely hits this on every rolling upgrade.
EXPECTED RESULTS
The destroy fails, the instance stays Running, and the caller retries once the host is back. Or the destroy waits for the host.
ACTUAL RESULTS
From a real occurrence on 4.22, during a rolling package upgrade:
13:39:02 WARN Unable to stop VM instance {"id":3645123,...,"state":"Stopping"} due to [AgentUnavailableException:
Resource [Host:110] is unreachable: Host 110: Host with specified id is not in the right state: Disconnected]
13:39:02 WARN Unable to actually stop VM instance {"id":3645123,...} but continue with release because it's a force stop
13:39:02 DEBUG VM instance {"id":3645123,...} is stopped on the host. Proceeding to release resource held.
13:39:02 DEBUG Successfully released network resources for the VM ...
13:39:15 DEBUG Expunged VM instance {"id":3645123,...}
13:41:58 WARN Host reports 9 instance(s) that do not exist in CloudStack DB, they are running unmanaged. host: hv104, instances: [i-625-3645123-VM, ...]
Across two rolling upgrades, 31 instances on 6 hosts were left running this way, and new instances were given their addresses within hours.
CAUSE
UserVmManagerImpl.destroyVm(DestroyVMCmd), VirtualMachineManagerImpl.destroy() and VirtualMachineManagerImpl.advanceExpunge() all stop the instance with cleanUpEvenIfUnableToStop = vm.destroy.forcestop. In advanceStop(), a forced stop that gets AgentUnavailableException or OperationTimedoutException releases the resources and marks the instance stopped:
} catch (AgentUnavailableException | OperationTimedoutException e) {
logger.warn("Unable to stop {} due to [{}].", ...);
} finally {
if (!stopped) {
if (!cleanUpEvenIfUnableToStop) { ... throw ... }
else { logger.warn("Unable to actually stop {} but continue with release because it's a force stop", vm); ... }
}
}
releaseVmResources(profile, cleanUpEvenIfUnableToStop);A forced stop means "the host cannot tell us, treat the instance as stopped". That is right when the host is gone (Down, Removed). It is wrong when the host is only unreachable for a while, whatever its status says: Disconnected, Connecting, Alert or Rebalancing, and also Up while a crashed management server's hosts have not yet been taken over or a disconnect investigation is inconclusive. Those hosts are expected back with their domains still running.
vm.destroy.forcestop is a global setting applied to every destroy, not a statement by the caller that it knows the host is gone. It should not bypass that distinction.
IMPACT
- an instance runs unmanaged and invisible
- its IP is reassigned, giving an address conflict between two live instances
- its root volume stays in
Destroyand the storage cleanup fails on it every run
Activity
engine: do not force the destroy stop while the host may reconnect
github-actions commentedon Sep 24, 2026
🎯 Triage report
When
vm.destroy.forcestop=trueand a KVM host is only brieflyDisconnected(e.g. during a rolling agent/management-server restart), destroying a running instance releases its NICs/IPs/volumes and marks it stopped without the domain actually being stopped. The domain keeps running unmanaged, its IP gets reassigned to a new instance, and its root volume is stuck inDestroy. This is a well-documented, code-level analysis with logs from a real occurrence and a clear root cause identified inVirtualMachineManagerImpl.advanceStop()/destroy().📊 Assessment
type:bugcomponent:kvmvm.destroy.forcestopprecondition.advanceStop()/destroy()should distinguish "host is gone" vs "host is temporarily unreachable" across all affected host states (Disconnected,Connecting,Alert,Rebalancing), which needs maintainer input before an agent should attempt an implementation.🔗 Similar issues
vm.destroy.forcestop. The reporter explicitly notes this is the same failure mode via a different trigger.💡 Notes and suggestions
Down/Removed, not merely unreachable) can address both.Disconnectedduring an agent/management-server restart while a destroy withexpunge=trueis in flight — this is timing-sensitive and may be easier to validate via targeted unit/integration tests aroundVirtualMachineManagerImpl.advanceStop()andreleaseVmResources()than live rolling-upgrade reproduction.Disconnected(e.g.,Connecting,Alert,Rebalancing) should also block the forced release path, since the issue proposes several.Add this agentic workflows to your repo
To install this agentic workflow, run