Skip to content

e2e: an install-phase stall records no Helm error text, no previous logs of restarted containers and no resource state #4628

Open

Description

When the e2e install phase stalls on a HelmRelease, three things that could explain the stall are never recorded. This is what is left of #3680 after the post-install-prep fixes, and it feeds #3368.

The Helm error object is empty

The HelmRelease shows warning: upgrade failed: {"error":{},"name":"linstor-scheduler"} (run 31288354552, E2E in-tree job), and the info-level helm-controller log has no reason either. In helm-controller v1.5.0, internal/action/log.go LogBuffer.Handle stores each slog attribute as Value.Any() and then json.Marshals it. A Go error has no exported fields, so it renders as {}. The buffer is a debug buffer, so the full record reaches the controller log only at debug level. The same code is on helm-controller main.

Fix shape: upstream, convert attributes that implement error to .Error() before marshalling. Until then, run helm-controller with --log-level=debug in the e2e lanes.

Previous-container logs are missing on passing runs

hack/cozytest.sh captures previous-instance logs only from its exit trap on a failing test. hack/cozyreport.sh runs on every job but reads --previous only for pods it selects as not Ready. A control-plane container that restarted twice and is Ready at report time (the kube-controller-manager and kube-scheduler restarts #3680 describes) gets no previous log on a green run, so there is nothing to compare a red run against.

Fix shape: cozyreport also reads --previous for every container with restartCount > 0, at least in kube-system, under the existing read and budget caps.

No node or pod resource state during install

cozyreport collects node.yaml and describe node only for nodes that are not Ready. In an install stall every node is Ready, so allocatable, allocated and node conditions are never recorded. Nothing in hack/ runs kubectl top. The install-failure branch of pull-requests.yaml dumps HelmReleases and events, but no resource state. The failing runs in #3680 show no OOMKilled or FailedScheduling, but that only shows the data isn't collected.

Fix shape: cozyreport records describe node for every node, plus kubectl top nodes and kubectl top pods -A --containers, under the existing bound. When metrics-server isn't available, record the refusal.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugtriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions