Describe the bug
verify-pvc-reclaim-on-delete deletes the Kafka CR and waits for the deletion to finish. The wait hits the 5m default and fails the suite. Thirteen seconds later the object is gone, so the budget and the teardown sit close enough that a run can land on either side.
Environment
- Cozystack:
main
- In-tree e2e lane, run 34891318171
To Reproduce
Run the full chainsaw suite. The kafka suite fails at verify-pvc-reclaim-on-delete after about 455s.
Expected behaviour
The teardown finishes inside the budget, or the budget reflects what a Kafka teardown costs.
Actual behaviour
The delete was accepted at 22:08:08 and the wait gave up exactly five minutes later:
22:08:08 | kafka | verify-pvc-reclaim-on-delete | DELETE | OK | apps.cozystack.io/v1alpha1/Kafka @ tenant-test/test
22:13:08 | kafka | verify-pvc-reclaim-on-delete | DELETE | ERROR | apps.cozystack.io/v1alpha1/Kafka @ tenant-test/test
=== ERROR
context deadline exceeded
By the time the catch block ran at 22:13:21, both the CR and its HelmRelease were gone:
Error from server (NotFound): helmreleases.helm.toolkit.fluxcd.io "kafka-test" not found
kafkas.apps.cozystack.io "test" not found
Logs
The step declares no timeout of its own, so it takes the suite default:
$ sed -n '95,103p' hack/e2e-chainsaw/kafka/chainsaw-test.yaml
- name: verify-pvc-reclaim-on-delete
try:
- delete:
ref:
apiVersion: apps.cozystack.io/v1alpha1
kind: Kafka
name: test
- error:
timeout: 2m
The default is five minutes, picked with finalizer-draining CRs in mind:
$ sed -n '21,28p' hack/e2e-chainsaw/.chainsaw.yaml
timeouts:
apply: 1m
assert: 5m
# delete/cleanup are generous: finalizer-draining CRs (VMDisk, FoundationDB,
# Tenant child namespaces) can take minutes to fully reclaim their backing
# resources, and Chainsaw waits for the delete to complete during cleanup.
delete: 5m
cleanup: 5m
Additional context
One observation, one run, no rate claimed. The log does not say whether the teardown finished just after the deadline or finished earlier and the wait missed it. Thirteen seconds fits both, and the catch block's own kubectl calls are the only evidence the object was gone.
Worth measuring before the number moves. Kafka drags Zookeeper PVCs and a HelmRelease behind it, so if a teardown near five minutes is normal here then the budget is wrong rather than the run unlucky. A few runs with the delete instrumented would say which, and that is cheaper than moving a timeout somebody set deliberately.
Checklist
Describe the bug
verify-pvc-reclaim-on-deletedeletes the Kafka CR and waits for the deletion to finish. The wait hits the 5m default and fails the suite. Thirteen seconds later the object is gone, so the budget and the teardown sit close enough that a run can land on either side.Environment
mainTo Reproduce
Run the full chainsaw suite. The
kafkasuite fails atverify-pvc-reclaim-on-deleteafter about 455s.Expected behaviour
The teardown finishes inside the budget, or the budget reflects what a Kafka teardown costs.
Actual behaviour
The delete was accepted at 22:08:08 and the wait gave up exactly five minutes later:
By the time the catch block ran at 22:13:21, both the CR and its HelmRelease were gone:
Logs
The step declares no timeout of its own, so it takes the suite default:
The default is five minutes, picked with finalizer-draining CRs in mind:
Additional context
One observation, one run, no rate claimed. The log does not say whether the teardown finished just after the deadline or finished earlier and the wait missed it. Thirteen seconds fits both, and the catch block's own kubectl calls are the only evidence the object was gone.
Worth measuring before the number moves. Kafka drags Zookeeper PVCs and a HelmRelease behind it, so if a teardown near five minutes is normal here then the budget is wrong rather than the run unlucky. A few runs with the delete instrumented would say which, and that is cheaper than moving a timeout somebody set deliberately.
Checklist