Skip to content

Commit 7585b0c

Browse files
myasnikovdaniilmyasnikovdaniil
authored
test(e2e): replace fixed timeouts with event-driven backstops (#2554)
## What this PR does Replaces fragile `sleep N` / `timeout N until …; do sleep …; done` patterns in `hack/*.bats` and `hack/*.sh` with `kubectl wait --for=condition=…` plus a small existence backstop: ```sh until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done kubectl wait <resource> --for=condition=Ready --timeout=… ``` ### Why the existence backstop matters `kubectl wait` against a not-yet-created resource sits silently for the full timeout and then fails with `error: timed out waiting for the condition` — useless for diagnosis. With the backstop, the test fails fast with a clear "resource never appeared" error. Existence backstop added at 14 sites in this sweep across: - `redis`, `kafka`, `openbao`, `harbor`, `mongodb`, `qdrant`, `clickhouse`, `mariadb`, `postgres`, `bucket`, `external-dns`, `foundationdb` bats - `hack/e2e-install-cozystack.bats` - `hack/e2e-apps/run-kubernetes.sh` - `hack/e2e-test-openapi.bats` (port-readiness wait replacing fixed `sleep 5` after `kubectl proxy`) The vm-disk hunk in `hack/e2e-apps/vminstance.bats` is included here; the vm-instance hunks land separately via the `daniil/split-vminstance` PR. ### Genuine sleeps annotated The 3 remaining genuine sleeps (LINSTOR node membership propagation, MetalLB end-to-end HTTP path, HR-count emission heuristic) are annotated with `# TODO` + rationale so a future reader knows they are intentional. Same green-path semantics; failures now surface in seconds instead of after a 5-minute timeout. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved e2e stability by replacing fixed sleeps with timeout-based polling across many integration tests, waiting for operator-created resources to exist before readiness checks to reduce race conditions and flakes. * **Documentation** * Added clarifying notes for LoadBalancer retry behavior and improved local proxy readiness checks in e2e scripts. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 8e78536 + cc7551c commit 7585b0c

16 files changed

Lines changed: 91 additions & 17 deletions

‎hack/e2e-apps/bucket.bats‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,11 @@ EOF
2020

2121
# Wait for the bucket to be ready
2222
kubectl -n tenant-test wait hr bucket-${name} --timeout=100s --for=condition=ready
23+
timeout 60 sh -ec "until kubectl -n tenant-test get bucketclaims.objectstorage.k8s.io bucket-${name} >/dev/null 2>&1; do sleep 2; done"
2324
kubectl -n tenant-test wait bucketclaims.objectstorage.k8s.io bucket-${name} --timeout=300s --for=jsonpath='{.status.bucketReady}'
25+
timeout 60 sh -ec "until kubectl -n tenant-test get bucketaccesses.objectstorage.k8s.io bucket-${name}-admin >/dev/null 2>&1; do sleep 2; done"
2426
kubectl -n tenant-test wait bucketaccesses.objectstorage.k8s.io bucket-${name}-admin --timeout=300s --for=jsonpath='{.status.accessGranted}'
27+
timeout 60 sh -ec "until kubectl -n tenant-test get bucketaccesses.objectstorage.k8s.io bucket-${name}-viewer >/dev/null 2>&1; do sleep 2; done"
2528
kubectl -n tenant-test wait bucketaccesses.objectstorage.k8s.io bucket-${name}-viewer --timeout=300s --for=jsonpath='{.status.accessGranted}'
2629

2730
# Get admin (readwrite) credentials

‎hack/e2e-apps/clickhouse.bats‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,12 @@ spec:
3535
resources: {}
3636
resourcesPreset: "nano"
3737
EOF
38-
sleep 5
38+
# Wait for the operator to materialise the HelmRelease before kubectl wait
39+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
40+
timeout 60 sh -ec "until kubectl -n tenant-test get hr clickhouse-$name >/dev/null 2>&1; do sleep 2; done"
3941
kubectl -n tenant-test wait hr clickhouse-$name --timeout=20s --for=condition=ready
4042
timeout 180 sh -ec "until kubectl -n tenant-test get svc chendpoint-clickhouse-$name -o jsonpath='{.spec.ports[*].port}' | grep -q '8123 9000'; do sleep 10; done"
43+
timeout 60 sh -ec "until kubectl -n tenant-test get statefulset.apps/chi-clickhouse-$name-clickhouse-0-0 >/dev/null 2>&1; do sleep 2; done"
4144
kubectl -n tenant-test wait statefulset.apps/chi-clickhouse-$name-clickhouse-0-0 --timeout=120s --for=jsonpath='{.status.replicas}'=1
4245
timeout 80 sh -ec "until kubectl -n tenant-test get endpoints chi-clickhouse-$name-clickhouse-0-0 -o jsonpath='{.subsets[*].addresses[*].ip}' | grep -q '[0-9]'; do sleep 10; done"
4346
timeout 100 sh -ec "until kubectl -n tenant-test get svc chi-clickhouse-$name-clickhouse-0-0 -o jsonpath='{.spec.ports[*].port}' | grep -q '9000 8123 9009'; do sleep 10; done"

‎hack/e2e-apps/external-dns.bats‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,9 @@ spec:
1818
- example.com
1919
EOF
2020

21-
sleep 5
21+
# Wait for the operator to materialise the HelmRelease before kubectl wait
22+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
23+
timeout 60 sh -ec "until kubectl -n tenant-test get hr ${name} >/dev/null 2>&1; do sleep 2; done"
2224
kubectl -n tenant-test wait hr ${name} --timeout=120s --for=condition=ready
2325
kubectl -n tenant-test wait hr ${name}-system --timeout=120s --for=condition=ready
2426
timeout 60 sh -ec "until kubectl -n tenant-test get deploy -l app.kubernetes.io/instance=${name}-system -o jsonpath='{.items[0].status.readyReplicas}' | grep -q '1'; do sleep 5; done"
@@ -41,7 +43,9 @@ spec:
4143
- example.org
4244
EOF
4345

44-
sleep 5
46+
# Wait for the operator to materialise the HelmRelease before kubectl wait
47+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
48+
timeout 60 sh -ec "until kubectl -n tenant-test get hr ${name} >/dev/null 2>&1; do sleep 2; done"
4549
kubectl -n tenant-test wait hr ${name} --timeout=120s --for=condition=ready
4650
kubectl -n tenant-test wait hr ${name}-system --timeout=120s --for=condition=ready
4751
timeout 60 sh -ec "until kubectl -n tenant-test get deploy -l app.kubernetes.io/instance=${name}-system -o jsonpath='{.items[0].status.readyReplicas}' | grep -q '1'; do sleep 5; done"

‎hack/e2e-apps/foundationdb.bats‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@ spec:
4242
imageType: "unified"
4343
automaticReplacements: true
4444
EOF
45-
sleep 15
45+
# Wait for the operator to materialise the HelmRelease before kubectl wait
46+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
47+
timeout 60 sh -ec "until kubectl -n tenant-test get hr foundationdb-$name >/dev/null 2>&1; do sleep 2; done"
4648

4749
# Wait for HelmRelease to be ready
4850
kubectl -n tenant-test wait hr foundationdb-$name --timeout=300s --for=condition=ready

‎hack/e2e-apps/harbor.bats‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,12 +38,16 @@ spec:
3838
size: 1Gi
3939
replicas: 1
4040
EOF
41-
sleep 5
41+
# Wait for the operator to materialise the HelmRelease before kubectl wait
42+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
43+
timeout 60 sh -ec "until kubectl -n tenant-test get hr $release >/dev/null 2>&1; do sleep 2; done"
4244
kubectl -n tenant-test wait hr $release --timeout=60s --for=condition=ready
4345

4446
# Wait for COSI to provision bucket
47+
timeout 60 sh -ec "until kubectl -n tenant-test get bucketclaims.objectstorage.k8s.io $release-registry >/dev/null 2>&1; do sleep 2; done"
4548
kubectl -n tenant-test wait bucketclaims.objectstorage.k8s.io $release-registry \
4649
--timeout=300s --for=jsonpath='{.status.bucketReady}'=true
50+
timeout 60 sh -ec "until kubectl -n tenant-test get bucketaccesses.objectstorage.k8s.io $release-registry >/dev/null 2>&1; do sleep 2; done"
4751
kubectl -n tenant-test wait bucketaccesses.objectstorage.k8s.io $release-registry \
4852
--timeout=60s --for=jsonpath='{.status.accessGranted}'=true
4953

@@ -64,8 +68,11 @@ EOF
6468
kubectl -n tenant-test get secret $release-registry-bucket -o jsonpath='{.data.BucketInfo}' 2>&1 | base64 -d 2>&1 || true
6569
false
6670
}
71+
timeout 60 sh -ec "until kubectl -n tenant-test get deploy $release-core >/dev/null 2>&1; do sleep 2; done"
6772
kubectl -n tenant-test wait deploy $release-core --timeout=120s --for=condition=available
73+
timeout 60 sh -ec "until kubectl -n tenant-test get deploy $release-registry >/dev/null 2>&1; do sleep 2; done"
6874
kubectl -n tenant-test wait deploy $release-registry --timeout=120s --for=condition=available
75+
timeout 60 sh -ec "until kubectl -n tenant-test get deploy $release-portal >/dev/null 2>&1; do sleep 2; done"
6976
kubectl -n tenant-test wait deploy $release-portal --timeout=120s --for=condition=available
7077
kubectl -n tenant-test get secret $release-credentials -o jsonpath='{.data.admin-password}' | base64 --decode | grep -q '.'
7178
kubectl -n tenant-test get secret $release-credentials -o jsonpath='{.data.url}' | base64 --decode | grep -q 'https://'

‎hack/e2e-apps/kafka.bats‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,12 @@ spec:
3939
partitions: 1
4040
replicas: 2
4141
EOF
42-
sleep 5
42+
# Wait for the operator to materialise the HelmRelease before kubectl wait
43+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
44+
timeout 60 sh -ec "until kubectl -n tenant-test get hr kafka-$name >/dev/null 2>&1; do sleep 2; done"
4345
kubectl -n tenant-test wait hr kafka-$name --timeout=30s --for=condition=ready
44-
kubectl wait kafkas -n tenant-test test --timeout=300s --for=condition=ready
46+
timeout 60 sh -ec "until kubectl -n tenant-test get kafkas $name >/dev/null 2>&1; do sleep 2; done"
47+
kubectl wait kafkas -n tenant-test $name --timeout=300s --for=condition=ready
4548
timeout 60 sh -ec "until kubectl -n tenant-test get pvc data-kafka-$name-zookeeper-0; do sleep 10; done"
4649
kubectl -n tenant-test wait pvc data-kafka-$name-zookeeper-0 --timeout=50s --for=jsonpath='{.status.phase}'=Bound
4750
timeout 40 sh -ec "until kubectl -n tenant-test get svc kafka-$name-zookeeper-client -o jsonpath='{.spec.ports[0].port}' | grep -q '2181'; do sleep 10; done"

‎hack/e2e-apps/mariadb.bats‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,13 +35,17 @@ spec:
3535
resources: {}
3636
resourcesPreset: "nano"
3737
EOF
38-
sleep 5
38+
# Wait for the operator to materialise the HelmRelease before kubectl wait
39+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
40+
timeout 60 sh -ec "until kubectl -n tenant-test get hr mariadb-$name >/dev/null 2>&1; do sleep 2; done"
3941
kubectl -n tenant-test wait hr mariadb-$name --timeout=30s --for=condition=ready
4042
timeout 80 sh -ec "until kubectl -n tenant-test get svc mariadb-$name -o jsonpath='{.spec.ports[0].port}' | grep -q '3306'; do sleep 10; done"
4143
timeout 80 sh -ec "until kubectl -n tenant-test get endpoints mariadb-$name -o jsonpath='{.subsets[*].addresses[*].ip}' | grep -q '[0-9]'; do sleep 10; done"
44+
timeout 60 sh -ec "until kubectl -n tenant-test get statefulset.apps/mariadb-$name >/dev/null 2>&1; do sleep 2; done"
4245
kubectl -n tenant-test wait statefulset.apps/mariadb-$name --timeout=110s --for=jsonpath='{.status.replicas}'=2
4346
timeout 80 sh -ec "until kubectl -n tenant-test get svc mariadb-$name-metrics -o jsonpath='{.spec.ports[0].port}' | grep -q '9104'; do sleep 10; done"
4447
timeout 40 sh -ec "until kubectl -n tenant-test get endpoints mariadb-$name-metrics -o jsonpath='{.subsets[*].addresses[*].ip}' | grep -q '[0-9]'; do sleep 10; done"
48+
timeout 60 sh -ec "until kubectl -n tenant-test get deployment.apps/mariadb-$name-metrics >/dev/null 2>&1; do sleep 2; done"
4549
kubectl -n tenant-test wait deployment.apps/mariadb-$name-metrics --timeout=90s --for=jsonpath='{.status.replicas}'=1
4650
kubectl -n tenant-test delete mariadbs.apps.cozystack.io $name
4751
}

‎hack/e2e-apps/mongodb.bats‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,14 +26,17 @@ spec:
2626
backup:
2727
enabled: false
2828
EOF
29-
sleep 5
29+
# Wait for the operator to materialise the HelmRelease before kubectl wait
30+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
31+
timeout 60 sh -ec "until kubectl -n tenant-test get hr mongodb-$name >/dev/null 2>&1; do sleep 2; done"
3032
# Wait for HelmRelease
3133
kubectl -n tenant-test wait hr mongodb-$name --timeout=60s --for=condition=ready
3234
# Wait for MongoDB service (port 27017)
3335
timeout 120 sh -ec "until kubectl -n tenant-test get svc mongodb-$name-rs0 -o jsonpath='{.spec.ports[0].port}' | grep -q '27017'; do sleep 10; done"
3436
# Wait for endpoints
3537
timeout 180 sh -ec "until kubectl -n tenant-test get endpoints mongodb-$name-rs0 -o jsonpath='{.subsets[*].addresses[*].ip}' | grep -q '[0-9]'; do sleep 10; done"
3638
# Wait for StatefulSet replicas
39+
timeout 60 sh -ec "until kubectl -n tenant-test get statefulset.apps/mongodb-$name-rs0 >/dev/null 2>&1; do sleep 2; done"
3740
kubectl -n tenant-test wait statefulset.apps/mongodb-$name-rs0 --timeout=300s --for=jsonpath='{.status.replicas}'=1
3841
# Cleanup
3942
kubectl -n tenant-test delete mongodbs.apps.cozystack.io $name

‎hack/e2e-apps/openbao.bats‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,9 @@ spec:
1818
external: false
1919
ui: true
2020
EOF
21-
sleep 5
21+
# Wait for the operator to materialise the HelmRelease before kubectl wait
22+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
23+
timeout 60 sh -ec "until kubectl -n tenant-test get hr openbao-$name >/dev/null 2>&1; do sleep 2; done"
2224
kubectl -n tenant-test wait hr openbao-$name --timeout=60s --for=condition=ready
2325
kubectl -n tenant-test wait hr openbao-$name-system --timeout=120s --for=condition=ready
2426

@@ -53,7 +55,9 @@ EOF
5355
kubectl -n tenant-test exec openbao-$name-0 -- bao operator unseal "$unseal_key"
5456

5557
# Now wait for pod to become ready (readiness probe checks seal status)
58+
timeout 60 sh -ec "until kubectl -n tenant-test get sts openbao-$name >/dev/null 2>&1; do sleep 2; done"
5659
kubectl -n tenant-test wait sts openbao-$name --timeout=90s --for=jsonpath='{.status.readyReplicas}'=1
60+
timeout 60 sh -ec "until kubectl -n tenant-test get pvc data-openbao-$name-0 >/dev/null 2>&1; do sleep 2; done"
5761
kubectl -n tenant-test wait pvc data-openbao-$name-0 --timeout=50s --for=jsonpath='{.status.phase}'=Bound
5862
kubectl -n tenant-test delete openbao.apps.cozystack.io $name
5963
kubectl -n tenant-test delete pvc data-openbao-$name-0 --ignore-not-found

‎hack/e2e-apps/postgres.bats‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,11 @@ spec:
4040
resources: {}
4141
resourcesPreset: "nano"
4242
EOF
43-
sleep 5
43+
# Wait for the operator to materialise the HelmRelease before kubectl wait
44+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
45+
timeout 60 sh -ec "until kubectl -n tenant-test get hr postgres-$name >/dev/null 2>&1; do sleep 2; done"
4446
kubectl -n tenant-test wait hr postgres-$name --timeout=100s --for=condition=ready
47+
timeout 60 sh -ec "until kubectl -n tenant-test get job.batch postgres-$name-init-job >/dev/null 2>&1; do sleep 2; done"
4548
kubectl -n tenant-test wait job.batch postgres-$name-init-job --timeout=50s --for=condition=Complete
4649
timeout 40 sh -ec "until kubectl -n tenant-test get svc postgres-$name-r -o jsonpath='{.spec.ports[0].port}' | grep -q '5432'; do sleep 10; done"
4750
timeout 40 sh -ec "until kubectl -n tenant-test get svc postgres-$name-ro -o jsonpath='{.spec.ports[0].port}' | grep -q '5432'; do sleep 10; done"

0 commit comments

Comments
 (0)