Skip to content

Commit 06d6106

Browse files
scooby87scooby87
committed
fix(bucket): clean up orphaned cert-manager ACME solver resources on delete (#3060)
cert-manager ACME http01-solver Ingress/Pod/Service objects were left orphaned after a Bucket app CR was deleted (a stuck challenge leaves them without a live owner to cascade-delete). Add a post-delete hook that deletes the release Certificate/Order/Challenge and sweeps leftover solver objects by the acme.cert-manager.io/http01-solver label. RBAC verified against the vendored cert-manager CRDs. The cleanup Job runs non-root with a read-only root fs and dropped capabilities; failures are logged but never block teardown. Signed-off-by: Алексей Артамонов <[email protected]>
1 parent 374cac1 commit 06d6106

3 files changed

Lines changed: 218 additions & 0 deletions

File tree

‎packages/apps/bucket/Makefile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
include ../../../hack/package.mk
22

3+
.PHONY: test
4+
test:
5+
helm unittest .
6+
37
generate:
48
cozyvalues-gen -m 'bucket' -v values.yaml -s values.schema.json -r README.md -g ../../../api/apps/v1alpha1/bucket/types.go
59
../../../hack/update-crd.sh

‎packages/apps/bucket/templates/hooks/cleanup-acme.yaml‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
apiVersion: batch/v1
3+
kind: Job
4+
metadata:
5+
name: {{ .Release.Name }}-cleanup-acme
6+
labels:
7+
app.kubernetes.io/instance: {{ .Release.Name }}
8+
annotations:
9+
"helm.sh/hook": post-delete
10+
"helm.sh/hook-weight": "10"
11+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
12+
spec:
13+
template:
14+
metadata:
15+
labels:
16+
app.kubernetes.io/instance: {{ .Release.Name }}
17+
policy.cozystack.io/allow-to-apiserver: "true"
18+
spec:
19+
serviceAccountName: {{ .Release.Name }}-cleanup-acme
20+
restartPolicy: Never
21+
securityContext:
22+
runAsNonRoot: true
23+
runAsUser: 65534
24+
runAsGroup: 65534
25+
seccompProfile:
26+
type: RuntimeDefault
27+
containers:
28+
- name: cleanup
29+
image: docker.io/clastix/kubectl:v1.32
30+
securityContext:
31+
allowPrivilegeEscalation: false
32+
readOnlyRootFilesystem: true
33+
capabilities:
34+
drop: ["ALL"]
35+
env:
36+
- name: HOME
37+
value: /tmp
38+
command:
39+
- /bin/sh
40+
- -c
41+
- |
42+
echo "Cleaning up orphaned cert-manager ACME resources for {{ .Release.Name }}..."
43+
# Delete the per-bucket Certificate/Order/Challenge. cert-manager
44+
# then cascade-removes the cm-acme-http-solver-* solver objects it
45+
# owns. The release's TLS Certificate is named after the secret it
46+
# populates ({{ .Release.Name }}-ui-tls).
47+
kubectl delete certificate {{ .Release.Name }}-ui-tls -n {{ .Release.Namespace }} --ignore-not-found \
48+
|| echo "WARNING: cert-manager cleanup failed for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
49+
kubectl delete certificate,order,challenge -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} --ignore-not-found \
50+
|| echo "WARNING: cert-manager cleanup failed for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
51+
# Backstop sweep: if the challenge never completed there may be no
52+
# live owner to cascade the deletion, so remove any leftover
53+
# http01 solver Ingress/Service/Pod directly. This label is set by
54+
# cert-manager on every solver object and is the reliable selector.
55+
echo "Sweeping leftover ACME http01-solver objects..."
56+
kubectl delete ingress,svc,pod -n {{ .Release.Namespace }} -l acme.cert-manager.io/http01-solver=true --ignore-not-found \
57+
|| echo "WARNING: ACME solver sweep failed for {{ .Release.Name }}; orphaned solver objects may remain" >&2
58+
echo "ACME cleanup complete."
59+
volumeMounts:
60+
- name: tmp
61+
mountPath: /tmp
62+
volumes:
63+
- name: tmp
64+
emptyDir: {}
65+
---
66+
apiVersion: v1
67+
kind: ServiceAccount
68+
metadata:
69+
name: {{ .Release.Name }}-cleanup-acme
70+
labels:
71+
app.kubernetes.io/instance: {{ .Release.Name }}
72+
annotations:
73+
"helm.sh/hook": post-delete
74+
"helm.sh/hook-weight": "0"
75+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
76+
---
77+
apiVersion: rbac.authorization.k8s.io/v1
78+
kind: Role
79+
metadata:
80+
name: {{ .Release.Name }}-cleanup-acme
81+
labels:
82+
app.kubernetes.io/instance: {{ .Release.Name }}
83+
annotations:
84+
"helm.sh/hook": post-delete
85+
"helm.sh/hook-weight": "5"
86+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
87+
rules:
88+
- apiGroups: ["cert-manager.io"]
89+
resources: ["certificates"]
90+
verbs: ["get", "list", "delete"]
91+
- apiGroups: ["acme.cert-manager.io"]
92+
resources: ["orders", "challenges"]
93+
verbs: ["get", "list", "delete"]
94+
- apiGroups: ["networking.k8s.io"]
95+
resources: ["ingresses"]
96+
verbs: ["get", "list", "delete"]
97+
- apiGroups: [""]
98+
resources: ["services", "pods"]
99+
verbs: ["get", "list", "delete"]
100+
---
101+
apiVersion: rbac.authorization.k8s.io/v1
102+
kind: RoleBinding
103+
metadata:
104+
name: {{ .Release.Name }}-cleanup-acme
105+
labels:
106+
app.kubernetes.io/instance: {{ .Release.Name }}
107+
annotations:
108+
"helm.sh/hook": post-delete
109+
"helm.sh/hook-weight": "5"
110+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
111+
roleRef:
112+
apiGroup: rbac.authorization.k8s.io
113+
kind: Role
114+
name: {{ .Release.Name }}-cleanup-acme
115+
subjects:
116+
- kind: ServiceAccount
117+
name: {{ .Release.Name }}-cleanup-acme

‎packages/apps/bucket/tests/cleanup_acme_test.yaml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
suite: bucket post-delete ACME cleanup hook
2+
templates:
3+
- templates/hooks/cleanup-acme.yaml
4+
5+
release:
6+
name: my-bucket
7+
namespace: tenant-root
8+
9+
tests:
10+
- it: renders a post-delete cleanup Job with the expected name and SA
11+
documentIndex: 0
12+
asserts:
13+
- equal:
14+
path: kind
15+
value: Job
16+
- equal:
17+
path: metadata.name
18+
value: my-bucket-cleanup-acme
19+
- equal:
20+
path: metadata.annotations["helm.sh/hook"]
21+
value: post-delete
22+
- equal:
23+
path: spec.template.spec.serviceAccountName
24+
value: my-bucket-cleanup-acme
25+
- equal:
26+
path: spec.template.spec.restartPolicy
27+
value: Never
28+
- equal:
29+
path: spec.template.metadata.labels["policy.cozystack.io/allow-to-apiserver"]
30+
value: "true"
31+
32+
- it: deletes cert-manager resources and sweeps http01-solver leftovers
33+
documentIndex: 0
34+
asserts:
35+
- matchRegex:
36+
path: spec.template.spec.containers[0].command[2]
37+
pattern: "kubectl delete certificate my-bucket-ui-tls -n tenant-root"
38+
- matchRegex:
39+
path: spec.template.spec.containers[0].command[2]
40+
pattern: "kubectl delete certificate,order,challenge -n tenant-root -l app.kubernetes.io/instance=my-bucket"
41+
- matchRegex:
42+
path: spec.template.spec.containers[0].command[2]
43+
pattern: "kubectl delete ingress,svc,pod -n tenant-root -l acme.cert-manager.io/http01-solver=true"
44+
45+
- it: creates a ServiceAccount for the cleanup job
46+
documentIndex: 1
47+
asserts:
48+
- equal:
49+
path: kind
50+
value: ServiceAccount
51+
- equal:
52+
path: metadata.name
53+
value: my-bucket-cleanup-acme
54+
55+
- it: grants RBAC on cert-manager, acme and networking/core resources
56+
documentIndex: 2
57+
asserts:
58+
- equal:
59+
path: kind
60+
value: Role
61+
- contains:
62+
path: rules
63+
content:
64+
apiGroups: ["cert-manager.io"]
65+
resources: ["certificates"]
66+
verbs: ["get", "list", "delete"]
67+
- contains:
68+
path: rules
69+
content:
70+
apiGroups: ["acme.cert-manager.io"]
71+
resources: ["orders", "challenges"]
72+
verbs: ["get", "list", "delete"]
73+
- contains:
74+
path: rules
75+
content:
76+
apiGroups: ["networking.k8s.io"]
77+
resources: ["ingresses"]
78+
verbs: ["get", "list", "delete"]
79+
- contains:
80+
path: rules
81+
content:
82+
apiGroups: [""]
83+
resources: ["services", "pods"]
84+
verbs: ["get", "list", "delete"]
85+
86+
- it: binds the Role to the cleanup ServiceAccount
87+
documentIndex: 3
88+
asserts:
89+
- equal:
90+
path: kind
91+
value: RoleBinding
92+
- equal:
93+
path: roleRef.name
94+
value: my-bucket-cleanup-acme
95+
- equal:
96+
path: subjects[0].name
97+
value: my-bucket-cleanup-acme

0 commit comments

Comments
 (0)