Repository navigation
SeaweedFS COSI provisioner permanently CrashLoops after one bad start (stale unix socket in emptyDir) #2429
Description
Activity
Confirming this is still present on a newer build and a different variant, plus two additions.
Environment (reproduced)
- Cozystack from
main(dev build);objectstorage-controllerandobjectstorage-sidecarv1.4.0-rc.2;seaweedfs-cosi-driverv0.3.0;seaweedfschrislusf/seaweedfs:4.05. - k3s
v1.36.1+k3s1,isp-full-genericvariant (non-Talos) — so this is not Talos-specific. - The
socketemptyDirat/var/lib/cosishared by both containers is present exactly as described.
Addition 1 — the stale Lease gates recovery independently of the socket. Even when the provisioner pod is recreated with a fresh
emptyDir(0 restarts, both containers Running/Ready, nobind: address already in use), reconciliation stays dead: the Leasetenant-root-seaweedfs-objectstorage-k8s-io-cosiremains held by the previous (now-deleted) pod, and the fresh pod does not take leadership for several minutes while that stale Lease (leaseDurationSeconds: 150) persists. So "delete the pod (wipes emptyDir)" does not restore service on its own — when pods are recreated rather than restarted in place, the Lease is the dominant blocker and must be cleared (or leadership handed over faster). The init-container socket cleanup is necessary but not sufficient for the recreate path.Addition 2 — silent false-success / data-integrity symptom (possibly a related controller-status bug). While the control plane is wedged, a
Bucketcreated via the cozystackBucketapp (withspec.users) lands in a misleading state: the COSIBucket(objectstorage.k8s.io) reportsstatus.bucketReady: truewith a bucketID while the bucket was never created in SeaweedFS — the filer has no buckets path, the granted identity'sListBucketsreturns empty, and every object op returnsNoSuchBucket. Meanwhile the matchingBucketClaim.status.bucketReadystaysfalse, so the driver refuses theBucketAccess(BucketAccess can't be granted to Bucket not in Ready state) and no credentials are issued. If the claim status is force-set to ready, credentials are then issued but are useless — they authenticate, yet every operation returnsNoSuchBucket/AccessDenied. Net effect: a consumer can see a "Ready" Bucket without a usable bucket behind it.Recovery confirms the fix direction. A full teardown of the S3 stack recovers cleanly: disable seaweedfs on the tenant, delete the stale Lease plus any orphaned COSI
Bucket/BucketClaim/BucketAccess(force-clearing finalizers) and the seaweedfs PVCs, then re-enable. A fresh provisioner then reconciles a newBucketand an S3 PUT/GET round-trip succeeds — so the SeaweedFS data plane is fine and the problem is entirely the COSI control plane. The failure recurs after the next provisioner pod recreation, so recovery is not automatic.+1 on the suggested init-container socket cleanup; please also consider the lease/leaderelection path so a recreated pod takes over without the multi-minute stall.
- Cozystack from
- added a commit that references this issue
on Jun 5, 2026 - added a commit that references this issue
on Jun 25, 2026 Fixed. Closing after verifying against
main(04b742670) — all three reported symptoms, including both additions in the thread.The socket half was fixed upstream in the driver, in the exact image
mainpins.packages/system/seaweedfs/values.yamlandpackages/extra/seaweedfs/images/seaweedfs-cosi-driver.tagboth pinghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1; at that tagcmd/seaweedfs-cosi-driver/main.gostats the endpoint, refuses to touch anything that is not a socket, dials it first so a live listener is left alone, and only then removes the path before binding (upstream PR seaweedfs/seaweedfs-cosi-driver#8). The reports here were all on v0.3.0, which unconditionally bound and so wedged forever on a leftover socket in theemptyDir.Addition 1, the ~150s leader-election stall after a pod recreate:
packages/system/objectstorage-controller/images/objectstorage/patches/93-shorten-leaderelection-lease.diffcutsLeaseDuration150s to 15s,RenewDeadline120s to 10s andRetryPeriod60s to 2s, so a recreated pod resumes reconciliation in about 15s.Addition 2, a
Bucketreported Ready with no bucket behind it:packages/system/objectstorage-controller/images/objectstorage/patches/92-bucketclaim-propagate-ready.diff.One thing from the discussion was deliberately not done: the chart-side init-container that would clean the socket before start. The
cosi.initContainershook exists in the vendored chart but cozystack never sets it, and the client-topology deployment still mounts a bareemptyDir. That is belt-and-braces now that the driver self-cleans, so it is not worth keeping this open — file it fresh if the driver ever regresses.Found during an audit of all open backup issues.
Summary
The
seaweedfs-objectstorage-provisionerDeployment from thecozy-seaweedfschart can enter an unrecoverableCrashLoopBackOffstate after a single transient driver crash. The driver's Unix socket is created in anemptyDirshared between theseaweedfs-cosi-driverandseaweedfs-cosi-sidecarcontainers, and the driver does not remove the socket file on shutdown — so every subsequent restart fails withbind: address already in use. SinceemptyDirsurvives container restarts (only pod deletion wipes it), a single bad start makes the socket unrecoverable for the lifetime of the pod.Reproduction
Observed on Cozystack
[email protected]+8ac248326acb(Cozystack v1.x, isp-full variant). The pod showed 72 driver restarts over 4d1h with no automatic recovery:```
$ kubectl -n tenant-root get pod -l app.kubernetes.io/component=objectstorage-provisioner
NAME READY STATUS RESTARTS AGE
seaweedfs-objectstorage-provisioner-699cb8d657-j5tzz 0/2 CrashLoopBackOff 72 (21s ago) 4d1h
$ kubectl -n tenant-root logs -c seaweedfs-cosi-driver --previous
E0420 18:26:32.659118 1 server.go:51] "Failed to start server" err="listen unix /var/lib/cosi/cosi.sock: bind: address already in use"
E0420 18:26:32.659194 1 main.go:63] "driver exited with error" err="Failed to start server: listen unix /var/lib/cosi/cosi.sock: bind: address already in use"
$ kubectl -n tenant-root logs -c seaweedfs-cosi-sidecar --previous
I0420 18:26:16.646213 1 cmd.go:79] "Attempting connection to driver" address="unix:///var/lib/cosi/cosi.sock"
E0420 18:28:46.646647 1 provisioner.go:80] "Connection failed" err="context deadline exceeded" address="unix:///var/lib/cosi/cosi.sock"
```
The two containers share an
emptyDir:```yaml
volumeMounts:
name: socket
volumes:
name: socket
```
Impact
BucketClaimreconciliation stops while the provisioner is down. New buckets can't be allocated.tenant-root/objectstore-systemHelmRelease wedges inRollbackSucceeded/UpgradeFailedflap because Helm waits 10m for the provisioner Deployment to become available.Workaround
Manually delete the pod (which wipes
emptyDir) and clear the stale leader Lease so the new pod can take over without waiting out the 150s lease:```bash
kubectl -n tenant-root delete pod -l app.kubernetes.io/component=objectstorage-provisioner
kubectl -n tenant-root delete lease tenant-root-seaweedfs-objectstorage-k8s-io-cosi
```
Suggested fix
Add an init container to the provisioner Deployment that removes any stale socket before the driver starts:
```yaml
initContainers:
image: busybox:stable
command: ["sh", "-c", "rm -f /var/lib/cosi/cosi.sock"]
volumeMounts:
mountPath: /var/lib/cosi
```
Alternatively (or additionally), the upstream COSI driver in
sigs.k8s.io/container-object-storage-interfaceshould callos.Remove(path)beforenet.Listen("unix", path). Either fix alone resolves it; both are zero-risk.Environment
isp-fullvariant[email protected]+8ac248326acb