Uncloud storage module for Caddy

This is a Caddy storage module for Caddy running as a reverse proxy in an Uncloud cluster. It lets Caddy instances on different machines share TLS certificates, private keys, and ACME challenge tokens through Uncloud's cluster store. It uses distributed locks to coordinate certificate issuance.

When you run multiple instances of Caddy behind a load balancer or multi-address DNS record, an ACME challenge may reach any Caddy instance. With Caddy's default local storage, only the instance that starts issuing a certificate has the token needed to answer the challenge. If it reaches another instance, validation fails and Caddy retries. This can take a long time or fail altogether, depending on how traffic is routed.

See issue #31 for more context on this problem.

With this shared storage, any Caddy instance can read the challenge token from the cluster store and complete the ACME challenge. Once one instance obtains a certificate, the others can use it too.

Uncloud runs Caddy as a global service on every machine in the cluster. To use this module, deploy a Caddy image that includes it and add storage uncloud to Caddy's global options. You can use the pre-built image or build your own if you need another compatible Caddy version or other modules.

See Managing Caddy in the Uncloud docs for more information on Caddy deployments.

ghcr.io/unlabs-dev/caddy-uncloud image is built from the Dockerfile in this repository and includes the Uncloud storage module and a few other popular modules such as caddy-l4 and cloudflare.

Create a Caddyfile with the global storage option:

{
    storage uncloud
}

Deploy Caddy with the pre-built image and your global config.

uc caddy deploy --image ghcr.io/unlabs-dev/caddy-uncloud:0.1.3 --caddyfile Caddyfile

Uncloud combines your global config with the sites it generates for published services. Check the resulting Caddyfile and service status:

uc caddy config
uc inspect caddy

Build your own image if you need other Caddy modules or want to choose a compatible Caddy version. Create the following .env, Dockerfile, and compose.yaml in the same directory.

Set the desired upstream Caddy version in .env:

CADDY_VERSION=2.11.4

The Dockerfile builds the chosen Caddy version with the Uncloud storage module using xcaddy. You can add any other modules to the xcaddy build command. Pin module versions if you want to avoid breaking changes in the future.

ARG CADDY_VERSION

FROM caddy:${CADDY_VERSION}-builder AS builder
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    xcaddy build \
        --with github.com/unlabs-dev/caddy-uncloud/storage
    # Include other Caddy modules you need here:
    #   --with github.com/caddy-dns/[email protected]

FROM caddy:${CADDY_VERSION}
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

The Compose file defines a custom build and deploy configuration for the Caddy service:

services:
  caddy:
    build:
      args:
        CADDY_VERSION: ${CADDY_VERSION}
      # If all your cluster machines use the same architecture as your local machine, you can omit `platforms`.
      platforms:
        - linux/amd64
        - linux/arm64
    image: local/caddy-uncloud:${CADDY_VERSION}-{{date "20060102-150405"}}
    environment:
      # unix// is not a typo. Caddy uses network/address format, not a unix:// URL.
      CADDY_ADMIN: unix//run/caddy/admin.sock
    healthcheck:
      test: curl -fsS -o /dev/null --unix-socket /run/caddy/admin.sock http://localhost/config/
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
      start_interval: 1s
    volumes:
      - /var/lib/uncloud/caddy:/etc/caddy:ro
      - /run/uncloud/api:/run/uncloud/api:ro
      - /run/uncloud/caddy:/run/caddy
    x-ports:
      - 80:80@host
      - 443:443@host
      - 443:443/udp@host
    x-caddy: |
      {
          storage uncloud
      }
    deploy:
      mode: global

Keep the admin address, mounts, and host ports as shown. Uncloud combines your global config x-caddy with the sites it generates for published services and writes the resulting Caddyfile to the mounted /etc/caddy. The storage module connects to Uncloud API through the Unix socket at /run/uncloud/api/uncloud.sock.

Run uc deploy from the directory containing these three files. Then check that storage uncloud appears in the updated Caddy config:

uc deploy
uc caddy config

uc deploy builds the image with your local Docker, pushes it directly to cluster machines, and updates the Caddy service. You do not need a registry.

The module has two optional settings:

For example, to set both options explicitly:

{
    storage uncloud {
        socket /run/uncloud/api/uncloud.sock
        lock_ttl 20s
    }
}

The module sends Caddy's storage operations to the local Uncloud API. Uncloud stores and replicates the data across the cluster using Corrosion. The replicated store is eventually consistent, so reads may return older data than the most recent write.

To ensure that any Caddy instance reads the latest data, it makes a best-effort attempt to catch up with the cluster for each read operation. It broadcasts a request for store versions, skips failed machine responses, and waits for the local replica to reach the versions reported by responding machines. If version collection or replication waiting fails, the module logs a warning and still proceeds with the local read.

Waiting normally makes the captured data available locally, but it does not guarantee an exact snapshot or cover writes on unavailable machines.

The module uses distributed locks to coordinate concurrent certificate issuance. When a Caddy instance needs to issue a certificate, it first acquires a lock for the domain. If another instance already holds the lock, it waits until the lock is released. The lock is automatically renewed while held, and it expires after the configured lock_ttl if the holder fails to renew it. This prevents multiple instances from issuing the same certificate at the same time.