Stop REST /health from leaking error text or taking every /api route down #405

Summary

The REST /health route runs a database query, a Redis PING and a Supabase call on every request. It skips the app rate limiter, and its Traefik router has no rate limit. The rest-api Docker healthcheck calls this same route. Traefik stops routing to a container whose Docker health is unhealthy. So a /health flood, or a Redis stall of about 50 s, can mark both replicas unhealthy and take every /api route down. The REST routes keep working without Redis, because the rate limiter fails open.

When a check fails, the response also carries the raw driver error text.

• Severity: Medium (availability), Low (information leak)
• Area: apps/hocuspocus.server health routes, docker-compose.prod.yml
• Source: security review of 2026-10-06, findings M12 and L16

Where

• Traefik router rest-api-health has security-headers@file only: docker-compose.prod.yml:228-233.
• The Traefik load-balancer probe already uses liveness (/health/live): docker-compose.prod.yml:234-241. This part is correct and stays.
• The Docker healthcheck calls /health: docker-compose.prod.yml:242-256. Its comment (:242-244) says the deploy waits on this status, so it must check a dependency, not liveness.
• The deploy waits on that status before it stops the old replicas: .github/workflows/prod.docs.plus.yml:502-518.
• The app limiter skips /health and /health/*: apps/hocuspocus.server/src/middleware/index.ts:248-251.
• The limiter fails open when Redis is down: apps/hocuspocus.server/src/middleware/index.ts:71-90.
• Checks: apps/hocuspocus.server/src/api/services/health.service.ts:39-124, combined in checkAllServices (:126-153). Redis counts as critical at :142-144.
• Raw error text: health.service.ts:51, :86 and :121 set error: errorMessage(error).
• Every /health/* sub-route also runs its check uncached: apps/hocuspocus.server/src/api/routers/health.router.ts:15-17.
• Consumers read the HTTP status only, never the body:
  • The admin dashboard: apps/admin-dashboard/src/services/api.ts:201-214 (checkApiHealth).
  • The blackbox probe: scripts/observability/prometheus/prometheus.yml:67-68.
  • The deploy smoke: .github/workflows/prod.docs.plus.yml:744.

Fix plan

1. Cache each dependency check for 5 s in health.service.ts.
   • Wrap checkDatabaseHealth, checkRedisHealth and checkSupabaseHealth in one small memo. It keeps the last promise and its start time, so concurrent callers share one check.
   • This covers /health and the /health/database, /health/redis and /health/supabase sub-routes.
   • Export a clearHealthCache() and call it in beforeEach of tests/integration/health.test.ts. Those tests swap the mocks per test.
2. Drop the error text from responses. In the three catch blocks, log the error with restApiLogger.warn({ err, check }) and return { status: 'unhealthy', lastCheck } with no error field.
   • Remove error from the HealthCheckResult schema in apps/hocuspocus.server/src/modules/openapi/domain/components.ts:124.
   • Remove error? from HealthCheckResult in src/types/health.types.ts:6. Nothing else sets it.
3. Point the Docker healthcheck at /health/database in docker-compose.prod.yml:251 and docker-compose.dev.yml:132.
   • The deploy still waits for a working database, which REST cannot serve without.
   • A Redis stall no longer marks the replicas unhealthy.
   • Update the comment at docker-compose.prod.yml:242-244 to say why Redis is left out.
4. Update apps/hocuspocus.server/API.md §Health. Say the checks are cached for 5 s and failures carry no error text.

Layer: the fix stays in health.service.ts and the compose healthcheck. Do not change the Traefik /health/live probe. Do not change what /health reports as degraded; the dashboard and the blackbox alert need Redis in it. This is the rest-api container only. The Next Health rule in CLAUDE.md §Next product APIs covers the webapp probe, not this one. Still get maintainer sign-off on the compose change, because its comment records a deliberate choice.

Out of scope

• A Traefik rate limit on rest-api-health. With the cache, a request costs no dependency call. Add one only if a flood still shows up in metrics.
• The /health/push body. Move the email and push gateway reports behind admin auth #433 moves the push and email gateway reports behind admin auth.
• Redis metadata in /health/redis. It holds connection state and queue lengths, not hosts or ports.

Acceptance criteria

• With Redis stopped for 60 s, the rest-api containers stay healthy, and /api routes still answer.
• With the database stopped, the rest-api containers turn unhealthy, as before.
• No /health or /health/* response contains a driver message, host name or port.
• 1000 rapid GET /health calls cause at most one database query per 5 s window.
• The admin dashboard System page and the blackbox probe still report status.

Verify

• cd apps/hocuspocus.server && bun test tests/integration/health.test.ts. Update the tests that assert data.error (:88-89, :103, :145-146, :160, :186) to assert that no error field exists.
• Add one test: two calls inside 5 s call $queryRaw once.
• Dev stack: docker compose -f docker-compose.dev.yml stop redis, wait 70 s, then docker inspect -f '{{.State.Health.Status}}' <rest-api container> reads healthy. Start Redis again.

Related

• Move the email and push gateway reports behind admin auth #433 (email and push gateway reports are public)
• Keep raw database error text out of production 500 responses #413 (raw error text in 500 responses, fixed)

Generated by Claude Code