Skip to content

Commit 1ad6f58

Browse files
HMarzbanclaude
andcommitted
fix(hocuspocus): cache REST health checks and keep error text out of them
Each REST health check now costs at most one dependency call every 5 s per process. Concurrent callers share one in-flight check, so a /health flood no longer reaches the database, Redis or Supabase on every request. A failed check returns only its status and time. The driver text could name a host and a port, so it now goes to the server log as a 'Health check failed' warning. The OpenAPI schema drops the error field, and the database probe no longer claims pool metadata. The rest-api container healthcheck now reads /health/database in both compose files. Both replicas share one Redis, so a Redis stall marked both unhealthy, and Traefik then had no server for any /api route. REST runs without Redis, because its limiter fails open. The public API guide now names /health/live for a load-balancer check. Refs #405 Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 966f2a9 commit 1ad6f58

12 files changed

Lines changed: 79 additions & 65 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,6 +287,9 @@ This file is the product changelog. The webapp package keeps [`apps/webapp/CHANG
287287
error from the shared error handler says only "Internal server error".
288288
A handler that sets its own message keeps it
289289
([#413](https://github.com/docs-plus/docs.plus/issues/413)).
290+
- Keep database and Redis error text out of the REST health responses, and
291+
stop a Redis stall from taking every `/api` route down
292+
([#405](https://github.com/docs-plus/docs.plus/issues/405)).
290293
- Show a plain, fixed message on the sign-in error page. An expired or used
291294
sign-in link says so. Every other error shows one general message
292295
([#411](https://github.com/docs-plus/docs.plus/issues/411)).

‎apps/hocuspocus.server/API.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ The documents controller emits this envelope on error and `{ "success": true, "d
7575
7676
## Health
7777

78-
Health routes are exempt from rate limiting. Each returns `200` when healthy and `503` otherwise. Source: `src/api/routers/health.router.ts`, `src/api/services/health.service.ts`.
78+
Health routes are exempt from rate limiting. Each returns `200` when healthy and `503` otherwise. Source: `src/api/routers/health.router.ts`, `src/api/services/health.service.ts`. Each dependency check is cached for 5 s per process, and concurrent callers share one check. A failed check returns no error text. The reason goes to the server log as a `Health check failed` warning. The `rest-api` container healthcheck reads `/health/database`, so a Redis stall does not mark the replicas unhealthy ([#405](https://github.com/docs-plus/docs.plus/issues/405)).
7979

8080
### GET /health
8181

@@ -86,7 +86,7 @@ Aggregate check across database, Redis, and Supabase. Returns `degraded` (`503`)
8686
"status": "ok",
8787
"timestamp": "2026-06-15T12:00:00.000Z",
8888
"services": {
89-
"database": { "status": "healthy", "lastCheck": "...", "metadata": { "pool": {} } },
89+
"database": { "status": "healthy", "lastCheck": "..." },
9090
"redis": { "status": "healthy", "lastCheck": "...", "metadata": {} },
9191
"supabase": { "status": "healthy", "lastCheck": "..." }
9292
}
@@ -95,9 +95,13 @@ Aggregate check across database, Redis, and Supabase. Returns `degraded` (`503`)
9595

9696
Per-service `status` is one of `healthy`, `unhealthy`, or `disabled` (Redis/Supabase when not configured).
9797

98+
### GET /health/live
99+
100+
Liveness only. Answers `200` while the process is up. It calls no dependency, so it never answers `503`. The Traefik probe reads it. Use it for a load-balancer check.
101+
98102
### GET /health/database
99103

100-
Database connectivity (`SELECT 1`) plus pool metadata.
104+
Database connectivity (`SELECT 1`).
101105

102106
### GET /health/redis
103107

‎apps/hocuspocus.server/CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -499,6 +499,15 @@ SMTP_PASS=...
499499
still answers `400` with its own message
500500
([#413](https://github.com/docs-plus/docs.plus/issues/413)).
501501

502+
- **REST health checks are cached, and their bodies carry no error text.**
503+
Each database, Redis and Supabase check is cached for 5 s per process. A
504+
`/health` flood then costs at most one call per dependency every 5 s on
505+
each replica. A failed check no longer returns driver error text. The
506+
server logs the reason instead. The `rest-api` container healthcheck now
507+
reads `/health/database`, so a Redis stall no longer marks both replicas
508+
unhealthy and drops every `/api` route
509+
([#405](https://github.com/docs-plus/docs.plus/issues/405)).
510+
502511
- **Service-role functions refuse `public`, `anon` and `authenticated`.** The
503512
Supabase migration `20261006130000_close_client_write_and_grant_gaps`
504513
revokes them. It also moves the three `28-ghost-accounts-audit.sql`

‎apps/hocuspocus.server/CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,7 @@ Moved verbatim out of the repo-root [AGENTS.md](../../AGENTS.md). The `### Supab
223223
### Production And Docker Compose
224224

225225
- Dev compose backend services need `context: .` at repo root to match `Dockerfile.bun`.
226+
- **The `rest-api` Docker healthcheck reads `/health/database` on purpose ([#405](https://github.com/docs-plus/docs.plus/issues/405)).** Do not point it back at `/health`. Both replicas share one Redis, so a Redis stall marks both unhealthy, and Traefik then has no server for any `/api` route. Do not point it at `/health/live` either. The deploy waits on this status, so it needs a real dependency. Each REST health check is cached for 5 s per process, and a failure body carries no error text. The Settled rule against a Docker HEALTHCHECK change covers only the webapp Next probe.
226227
- Hocuspocus image:
227228
- `migration-extensions.ts` imports `@docs.plus/extension-hypermultimedia` and `@docs.plus/extension-inline-code` at runtime.
228229
- Root `.dockerignore` excludes `**/dist`.

‎apps/hocuspocus.server/src/api/services/health.service.ts‎

Lines changed: 38 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import type { PrismaClient } from '@prisma/client'
22

3+
import { restApiLogger } from '../../lib/logger'
34
import { getRedisStats } from '../../lib/redis'
45
import { getAnonClient } from '../../lib/supabase'
56
import type { HealthCheckResult, OverallHealthResult, RedisClient } from '../../types'
@@ -10,12 +11,28 @@ import type { HealthCheckResult, OverallHealthResult, RedisClient } from '../../
1011
// default is 60 s, so a slow dependency failed the probe on the clock.
1112
const HEALTH_CHECK_TIMEOUT_MS = 2000
1213

13-
// postgrest-js rejects with a plain object, not an Error, so an instanceof test
14-
// alone dropped the real reason and reported "Unknown error" to the operator.
15-
const errorMessage = (error: unknown): string =>
16-
error instanceof Error
17-
? error.message
18-
: ((error as { message?: string } | null)?.message ?? 'Unknown error')
14+
// A /health flood must not cost a dependency call per request (#405). Each check
15+
// catches its own error and never rejects, so a cached promise never holds a rejection.
16+
const HEALTH_CACHE_MS = 5000
17+
const cache = new Map<string, { at: number; result: Promise<HealthCheckResult> }>()
18+
19+
export const clearHealthCache = (): void => cache.clear()
20+
21+
// Keyed on the check alone: prod hands every request the same clients.
22+
const memo = (check: string, run: () => Promise<HealthCheckResult>): Promise<HealthCheckResult> => {
23+
const hit = cache.get(check)
24+
if (hit && Date.now() - hit.at < HEALTH_CACHE_MS) return hit.result
25+
const result = run()
26+
cache.set(check, { at: Date.now(), result })
27+
return result
28+
}
29+
30+
// The body carries no driver text, because it can name a host and a port. The
31+
// reason goes to the log instead.
32+
const unhealthy = (check: string, error: unknown): HealthCheckResult => {
33+
restApiLogger.warn({ err: error, check }, 'Health check failed')
34+
return { status: 'unhealthy', lastCheck: new Date() }
35+
}
1936

2037
const withDeadline = async <T>(work: Promise<T>, label: string): Promise<T> => {
2138
let timer: ReturnType<typeof setTimeout> | undefined
@@ -36,7 +53,7 @@ const withDeadline = async <T>(work: Promise<T>, label: string): Promise<T> => {
3653
}
3754
}
3855

39-
export const checkDatabaseHealth = async (prisma: PrismaClient): Promise<HealthCheckResult> => {
56+
const runDatabaseCheck = async (prisma: PrismaClient): Promise<HealthCheckResult> => {
4057
try {
4158
await withDeadline(prisma.$queryRaw`SELECT 1`, 'database')
4259

@@ -45,15 +62,11 @@ export const checkDatabaseHealth = async (prisma: PrismaClient): Promise<HealthC
4562
lastCheck: new Date()
4663
}
4764
} catch (error) {
48-
return {
49-
status: 'unhealthy',
50-
lastCheck: new Date(),
51-
error: errorMessage(error)
52-
}
65+
return unhealthy('database', error)
5366
}
5467
}
5568

56-
export const checkRedisHealth = async (redis: RedisClient | null): Promise<HealthCheckResult> => {
69+
const runRedisCheck = async (redis: RedisClient | null): Promise<HealthCheckResult> => {
5770
if (!redis) {
5871
return {
5972
status: 'disabled',
@@ -80,15 +93,11 @@ export const checkRedisHealth = async (redis: RedisClient | null): Promise<Healt
8093
: undefined
8194
}
8295
} catch (error) {
83-
return {
84-
status: 'unhealthy',
85-
lastCheck: new Date(),
86-
error: errorMessage(error)
87-
}
96+
return unhealthy('redis', error)
8897
}
8998
}
9099

91-
export const checkSupabaseHealth = async (): Promise<HealthCheckResult> => {
100+
const runSupabaseCheck = async (): Promise<HealthCheckResult> => {
92101
try {
93102
// Inside the try on purpose. SUPABASE_URL is validated as a non-empty string
94103
// only, so a malformed value throws here, and checkAllServices below would
@@ -115,14 +124,19 @@ export const checkSupabaseHealth = async (): Promise<HealthCheckResult> => {
115124
lastCheck: new Date()
116125
}
117126
} catch (error) {
118-
return {
119-
status: 'unhealthy',
120-
lastCheck: new Date(),
121-
error: errorMessage(error)
122-
}
127+
return unhealthy('supabase', error)
123128
}
124129
}
125130

131+
export const checkDatabaseHealth = (prisma: PrismaClient): Promise<HealthCheckResult> =>
132+
memo('database', () => runDatabaseCheck(prisma))
133+
134+
export const checkRedisHealth = (redis: RedisClient | null): Promise<HealthCheckResult> =>
135+
memo('redis', () => runRedisCheck(redis))
136+
137+
export const checkSupabaseHealth = (): Promise<HealthCheckResult> =>
138+
memo('supabase', runSupabaseCheck)
139+
126140
export const checkAllServices = async (
127141
prisma: PrismaClient,
128142
redis: RedisClient | null

‎apps/hocuspocus.server/src/modules/openapi/domain/components.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,6 @@ const healthCheckResult: JsonSchema = {
121121
properties: {
122122
status: { type: 'string', enum: ['healthy', 'unhealthy', 'disabled'] },
123123
lastCheck: { type: 'string', format: 'date-time' },
124-
error: { type: 'string' },
125124
metadata: { type: 'object', additionalProperties: true }
126125
},
127126
required: ['status', 'lastCheck']

‎apps/hocuspocus.server/src/modules/openapi/domain/paths/health.ts‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -68,11 +68,7 @@ export const healthPaths: OpenApiPaths = {
6868
}
6969
},
7070
'/health/database': {
71-
get: probe(
72-
'getHealthDatabase',
73-
'Database health',
74-
'Connectivity (`SELECT 1`) plus pool metadata.'
75-
)
71+
get: probe('getHealthDatabase', 'Database health', 'Connectivity (`SELECT 1`).')
7672
},
7773
'/health/redis': {
7874
get: probe(

‎apps/hocuspocus.server/src/types/health.types.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ export type HealthStatus = 'healthy' | 'unhealthy' | 'disabled'
33
export interface HealthCheckResult {
44
status: HealthStatus
55
lastCheck: Date
6-
error?: string
76
metadata?: any
87
}
98

‎apps/hocuspocus.server/tests/integration/health.test.ts‎

Lines changed: 13 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import { describe, test, expect, beforeAll, beforeEach, mock } from 'bun:test'
22
import { Hono } from 'hono'
33
import healthRouter from '../../src/api/routers/health.router'
4+
import { clearHealthCache } from '../../src/api/services/health.service'
45
import { TestServer, createMockPrisma, createMockRedis } from '../helpers/test-server'
56

67
// Mutable reference updated per-test so the mock factory always reads current value.
@@ -24,6 +25,8 @@ describe('Health Check API', () => {
2425
})
2526

2627
beforeEach(() => {
28+
// Each test swaps the mocks, so a cached result would read the last test.
29+
clearHealthCache()
2730
mockPrisma = createMockPrisma()
2831
mockRedis = createMockRedis()
2932
// Default: no Supabase client → checkSupabaseHealth returns 'disabled' instantly.
@@ -85,22 +88,21 @@ describe('Health Check API', () => {
8588

8689
expect(response.status).toBe(503)
8790
expect(data.status).toBe('unhealthy')
88-
expect(data).toHaveProperty('error')
89-
expect(data.error).toBe('Database connection failed')
91+
expect(data).not.toHaveProperty('error')
9092
expect(data).toHaveProperty('lastCheck')
9193
})
9294

93-
test('should handle non-Error exceptions', async () => {
95+
test('should share one database query across calls inside the cache window', async () => {
96+
let calls = 0
9497
mockPrisma.$queryRaw = async () => {
95-
throw 'String error'
98+
calls += 1
99+
return [{ '?column?': 1 }]
96100
}
97101

98-
const response = await testServer.get('/health/database')
99-
const data = await response.json()
102+
await testServer.get('/health/database')
103+
await testServer.get('/health/database')
100104

101-
expect(response.status).toBe(503)
102-
expect(data.status).toBe('unhealthy')
103-
expect(data.error).toBe('Unknown error')
105+
expect(calls).toBe(1)
104106
})
105107
})
106108

@@ -142,23 +144,9 @@ describe('Health Check API', () => {
142144

143145
expect(response.status).toBe(503)
144146
expect(data.status).toBe('unhealthy')
145-
expect(data).toHaveProperty('error')
146-
expect(data.error).toBe('Redis connection refused')
147+
expect(data).not.toHaveProperty('error')
147148
expect(data).toHaveProperty('lastCheck')
148149
})
149-
150-
test('should handle non-Error exceptions in redis', async () => {
151-
mockRedis.ping = async () => {
152-
throw 'String error'
153-
}
154-
155-
const response = await testServer.get('/health/redis')
156-
const data = await response.json()
157-
158-
expect(response.status).toBe(503)
159-
expect(data.status).toBe('unhealthy')
160-
expect(data.error).toBe('Unknown error')
161-
})
162150
})
163151

164152
describe('GET /health/supabase', () => {
@@ -183,7 +171,7 @@ describe('Health Check API', () => {
183171

184172
expect(response.status).toBe(503)
185173
expect(data.status).toBe('unhealthy')
186-
expect(data).toHaveProperty('error')
174+
expect(data).not.toHaveProperty('error')
187175
expect(data).toHaveProperty('lastCheck')
188176
})
189177

‎docker-compose.dev.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ services:
129129
'CMD',
130130
'bun',
131131
'-e',
132-
"fetch('http://localhost:4000/health').then(r => r.ok ? process.exit(0) : process.exit(1)).catch(() => process.exit(1))"
132+
"fetch('http://localhost:4000/health/database').then(r => r.ok ? process.exit(0) : process.exit(1)).catch(() => process.exit(1))"
133133
]
134134
interval: 10s
135135
timeout: 5s

0 commit comments

Comments
 (0)