2026-08-26T23:34:40.468ZTimeline. First signs appeared Jul 12 around 14:00 UTC, when /health began returning 500s intermittently under load. Through Jul 13 the issue simmered — /status.json occasionally threw 3-second Promise.race timeouts on slow Neon round-trips. By Jul 14 the service was hard down for most tenants, and we held in that state through Jul 15 while we worked the migration rollback. Full restoration completed Jul 16 at 00:00 UTC.
Root cause. A routine schema migration triggered migrate.js to fatal-exit partway through applying changes. The Node process crashed, but the in-flight connection pool maintained by @neondatabase/serverless held open sockets waiting for a process that would not return. Subsequent retry storms from monitoring and webhook delivery exhausted the remaining capacity, after which every request either wedged or timed out.
Resolution. Tier 1 shipped the migrate.js fix (idempotent guard + structured rollback so a partial application cannot crash the process). Diagnostics were added: a /__healthz endpoint with a 2s ceiling on SELECT 1, an in-memory 30s status cache, and a per-check 3s ceiling covering both the snapshot aggregation and the JSON endpoint. We walked every dependency to confirm none could wedge indefinitely.
Permanent fixes shipped. (1) routes/startup-diagnostics.js:probeDb() — 2s Promise.race ceiling. (2) routes/status.js — 30s in-memory cache extended to 5 minutes when DB wedges. (3) getStatusSnapshot — per-check 3s ceiling with capped() helper. (4) /api/status.json — Cache-Control: no-store and always 200, so external widgets and procurement screenshots never 5xx during a Neon blip.
Forward posture. This status page is part of the fix. We log every incident publicly — including this one — because buyers screenshot status pages into procurement decks and the audit trail matters. You can subscribe below to get an email when something breaks.