Skip to content

Documentation and content pages answered 500 in production

published · September 3, 2026

A class the app loads at runtime was only installed for development, so every documentation page, content page and the sitemap failed on the production server while the lookup endpoints kept answering.

Impact
Every documentation page, content page and the sitemap answered 500 in production
Not affected
The lookup endpoints, which read none of those files
Found
3 September 2026
Resolved
3 September 2026
Cause
A runtime dependency that was only installed for development

Every documentation page, every Markdown content page and the sitemap answered 500 on the production server. Anyone reading the docs got an error page instead. The API itself was unaffected: a lookup reads none of the files involved, so calls kept being answered and billed normally throughout.

What caused it

The pages that failed all read Markdown front matter or the OpenAPI document, and both are parsed with a YAML library. That library was never declared as something the application needs. It was present on a development machine and in the test suite only because two development tools happened to pull it in, and a production deploy installs dependencies without those tools. So the class existed everywhere it was tested and nowhere it mattered.

Nothing caught it before the deploy for exactly that reason: locally, in the tests and in CI, the code worked.

What changed

The parser is now a first-class requirement of the application rather than a coincidence of somebody's tooling.

More usefully, this class of fault can no longer reach production unnoticed. A test reads the list of packages that are installed for development only, walks every class the application imports, and fails the build if any of them resolves to one of those packages. The failure now happens on a machine where somebody is watching, which is the whole difference between a mistake and an outage.

What it does not tell you

This report was written from the engineering record. The day the fault was found and fixed is known; how many hours the pages were failing before that is not, because nothing was watching them at the time. That gap is why the status page now publishes a measurement rather than a promise — and why the availability figure it shows is the share of requests that reached us, not a claim about the times nothing did.

Related

markdown version: /incidents/2026-09-03-documentation-pages-failed-in-production.md

Verify addresses the same way

The Spaw API runs every check described here on each lookup, with an SMTP handshake that never sends mail. 10 free lookups a month, no card required.

Get your API key

More incident history

  • Two datasets had stopped syncing and nothing said so

    Two weekly data syncs were failing, and because a scheduled background task throws its output away, the failures left no trace anywhere. The datasets were missing from the status page before anyone noticed.

  • IP answers lost their location and network flags

    Files a scheduled job had synced correctly were written into directories the web process could not enter, so IP lookups answered with no location and no network operator, and the datacenter, VPN and Tor flags read false.