Skip to content

Two datasets had stopped syncing and nothing said so

published · September 8, 2026

Two weekly data syncs were failing, and because a scheduled background task throws its output away, the failures left no trace anywhere. The datasets were missing from the status page before anyone noticed.

Impact
Discovered operator geofeeds and the Norwegian address register were not installed, so answers that would have used them fell back to what was left
Found
8 September 2026
Resolved
8 September 2026
Cause
Failing background tasks wrote their output to nowhere

Two of the weekly data syncs were failing: the discovery run that collects the geofeeds network operators publish about their own address ranges, and the Norwegian national address register. Both showed on the status page as datasets that had never been installed, which was true, and neither had reported a failure anywhere, which is the actual incident.

Nobody was told a wrong answer. What they got was a thinner one: an IP inside a range whose operator publishes its own location was located from the geolocation database like any other, and a Norwegian address was checked against its country's postal format rather than against the register of real addresses.

What caused it

A scheduled command sends its output nowhere unless it is told otherwise, and the two syncs are also run in the background because they download and compile very large files. That hides the output a second time. So a task could fail every week, for as long as it liked, and produce not one line anywhere that anybody would ever read.

What changed

Every scheduled task that runs in the background now appends its output to a log file, and a test walks the schedule and fails, naming the task, if one of them does not. That is the direct fix, and it only helps somebody who goes looking.

The check that does not need anybody to go looking is the daily freshness sweep. Once a day it reads the same list of datasets the status page publishes and warns about anything that is not installed, or whose version has aged past the window that dataset is expected to be refreshed within. It reads the published list rather than a list of its own, so a dataset added later is covered the day it appears on the page. And because it judges the data rather than the job, it still fires when the sync command is not running at all — which was precisely the case here.

What it does not tell you

How long the two syncs had been failing is not recorded, because nothing was recording it. That is the reason the incident is worth publishing: the fault was not that a download broke, it was that a download could break in silence.

Related

markdown version: /incidents/2026-09-08-two-datasets-had-stopped-syncing.md

Verify addresses the same way

The Spaw API runs every check described here on each lookup, with an SMTP handshake that never sends mail. 10 free lookups a month, no card required.

Get your API key

More incident history

  • Four datasets stopped refreshing and nobody was told

    Four datasets behind the answers had stopped refreshing, the oldest copy dating from 6 September, and the daily check that exists to catch exactly that reached nobody. A fifth was flagged by a freshness window set too tight.

  • IP answers lost their location and network flags

    Files a scheduled job had synced correctly were written into directories the web process could not enter, so IP lookups answered with no location and no network operator, and the datacenter, VPN and Tor flags read false.

  • Documentation and content pages answered 500 in production

    A class the app loads at runtime was only installed for development, so every documentation page, content page and the sitemap failed on the production server while the lookup endpoints kept answering.