Skip to content

Four datasets stopped refreshing and nobody was told

published · October 14, 2026 · updated · October 16, 2026

Four datasets behind the answers had stopped refreshing, the oldest copy dating from 6 September, and the daily check that exists to catch exactly that reached nobody. A fifth was flagged by a freshness window set too tight.

Impact
Answers drew on older copies of four datasets, the oldest from 6 September, and a Danish address created after 26 September could be answered as one that does not exist
Not affected
Every other dataset on the status page, including the Norwegian address register, which was flagged but refreshing on schedule
Found
14 October 2026
Resolved
Partly. The United States address register and the UK number ranges were restored on 15 October and the geofeeds on 16 October; the Danish register waits until the sync reads its new files
Cause
Four unrelated refresh failures, and an alert path nobody had verified

By 14 October four datasets behind the answers had stopped refreshing, and the status page said so, but nobody here had been told. The last copies installed were these: the geofeeds network operators publish about their own address ranges, from 20 September; the UK number ranges, from 27 September; the Danish address register, from 26 September; and the United States address register, from 6 September. A fifth row, the Norwegian address register, read stale as well. It was being refreshed every week as scheduled, and was flagged because its freshness window was set too tight.

Lookups kept answering throughout, from the copies already installed, and nothing was refused. Most answers were accurate about the copy they came from, which was older than the status page allows. One kind was wrong. The Danish register was treated as complete, because its publisher lists every address in the country, so an address created there after 26 September could be answered as one that does not exist. Every answer names the datasets and versions behind it in its sources field, so a caller can tell which of their answers came from an older copy.

What happened to each dataset

  • United States address register. On 30 September, hours before the monthly sync, the publisher replaced its file with a new release. The release is packed with a compression method, Deflate64, that the sync's archive reader did not support, so the run of 1 October could not install it whether or not the 8.5 GB download completed, and the copy of 6 September stayed. The reader now inflates Deflate64, and this register's download has a longer timeout to match the larger file. The archive also gives its disk count as 0 where the format requires 1, which the server's unzip tool rejects, so the reader corrects it first. The register was restored on 15 October.
  • Danish address register. The Danish agency that publishes the register closed the interface the sync read on 1 October 2026, and every request to it now answers that the resource is gone for good. The register is still published, as separate files through the national data distributor, but the sync does not read those yet. Until it does, Spaw marks the register partial, so a Danish address the copy of 26 September does not hold now reads as unknown, not absent.
  • UK number ranges. In September the regulator's website was recorded answering the server with a bot check instead of the files, and no weekly sync has replaced the copy of 27 September since. The files are fetched elsewhere and installed by hand, and a copy is now dated by the day the regulator published it, read from the files' own timestamps, rather than by the day it was installed. The regulator's files of 14 October were installed on 15 October.
  • Operator geofeeds. These come from a long crawl every Sunday, and no crawl after 20 September installed a result. To find the smaller blocks inside one regional registry's 88,786 allocations, the crawl ran a lookup for each of them with a command-line tool that was not installed on the server. Every lookup failed at once, but together they held the crawl up for about 19 minutes. Another registry's download, running alongside, stalled in that time and its server dropped the connection, which the crawl only noticed when the connection timed out about two hours later. It then discarded every registry's data, not only the one that failed. The crawl now skips that step, which could find nothing on this server, and keeps its own output, so a failure names its cause. A crawl run by hand on 16 October installed a fresh copy.
  • Norwegian address register. Not a failure. Its publisher releases each weekly extract on Saturday afternoon, hours after the Saturday sync, so every sync installs the extract from the week before. The window assumed a daily release, so it called a copy that was on schedule stale every Wednesday, Thursday and Friday, and it now allows for the weekly release. Two registers published twice a year were days from the same mistake. Their windows now follow their publishers' calendars.

Why nobody heard

The freshness sweep runs every morning. It reads the same list of datasets the status page publishes, and writes a warning to the application log for each one past its window. It sends an email only when the server has an alert address configured. That email is queued rather than sent on the spot, so it also needs something on the server sending queued mail, and a mail transport that delivers it. If any one of those is missing, the warning ends as a line in a log file that nobody reads, and nothing reports that it went no further.

The report of 8 September said the sweep warns. That was true of the code. On the server, the warning depended on settings that nothing checked.

Which of those links broke in production is not yet established: whether an alert address was set, whether queued mail was being sent, or whether the transport delivered it. This report will be updated when it is.

What changed

When no alert address is configured, every operational alert now goes to the people who run the admin panel instead of only to the log: stale datasets, a queue that has stopped moving, failing phone feeds and low balances alike. A server with neither set still only logs, and the page an operator checks now says so.

That page, the admin panel's operations page, now lists every dataset past its window, from the same assessment the sweep and the status page use. It also states where alert mail goes and how it is sent, so someone who never opens the mail still sees what it would have said.

A command sends a test alert along the same path, through the queue by default or directly on request, so a server's alerting can be checked rather than assumed.

What it does not tell you

How long warnings had been going nowhere is not known, because the sweep's only record of a run is the log on the server. This report does not count how many answers drew on the older copies, or how many Danish addresses were answered as absent.

The fault is the one the September report described. A sync breaking was not the problem. The problem was that a broken sync could go unreported, this time past a check that exists to report it.

Related

markdown version: /incidents/2026-10-14-four-datasets-stopped-refreshing-and-nobody-was-told.md

Verify addresses the same way

The Spaw API runs every check described here on each lookup, with an SMTP handshake that never sends mail. 10 free lookups a month, no card required.

Get your API key

More incident history