The user agent string is free text. A scraper that wants to be let through says it is Googlebot, and a log that trusts the string will believe it. Two proofs do hold, both published by the crawler operators themselves, and one IP lookup applies both. This guide is about what they prove, what they do not, and how to read the three fields that carry the answer.
Proof one: the operator's published ranges
The large crawler operators publish the IP ranges their crawlers fetch from, precisely so that sites can verify them. The files are public, machine-readable and refreshed by the operators as their fleets change:
- Google:
https://developers.google.com/static/crawling/ipranges/common-crawlers.jsonfor Googlebot, plus separate files for its special crawlers and its user-triggered fetchers. - Microsoft:
https://www.bing.com/toolbox/bingbot.jsonfor Bingbot. - OpenAI:
https://openai.com/gptbot.jsonfor GPTBot,https://openai.com/chatgpt-user.jsonfor ChatGPT-User andhttps://openai.com/searchbot.jsonfor OAI-SearchBot. - Perplexity:
https://www.perplexity.ai/perplexitybot.jsonandhttps://www.perplexity.ai/perplexity-user.json. - DuckDuckGo:
https://duckduckgo.com/duckduckbot.json. - Ahrefs:
https://api.ahrefs.com/v3/public/crawler-ip-ranges.
Spaw syncs every one of these daily and compiles them into one feed. An address inside a published range answers is_crawler: true, crawler names the operator's crawler (Googlebot, Bingbot, GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, DuckDuckBot, AhrefsBot, or Google's special crawlers and user-triggered fetchers), and crawler_kind says what it does: search for an index crawler, ai_training for a crawler collecting training data, ai_assistant for a fetcher acting on a user's request in an assistant, seo for a link-index crawler, fetcher for Google's on-demand fetchers. The feed is tri-state like every other: false is a checked negative against every published file, null means no file covered the address family. The current ranges, with the date each file was synced, are published on the crawler pages.
Proof two: forward-confirmed reverse DNS
Some operators document a second proof that works even for addresses their files do not list. Look up the PTR record that names the address, then resolve that name forward and check the address is among its answers. Anyone can publish a PTR name for a block they control; only the owner of the name's domain can make it resolve back. Google documents this for Googlebot (names under googlebot.com and google.com), Microsoft for Bingbot (search.msn.com), Apple for Applebot (applebot.apple.com — Apple publishes no range file, so this is the only proof for it), and Yandex and Baidu for theirs.
Spaw runs the check when a lookup asks for it with hostname: true, because it is a live DNS query with a two-second budget. hostname is the PTR name and hostname_verified whether it forward-confirmed; a verified name under one of those documented domains proves the crawler and sets is_crawler even where no published range does. An unverified name proves nothing and is never used as evidence. OpenAI documents ranges and not reverse DNS, so for GPTBot the range file is the proof.
What the proofs do and do not say
A verified crawler is the operator's machine. That is all. It says nothing about intent: a real Googlebot address fetches whatever Google's systems decided to fetch, and whether you want GPTBot collecting training data from your pages is a policy question that robots.txt and the operators' documented directives answer, not the IP.
That is why a crawler carries no risk weight, and why the datacenter weight is skipped for it. Every crawler lives on infrastructure by definition; scoring it as a datacenter visitor would flag exactly the traffic a publisher most wants to recognise. A crawler on a published range scores 0 and reads low.
The reverse holds too. A request that claims to be Googlebot from an address outside Google's ranges, whose PTR name does not forward-confirm, is not Googlebot. It may be a benign tool, a research crawler, or a scraper hoping the name opens doors; the answer says only that the operator's own proofs do not vouch for it.
Recipes
Per request. Look the address up and branch on crawler and crawler_kind. Serve search crawlers what you serve everyone; decide separately, per crawler_kind, what AI training crawlers and AI assistants may have; treat an unverified "Googlebot" as an ordinary visitor and let the rest of the answer — datacenter, attack-source, your own lists — speak.
With your own lists. If you have decided to allow a crawler through a stricter policy, an allow list keyed on its published ranges makes the decision explicit and auditable; the ranges are on the crawler pages, dated, and the list is yours to update when the operator changes its fleet.
In logs. For traffic already logged, a bulk run over the addresses answers crawler and crawler_kind for every row and the run's summary counts them, so a week of "Googlebot" hits resolves into the share that was Google and the share that was pretending.
In robots.txt. Verification tells you who is at the door; the directives tell them what they may take. The two are complementary, and the operators' documentation for each crawler names the user agent tokens the directives apply to.
For the operators without a file. Applebot, YandexBot and Baiduspider publish no range file, so the only proof for them is the reverse DNS check, which costs a live query. Ask for it only on the requests where it matters — a suspicious burst claiming one of those names — rather than on every hit, and remember that a two-second resolver stall answers nothing rather than delaying the request twice.
Sources
- Google's verification guide: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
- Microsoft's Bingbot verification guide: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26
- OpenAI's crawler documentation and range files: https://platform.openai.com/docs/bots
- The range file URLs above, as synced daily on 12 September 2026; the current copies and their dates are on the crawler pages.
The IP intelligence guide documents is_crawler, crawler, crawler_kind, hostname and hostname_verified in full.