The SEO's Guide To Web Bot Auth

One of the sillier things about the internet is that a request can turn up at your website wearing a namebadge that says “Googlebot”, and quite a lot of people will take that at face value.

You can change a user agent in about three seconds. Your crawler can claim to be Googlebot. So can somebody’s script running on a computer in their shed, and to be honest the guy checking the namebadge is not doing a tremendous amount of security work. You may have seen that I've written about this before, mainly in the context of getting your perfectly legitimate SEO tools past a firewall that has decided they look a bit shifty (or sus, if you're Gen Z).

So, Hallelujah, glory be: Web Bot Auth is here, and it is an attempt to give automated requests something more convincing than a namebadge, and while the name suggests something you can safely leave to a person with more monitors than you, there are some fairly direct implications for SEO.

So what actually is it?

It's user agents on HGH.

Web Bot Auth lets bots sign HTTP requests cryptographically, so the website receiving them has a way to verify an identity. The underlying signing standard is, say it with me: HTTP Message Signatures, RFC 9421 - A catchy name I'm sure we all agree.

Web Bot Auth builds on that for automated traffic - or rather, is building on it: As of November 2026, the Web Bot Auth protocol is still an IETF draft, so expect details to move around.

Broadly, the bot keeps a private key and makes the corresponding public key available. It signs selected parts of its request, and your infrastructure checks that signature using the public key associated with the claimed identity. You can share the public key, hence the name, and surprise surprise, the private key is the bit you very much want to keep private.

The headers you care about are:

The first two come from the HTTP Message Signatures standard; Web Bot Auth adds the bot-specific conventions around them; it's basically your new UA.

You don't need to memorise the headers to have a useful conversation about this, but you do need to understand that checking whether a header exists is very different from validating what it contains, otherwise we've just given the namebadge a more expensive printer.

But we already verify bots, don't we?

Yes. Or at least, hopefully.

Google already provides ways to check whether requests claiming to be from its crawlers actually are. You can compare the source IP against its published ranges, or perform reverse DNS followed by a forward lookup to confirm that everything matches. Google documents both approaches - those checks are for more useful than a simple UA check for all the reasons we mentioned above.

IP-based verification does, however, give you lists to maintain and distinctions to understand: For instance, moving away from Google's equipment, “This IP belongs to a cloud provider” doesn't necessarily tell you which of that provider's customers is making the request, which is a problem when you're trying to identify a known friendly crawler that uses AWS from malicious (or at least nuisance) traffic that's coming from the same source. The Web Bot Auth draft explicitly identifies shared infrastructure and self-declared user agents as problems it's trying to address, which is great - a verifiable bot identity gives website owners another basis for recognising traffic, without making the source IP its entire identity. That could make life considerably less annoying for everybody currently maintaining a spreadsheet called crawler_ips_FINAL_actually_final.csv.

Why should an SEO care?

Because somebody can build you a beautiful, fast, server-rendered website with impeccable internal linking, and a security rule can still tell the crawler to sod off before it sees any of it - and obviously the particularly irritating/dangerous extreme of this is when everything looks fine in your browser, but the automated request doesn't necessarily get the same experience.

This is why I think SEOs need to be involved in conversations about bot authentication. We have a fairly obvious interest in which automated clients can retrieve our pages, what they receive, and what happens when a security setting changes.

Better identification could help with that. It could also make your reporting more useful - I'd much rather investigate traffic attributed to a verified identity than traffic attributed to whichever bot somebody felt like impersonating that morning.

HOWEVER!

Knowing who it is doesn't mean you have to let it in

A valid signature doesn't tell you that a bot is well behaved, commercially useful, or entitled to your content. The protocol separates identity verification from the website's decision about what that identity may do. A person can show you a perfectly legitimate driving licence and still be someone you don't want rummaging through your fridge.

You might want to let one bot retrieve public product pages, rate-limit another, and block a third. Those are still decisions you need to make, and be careful about the level at which you're making them. Recognising a provider doesn't, by itself, tell you that every request serves the same purpose. Search discovery, model training and an agent fetching something because a user asked it to are different activities with potentially very different value to your business. Please don't collapse them into a single checkbox labelled “trusted AI”.

This also (possibly) matters for the content-payment conversation. Identifying a caller could be useful when deciding which access terms to offer it, maybe, but that's a future where 402 takes off and I'm not sure that'll happen any time soon.

What is Google actually doing?

This is the bit I'd want everyone to read before doing anything - in fact I'll put it in bold:

Google describes its Web Bot Auth implementation as experimental. Some requests made by Google-Agent are signed using the https://agent.bot.goog identity. Google explicitly says it isn't signing every request from participating agents, and recommends retaining established verification methods.

Does this make it annoyingly useless at the moment? Kinda, yeah.

So, if your takeaway is "Google has moved crawling to signed requests; we can block anything unsigned", please put the mouse down; you're about to create a problem that will eventually become somebody else's P1 in Jira.

So What Should I Actually Do?

First, I'd talk to whoever owns the CDN, WAF or bot-management platform, find out if they have any kooky ideas about using them, and if yes, find out how they plan. on implementing it.

Then I'd want answers to a few practical questions:

None of this requires the SEO team to invent its own cryptography. There are plenty of exciting ways for us to break websites already.

Where I think this gets useful

I'd like fewer conversations where the SEO team asks security to change a rule because a crawler needs access, and security quite reasonably asks how we're going to distinguish that crawler from somebody pretending to be it., and Web Bot Auth gives us another way to answer that question. Good.

We'll still need to decide which bots we want, how fast they're allowed to crawl, and whether the resources they're consuming justify the value they're bringing back. We'll still need to check that the content they receive is actually the content we intended to serve, but being able to recognise the caller gives us a firmer starting point for those decisions.

However: I remain entirely capable of knowing who's knocking and deciding I'm not answering the door.