402 High and Rising
I keep thinking about the (potential) rise of the 402 (payment required) status.
I've seen a few posts on here saying things like "ah yes, but are you offering anything worth paying for?", which I think kind of speaks to a lack of understanding with regards to just how hard LLM user agents hit some websites.
Stupid analogy (my favourite kind): imagine your website is a house, and imagine instead of a user requesting a webpage, they are knocking on the door of your house and asking you to show them something in your house.
Your average user might ask to see, I dunno, 3, 4 things, then they're gone, satisfied that they have learned what they wanted. Maybe a recipe, or the manual for your toaster. You didn't charge them, but if you did, they'd probably be happy to pay, I dunno, 50p? it if they REALLY wanted to see a particular thing.
An LLM can spend hours knocking over and over and over again, asking to see your spoons and your coats and your dog's vaccination certificate and some stuff that you don't have but it just feels like you might have because the last house it did this to had them and yours looks a bit like that one - eventually you'd get sick of this ridiculous caper and say "No, do you know what, every time you knock I have to pause The Bill, my brew goes cold, and frankly the bottoms of my slippers are getting too much extra mileage put on them - if you want anything else, you're going to have to pay".
And you'd be right to do so - and that's where we're currently at. At the moment, LLMs absolutely batter websites (I'm sure I'm not the only person who's noticed an increase in Cloudflare humanity challenges in recent months) because there's no jeopardy in doing so - especially if you're a good little SEO and you're rendering the site on the server and thus taking on all the cost of actually creating a webpage to send - and so, by asking for a payment (even if it's tiny) what you're actually doing is getting the LLMs to put a value on your content, not the other way round. Up until now it's been the last night of a stag do in an all-inclusive bar in Magaluf, and while I'm not here to pretend my own personal blog is Claridge's, it's at least a provincial Wetherspoons that's architecturally interesting because it used to be a bank.
The 402 isn't there to make site owners rich, nor is it based on a level of arrogance from content creators that believe LLMs simply cannot survive without this one particular blog post - instead it's a shield, a request to the LLMs to remember that they aren't owed access to every single resource on the web; that behind some incredible resources are volunteer teams nursing aging CMSes into a new world of online interactions; that the internet was built to allow humans to communicate with other humans.
So, if you want to look at the contents of my spice rack four thousand times in a minute, it's going to cost you 0.001p each time you do it, and I think that's fair enough.