Should You Block AI Bots From Your Website?

At the root of nearly every website sits a plain text file most owners have never opened. In 2026 that file quietly decides whether the machines answering your buyers' questions are allowed to read your proof. So, should you block AI bots? It depends on a prior question almost nobody asks first: which bot, and what does it feed?

What does robots.txt actually control?

Robots.txt is a convention, not a lock. It has existed since 1994, long before anyone imagined a chatbot summarising the web, and it works on an honour system. A crawler arrives at your site, requests the file, and reads a short list of rules: this user agent may fetch these paths, that one may not. Everything that happens next depends on the crawler choosing to obey.

The major AI companies say theirs do. OpenAI documents its crawlers and the exact lines that turn each one away. Anthropic does the same for ClaudeBot. Google publishes tokens that separate its search crawler from its AI training pipeline. So the lever is real, and for the named, reputable bots it genuinely works.

It is equally worth being clear about what robots.txt is not. It is not retroactive: a model that read your pages during last year's training run keeps whatever it learned. It is not selective about facts: you cannot block the paragraph you regret and allow the rest. And it is not security: a rule in a text file stops polite software, not determined scraping. It is a posted sign, honoured by the crawlers that matter most and ignored by the ones that never mattered anyway.

Which AI bots are actually reading your site?

The phrase "AI bots" hides a crucial distinction, because the big AI companies run several crawlers with entirely different jobs. OpenAI alone operates three. Deciding whether to block "AI" without knowing which door each bot feeds is like locking a random third of your building.

Notice how different the stakes are per row. Blocking GPTBot is a slow, philosophical decision about future models. Blocking OAI-SearchBot or ChatGPT-User is an immediate, practical decision about whether the world's most used assistant can show or read your pages today.

Why do serious publishers block AI crawlers?

The case for blocking deserves a fair hearing, because it is not paranoia. If your content is the product, an AI answer that summarises your article is a substitute for the visit you sell advertising against or the subscription you charge for. News organisations, reference publishers and owners of original datasets watched language models train on decades of their work without payment, and reached a rational conclusion: withholding the archive is negotiating leverage. Several have since converted blocked crawlers into licensing deals, which rather proves the point.

There are quieter reasons too. Some site owners object on principle to their writing shaping commercial models. Some run infrastructure where aggressive crawling genuinely strains servers. Some hold client work or unpublished research that was never meant for any index, machine or human. All of these are legitimate, and robots.txt, with path-level rules, handles them well.

The trouble starts when a freelance consultant reads a headline about a newspaper's crawler policy and copies it. The newspaper is protecting inventory. The consultant is deleting their own evidence, and the two situations only look similar from a distance.

What do you lose when you block?

Think of the three OpenAI crawlers as three doors into an AI answer. Training is the slow door: what a model absorbs about you becomes part of its long memory, the thing it says when nobody searches, a mechanism we unpack in what a knowledge cutoff means for your name. Search indexing is the fast door: it decides whether your pages surface as cited, linked sources. Live browsing is the immediate door: when a buyer asks "tell me more about this person", the assistant tries to open your site right then. Block all three and you have not made yourself private; you have made yourself hearsay, described only through whatever third parties say about you.

The traffic numbers make the trade concrete. AI referrals are still a small stream, around 1.08% of website traffic across ten industries, but the visitors they deliver convert at extraordinary rates: 14.2 to 15.9% from ChatGPT and up to 16.8% from Claude, against roughly 1.76% for Google organic, according to Conductor benchmarks collected by SEO Sherpa's AI search statistics. And since ChatGPT drives about 87.4% of all AI referral traffic, blocking OpenAI's search and browsing bots specifically means declining the channel with the highest-intent visitors you are likely to receive this year.

There is also the question blocking cannot answer: if the engines cannot read you, whom do they read instead? We looked at where AI looks before recommending someone, and the uncomfortable pattern is that an engine denied your site does not go silent about you. It simply reconstructs you from weaker sources.

Should a personal website ever block AI bots?

Sometimes, yes. The honest way to decide is to answer five questions in order, and let the answers overrule your mood about AI in general.

1. Does new business find you through reputation? If clients arrive by referral, search or recommendation, every AI answer is a referral channel. Leave search and browsing bots alone.
2. Is the content itself what you sell? If your site hosts paid courses, gated research or original data, blocking training bots on those paths is defensible. Keep your about, bio and proof pages open regardless.
3. Are you worried about being misquoted? Blocking makes this worse, not better. An engine that cannot read your canonical pages leans harder on third-party sources you control even less.
4. Do you hold licensing leverage? A publisher with a famous archive can trade access for money. An individual expert almost never can, so the leverage argument does not transfer.
5. Would you notice if ChatGPT stopped citing you? Ask the major assistants about your specialty before changing anything, note where you appear, then decide. Never block blind. Our piece on whether ChatGPT recommends you shows how to run that check properly.

How do you write the rules once you have decided?

Robots.txt syntax is mercifully simple: a user-agent line names the bot, and a disallow line names the paths it may not fetch. In practice, personal sites end up in one of three postures.

• Open. Do nothing, or explicitly allow everything. The default, and for most experts the right call. Every door stays open, every crawler reads your best case for yourself.
• Training-shy. Disallow GPTBot, ClaudeBot and Google-Extended sitewide, but leave OAI-SearchBot, ChatGPT-User and PerplexityBot untouched. Your work stays out of future training runs while remaining fully visible in AI search and live browsing. A principled middle position, at the cost of the slow familiarity that training builds.
• Closed. Disallow every documented AI user agent. Rational for pure publishers, and close to self-erasure for anyone whose name is the business.

Remember the rules take paths, not just sites. Blocking /clients/ or /drafts/ while leaving the rest open is often the precise instrument people actually wanted when they reached for the sledgehammer. And note that Google-Extended affects Gemini training only; your ordinary Google rankings do not move either way, as Search Engine Land's AI crawler guide lays out.

What can blocking never do for you?

Three things, and they are the three people most often want. It cannot make a model unlearn what it already trained on; that knowledge persists until model retirement, whatever your robots.txt says today. It cannot touch what the rest of the web says about you; the engines will keep reading directories, press, reviews and old bios, which is why fixing errors means correcting sources, a process we walk through in can you make AI forget you. And it cannot bind the impolite: a crawler that never announces itself never reads your rules.

So if your real worry is wrong information rather than principled objection, the wall is the wrong tool. Corrections travel through open doors.

A quick way to picture the whole decision

Think of your website as a shop with three separate entrances, one for a researcher taking notes for a future encyclopedia, one for a librarian building today's card catalogue, and one for a customer walking in right now to check a specific fact before buying. Blocking GPTBot closes the researcher's door, a slow, philosophical choice about future editions. Blocking OAI-SearchBot closes the librarian's door, so your shop stops appearing in the catalogue people browse. Blocking ChatGPT-User closes the door on the customer mid-decision, the one most likely to have been about to trust you. Most personal brands, on reflection, only ever meant to worry about the first door, and end up locking all three because "AI bots" sounded like one thing instead of three very different visitors.

So, block or not?

If you publish content that is itself the product, block the training bots, negotiate if you can, and keep your commercial pages discoverable. If you are a consultant, adviser, founder or expert whose name is the asset, the arithmetic points the other way: the referral stream is small but converts like nothing else you run, the reputational risk of being described only second-hand is real, and the thing you would be protecting is precisely the thing you need machines to read. Keep the doors open and make what is behind them worth citing. If you would like a second pair of eyes on your own crawler posture and what the engines currently say about you, that is part of what our services cover, and the rest of the journal digs into the neighbouring decisions.