Should you block AI bots from your website?

If your website runs behind Cloudflare or another security service, you have probably seen a setting that offers to block AI bots. It looks like an easy call: AI companies are crawling websites, and if you do not want your content taken, block them. The complication is that AI bots do different jobs. Some collect content to train future models. Some index your site so it can appear in AI search answers. Some open a page because a person asked an assistant to look something up. Block all of them and you protect your content from one use while removing your business from another. For most business websites, that trade matters.

"AI bot" now means three different things

Cloudflare sorts AI traffic into three categories, Search, Agent and Training, and lets you set a policy for each. The categories are useful whether or not you use Cloudflare.

Search

Search crawlers index your content so an AI service can answer questions about it later. This is the category closest to ordinary search engine crawling. If someone asks an assistant for a business like yours, allowing search crawlers is what makes it possible for your site to be found and cited.

Training

Training crawlers collect content that may be used to train or improve a model. That is a different transaction. Letting a company train on your website does not, by itself, send you a customer. For publishers, photographers, writers and any business whose content is the product, this distinction carries real weight.

Agent

Agent traffic is an automated system acting for a person in real time: someone asks an assistant to open a page, find a detail, compare products or complete a task. Cloudflare describes agents as activity on a person's behalf, and blocking them can get in the way of customers who are deliberately using an assistant to deal with your business.

OpenAI's three bots show why one switch is too blunt

OpenAI runs three user agents with three purposes. OAI-SearchBot indexes pages for ChatGPT search, and OpenAI says sites that opt out will not be shown as sources in search answers, though a URL can still appear as a navigational link. GPTBot collects content that may be used to train OpenAI's models. ChatGPT-User fetches a page when a user asks ChatGPT to, and OpenAI says that is user-initiated rather than automatic crawling and plays no part in whether a site appears in search. You can allow OAI-SearchBot and disallow GPTBot; each setting is independent. Anthropic draws the same lines with ClaudeBot for training, Claude-SearchBot for search and Claude-User for pages a person asks Claude to open. "Do we allow ChatGPT?" is no longer a precise enough question to act on.

For most businesses, blocking AI search works against you

A contractor, an accountant, a medical practice, a manufacturer, a hotel: the website exists to be found. It needs to tell prospective customers what the business does, where it operates, what it offers, why to choose it and how to make contact. If AI assistants are becoming one more way people find businesses, shutting out their search crawlers works against the purpose of the site. Access does not mean an assistant will recommend you. It means you are at least in the running. OpenAI is explicit: block OAI-SearchBot and your site is not eligible to be surfaced as a source in ChatGPT search answers. Giving that up should be a deliberate decision with a reason behind it.

Training is a separate decision

The case for allowing training crawlers is weaker. Suppose a landscaping company's site holds service descriptions, project photos and a library of articles. Would letting that material train a future model bring in customers? Possibly, indirectly. There is no clear line from training access to traffic. Blocking training crawlers is a reasonable policy for many businesses, especially when the same company offers a separate search crawler. For businesses whose content is the product, the case is stronger still. A photographer can want images visible in search and not used to train image generators. A publisher can want articles findable and its archive left alone. Search visibility and model training do not have to be the same permission.

Agents need more thought

Agents sit in the middle. Someone asks an assistant to find a warranty page, compare three room types, check whether you serve a ZIP code or find the service department's phone number. Those interactions can bring you business, and assistants will increasingly handle appointments, information requests and similar tasks. Blocking all agents could eventually make your site hard to use for customers who reach the web through one. There are also legitimate security reasons to control agents when they can submit forms, sign in or complete transactions rather than read public pages. For a public marketing site, treating every user-directed request as hostile is more restriction than the risk calls for.

Cloudflare changed its own default

Cloudflare no longer treats AI crawlers as one category. In July 2026 it introduced separate Search, Agent and Training controls for every plan, including Free. Each can be allowed, blocked across the site or blocked only on pages that carry ads. For domains added from September 15, 2026, the default keeps Search allowed and restricts Training and Agent traffic on ad pages. The signal is that blocking every AI crawler is no longer the sensible default. Cloudflare also offers AI Crawl Control, which shows which AI crawlers are reaching your site and lets you allow or block each one. It is available on all Cloudflare plans and is far more useful than a single switch.

A robots.txt file is not a lock

Your robots.txt file tells crawlers what you permit, and legitimate search crawlers generally follow it. Cloudflare notes that compliance is voluntary: a crawler can ignore the file and request the page anyway. A firewall or crawler-control rule actually stops the request. Use robots.txt to state your policy for reputable crawlers, and real blocking for scrapers that ignore it.

A starting policy for a business website

There is no single right configuration, but for a business whose website exists to bring in customers, this is where we would start.

• AI search: allow. Blocking the crawlers behind services like ChatGPT search removes you from consideration.
• AI training: your call. You do not have to allow training to take part in AI search when the provider separates the two. If your content is valuable in itself, there is a good reason to say no.
• AI agents: allow public pages, look closely at anything interactive. Customer-directed lookups have value. Forms, accounts and transactions deserve more caution.
• Unknown or abusive scrapers: block. A bot claiming an AI purpose has not earned access. Crawlers that hide their identity, ignore your policy or hammer your server are a security matter, not a marketing opportunity.

Does allowing crawlers create a security risk?

A legitimate crawler reading the same public pages as any visitor gets no special access to private data. That is not a reason to trust every automated request. Sensitive information should never rely on robots.txt or crawler policies for protection. Authentication, permissions and server configuration decide what is private. Crawler management controls access to public content. It is not a substitute for security.

Look before you throw the big switch

AI Crawl Control shows which AI services are requesting your content, how often, and whether they respect your robots.txt, and it lets you write rules per crawler. See what is actually happening, keep the uses that bring value, restrict the ones that do not, and do not block useful discovery just because the traffic involves AI.

The question is not whether to "allow AI"

AI systems are becoming part of how people use the web, which makes "should I block AI bots?" the wrong question. The better one is: what do I want these systems to be allowed to do with my website? Index your public pages so customers can find you: probably yes. Open a page because a customer asked: usually yes. Train a commercial model on your whole content library: a separate decision. Scrape the site while ignoring your rules: no. The principle is old even if the technology is new. Give legitimate systems enough access to make your site useful and findable, and keep control over the uses that do not serve you.

The final article brings this together: what to actually do to make a website visible in AI search without chasing AI SEO tricks.

In this series

1. SEO and AI: what's real and what's hype
2. How does an AI assistant decide which businesses to recommend?
3. Should you block AI bots from your website? (this article)
4. How to make your website visible in AI search without chasing AI SEO tricks