AI crawlers have changed the way websites need to think about content access.
For decades, most companies have asked themselves one fundamental question: how do I make Google crawl and index my website? In 2026, this question has grown more complex. An internet site may be crawled by search bots, AI search bots, model training bots, user-initiated AI bots, data aggregators, browser bots, and scrapers.
Not all the bots have the same intended use, and such an assumption can be very costly.
One business could need Googlebot to crawl their entire set of important product pages, yet deny a model training bot access to any proprietary research they conduct. On the other hand, another enterprise may wish to have ChatGPT and Perplexity find its public posts because these artificial intelligence algorithms can send referral traffic but can’t scrape large amounts of low-value URLs. A SaaS firm would wish to have its public documents made clear to the AI but protect its customer-related information and proprietary products.
That is where AI crawler management becomes a real technical and strategic issue.
llms.txt is part of this discussion, but it is only one piece of the system. It should not be confused with robots.txt, a sitemap, an indexing directive, or a security control.
The more useful way to think about the modern AI web is this:
Robots.txt tells a crawler what it should access. Security controls determine what it can access. llms.txt helps an agent understand what is worth reading. Your content strategy decides what information the AI system should know about your company.
This article will cover how these pieces connect, when they don’t, and how businesses can create an AI crawler policy that doesn’t harm their SEO or AI visibility.
What Is The AI Crawl Budget?
An AI crawler budget is an amount of crawling, processing power, exposure, and focus that a particular website dedicates to AI automation.
One thing is certain from the very start. There is no such thing as a budget number for the AI crawler that applies to all websites, whether provided by companies like OpenAI, Google, or Anthropic.
The concept is best understood as an approach to planning.
Traditional search engines have formal concepts around crawl capacity and crawl demand. For instance, Google says that the crawl budget is defined by the URLs that Google will be able to and willing to crawl, where the key factors will be crawl capacity and crawl demand. At the same time, Google says that crawl-budget optimization applies only to the biggest websites.
AI crawlers create a related but different problem.
There can be multiple AI crawlers that visit a website at once, with their own goals. While one of the crawlers might gather information for training models, another one might look for pages to include in AI search, and yet another might simply fetch a web page due to a user request to the AI assistant.
Instead of asking:
“How much AI crawl budget do I have?”
A better question is:
“What automated systems will be allowed access to what parts of my website for what reason, and at what cost to operate?”
That question produces a much better strategy.
The AI crawler budget equation
For practical planning, think about AI crawler exposure as:
AI crawler budget = crawler demand + URL inventory + server cost + content value + business value
A large-scale crawler that visits 100,000 URLS of products is not the same thing as an AI search crawler that visits 500 highly-maintained editorial pages.
The number of requests made does not give a hint of the quality of action performed.
When AI crawler activity becomes a problem
AI crawling deserves closer attention when you see:
- High request volumes from automated agents
- Repeated crawling of unchanged pages
- Crawling of faceted or parameter-based URLs
- Access to proprietary content
- Server performance issues
- Increased bandwidth costs
- AI crawlers ignoring your stated preferences
- Scrapers pretending to be legitimate browsers
- Sensitive information appearing in AI-generated answers
- Content being reused without a clear commercial benefit
- Large amounts of crawling with little or no referral traffic
What Is llms.txt and What Does It Actually Do?
llms.txt is a proposed convention for placing a Markdown file at the root of a website, usually at:
https://example.com/llms.txt
The proposal was initially proposed by Jeremy Howard of Answer. AI in September 2024 and updated to version 2 in August 2026.
It is quite easy to understand.
Rather than having the AI agent scan the entire site using navigation menus, scripts, redundant boilerplate, categories, and many other URLs, the site gives the AI a brief description of what the site is all about, the context it operates in, and its best links.
It is a reading list for AI agents, rather than a security policy.
What does llms.txt solve?
Today’s websites are designed for people and browsers.
The human visitor knows how to use the navigation bar, sees the headings, understands that the content is related, disregards the ads, and understands what pages are important.
The AI agent needs to do all of those things programmatically.
This is the problem that llms.txt attempts to simplify.
For example, a software company could use it to identify:
- Product documentation
- API documentation
- Authentication guides
- Pricing information
- Integration documentation
- Security documentation
- Compliance information
- Product limitations
- Developer resources
Instead of leaving an agent to discover those pages independently, the business provides a curated map.
What llms.txt does not do
This is where many articles become misleading.
llms.txt does not:
- Block a crawler
- Prevent scraping
- Guarantee AI citations
- Stop model training
- Replace robots.txt
- Replace authentication
- Protect confidential information
- Create a legal license
- Force an AI engine to use specific pages
- Guarantee better rankings
- Guarantee visibility in ChatGPT, Gemini, Claude, or Perplexity
Publishing the file is therefore not equivalent to telling AI companies:
“You may access these pages and nothing else.”
It is closer to saying:
“If an agent is trying to understand this website, these are the pages we consider most useful.”
That distinction should guide your implementation.
llms.txt, robots.txt, sitemap.xml, and noindex
Even though these files share some similarities, as they all affect the behavior of machines on websites, their purposes are quite different.
| Tool | Primary purpose | Controls access? | Controls indexing? | Helps AI understand priority? |
|---|---|---|---|---|
| robots.txt | Crawler access instructions | Yes, where respected | Not reliably | Indirectly |
| sitemap.xml | URL discovery | No | No | Indirectly |
| noindex | Search/index exclusion | No | Yes | Sometimes |
| nosnippet | Search preview control | No | Partly | Can limit displayed content |
| Google-Extended | Google AI use controls | Through Google crawler policy | No | No |
| llms.txt | Curated AI-readable site guide | No | No | Yes |
| Authentication | Restrict access | Yes | Yes | No |
| WAF/server rules | Technical enforcement | Yes | Indirectly | No |
The strategic error lies in using one tool to address another tool’s problem.
If you wish to stop people from accessing your private information, then use access control.
If you wish to inform Google not to index that particular page, then use indexing controls.
If you wish to provide AI agents with an overview, then llms.txt would be helpful.
If you wish to minimize automated traffic, then you should control the crawlers.
The simplest mental model
Think of your website as a building.
- Sitemap: building directory
- llms.txt: visitor guide explaining which rooms matter
- robots.txt: sign telling certain visitors which rooms they should not enter
- Authentication: locked door
- WAF: security guard
- Server logs: security cameras
- Content strategy: deciding what should actually be inside the building
You need all of these concepts to manage a modern website properly.
The Importance of AI Crawler Management in 2026
There used to be four parts to search: the crawler would find your page, index it, and rank it, and a human would click on your page. With AI-based search, this sequence is broken. It would be able to collect your content, analyze it, answer the question and even point back to your site.
This is why it is not enough to think about bots equally.
Visibility is not the same as access
You can want to be cited by an AI search engine and still not want your content used to train a model. These are separate goals.
Take an accounting software company. It probably wants its public tax guide quoted in AI answers, since a citation can introduce the brand to a new buyer. It probably doesn’t want these things floating around:
- Internal product documentation
- Customer-specific examples
- Proprietary research
- Draft material
- Competitive intelligence
- Private implementation guides
So crawler policy should depend on what the content is worth and what the bot is doing with it. Fear of AI is a bad basis for it.
robots.txt cannot serve as a blanket for security.
There is a trap in the accounting example. Any form of documentation, including working drafts and guidelines, posted on a public link without login requirements will still be susceptible to attacks even with the use of robots.txt file. The purpose of this file is to instruct the search engine about which pages not to index.
Truly sensitive content belongs behind authentication. Robots.txt is for content that is public but where you have a preference about how it gets used.
AI crawlers are not one category.
Most big AI companies run more than one bot, and each bot does a different job. All these organizations, among others, do this to some extent.
These bots can be classified into three categories:
- Training crawlers gather data that can be used to train or develop models.
- Search crawlers build an index so a product can surface and cite your pages.
- User-triggered fetchers load a page because a person asked an assistant about it.
Blocking the first group has a very different business effect from blocking the second.
OpenAI: the clearest example
OpenAI’s documentation shows how split these roles are.
- GPTBot is tied to collecting content that may be used for model training.
- OAI-SearchBot is tied to search. OpenAI uses it to surface sites in ChatGPT’s search features, and recommends allowing it in robots.txt if you want to appear in those results.
- ChatGPT-User handles certain user-initiated actions in ChatGPT and custom GPTs. OpenAI says robots.txt rules may not apply to these, because a user starts the action.
A third-party analysis adds two details worth checking against OpenAI’s pages. It says OpenAI revised its documentation on 9 December 2025 and removed the robots.txt compliance language for ChatGPT-User, and that a fourth crawler, OAI-AdsBot, appeared in the documentation in late April 2026. I couldn’t confirm either on OpenAI’s own page, so verify before you publish them as fact.
That makes the draft’s “three bots” line shaky. Check OpenAI’s live crawler page before you finalize the list.
Anthropic follows a similar split.
- Three bots have been listed by Anthropic on its support website. These bots are ClaudeBot, Claude-User, and Claude-SearchBot.
- The role of Claude-User bot is to gather information from different pages whenever Claude receives queries related to them.
- Claude-SearchBot is used to improve search result quality, and disabling it can reduce your visibility and accuracy in search results.
- ClaudeBot is the training crawler, according to a third-party guide. Blocking Claude-SearchBot does not affect training.
Anthropic claims that its bots adhere to robots.txt guidelines, and it recommends not using IP blocking since it could prevent the company from even accessing your robots.txt file. That second point matters. A blanket IP block can break the very opt-out you’re trying to set.
Google, Apple, and the “extended” tokens
Some companies use a separate robots.txt token for AI use instead of a separate bot. Google-Extended and Applebot-Extended are the common ones.
Google-Extended covers certain Gemini uses. As far as I know, blocking it does not remove you from Google Search or change how AI Overviews draw on your pages, because those run through Googlebot. I’m working from general knowledge here, so check Google’s current crawler documentation before you rely on it.
The practical point is that blocking an AI token and blocking a search bot are different moves with different costs.
Not every bot plays by the rules
This is the part the original skips. A few patterns show up across the industry.
Training bots from major vendors generally honor robots.txt.
- User-triggered fetchers may not. The documentation of Perplexity itself has mentioned that the agent activated by users mostly does not pay heed to robots.txt as the request was made by a human user, as per one third-party guide.
- There are such crawlers which remain undisclosed or hidden. One reference guide has pointed out that Perplexity is known to use such undisclosed crawlers to bypass robots.txt.
- Some bots are listed as blockable but ignore the rule in practice. One roster flags ByteDance’s Bytespider this way.
Therefore, the robots.txt file is effective on those bots which respect it. As for the rest, there is a need for rate limiting, firewalls, and bot verification. The latter can be done through verifying requests against the ranges of IPs released by vendors for their bots. For instance, OpenAI releases separate range files for different bots, and user-agent strings can be faked.
What Blocking Actually Costs?
Many sites block first and think later. One June 2026 analysis of robots.txt files across the top 10,000 sites reported that 17.6% blocked GPTBot while only 6.6% blocked OAI-SearchBot. That gap suggests most publishers do see the difference between training and search. This figure has come from a secondary source and should be taken as indicative only.
The cost of blocking depends on which bot you block:
| Bot Type | Impact of Blocking | Typical Action |
|---|---|---|
| Training crawler | Little visibility loss, as it does not power citations directly. | Block if content is valuable or sensitive. |
| Search crawler | Loss of presence in the product’s answers and citations. | Usually allow. |
| User-triggered fetcher | Loss of reach when a user specifically asks about your page. | Usually allow (do not rely on robots.txt to stop it). |
Allowing search bots has its own risks. A citation doesn’t guarantee a click. If an AI answer fully satisfies the question, the visit may never happen. The accounting company is betting that brand exposure from citations beats the lost clicks. That bet needs tracking, not faith.
A practical way to decide
- Don’t set one policy for the whole site. Sort your content first.
- Public marketing and educational content. You want this found and cited. Allow search and user-triggered bots. Decide on training separately.
- Premium or original research. Training use may undercut your value. Consider blocking training crawlers and keeping search bots.
- Sensitive or internal content. Use a login. Do not rely on robots.txt.
- Content you can’t easily replace. Check server logs for heavy crawling and set rate limits.
Then ask the sharper question:
Which crawler, reaching which content, for which purpose, creates value or risk for us?
Keep the policy alive.
Crawler lists change often. Bot names are changed, new bots appear, and changes take place in the documentation.
- Check robots.txt file each quarter.
- Check the server logs to know which bots come to your site, not the bots mentioned in the guidelines.
- Bots can be verified by their IP addresses.
- Be patient; give enough time for updates. According to OpenAI, robots.txt updates reflect in search appearance within 24 hours.
- superblog
- Keep in mind that each subdomain requires its own robots.txt file.
Different Types of AI Crawlers
Managing crawlers becomes simpler once you differentiate between all kinds of requests made automatically. Some bots use your content for training. Some bots have a search engine behind them that will bring you traffic. Some bots operate under a single user. And some bots misrepresent their identities.
Here are five categories, what they do and what you lose by blocking them:
1. Training crawlers
Training bots gather publicly available web content that can potentially be used to develop an AI model. Common examples:
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- CCBot (Common Crawl)
- Bytespider (ByteDance)
- Google-Extended and Applebot-Extended, which are robots.txt tokens, are not separate crawlers.
What you get from allowing them: very little you can measure. Any advantage would be an indirect one. Your thinking may have some impact on the knowledge a model has of your field, but no referral will show in your analytics.
Your risk: your content being part of a service which answers questions without having to send someone to you.
People who should block: publishers of original research, those behind a paywall or content that is the product itself. A free tax guide meant to attract customers is a weaker case for blocking than a $5,000 industry report.
Two details people miss:
- Google-Extended is not what many assume. Google’s own documentation says it doesn’t affect inclusion in Search and isn’t a ranking signal. It controls use of content for training future Gemini models and for grounding in Gemini apps and the Vertex API. It does not switch off AI Overviews.
- Blocking one training bot doesn’t block the others. Your content can still reach models through other routes, such as Common Crawl, which many labs have drawn on. Blocking CCBot is a separate decision.
2. AI search crawlers
These bots build the index behind an AI product’s search feature. Common examples:
- OAI-SearchBot (ChatGPT search)
- Claude-SearchBot (Anthropic)
- PerplexityBot (Perplexity)
Why the value is more direct: an AI answer built from your page can cite you and link to you. That gives you a path to brand exposure and visits, which training crawlers don’t.
What are the blocking costs? OpenAI says sites that opt out of OAI-SearchBot won’t be shown in ChatGPT search answers. This means anthropocentric beliefs that disabling Claude-SearchBot can decrease your visibility and accuracy when appearing in user search results. In both these scenarios, the robot that gets blocked is the robot that decides whether you are going to appear or not.
However, a citation does not equal a click. If you solve the problem completely, you will never be clicked on. Nobody has settled how much traffic these citations send, and the answer varies by industry. So don’t allow these bots. Check your referral data for visits from ChatGPT, Perplexity, and similar sources, and judge the channel on what you see.
One more number gives context. A study done in June of 2026 regarding the top 10,000 sites indicated that GPTBot was blocked by 17.6%, whereas OAI-SearchBot was only blocked by 6.6%. This is a secondary source, but the discrepancy indicates that many organizations have been treating both in different ways.
3. User-triggered AI agents
These bots fetch a page because a person asked an assistant to. Common examples:
- ChatGPT-User (OpenAI)
- Claude-User (Anthropic)
- Perplexity-User (Perplexity)
How they differ: a training crawler works through your site on its own schedule. A user-triggered agent loads one page because a human wants to read it, or wants the assistant to read it for them.
Why a blanket block backfires: you’d be turning away a real person who is interested in your page. Anthropic says disabling Claude-User prevents Claude from retrieving your content in response to a user query, which may reduce your visibility for user-directed search.
The compliance problem: these agents don’t all follow robots.txt.
- OpenAI’s documentation says robots.txt rules may not apply to ChatGPT-User because a user starts the action.
- A third-party guide reports that OpenAI revised its documentation on 9 December 2025 to remove the robots.txt compliance language for that bot. I couldn’t confirm that on OpenAI’s own page, so check it before you cite it.
- The same guide says Perplexity’s documentation states that its user agent generally ignores robots.txt for the same reason.
- Anthropics takes a different position. It says its bots, Claude-User included, honor robots.txt.
So a Disallow rule for these agents is a coin flip, depending on the vendor. If you truly need to keep them out, use a firewall rule or a login, and accept that you’ll turn away real readers.
4. General-purpose scrapers
These bots don’t behave. Some collect data to sell, some feed unlicensed AI datasets, and some copy content for spam sites.
- How they hide: they often present themselves as an ordinary browser, such as Chrome on Windows. Some copy the name of a legitimate bot, including Googlebot.
- Why robots.txt fails here: it only works on bots that choose to read it. A scraper that wants your content won’t check.
Some named bots belong here too. One reference list flags Bytespider as a bot to block that often ignores the rule. And one guide warns that Perplexity has reportedly used undeclared crawlers to get around robots.txt, a claim you should treat as reported, not proven.
What does work:
- Verify the claimed identity. OpenAI publishes IP range files for each of its bots. Anthropic publishes a list of its source IP addresses. Check a request against the vendor’s list before you believe the user-agent name.
- Use rate limits. Thousands of requests a minute from one source is a bad sign, whatever it calls itself.
- Apply the web application firewall (WAF). Many hosting and CDN providers provide you with bot management, which grades your traffic by behavior.
- Do apply authentication to everything that shouldn’t be public.
However, please be cautious: Anthropic doesn’t recommend applying the IP blocking rule against its bots. Doing that will prohibit it from reading your robots.txt file and, therefore, implementing opt-outs. Apply IP to validate the identity of the visitors and robots.txt to opt out from compliant vendors.
5. Search Engine Crawlers
Googlebot and Bingbot are still the foundation of traditional search visibility. Blocking them because you want less AI crawling can cost you far more than the problem you’re solving.
The draft calls this a risk, which is right. It then leaves out why the line between “AI” and “search” is so blurry here.
Google is the hard case. AI Overviews and AI Mode draw on the regular search index through Googlebot. So you can’t block Google’s AI answers by blocking a separate AI bot, because there isn’t one. Your options, as of now:
- nosnippet, data-nosnippet, max-snippet, or noindex limit what Google shows from your pages. The catch is that using them to stay out of AI answers also removes the snippet from ordinary search results.
- Search Console’s generative AI setting. Google announced a setting that lets site owners pull their content out of AI Overviews, AI Mode, and Discover’s AI features without leaving regular Search. One report dates the rollout to 3 June 2026 and says it began with a subset of site owners, so not every account has it yet. A Google statement says sites that opt out won’t receive traffic or impressions from those features, and that the setting isn’t a ranking signal for results outside them.
- Possible side effects. One analyst quoted in coverage believes the opt-out may also remove publishers from Top Stories inside AI Overviews. That’s an expert’s reading, not confirmed, so test before you assume.
- Google-Extended does not do this job, as covered above.
An Improved Perspective on AI Crawler Budget
Rather than having one global policy, split up the contents of your website into four categories.
Class 1: Public growth content
This includes content designed to attract customers.
Examples include:
- Product pages
- Service pages
- Educational articles
- Buying guides
- Public documentation
- Case studies
- Research summaries
This content usually deserves the highest level of discoverability.
Class 2: Controlled commercial content
This content is public but commercially sensitive.
Examples include:
- Detailed pricing
- Competitive comparisons
- Advanced implementation guides
- Proprietary calculators
- High-value reports
- Specialized research
This category requires a business decision rather than an automatic block.
Class 3: Restricted content
Examples include:
- Customer portals
- Internal documentation
- Private dashboards
- Account information
- Draft content
- Employee resources
This content should not depend on robots.txt for protection.
It should be protected technically.
Class 4: Low-value crawl inventory
This includes:
- Filter combinations
- Internal search results
- Session URLs
- Duplicate parameters
- Temporary pages
- Tracking URLs
- Empty category pages
AI Crawlers to Permit or Block?
It is impossible to have a universal allow list or block list for all businesses.
A better approach is to score each crawler according to four factors:
| Factor | Question |
|---|---|
| Referral value | Does this crawler lead to measurable discovery or traffic? |
| Content-use value | Does access provide a business benefit? |
| IP risk | Does the crawler access valuable proprietary information? |
| Infrastructure cost | Does the activity create meaningful server or bandwidth cost? |
Then assign a policy:
Allow → Monitor → Restrict → Block
Example
Suppose an enterprise software company discovers:
- AI search crawler: moderate traffic, high citation value
- Model-training crawler: high crawl volume, unclear direct benefit
- Unknown scraper: high volume, no referrals
- Search crawler: high business value
A reasonable policy might be:
| Crawler type | Decision |
|---|---|
| Search crawler | Allow |
| AI search crawler | Allow and monitor |
| Model-training crawler | Restrict or evaluate |
| Unknown scraper | Investigate and potentially block |
This is far better than implementing a blanket AI block.
How to Control OpenAI Crawlers?
OpenAI’s current crawler documentation distinguishes among several user agents.
The most important distinction for businesses is between GPTBot and OAI-SearchBot.
GPTBot is associated with crawling content that may be used for training OpenAI’s generative AI foundation models.
OAI-SearchBot is used for search functionality in ChatGPT.
These controls are independent.
That creates an important strategic option: a company can allow OAI-SearchBot while disallowing GPTBot.
Example policy
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Such a setup is indicative of a company’s desire:
“We need to stay visible on ChatGPT search, but we do not want our content to be used for training via GPTBot.”
That is a much more targeted approach than completely restricting access from OpenAI.
But how about ChatGPT-User?
ChatGPT-User stands for user-initiated access rather than automated crawling.
That is important since restricting automated crawlers does not always imply that one needs to restrict user-initiated access.
Before setting up a generic restriction, learn which OpenAI user agent you see in your server logs.
How Google-Extended Works?
Google requires especially careful handling because Google Search and Google’s AI systems are not controlled through one universal switch.
Googlebot is the primary crawler used for Google Search.
Google-Extended is a separate robots.txt product token that publishers can use to manage how content Google crawls may be used for certain Gemini and Vertex AI purposes.
This produces an important distinction:
- Google-Extended is not a replacement for Googlebot controls.
Google states that Google-Extended does not affect inclusion in Google Search and is not a Google Search ranking signal.
Why does this matter?
Suppose a company wants:
- Google Search visibility
- Google AI Overviews visibility
- Less use of its content for certain Gemini model-development purposes
Blocking Googlebot would be too extreme and perhaps harmful.
The right way would be to know which Googlebot or product token dictates the behavior that needs to be changed.
AI Overviews should not be confused with model training.
AI search visibility and model training are separate questions.
A business can want its content to be found and cited in Google’s search ecosystem while still evaluating whether its content should contribute to future model development.
That distinction should be part of every enterprise crawler policy.
How to Control Perplexity, Microsoft, Meta, and Other Crawlers?
The AI crawler landscape extends well beyond the largest model providers.
Depending on the website, you may see crawlers associated with:
- Perplexity
- Microsoft
- Meta
- Amazon
- Apple
- ByteDance
- Common Crawl
- DuckDuckGo
- Mistral
- Other AI and data services
Key operational rule:
It is very easy:
Never formulate the crawler policy based on some randomly generated list posted on a blog.
The names, functions, and infrastructure of crawlers may change.
Instead:
- Review your server logs.
- Identify the user agent.
- Verify the operator where possible.
- Check the operator’s official crawler documentation.
- Determine the business purpose of the crawler.
- Decide whether access creates value or risk.
- Apply the narrowest practical rule.
- Monitor the result.
This process is more durable than maintaining a static “AI bot blacklist.”
What Should Go Inside llms.txt?
Include
- Core product pages
- Service descriptions
- High-quality documentation
- Authoritative research
- Pricing information where appropriate
- Security documentation
- Compliance information
- Important policies
- Original studies
- Strong educational resources
- Frequently referenced product information
Pages that contain original research, proprietary technology, or industry-specific data created by your business need to be especially emphasized.
This is where llms.txt can support information architecture.
Rather than simply listing:
Blog
you can point agents toward the pages that contain your strongest first-party knowledge.
Use descriptions strategically
Compare:
Blog
with:
AI Customer Service Guide: Guide on AI customer service support automation, considerations for its use, limitations, and potential risks for customers.
The second description provides much more information to an agent.
How to Protect Your Intellectual Property Against AI Scrapers?
First things first, figure out what it is you’re trying to protect.
Most companies talk about wanting to “protect their content” but they actually want to protect four different things:
- Protect confidential information
- Stop the unauthorized reuse of content
- Decrease infrastructure costs
- Protect the value of proprietary intellectual property
Each problem has a different solution.
Protect confidential information
Ignore all crawler directives.
Confidential information must be protected by authentication.
Reduce undesirable automated crawling
Make use of robots.txt whenever possible, and ensure that enforcement of your policy is done through your CDN, WAF, or your server stack.
For example, CloudFlare already has an AI Crawl Control solution which enables you to monitor bot activity on your site and control bot access.
Reduce exposure of your IP
Identify your highest-value public assets.
These may include:
- Original research
- Proprietary datasets
- Industry benchmarks
- Detailed frameworks
- Paid reports
- Premium tutorials
- Unique product documentation
Then determine whether AI visibility is more valuable than restricting automated access.
Do not assume blocking equals protection
A determined scraper can ignore robots.txt.
This is why IP protection should include:
- Terms of use
- Copyright notices
- Access controls
- Rate limiting
- Bot detection
- WAF rules
- Monitoring
- Legal review where necessary
Technical controls and legal rights solve different problems.
AI Crawler Decision Framework
Utilize this approach before blocking any crawler.
Question 1: What is it owned by?
Does there appear to be a specific firm or corporation running this site?
If not, research to see if it’s legitimate.
Question 2: What does it do?
Is it used for:
- Search?
- Training?
- User-triggered retrieval?
- Data collection?
- Preview generation?
- Commercial intelligence?
Question 3: What does it access?
Is it accessing:
- Public marketing pages?
- Product pages?
- Research?
- Documentation?
- Private information?
- Low-value URLs?
Question 4: What is the business value?
Does access result in:
- Citations?
- Referral traffic?
- Product discovery?
- Customer acquisition?
- Better AI visibility?
Question 5: What does access cost?
Consider:
- Bandwidth
- Origin requests
- Database queries
- Rendering resources
- CDN costs
- Security overhead
A More Useful Model: The AI Content Access Funnel
A practical way to think about your website is as an access funnel.
Level 1: Discover
Can an AI system find the business?
Use:
- Search visibility
- Sitemaps
- Crawlable links
- Public content
Level 2: Understand
Can the AI system comprehend the real function of the business?
Use:
- Clear information architecture
- Strong entity references
- Consistent terminology
- Structured content
- llms.txt where appropriate
Level 3: Verify
Can the AI system find authoritative evidence?
Use:
- Original research
- Documentation
- First-party data
- Author credentials
- Case studies
- Transparent sourcing
Level 4: Cite
Can an AI system identify a specific page as a useful source?
Use:
- Clear answers
- Specific claims
- Strong topical relevance
Level 5: Convert
Can the user move from an AI answer to your business?
Use:
- Clear calls to action
- Strong landing pages
- Product information
- Trust signals
- Easy navigation
llms.txt can support the understanding stage.
It does not replace the other four.
Frequently Asked Questions
Is the llms.txt a recognized SEO standard?
Absolutely not. The llms.txt is more of a recommendation than a universal web standard.
Its objective is to serve as a summary of vital information on a website for the benefit of AI bots.
A business can upload a file, although they shouldn’t expect all major AI systems to read it or use it to influence their ranking.
Does llms.txt improve Google rankings?
There is no established basis for treating llms.txt as a Google ranking factor.
Google’s documented controls for Search remain focused on established crawling, indexing, serving, and search systems.
Does llms.txt stop AI scraping?
No.
“llms.txt” is not an access control system.
For controlling access by crawlers, employ suitable directives in robots.txt and technical controls if stricter control is needed.
Should every website have an llms.txt file?
Absolutely not.
For an informational brochure website, which has few pages, the usefulness might be minimal.
If it is a SaaS company or any other that has lots of documents or information and has to provide it all, then this map might be very useful to have.
It is a relatively inexpensive process, although it must only be done if one can keep it accurate.
Should I block GPTBot?
That will depend upon your business model.
However, if you do not wish to have content considered for training the OpenAI models, then blocking GPTBot is a good idea.
However, that decision should not automatically lead to blocking OAI-SearchBot, because search and model-training access serve different purposes.
Should I block AI crawlers from my entire website?
Usually, that is too broad a starting point.
First identify:
- Which crawlers are accessing your site
- Why they are accessing it
- Which pages they are requesting
- What value they create
- What risk they create
Then create a narrower policy.
Can robots.txt protect proprietary content?
Not consistently.
In cases where secrecy has to be ensured, one should ensure that information is not made public and should depend on robots.txt only.
Proper access control and authentication mechanisms have to be used.
Does blocking AI crawlers hurt SEO?
It can, depending on which crawler you block.
Blocking the training crawler is a different process from blocking other crawlers like Googlebot or Bingbot.
First, always know what the crawlers are doing before you create the rule.
Am I allowed to enable search by AI and disable the training of the model?
Sometimes yes.
This is the most helpful change in the field of crawlers because several large AI vendors make distinctions between the two crawlers.
The exact user-agent names and policies should be verified against the provider’s current documentation before implementation.




