A comprehensive analysis of 66.7 billion bot requests across more than 5 million websites has revealed a fundamental transformation in how artificial intelligence systems access and index web content. According to data published by Hostinger in January 2026, OpenAI’s OAI-SearchBot has achieved 55.67% average coverage across monitored websites, marking a substantial expansion in AI-driven web discovery.
The study, which examined anonymized server logs from three separate six-day periods between June and November 2025, documents diverging patterns between different categories of AI crawlers. While training bots face increasing resistance from content publishers, assistant-facing crawlers that power AI search tools are rapidly expanding their reach across the internet.
Understanding the Dual Nature of AI Crawlers
The Hostinger analysis categorizes AI-related bots into two distinct functional groups with notably different reception from website operators. Training bots collect data to improve large language models, while assistant bots retrieve content in real-time to answer specific user queries. This functional difference has created a clear division in how sites manage crawler access.
OpenAI operates multiple crawlers with distinct purposes. GPTBot, which collects training data for model development, experienced a precipitous decline from 84% coverage to just 12% over the study period. In contrast, OAI-SearchBot, which fetches content for ChatGPT’s search feature, maintained 55.67% average coverage with 279 million requests.
This divergence reflects a strategic choice by website operators. Training bots extract content without driving traffic back to source sites, creating a value asymmetry that many publishers find unacceptable. Assistant bots, however, surface content in direct response to user queries, potentially driving discovery and engagement similar to traditional search engines.
Meta’s ExternalAgent emerged as the largest training-category crawler by request volume in Hostinger’s dataset. The training-bot category overall showed the strongest declines, driven by explicit blocking through robots.txt files and server-level configurations. According to BuzzStream research cited in the analysis, 79% of top news publishers now block at least one training bot.
Traditional Search Crawlers Maintain Stability
While AI crawlers undergo rapid transformation, traditional search engine bots demonstrated remarkable stability throughout the study period. Googlebot maintained 72% average coverage with 14.7 billion requests across the three measurement windows. This consistency reflects Google’s unique position in the web ecosystem, where blocking the primary search crawler directly affects visibility in search results.
Bingbot held 57.67% coverage with 4.6 billion requests, representing the second-largest traditional search presence. Other established search crawlers including Yandex (19.33% coverage), DuckDuckGo (9% coverage), and Baidu (5.67% coverage) maintained smaller but consistent footprints.
The stability of traditional search crawlers contrasts sharply with changes in the AI category. Google’s crawler faces fundamentally different incentives than emerging AI bots. Decades of search engine optimization practice have established an implicit agreement: sites provide access to crawlers in exchange for potential traffic from search results. This established relationship provides traditional search bots with more predictable and stable access patterns.
Baidu’s data showed a sharp spike in November, potentially representing either expanded global indexing efforts or a temporary crawl burst. The pattern will require additional monitoring to determine whether this represents a sustained change in strategy or a one-time event.
The Resource Cost Problem
Server resource consumption emerged as a significant factor influencing blocking decisions. Vercel data from December 2024 showed OpenAI’s GPTBot generating 569 million requests across their network in a single month. For publishers operating on usage-based infrastructure pricing, this level of automated traffic created measurable business costs without corresponding revenue.
AI crawlers can significantly impact website bandwidth and server load. Unlike human visitors who browse selectively, automated crawlers systematically request multiple pages in rapid succession. For sites with limited server capacity or bandwidth caps, high-volume crawler activity can degrade performance for human visitors or trigger overage charges.
Cloudflare data from their 2025 Year in Review indicated that AI crawler traffic can account for substantial portions of total bot activity. From May 2024 to May 2025, overall crawler traffic rose 18%, with GPTBot growing 305% and Googlebot increasing 96%. This rapid growth in automated requests forced website operators to make explicit decisions about which crawlers to allow.
The bandwidth concern extends beyond pure volume. AI crawlers often request complete page content including images, scripts, and other assets required to render pages fully. JavaScript-heavy sites face particularly high resource costs as crawlers execute client-side code to access dynamically loaded content. This technical reality makes crawler management a practical infrastructure consideration rather than purely a strategic decision.
AI Assistant Bots Expand Discovery Role
Group 5 crawlers in Hostinger’s classification—those powering AI assistants and search tools—showed the strongest growth signals. These bots fetch content on demand to answer specific user queries rather than building comprehensive training datasets. The category collectively generated 4.6 billion requests (6.9% of total traffic) while achieving substantial coverage rates.
Beyond OpenAI’s 55.67% coverage, TikTok’s bot reached 25.67% coverage with 1.4 billion requests. Apple’s Applebot achieved 24.33% coverage with 1.3 billion requests, reflecting Siri’s integration of web search capabilities. PetalSearch, Huawei’s search engine, reached 18.33% coverage with 675 million requests.
These assistant crawls differ fundamentally from training operations in their triggering mechanism and scope. Rather than systematically scanning the entire web, assistant bots target specific content in response to user queries. This user-triggered behavior means each request potentially represents genuine interest in the content, making these crawls more similar to traditional search referrals.
Amazon’s bot achieved 4.67% coverage with 581 million requests, likely supporting Alexa’s knowledge base and product recommendation systems. Google’s Read Aloud feature generated 4.33% coverage with 225 million requests. The diversity of assistant-facing crawlers indicates broad industry movement toward AI-enhanced search and discovery experiences.
ChatGPT-User, which handles user-initiated browsing within ChatGPT, reached 9.33% coverage with 137 million requests. OpenAI’s documentation clarifies that ChatGPT-User and OAI-SearchBot serve different functions. OAI-SearchBot specifically controls inclusion in ChatGPT search results and respects robots.txt directives, while ChatGPT-User may have different governance mechanisms for user-initiated actions.
SEO and Marketing Crawler Decline
The SEO and marketing crawler category experienced declining coverage despite maintaining high request volumes. Ahrefs maintained the largest footprint at 60% average coverage with 3.1 billion requests, but other tools in this category showed shrinking reach. Majestic achieved 27.7% coverage, Semrush reached 25% coverage, and various monitoring and audit tools maintained single-digit coverage rates.
Two factors drive this decline. First, SEO tools increasingly focus their crawling efforts on sites actively engaged in search optimization work. Rather than attempting comprehensive web coverage, these services concentrate resources on customers and potential customers. This business logic creates natural boundaries around crawling scope.
Second, website owners have become more aggressive about blocking resource-intensive crawlers that don’t directly support their business objectives. SEO tool crawlers provide value primarily to the analyzing party rather than the site being crawled. Unlike search engine crawlers that can drive traffic, or assistant bots that surface content to users, SEO crawlers extract competitive intelligence without offering direct reciprocal value to the target site.
The category generated 6.4 billion total requests (9.7% of total traffic), indicating these services remain highly active despite reduced coverage. The pattern suggests SEO tools are crawling their target sites more intensively rather than attempting broader but shallower web coverage.
LLM Training Bots Face Growing Resistance
The most dramatic shifts appeared in Group 3—LLM training and data collection crawlers. This category generated 10.1 billion requests (15.1% of total traffic) but experienced steep coverage declines as publishers implemented blocks.
Meta’s ExternalAgent maintained relatively high 57.33% coverage despite the category’s overall struggles, likely due to its recency and the complexity of blocking Facebook-related bots without affecting social media sharing functionality. However, the bot faces growing scrutiny as publishers become more sophisticated about distinguishing training crawlers from social preview bots.
OpenAI’s GPTBot provided the clearest signal of publisher resistance. The dramatic drop from 84% coverage in the early measurement period to 12% in the final window represents active decisions by website operators to prevent training data collection. This blocking occurred despite GPTBot maintaining 1.7 billion total requests across the study period, indicating the crawler continued attempting access but found itself denied entry at most sites.
Anthropic’s ClaudeBot achieved only 9.33% coverage with 1.4 billion requests. Perplexity’s bot, despite controversy over crawling practices, showed minimal 1.67% coverage with 13 million requests. CommonCrawl, the open web archive used by numerous AI companies, reached just 1% coverage with 30 million requests.
Google’s catch-all “google-other” category, which includes various research and AI development crawlers beyond the primary Googlebot, achieved 9.67% coverage with 2.9 billion requests. This represents an exception to the declining pattern, potentially because publishers remain cautious about blocking anything associated with Google’s primary search functions.
The training bot decline represents a significant shift in web ecosystem dynamics. For decades, the internet operated under an implicit open access model where automated crawling was generally accepted as necessary infrastructure for search and discovery. AI training bots disrupted this equilibrium by collecting data for commercial model development without providing direct value back to content creators.
Strategic Implications for Website Operators
The Hostinger data reveals website operators implementing a nuanced approach to crawler management rather than blanket policies. The middle path—allowing assistant bots while blocking training bots—appears to be emerging as the standard practice.
This strategy reflects a value calculation. Assistant bots that power search features in ChatGPT, Perplexity, and other AI tools represent potential discovery channels for content. Users asking questions through these interfaces may encounter site content in AI-generated responses, potentially driving traffic or building brand awareness. The relationship mirrors traditional search engine dynamics where crawler access enables visibility in results.
Training bots present a different proposition. They extract content to build datasets that improve AI models, but don’t create direct pathways for users to reach source websites. The asymmetry becomes particularly stark when AI systems generate comprehensive answers that satisfy user queries without requiring click-throughs to source content.
Publishers face additional concerns around licensing and attribution. When training bots incorporate content into model weights, the original source becomes embedded in the AI’s knowledge base without clear attribution or compensation mechanisms. Some publishers view this as unauthorized commercial use of their intellectual property.
For content sites and publishers, visibility in AI assistant responses may increasingly compete with traditional search traffic as a discovery channel. OpenAI recommends allowing OAI-SearchBot for sites that want to appear in ChatGPT search results, even if they block GPTBot to prevent training data collection.
Sites with proprietary content or APIs may implement more restrictive policies, blocking all AI crawlers to prevent commercial use of their data without explicit licensing agreements. This approach prioritizes protecting intellectual property over potential discovery benefits.
High-traffic sites concerned primarily about server resources can use selective blocking to manage infrastructure costs. Content Delivery Networks (CDNs) and web application firewalls provide granular controls for managing crawler access based on rate limits, resource consumption, or specific user agents.
Technical Implementation Considerations
Website operators implement crawler access controls through multiple technical mechanisms, each with distinct characteristics and trade-offs. The robots.txt protocol provides the most widely recognized method for communicating access policies to automated systems.
A robots.txt file placed in a website’s root directory specifies which crawlers can access which content. The file uses a simple syntax to define user agents (crawler identifications) and the paths they may or may not access. Legitimate crawlers that respect the protocol check this file before crawling and honor the specified directives.
To block specific AI crawlers while allowing others, robots.txt entries target individual user agents:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
This configuration would block GPTBot and CommonCrawl’s CCBot from accessing site content while explicitly allowing OAI-SearchBot. The distinction enables sites to prevent training data collection while maintaining visibility in ChatGPT search results.
OpenAI’s documentation clarifies that OAI-SearchBot respects robots.txt directives and controls inclusion in ChatGPT search results. Sites wanting to appear in AI search answers should ensure this crawler has access even if they block training-focused bots.
However, robots.txt operates on an honor system. The protocol has no enforcement mechanism, and crawlers can choose to ignore directives. Some AI companies have faced criticism for circumventing robots.txt restrictions or using crawlers with undisclosed user agent strings that bypass blocking rules.
Server-level blocking provides more robust enforcement. Web servers and CDNs can identify crawler user agents and refuse requests before serving any content. This approach prevents resource consumption by blocked crawlers rather than merely requesting they respect access limitations.
Rate limiting offers a middle ground for managing crawler impact without complete blocking. Servers can restrict how many requests a particular user agent makes within a time window, preventing resource exhaustion while still allowing some level of access. This approach works well for managing server load without making binary allow/deny decisions.
More sophisticated implementations use behavioral analysis to identify crawlers that misrepresent their user agents or attempt to circumvent blocks. Web application firewalls can detect automated access patterns and apply restrictions regardless of the declared user agent string.
The Competitive Landscape of AI Search
The Hostinger data documents a transitional moment in web discovery. AI-powered search tools are expanding from experimental features to mainstream user behaviors. ChatGPT’s search feature, Perplexity’s answer engine, and similar services represent a new category of discovery that operates alongside traditional search rather than simply replacing it.
For users, AI search provides a fundamentally different experience. Instead of receiving a ranked list of links, they get direct answers synthesized from multiple sources. The system attempts to understand query intent and construct comprehensive responses rather than matching keywords to documents.
This shifts the value exchange that underlies web discovery. Traditional search directs users to websites, where publishers can engage visitors, build audiences, and monetize attention through advertising or conversions. AI search may provide complete answers without requiring users to visit source sites, fundamentally disrupting the traffic-for-access agreement.
Citation practices vary widely across AI search implementations. Some systems prominently link to sources used in generating answers, creating potential pathways for user traffic. Others provide minimal attribution or bury source links in ways that discourage click-throughs. This variation creates uncertainty for publishers evaluating whether AI search represents a meaningful discovery channel.
The competition between AI search and traditional search intensifies as both Google and Microsoft integrate AI capabilities into their core products. Google’s Search Generative Experience (SGE) adds AI-generated answers above traditional results. Microsoft’s Bing integrates ChatGPT functionality directly into search. These hybrid approaches attempt to combine the traffic-driving function of traditional search with the answer-completion capability of AI systems.
For website operators, this competitive dynamic creates strategic complexity. Blocking AI crawlers might protect content from training data extraction but could forfeit visibility in a growing discovery channel. Allowing all AI access maximizes potential reach but enables training without compensation. The optimal strategy depends on individual business models, content types, and competitive positioning.
Data Methodology and Limitations
Hostinger’s analysis drew from 66.7 billion anonymized log entries from 5 million websites hosted on their platform. The study examined three six-day windows: June 13-18, August 20-25, and November 20-25, 2025. All dates are inclusive, meaning each window captured 144 hours of traffic data.
Bot grouping relied on publicly documented user-agent descriptions, classifications from the AI.txt project, and observed crawling behavior patterns. The analysis filtered out traffic identified as likely human, focusing exclusively on verified bot activity. Websites were counted as “covered” if they received at least one request from a particular crawler during the measurement window.
Several limitations affect interpretation of these findings. The dataset represents Hostinger’s specific customer base rather than a randomized sample of all websites. Sites hosted on Hostinger may have different characteristics, traffic patterns, and crawler exposure than the broader web population.
The six-day measurement windows provide snapshots rather than continuous monitoring. Crawlers with irregular schedules might be underrepresented if their activity didn’t align with measurement periods. Conversely, temporary spikes or anomalies during measurement windows could distort coverage estimates.
Coverage percentages indicate whether a crawler accessed each site, not the depth or completeness of crawling. A crawler counted as accessing a site might have requested a single page or conducted a comprehensive crawl of thousands of pages. The metric doesn’t distinguish between these scenarios.
Request volumes face similar interpretation challenges. Total requests indicate activity levels but don’t specify whether crawlers accessed diverse content or repeatedly requested the same resources. Some crawlers might generate high request counts through aggressive retry logic or frequent recrawling of updated content.
User agent strings are self-reported and potentially spoofable. The analysis assumes crawlers accurately identify themselves through user agent declarations, but sophisticated actors could misrepresent their identity to bypass blocking rules. The study can’t detect or account for this possibility.
Despite these limitations, the scale and scope of Hostinger’s dataset provide valuable insights into crawler behavior patterns and the evolving dynamics of web access. The consistent methodology across three measurement windows enables observation of trends even if absolute coverage numbers carry uncertainty.
Broader Industry Context
Hostinger’s findings align with data from other sources tracking AI crawler activity. Cloudflare’s 2025 Year in Review reported that crawler traffic rose 18% from May 2024 to May 2025, with GPTBot growing 305% and Googlebot increasing 96%. This corroborates the Hostinger observation that AI crawlers are expanding rapidly relative to traditional search.
Cloudflare data also showed GPTBot’s share of crawler traffic growing from 4.7% in July 2024 to 11.7% in July 2025. ClaudeBot increased from 6% to nearly 10%, while Meta’s crawler jumped significantly. These figures reflect growing activity levels even as coverage rates decline due to blocking, suggesting AI companies are crawling more intensively on sites that allow access.
A Senthor analysis from November 2025 found OpenAI dominates AI crawler traffic with 47% of requests (median), confirming OpenAI’s position as the most active player in AI web crawling. The study noted market evolution between 2024 and 2025 showed overall crawl traffic growth of 18%, exactly matching Cloudflare’s figures.
BuzzStream research on news publishers found 79% of top news sites block at least one AI training bot via robots.txt. GPTBot emerged as the most-blocked AI crawler at 49.4% of sites, while PerplexityBot faced blocks from 67% of publishers despite its role in content indexing rather than pure training. These publisher-focused statistics complement Hostinger’s broader hosting data.
Arc XP analysis of AI bot traffic trends found similar patterns, noting training bots face disproportionate blocking while retrieval bots gain acceptance. The convergence of findings across multiple independent data sources strengthens confidence in the underlying trends despite variations in methodology and sample populations.
Vercel’s December 2024 report on AI crawler rise documented GPTBot generating 569 million requests across their network. This specific quantification of resource impact provided concrete data supporting anecdotal reports from publishers about bandwidth costs and server strain.
The Evolving Crawler Ecosystem
Beyond the major AI players, numerous specialized crawlers serve niche functions. Social media platforms operate bots for generating link previews when users share URLs. Meta’s fbexternalhit achieved 69% average coverage with 1.3 billion requests in Hostinger’s data, reflecting Facebook’s role in content sharing.
Advertising and analytics services crawl to verify ad placement, monitor brand mentions, or gather competitive intelligence. Google’s adsbot achieved 9.33% coverage, while various other advertising verification crawlers appeared throughout the dataset at lower coverage rates.
Messaging platforms including WhatsApp (5% coverage) and iMessage (5% coverage) operate crawlers that fetch content for link previews within messages. Pinterest’s bot reached 4% coverage supporting visual bookmarking functionality.
This crawler diversity creates complexity for website operators attempting to implement nuanced access policies. Distinguishing between crawler types requires understanding technical identifiers and the business purposes behind each bot. Many sites lack the expertise or resources to maintain sophisticated crawler management policies, leading to either overly permissive (allow all) or overly restrictive (block all) approaches.
The emergence of specialized tools for crawler management reflects growing demand for middle-ground solutions. Services like Cloudflare’s AI audit features enable sites to monitor crawler activity and implement targeted blocking rules without requiring deep technical expertise. These tools provide visibility into which crawlers access sites and what resources they consume.
Future Trajectory and Open Questions
The diverging paths of training bots versus assistant bots raise questions about sustainable models for AI development and deployment. If training data sources become increasingly restricted, how will AI companies maintain model quality and coverage? Will this drive consolidation around companies that secured training data access through early partnerships or acquisitions?
The assistant bot expansion suggests AI search may establish itself as a legitimate discovery channel. But citation practices, click-through rates, and actual traffic value remain uncertain. Publishers need concrete data on whether AI search visibility translates into meaningful business outcomes before fully embracing these new crawlers.
Technical enforcement of access controls will likely escalate. As robots.txt proves insufficient for managing crawler access, more sites may implement server-level blocking or behavioral analysis. This could trigger counter-measures from AI companies seeking training data, potentially creating an adversarial dynamic similar to ad blocking and ad block detection.
Regulatory frameworks may emerge to codify crawler access rights and responsibilities. The European Union’s AI Act and similar legislation in other jurisdictions could establish legal obligations around training data collection and attribution. These regulatory interventions might resolve tensions that technical and market mechanisms have failed to address.
Economic models for compensating content creators when their work trains AI systems remain underdeveloped. Some publishers have negotiated licensing deals with AI companies, but comprehensive marketplace mechanisms don’t yet exist. Whether such systems develop, and what form they take, will significantly influence crawler access patterns.
The sustainability of free, open web content faces questions in an AI era. If automated systems extract value from content without driving proportional traffic or revenue back to creators, what incentives remain for high-quality content production? This economic tension could reshape the web’s information architecture over time.
Practical Recommendations for Website Operators
Website operators should start by assessing their specific situation and goals. Different sites have different relationships to search, discovery, and content value. A news publisher monetizing through advertising has different incentives than a SaaS company using content for lead generation, which differs from an e-commerce site relying on product discoverability.
Begin by monitoring current crawler activity. Server logs reveal which bots access your site, how frequently they crawl, and what resources they consume. This baseline data enables informed decisions rather than reactive policies based on general principles or anecdotes.
For most content-focused sites, the middle-path strategy makes sense: allow assistant bots while blocking training bots. This maximizes potential discovery through AI search while preventing training data extraction. Implement this through robots.txt entries that specifically disallow GPTBot, CCBot, and similar training crawlers while explicitly allowing OAI-SearchBot, and other assistant-facing bots.
Sites with exceptional server load concerns should implement rate limiting before outright blocking. This manages resource consumption while maintaining some level of crawler access. Monitor the impact of rate limits and adjust thresholds based on actual performance metrics rather than precautionary restrictions.
Consider your content’s competitive positioning. If competitors allow AI crawler access and achieve visibility in AI search results while you block crawlers, you may lose discovery opportunities. Conversely, if your content represents unique intellectual property or proprietary data, protecting it from training may outweigh discovery benefits.
Implement crawler policies as part of a documented content strategy rather than ad-hoc technical decisions. Ensure marketing teams, technical staff, and business leadership share understanding of the trade-offs involved. Crawler access policies affect discoverability, infrastructure costs, and content protection—concerns that span multiple organizational functions.
Stay informed about changes in crawler behavior and AI search implementations. The landscape evolved dramatically in 2024-2025 and will continue shifting as AI search tools mature. Policies that make sense today may need revision as the competitive dynamics and business models stabilize.
The Hostinger analysis of 66.7 billion bot requests documents a fundamental restructuring of web discovery mechanisms. AI assistant crawlers are establishing themselves as significant access patterns alongside traditional search engines, while training bots face growing resistance from content publishers concerned about unauthorized data extraction.
The divergence between crawler types reflects underlying economic tensions. Web content has historically been open to crawlers because search engines drove valuable traffic back to sources. AI systems that generate complete answers without requiring source visits disrupt this reciprocal relationship, forcing publishers to reevaluate access policies.
OpenAI’s OAI-SearchBot achieving 55.67% coverage indicates AI search has reached meaningful scale. Website operators can no longer treat AI crawlers as experimental edge cases; they represent substantial traffic and potential discovery channels. Strategic decisions about crawler access will increasingly influence content visibility and business performance.
The data shows website operators implementing nuanced approaches rather than blanket policies. The emerging middle path—allowing assistant bots while blocking training bots—attempts to balance discovery benefits against content protection concerns. This strategy’s effectiveness depends on whether AI search delivers meaningful traffic and whether training blocks remain technically enforceable.
Traditional search maintains stability with Googlebot’s 72% coverage and established position in the ecosystem. The complementary growth of AI search alongside traditional search suggests a future of diverse discovery mechanisms rather than simple replacement of old by new.
For businesses navigating this transition, crawler management has evolved from a narrow technical consideration to a strategic choice affecting content visibility, infrastructure costs, and intellectual property protection. The 2026 web discovery landscape requires explicit policies aligned with business objectives rather than default acceptance of all crawler access.
About ALM Corp
ALM Corp helps businesses navigate the evolving digital landscape with comprehensive search engine optimization and digital marketing solutions. As AI search tools reshape content discovery, our SEO services ensure clients maintain visibility across both traditional search engines and emerging AI-powered discovery channels.
Our technical SEO expertise includes robots.txt configuration, crawler management, and server optimization to balance discoverability with resource protection. We help clients implement strategic crawler access policies that maximize visibility in AI search results while protecting intellectual property and managing infrastructure costs.
Beyond crawler management, ALMCorp’s integrated digital marketing approach combines SEO, paid media, analytics, and content strategy to drive measurable business results. We stay ahead of industry shifts including AI search adoption to ensure our clients’ content reaches audiences regardless of how discovery patterns evolve.
With over 5,700 websites launched and more than 1 million leads generated for clients, ALMCorp brings proven expertise to the complex challenge of digital visibility. Our data-driven methodology and results-obsessed approach help businesses adapt to changing search dynamics while maintaining focus on revenue and growth.
Whether optimizing for traditional search, preparing for AI discovery, or developing comprehensive digital strategies, ALMCorp provides expert guidance backed by industry insight and technical execution. Contact us to discuss how we can help your business navigate the future of search and discovery.
Frequently Asked Questions
What is the difference between OpenAI’s GPTBot and OAI-SearchBot?
GPTBot collects web content to train OpenAI’s language models, extracting data to improve model capabilities without directly serving user queries. OAI-SearchBot retrieves specific content in real-time to answer user questions in ChatGPT’s search feature. GPTBot has experienced significant blocking by publishers, dropping from 84% coverage to 12%, while OAI-SearchBot maintains 55.67% coverage because it provides direct user value similar to traditional search engines.
Should I block AI training crawlers from accessing my website?
The decision depends on your business model and content strategy. Most publishers are adopting a middle-path approach: blocking training bots like GPTBot and CCBot while allowing assistant bots like OAI-SearchBot. This prevents training data extraction while maintaining visibility in AI search results. Sites with proprietary content may block all AI crawlers, while those prioritizing discovery may allow broader access.
How do I configure robots.txt to allow AI search crawlers while blocking training bots?
Add specific user-agent directives to your robots.txt file. To block training bots while allowing search crawlers, use entries like:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
This configuration blocks GPTBot and CommonCrawl while explicitly allowing OAI-SearchBot for ChatGPT search visibility.
Can AI crawlers significantly increase my server costs?
Yes, AI crawlers can substantially impact bandwidth and server resources. Vercel reported OpenAI’s GPTBot generated 569 million requests in a single month. For sites on usage-based hosting or with bandwidth limits, high crawler activity can trigger overage charges or degrade performance. Implement rate limiting or selective blocking if crawler traffic creates cost concerns.
Do AI crawlers respect robots.txt directives?
Legitimate AI crawlers from major companies like OpenAI, Anthropic, and Google typically respect robots.txt directives. However, the protocol operates on an honor system with no enforcement mechanism. Some crawlers have been documented circumventing robots.txt restrictions. For more robust control, implement server-level blocking through web application firewalls or CDN configurations rather than relying solely on robots.txt.
Will blocking AI crawlers hurt my SEO?
Blocking training crawlers like GPTBot doesn’t affect traditional search engine rankings since these bots aren’t used for Google, Bing, or other search engine indexing. However, blocking assistant bots like OAI-SearchBot prevents your content from appearing in AI search results (ChatGPT search, etc.). If AI search becomes a significant discovery channel, blocking these crawlers could reduce overall visibility, though this differs from traditional SEO impact.
What percentage of websites currently block AI training bots?
BuzzStream research found 79% of top news publishers block at least one AI training bot via robots.txt. GPTBot is blocked by approximately 49% of sites. The Hostinger study showed GPTBot’s coverage dropped from 84% to 12% over six months, indicating widespread adoption of blocking practices. However, practices vary significantly by industry and site type.
How can I monitor which AI crawlers are accessing my website?
Check your server access logs for bot user agents. Most hosting control panels provide log analysis tools. Look for user agents including “GPTBot,” “ClaudeBot,” “CCBot,” “OAI-SearchBot,” and other AI crawler identifications. Tools like Google Analytics filters bot traffic by default, so you’ll need raw server logs. Some CDNs including Cloudflare offer crawler monitoring dashboards.
What’s the difference between AI search and traditional search engines?
Traditional search engines return ranked lists of links to websites, directing users to sources where they engage with full content. AI search generates direct answers synthesized from multiple sources, potentially satisfying queries without requiring users to visit source sites. AI search may cite sources, but the primary value delivery happens within the AI interface rather than through referral traffic to websites.
Are there legal requirements for allowing or blocking AI crawlers?
Currently, no specific laws require websites to allow or block AI crawlers in most jurisdictions. Websites generally have the right to control automated access through technical means. However, legal frameworks are evolving—the EU’s AI Act and similar regulations may establish requirements around training data collection and attribution. Some AI companies have faced lawsuits over training data usage, but legal precedents remain unsettled.
How do assistant AI bots differ from training AI bots technically?
Training bots systematically scan large portions of the web to build comprehensive datasets for model development, crawling at scale to gather diverse content. Assistant bots retrieve specific content in response to individual user queries, making targeted requests based on what information is needed to answer particular questions. Assistant bots are user-triggered and selective, while training bots follow systematic crawling schedules.
Can blocking certain AI crawlers affect social media sharing?
Be careful when blocking bots from Meta, Twitter, or other social platforms. Their crawlers serve multiple purposes including generating link previews when users share content. Blocking Meta-ExternalAgent (training bot) differs from blocking fbexternalhit (link preview bot). Ensure your blocking rules target specific training crawlers without affecting social sharing functionality important for content distribution.
What is the optimal crawl rate limit for AI bots?
Optimal rate limits depend on your server capacity and content update frequency. A common approach allows 1-5 requests per second per crawler, with daily request caps (e.g., 10,000 requests per day). Sites with dynamic content that changes frequently may allow higher limits, while sites with static content can implement stricter restrictions. Monitor server performance and adjust limits based on observed impact.
How long does it take for robots.txt changes to take effect?
Most AI crawlers check robots.txt before crawling, so changes take effect on the next crawl attempt. However, crawlers cache robots.txt files for performance, typically checking for updates daily or weekly. Changes may take 24-48 hours to fully propagate. For immediate enforcement, implement server-level blocking rules that take effect instantly without requiring crawler cooperation.
Will AI search replace traditional search engines?
Current data suggests AI search will complement rather than replace traditional search. Google maintains 72% crawler coverage and dominant market position. Users employ AI search for different query types than traditional search—often using AI for complex questions requiring synthesized answers and traditional search for navigational queries or browsing. The two discovery methods are evolving to serve different user needs.
Sources
- Search Engine Journal – “OpenAI Search Crawler Passes 55% Coverage In Hostinger Study” (January 2026)
- Hostinger – “66 billion bot requests analysis: AI bots rise, SEO tools shrink, and search engines hold their ground” (January 2026)
- Cloudflare – “The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals” (August 2025)
- Cloudflare – “From Googlebot to GPTBot: who’s crawling your site in 2025” (July 2025)
- Cloudflare – “The 2025 Cloudflare Radar Year in Review” (December 2025)
- BuzzStream – “Which News Sites Block AI Crawlers in 2025?” (2025)
- Vercel – “The rise of the AI crawler” (December 2024)
- OpenAI Platform Documentation – “Overview of OpenAI Crawlers” (2025)
- Senthor – “State of AI Bots 2025” (November 2025)
- MediaNama – “User-Driven AI Bots Crawling Grows 15x in 2025: Cloudflare Report” (December 2025)
- Ars Technica – “AI bots strain Wikimedia as bandwidth surges 50%” (April 2025)
- Arc XP – “How AI Bots Crawl News Content: A Look at AI Trends” (2025)
