Microsoft Clarity recently added a new bot violation tracking feature to its Bot Analytics dashboard which identifies cases of violations of robot exclusion protocol rules. The feature allows seeing requests made to paths that were disallowed by the website’s robots.txt file, allowing site owners to gain insight into the crawler behavior they would otherwise need to detect by analyzing server logs. It arrives at a moment when AI crawler compliance with robots.txt has become a pressing concern across the web.
The new violation tracking follows other new features that have been rolled out to Clarity’s Bot Analytics dashboard in recent months. In May, Clarity started showing the queries used by AI to ground their citations on websites. The feature that tracks bot violations is another step on the road, shifting the focus from the citation usage to crawling compliance.
What the Violation View Actually Shows
Whenever a request is made by a bot to the site connected to Microsoft Clarity, it is compared to the directives contained in the robots.txt file. In case the request is violating any of the directives, the number of such requests is calculated and shown as a percentage of the overall bot activity in a chosen time period.
Users are allowed to filter bot violation data based on bot operator, bot name, the type of the request activity, and even the exact URLs or paths requested by the bot. The ability to filter this data makes it possible not only to see if any violations have occurred but also to be able to compare different crawlers and their behavior.
Not all bots are equal. Different violation types and different violators have different importance and consequences for the website owner. It allows comparing the requests of known search engine bots, which are normally compliant with robots.txt rules, and violations performed by other bots that might not comply with such rules, using the side-by-side comparisons provided by Clarity.
How to Enable It
The violation tracking feature is not enabled for all the sites by default. It needs to be enabled manually in the AI Visibility section of Project Settings, and it is only available for sites that use a supported content delivery network.
Those networks include Fastly, Amazon CloudFront, Cloudflare, Azure Front Door, and Akamai. Also, WordPress sites that use the latest version of the Microsoft Clarity plugin are supported. Other sites that don’t use a supported CDN or Clarity WordPress plugin cannot currently use the feature. This limitation is worth mentioning since it means that the feature cannot be used out-of-the-box for every site.
Why This Matters Now
The reason this update matters is straightforward. The issue that the new feature addresses – the problem of AI crawlers ignoring robots.txt rules – has become especially noticeable and pressing in recent times.
AI crawlers used by companies building large language models and AI search products have been documented repeatedly ignoring robots.txt directives and requesting disallowed URLs. As a result, site owners experience additional server load from bots that shouldn’t access certain URL paths, get inaccurate analytics due to inflated numbers of non-human traffic requests, and experience scraping of their content from the URL paths which they’ve restricted for access.
The robots.txt rules are never enforceable and just contain information on which URL paths the bots are supposed to avoid. If some bot chooses to ignore the file, it can access the disallowed URL. What Clarity provides is a way to measure how often that is happening and which crawlers are responsible.
That information is quite important itself. Being aware of the existence of violations is one thing, but knowing the exact bots causing the violations, how frequent it happens, and which URL paths are accessed – that information is actionable. It does not prevent violations, but documents them in a way that can support complaints or decisions about stronger blocking measures.
The Limits of What the Tool Measures
Microsoft Clarity makes it very clear what this feature can and cannot measure. The violation data reflects the requests that reached the disallowed URL paths. This is just information on what happened. The bot in violation view does not necessarily act maliciously. Some violations could happen due to misconfiguration of the crawlers, old cached copy of the robots.txt file, or bots that just haven’t updated their crawling behavior to the current rules.
Also, Clarity does not block any requests. It just measures them. Site owners who want to prevent bots from accessing restricted paths need to use the proper blocking measures through the CDN, server configuration, or firewall. The Clarity violation data can be used to help decide about what blocking measures to implement, but it doesn’t execute any actions.
This transparency from Microsoft about what the tool actually does is worth noting. The violation view provides visibility, not enforcement. For some site owners who were flying blind on this issue, it can be quite valuable. For those who expect this tool to enforce the robots.txt rules, the expectation should be adjusted.
Why Manual Log Analysis Was Not Scaling
Prior to this update, the real-world method of monitoring bot violations entailed parsing through server logs to find the bots and then checking to see if each URL they requested violated the rules defined in the website’s robots.txt configuration. While this is feasible for small websites with minimal bot activity, it becomes increasingly impractical for more substantial operations.
The utility of this new functionality is automation and data aggregation. Rather than someone manually spending hours sifting through logs, Clarity compares the URL requests with the rules of a site’s robots.txt file continuously and presents the findings in an easily filterable manner.
This functionality is especially relevant for publishers and content-focused websites that experienced a massive uptick in AI crawler traffic in the last two years. Publishers are known for having rather complex robots.txt configurations, which forbid AI training crawlers from accessing large parts of the site, but have had very few ways to monitor compliance with those restrictions. Clarity solves this problem with its new feature.
What This Means for Site Owners and SEO Teams
What this means for SEOs and site owners is that there is now a free tool available to provide visibility into bot compliance. Microsoft Clarity is a free service, and the bot violation tracking does not require any paid plans. This makes Clarity available to all publishers and site owners that are looking to improve their bot visibility, even if they cannot afford enterprise-level bot management.
The violation view offers valuable data that can be leveraged in several ways. Identifying which bots are persistently violating disallowed paths is valuable to decide which crawlers should be blocked at the CDN level. Knowing which specific paths are hit the most by non-compliant bots can give site owners ideas about which areas of the website need additional security measures. Having a timestamped record of violation activity might also be helpful if one wants to escalate any compliance issues to a bot operator.
This new functionality raises the question whether increased visibility into violation activity will lead to any changes in crawlers’ behavior. Increased visibility increases bot operators’ accountability, but it is still up to operators themselves if they want to address this accountability in some meaningful way. What is clear is that site owners now have better tools to see what is happening on their own sites..
Frequently Asked Questions
Q 1: What does Microsoft Clarity’s new bot violation feature do?
It identifies cases where bots are requesting URLs that are disallowed by the site’s robots.txt configuration and shows these violations as a percentage of total bot activity in the Bot Analytics dashboard. Violations can be filtered by the name of the bot, operator, activity type, and specific paths.
Q 2: Does the feature automatically activate for all Clarity users?
No. It must be turned on by the project administrator in the AI Visibility section of Project Settings. This functionality is only available for sites using a supported CDN, including Fastly, Amazon CloudFront, Cloudflare, Azure Front Door, Akamai, and WordPress sites with the latest Clarity plugin.
Q 3: Does the tool block bots from accessing disallowed URLs?
No. Clarity simply records and shows violations but does not block anything. If a site owner wants to prohibit access to disallowed paths, he or she has to implement blocking at CDN or server level manually.
Q 4: Why is robots.txt compliance a growing concern right now?
Repeated documentation of AI training crawlers accessing the content disallowed by robots.txt has resulted in increased server load, distorted analytics, and scraping of restricted parts of websites, thus increasing the need for more visibility into crawlers’ behavior.
Q 5: What can site owners do with the violation data?
Identify which specific crawlers violate the robots.txt rules, see which paths are hit the most, and base their decision to either block bots at CDN level or escalate violations to bot operators on that data.
Q 6: Is Microsoft Clarity free to use?
Yes. Microsoft Clarity is a free tool, and the bot violation tracking feature does not require any paid plans.
Q 7: What is the difference between the violation view and what Clarity added in May?
In May, Clarity started offering the ability to see grounding queries behind AI citations, showing how AI products utilize content on a website. The bot violation feature monitors access behavior, specifically whether crawlers follow the instructions in a website’s robots.txt file.
Q 8: Does seeing a bot in the violation view mean it is acting maliciously?
Not necessarily. Some violations might happen due to a misconfigured crawler, out-of-date cached robots.txt file, or bots that have not updated their behavior according to recent changes. The tool just records the violations but does not imply any intentions.
