Check your crawler rules. See what they allow, understand what they restrict, and preview a change.
When a search service wants to fetch a public page, it may first read the site's robots.txt file. That file contains crawler names and paths with Allow or Disallow rules. Entering a page above lets you see how those rules apply to that specific path. We fetch only robots.txt, never the page itself.
OAI-SearchBot is OpenAI's search crawler. Googlebot crawls for Google Search, including the index that supports Google's AI features. This checker shows each policy separately, because the same website may have different rules for each crawler.
A permitted robots.txt path remains only a permission signal. A firewall can still block a request. A page can also have other indexing restrictions, and permission never guarantees an AI citation or recommendation.
GPTBot is associated with potential model training. Its rule is displayed separately from OAI-SearchBot. You can choose to restrict training while allowing search access.
If the file is missing or cannot be retrieved, the result says the rules were not verified. It does not label the website as blocked without evidence.
If OAI-SearchBot has Disallow: /, the search crawler is restricted from a page such as /services. A narrower Allow: /services$ in the appropriate group changes that path's interpretation. A separate GPTBot restriction can remain in place.
User-agent: OAI-SearchBot Disallow: / Allow: /services$ User-agent: GPTBot Disallow: /
Public HTTPS domains only, no IP addresses, ports or query strings. robots.txt up to 500 KB, up to three same-site redirects, a few checks per minute per visitor. No page meta or header inspection, firewall proof, bot visits, indexing or AI-answer measurement.
No. This checks robots.txt permissions for three named crawlers. Permission does not establish whether your business appears in an AI answer. That requires separate answer measurements.
We report that we could not verify the rules. You can paste the file instead. A failed request is not automatically a blocked crawler.
We select the most specific matching crawler groups, combine matching groups, and use the longest matching path rule. Allow wins a tie. Paths are case-sensitive. Wildcards and end-of-path markers are supported. This is a policy interpretation, not a simulation of every crawler implementation.
Pasted rules stay in your browser and are never uploaded. Website checks return current rules at the time shown in the result and are not stored. Downloaded reports stay on your device. There is no account history or automatic CRM save in this tool.
Sources: Google robots.txt documentation and OpenAI crawler documentation. Rules engine v1.0. Next: review a page or build a prompt plan.