AI Crawler
A bot operated by an AI company to fetch, index, or retrieve web content for use in training or live answer generation.
Definition
An AI crawler is an automated agent that fetches web content on behalf of an AI company. AI crawlers serve two broad purposes: training crawlers collect content to be used in model training datasets; retrieval crawlers fetch content in real time to ground live AI answers (as in retrieval-augmented generation). Examples include GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google's various AI-related crawlers. AI crawlers typically identify themselves via user-agent strings, but verification requires checking against known IP ranges and stated purposes.
Why it matters
Understanding which AI crawlers visit a site — and why — tells businesses whether their content is being ingested for training, for live answers, or both. It also reveals whether a site is inadvertently blocking AI crawlers via robots.txt rules that haven't been updated for the AI era, which would limit their AI visibility regardless of content quality.
Example
A business checks its CDN logs and notices PerplexityBot fetching its homepage and service pages frequently. This is a retrieval crawl — Perplexity is fetching content to ground live answers. The business can confirm that their content is being read, but whether it is being used in responses requires AI visibility monitoring, not just crawler observation.
How it differs from related terms
An AI crawler fetches content — it is infrastructure. Agent Traffic (GeoScript's planned capability) refers to the broader category of AI agent and crawler visits to a site, including intent classification and verification. AI citation is an output — the result of a crawler successfully reading and an AI engine choosing to reference that content. Crawling is necessary but not sufficient for citation.
Related terms
AI crawler and agent visits to a client's website, classified by identity and purpose. Also an upcoming GeoScript capability.
AI CitationA named reference to a specific business, brand, or source within an AI engine's generated response.
AI VisibilityWhether and how prominently a business appears in AI engine responses to relevant queries.