1. Request
A server saw a matching crawler token request a specific path.
AI crawler directory
Identify the bots requesting your website, understand why they crawl, and find the verification sources available for each crawler.
9 reviewed crawlers across 3 providers
OpenAI
OpenAI publishes separate crawlers for model training, ChatGPT search, user-triggered page retrieval, and advertising systems.
CrawlerPurposeWhat it meansVerification
GPTBotTrainingGPTBot is OpenAI’s automated crawler for content that may be used to improve its generative AI foundation models.Published IP sourceOAI-SearchBotIndexingOAI-SearchBot discovers and indexes web content for search experiences in OpenAI products.Published IP sourceChatGPT-UserAnswerChatGPT-User represents page retrieval associated with an action a ChatGPT user initiated.Published IP sourceOAI-AdsBotOtherOAI-AdsBot is an OpenAI crawler associated with advertising systems rather than model training or search indexing.Published IP sourceAnthropic
Anthropic distinguishes automated model-development crawling, search discovery, and page retrieval initiated by a Claude user.
CrawlerPurposeWhat it meansVerification
ClaudeBotTrainingClaudeBot is Anthropic’s automated crawler for model development and improvement.Official guidanceClaude-SearchBotIndexingClaude-SearchBot accesses pages to improve search relevance and discovery for Claude experiences.Official guidanceClaude-UserAnswerClaude-User represents a page retrieval initiated by a person using Claude.Official guidancePerplexity
Perplexity documents an automated search crawler and a distinct user-triggered fetcher used when people request pages through its products.
CrawlerPurposeWhat it meansVerification
PerplexityBotIndexingPerplexityBot is Perplexity’s automated crawler for search discovery and index maintenance.Published IP sourcePerplexity-UserAnswerPerplexity-User represents a page fetch associated with a person using a Perplexity product.Official guidanceHow to read crawler activity
2. Verification
Published network evidence can strengthen identity confidence when available.
3. Outcome
A successful response still does not prove indexing, citation, or recommendation.
See these crawlers in your own traffic
Start tracking which AI crawlers visit your website.