AI crawlers
Term from the world of AI visibility – explained simply by the GeoTrova team.
Programs that AI providers use to fetch websites – for model training, for a search index, or for a live fetch during a user request.
Want to know how your brand stands on this topic? GeoTrova measures your AI visibility across ChatGPT, Google AI Overviews, Gemini, Perplexity, Microsoft Copilot and Google AI Mode – First measurement in minutes · 7 days free, charged only afterwards.
Three types of AI crawlers
Training crawlers like GPTBot or ClaudeBot collect content for future models. Search crawlers like OAI-SearchBot or PerplexityBot build the index that answers draw their sources from. User fetches like ChatGPT-User retrieve a page when someone asks for it. For visibility today, the search index and live fetch matter most. Many AI crawlers don't execute JavaScript – content that only loads later in the browser often goes unseen.
How to keep track of them
Server logs show which bots fetch which pages, and when. GeoTrova' bot analysis evaluates them, recognises 48 AI bots, compares the visits with your robots.txt, and links the first bot visit to the first citation for each URL.
More in the Glossary.
GPTBot · PerplexityBot · robots.txt for AI bots · Time-to-First-Citation
Set your course
today.
Start free – in a few minutes you will know where your brand stands across the AI universe.