How to See If AI Bots Like GPTBot Visit Your Website
Before an AI assistant can mention your business, it has to read about it. AI companies send their own bots to read web pages - and most now have separate bots for different jobs: one that collects pages for training, one that builds the index their search answers come from, and one that fetches a page when a user asks about it. Knowing which visit you, and which you allow, is one of the simplest GEO checks there is.
The AI bots worth knowing
- OpenAI (ChatGPT): GPTBot collects pages that may be used for training. OAI-SearchBot decides whether you can appear in ChatGPT's search answers - OpenAI says sites that block it won't be shown there. ChatGPT-User fetches a page when a user asks about it; OpenAI says robots.txt rules may not apply to it.
- Anthropic (Claude): ClaudeBot for training, Claude-SearchBot for Claude's search, Claude-User for pages a user asks about. The older names anthropic-ai and Claude-Web are no longer used - a robots.txt that only blocks those doesn't block today's bots.
- Perplexity: PerplexityBot builds Perplexity's search index (Perplexity says it isn't used for training). Perplexity-User fetches pages for a user and generally ignores robots.txt.
- Others: Applebot, Amazonbot, Meta-ExternalAgent, Bytespider and CCBot (Common Crawl).
Google is different: AI Overviews and AI Mode use Google's normal crawler, Googlebot, and its normal index. Google-Extended is only a robots.txt setting for whether your content may be used to train and ground Gemini - it doesn't keep you out of AI Overviews or AI Mode. For those, Google Search Console now has its own setting.
Train or search: you can choose
Because the jobs are split, you can decide separately. Many businesses that don't want their content used for training block GPTBot and ClaudeBot but keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed - so they can still appear in the assistants' answers. Blocking the search bots is what keeps you out.
Three ways to see the visits
- Your server's access log. Each visit is a line with the visitor's "user agent". Search it for the names above. Many hosts (cPanel, Plesk) let you download the log.
- Your CDN or host. Cloudflare and similar services can show bot traffic, sometimes with a ready-made AI bots view.
- A tool that counts them for you. BusinessVeno's AI traffic page shows which AI bots read which pages, day by day - from a WordPress plugin, a Cloudflare Worker, a Next.js middleware or a log file you upload.
Beware of fakes
Anyone can call their scraper "GPTBot". OpenAI, Perplexity and others publish the IP addresses their bots use, so a real visit can be checked. A good tool checks the address, not just the name - BusinessVeno leaves out visits from a "GPTBot" whose address OpenAI doesn't use.
Are you blocking them by mistake?
Open yourdomain.com/robots.txt. A line like User-agent: OAI-SearchBot followed by Disallow: / keeps you out of ChatGPT's search answers. Bot protection and firewalls can do the same without you noticing. After a change, give the bots a day or so to pick it up.
People who arrive from AI answers
The other half of AI traffic is people: visitors who clicked a link in a ChatGPT, Perplexity or Gemini answer. They show up with a referrer such as chatgpt.com or perplexity.ai, or a utm_source tag. Counting them shows whether AI answers already send you customers.
See it for your own website with BusinessVeno's AI traffic - or start with a free AI visibility check.