Bot & crawler traffic detection
Search indexers, social preview bots, SEO crawlers, uptime monitors, AI training crawlers — something is always hitting your site that isn't a person. Left mixed in with real visitors, it inflates request counts and quietly distorts conversion rate math. Measurebase identifies this traffic by name and charts it separately, automatically, for every domain.
What gets detected
Every request is matched by name against a broad, regularly extended set of known bots and crawlers, including:
- Search engines — Googlebot, Bingbot, DuckDuckBot, YandexBot, Baiduspider, and others
- Social & preview bots — Meta's link crawler, LinkedInBot, Twitterbot, Slackbot, Discordbot, WhatsApp, Telegram
- AI & LLM crawlers — GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Amazonbot, CCBot, and others actively crawling for AI training or live answers
- SEO & marketing tools — AhrefsBot, SemrushBot, MJ12bot, DotBot, and similar crawlers used by SEO platforms
- Security scanners & uptime monitors — UptimeRobot, Pingdom, StatusCake, and various vulnerability/asset scanners
- Headless browsers — Puppeteer, Playwright, Selenium and similar automation tools
Anything else that merely looks bot-like — a generic "bot", "crawl", "spider" or similar token in its User-Agent with no specific match above — still gets caught and grouped as Other bot, rather than silently passing through as an unidentified visitor.
Reading the chart
Each domain's Domain statistics page has a Bot traffic panel: a stacked bar chart, one bar per day over your selected date range, broken down by bot name. The busiest bots for the period get their own segment and legend entry with total requests and share of bot traffic; anything past the top few folds into an "Other" segment so a long tail of one-off crawlers doesn't crowd out the bots that actually matter for that domain.
Why this matters
Bot traffic is normal and mostly harmless — you want Googlebot indexing your site. The value is in separating it out: knowing what share of your requests are crawlers rather than people keeps conversion rates and funnel numbers honest, and a sudden new entrant in the chart — an AI crawler you didn't expect, a scanner probing your endpoints — is worth a second look the moment it shows up rather than months later.