What AI Crawlers Look For

Knowing that AI crawlers are visiting your site is useful. Knowing exactly which pages they are reading is where it gets actionable.

The pages AI crawlers visit most are the pages that have the highest chance of influencing what those models say about your brand. If your most-crawled page is a comparison article, AI models are likely using that content to formulate answers about your category. If a key product page is getting zero crawler visits, that page is essentially invisible to AI platforms regardless of how well it ranks on Google.

The Crawled Pages table

Scroll past the donut charts on the AI Traffic page and you will reach the Crawled Pages table. This table lists every page on your site that AI bots have visited during the selected period, ranked by total request count.

Each row in the table shows you three things: the page path, the specific bots that visited that page along with their individual request counts, and the total number of requests that page received across all bots combined.

For example, a page might show ChatGPT Browser with 370 visits, OpenAI Training Crawler with 14 visits, ByteDance Crawler with 9 visits, Claude Training Crawler with 7 visits, and Amazon Crawler with 6 visits, totaling 425 requests for that page. That level of detail tells you not just that the page is being crawled but which AI systems specifically are interested in it and how often each one returns.

How to read this data

Start by looking at your top five most-crawled pages. Ask yourself whether these are the pages you would want AI platforms reading if you were trying to control the narrative about your brand.

If your top crawled pages are blog posts comparing you to competitors, that content is actively shaping how AI models understand your positioning. If those articles present your brand favorably and accurately, that is a good sign. If they are outdated or written by a third party with an agenda, that is worth knowing.

If your most important pages, like your homepage, your core product pages, or your case studies, are sitting low in this list or not appearing at all, AI platforms are not reading the content you most want them to see. That gap is one of the most direct things you can act on to improve your AI visibility over time.

Looking at bots per page

The bot breakdown per page tells you something specific about intent. A page that is mostly visited by AI Training crawlers is being used to train AI models. A page that is mostly visited by AI Search crawlers like Perplexity Search or OpenAI Search is being actively consulted to answer user questions right now.

Pages with a healthy mix of both training and search crawlers are the most valuable pages you have from an AI visibility perspective. They are shaping both what models have learned about your brand and what they are saying about it in real conversations today.

What a healthy crawl pattern looks like

A healthy pattern has a few characteristics. Your most important pages appear near the top of the crawled pages list. Multiple different bots are visiting your site regularly, not just one or two. Your daily crawler activity is consistent rather than spiking occasionally and then disappearing for days.

If you see only one or two bots visiting your site and ignoring everything except your homepage, your content is likely not structured in a way that AI crawlers find easy to process and reference. That is a signal to look at how your content is written, structured, and linked internally.

What comes next

You now have a complete picture of where you stand. You know how often your brand is mentioned in AI responses, how your visibility compares to competitors, what sentiment surrounds your brand, and which AI platforms are actively reading your site.

The next step is doing something with that knowledge. The Start Showing Up section walks you through exactly how Zerply helps you close the gaps you have just identified.

Start closing your visibility gaps