We spent a week reading crawler logs so you do not have to: 90 days of server logs across our site and six client sites, filtered to the user agents that feed AI answer engines. The headline is that AI crawlers behave nothing like Googlebot, and most optimization advice quietly assumes they do.
They fetch raw HTML and mostly do not run JavaScript. They visit more often than you think and read less than you hope. And they skip predictable parts of every page.
Here is what the logs show, and the five-minute audit that tells you what the machines actually see.
Who is crawling: a 90-day census
Across seven sites, five user agents did almost all the AI crawling: GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended, and CCBot from Common Crawl. Together they averaged 410 hits per site per week, small sites included. GPTBot was the heaviest at roughly 38% of AI-crawler traffic, and PerplexityBot was the most responsive, often refetching a page within 48 hours of an update.
Amazonbot and Bytespider showed up too, but their volume was noise by comparison, under 4% combined.
The volume is what surprised our clients most. One 40-page consulting site logged 1,900 AI-crawler hits in a month. Nobody was watching, because analytics dashboards do not show bots.
If you publish regularly, you are already being read weekly. The only question is what the reader gets.
Most AI crawlers do not run your JavaScript
GPTBot, ClaudeBot, CCBot, and most of their peers fetch your HTML and parse it as delivered. They do not execute JavaScript, wait for hydration, or trigger lazy loading. If your content renders client-side, through a React app filling an empty div, testimonials that load on scroll, or tabs that fetch on click, that content does not exist for them. The page they read is the page curl sees, not the page Chrome sees.
This is the single most common failure we find in GEO audits. A beautiful site, strong content, and a view-source that shows a spinner.
Server-side rendering or static generation fixes it wholesale. If you cannot re-platform this quarter, at least render your core answers, headings, dates, and FAQ into the initial HTML.
If it is not in view-source, it is not in the answer.
They read shallow, so front-load the HTML
AI crawlers spend little time per page and favor content that appears early in the document. In our logs, most visits were quick single fetches with no follow-up requests for assets. Pages whose main content sat deep in the DOM, after 200 lines of nav markup, banners, and script tags, got cited less than structurally lighter pages carrying the same words. Put the substance in the first screenful of HTML, not the last.
The practical version: heading, byline, updated date, and a 40-90 word answer before anything decorative. Push the cookie banner markup and the mega-menu lower in the source order.
We moved one client's summary block from the bottom of the page template to the top. Same words, same URL. Perplexity started quoting it 12 days later.
Weight matters too. The fastest-cited pages in our sample were under 300 KB of HTML. Bloat is not neutral when the reader is on a budget.
What they read first: headings, dates, and definitions
Structure is the crawler's reading order. Question-shaped h2s, a visible published and updated date, definition sentences that stand alone, and FAQ pairs get extracted far more reliably than flowing prose. Schema helps as confirmation, Article and FAQPage mainly, because it restates the same facts in a format that needs no inference. The pattern across engines is consistent: they lift passages that already look like answers.
That is why a tight 50-word paragraph directly under a clear heading beats 500 words of buildup. The engines are not impressed by buildup. They never read it.
Dates matter more than most people believe. Pages showing an updated date in the first screenful were refetched noticeably more often in our logs, which is exactly what you want.
Write for a reader who grabs the first clean answer and leaves, because that is the literal behavior.
What they skip: nav, footer, hero, carousel
Every extraction pipeline strips boilerplate before the model reads anything. Navigation menus, footers, sidebars, cookie notices, and repeated calls to action are discarded wholesale. Hero sections usually die too, because short punchy fragments carry no extractable claim. Carousels and tab panels often never render at all without JavaScript. If your best proof points live in a footer badge or a sliding testimonial, the engines have never read them.
The test is brutal and useful. Strip your page to plain paragraphs and headings, and whatever survives is your actual content. On many marketing sites, almost nothing survives.
One client had their strongest fact, 214 projects delivered with a 94% retention rate, living exclusively in a footer graphic. Zero citations mentioned it until we wrote it into body prose.
Move the load-bearing facts, who you are, what you do, and the numbers that prove it, into plain sentences on the page.
robots.txt and your firewall are quietly deciding for you
The major AI crawlers largely respect robots.txt and identify themselves honestly, which cuts both ways. A disallow rule someone added in 2023 to block AI may now be removing you from the engines your buyers ask. Worse, CDN bot protection blocks silently: we found two client sites whose WAF returned 403s to GPTBot on every single request. The owners had opted out of AI search without ever deciding to.
Check three layers: the robots.txt rules themselves, your CDN or WAF bot settings, and any rate limiting. Any one of them can make you invisible while competitors get cited.
Blocking is a legitimate choice for some businesses with content worth protecting. Make it a choice.
Both 403ing clients unblocked the bots in June. Crawl volume returned within a week, and first citations followed inside a month.
Run the five-minute crawler audit
Run four checks. One: curl your most important page and confirm the core answer appears in the raw HTML. Two: grep 30 days of access logs for GPTBot, ClaudeBot, and PerplexityBot to confirm you are being crawled at all. Three: request your page with each bot's user agent and check you get a 200, not a 403. Four: read your robots.txt and CDN bot rules and confirm they say what you intend.
Total time is about five minutes with server access, fifteen without. It is the highest-leverage quarter hour in GEO.
Repeat it quarterly. Crawler behavior shifts fast, and half of what these logs taught us was not true a year ago.
The machines are already reading. Decide what they find.