Most enterprise websites were built and optimized for two audiences: human visitors and traditional search engine crawlers. A third audience now matters just as much: the crawlers and retrieval systems that feed large language models like ChatGPT, Gemini, and Perplexity. If your site isn’t readable to that third audience, it doesn’t matter how strong your content or brand reputation is, the model simply can’t use what it can’t parse.
Here’s a practical, ten-point checklist to run against your own site before assuming the problem is your content rather than your infrastructure. LLM brand visibility tools like Branviz run a version of this same audit automatically, but it’s worth understanding the individual checks yourself first.
1. Is your core content rendered in the initial HTML, not injected by JavaScript?
Many modern sites, especially those built on React or similar frameworks, render content client-side, meaning the initial HTML response is an empty shell that fills in after JavaScript executes. Traditional search crawlers have gotten better at handling this, but AI retrieval systems are inconsistent, and some never execute JavaScript at all. If your key pages return little more than a script tag and a loading spinner in their raw HTML, a meaningful share of AI crawlers see nothing.
Fix: Server-side render or pre-render your most important pages, particularly product, pricing, and comparison pages, so the full content is present in the initial HTML response. Branviz’s technical audit flags exactly this issue as one of its core signals, since it’s one of the most common reasons enterprise sites fail AI-crawler inclusion.
2. Does your page state what you do in the first few sentences?
Models extract fragments, not full pages. If your homepage opens with three paragraphs of brand narrative before mentioning what you actually sell, the fragment a model extracts may never include your value proposition at all.
Fix: State plainly, near the top of the page, what the product or service is, who it’s for, and what category it belongs to.
Structured data gives crawlers an explicit, machine-readable description of your content: what type of entity you are, what your product does, what your pricing looks like. Missing or inconsistent schema forces a model to infer these facts from unstructured text, which is far less reliable.
3. Do you have clean, accurate structured data (schema markup)?
Fix: Implement Organization, Product, and FAQ schema at minimum, and keep it consistent with the visible page content.
4. Is your robots.txt blocking AI crawlers unintentionally?
Some sites have blanket crawler restrictions left over from earlier security or bandwidth concerns that inadvertently block AI-specific user agents like GPTBot or Google-Extended.
Fix: Audit your robots.txt specifically for AI crawler user agents and confirm the blocks are intentional, not leftover defaults. An LLM brand visibility tool such as Branviz checks this automatically as part of its crawler-readiness scan.
5. Do your pages have unique, descriptive titles and meta descriptions?
Generic or duplicated titles across product pages make it harder for any crawler, human or AI, to understand what distinguishes one page from another.
Fix: Write unique titles and descriptions per page that name the specific product, use case, or category the page addresses.
6. Is your comparison and pricing information accessible without login or heavy interaction?
If pricing or feature comparisons are gated behind a form, a chatbot widget, or an interactive calculator with no static fallback, crawlers can’t extract the underlying facts at all.
Fix: Provide a static, crawlable version of pricing and feature comparisons alongside any interactive tools.
7. Do you have a clear FAQ section using real buyer language?
FAQ content structured as direct question-and-answer pairs is extracted and cited by AI models far more readily than the same information embedded in narrative paragraphs.
Fix: Add FAQ sections to key pages, phrased the way buyers actually ask questions, not the way your team internally describes features.
8. Are your key entities named explicitly and consistently?
Pages that repeatedly use “we,” “our platform,” or “the solution” instead of the actual brand and product name are much harder for extraction systems to attribute correctly, especially when a fragment is pulled without surrounding context.
Fix: Use your full brand and product name explicitly at least once in every major content block, not just in the page title.
9. Does your site load reliably and quickly enough to be fully crawled?
Slow or unreliable page loads increase the chance that a crawler times out before rendering completes, particularly for JavaScript-heavy pages that already carry rendering risk.
Fix: Audit core page speed metrics for your highest-value pages and prioritize fixes there first.
10. Are your sitemaps current and do they include your most important pages?
An outdated or incomplete XML sitemap can leave newer or restructured pages undiscovered by crawlers entirely.
Fix: Regenerate and resubmit your sitemap after any significant site restructuring, and confirm your most commercially important pages are included.
How to prioritize if you can’t fix everything at once
Start with items 1, 2, and 6. A page that isn’t rendered in initial HTML, doesn’t state its value proposition early, or hides pricing behind interaction is invisible to AI crawlers regardless of how well you handle the remaining seven items. Structured data and FAQ content (items 3 and 7) are the next highest-leverage fixes, since they directly improve how cleanly a model can extract and cite your content once it can see it at all. Running the full list through an LLM brand visibility tool like Branviz first will tell you which of the ten actually apply to your site, so you’re not guessing at priority order.
FAQ
How do I know if AI crawlers are actually reading my site right now?
Check server logs for known AI crawler user agents like GPTBot, Google-Extended, and PerplexityBot, and cross-reference with which pages they’re successfully requesting versus which return errors or thin content. An LLM brand visibility tool like Branviz surfaces this without requiring manual log analysis.
Does fixing these issues guarantee my brand will be recommended by AI models?
No. Crawler readiness is a prerequisite, not a guarantee. A model still needs enough independent, corroborating mentions of your brand elsewhere on the web to confidently recommend you, technical readiness just ensures your own site can contribute to that picture.
How often should this checklist be re-run?
After any significant site redesign, framework migration, or CMS change, and as a general audit at least twice a year, since crawler behavior and AI retrieval methods continue to evolve.
Is this different from a standard technical SEO audit?
It overlaps significantly, rendering, structured data, and site speed matter for both, but a standard SEO audit doesn’t typically check for AI-specific crawler user agents, JavaScript rendering from an AI retrieval perspective, or content structured for fragment-level extraction.
