Scored checks
llms.txt · weight 18/100
What it is. A markdown file at /llms.txt that gives AI systems a curated map of your site: what you are, and which pages matter.
Why it matters. It is the emerging convention AI crawlers and agents look for when they want an overview rather than a scrape. Without it, engines assemble their understanding of you from whatever HTML they happen to parse — and you lose control of the narrative.
How to fix it. Publish /llms.txt as real markdown: an H1 with your name, a one-line summary in a blockquote, then grouped links with short descriptions. Serve it as text/markdown or text/plain, and make sure a missing file returns a real 404 rather than your app shell.
Reference: llmstxt.org/
robots.txt AI bots · weight 18/100
What it is. The rules in /robots.txt that apply to AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and others.
Why it matters. Assistants that respect robots.txt decide from this file whether your content can appear in their answers at all. A blanket disallow, or rules written years ago for search engines, can silently remove you from AI search entirely.
How to fix it. Add explicit allow rules for the AI agents you want citing you. Being silent is not neutral — it leaves each crawler to apply its own default.
Reference: www.robotstxt.org/
Schema.org JSON-LD · weight 14/100
What it is. Structured data embedded in the page as JSON-LD, describing what the page and the organisation behind it actually are.
Why it matters. It is the strongest machine-readable statement of identity you can make. Generative engines lean on it to tell your brand apart from similarly named products — which matters more the more generic your name is.
How to fix it. Add JSON-LD with Organization and WebSite at minimum. Add Product, Article or FAQPage where they genuinely apply.
Reference: schema.org/
Sitemap · weight 12/100
What it is. A valid, reachable sitemap.xml — either at the root or declared in robots.txt.
Why it matters. It is how crawlers, classic and AI alike, find pages beyond whatever is linked from your homepage. A missing or broken sitemap means partial coverage of your content, and partial coverage means partial answers.
How to fix it. Serve a valid /sitemap.xml and declare it in robots.txt. We follow the declared URL, so a sitemap index living elsewhere still counts.
Title & meta description · weight 10/100
What it is. The document title and meta description of the page being scanned.
Why it matters. Still the primary snippet source for classic results and a common source for AI citations. Thin or missing descriptions produce vague citations that earn fewer clicks.
How to fix it. Write a descriptive title and a meta description of roughly 50–160 characters that says what the page is, not what the company aspires to be.
llms-full.txt · weight 8/100
What it is. An optional companion to llms.txt carrying extended context — deeper product facts, documentation, positioning.
Why it matters. Where llms.txt is the map, llms-full.txt is the territory. Agents that want detail before answering a question about you have somewhere to find it.
How to fix it. Add /llms-full.txt with the fuller version of your story. Optional, but cheap insurance against an engine guessing.
Reference: llmstxt.org/
Canonical URL · weight 8/100
What it is. A link rel="canonical" declaring the authoritative URL for the page.
Why it matters. Without it, duplicate and parameterised URLs compete with one another and split whatever authority the page earns — in AI indexes as much as classic ones.
How to fix it. Add a canonical link to every significant page, pointing at the URL you want cited.
Open Graph · weight 6/100
What it is. og:title, og:url and og:image metadata.
Why it matters. Open Graph controls how your pages appear when shared, and several AI crawlers read it as supplementary metadata. Missing OG data weakens both.
How to fix it. Add og:title, og:url and og:image. Keep them consistent with the page title and canonical URL.
Indexable page · weight 6/100
What it is. Whether the page carries a noindex directive in its robots meta tag.
Why it matters. A noindex removes the page from search and most AI retrieval pipelines. When it is unintentional — a staging rule that shipped to production — it is the single most damaging finding on this list.
How to fix it. Remove noindex from any page that should be discoverable. If you find one here you did not expect, check what else your last deploy carried over from staging.
