Your next customer might never open Google.
A growing share of buyers now research and compare vendors through AI tools like ChatGPT, Claude, Perplexity, and Gemini. These tools pull information from the web, summarize it, and present it directly in conversation. No search results page. No ten blue links. Just an answer.
For years, being found meant ranking on Google. Now, being found also means being readable by language models. And most websites are not.
If an AI tool cannot properly parse your site, your brand does not exist in that conversation. Your competitor’s might.
How AI tools actually read your website
AI models do not browse the web the way Google does.
Google’s crawler was built to understand web pages. It executes JavaScript, renders layouts, and processes complex structures. AI crawlers are far simpler. Think of them as basic HTTP clients that download your HTML and try to extract meaning from it within a limited context window.
That context window is the critical constraint. It is the chunk of text the model can process at once. When a page is too large, only part of it gets loaded. If your main content sits outside that window, the model never sees it.
In our testing, pages weighing 500 KB or more of HTML already caused significant problems. Models lost headings, missed core claims, and in one case could not even find the main <h1> tag because it was buried too deep in the document structure. The model started hallucinating instead of quoting the page.
This happens whenever pages are built for browsers and not for machines that read text.
Why clean markup matters more than ever
What makes pages large in the first place? Bloated markup, excessive wrapper divs, inline styles, and CSS class names that carry no semantic meaning. All of this is noise for an AI model trying to understand your page.
Tomasz Guz from kodem.dev measured this directly in his analysis of AI website optimization. His research showed that cluttered markup with frameworks like Tailwind resulted in the model consuming 63.57% more tokens compared to clean, semantic HTML. That is not a minor overhead. It is the difference between an AI tool reading your full page or cutting it short halfway through.
As he puts it:
“(…) content is still king, but if great content is placed inside a chaotic and non-semantic structure, we lose additional opportunities. We may lose users, and we may also lose visibility in AI-powered search experiences.”
Enter Markdown for Agents
Cloudflare introduced a feature called Markdown for Agents that addresses the delivery problem directly.
The concept is content negotiation. When an AI agent sends a request with the Accept: text/markdown header, the server returns the page as structured markdown instead of full HTML. Same URL, same content, but delivered in a format that language models can work with far more efficiently.
The token savings are significant. A heading like ## About Us in markdown takes roughly 3 tokens. The same heading in HTML with class names and IDs can take 12 to 15 tokens. Across a full page, Cloudflare reports token reductions of up to 80%.
Two things follow from this.
If you are building your own AI-powered workflow that pulls content from web pages, this is a clear win. Configure your agent to send the Accept: text/markdown header and you get clean content directly. No need to rely on the model parsing raw HTML. Fetch the markdown, pass it to the model, and your agent gets exactly what matters without the noise.
But do not expect AI crawlers to switch overnight. Enabling Markdown for Agents will not magically make your site appear in AI chat answers for every query. Cloudflare may be starting a shift in how crawlers consume content, but it will take time before this becomes a broadly adopted standard.
In other words, if you care about AI visibility and GEO (Generative Engine Optimization), your website’s HTML should be clean and well-structured in the first place. Markdown for Agents is the next step after that, not a substitute for it.
How we built it for Nuxt projects for free
Ok, let’s assume you want to serve a markdown version of your website to AI agents, but you’re not proxying your website through Cloudflare. Or you are on Cloudflare, but you don’t want to pay for their Pro plan (which is required for turning on the Markdown for Agents feature). We built a custom solution that does the same thing independently.
Quick side note: we recently started a new brand, coditive.ai, focused on AI automations, AI-driven enhancements, and LLM visibility. On the Generative Engine Optimization side, we look at how models read your content, how your site shows up in AI-driven conversations, and what you can do about it.
Back to the implementation. One of our clients had a Nuxt-based project and wanted markdown delivery without paying for Cloudflare Pro. The site was fully prerendered, so we created a post-build step that automatically converts each HTML file to clean markdown with metadata, then added server configuration to handle content negotiation and serve the right format based on request headers.
This pattern works with any static-site generator or prerendered framework. You need two things: a build-time conversion step and a server rule that routes based on the Accept header. In our case, we used Nitro hooks to create a custom conversion step, and Vercel routing to serve the right format based on the request headers.
We are also building a WordPress plugin that does this for free, because most business sites should not need a paid solution to be readable by AI.
What to do now
AI visibility matters now, not later. Companies that make their content easy for AI tools to read are already showing up in AI-driven conversations where buying decisions happen.
Start with three things. Check your page weight. If important pages exceed 300 to 500 KB of HTML, AI tools are likely losing content. Test your pages with an LLM by providing the URL or raw HTML and asking it to extract your main heading. If it struggles, so will AI crawlers. And consider markdown delivery, whether through Cloudflare, or a custom solution.
One thing is certain: the internet is moving toward agentic workflows, and that trend will only accelerate. The more a website, service, or application is built to integrate with language models, the more value its owners will get from it. Over time, businesses will pay attention to how visible their sites are inside LLMs, because that is where their customers are going.
Build your business where people are. More and more of them are using ChatGPT and similar tools to research, compare, and buy. Increasing your presence in large language models is just the natural next step.
If you need help, we offer AI Web Enhancements or Technical GEO audits at coditive.ai.
