Feature Request: Native Support for Serving Markdown to LLMs Based on Accept Headers

[faiz9068]

Body

Problem Statement

LLM agents and AI crawlers are becoming a significant portion of web traffic. These agents don't need HTML markup, CSS, or JavaScript - they only need semantic content. Currently, serving full HTML to LLMs results in:

• 10x higher token costs for AI companies crawling sites
• Reduced crawl frequency due to higher costs
• Less content being included in training datasets
• Worse performance in AI-powered search and assistants

Proposed Solution

Add native support in Astro and Starlight to automatically serve markdown versions of pages when the Accept header indicates preference for text/plain or text/markdown over text/html.

Example behavior:

# Returns markdown version
curl -H "Accept: text/markdown" https://example.com/blog/post

# Returns HTML version (default)
curl https://example.com/blog/post

Implementation Ideas

1. Build-time generation: During astro build, automatically generate parallel markdown versions of all HTML pages
2. Middleware support: Built-in middleware that detects Accept headers and serves appropriate format
3. Configuration options in astro.config.mjs:

export default defineConfig({
  output: 'static',
  markdown: {
    servePlaintext: true, // Enable markdown serving
    plaintextPriority: ['text/markdown', 'text/plain'], // Accept header priority
    excludePaths: ['/admin/*'], // Paths to exclude
  }
});

Benefits

• GEO (Generative Engine Optimization): Better visibility in AI search results
• Reduced infrastructure costs: Lower bandwidth for AI crawlers
• Better accessibility: Cleaner content for screen readers and text-based browsers
• Future-proof: Prepare for an AI-first web ecosystem
• Developer experience: Zero-config solution vs current complex workarounds

Prior Art

• The Bun documentation site implements this and reported 10x token reduction
• Several developers are independently implementing similar solutions

Community Impact

This feature would position Astro as the first major SSG with native LLM optimization, potentially driving adoption among:

• Technical documentation sites
• Blogs and content-heavy sites
• Companies focused on AI discoverability
• Developers building for the AI-powered web

[re-xyr]

The idea of serving different versions of the page depending on the Accept header is nice, but I don't think your concept is implementable:

• Middlewares don't run for statically rendered pages.
• Even if they do, there is no 1:1 correspondence between pages and Markdown files, and therefore no way for the framework to do this automatically for all pages.
• Ignoring that, MDX can contain server-side code which you do not necessarily want the user to see.

As an alternative, just SSR your route, and when the Accept header requests plaintext, redirect to an endpoint that returns the Markdown source.

[SanariSan]

• Middlewares don't run for statically rendered pages.

Serving static files can always be configured be it edge function or a custom nginx instance. I'd say main problem is the second bullet of yours. Although, generating MD or even plain text based on semantics of prerendered html sounds pretty real to me, but requires devs to pay attention to tags so the output is structured properly.

[jasikpark]

Yeah, this seems more likely something that could get fleshed out in a blog post or an external plugin, if someone wanted to build that. Maybe a starlight plugin?

[ematipico]

Just to let you know, internally, we already started discussing how Astro can provide such content for LLMs.

We aren't ready for a proposal yet because we want to frame where and how the feature will stand inside the Astro code base e.g. scope of the feature, core VS integration, what about MDX, etc.

There's a lot to consider, but rest assured that it's on our radars