Markdown Content Negotiation

Why It Matters

Retrieval systems and agents that fetch live pages often parse HTML, which adds boilerplate, layout noise, and ambiguity. Serving Markdown when requested reduces extraction errors, aligns headings and lists with logical document structure, and can improve passage quality for models that cite or summarize your content.

How It Works

The client sends an `Accept` header listing supported media types and q-values. The server or edge layer matches the best available variant, sets `Content-Type` to the chosen format (for example `text/markdown; charset=utf-8`), and returns the body in that format. Correct caching usually requires a `Vary: Accept` response header so intermediaries do not serve the wrong variant. Markdown may be authored natively, generated from a CMS, or converted from a structured source document.

Use Cases

• Documentation sites that return Markdown to developer tools and HTML to humans from one URL
• Blogs or knowledge bases exposing agent-consumable text for AI assistants and research workflows
• APIs and gateways that normalize web pages to Markdown for downstream RAG or monitoring pipelines
• Reducing custom scraper logic by offering a first-class text format instead of DOM heuristics

Best Practices

• Advertise Markdown support consistently and document the expected `Accept` values for integrators
• Return accurate `Content-Type` and charset metadata and use `Vary: Accept` when multiple representations exist
• Keep Markdown faithful to the authoritative meaning of the page (titles, links, code blocks, tables)
• Ensure authentication, paywall, and rate-limit rules apply equally to negotiated variants
• Test with real agent and crawler `Accept` headers to confirm the server selects the intended variant