Building the AI-Native Web with EDS, llms.txt

Jeremy Howard's llms.txt is a proposed standard https://llmstxt.org/#proposal for helping LLMs understand and work with websites and new software at inference time, particularly for projects/tools released after an LLM's training cutoff date, it is a better guide to your site than sitemap.xml; it allows the content creator to decorate and enhance the index to your site or API.

This isn't just about technical specifications—it's about creating websites that serve both human readers and AI systems effectively.

Adobe's Edge Delivery Services (EDS) is at the cutting edge of content delivery, staying ahead of the curve by implementing native support for the latest and most innovative standards. This commitment to staying at the forefront of industry trends allows EDS to provide superior service and ensure that its customers have access to the most advanced solutions available.

As a part of my ongoing research, I have looked into the new llms.txt standard, including insights on security best practices. You can read my findings at the following link: https://allabout.network/blogs/ddt/creating-an-llms-txt.

Beyond the information provided in that document, this guide will describe the practical aspects of utilizing llms.txt with EDS. By understanding how these elements work together, you can optimize your content delivery strategy and maximize the benefits of using EDS.

Making AI-Ready Content Simple with EDS

The path to AI-optimized content is already built into Edge Delivery Services. Every page on your EDS-powered site automatically has a markdown version—simply append .md to any URL. This native capability, combined with llms.txt implementation, creates a powerful foundation for AI-ready content delivery.

Understanding llms.txt: The Foundation

Think of llms.txt as a digital welcome mat for AI visitors to your website. Just as robots.txt guides crawlers and sitemap.xml directs search engines, llms.txt helps AI systems navigate and comprehend your site's content. The standard's power lies in its dual approach: a root-level llms.txt file and markdown versions of web pages.

The AI-Ready Trifecta: Powered by EDS

Edge Delivery Services makes implementing the three core components of AI-ready content straightforward:

llms.txt at your site's root - easily managed through EDS's content structure
Markdown versions of pages - automatically generated by EDS
Schema.org integration - seamlessly indicating llms.txt support

This combination creates a semantic layer that both humans and AI can navigate with ease, all powered by EDS's robust infrastructure.

Structured Data: Speaking AI's Language

Modern websites need to be bilingual - speaking both human and machine languages. The llms.txt standard can be indicated through simple Schema.org markup:

{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "ai-standard": "llms.txt",
  "ai-lastUpdated": "2024-11-27"
}

This structured data helps AI systems identify llms.txt-enabled websites efficiently.

Extending EDS: AI-Native Error Handling

While the llms.txt standard provides a foundation for AI-web interaction, I propose extending Edge Delivery Services' error handling to better serve both human visitors and AI systems.

Here's my proposed enhancement to 404.html

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <meta name="llms-section" content="/llms.txt">
        <meta name="ai-error-code" content="404">
        <link rel="alternate" type="text/markdown" href="/error.md">
        <title>404 Not Found</title>
        <style>
            body {
                display: flex;
                justify-content: center;
                align-items: center;
                height: 100vh;
                margin: 0;
                font-family: Arial, sans-serif;
                background-color: #f0f0f0;
            }
            .container {
                text-align: center;
            }
            h1 {
                font-size: 3em;
                margin: 0;
            }
            p {
                font-size: 1.2em;
                color: #666;
            }

            a {
                color: #007BFF;
                text-decoration: none;
            }
            a:hover {

                text-decoration: underline;
            }
        </style>
    </head>
    <body>
        <div class="container">
            <h1>404</h1>
            <p>Page Not Found</p>
            <p><a href="/">Go back to Home</a></p>
            <!-- Machine-readable info pointing to llms.txt -->
            <script type="application/json" data-ai-error hidden>
            {
                "errorCode": 404,
                "suggestedAction": "/llms.txt"
            }
            </script>
        </div>
    </body>
</html>

When a 404 error occurs, humans continue to see EDS's clean, user-friendly error page with clear navigation options. Obviously you will have restyled the 404.html to meet your clients aspirations, adding these meta elements is simple

My proposed extension adds parallel AI-specific handling through two key mechanisms:

A llms-section meta tag that directs AI systems to the llms.txt file for standardized handling instructions

 <meta name="llms-section" content="/llms.txt">
 <meta name="ai-error-code" content="404">

Structured machine-readable error data that provides immediate context and guidance

      <!-- Machine-readable info pointing to llms.txt -->
      <script type="application/json" data-ai-error hidden>
        { 
           "errorCode": 404,
           "suggestedAction": "/llms.txt"
         }
       </script>

This proposed enhancement:

Maintains EDS's excellent human user experience
Adds standardized AI guidance
Leverages existing llms.txt implementation
Creates clear paths for both human and AI visitors

Why EDS is Your AI-Ready Platform

Edge Delivery Services isn't just supporting these standards—it's helping define them:

Native markdown support built into the platform
Automatic content optimization for both human and AI consumption
Seamless Schema.org integration
Built-in performance optimization
Global edge presence for faster AI interaction

Implementation Guide with EDS

Starting your AI-ready journey with Edge Delivery Services:

Deploy your content through EDS
Add llms.txt to your root
Leverage automatic markdown versions
Implement standard Schema.org markup

The platform handles the complexity, letting you focus on creating great content.

Key Benefits

Improved AI Understanding: Clear indication of llms.txt support helps AI systems better interpret your content
Standardized Format: Simple, markdown-based structure ensures consistent interpretation
Integration Ready: Works alongside existing standards like robots.txt and sitemap.xml
Error Handling: Built-in support for 404 errors and API failures

Business Impact

Organizations adopting these standards with EDS need to consider:

Hiring AI expertise
Updating content management processes
Implementing structured data strategies
Creating new roles like "AI Evangelist"
Optimizing sites for AI

Bridging Robots.txt and llms.txt: A Unified Approach

The implementation of llms.txt marks an important step toward creating AI-ready websites, but we must carefully consider how this new standard interacts with existing web protocols. For decades, robots.txt has served as the primary guide for web crawlers. Now, it can evolve to create a seamless bridge to AI interaction.

Traditional robots.txt files focus solely on basic crawler access control through simple directives:

User-agent: *
Disallow: /drafts/

However, in an AI-ready website, robots.txt can serve as the first point of contact for both traditional crawlers and sophisticated AI systems. Our proposed enhancement creates a clear hierarchy of machine-readable resources:

User-agent: *
Disallow: /drafts/
# AI-Documentation: /llms.txt
# Block specific LLM crawlers if desired
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
Allow: /llms.txt
Allow: /*.md$
Sitemap: [yoursitemap.xml]

This enhanced structure achieves several crucial objectives. It maintains backward compatibility with traditional web crawlers through standard directives, ensuring existing systems continue to function as expected. The new (commented out) AI-Documentation directive creates a clear path for AI systems to discover AI-ready features, acting as a signpost to the llms.txt file. Perhaps most importantly, it explicitly allows access to both the llms.txt file and markdown versions of pages, ensuring AI systems can access these resources even when other content access is restricted.

Think of this enhanced robots.txt as a sophisticated traffic controller for machine visitors. While traditional web crawlers follow familiar disallow rules, AI systems discover paths to richer interaction through the AI-Documentation directive. The explicit allowing of markdown files (/*.md$) ensures AI systems can access clean, structured content optimized for their consumption.

API Errors

You may have an API to serve your EDS as well as the content in your EDS Bus, if you do you should add AI Error handling there too.

API 404 Error Handling Implementation

# Enhanced Nginx configuration for contextual error handling, 
# Add header to 404 page -- other patterns for other CDNs

location @llms_fallback {
try_files /llms.txt =404;
add_header Content-Type text/markdown;
add_header X-Content-Section "optional-details";
}

Server Side Access, provide both

// 404 handler - this should be after all other routes
app.use((req, res, next) => {
res.status(404)
.setHeader('X-llms-Section', '/llms.txt')
.setHeader('ai-error-code', '404')
.sendFile('/404.html'); // or render your 404 page
});

// General error handler should be last
app.use((err, req, res, next) => {
// Handle other errors
});

Implementation with Edge Delivery Services

Edge Delivery Services makes implementing this enhanced robots.txt straightforward through a simple process:

First, create your robots.txt file in your project root. Then include your traditional crawler directives, followed by adding the AI-Documentation directive pointing to your llms.txt. Finally, ensure markdown versions are explicitly allowed through the appropriate directives.

This creates a clear hierarchy for machine interactions, where robots.txt provides initial access rules and AI guidance, then AI systems are directed to llms.txt for detailed interaction rules, while markdown versions remain accessible for efficient processing.

The result is what we call a "structured AI interaction pathway." While general crawling might be restricted, AI systems can still access:

The llms.txt file, providing structured guidance about your content
Markdown versions of pages, offering clean, semantic content
Your sitemap, helping understand the overall content structure

The robots.txt architecture naturally supports this hierarchical access control, allowing you to maintain different rules for different types of machine visitors while ensuring your content remains optimally accessible through your chosen channels.

This approach represents a fundamental shift from simple binary access control (allow/disallow) to a more nuanced system where website owners can guide AI systems to interact with their content through structured channels. It's comparable to creating a specialized entrance for AI systems that ensures they interact with your content as intended, while maintaining traditional access patterns for conventional web crawlers.

Looking Ahead

As AI continues to evolve, websites that embrace structured data and semantic markup will be better positioned to serve both human and AI users effectively. Edge Delivery Services provides the ideal platform for this transformation, offering built-in support for the features that define AI-ready content.

When AI searches and agents begin to overtake human use of the web, EDS customers will already be prepared. The question isn't whether to make your content AI-ready, but how quickly you can leverage EDS to get there. This technique is perfect for brand new websites.

Particularly valuable for new projects not in LLM training data

Helps with documentation and library usage

Improving AI-assisted development workflows

Enabling more autonomous AI tool usage

Its early days for llms.txt;It was proposed in September 2024, ChatGPT’s cutoff is October 2023 - Antropic is May 2024; so there is a gap between the standard , the AI’s training and now.

Lars Trieloff – Principal – Adobe | LinkedIn confirms that there are no hits against llms.txt in EDS as of publishing date (Jan 2025); The infancy of generative AI search adoption can be attributed to several factors. These include the need for further technological advancements, user hesitations due to unfamiliarity with the technology, and the time it takes for businesses and individuals to adapt to new search paradigms. AI changes rapidly, faster than humans comprehend, and as this barrier will drop and more users will switch to AI search; EDS helps you get your website ready.

Notes

Suggestions made for robots.txt, meta keywords, and headers are to guide AI.
These suggestions are not part of the official standards.
Monitor the llms.txt specification for updates and adjust implementation accordingly.

Resources

Original Proposal
Discord Channel
The /llms.txt file, helping language models use your website
llms.txt directory
https://allabout.network/blogs/ddt/creating-an-llms-txt

What’s the impact of the new Robot-First Web? — Boye & Company

Thanks for reading.

Digital Domain Technologies provides expert Adobe Experience Manager (AEM) consultancy. We have collaborated with some of the world’s leading brands across various AEM platforms, including AEM Cloud, on-premise solutions, Adobe Managed Services, and Edge Delivery Services. Our portfolio includes partnerships with prominent companies such as Twitter (now X), EE, Nissan/Renault Alliance Ford, Jaguar Land Rover, McLaren Sports Cars, Hyundai Genesis, and many others.

/fragments/ddt/proposition

style

bg-dark