add llms.txt to repo

[sarahsanders-docker]

Description

• Adds .md versions of manuals and guide pages
• Adds site-wide llms.txt file to footer (this is essentially a sitemap that LLMs can crawl)
• Adds options for llms/AI agents to better crawl docs, users to ask about docs in Docs AI
• Preview: https://deploy-preview-22298--docsdocker.netlify.app/admin/organization/onboard/

Related issues or tickets

https://docker.atlassian.net/browse/ENGDOCS-2454

Reviews

• Technical review
• Editorial review
• Product review

[netlify]

✅ Deploy Preview for docsdocker ready!

Name	Link
🔨 Latest commit	5f84022
🔍 Latest deploy log	https://app.netlify.com/sites/docsdocker/deploys/67e5635e71a7d0000836467c
😎 Deploy Preview	https://deploy-preview-22298--docsdocker.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[craig-osterhout]

It's looking good.
One blocker: The button appears on top of the header bar when scrolling down.

Some nits to consider, but not blockers:
The button text is "Copy page", but only one option is about copying. Maybe call it "Page options" or something more umbrella-y.
Can the copy success be more like the codeblock copy and not a blocking modal.
The bold text in the dropdown items feels too strong and is different than the left-nav dropdown component.
The carrot is a different style than the left-nav dropdown component.
The description text seems redundant for each item and can maybe be removed.
The verb in "Search Docs AI" is inconsistent with "Ask AI". Also the description text uses a different verb, "Search Docs AI" vs. "Ask question with Docs AI".
If intended for LLM scrapers, does it need to be so prominent? To be blunt, if the use-case is for LLM scrapers and based on many of the proposed standards, it feels like this should just be an /llms.txt at the root and be invisible to users, like our /sitemap.xml and /robots.txt.

[sarahsanders-docker]

• https://deploy-preview-22298--docsdocker.netlify.app/admin/organization/onboard/

all feedback addressed!

I had been a bit confused by the standard when we discussed in Docs huddle. I first though llms.txt was supposed to be at the page level. But after doing some more digging, the standard proposes that:

• llms.txt is one file that is reserved for scraping the entire site (see: https://docs.stripe.com/llms.txt), but some sites (Stripe, Cloudflare) have this visible, not hidden like robots.txt. Other sites have it hidden. There isn't a real standard for whether this should be hidden or available somewhere on the site. I guess this should be a discussion whether we expose this file (I have it in the footer right now) @usha-mandya @aevesdocker any strong feelings on whether to keep the llms.txt file in the footer exposed, or hide it?

the individual page level copy as markdown/view plaintext/ask AI options are so that actual users can copy the markdown files into an LLM of their choice. this is not part of the llms.txt standard, but rather a user experience/developer experience improvement in general. I am in strong support of giving users the flexibility to use docs in the LLM of their choice (by copying the markdown file or viewing in plaintext/copying).

[aevesdocker]

I wonder if you want to make the option description the actual option, otherwise it's a bit repetitive and is a lot of text all in that little dropdown. You could make it more efficient with:

• Copy page as Markdown for LLMs
• View page as plain text
• Ask questions with Docs AI

What would that look like?

Re the llm.txt file in the footer - no strong feelings. Only thought is, do we look 'hip and cool and with it' if we expose it? 😎

[aevesdocker]