Provide entire docs in a single text file for LLMs.

[lucasguellis]

Solves #958

This PR introduces support for the LLM-ready website standard to improve how language models (LLMs) access and suggest information about ReScript code.

The following files are now generated via a script, for both ReScript language and ReScript React package:

• llm-full.txt: Contains the full documentation, including all markdown files from pages/docs/manual and pages/docs/react, along with examples.
• llm-small.txt: A compressed version of llms-full.txt, excluding examples, to provide a more concise reference.
• llms.txt: Provides metadata to inform LLMs about the available files for reference.

As an example of implementation, this aligns with the approach used by the Svelte documentation.

The script runs on build with the existing update-index script, but can be used alone with:

npm run generate-llms

The files are generated separately for each documentation version, ensuring LLMs can reference version-specific information, so the final files should look like this:

Also, the llm.mdx file is generated by the script for each version separately, as the links to the files depends on the version number. This automation ensures the links are always accurate and eliminates the need for manual updates with each new release.

[vercel]

@lucasguellis is attempting to deploy a commit to the ReScript Association Team on Vercel.

A member of the Team first needs to authorize it.

[fhammerschmidt]

Great work! Let's merge this and see whether it improves the LLM's knowledge of ReScript in the future.

Thank you very much!