Add agent guidelines and static llms.txt

[CoderOMaster]

Add AGENTS.md, CLAUDE.md, and serve a static llms.txt in the Next.js public folder. Also add an HTML head discovery tag for llms-txt.

Description

This PR introduces tooling guidelines and discovery configs for AI coding assistants working in the repository:

1. AI Agent Guidelines: Added AGENTS.md and CLAUDE.md to establish development rules (such as enforcing nativelink-util::task wrappers, signed commits, and formatting conventions).
2. Static llms.txt: Added a curated static /llms.txt file inside the Next.js docs public folder containing a high-level overview, architecture mappings, JSON5 configurations, and recent changelog highlights.
3. Metadata Discovery: Added an HTML discovery <link rel="llms-txt" href="/llms.txt" /> tag to the global Next.js page layout to allow automated AI crawlers to discover the context file.

Fixes #2508

Type of change

• This change requires a documentation update

How Has This Been Tested?

• Ran pre-commit run -a inside the nix develop environment. All formatting, coding structure, and Vale spelling/spelling-exclusion checks passed successfully.
• Verified that the static /llms.txt file and discovery header metadata are compiled and served correctly by Next.js.

Checklist

• Updated documentation if needed
• Tests added/amended
• bazel test //... passes locally
• PR is contained in a single commit, using git amend

---

This change is 

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview, Comment	Jul 10, 2026 11:09am
nativelink-aidm	Ready	Preview, Comment	Jul 10, 2026 11:09am

[palfrey]

It's an interesting idea, but has the problem that there's a lot of things here we'd have to keep manually up-to-date and will inevitably drift otherwise, and in that case it'd be better for the LLMs to refer to the up-to-date docs, not this file.

If you can make the various items I've flagged here generated rather than hardcoded, this becomes significantly more useful!

[palfrey]

Suggested change

	NativeLink speaks the standard Remote Execution API, so it works with any REAPI client:
	NativeLink speaks the standard Remote Execution API, so it works with any REAPI client, including:

[palfrey]

web/apps/docs/scripts/gen-changelog.mjs shows how we've done existing changelog generation. Worth cribbing from there.

[CoderOMaster]

will check!

[palfrey]

This should really just refer to web/README.md as that's the canonical info on this

[CoderOMaster]

the whole agents.md or just these lines?

[palfrey]

The set of items under Documentation site

[CoderOMaster]

@palfrey let me have a check and will get back to you very soon on automatic updates for llms.txt rather than hardcode, i agree would be tiring to maintain it lol

[CoderOMaster]

@palfrey @MarcusSorealheis folks i am working on automating it via these:

1. Made web/apps/docs/public/llms.txt fully generated by a new script scripts/gen-llms-txt.mjs (wired into dev/build), so it can't drift only a short editorial header stays hardcoded.

2. The version/date come from CHANGELOG.md, the sample config + examples list from nativelink-config/examples/, and the docs table of contents from the content/docs tree; the changelog is now a link instead of copied bullets so that will hard coded dependency here

3. The store list is derived from the Rust StoreSpec enum via build-schema, cached to a committed lib/store-specs.json snapshot so normal builds need no cargo (refresh it with gen:llms-txt -- --schema in the Nix shell).

4. public/llms.txt is now a gitignored build output (like the generated changelog page); store-specs.json and the script are the committed inputs.

Why store-specs.json is committed: the store list can only be extracted from the Rust StoreSpec enum by compiling it (cargo run --bin build-schema), which needs the Nix dev shell but the docs site builds on Vercel/CI, where there's no Rust toolchain. If the snapshot weren't committed, every docs build would have to compile a Rust crate, which Vercel simply can't do i think. So we commit the snapshot as a build input: it's regenerated occasionally in the Nix shell (gen:llms-txt -- --schema, only when StoreSpec changes), and consumed on every toolchain-free web build.

[CoderOMaster]

store-spec.json is only complied using (--schema header below)
nix develop -c bun --filter @nativelink/docs gen:llms-txt -- --schema

That's the only path that compiles the nativelink-config crate (build-schema) and re-derives the list from StoreSpec. It's a manual step, requires the Nix dev shell + cargo, and isn't triggered by ordinary repo changes. And can go stale If someone adds or edits a store in StoreSpec and doesn't run --schema..so i was think to Add a CI check in the Nix lane ?

[palfrey]

store-spec.json is only complied using (--schema header below) nix develop -c bun --filter @nativelink/docs gen:llms-txt -- --schema

That's the only path that compiles the nativelink-config crate (build-schema) and re-derives the list from StoreSpec. It's a manual step, requires the Nix dev shell + cargo, and isn't triggered by ordinary repo changes. And can go stale If someone adds or edits a store in StoreSpec and doesn't run --schema..so i was think to Add a CI check in the Nix lane ?

The way we've done ones like this before is demoed as generate-stores-config in https://github.com/TraceMachina/nativelink/blob/main/tools/pre-commit-hooks.nix#L190. That command regenerates the stores-config.json5 from comments in stores.rs, primarily so we can then shove the config through the examples testing code. This will run both in CI and locally with pre-commit.

Something like this at the end should do it:

  # Regenerate store-spec.json for llms-txt
  generate-store-spec = {
    enable = true;  
    entry = let
      script = pkgs.writeShellScriptBin "generate-store-spec" ''
        set -eu
        ${pkgs.nodejs}/bin/node web/apps/docs/scripts/gen-llms-txt.mjs --schema
      '';
    in "${script}/bin/generate-store-spec";
    name = "generate-store-spec";
    files = "nativelink-config/src/stores.rs|web/apps/docs/lib/store-specs.json";
    pass_filenames = false;    
  };

[palfrey]

Like the improvements, this is looking good, but needs a few more things

[palfrey]

The version and MSRV here are hardcoded and that should be fixed.

[CoderOMaster]

agents.md is usually hardcoded with info though. it doesnt have to be autoupdated like llms.txt as far as i have seen. else we can remove version and MSRV from it then

[CoderOMaster]

oh just saw pre commit failing, folks let me get back on how to fix this.

[palfrey]

pre-commit was failing due to network issues. I've just redone things to run this via Bazel which I think will help

[palfrey]

This is a pretty tricky one. So the pre-commit environment under nix doesn't permit any downloads, so bazelisk breaks. To fix this, we need the nix bazel 9, but that's not present in our current nixpkgs. #2537 will fix that though, so just need to go through review of that.

[CoderOMaster]

okay sounds good. I have been trying to understand lol but wasn't able to figure it out lol :)
Once 2537 is merged i can update my branch accordingly then