Skip to content

feat(iri): MCP server + skill card for ALCF IRI tools - #199

Open
JinchuLi2002 wants to merge 14 commits into
argonne-lcf:mainfrom
JinchuLi2002:jinchu/iri-mcp-and-skill
Open

feat(iri): MCP server + skill card for ALCF IRI tools#199
JinchuLi2002 wants to merge 14 commits into
argonne-lcf:mainfrom
JinchuLi2002:jinchu/iri-mcp-and-skill

Conversation

@JinchuLi2002

@JinchuLi2002 JinchuLi2002 commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Follows up merged PR #173:

  • MCP server for the ALCF IRI Facility API: chemgraph.mcp.alcf_iri_mcp — 43 flat tools, one per endpoint, launchable via python -m chemgraph.mcp.alcf_iri_mcp (stdio) or --transport streamable_http --port 9010. Wraps the same alcf_iri_core implementation as single_agent_iri, so auth / refresh / unsafe-write gating / error surface are identical. Any MCP-speaking client (Claude Desktop, main_agent's MCP wiring, other agent frameworks) can now reach the IRI API without going through LangChain.
  • Skill card at src/chemgraph/skills/alcf_iri.md — first entry in a new skills/ directory, following the Anthropic Skills convention (YAML frontmatter + when-to-use + auth + example flows + failure modes). Intended for main_agent-style skill routing.

Also folds in one small fix carried over from #173 review:

  • Variant-aware auth reprompts in alcf_iri_core.py: _start_reauth and the 401 recovery hint now emit tool names that match whichever wrapper module is on the call stack (flat vs category). Was producing category-flavoured instructions ("invoke alcf_auth with action='complete_reauth'") under the flat variant, which caused gpt-4o to hallucinate "session expired" instead of calling alcf_auth_complete_reauth(auth_code=...).

Files

Kind Path
MCP server src/chemgraph/mcp/alcf_iri_mcp.py (new)
Skill card src/chemgraph/skills/alcf_iri.md (new dir + file)
Fix src/chemgraph/tools/alcf_iri_core.py
Docs docs/mcp_servers.md, examples/iri/README.md

Test plan

  • python -c "from chemgraph.mcp import alcf_iri_mcp; ..." — server module imports cleanly, registers 43 tools with the expected names (alcf_facility_get, alcf_compute_submit_job, alcf_auth_start_reauth, alcf_auth_complete_reauth, ...).
  • Live call to public /status/resources endpoint via mcp.call_tool('alcf_status_list_resources', {}) returns the same payload as the LangChain flat tools and category tools (HTTP 200, 9 resources, 4 up: Sophia, Crux, Aurora, Polaris).
  • Per-tool JSON schemas match params_schema in alcf_iri_core.CATEGORIES — required fields marked required, optional fields typed Optional[T] with default: null.
  • Variant detection unit-tested against synthetic stack frames simulating both wrapper modules; returns correct value and falls back to category when neither module is present.
  • End-to-end MCP handshake via main_agent — deferred until main_agent's MCP wiring for IRI lands; nothing here should require changes there.

Open questions for review

  • The skills/ directory format is a first pass following the Anthropic Skills convention. Happy to rename frontmatter fields (mcp_server, tool_prefix) or restructure entirely to whatever main_agent ends up expecting.
  • MCP server defaults to port 9010 for streamable_http. If there's an existing port-allocation table for ChemGraph MCP servers, easy to move.

🤖 Generated with Claude Code

JinchuLi2002 and others added 4 commits August 20, 2026 13:05
@JinchuLi2002 @claude
fix(iri): variant-aware auth reprompts (flat vs category)
e59c86c 
_start_reauth and the 401 recovery hint used to always emit
"invoke alcf_auth with action='complete_reauth'..." -- correct for
the category tool set but wrong for flat, where the tool is
alcf_auth_complete_reauth(auth_code=...). Under flat, gpt-4o would
read the mismatched instruction, fail to find the named tool, and
either hallucinate "session expired" or restart the whole reauth
loop.

Walk the call stack to detect which tool-wrapper module is on it,
then emit the matching phrasing. Default is category when neither
module is on the stack (e.g. bare-CLI use).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
feat(mcp): ALCF IRI Facility API MCP server
f02475b 
Wraps the same 43 flat tools shipped by single_agent_iri as a
standalone MCP server (chemgraph.mcp.alcf_iri_mcp) so any MCP-speaking
client -- Claude Desktop, other agent frameworks, or main_agents MCP
wiring -- can reach the ALCF IRI API without going through LangChain
directly.

- Reuses chemgraph.tools.alcf_iri_core.CATEGORIES verbatim; the MCP
  file is a thin adapter that builds per-action wrapper functions with
  the right signatures and registers them via FastMCP.add_tool.
- Tool naming matches the LangChain flat set (alcf_<category>_<action>)
  so agents that switch backends can reuse the same tool references.
- Auth, refresh, unsafe-write gating, and error surface are inherited
  from alcf_iri_core -- no divergence to maintain.
- Default port 9010 for streamable_http.

Verified end-to-end against the live public /status/resources endpoint
(no token required): list_tools returns 43 entries with correct JSON
schemas, and alcf_status_list_resources returns the same payload the
category tools and LangChain flat tools already return.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
feat(skills): ALCF IRI capability card for main_agent
e69e480 
Creates src/chemgraph/skills/ (new dir) with alcf_iri.md, a
capability card the main_agent workflow can read to decide when to
route a query at the ALCF IRI tool set.

Format follows the Anthropic Skills convention: YAML frontmatter
(name, description, mcp_server, tool_prefix) plus markdown sections
for when-to-use, auth, safety, tool naming, example flows, and
common failure modes. This is a first pass; the exact fields are
open to renaming once main_agent settles on a schema.

Content is skill-router-oriented (not developer-oriented like the
opencode SKILL.md): tells the agent what problems this skill
handles, what problems it does NOT handle, the auth prerequisites
it enforces, and the multi-hop patterns that come up most often.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
docs(iri): document the new MCP server and skill card
0bdea7f 
- docs/mcp_servers.md: adds an ALCF IRI section under "Specialized
  servers" with the launch commands, auth summary, and cross-refs
  to the LangGraph workflow and the skill card.
- examples/iri/README.md: appends a short "Also available as an MCP
  server" section so readers who land on the LangGraph example
  discover the MCP path (and vice versa).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
feat(mcp/iri): support both flat and category variants; default flat
ed95b63 
MCP server now mirrors the LangGraph workflow's flat-vs-category
split. Selection precedence: CLI --variant > $CHEMGRAPH_IRI_MCP_VARIANT
> "flat". The env var exists for MCP client configs that pass env
but not argv (e.g. Claude Desktop mcpServers entries).

- register(variant) is public so embedders that import the module
  can pick either shape after construction.
- Default (flat) is registered at import time so plain
  from chemgraph.mcp import alcf_iri_mcp still yields a populated
  server, matching the previous behaviour of this file.
- CLI path re-parses --variant and only rebuilds the tool set if it
  differs from what import time chose; FastMCPs tool store is not
  part of its public API so we swap in a fresh ToolManager rather
  than mutate.

Verified end-to-end: flat registers 43 tools (alcf_facility_get, ...,
alcf_task_list), category registers 7 dispatchers (alcf_facility, ...,
alcf_auth), env var and CLI flag both work, --variant is stripped
from sys.argv before run_mcp_server sees it.

Docs (docs/mcp_servers.md, examples/iri/README.md, skill card)
updated to describe both variants and how to pick between them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002
JinchuLi2002 marked this pull request as ready for review August 20, 2026 19:46
tdpham2
tdpham2 reviewed Aug 20, 2026
View reviewed changes
Comment thread src/chemgraph/skills/alcf_iri.md

@tdpham2 tdpham2 Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you make an additional skill.md file that is aimed for a coding agent to launch bash command (curl) to use the IRI, not describing the MCP.

Please test it with Claude code to make sure it works well based on the benchmark you created earlier. For the benchmark scoring, make it score either 0 or 1, so that the style doesn't matter, only final answer matters.

I will try to give you a deepagent graph for you to test (similar to Claude code) for you to test next week with the skills.

@tdpham2 tdpham2 Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JinchuLi2002 Just want to follow up if you have the skills ready. Thanks!

@JinchuLi2002 @claude
feat(skills): bash/curl skill card + Claude Code benchmark harness
4ea54fc 
Followup on PR argonne-lcf#199: coding agents whose runtime is bash+curl
(not the ChemGraph MCP server) also need an ALCF IRI skill card. The
target audience is Claude Code, or a forthcoming deepagent-style
graph. This adds:

- src/chemgraph/skills/alcf_iri_bash.md -- curl-oriented capability
  card. Same when-to-use / auth / safety framing as alcf_iri.md, but
  the endpoint recipes are runnable bash (curl + jq) rather than MCP
  tool names. Covers all 43 IRI actions grouped by category, with
  multi-hop patterns (resolve names to UUIDs, paginate list_jobs) and
  identity-picker guidance for the Globus login flow.

- examples/iri/bench_claude_code.py -- headless-Claude-Code harness.
  Loads the skill via --append-system-prompt, restricts tools to Bash,
  runs each of the 15 qeval questions, and writes a JSONL with the
  answer + full tool-call trace + wall time. Designed to feed the
  notebooks binary judge cell so Claude+curl and single_agent_iri can
  be compared under identical scoring.

- examples/iri/README.md -- documents both.

The notebooks binary-judge cell itself lives in the untracked
notebooks/iri_qeval.ipynb (per PR argonne-lcf#173 policy that notebooks are
personal analysis, not shipped). The judge system prompt is:

  Return 1 if the final answer factual claims match the trace,
  else 0. Ignore formatting and style. Ignore verbosity.
  Pagination gotcha: reporting N from a single page of ~100 is 0.

Same rubric idea, style-blind, so cross-runtime comparisons are fair.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
JinchuLi2002 added a commit to JinchuLi2002/ChemGraph that referenced this pull request Aug 25, 2026
@JinchuLi2002 @claude
feat(skills): bash/curl skill card + Claude Code benchmark harness
de8f7c0 
Per PI followup on PR argonne-lcf#199: coding agents whose runtime is bash+curl
(not the ChemGraph MCP server) also need an ALCF IRI skill card. The
target audience is Claude Code, or PIs forthcoming deepagent-style
graph. This adds:

- src/chemgraph/skills/alcf_iri_bash.md -- curl-oriented capability
  card. Same when-to-use / auth / safety framing as alcf_iri.md, but
  the endpoint recipes are runnable bash (curl + jq) rather than MCP
  tool names. Covers all 43 IRI actions grouped by category, with
  multi-hop patterns (resolve names to UUIDs, paginate list_jobs) and
  identity-picker guidance for the Globus login flow.

- examples/iri/bench_claude_code.py -- headless-Claude-Code harness.
  Loads the skill via --append-system-prompt, restricts tools to Bash,
  runs each of the 15 qeval questions, and writes a JSONL with the
  answer + full tool-call trace + wall time. Designed to feed the
  notebooks binary judge cell so Claude+curl and single_agent_iri can
  be compared under identical scoring.

- examples/iri/README.md -- documents both.

The notebooks binary-judge cell itself lives in the untracked
notebooks/iri_qeval.ipynb (per PR argonne-lcf#173 policy that notebooks are
personal analysis, not shipped). The judge system prompt is:

  Return 1 if the final answer factual claims match the trace,
  else 0. Ignore formatting and style. Ignore verbosity.
  Pagination gotcha: reporting N from a single page of ~100 is 0.

Same rubric idea, style-blind, so cross-runtime comparisons are fair.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
JinchuLi2002 and others added 7 commits August 25, 2026 11:02
@JinchuLi2002 @claude
feat(iri): run-all notebook for the Claude Code / bash-skill benchmark
d098295 
Wraps bench_claude_code.py in a self-contained Jupyter notebook so
reviewers can Run All instead of stitching together a 5-step manual
procedure across notebook + CLI + judge cell.

Also threads a --bare / bare=True flag through the harness so each
Claude Code subprocess runs isolated from ambient project state
(memory, CLAUDE.md, hooks, plugins). That gives the fair signal originally requested -- only what the skill file teaches the agent should count.

Notebook flow (Run All):
1. Preflight (claude/curl/jq on PATH, ALCF_API_TOKEN or on-disk cache,
   ANTHROPIC_API_KEY, harness importable).
2. Config (qids, trials, concurrency, bare, judge model, output paths).
3. Sweep: run_all() spawns one fresh `claude -p --bare` per (q, trial)
   with only Bash enabled and the skill appended to the system prompt.
   Live tqdm updates with per-question wall+cost.
4. Binary judge: AsyncAnthropic against claude-opus-4-20250514, strict
   0/1 style-blind rubric identical to the one in iri_qeval.ipynb.
5. Report: per-question pass table with turns/tokens/wall/cost + one
   representative failure rationale.
6. Drill-down: prints question, answer, and trace tail for each failing
   run so the skill can be iterated on cheaply.

Also fixes a text bug -- the benchmark has 16 questions (q1-q15 + q20),
not 15 as the earlier README claimed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
fix(iri/bench nb): argo-shim judge support + soft Claude Code auth check
ac2d966 
Two blockers for running the notebook without ANTHROPIC_API_KEY:

1. Preflight hard-asserted ANTHROPIC_API_KEY, but Claude Code accepts
   OAuth via `claude login` too. Softened to a warning -- if the
   sweep needs the key it will surface a clear Claude-Code error.

2. Binary judge was hardcoded to AsyncAnthropic + Anthropic model id.
   ALCF users typically dont have ANTHROPIC_API_KEY; they route
   through the argo shim (as iri_qeval.ipynb does). Rewired to use
   chemgraph.models.loader.load_chat_model which dispatches by model
   name prefix, so the same call site works for both.

New config cell exposes JUDGE_PROVIDER with auto-detect:
  argo -- if $ARGO_USER set or local proxy at 127.0.0.1:18085 responds
  anthropic -- if $ANTHROPIC_API_KEY set
Override with $BENCH_JUDGE_PROVIDER=argo|anthropic.

Judge model defaults follow the provider:
  argo      -> argo:claude-opus-4.7
  anthropic -> claude-opus-4-20250514

Argo path monkey-patches ARGO_LOCAL_OPENAI_MODEL_MAP with the current
claude-opus family entries (same lift as iri_qeval.ipynb) so the shim
recognises the judge model id.

README bumped with the auto-detect behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
fix(iri/bench): close subprocess stdin so claude -p actually runs
59f0d86 
Without stdin redirected, claude -p waits ~3s for piped input, warns
"no stdin data received in 3s, proceeding without it", and exits 1
without processing the prompt. Every question in the sweep failed
silently this way (16/16 wall=3s, no answer, no cost).

Set stdin=DEVNULL on the create_subprocess_exec call. Verified the
warning is gone and subprocesses proceed to actual inference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
fix(iri/bench): auto-disable --bare when no ANTHROPIC_API_KEY
2c52e52 
Claude Code's --bare explicitly disables OAuth keychain reads, so
running `claude -p --bare` without ANTHROPIC_API_KEY returns
"Not logged in · Please run /login" and exit 0 (with is_error=true
in the JSON). Previous sweep failed 16/16 with $0 cost and no traces.

Two changes:

1. Notebook cell 2 gates BARE on presence of ANTHROPIC_API_KEY.
   BARE_REQUESTED preserves user intent; effective BARE falls back
   to False with a printed NOTE explaining the tradeoff (ambient
   project state may leak into results). Set ANTHROPIC_API_KEY to
   get truly isolated skill-only measurement.

2. Harness now respects the JSON is_error field on claude subprocess
   output -- treats it as failure and surfaces the message as the
   row's error, so the JSONL is diagnostic instead of silently ok.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
fix(iri/bench): switch to stream-json for actual tool traces
92b924e 
claude -p --output-format json returns only {result, usage, cost,
num_turns} -- no message history. That made every runs trace_rendered
come back (no tool calls) regardless of whether Bash actually ran.
The binary judge then scored those as unsupported / hallucinated,
even when curl had genuinely executed and returned the correct
answer. Diagnosed by smoke-testing a raw curl on q1: sweep answer 9
matched the live API count, but trace was empty -> false-negative
fabrication verdict.

Switch to --output-format stream-json --verbose which emits one JSON
object per line, including tool_use and tool_result events. Rewrote
_render_trace to iterate the flat event list, still emitting the
CALL <tool>(args) / RESULT: <text> shape the judge expects. Also
grabs usage/cost/num_turns from the final result event.

Smoke q1 (bare=False, ALCF_API_TOKEN via refreshed nested-shape cache):
  ok: True
  answer: 9
  turns: 2  cost: $0.308
  trace: CALL Bash({...curl .../status/resources | jq length})
         RESULT: 9

Not pushing per user instruction.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
fix(iri/bench): raise CALL truncation 500 -> 2500 chars
b61aa16 
Compound shell scripts Claude Code emits for multi-hop questions
(env-var setup + curl + jq) run 1-2kB. Truncating at 500 chars cuts
off the actual jq filter, so the binary judge cant verify whether
the query was correct -- observed on q15 where the answer was
plausibly right but rationale said 'no visible query filtering
queued jobs and sorting by age.'

Not pushing per user instruction.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002 @claude
chore(iri): sanitize personal identifiers from PR
e174e56 
Pre-ship cleanup for reviewer-visible surfaces:

- src/chemgraph/tools/alcf_iri_core.py: change hard-coded
  jinchuli@alcf.anl.gov in the LLM re-auth prompts to
  <user>@alcf.anl.gov placeholder. This text goes to the LLM
  as guidance and shouldnt name any specific user.
- examples/iri/bench_claude_code.ipynb: change ARGO_USER default
  from a hard-coded name to empty string (still reads $ARGO_USER
  from env). Adds a warning when the argo judge path is selected
  without ARGO_USER set. Clears all cell outputs (they contained
  local paths and argo user identity from a prior run).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JinchuLi2002
JinchuLi2002 force-pushed the jinchu/iri-mcp-and-skill branch from 86dccf8 to e174e56 Compare August 25, 2026 17:21
@JinchuLi2002 @claude
style(iri/bench): satisfy ruff lint
d5bc67e 
- examples/iri/bench_claude_code.py: drop unused import subprocess
  (last use went away when we switched to asyncio.subprocess).
- examples/iri/bench_claude_code.ipynb: split multi-import in the
  preflight cell (ruff E401), and drop the redundant import asyncio
  from the judge cell (ruff F811 -- already imported in the sweep
  cell; ruff treats notebook cells as one logical file).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tdpham2 tdpham2 tdpham2 left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@JinchuLi2002 @tdpham2