diff --git a/README.md b/README.md index 0d3f4f5..98c007f 100755 --- a/README.md +++ b/README.md @@ -13,6 +13,8 @@ Each tool **provider** is a single YAML file under `tools/`. The YAML contains: - Per-tool input schemas, secrets, and auth metadata - Or a `package:` block to delegate to any existing MCP subprocess server — launched via `npx`, `uvx`, `python -m`, or any installed binary +- Or a `package:` + `repository:` pair to clone a git repo, run build commands, and + spawn the resulting stdio MCP server — useful for servers distributed only as source `server.py` loads every YAML at startup, installs declared `requirements` (pip packages), runs `setup_commands`, then registers each tool automatically — no Python files to @@ -107,6 +109,7 @@ Click **+ New Provider** and choose a provider type: |---|---| | **Python code** | Write `async def` functions; the UI lists the ones it finds as you type. Each becomes a tool entry. | | **Package** | Enter any command that launches a stdio MCP server (`npx`, `uvx`, `python -m`, or an installed binary). When you click **Next**, mcpproxy auto-introspects the command and pre-populates the tool list; if introspection fails you can still proceed and add tools by hand. | +| **Repository** | Provide a git URL and a list of build commands. mcpproxy clones the repo, runs the build commands, then introspects the resulting stdio MCP server. The URL and build commands are persisted in YAML so the repo can be re-cloned and re-built automatically on every container restart. | After the provider step, the wizard shows a **Secrets** step: any `secrets.env` entries in the provider are listed, and you can fill in their values to save them directly to `.env`. @@ -725,6 +728,107 @@ tools: [] The server spawns the process, performs the MCP handshake once, then forwards every tool call to it. The process is reused across calls (started lazily on the first tool call). +--- + +### Part 3.5 — a repository provider (clone + build + introspect) + +For MCP servers that are published only as source code (no `npx` / `uvx` / pip distribution), +use a **repository provider**. mcpproxy will: + +1. `git clone` the repo into a workdir under `MCPPROXY_REPOS_DIR` (default `/app/repos/`). +2. Run each entry of `build_commands` inside that workdir (e.g. `npm install`, `npm run build`). +3. Spawn the `package.command` from inside the workdir and introspect tools the same way as a + package provider. +4. Re-run steps 1–3 on every server start so ephemeral containers always have a fresh build. + +#### Adding one via the wizard + +1. Click **+ New Provider** → choose **📂 Repository**. +2. Fill in: + - **Provider name** — e.g. `linkedin`. + - **Git URL** — `https://github.com/felipfr/linkedin-mcpserver` (https or ssh). + - **Ref** *(optional)* — branch, tag, or commit SHA. Defaults to the repo's default branch. + - **Build commands** — one per row, e.g. `npm install`, then `npm run build`. + - **Spawn command** — the stdio MCP launch command, e.g. `node dist/main.js`. Runs inside the workdir. +3. Click **Next** — mcpproxy clones, builds, and introspects. The tool list is auto-populated. + +#### YAML produced + +```yaml +package: + command: node dist/main.js # spawn command, run inside the workdir +repository: + url: https://github.com/felipfr/linkedin-mcpserver + ref: main # optional + workdir: /app/repos/linkedin # optional — defaults to / + build_commands: + - npm install + - npm run build +tools: + - name: search_jobs # advertised as linkedin__search_jobs + description: Search LinkedIn job postings. + input_schema: + type: object + properties: + query: {type: string, description: "Search query"} + required: [query] +``` + +The `package.command` is what spawns the MCP server (just like a regular package provider). +The new `repository:` block tells the server **how to materialize the workdir on startup**. + +#### Editing a repository provider + +The editor shows a **📂 Repository** box with the git URL, ref, and build commands. +- **↻ Re-clone & build** — re-runs `git pull` (or `git clone` on a fresh container) + and the build commands, then re-introspects the spawn command. +- After saving, click **Restart MCP Server** to apply changes — on startup the server + walks every repository provider and re-runs clone/build before registering tools. + +#### Environment variables + +| Variable | Default | Description | +|---|---|---| +| `MCPPROXY_REPOS_DIR` | `/app/repos` | Base directory for cloned repos. | + +For persistent build caches across container restarts, mount this directory as a +volume: + +```yaml +volumes: + - mcpproxy-repos:/app/repos +``` + +Without a mount, every container start re-clones and re-builds — exactly what's wanted +for an ephemeral / disposable container. + +#### Lifecycle on container restart + +On every server start, `server.py` walks each YAML provider and: +- If the spec has a `repository:` block, runs `git clone` (or `git pull` if the workdir + already contains `.git`), then re-runs every entry in `build_commands` with + `cwd=`. +- Then runs the standard `requirements:` (pip) and `setup_commands:` lists. +- Then registers the tools and spawns the MCP subprocess (lazily, on first tool call). + +#### Security notes + +- Build commands run as the server user with full shell-style splitting via `shlex.split`. + Do **not** paste untrusted commands. +- The git URL is passed directly to `git clone`. Private repos require SSH keys or a + credential helper to be configured inside the container. + +#### Troubleshooting + +| Symptom | What to check | +|---|---| +| Clone hangs or fails | The container must have outbound HTTPS / SSH to the git host. For SSH, mount your `~/.ssh` and configure `known_hosts`. | +| `npm install` / build fails | View container stdout: `docker compose logs -f`. All build output is streamed unbuffered. | +| Spawn / introspect fails | The repo must produce a working stdio MCP server. Check the spawn command resolves inside the workdir (e.g. `dist/main.js` only exists after a successful build). | +| Tools not appearing after edit | Click **Restart MCP Server** so the YAML is re-loaded and the workdir re-materialized. | + +--- + ### pip Requirements vs setup_commands | Feature | Use for | @@ -861,7 +965,19 @@ package: # "python -m mcp_server_github" # "mcp-server-github" -# ── Shared optional fields (both provider types) ────────────────────────────── +# ── Repository provider (clone + build, spawned from inside the workdir) ────── +# When `repository:` is present, the `package.command` above is run with cwd +# set to the cloned workdir. Clone + build re-runs on every server start. + +repository: + url: string # e.g. "https://github.com/owner/repo" + ref: string # optional — branch, tag, or commit SHA + workdir: string # optional — defaults to / + build_commands: # shell commands run in before spawn + - npm install + - npm run build + +# ── Shared optional fields (all provider types) ─────────────────────────────── requirements: # pip packages installed before the server starts - package-name diff --git a/config.py b/config.py index 4a57008..7695dec 100644 --- a/config.py +++ b/config.py @@ -12,6 +12,11 @@ # immediately accessible. Override with MCPPROXY_FILES_DIR. FILES_DIR = Path(os.environ.get("MCPPROXY_FILES_DIR", ".playwright-mcp")) +# Base directory where repository providers clone their git repos. Each +# provider gets a subdirectory named after the provider (e.g. /app/repos/linkedin). +# Override with MCPPROXY_REPOS_DIR. +REPOS_DIR = Path(os.environ.get("MCPPROXY_REPOS_DIR", "/app/repos")) + UI_HOST = os.environ.get("MCP_UI_HOST", "0.0.0.0") UI_PORT = int(os.environ.get("MCP_UI_PORT", "8889")) diff --git a/frontend/app.py b/frontend/app.py index 96af89c..44a488b 100644 --- a/frontend/app.py +++ b/frontend/app.py @@ -36,7 +36,7 @@ from fastapi import FastAPI, HTTPException, Request from fastapi.responses import HTMLResponse, JSONResponse, StreamingResponse -from config import CONFIG_DIR, ENV_FILE +from config import CONFIG_DIR, ENV_FILE, REPOS_DIR # --------------------------------------------------------------------------- @@ -116,6 +116,25 @@ def _get_package_spec(spec: dict[str, Any]) -> dict[str, Any] | None: return spec.get("package") or None +def _get_repository_spec(spec: dict[str, Any]) -> dict[str, Any] | None: + """Return the repository sub-dict (repository:), or None when absent.""" + return spec.get("repository") or None + + +def _safe_provider_dirname(name: str) -> str: + """Normalize a provider name into a safe single-segment directory name.""" + safe = re.sub(r"[^a-zA-Z0-9_-]", "-", name or "").strip("-") + return safe or "repo" + + +def _repository_workdir(name: str, explicit: str | None = None) -> str: + """Resolve the on-disk workdir path for a repository provider.""" + explicit = (explicit or "").strip() + if explicit: + return explicit + return str(REPOS_DIR / _safe_provider_dirname(name)) + + def _provider_to_structured(name: str, spec: dict[str, Any]) -> dict[str, Any]: """Convert a loaded YAML spec into the structured JSON the UI works with.""" tools_out = [] @@ -146,12 +165,29 @@ def _provider_to_structured(name: str, spec: dict[str, Any]) -> dict[str, Any]: }) pkg_sub = _get_package_spec(spec) - if pkg_sub is not None: + repo_sub = _get_repository_spec(spec) + if repo_sub is not None: + ptype = "repository" + command = (pkg_sub.get("command") if pkg_sub else "") or "" + command = command.strip() + repo_url = (repo_sub.get("url") or "").strip() + repo_ref = (repo_sub.get("ref") or "").strip() + build_commands = list(repo_sub.get("build_commands") or []) + workdir = _repository_workdir(name, repo_sub.get("workdir")) + elif pkg_sub is not None: ptype = "package" command = (pkg_sub.get("command") or "").strip() + repo_url = "" + repo_ref = "" + build_commands = [] + workdir = "" else: ptype = "code" command = "" + repo_url = "" + repo_ref = "" + build_commands = [] + workdir = "" return { "name": name, @@ -161,6 +197,10 @@ def _provider_to_structured(name: str, spec: dict[str, Any]) -> dict[str, Any]: "code": spec.get("code", ""), "requirements": list(spec.get("requirements") or []), "setup_commands": list(spec.get("setup_commands") or []), + "repo_url": repo_url, + "repo_ref": repo_ref, + "build_commands": build_commands, + "workdir": workdir, "tools": tools_out, } @@ -177,6 +217,21 @@ def _structured_to_yaml(provider: dict[str, Any]) -> str: if ptype == "package": spec["package"] = {"command": (provider.get("command") or "").strip()} + elif ptype == "repository": + spec["package"] = {"command": (provider.get("command") or "").strip()} + repo_block: dict[str, Any] = { + "url": (provider.get("repo_url") or "").strip(), + } + ref = (provider.get("repo_ref") or "").strip() + if ref: + repo_block["ref"] = ref + workdir = (provider.get("workdir") or "").strip() + if workdir: + repo_block["workdir"] = workdir + build_commands = [c for c in (provider.get("build_commands") or []) if c] + if build_commands: + repo_block["build_commands"] = build_commands + spec["repository"] = repo_block else: code = (provider.get("code") or "").strip() if code: @@ -236,6 +291,14 @@ def _validate_provider(provider: dict[str, Any]) -> dict[str, Any]: if ptype == "package": if not (provider.get("command") or "").strip(): errors.append("command is required for package providers") + elif ptype == "repository": + if not (provider.get("repo_url") or "").strip(): + errors.append("repo_url is required for repository providers") + if not (provider.get("command") or "").strip(): + errors.append("command is required for repository providers") + build_commands = provider.get("build_commands") + if build_commands is not None and not isinstance(build_commands, list): + errors.append("build_commands must be a list") else: if not (provider.get("code") or "").strip(): errors.append("code is required for code providers") @@ -328,6 +391,7 @@ async def list_tools() -> list[dict]: structured = _provider_to_structured(path.stem, spec) validation = _validate_provider(structured) is_package = bool(_get_package_spec(spec)) + is_repository = bool(_get_repository_spec(spec)) out.append({ "name": path.stem, "file": path.name, @@ -335,6 +399,7 @@ async def list_tools() -> list[dict]: "tool_names": [t.get("name") for t in tool_entries], "provider_type": structured["type"], "is_package": is_package, + "is_repository": is_repository, "secret_keys": secret_keys, "missing_secrets": missing_secrets, "validation_errors": validation["errors"], @@ -418,6 +483,7 @@ async def introspect_package(request: Request) -> dict: command = (body.get("command") or "").strip() requirements: list[str] = body.get("requirements") or [] setup_commands: list[str] = body.get("setup_commands") or [] + cwd = (body.get("cwd") or "").strip() or None if not command: raise HTTPException(400, "command is required") @@ -433,20 +499,70 @@ async def introspect_package(request: Request) -> dict: check=True, ) - # 2. Run setup commands + # 2. Run setup commands (in cwd when one is supplied — e.g. a repo workdir) for cmd in setup_commands: if not cmd: continue - subprocess.run(shlex.split(cmd), check=True) + subprocess.run(shlex.split(cmd), check=True, cwd=cwd) # 3. Introspect the MCP server from process_runner import introspect - tools = await introspect(command) + tools = await introspect(command, cwd=cwd) return {"ok": True, "tools": tools, "package_manager": pm} except Exception as exc: traceback.print_exc() return {"ok": False, "error": str(exc), "tools": [], "package_manager": pm} + # ── Repository clone-and-build ─────────────────────────────────────────── + + @app.post("/api/clone-and-build") + async def clone_and_build(request: Request) -> dict: + """Clone (or pull) a git repo and run build_commands inside the workdir. + + Body: + name — provider name (used to derive the default workdir) + repo_url — git URL (required) + ref — optional branch/tag/commit + build_commands — optional list of shell commands run inside the workdir + workdir — optional explicit workdir path (overrides default) + + Idempotent: if ``/.git`` exists, runs ``git pull`` instead of + ``git clone`` so persistent volumes pick up upstream changes. + """ + body = await request.json() + name = (body.get("name") or "").strip() + url = (body.get("repo_url") or "").strip() + ref = (body.get("ref") or "").strip() + explicit_workdir = (body.get("workdir") or "").strip() + build_commands: list[str] = body.get("build_commands") or [] + if not name: + raise HTTPException(400, "name is required") + _guard_name(name) + if not url: + raise HTTPException(400, "repo_url is required") + + workdir = _repository_workdir(name, explicit_workdir) + + try: + Path(workdir).parent.mkdir(parents=True, exist_ok=True) + if (Path(workdir) / ".git").exists(): + subprocess.run(["git", "-C", workdir, "pull", "--ff-only"], check=True) + else: + subprocess.run(["git", "clone", url, workdir], check=True) + + if ref: + subprocess.run(["git", "-C", workdir, "checkout", ref], check=True) + + for cmd in build_commands: + if not cmd: + continue + subprocess.run(shlex.split(cmd), check=True, cwd=workdir) + + return {"ok": True, "workdir": workdir} + except Exception as exc: + traceback.print_exc() + return {"ok": False, "error": str(exc), "workdir": workdir} + # ── Function extractor (code providers) ────────────────────────────────── @app.post("/api/extract-functions") @@ -665,6 +781,7 @@ async def index(): /* badges */ .badge-pkg{background:#cba6f7;color:#1e1e2e;font-size:.65em;padding:2px 6px;border-radius:3px;font-weight:700} .badge-code{background:#89b4fa;color:#1e1e2e;font-size:.65em;padding:2px 6px;border-radius:3px;font-weight:700} +.badge-repo{background:#a6e3a1;color:#1e1e2e;font-size:.65em;padding:2px 6px;border-radius:3px;font-weight:700} .badge-count{background:#45475a;color:#cdd6f4;font-size:.65em;padding:2px 6px;border-radius:3px} /* modal */ .modal-content{background:var(--bg);border:1px solid var(--border);color:#cdd6f4} @@ -792,6 +909,36 @@ async def index():
Any command that spawns a stdio MCP server: npx, uvx, python -m, or an installed binary. The process is started on demand and kept alive between calls.
+ + +