Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ This file is read automatically by Claude Code at the start of every session. Fo

**Every PR that adds, modifies, or removes an MCP tool must update documentation.** The full set of places to update:

| What changed | Where to update |
|---|---|
| New MCP tool | `docs/mcp.md` (add tool entry with schema + example), `docs/mcp-pilot-guide.md` (add evaluation scenario if user-facing), `README.md` (update tool count if referenced) |
| Modified tool description / parameters / errors | `docs/mcp.md` (update the relevant tool section) |
| Removed tool | `docs/mcp.md`, `docs/mcp-pilot-guide.md`, `README.md` |
| New error code or suggestion field | `docs/mcp.md` (Troubleshooting or Error Codes section) |
| Security model change (path policy, licence, transport) | `docs/mcp.md` (Security Model section), `docs/mcp-pilot-guide.md` (Security Model section) |
| New NitroX tool or schema rule | `docs/mcp.md` (NitroX section), `docs/mcp-pilot-guide.md` (Scenario 7) |
| Smoke test count change | Update `TOTAL_EXPECTED` in `scripts/mcp-smoke.cjs` |
| What changed | Where to update |
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| New MCP tool | `docs/mcp.md` (add tool entry with schema + example), `docs/mcp-pilot-guide.md` (add evaluation scenario if user-facing), `README.md` (update tool count if referenced) |
| Modified tool description / parameters / errors | `docs/mcp.md` (update the relevant tool section) |
| Removed tool | `docs/mcp.md`, `docs/mcp-pilot-guide.md`, `README.md` |
| New error code or suggestion field | `docs/mcp.md` (Troubleshooting or Error Codes section) |
| Security model change (path policy, licence, transport) | `docs/mcp.md` (Security Model section), `docs/mcp-pilot-guide.md` (Security Model section) |
| New NitroX tool or schema rule | `docs/mcp.md` (NitroX section), `docs/mcp-pilot-guide.md` (Scenario 7) |
| Smoke test count change | Update `TOTAL_EXPECTED` in `scripts/mcp-smoke.cjs` |

**External / customer-facing docs** (`docs/provar-mcp-public-docs.md`, `docs/university-of-provar-mcp-course.md`) are maintained separately by the Provar team — flag changes that affect public-facing behaviour in your PR description so those can be updated manually.

Expand All @@ -32,6 +32,7 @@ Every PR must include tests for new or changed behaviour:
- **Smoke test** → add an entry in `scripts/mcp-smoke.cjs` for each new tool; update `TOTAL_EXPECTED`

Run before every commit:

```sh
yarn test:only # unit tests — must all pass
node scripts/mcp-smoke.cjs 2>/dev/null # smoke — must show all PASS
Expand All @@ -41,9 +42,11 @@ yarn compile # TypeScript — must be clean
> **Wireit caching gotcha:** `yarn test:only` is wired through wireit and can return a cached result
> after changes (e.g. after `git stash pop` or switching branches). When you need a guaranteed fresh
> run, bypass wireit entirely:
>
> ```sh
> node_modules/.bin/nyc node_modules/.bin/mocha "test/**/*.test.ts"
> ```
>
> This is equivalent to `yarn test:dev` and always re-executes against the current source.

---
Expand Down
12 changes: 6 additions & 6 deletions docs/mcp-pilot-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ The server runs **locally on your machine**. It does not phone home, transmit yo

## Prerequisites

| Requirement | Version | Notes |
| --------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Provar Automation IDE | ≥ 2.x | Must be installed with an **activated licence** on the same machine. The MCP server reads the licence from `~/Provar/.licenses/`. |
| Salesforce CLI (`sf`) | ≥ 2.x | `npm install -g @salesforce/cli` |
| Provar DX CLI plugin | ≥ 1.5.0 | `sf plugins install @provartesting/provardx-cli@beta` |
| An MCP-compatible AI client | — | Claude Desktop, Claude Code, or Cursor |
| Requirement | Version | Notes |
| --------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Provar Automation IDE | ≥ 2.x | Must be installed with an **activated licence** on the same machine. The MCP server reads the licence from `~/Provar/.licenses/`. |
| Salesforce CLI (`sf`) | ≥ 2.x | `npm install -g @salesforce/cli` |
| Provar DX CLI plugin | ≥ 1.5.0 | `sf plugins install @provartesting/provardx-cli@beta` |
| An MCP-compatible AI client | — | Claude Desktop, Claude Code, or Cursor |
| Node.js | 18–24 | Installed automatically with the SF CLI. **Node 25+ is not supported** — a transitive dependency crashes on startup. Use Node 22 LTS. |

---
Expand Down
1 change: 1 addition & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,7 @@ Validates an XML test case for schema correctness (validity score) and best prac
**Key schema rules:** TC_001 (missing XML declaration), TC_002 (malformed XML), TC_003 (wrong root element), TC_010/011/012 (missing/invalid id/guid), TC_031 (invalid apiCall guid), TC_034/035 (non-integer testItemId).

**Warning rules:**

- **DATA-001** — `testCase` declares a `<dataTable>` element. CLI standalone execution does not bind CSV column variables; steps using variable references will resolve to null. Use `SetValues` (Test scope) steps instead, or add the test to a test plan.
- **ASSERT-001** — An `AssertValues` step uses the `argument id="values"` (namedValues) format, which is designed for UI element attribute assertions. For Apex/SOQL result or variable comparisons this silently passes as `null=null`. Use separate `expectedValue`, `actualValue`, and `comparisonType` arguments instead.

Expand Down
8 changes: 4 additions & 4 deletions docs/university-of-provar-mcp-course.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,10 +275,10 @@ A file can be valid (no schema errors) but have a low quality score (missing des

The `provar.testcase.validate` tool operates in one of two modes depending on whether a Quality Hub API key is configured:

| Mode | `validation_source` | Rules | Requires |
|---|---|---|---|
| **Quality Hub API** | `quality_hub` | 170+ rules, full quality scoring | `sf provar auth login` (once) |
| **Local only** | `local` | Structural and schema rules | Nothing |
| Mode | `validation_source` | Rules | Requires |
| ------------------- | ------------------- | -------------------------------- | ----------------------------- |
| **Quality Hub API** | `quality_hub` | 170+ rules, full quality scoring | `sf provar auth login` (once) |
| **Local only** | `local` | Structural and schema rules | Nothing |

If a key is configured but the API is temporarily unreachable, the tool falls back to local rules and sets `validation_source: "local_fallback"` with a `validation_warning` explaining why.

Expand Down
3 changes: 3 additions & 0 deletions messages/sf.provar.auth.clear.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
# summary

Remove the stored Provar API key.

# description

Deletes the API key stored at ~/.provar/credentials.json. After clearing, the
provar.testcase.validate MCP tool falls back to local validation (structural rules only,
no Quality Hub quality scoring).

The PROVAR_API_KEY environment variable is not affected by this command.

# examples

- Clear the stored API key:
<%= config.bin %> <%= command.id %>
3 changes: 3 additions & 0 deletions messages/sf.provar.auth.status.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
# summary

Show the current Provar API key configuration status.

# description

Reports where the active API key comes from (environment variable or stored file),
shows the key prefix and when it was set, and states whether validation will use the
Quality Hub API or local rules only. The full key is never printed.

If no key is configured, guidance is shown for logging in or requesting access.

# examples

- Check auth status:
<%= config.bin %> <%= command.id %>
7 changes: 3 additions & 4 deletions src/commands/provar/auth/login.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,10 +88,9 @@ export default class SfProvarAuthLogin extends SfCommand<void> {
keyData = await qualityHubClient.exchangeTokenForKey(tokens.access_token, baseUrl);
} catch (err) {
if (err instanceof QualityHubAuthError) {
this.error(
`No Provar MCP account found for this login.\nRequest access at: ${REQUEST_ACCESS_URL}`,
{ exit: 1 }
);
this.error(`No Provar MCP account found for this login.\nRequest access at: ${REQUEST_ACCESS_URL}`, {
exit: 1,
});
}
throw err;
}
Expand Down
Loading
Loading