Skip to content

test: add property and mutation coverage for core invariants - #506

Open
mldangelo-oai wants to merge 12 commits into
mainfrom
mdangelo/codex/test-properties-and-mutation
Open

test: add property and mutation coverage for core invariants#506
mldangelo-oai wants to merge 12 commits into
mainfrom
mdangelo/codex/test-properties-and-mutation

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add deterministic, behavior-focused tests for the SDK's core invariants and a small mutation-testing trial.

Part 2 of 4 in the test-quality stack. #504 has merged; this PR is now based on main.

Changes

  • Extract the pure cost calculations from filesystem-backed cost tracking while preserving the existing exports.
  • Add seeded property tests for token costs, error handling, progress events, finding identity, sealed artifacts, and safe artifact paths.
  • Test SQLite migration rollback, idempotence, and integrity against a real database.
  • Strengthen malformed-event and Markdown-boundary cases, and make generated filenames portable to Windows.
  • Add pinned development-only fast-check and Stryker dependencies. Keep mutation testing scoped to three pure modules.
  • Preserve the new live-session events when merging the refreshed parent.

Testing

  • Refreshed head eb839e7b, Bun 1.3.14, seed 1: 115 focused cost, error, progress, contract, publication, and comparison tests passed.
  • Independent pure-module rerun, seed 12345: 28 passed, 0 failed.
  • Type checking, formatting, and git diff --check: passed.
  • Initial mutation baseline: 487 killed, 18 survived, 0 timed out; 96.44% score. This refresh does not introduce a mutation-score gate.
  • Three fresh native reviews and independent verification of eb839e7b: no findings.
  • The prior tree-equivalent head e4b3ef94 hit an unchanged Windows subprocess test and then a pre-existing replacement-lock test in its CI retry. The lock correction is handled separately in fix: preserve exact multiscan lock identities #563. CI will rerun on the new main-based head.

Risk and rollout

The production change is a pure cost-module extraction with the existing public exports retained. The package allowlist includes the new emitted module. Property and mutation tooling are development-only. The refreshed parent preserves default test randomization and the Windows timeout allowance. When replaying a shrink path, select the failing file and test name. Mutation testing remains diagnostic until repeated runs establish a useful baseline.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head e038344e35131ad0f6430df7786a18bf62b9aa7d.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 17, 2026

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: e038344e35

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: e038344e35

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai
mldangelo-oai marked this pull request as ready for review August 17, 2026 22:21
faizan-oai
faizan-oai previously approved these changes Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the refreshed head 40b23460a1012505d4c41176a336b7074839f90b. This additive merge preserves the current parent, the 120-second Windows allowance, and the upstream live-session events. Normal and randomized SDK tests, packed-package checks, and the fresh native review gate passed.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 40b23460a1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 17, 2026

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 40b23460a1

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the additive refresh at e4b3ef94bc73749956952a5181d6cdea6e2bb029 against the updated parent. The existing cost exports, package boundary, seeded properties, and diagnostic-only mutation setup are preserved.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: e4b3ef94bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@codex security review

Please review the exact refreshed head e4b3ef94bc73749956952a5181d6cdea6e2bb029, including the pure cost-module extraction, artifact-path properties, and package allowlist.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: e4b3ef94bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Base automatically changed from mdangelo/codex/test-fixture-isolation to main August 18, 2026 17:17
@mldangelo-oai
mldangelo-oai dismissed faizan-oai’s stale review August 18, 2026 17:17

The base branch was changed.

@github-actions github-actions Bot added the skip-release-notes Omit internal changes from generated release notes label Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the main-based head eb839e7b10186fcf4c91d1efe99b15945e8b0271. #504 has merged. This additive restack has the same Git tree as e4b3ef94; the 115 focused tests and three fresh native reviews with independent verification passed again. The separately discovered, pre-existing lock-identity correction is in #563.

Copy link
Copy Markdown
Collaborator Author

@codex security review

Please review exact main-based head eb839e7b10186fcf4c91d1efe99b15945e8b0271, including the cost-module extraction, package allowlist, and artifact-path regression coverage.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: eb839e7b10

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-release-notes Omit internal changes from generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants