Skip to content

New Benchmark: Add SciKG Verify MatBench dielectric result - #366

Open
Demfier wants to merge 3 commits into
materialsproject:mainfrom
Demfier:submit-scikg-verify-dielectric
Open

New Benchmark: Add SciKG Verify MatBench dielectric result#366
Demfier wants to merge 3 commits into
materialsproject:mainfrom
Demfier:submit-scikg-verify-dielectric

Conversation

@Demfier

@Demfier Demfier commented Sep 2, 2026

Copy link
Copy Markdown

Benchmark submission

This adds a single-task MatBench v0.1 submission for the
matbench_dielectric per-task leaderboard using SciKG Verify. MatBench's site
builder includes every recorded task in its per-task leaderboard, while its
main general-purpose suite table separately requires a complete, regression-
complete, or structure-complete submission. This PR targets the former; the
full-suite submission is separate ongoing work.

Headline result

SciKG Verify reduces the five-fold mean MAE of the frozen MODNet v0.1.12
predictions from 0.27110192 to 0.24932957. That is an 8.03% error
reduction
and a 0.02177 absolute improvement over the leader in the frozen
MatBench snapshot used for this study. Every one of the five official folds
improves. The paired reduction has a 95% bootstrap interval of
0.01762--0.02600 and a 20,000-draw sign-flip p-value of 0.00005.

Cross-model evidence

This is not a one-model correction discovered specifically for MODNet. The
same frozen nested verifier was transferred to all 16 archived dielectric
models without model-specific retuning and improved 16/16 models and 80/80
model-fold pairs
. The median relative MAE reduction was 13.85%; for example,
coGN improved from 0.30877177 to 0.27063735 (12.35%).

A later, separately versioned compatibility-aware diagnostic exposed the
mechanism on a held-out ZnO row: coGN predicted 34.8564 against 2.4092, while a
structure-matched pre-cutoff JARVIS record reported 2.3742. The nested verifier
corrected the prediction to 2.3742, eliminating 99.89% of the absolute
error. This row is supporting diagnostic evidence and is not substituted for
the submitted five-fold aggregate. The later method variant also is not mixed
post hoc into this submission.

Algorithm and evaluation contract

SciKG Verify applies a temporally filtered scientific-evidence verifier to
frozen MODNet predictions. It retrieves compatible measurements and relations
published before the evaluation cutoff, enforces structure, property and
source compatibility, and uses nested out-of-fold selection to decide whether
and how strongly to correct each prediction. The outer test fold is excluded
from every selector, gate and parameter choice.

This is intentionally disclosed as an external-evidence-assisted result,
not a closed-book model result. The distinction is central to the method: it
tests whether a separately trained scientific model can be corrected at test
time using admissible prior scientific evidence.

Included files

Only the required public benchmark folder is added:

  • results.json.gz: native MatBench recording for the dielectric task
  • info.json: method, metric, dependency, and evidence-contract metadata
  • submission_client.py: thin standard-library client for reproduction
    against the hosted verifier

The knowledge graph, literature corpus, extraction prompts, model weights,
matching rules and credentials are private implementation IP and are not
included. A revocable benchmark API credential can be provided to maintainers
for result reproduction.

Validation

  • Result archive SHA-256: 3c99171a6a140fda4aaee667445c6d4fa6e3ff20c62836962e50f06d49283d75
  • All five official dielectric folds are recorded.
  • Outer-fold labels were excluded from all inner fitting and selection.
  • The public client reproduced the frozen predictions exactly against the
    deployed HTTPS endpoint.

Please apply the new_benchmark label.

@Demfier

Demfier commented Sep 2, 2026

Copy link
Copy Markdown
Author

CI note: the benchmark-submission job fails before inspecting this submission. The workflow installs current matminer==0.9.3 / pymatgen==2024.8.9, then requirements-dev.txt downgrades monty to 2022.4.26. Those packages require newer Monty, and import consequently fails with TypeError: deprecated() got an unexpected keyword argument 'deadline' while importing pymatgen. The submission itself is not reached by scripts/test_submission.py.

The added results.json.gz was independently loaded with MatBench 0.6 on the x86 evaluation environment and reports is_valid=True, all five matbench_dielectric folds recorded. Artifact SHA-256: 3c99171a6a140fda4aaee667445c6d4fa6e3ff20c62836962e50f06d49283d75.

@Demfier

Demfier commented Sep 2, 2026

Copy link
Copy Markdown
Author

Maintainer note: the submission guide requires the new_benchmark label. GitHub does not permit this external contributor account to add repository labels, so please add new_benchmark during review. The currently failing checks are upstream CI failures addressed separately in #367.

@Demfier Demfier changed the title Add SciKG Verify MatBench dielectric result New Benchmark: Add SciKG Verify MatBench dielectric result Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant