You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This adds a single-task MatBench v0.1 submission for the matbench_dielectric per-task leaderboard using SciKG Verify. MatBench's site
builder includes every recorded task in its per-task leaderboard, while its
main general-purpose suite table separately requires a complete, regression-
complete, or structure-complete submission. This PR targets the former; the
full-suite submission is separate ongoing work.
Headline result
SciKG Verify reduces the five-fold mean MAE of the frozen MODNet v0.1.12
predictions from 0.27110192 to 0.24932957. That is an 8.03% error
reduction and a 0.02177 absolute improvement over the leader in the frozen
MatBench snapshot used for this study. Every one of the five official folds
improves. The paired reduction has a 95% bootstrap interval of
0.01762--0.02600 and a 20,000-draw sign-flip p-value of 0.00005.
Cross-model evidence
This is not a one-model correction discovered specifically for MODNet. The
same frozen nested verifier was transferred to all 16 archived dielectric
models without model-specific retuning and improved 16/16 models and 80/80
model-fold pairs. The median relative MAE reduction was 13.85%; for example,
coGN improved from 0.30877177 to 0.27063735 (12.35%).
A later, separately versioned compatibility-aware diagnostic exposed the
mechanism on a held-out ZnO row: coGN predicted 34.8564 against 2.4092, while a
structure-matched pre-cutoff JARVIS record reported 2.3742. The nested verifier
corrected the prediction to 2.3742, eliminating 99.89% of the absolute
error. This row is supporting diagnostic evidence and is not substituted for
the submitted five-fold aggregate. The later method variant also is not mixed
post hoc into this submission.
Algorithm and evaluation contract
SciKG Verify applies a temporally filtered scientific-evidence verifier to
frozen MODNet predictions. It retrieves compatible measurements and relations
published before the evaluation cutoff, enforces structure, property and
source compatibility, and uses nested out-of-fold selection to decide whether
and how strongly to correct each prediction. The outer test fold is excluded
from every selector, gate and parameter choice.
This is intentionally disclosed as an external-evidence-assisted result,
not a closed-book model result. The distinction is central to the method: it
tests whether a separately trained scientific model can be corrected at test
time using admissible prior scientific evidence.
Included files
Only the required public benchmark folder is added:
results.json.gz: native MatBench recording for the dielectric task
info.json: method, metric, dependency, and evidence-contract metadata
submission_client.py: thin standard-library client for reproduction
against the hosted verifier
The knowledge graph, literature corpus, extraction prompts, model weights,
matching rules and credentials are private implementation IP and are not
included. A revocable benchmark API credential can be provided to maintainers
for result reproduction.
Validation
Result archive SHA-256: 3c99171a6a140fda4aaee667445c6d4fa6e3ff20c62836962e50f06d49283d75
All five official dielectric folds are recorded.
Outer-fold labels were excluded from all inner fitting and selection.
The public client reproduced the frozen predictions exactly against the
deployed HTTPS endpoint.
CI note: the benchmark-submission job fails before inspecting this submission. The workflow installs current matminer==0.9.3 / pymatgen==2024.8.9, then requirements-dev.txt downgrades monty to 2022.4.26. Those packages require newer Monty, and import consequently fails with TypeError: deprecated() got an unexpected keyword argument 'deadline' while importing pymatgen. The submission itself is not reached by scripts/test_submission.py.
The added results.json.gz was independently loaded with MatBench 0.6 on the x86 evaluation environment and reports is_valid=True, all five matbench_dielectric folds recorded. Artifact SHA-256: 3c99171a6a140fda4aaee667445c6d4fa6e3ff20c62836962e50f06d49283d75.
Maintainer note: the submission guide requires the new_benchmark label. GitHub does not permit this external contributor account to add repository labels, so please add new_benchmark during review. The currently failing checks are upstream CI failures addressed separately in #367.
Demfier
changed the title
Add SciKG Verify MatBench dielectric result
New Benchmark: Add SciKG Verify MatBench dielectric result
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmark submission
This adds a single-task MatBench v0.1 submission for the
matbench_dielectricper-task leaderboard using SciKG Verify. MatBench's sitebuilder includes every recorded task in its per-task leaderboard, while its
main general-purpose suite table separately requires a complete, regression-
complete, or structure-complete submission. This PR targets the former; the
full-suite submission is separate ongoing work.
Headline result
SciKG Verify reduces the five-fold mean MAE of the frozen MODNet v0.1.12
predictions from 0.27110192 to 0.24932957. That is an 8.03% error
reduction and a 0.02177 absolute improvement over the leader in the frozen
MatBench snapshot used for this study. Every one of the five official folds
improves. The paired reduction has a 95% bootstrap interval of
0.01762--0.02600 and a 20,000-draw sign-flip p-value of 0.00005.
Cross-model evidence
This is not a one-model correction discovered specifically for MODNet. The
same frozen nested verifier was transferred to all 16 archived dielectric
models without model-specific retuning and improved 16/16 models and 80/80
model-fold pairs. The median relative MAE reduction was 13.85%; for example,
coGN improved from 0.30877177 to 0.27063735 (12.35%).
A later, separately versioned compatibility-aware diagnostic exposed the
mechanism on a held-out ZnO row: coGN predicted 34.8564 against 2.4092, while a
structure-matched pre-cutoff JARVIS record reported 2.3742. The nested verifier
corrected the prediction to 2.3742, eliminating 99.89% of the absolute
error. This row is supporting diagnostic evidence and is not substituted for
the submitted five-fold aggregate. The later method variant also is not mixed
post hoc into this submission.
Algorithm and evaluation contract
SciKG Verify applies a temporally filtered scientific-evidence verifier to
frozen MODNet predictions. It retrieves compatible measurements and relations
published before the evaluation cutoff, enforces structure, property and
source compatibility, and uses nested out-of-fold selection to decide whether
and how strongly to correct each prediction. The outer test fold is excluded
from every selector, gate and parameter choice.
This is intentionally disclosed as an external-evidence-assisted result,
not a closed-book model result. The distinction is central to the method: it
tests whether a separately trained scientific model can be corrected at test
time using admissible prior scientific evidence.
Included files
Only the required public benchmark folder is added:
results.json.gz: native MatBench recording for the dielectric taskinfo.json: method, metric, dependency, and evidence-contract metadatasubmission_client.py: thin standard-library client for reproductionagainst the hosted verifier
The knowledge graph, literature corpus, extraction prompts, model weights,
matching rules and credentials are private implementation IP and are not
included. A revocable benchmark API credential can be provided to maintainers
for result reproduction.
Validation
3c99171a6a140fda4aaee667445c6d4fa6e3ff20c62836962e50f06d49283d75deployed HTTPS endpoint.
Please apply the
new_benchmarklabel.