Every variant so far splits its catalog along two axes, but core only models one of
them (Category), so each variant bends that single axis into a different shape:
| Variant |
Categories |
Groups within a category |
| VNN-COMP |
one (default) |
three: test / regular / extended |
| ARCH-COMP |
many (AFF, NLN, AINNCS, …) |
one (implicit default) |
| CORA-COMP |
one (contSet) |
three: test / sets / sets-batched |
The two axes are genuinely different things:
- A category defines the set of benchmarks
- A group partitions this set logically
A first cut is on feature/benchmark-groups: Competition.benchmark_groups() returns
the ordered group names, benchmarks name their group in extra["group"], and the toolkit
submission form renders one subheading per group (default: none, so existing variants
keep a flat list). CORA-COMP uses it today.
What that branch does not do, and what this issue is about:
- the group is a
Benchmark.extra convention rather than a field, so nothing validates
it and nothing can filter or order by it in the database;
- only the toolkit submission form renders it — task details, the benchmark overview and
the scoreboard still ignore groups (make it similar to how VNN-COMP shows different tracks: | in overview to separate benchmark lists, horizontal dividers on details page, ...);
- VNN's test/regular/extended and ARCH's implicit single group are not expressed in it,
so the table above is still three different mechanisms rather than one.
Every variant so far splits its catalog along two axes, but core only models one of
them (
Category), so each variant bends that single axis into a different shape:default)default)contSet)The two axes are genuinely different things:
A first cut is on
feature/benchmark-groups:Competition.benchmark_groups()returnsthe ordered group names, benchmarks name their group in
extra["group"], and the toolkitsubmission form renders one subheading per group (default: none, so existing variants
keep a flat list). CORA-COMP uses it today.
What that branch does not do, and what this issue is about:
Benchmark.extraconvention rather than a field, so nothing validatesit and nothing can filter or order by it in the database;
the scoreboard still ignore groups (make it similar to how VNN-COMP shows different tracks: | in overview to separate benchmark lists, horizontal dividers on details page, ...);
so the table above is still three different mechanisms rather than one.