Skip to content

Benchmarks need a grouping axis within a category #10

Description

@toladnertum

Every variant so far splits its catalog along two axes, but core only models one of
them (Category), so each variant bends that single axis into a different shape:

Variant Categories Groups within a category
VNN-COMP one (default) three: test / regular / extended
ARCH-COMP many (AFF, NLN, AINNCS, …) one (implicit default)
CORA-COMP one (contSet) three: test / sets / sets-batched

The two axes are genuinely different things:

  • A category defines the set of benchmarks
  • A group partitions this set logically

A first cut is on feature/benchmark-groups: Competition.benchmark_groups() returns
the ordered group names, benchmarks name their group in extra["group"], and the toolkit
submission form renders one subheading per group (default: none, so existing variants
keep a flat list). CORA-COMP uses it today.

What that branch does not do, and what this issue is about:

  • the group is a Benchmark.extra convention rather than a field, so nothing validates
    it and nothing can filter or order by it in the database;
  • only the toolkit submission form renders it — task details, the benchmark overview and
    the scoreboard still ignore groups (make it similar to how VNN-COMP shows different tracks: | in overview to separate benchmark lists, horizontal dividers on details page, ...);
  • VNN's test/regular/extended and ARCH's implicit single group are not expressed in it,
    so the table above is still three different mechanisms rather than one.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions