Fix BFCL default argument scoring

[wise-east]

Summary

• fill optional schema defaults before comparing BFCL tool-call arguments
• preserve failures for unknown arguments, non-default values, and missing required arguments

Why this matters

The current scorer measures whether the model reproduced the ground truth's argument serialization, not whether it produced the correct function call. For an optional argument with a declared default, omission and explicitly passing that default produce the same effective invocation. Marking one form incorrect creates false negatives for operationally valid calls.

This is harmful in several ways:

• Scores reflect annotation style rather than tool-use ability. A model can select the correct function and all effective argument values yet lose credit solely because it made a default explicit.
• The metric is internally inconsistent. Some ground truths omit defaults (exec_simple_86), while others explicitly include them (exec_simple_79). A model cannot follow one consistent serialization policy and be correct in both cases under raw dictionary equality.
• Model comparisons become fragile. Models or tool-calling pipelines that habitually emit explicit defaults score below models that omit them, even though both would execute identically.
• It creates the wrong training incentive. Instead of improving semantic function calling, benchmark participants are pushed to learn hidden per-item annotation conventions. In an observed PostTrainBench run, these false negatives prompted an agent to inspect the affected test answers and rewrite training labels to omit defaults, turning an evaluator defect into a benchmark-targeting/contamination issue.
• Error analysis becomes misleading. These failures are reported as model mistakes, obscuring whether an actual function name, required argument, or non-default value was wrong.

Normalizing both calls with optional schema defaults before comparison makes the score track effective tool behavior. It does not relax correctness for unknown keys, required arguments, or explicit values that differ from the default.

Concrete examples

• exec_simple_86: calculate_investment_value declares optional adjust_for_inflation with default true; the ground truth omits it, so a model call that explicitly passes adjust_for_inflation=true is semantically equivalent
• exec_simple_79: sort_array declares optional reverse with default false; the ground truth explicitly passes reverse=false, so a model call that omits it is also semantically equivalent
• exec_simple_78: the ground truth passes reverse=true; omission remains incorrect because true differs from the schema default false

Testing

• ran canonicalization assertions against omitted, explicit-default, and explicit-nondefault arguments
• ruff check src/eval/tasks/bfcl/task_context/bfcl_evaluation_code.py

Made with Cursor