Skip to content

Comparison: expected-side error is re-raised instead of producing an assertion failure #12

Description

@jcouball

Summary

When the expected expression evaluates to a StandardError (e.g. #=> raise ZeroDivisionError, "divided by 0") but the actual expression returns a normal value, compare_values re-raises the expected-side error via only_one_is_error?. Minitest reports this as an Error, making it look like the user's code crashed — when in fact the code returned a value and the comparison simply didn't match. The actual return value is never shown to the user.

Location

lib/yard_example_test/example/comparison.rbcompare_values and only_one_is_error?

def compare_values(expected, actual)
  if both_are_errors?(expected, actual)
    assert_equal(...)
  elsif (error = only_one_is_error?(expected, actual))
    raise error
  ...
end

only_one_is_error? is symmetric — it returns whichever side is a StandardError — but the two cases have different semantics:

Expected Actual Current behavior Correct behavior
Error Value Re-raises expected error → shows as Error Should produce an assertion failure comparing the two
Value Error Re-raises actual error → shows as Error Correct — the user's code raised unexpectedly

Concrete example

# @example
#   divide(1, 0) #=> raise ZeroDivisionError, "divided by 0"
def divide(a, b)
  0  # bug: returns 0 instead of raising
end

Current output

1) Error:
#divide#test_0001_:
ZeroDivisionError: divided by 0

The user sees a ZeroDivisionError and thinks divide raised — but divide returned 0.

Expected output

An assertion failure showing something like:

1) Failure:
#divide#test_0001_:
Expected: #<ZeroDivisionError: divided by 0>
  Actual: 0

Proposed solution

Split the only_one_is_error? branch into two distinct cases:

def compare_values(expected, actual)
  if both_are_errors?(expected, actual)
    assert_equal("#<#{expected.class}: #{expected}>", "#<#{actual.class}: #{actual}>")
  elsif expected.is_a?(StandardError) && !actual.is_a?(StandardError)
    # Expected an error but got a value — assertion failure, not re-raise
    assert_equal("#<#{expected.class}: #{expected}>", actual.inspect)
  elsif !expected.is_a?(StandardError) && actual.is_a?(StandardError)
    # Actual raised unexpectedly — re-raise so Minitest reports as Error
    raise actual
  elsif expected.nil?
    assert_nil(actual)
  else
    assert expected === actual, diff(expected, actual)
  end
end

Acceptance criteria

  • When expected evaluates to an error but actual returns a value, Minitest reports a Failure (not an Error) with a meaningful diff
  • When actual raises unexpectedly, the error is still surfaced as an Error
  • When both sides are errors, they are still compared via assert_equal
  • All existing cucumber scenarios continue to pass

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions