fix: clarify Python pytest failure messages

[kbuffardi]

Closes #23

Summary

• preserve custom assertion feedback without pytest framing
• retain complete default pytest E-line diagnostics for assertions and exceptions
• add dependency-free Node 16-compatible parser regressions
• count pytest errors alongside failed tests

Testing

• npm test

[Copilot]

Pull request overview

Improves Python pytest result parsing so learners receive more constructive, non-truncated failure feedback (custom assertion messages and full E ... diagnostic lines), aligning with Issue #23’s request for clearer outcome messaging.

Changes:

• Reworked parsePytestOutput to extract per-failure error message blocks (including multiline assertion details) and distinguish failures vs errors for counting.
• Added dependency-free Node regression tests covering custom assertion messages, default assertion diffs, exceptions, multiple failures, collection errors, and mixed failure/error runs.
• Updated npm test to run the new regression suite.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
executor.js	Updates pytest output parsing to preserve detailed assertion/exception diagnostics and count errors alongside failures.
test/executor.test.js	Adds Node-based regression tests for pytest parsing behavior across failure/error scenarios.
package.json	Wires npm test to run the new regression tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

test/pytest-integration.test.js:12

• npm test now runs test:integration, which unconditionally spawns the pytest binary. If pytest is not installed / not on PATH (common outside the Docker image), spawnSync will return an error and this test will fail (or produce confusing assertions) even though the Node code is fine. Consider skipping this integration regression when pytest is unavailable, or failing with an explicit message.

const versionResult = spawnSync('pytest', ['--version'], { encoding: 'utf8' });
const pytestVersion = (versionResult.stdout || versionResult.stderr).trim();

[kbuffardi]

Follow-up plan: preserve multiline failure formatting in codewit.us

PR #24 correctly returns the complete failure_details[].error_message; the remaining issue is presentation in CodeSubmission.tsx, where the message is rendered in a normal <span> and browser whitespace collapsing removes the pytest line structure. No additional codeval parser change is needed.

1. Render assertion details as preformatted text

   • Replace the failure-message <span> with a semantic <pre>.
   • Reuse the component’s existing output convention: font-mono whitespace-pre-wrap, with horizontal overflow or safe word wrapping so long diagnostics do not expand the results panel.
   • Preserve the message exactly as received; do not split, trim, or reinterpret pytest output in the client.

2. Add a focused component regression

   • Add CodeSubmission.spec.tsx beside the component using the existing Vitest/Testing Library setup.
   • Render an EvaluationResponse containing a multiline assertion diff.
   • Assert that the complete error_message remains the element’s exact textContent and is rendered by a <pre> with whitespace-pre-wrap.
   • Include a long diagnostic line to verify the container does not cause page-level horizontal overflow.

3. Verify the integrated learner flow

   • Run the focused Vitest test and the client production build.
   • Submit a faulty Python solution against codeval PR fix: clarify Python pytest failure messages #24 and confirm the Results → Outcome panel visibly preserves each assertion/diff line, retains all expected/actual text, and still works when navigating multiple failures.

Acceptance criteria

• Newlines and indentation in failure_details[].error_message are visibly preserved.
• No assertion content is truncated or altered.
• Long diagnostics remain usable without breaking the results layout.
• Existing compilation/runtime error and raw-output presentation remains unchanged.

This should be implemented as a small follow-up PR in codewit-us/codewit.us and linked here for end-to-end verification.

[kbuffardi]

@DAC098 I'm curious about your thoughts on multi-line support once you have some time testing this out with real exercises. I'm tempted to push the multi-line output to later (or not planned) because I suspect most of our failure messages should be more simplistic and not depend on formatting across multiple lines. I can imagine some situations where that might not be the case, but shipping this feature soon is a higher priority.

[DAC098]

I could see having multi-line messages being useful for more complicated tests but should probably keep it to simple messages as much as possible. if the message is complicated enough then maybe we could make notes or hints about possible errors that could happen in the prompt of the exercise. I am going to look more into the PR and test it against the exercises that I updated since I can just directly send tests to codeval for quicker results.

I was curious if pytest supported doing like a json output (or structured output of any kind) to see if we could reduce the amount of stuff that we would need to parse. pytest does support doing --junit-xml=PATH for CI/CD related things (they mention Jenkins). we are able to parse that as it also provides additional data about tests and output but could also include custom properties that we can specify in the test itself. we would still want to have the ability to parse the output if needed but having the structured output could help with providing extra detail without having to parse the current stdout text.

[kbuffardi]

I could see having multi-line messages being useful for more complicated tests but should probably keep it to simple messages as much as possible. if the message is complicated enough then maybe we could make notes or hints about possible errors that could happen in the prompt of the exercise. I am going to look more into the PR and test it against the exercises that I updated since I can just directly send tests to codeval for quicker results.

I was curious if pytest supported doing like a json output (or structured output of any kind) to see if we could reduce the amount of stuff that we would need to parse. pytest does support doing --junit-xml=PATH for CI/CD related things (they mention Jenkins). we are able to parse that as it also provides additional data about tests and output but could also include custom properties that we can specify in the test itself. we would still want to have the ability to parse the output if needed but having the structured output could help with providing extra detail without having to parse the current stdout text.

Good points.

Let's wrap up this PR without those features, but I'll note both Multi-line messages and possible json results as potential future features.

[DAC098]

I have been looking through the PR, once of the exercises I have multiple tests for and while to collects the tests individually, it copies the raw out for each test which creates a fairly large json to be created when responding. not sure if this is something that you want or if you want each individual test to only contain the rawOut for it's specific test. also that when an assertion has a custom message the expected and received fields are left blank, was this also desired or did you want to include that in the response.

example output that I have as a json and includes the command to run the test:

user@noop ~/r/c/exercises (python_exercise_updates)> node ./scripts/req_test.js -p 3000 python/uRDYSScpmso
{
  state: 'failed',
  tests_run: 4,
  passed: 1,
  failed: 3,
  errors: 0,
  no_tests_collected: false,
  exit_code: 1,
  failure_details: [
    {
      test_case: 'test_correct_first_try',
      expected: "'Correct! The word was puy\\n'",
      received: "'Correct! The word was puyp\\n'",
      error_message: "AssertionError: assert 'Correct! The word was puyp\\n' == 'Correct! The word was puy\\n'\n" +
        '  \n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +',
      rawout: '============================= test session starts ==============================\n' +
        'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0 -- /usr/bin/python3\n' +
        'cachedir: .pytest_cache\n' +
        'rootdir: /usr/src/app/temp/9c3805ed-3060-4868-a332-4c4e0fa580f2\n' +
        'collecting ... collected 4 items\n' +
        '\n' +
        'test_program.py::test_correct_first_try FAILED                           [ 25%]\n' +
        'test_program.py::test_incorrect_first_try FAILED                         [ 50%]\n' +
        'test_program.py::test_incorrect_no_letters FAILED                        [ 75%]\n' +
        'test_program.py::test_fail_all_guesses PASSED                            [100%]\n' +
        '\n' +
        '=================================== FAILURES ===================================\n' +
        '____________________________ test_correct_first_try ____________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933d629d0>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b9e490>\n' +
        '\n' +
        '    def test_correct_first_try(monkeypatch, capsys):\n' +
        '        # Provide the correct word immediately to trigger the success branch.\n' +
        '        run_program_with(monkeypatch, iter(["puy"]))\n' +
        '    \n' +
        '>       assert capsys.readouterr().out == f"Correct! The word was puy\\n"\n' +
        "E       AssertionError: assert 'Correct! The word was puyp\\n' == 'Correct! The word was puy\\n'\n" +
        'E         \n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:21: AssertionError\n' +
        '___________________________ test_incorrect_first_try ___________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933b80a90>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b82a10>\n' +
        '\n' +
        '    def test_incorrect_first_try(monkeypatch, capsys):\n' +
        '        run_program_with(monkeypatch, iter(["p", "puy"]))\n' +
        '    \n' +
        '        output = capsys.readouterr().out\n' +
        '    \n' +
        '        expected = (f"p\\n"\n' +
        '        f"Guesses remaining: 4\\n"\n' +
        '        f"Correct! The word was puy\\n")\n' +
        '    \n' +
        '>       assert output == expected, "make sure to handle failure cases properly"\n' +
        'E       AssertionError: make sure to handle failure cases properly\n' +
        "E       assert 'p\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'p\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        'E         \n' +
        'E           p\n' +
        'E           Guesses remaining: 4\n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:32: AssertionError\n' +
        '__________________________ test_incorrect_no_letters ___________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933b82590>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b83f10>\n' +
        '\n' +
        '    def test_incorrect_no_letters(monkeypatch, capsys):\n' +
        '        run_program_with(monkeypatch, iter(["a", "puy"]))\n' +
        '        output = capsys.readouterr().out\n' +
        '    \n' +
        '>       assert output == (f"None of the letters of your word was correct\\n"\n' +
        '        f"\\n"\n' +
        '        f"Guesses remaining: 4\\n"\n' +
        '        f"Correct! The word was puy\\n")\n' +
        "E       AssertionError: assert 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        'E         \n' +
        'E           None of the letters of your word was correct\n' +
        'E           \n' +
        'E           Guesses remaining: 4\n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:38: AssertionError\n' +
        '=========================== short test summary info ============================\n' +
        "FAILED test_program.py::test_correct_first_try - AssertionError: assert 'Correct! The word was puyp\\n' == 'Correct! The word was puy\\n'\n" +
        '  \n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        'FAILED test_program.py::test_incorrect_first_try - AssertionError: make sure to handle failure cases properly\n' +
        "assert 'p\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'p\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        '  \n' +
        '    p\n' +
        '    Guesses remaining: 4\n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        "FAILED test_program.py::test_incorrect_no_letters - AssertionError: assert 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        '  \n' +
        '    None of the letters of your word was correct\n' +
        '    \n' +
        '    Guesses remaining: 4\n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        '========================= 3 failed, 1 passed in 0.03s ==========================\n'
    },
    {
      test_case: 'test_incorrect_first_try',
      expected: '',
      received: '',
      error_message: 'make sure to handle failure cases properly\n' +
        "assert 'p\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'p\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        '  \n' +
        '    p\n' +
        '    Guesses remaining: 4\n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +',
      rawout: '============================= test session starts ==============================\n' +
        'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0 -- /usr/bin/python3\n' +
        'cachedir: .pytest_cache\n' +
        'rootdir: /usr/src/app/temp/9c3805ed-3060-4868-a332-4c4e0fa580f2\n' +
        'collecting ... collected 4 items\n' +
        '\n' +
        'test_program.py::test_correct_first_try FAILED                           [ 25%]\n' +
        'test_program.py::test_incorrect_first_try FAILED                         [ 50%]\n' +
        'test_program.py::test_incorrect_no_letters FAILED                        [ 75%]\n' +
        'test_program.py::test_fail_all_guesses PASSED                            [100%]\n' +
        '\n' +
        '=================================== FAILURES ===================================\n' +
        '____________________________ test_correct_first_try ____________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933d629d0>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b9e490>\n' +
        '\n' +
        '    def test_correct_first_try(monkeypatch, capsys):\n' +
        '        # Provide the correct word immediately to trigger the success branch.\n' +
        '        run_program_with(monkeypatch, iter(["puy"]))\n' +
        '    \n' +
        '>       assert capsys.readouterr().out == f"Correct! The word was puy\\n"\n' +
        "E       AssertionError: assert 'Correct! The word was puyp\\n' == 'Correct! The word was puy\\n'\n" +
        'E         \n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:21: AssertionError\n' +
        '___________________________ test_incorrect_first_try ___________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933b80a90>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b82a10>\n' +
        '\n' +
        '    def test_incorrect_first_try(monkeypatch, capsys):\n' +
        '        run_program_with(monkeypatch, iter(["p", "puy"]))\n' +
        '    \n' +
        '        output = capsys.readouterr().out\n' +
        '    \n' +
        '        expected = (f"p\\n"\n' +
        '        f"Guesses remaining: 4\\n"\n' +
        '        f"Correct! The word was puy\\n")\n' +
        '    \n' +
        '>       assert output == expected, "make sure to handle failure cases properly"\n' +
        'E       AssertionError: make sure to handle failure cases properly\n' +
        "E       assert 'p\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'p\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        'E         \n' +
        'E           p\n' +
        'E           Guesses remaining: 4\n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:32: AssertionError\n' +
        '__________________________ test_incorrect_no_letters ___________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933b82590>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b83f10>\n' +
        '\n' +
        '    def test_incorrect_no_letters(monkeypatch, capsys):\n' +
        '        run_program_with(monkeypatch, iter(["a", "puy"]))\n' +
        '        output = capsys.readouterr().out\n' +
        '    \n' +
        '>       assert output == (f"None of the letters of your word was correct\\n"\n' +
        '        f"\\n"\n' +
        '        f"Guesses remaining: 4\\n"\n' +
        '        f"Correct! The word was puy\\n")\n' +
        "E       AssertionError: assert 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        'E         \n' +
        'E           None of the letters of your word was correct\n' +
        'E           \n' +
        'E           Guesses remaining: 4\n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:38: AssertionError\n' +
        '=========================== short test summary info ============================\n' +
        "FAILED test_program.py::test_correct_first_try - AssertionError: assert 'Correct! The word was puyp\\n' == 'Correct! The word was puy\\n'\n" +
        '  \n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        'FAILED test_program.py::test_incorrect_first_try - AssertionError: make sure to handle failure cases properly\n' +
        "assert 'p\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'p\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        '  \n' +
        '    p\n' +
        '    Guesses remaining: 4\n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        "FAILED test_program.py::test_incorrect_no_letters - AssertionError: assert 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        '  \n' +
        '    None of the letters of your word was correct\n' +
        '    \n' +
        '    Guesses remaining: 4\n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        '========================= 3 failed, 1 passed in 0.03s ==========================\n'
    },
    {
      test_case: 'test_incorrect_no_letters',
      expected: "'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'",
      received: "'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n'",
      error_message: "AssertionError: assert 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        '  \n' +
        '    None of the letters of your word was correct\n' +
        '    \n' +
        '    Guesses remaining: 4\n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +',
      rawout: '============================= test session starts ==============================\n' +
        'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0 -- /usr/bin/python3\n' +
        'cachedir: .pytest_cache\n' +
        'rootdir: /usr/src/app/temp/9c3805ed-3060-4868-a332-4c4e0fa580f2\n' +
        'collecting ... collected 4 items\n' +
        '\n' +
        'test_program.py::test_correct_first_try FAILED                           [ 25%]\n' +
        'test_program.py::test_incorrect_first_try FAILED                         [ 50%]\n' +
        'test_program.py::test_incorrect_no_letters FAILED                        [ 75%]\n' +
        'test_program.py::test_fail_all_guesses PASSED                            [100%]\n' +
        '\n' +
        '=================================== FAILURES ===================================\n' +
        '____________________________ test_correct_first_try ____________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933d629d0>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b9e490>\n' +
        '\n' +
        '    def test_correct_first_try(monkeypatch, capsys):\n' +
        '        # Provide the correct word immediately to trigger the success branch.\n' +
        '        run_program_with(monkeypatch, iter(["puy"]))\n' +
        '    \n' +
        '>       assert capsys.readouterr().out == f"Correct! The word was puy\\n"\n' +
        "E       AssertionError: assert 'Correct! The word was puyp\\n' == 'Correct! The word was puy\\n'\n" +
        'E         \n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:21: AssertionError\n' +
        '___________________________ test_incorrect_first_try ___________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933b80a90>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b82a10>\n' +
        '\n' +
        '    def test_incorrect_first_try(monkeypatch, capsys):\n' +
        '        run_program_with(monkeypatch, iter(["p", "puy"]))\n' +
        '    \n' +
        '        output = capsys.readouterr().out\n' +
        '    \n' +
        '        expected = (f"p\\n"\n' +
        '        f"Guesses remaining: 4\\n"\n' +
        '        f"Correct! The word was puy\\n")\n' +
        '    \n' +
        '>       assert output == expected, "make sure to handle failure cases properly"\n' +
        'E       AssertionError: make sure to handle failure cases properly\n' +
        "E       assert 'p\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'p\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        'E         \n' +
        'E           p\n' +
        'E           Guesses remaining: 4\n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:32: AssertionError\n' +
        '__________________________ test_incorrect_no_letters ___________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5933b82590>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7f5933b83f10>\n' +
        '\n' +
        '    def test_incorrect_no_letters(monkeypatch, capsys):\n' +
        '        run_program_with(monkeypatch, iter(["a", "puy"]))\n' +
        '        output = capsys.readouterr().out\n' +
        '    \n' +
        '>       assert output == (f"None of the letters of your word was correct\\n"\n' +
        '        f"\\n"\n' +
        '        f"Guesses remaining: 4\\n"\n' +
        '        f"Correct! The word was puy\\n")\n' +
        "E       AssertionError: assert 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        'E         \n' +
        'E           None of the letters of your word was correct\n' +
        'E           \n' +
        'E           Guesses remaining: 4\n' +
        'E         - Correct! The word was puy\n' +
        'E         + Correct! The word was puyp\n' +
        'E         ?                          +\n' +
        '\n' +
        'test_program.py:38: AssertionError\n' +
        '=========================== short test summary info ============================\n' +
        "FAILED test_program.py::test_correct_first_try - AssertionError: assert 'Correct! The word was puyp\\n' == 'Correct! The word was puy\\n'\n" +
        '  \n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        'FAILED test_program.py::test_incorrect_first_try - AssertionError: make sure to handle failure cases properly\n' +
        "assert 'p\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'p\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        '  \n' +
        '    p\n' +
        '    Guesses remaining: 4\n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        "FAILED test_program.py::test_incorrect_no_letters - AssertionError: assert 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puyp\\n' == 'None of the letters of your word was correct\\n\\nGuesses remaining: 4\\nCorrect! The word was puy\\n'\n" +
        '  \n' +
        '    None of the letters of your word was correct\n' +
        '    \n' +
        '    Guesses remaining: 4\n' +
        '  - Correct! The word was puy\n' +
        '  + Correct! The word was puyp\n' +
        '  ?                          +\n' +
        '========================= 3 failed, 1 passed in 0.03s ==========================\n'
    }
  ],
  compilation_error: '',
  runtime_error: '',
  execution_time_exceeded: false,
  memory_exceeded: false
}

[kbuffardi]

@DAC098 I think a lengthy raw out might be unavoidable and kind of serves the purpose in the interface where Output has the really detailed information (in cases where more info might be necessary) while the Outcome has a more concise result. In the outcome, either error_message: or a combination of the test_case expected and received would get the job done... and are a vast improvement on the truncated messages we've been getting.

When custom assertion messages are used, are they displaying (in full) in the outcome? If so, that'd be perfect because that'd give us the power to choose custom messages that make the most sense for that particular exercise / test case!