Call record_workflow_creation on every run

[annagiroti]

Claude summarized that restarting a Toil workflow from a different container crashes at exit with a FOREIGN KEY constraint failed - the new container's local history database never recorded the workflow's creation, since that only happened on first attempts, not restarts.

Make record_workflow_creation idempotent, and call it on every run (not just first attempts) so a restart backfills the missing row before the workflow finishes.

We thought through whether this would cause trouble for Dockstore and decided it would not since Dockstore deals in attempts and not entire workflows.

Address #5544

Changelog Entry

To be copied to the draft changelog by merger:

• Toil no longer crashes when restarting a Toil workflow from a different machine

Reviewer Checklist

• Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • If there is no associated issue, create one.
• Read through the code changes. Make sure that it doesn't have:
  • Addition of trailing whitespace.
  • New variable or member names in camelCase that want to be in snake_case.
  • New functions without type hints.
  • New functions or classes without informative docstrings.
  • Changes to semantics not reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • New features without tests.
• Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• Finish the review with an overall description of your opinion.

Merger Checklist

• Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
• Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
• Merge with the Github "Squash and merge" feature.
  • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• Copy its recommended changelog entry to the Draft Changelog.
• Append the issue number in parentheses to the changelog entry.

[adamnovak]

This looks OK to me.

[adamnovak]

This is odd because the job store really shouldn't move like that.

[adamnovak]

I guess it helps us see which call actually created the database entry, for the test.

[adamnovak]

It looks like this is breaking some of the existing tests for this system:

SUBFAILED[Testing input test/cwl/whalesay.json for status 0] src/toil/test/cwl/cwlTest.py::TestCWLWorkflow::test_dockstore - AssertionError: Expected 'record_workflow_metadata' to be called once. Called 0 times.
SUBFAILED[Testing input test/cwl/empty.json for status 1] src/toil/test/cwl/cwlTest.py::TestCWLWorkflow::test_dockstore - AssertionError: Expected 'record_workflow_metadata' to be called once. Called 0 times.
FAILED src/toil/test/cwl/cwlTest.py::TestCWLWorkflow::test_dockstore - contains 2 failed subtests

[adamnovak]

I think merging this might have caused the CI failures we are seeing in mainline Toil in https://ucsc-ci.com/databiosphere/toil/-/pipelines/12468

Two different jobs failed with:

FAILED src/toil/test/lib/test_history.py::TestHistory::test_history_attempt_after_restart_backfill - sqlite3.OperationalError: table workflows already exists

I think the test maybe didn't work to start with but wasn't actually run in CI for this PR, because when I run the test with:

make test cov='' tests='src/toil/test/lib/test_history.py::TestHistory::test_history_attempt_after_restart_backfill'

I am able to see the same failure.

I think we're not rolling back the transaction that produced the IntegrityError before trying to do more things with the caching database.