Celery integration: Add task_id to task_failed log - #1043
Open
ento wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is helpful when a worker crashes and the
task_failurereceiver gets called with aWorkerLostErrorexception. Because the receiver gets called on the master Celery process, context vars that were bound by previous receivers are not there, and there's barely any information that helps quickly pinpoint which task the worker was processing.Example event without
task_id:{ "event": "task_failed", "level": "error", "logger": "django_structlog.celery.receivers", "error": "Worker exited prematurely: signal 9 (SIGKILL) Job: 1012.", "exception": [ { "exc_type": "WorkerLostError", "exc_value": "Worker exited prematurely: signal 9 (SIGKILL) Job: 1012.", "frames": [ { "filename": "[...]/site-packages/billiard/pool.py", "line": "", "lineno": 1265, "locals": { "exitcode": "-9", "job": "<%s: 1012 ack:True ready:True>", "self": "<celery.concurrency.asynpool.AsynPool object at 0xffff677a24a0>" }, "name": "mark_as_worker_lost" } ], "is_cause": false } ] }The line in Celery that sends the
task_failuresignal when a worker is lost: https://github.com/celery/celery/blob/v5.6.3/celery/worker/request.py#L645Confirmed that the
task_idin this case matches the one included in thetask_startedevent using a local project: