fix: freeze set during ticks iter in async hub #1830
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relates to #1774
With redis broker, when the redis server reaches timeout and celery worker is running a long cpu-intensive task, the transport tick automatically disconnects, and that's expected.
The connection error is thrown by the
redis-py
lib, but before that happens, the redis transport layer is auto-disconnected.Instead of propagating the
ConnectionError
, the async hub gets its ticks set changed in the middle of the for loop that executes its callbacks, resulting in an error such asRuntimeError: Set changed size during iteration
.This happens because one of these callbacks belongs to the transport layer itself.
This solution basically freezes those ticks sets so they keep their consistency until the
ConnectionError
is thrown, which allows the celery worker to automatically recover itself, rather than just teardown due to an unrecoverable error like that one above.