Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: freeze set during ticks iter in async hub #1830

Merged
merged 2 commits into from Dec 19, 2023

Conversation

claudinoac
Copy link
Contributor

Relates to #1774

With redis broker, when the redis server reaches timeout and celery worker is running a long cpu-intensive task, the transport tick automatically disconnects, and that's expected.

The connection error is thrown by the redis-py lib, but before that happens, the redis transport layer is auto-disconnected.
Instead of propagating the ConnectionError, the async hub gets its ticks set changed in the middle of the for loop that executes its callbacks, resulting in an error such as RuntimeError: Set changed size during iteration.
This happens because one of these callbacks belongs to the transport layer itself.

This solution basically freezes those ticks sets so they keep their consistency until the ConnectionError is thrown, which allows the celery worker to automatically recover itself, rather than just teardown due to an unrecoverable error like that one above.

Copy link
Member

@Nusnus Nusnus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you and sorry for the delay!

@Nusnus Nusnus merged commit 8aff4e7 into celery:main Dec 19, 2023
17 checks passed
@auvipy auvipy added this to the 5.3.x milestone Dec 19, 2023
@claudinoac claudinoac deleted the fix-async-hub branch December 19, 2023 23:03
@claudinoac
Copy link
Contributor Author

@Nusnus Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants