[Bug]: Redis Connections Leak, especially with queue api calls. #1065
Assignees
Labels
bug
Something isn't working
Comments
ikreymer
added a commit
that referenced
this issue
Aug 13, 2023
- use contextmanager for accessing redis to ensure redis.close() is always called - add get_redis_client() to k8sapi to ensure unified place to get redis client - use connectionpool.from_url() until redis 5.0.0 is released to ensure auto close and single client settings are applied - also: catch invalid regex passed to re.compile() in queue regex check, return 400 instead of 500 for invalid regex - redis requirements: bump to 5.0.0rc2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Browsertrix Cloud Version
v1.6.0
What did you expect to happen? What happened instead?
It appears that number of redis connections keeps growing as the crawl is running!
It appears that we are missing redis.close() calls for most operator and queue API calls.
Also, due to an issue with
aioredis.from_url, connections are not being auto-closed even with redis.close().Looks like this will be fixed in upcoming redis 5.0.0 release, but should be possible to add a workaround. See redis/redis-py#2831.
Step-by-step reproduction instructions
redis-cliinfo clientsconnected_clientsnumber going upAdditional details
No response
The text was updated successfully, but these errors were encountered: